Ways to load data in DW from External Data Source

Introduction After performing transformation on data, then it needs to save it in Data Warehouse for further analysis. This process comes under ETL (Extract, Transform, Load). It involves extracting data from various sources, transforming it as per business need, and...

July 21, 2024

News

MapReduce Vs Tez

Apache Hadoop uses MapReduce as it’s programming model for distributed processing of Big Data, but instead of writing multiple MapReduce jobs, we can also utilize the power of Hive or Pig. Hive: Hive gives an SQL-like interface to query data...

July 7, 2024

Top 10 Common Data Engineers and Scientists Pain Points in 2024

Development News

Top 10 Common Data Engineers and Scientists Pain Points in 2024

As we navigate through 2024, the landscape of data engineering and science continues to evolve at a breakneck pace. With advancements in AI technology come new challenges, and professionals in these fields are grappling with a unique set of challenges....

April 11, 2024

How to Convert Dates in One Group into An Interval

Development News

How to Convert Dates in One Group into An Interval

We have a database table TBLTEST. Below is part of its data: The data is ordered by date. We are trying to group rows by the first five columns, convert dates in each group into an interval, and record the...

March 28, 2024

“Data has a Dream” — A Short comic about data mesh and how it can transform your company

Development News

“Data has a Dream” — A Short comic about data mesh and how it can transform your company

a little data story by Markus Schüler (Director of Data Strategy Adevinta) with drawings by Gitanjali Venkatraman (Technology Writer and Illustrator at ThoughtWorks) The profound impact of Data Mesh and its associated principles domain data ownership data as a product...

March 19, 2024

How to Transpose Columns in Each Group to a Single Row

Development News

How to Transpose Columns in Each Group to a Single Row

We have a database table STAKEHOLDER as follows: We are trying to group the table by CLASS and convert all columns to a same row. Below is the desired result set: SQL code written in Oracle: WITH CTE AS( SELECT UP.CLASS, UP.NS...

March 18, 2024

Different file formats, a benchmark doing basic operations

Development News

Different file formats, a benchmark doing basic operations

Recently, I’ve been designing a data lake to store different types of data from various sources, catering to diverse demands across different areas and levels. To determine the best file type for storing this data, I compiled points of interest,...

March 11, 2024

About job offers, innovation & data strategy

Development News

About job offers, innovation & data strategy

Shift-left thoughts In IT, we usually think about Shift-left for our stack: “Shift left is the practice of moving testing, quality, and performance evaluation early in the development process, often before any code is written.” And what if I told...

March 5, 2024

Xavier's Insight: Overcoming Data Hoarding Disorder

Development News

Xavier's Insight: Overcoming Data Hoarding Disorder

Intro: In our digital age, the art of capturing and storing data in the vast expanse of the cloud has become second nature. It’s a reflex born out of the fear of missing out (FOMO) on a valuable piece of...

March 4, 2024

Development News

Extracting data with dlt

Last week we learned about dlt (data load tool) in a workshop conducted by Adrian Brudaru and hosted by Data Talks Club. dlt can extract data from APIs in a way that’s scalable. It can normalize and load data, and...

February 15, 2024

1 2