Background
Our client works in the automotive industry. They get a lot of data sources from the Benelux region that come in different formats, times and origins. To get useful data insights, these sources need to be combined into a data warehouse.
Challenge
Our client's infrastructure was outdated and not efficient as per the current standards. It was difficult, tedious, and prone to mistakes to add or handle hundreds of ingestions. By making the data ingestion flow metadata-driven and automated, it becomes simpler and more informative to monitor and manage the flow.
Approach
Detail the specifications for each data source: Column names and their data types, dimension types of the data source, ingestion and storage requirements, etc. Design a flexible data ingestion process that gathers the different data sources from a source location and loads them into the DWH. Create a foundational structure to manage the flow of the data sources.
Key results
Benefits
The automated data ingestion flow sets up a solid data warehouse ingestion infrastructure. This enables more accurate and more insightful queries of the existing data. The flow also makes it easier to add new data sources, which expands the possibilities for data analysis.
Share this key play
