From Manual to Magic: File Ingestion Pipeline with Anaplan

Background

This is a multinational mining company based in Canada as part of an Anaplan implementation required some work to support their ingestion and transformation process by leveraging their new Databricks environment and loading Excel files (stored in SharePoint for their multiple mining sites), transformed into Anaplan for consumption.

Challenge

The challenge here is Local Excel files that only have a manual way of uploading information to Anaplan, causing a lack of insight into site data for the analytics team.

Approach

Keyrus implemented a synchronized S3 bucket with SharePoint to directly access the files from the S3 bucket in real time. The team developed several Databricks notebooks to load the files, triggered whenever they are updated, into a medallion architecture in the unity catalog. The Bronze layer is used to store the raw data and the Silver to load the transformed data. The transformations were different for each category and were made using Spark. Once the data was transformed, we had to upload it in CSV files to S3 to load them into Anaplan. The loading process was made with Anaplan “CloudWorks” by choosing the right file in the s3 and the right action (process). This Silver layer data was also made available for the analytics team providing site insights they never had to use across other business use cases like ESG Reporting.

Key results

Benefits

Robust pipeline delivering required information with actions to process it with Anaplan. Access to data that was not available or being collected from sites previously.

Share this key play