In most projects, we leverage .yxdb files stored on the disk on the Alteryx server. But when a project calls for very granular reporting on large datasets, Calgary files can be used to speed up performance, allow for more granular reporting, and leverage macros in Alteryx, which broaden reporting capabilities.
Rather than reading large data into your workflow and then filtering it out, Calgary files allow you to only read the relevant records of data.
Use Case
Our client had weekly sets of sales data, each containing hundreds of millions of rows from thousands of stores across the timeframe of two years. Their existing ecosystem was in Alteryx gallery but transfer over the internet or intranet remained a huge bottleneck in the company’s workflows. They were handling 1B+ rows of data!
Because of their reporting requirements, there were no opportunities to aggregate data in .yxdb format since the company wanted to slice it by every dimension. In flat-file format, these reports would take a couple of hours to complete. Using Calgary files, the same reports only take thirty seconds to a minute to complete.
Creating a Calgary file
Prior to the reporting layer, we implemented some pre-processing workflows to create these Calgary databases. These Calgary database files are fed into the workflow using the Calgary Join tool.
This tool takes an input file and performs joins against a Calgary database where an input record matches a Calgary database record based on user-defined join criteria.
We then used an extract macro to pass in filters and only scan data that are required, rather than reading in all data and then filtering down in workflow.
Calgary macro has its own filters that can be controlled by user selection. Being able to control the query streamlines the process. The workflow runs faster, as the macro gets rid of extraneous fields and allows end-users to focus only on necessary fields.
Reporting capabilities
Reporting with Calgary files is comprehensive and intuitive. Unlimited filtering capabilities and historical data access are available reporting capabilities.
These capabilities when combined with reporting, geo-spatial, or other Alteryx components can provide powerful analytic functions on large data volumes much faster than traditional workflows.
Have a similar project? Chat with us today!