Success story

Modernizing a data platform at a global trade organization

by Norman Chea

Background

A large global trade organization (GTO) has a mission to promote the sciences in order to improve the quality of life for all. The organization focuses on publications and membership, with a membership exceeding 150,000 in more than 140 countries. They publish over 70 peer-reviewed journals that are considered cutting edge in research within their domain. Additionally, they organize national and regional conferences to promote the sciences.

Challenge

Keyrus was brought in to work with the GTO to address very specific meta-data management and data cataloging concerns that their business stakeholders had identified. The existing data platform was a 15+-year-old Oracle system that had supported the business well over the years, but had complex logic that was not documented. 

Whenever there were questions on how KPIs were being added into the system, it was a very time-consuming process to figure out what the existing data transformations were doing. To understand what was done, it would require IT to reverse engineer the layers upon layers of code and then document these findings back to the business. 

This ultimately created bottlenecks for their lean development team and instilled a lack of confidence in some of the metrics, due to the black box logic. 

Approach and Key Success Factors

Discovery and Learning the Business

Keyrus executed a data discovery to assess their entire data platform and provide recommendations on how to resolve their meta-data management needs. This was a 5-week discovery where our team conducted interviews across their key department stakeholders to understand existing data processes, how they use the data, and what capabilities they would like in the future. 

We determined that meta-data management was only part of the challenge, and that a recommendation for a data cataloging tool wouldn’t solve all the core issues. While the 15+-year-old system did its job at meeting existing business requirements, there was a need to modernize the platform to accommodate future requirements. 

Some of the core future requirements that were discovered were:

  • A data platform that can scale for growing data volumes and connections without spending significant overhead on IT time

  • Modernize their BI consumption layer and provide self-service capabilities for business users to generate their own insights

  • Remove the need to provide manual reports and dashboards to external stakeholders

  • Provide a more transparent data lineage capability to identify issues

At the end of the discovery, we presented our findings to the main stakeholders at the GTO and proposed a data platform modernization to address the core challenges that expanded outside of the data lineage challenges.

We provided the GTO leadership team with recommendations on future technology stack options, based on the requirements we gathered from business and IT. To supplement the tool comparison, we also conducted a two-week proof-of-concept to demonstrate the capabilities of handling the GTO’s current and future state requirements. We chose Snowflake, Tableau, and Talend and agreed on a plan to execute the modernization effort.

Data Platform Modernization

At the implementation stage, our team collaborated with their stakeholders to design and build the future state data platform. First, we reverse engineered the 15-year-old system and looked at a large library of their Java classes and Oracle stored procedures to translate complex and layered logic into readable Talend data pipelines. Creating these readable flows was easier to follow and maintain than existing Java logic, and since Talend leverages a graphical user interface, it enables a larger pool of their developers to generate their own data pipelines, as opposed to being reserved for Java developers. 

Lastly, Talend’s ETL tool offers great native connectivity to all of the connections that the GTO would like to leverage in the future, which reduces the time developers would have to spend building a custom connection in the existing solution.

To address their scalability requirement, we leveraged a fully cloud solution, leveraging Amazon Web Services (AWS) to host the Talend ETL engine and Snowflake for their data warehouse. These tools provided a highly elastic solution that would enable the team to scale their resources up and down accordingly. 

Leveraging a cloud-based and managed solution allows the GTO to focus on generating business value as opposed to upkeep and patching on-premise infrastructure. On top of the natural scalability of the tools, Keyrus also leveraged Cloud Formation scripts on the AWS side that helps automate standing up the infrastructure.

To create a modern BI consumption layer, we leveraged Tableau as the enterprise reporting tool. Our first effort on the consumption layer was to factor the redundant reports and dashboards that existed within their legacy BusinessObjects stack. 

Our team worked closely with key stakeholders to consolidate 30 dashboards and reports into six Tableau dashboards. This instantly created a better user experience, so that users can go to one dashboard as opposed to several to get the data sets they are looking for. 

This also reduces the maintenance burden of IT by consolidating these objects. We also collaborated with the GTO’s infrastructure staff to install and deploy a Tableau external server, which allows end users who are outside of their organization’s network to access dashboards and reports. With this external server, their business intelligence team can now deploy an automated dashboard that their external users can access at any time, as opposed to building out manual reports every month.

To tie this all together, we deployed a Talend Data Catalog solution to address the data lineage requirements. This tool provides great native integration with our entire recommended stack (Talend ETL, Snowflake, and Tableau). 

Keyrus configured the data catalog to automatically harvest and read the metadata across these different technologies, which automatically create end-to-end data lineage. This allows business users to see the entire journey of all their data assets - how a data element goes from source to data warehouse to dashboard - and how it gets transformed in the entire data process. This provides great visibility that the GTO stakeholders never had before, allowing them to better validate and trust their data.

 What Made Us Successful

  • Our team leveraged best-in-class technologies and layered them together to provide GTO with a custom solution that met a wide range of business needs. 

  • Through proven technical expertise implementing data platform modernizations, we leveraged accelerators (i.e., Cloud Formation scripts) to reduce development time.

  • Managing different software vendors with different license and pricing structures can be complex. Keyrus operated as a reseller thanks to our relationship with different software partners. This created a one-stop-shop for our customer to get all of their software licenses they needed across different vendors.  

Key Deliverables 

  • Comprehensive assessment of existing infrastructure (ERD diagrams, workflows, interview notes)

  • Vendor technology comparisons for their entire data infrastructure stack (i.e., data visualization and analytics, data warehousing, ETL, and infrastructure), including executing a proof-of-concept

  • Detailed design of the new solution inclusive of new server/network infrastructure, data warehouse structure and models, error handling, and reporting principles

  • Reverse engineered a 15+-year-old system built on legacy Oracle and Java code into a modernized data platform

  • Stood up an external Tableau server to enable the new data platform to be accessed by their end users outside the company’s network

  • Conducted training on the new platform to a wide range of audiences, including a larger Tableau training

whatsapptwitter
linkedinfacebookworkplace
newsletter.svg

Never miss an insight

Stay updated on the latest articles, events, and more

Your email address is only used to send you the Keyrus newsletter and for commercial prospecting purposes. You can use the link in our emails to opt-out at any time. Learn more about the management of your data and your rights.

Continue reading

Blog post

5 Insights Your Organization Can Gain from an ESG Solution

April 22, 2022

Environmental, Social, and Corporate Governance (ESG) disclosures have become increasingly sought after in the last 5 years, with heavy forecasted growth expected in the coming two to three.

Event

Keyrus & Anaplan Sponsor Life Sciences Accounting & Reporting Congress 2022

March 11, 2022

Keyrus & Anaplan Sponsor Life Sciences Accounting & Reporting Congress

Success story

Leverage Workiva and Anaplan to automate dozens of FP&A and demand planning reports

March 3, 2022

How a front-line service provider leveraged Workiva and Anaplan to automate dozens of reports and free up 14 days each month.

Podcast

How to craft effective data quality and MDM strategies

March 1, 2022

In this podcast episode, we'll define data quality and MDM as part of data governance. We’ll explore how to create effective data quality and MDM strategies.

Success story

Improving sales reporting to maximize deals at a fintech company

February 23, 2022

A global financial technology company sought to streamline, automate, and optimize how they collect and report on data to keep a steady pulse on the health of their business. Specifically, their sales team was looking for innovative ways to utilize data to enable their sales reps to maximize deal volumes. 

Blog post

Facing the transit industry's challenges head on with data

January 7, 2022

Welcome to another installment of our Evolving Industries series. Join us as we take a deep dive into changing markets, technologies, and consumer expectations, and learn how leading companies are surviving and thriving in some of the world’s most unpredictable industries. In this edition, we sit down with Jeff Hiott, Vice President of Imagination, Research, and Industry Benchmarking for the Capital Metropolitan Transit Authority (Capital Metro).

Press release

Keyrus wins recognition from Clutch as a Leading 2021 Global Provider of Data Analytics

December 16, 2021

We’re excited to share with you that we’ve been named among the top data analytics service providers globally. According to Clutch’s B2B research findings, we’re one of the global leaders for 2021!

Blog post

Keyrus (officially) recognized as a Great Place to Work

November 30, 2021

Keyrus is proud to announce that we are now a certified Great Place to Work! We already knew that we love working at Keyrus, but as a data and digital consulting firm, we like our assumptions to be backed by data. That’s why we brought in the experts at Great Place to Work to survey our team to see how we’re doing and where we can improve.

Blog post

10 can’t-miss episodes from Tableau Conference 2021

November 15, 2021

The Tableau Conference wrapped up on November 12th and, for the second year in a row, it was a fully virtual event. With 308 episodes and five concurrent channels of content, it’d be impossible to watch all of the sessions quickly. To help you navigate, we’ve compiled a list of the ten most essential sessions that provide excellent range and an overview of the entirety of the three-day event.

Event

Pharma/Biotech GTN Summit 2021

November 5, 2021

Keyrus & Anaplan Sponsor Life Science Gross-to-Net (GTN) Summit