Background
Our client is a regional transmission organization (RTO) that coordinates the movement of wholesale electricity across several states. To comply with federal regulations, our client is required to post real-time and historical data on its public website. The organization sought to enhance its data governance capabilities to provide a better experience for end users and remain compliant with all regulations.
Challenge
Prior to this initiative, our client had multiple legacy systems and faced data quality challenges, which were impacting the posting time and reliability of data. Because these legacy systems were not centralized, the data postings were inconsistent and there was no standard method of managing reference data.
The client data was vast and provided business-critical information related to various aspects of the RTO’s operations. Each dataset involved different source systems, loading cadences, quality checks, and other requirements.
This initiative required cleaning and processing bid and offer data, load forecasts, settlements, and sensor data from a Plant Information (PI) Management System. The client’s largest dataset showed day-ahead and real-time Locational Marginal Pricing (LMP) across a power grid for each node where electricity could be purchased across the entire territory.
Solution
After performing a discovery exercise, Keyrus determined that the goal of the project was to create a system that allowed end users to access and understand current and historical data for 55 datasets from the past thirty years to satisfy all regulatory requirements.
Using Alteryx, the Keyrus team built custom workflows to first extract and transform data. To prove the reliability of these new workflows, each dataset completed an extensive QA process that was conducted automatically and programmatically using Quilliup prior to being published on the client’s website. The tool also logged data and created audit trails to ensure that data remained reliable and accessible to meet federal regulatory requirements. Finally, the verified outputs were loaded into an Azure Data Lake and posted to the public website.
In addition to the technical implementation, the Keyrus team also developed extensive documentation and provided training resources to the client, to ensure that the operational processes worked smoothly.
Results
After the implementation of this process, our client can be confident that the requisite data is being posted to their public website in a timely fashion so that they remain compliant with federal regulations.
The solution created reliable, uninterrupted service, and an easy-to-maintain hub. In just six months, Keyrus delivered over 50 different workflows to help the client serve end users.