Introduction
This title will sound provocative to many of you. Why on earth would someone want to build a Master Data Management solution when there are so many off-the-shelf products that can do this for you? Why not simply installing a turnkey software from a market leader such as Informatica, Stibo, Semarchy, SAP, or IBM just to name a few?
That would certainly save costs, speed up the time-to-market, and provide a much more robust and stable solution than a homemade application.
Are we so presumptuous as to consider that we can do better? And if not, why then reinvent the wheel? In this series, we will explore this question through three distinct articles, each shedding light on a different facet of the decision to build a custom Master Data Management solution.
The context
Internal
We ran this project for a global construction company present on every continent. Through its strategy of growing by acquisition, many of the company’s local subsidiaries retained their own ERP and production systems.
Needless to say that negotiating better deals with suppliers or sharing up to date information about customers across the company is rather challenging in such an environment!
Another limitation was the lack of integration across systems even within the same legal entity. Next to the ERP which is essentially used by accounting and cost control, additional stand-alone tools are being used to cater for each department’s specific processes. There is a tool for recruitment, another for people management, one (actually many) for tendering, some more for document management, etc…
In this ecosystem, finding its way becomes an art in itself!
External
The economical context at the time when the MDM initiative started was quite unfavorable. In the midst of the Covid crisis, the company faced the financial impact of lockdown and was forced to take drastic cost saving measures to keep the boat afloat.
These uncertain times were far from being the ideal period to suggest the funding of an MDM project, especially knowing that the ROI of such a solution is difficult to estimate and its positive effects on the business not always immediately tangible.
The need
What was obvious though is the reality of the need for an MDM solution.
So many hours were wasted by associates in numerous departments, to compile Excel reports where extractions from various sources were aggregated and cleansed, month after month.
It could take literally hours to find information on a running construction project, just because its name is different in every system.
Information was scattered around, sometimes complementary, eventually conflicting, seldom aligned.
In order to provide easy and accurate corporate reporting, it was crucial to combine all of the data generated by the company activity into a single harmonized view, for each of core business entities, which were in our case:
clients
legal entities (subsidiaries)
(construction) projects
resources (employees, contractors)
vendors
As the business was already suffering from the lack of integration, the aim was ideally to deliver value on a fairly short term.
The decision
BYO
The first question that came up was related to the strategy.
Not doing anything was definitely not an option. Should we then buy a solution off the shelf to address the needs, or develop our own custom solution?
A series of criteria were analyzed to compare and weight the opportunity of going one way or the other. 2 of them were considered as key decision drivers:
cost
speed of delivery
Home-built | Off the shelf | |
|---|---|---|
Cost | + Lower software costs - Higher implementation costs as more analysis, design, testing and debugging are required | + Implementation generally cheaper thanks to the supplier/partner’s experience with the software - Higher initial cost due to the investment in software licenses |
Speed of delivery | - With an equal amount of time, this solution contains a narrower scope than the off the shelf solution. Initially, there is hardly any business value During the live phase, small changes have a longer implementation time | + Shorter time to market and more scalability provided by turnkey solutions - Specific needs may however take longer or may never be available |
With Covid restrictions knocking at the door, the initial investment required for an off the shelf solution was deemed too expensive by the board, leaving no other option than the BYO model.
The conditions for such a choice to take shape were quite good:
experienced in-house software development team
availability of key functional roles (functional analysts, testers & project manager)
ETL and modeling knowledge
Cloud environment ready to support the initiative
Analytical MDM
The next question that needed answering was the type of the MDM solution. Should it be an Analytical or Operational MDM?
For those who are not yet familiar with these terms, an analytical MDM is primarily focused at improving reporting and analytics by harmonizing source data. It usually sits in between business applications and the data warehouse.
An operational MDM is however an integral part of the operational process. It is used to manage centrally the master data and acts as a unique source for downstream business applications.
For our project, the decision was to go for an analytical MDM rather than operational MDM in order to be the least disrupting to business processes. In the back of our minds, the idea of making MDM a core operational element was still strongly anchored but we would first need to show evidence of how powerful such an application can be, by starting with its positive impact on reporting.
The functionalities
With the frame now set, what finally had to be decided were the deliverables. Which functionalities should the tool provide? What should be the scope?
The plan was to start small by focusing on an initial narrow deliverable:
1 data entity
Projects proved to be the most valuable to the business on short term
1 MDM feature
Matching entities from different sources together to produce a single view
Web application
User interface allowing data stewards to interact with the system
This small scope allows to work agile, providing a fully functional deliverable in a short timeline. Building on this initial implementation, other data entities and MDM features can be progressively added as extensions of the core module.
In theory, this looks great. Dependencies and enthusiasm however surreptitiously forced on additional items to the scope, as we realized that the project entity was connected to legal entities, clients and resources. Delivering the project entity therefore required that we first dealt with 3 of the remaining 4 data entities that composed the full scope!
As we opted for an analytical MDM, existing processes didn’t require a redesign, no integration to existing systems other than the data warehouse were needed, and real-time updates were not mandatory.
That being said, the frontend provided to the data stewards should not only let them consult MDM data but also interact with it by matching data entities together on demand. The output would be stored in a new database and ETL processes were to be foreseen to ingest source system data and apply the actions performed by frontend users.
Technology
There are hundreds of tools out there and as many programming languages, more or less fit for what we wanted to achieve. How could we choose the most appropriate?
What is important when building your own solution is to capitalize on the internal skills and knowledge. Unless these skills refer to a technology no longer supported, or so obscure that it is hardly known outside of your team, don’t try to be too creative and stick to what you know best! Development will be faster and more stable.
For this project, several elements encouraged us to go for an Azure based solution:
The team was already familiar with the Azure cloud environment
The data warehouse was already running in Azure. As mentioned earlier, an analytical MDM is built to support reporting & analysis. It therefore belongs close to the data warehouse
Using a cloud architecture allows to quickly deploy a mockup while limiting upfront costs and yet supporting a future increase of scope
Below is the architecture that we designed:
As we conclude this initial exploration, the question lingers: why choose to forge a unique path in the realm of Master Data Management? Part 2 of our journey takes us deeper into the heart of this decision-making process, focusing on the intricacies of design. Here, we unravel the blueprint that guides the creation of a bespoke MDM solution. Delving into the architecture, customization, and scalability considerations, we aim to shed light on the nuanced artistry that distinguishes a purpose-built solution. Join us in Part 2 as we navigate the captivating landscape of design, where innovation meets necessity, and where the decision to craft a distinctive MDM solution begins to unfold its true potential.
Stay connected for part 2 that appears next week! If you have any business-related queries about the content, do not hesitate to reach out. Questions or concerns? Contact the author, Fabien Arnaud, directly at fabien.arnaud@keyrus.com.