The Data Mesh way of Data Governance: Focus on Federation and Data Constitution
In today’s dynamic business environment, the traditional centralized data governance model(Fig1), characterized by a hierarchical structure with a central team overseeing all data management and decision-making, is no longer sufficient for large organizations. As the number of business units grows, the complexity of the technology stack increases, and the volume and diversity of data proliferates, this centralized approach leads to inefficiencies, communication gaps, and a lack of agility. Implementing data management policies that ensure compliance becomes increasingly challenging given the numerous touchpoints throughout the data lifecycle.
To address this issue, Data Mesh proposes a federated approach to data governance(Fig2), allowing governance to be managed by domain teams. In a federated data mesh, data is governed centrally but accessed and managed locally. This approach combines the benefits of both centralization and decentralization, enabling organizations to strike a balance between control and agility.
Within the context of embracing a federated approach to data governance, it is imperative to consider the establishment of a Center of Excellence (CoE). This entity will serve as the primary source for reusable templates, organizational key performance indicators (KPIs), and baselines necessary for successful implementation. Additionally, the CoE should initiate the process of defining a comprehensive document that clearly outlines the fundamental principles, policies, and standards pertaining to data governance across the organization. The purpose of this document is to establish a framework for consistent and effective data management practices throughout the organization. We will refer to this document as the “Data Constitution”.
The Data Mesh — A Nation-building Story
The Data Mesh architecture can be conceptualized as an analogy to the evolutionary processes of nations and governments. Imagine a recently discovered country, where settlement has just begun. Initially, the small population allows for smooth functioning without the need for extensive management. However, as the population grows, the habitable land must be expanded, improved systems must be established, and some form of governance and consistency in management is required. A centralized government may be established to address these needs. As the population continues to grow exponentially, the centralized system becomes inadequate, necessitating the decentralization into states, each with its own governance system. To ensure harmony and effective management, a constitution is essential, allowing each state to retain autonomy while adhering to the central constitution. Without central control and a constitution, individual states can create chaos. The Data Mesh architecture follows the same principle, enabling the scaling of data-driven organizations. Therefore, when considering the Data Mesh architecture, it is imperative to think along these lines. Domains are analogous to states with autonomy, while the central team retains certain controls and responsibilities to standardize and facilitate each domain. When adopting the Data Mesh architecture, it is crucial to clearly define the DATA CONSTITUTION.
Summary
Federated Data Governance on a self service domain-driven platform with a well defined data constitution is an ideal solution for organizations seeking to scale and maintain data management practices. As an example, refer to this article which talks about setting up a federated Data Quality framework.