What is Data Mesh Architecture? When Do You Need To Start Implementing Data Mesh Architecture?

Data mesh is still a relatively new concept, and there are not yet many well-defined patterns of best practices. However, some companies already use data mesh to build scalable and reliable data architecture. One of the underlying principles is “Domain Data Products.” This principle means that data producers own the data and treat it as a product.

An example will look like this:

There is an HR department within a company, and the department has a recruitment application, benefits platform, and payroll application. Following the data mesh approach, all data generated by the above apps in the HR stack should be owned and governed by the HR team. This guide on data ownership is how all departments should treat the data in the rest of the organization. 

In this data mesh configuration, the data engineering and analytics team exists as separate domains of aggregated data. The data aggregates team will collect generated data in a single storge for data modeling. For instance, the data analyst team will aggregate data from the marketing and HR domains to understand what marketing managers meet their KPIs and are entitled to bonuses. Then they will make it accessible across domains back to the HR department. 

The major shift happens in interaction with data generated by the engineering team. In this case, the data engineer’s role is to build and support the data infrastructure needed for the production team, as a domain team, to produce and share their data. In this case, data engineers set up a data lake and a metadata layer–for data governance and cataloging of what data sets are stored. From that point, domain teams produce and put their data sets into the data lake storage. The metadata layer includes settings–like access controls and contracts–to help integrate and deliver data correctly between domains. Then, the data analyst team is responsible for building reports and the data processing layer. This freedom after the report means it is up to users how they want to analyze the data. It also means that the engineering team is there to manage the infrastructure but not get involved in the data. This process results in decentralized data storage, centralized data infrastructure, decentralized data ownership, and centralized data governance enforced through a metadata layer. 

Implementing data mesh requires many resources, high data literacy within an organization, and the commitment of every employee. It is hard to imagine that an employee in a larger company would be responsible for generating fields and automatically loading them into systems—like Google Data Storage–because of the complexity of the field and the stigma of technological compartmentalization. 

We can argue that the data mesh concept may not be implementable, however, organizations can still leverage some incremental ideas and concepts to improve data quality and reliability.  Data mesh is a complex, multi-layer concept that most companies will struggle to implement, due to the cost of its implementation and understanding of the return of investment. Embracing a few principles seems like baby steps toward decentralization, however, it can help promote comprehensive data ownership within an organization, where each department treats the data as a product. The initial step could be implementing data contracts between the data engineering and production teams. An example is a “Protobuf Contract,“ which helps enforce the data schema of data delivered to a centralized storage.