The data mesh concept is on its way to replacing the central data lake as the predominant architecture paradigm in the analytics sector. Because it enables even large volumes of data in companies to be structured in a practical manner and made effectively usable in self-service. A current white paper shows the contexts and possibilities.
Creating structures instead of boundaries
In order to democratize data management and use in the enterprise, the data lake concept has long been considered the more scalable successor to the data warehouse approach. However, the great expectations placed on data lakes have only been partially fulfilled. Initially, the approach of bringing together data from multiple sources and formats to generate new insights proved to be quite promising.
However, collected without an organizing principle, this data eludes use as volumes and complexity increase. And the data lake tends to turn into a swamp that can no longer be managed centrally, but which no one wants to shut down either. On the contrary, (too) many companies continue to invest in centralized data lakes in the hope of democratizing data on a large scale and making it usable for analytics.
Data products for self-service analytics
The Data Mesh concept remedies this situation! It marks the fundamental paradigm shift from a centralized to a decentralized approach and enables large data volumes to be structured and made usable in a practical way as a “meshed” network aka. the data mesh.
The resulting solution lies in a distributed, domain-oriented data architecture that makes data available for convenient self-service analyses by the various departments. What is new and groundbreaking is that the data is treated as “products” that can be developed and maintained holistically, across all departments, and can be networked and combined as desired.
So instead of data being managed centrally, or no longer being able to do so, it is oriented around domains and owned by independent and cross-functional teams that can deliver ready-to-use data products with immediate and best value.
Transformed structures and new opportunities
Quite possibly the biggest effect of the data mesh concept is the change in team structures and new opportunities for collaboration: instead of highly specialized central engineering teams, there will be cross-functional teams with domain knowledge.
This is accompanied by a cultural, organizationally driven change. Instead of rigid departments, there will be flexible responsibilities and everything will be domain-oriented.
This new way of thinking about data and acting on data can be an impetus for companies to deal much more creatively with their own data in the future. The result can be new digital services and business models that significantly improve transparency, efficiency and value creation.
In a recent whitepaper, Georg Roesch, Senior Data Engineer and data mesh pioneer at The unbelievable Machine Company (*um), explains the limitations of data lakes or centrally controlled platforms, what a data mesh is, and how it can be applied in a meaningful way.
This post is also available in: German