Data mesh
 | This article has multiple issues. Please help improve it or discuss these issues on the talk page. (Learn how and when to remove these template messages)
|
Data mesh is a
data silos, due to the presence of a centralized system that ensures the consistent sharing of fundamental principles across various nodes within the data mesh and allows for the sharing of data across different areas.[6]
History
The term data mesh was first defined by Zhamak Dehghani in 2019 and others.
In 2022, Dehghani left Thoughtworks to found Nextdata Technologies to focus on decentralized data.[16]
Principles
Data mesh is based on four core principles:[17]
- Domain ownership
- Data as a product[18]
- Self-serve data platform
- Federated computational governance
In addition to these principles, Dehghani writes that the data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[19] In other words, the data should be treated as a product that is ready to use and reliable.[20]
Data mesh in practice
After its introduction in 2019[7] multiple companies started to implement a data mesh[12][14][15] and share their experiences. Challenges (C) and best practices (BP) for practitioners, include:
- C1. Federated data governance
- Companies report difficulties to adopt a federated governance structure for activities and processes that were previously centrally owned and enforced. This is especially true for security, privacy, and regulatory topics.[21][22][23]
- C2. Responsibility shift
- In data mesh individuals within domains are end-to-end responsible for data products. This new responsibility can be challenging, because it is rarely compensated and usually benefits other domains.[21][22]
- C3. Comprehension
- Research has shown a severe lack of comprehension for the data mesh paradigm among employees of companies implementing a data mesh.[21]
- BP1. Cross-domain unit
- Addressing C1, organizations should introduce a cross-domain steering unit responsible for strategic planning, use case prioritization, and the enforcement of specific governance rules—especially concerning security, regulatory, and privacy-related topics. Nevertheless, a cross-domain steering unit can only complement and support the federated governance structure and may grow obsolete with the increasing maturity of the data mesh.[21][24]
- BP2. Track and observe
- Addressing C2., organizations should observe and score data product quality as tracking and ranking key data products can encourage high-quality offerings, motivate domain owners, and support budget negotiations.[21]
- BP3. Conscious adoption
- Organizations should thoroughly assess and evaluate their existing data systems, consider organizational factors, and weigh the potential benefits before implementing a data mesh. When introducing data mesh, it is advised to carefully and consciously introduce data mesh terminology to ensure a clear understanding of the concept (C3).[21]
Community
Scott Hirleman has started a data mesh community that contains over 7,500 people in their Slack channel.[25]
See also
- Data management
- Data platform
- Data vault modeling, method of data modeling with storage of data from various operational systems and tracing of data origin, facilitating auditing, loading speeds and resilience
- Data warehouse, a well established type of database system for organizing data in a thematic way
- ETL and ELT
References
- OCLC 52134890.
- OCLC 1108538721.)
{{cite book}}: CS1 maint: location missing publisher (link - S2CID 245864612.
- ^ "Data Mesh Architecture". datamesh-architecture.com. Retrieved 2022-06-13.
- OCLC 1260236796.)
{{cite book}}: CS1 maint: location missing publisher (link - ISSN 1877-0509.
- ^ a b "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh". martinfowler.com. Retrieved 28 January 2022.
- ^ Baer (dbInsight), Tony. "Data Mesh: Should you try this at home?". ZDNet. Retrieved 2022-02-10.
- ^ Andy Mott (2022-01-12). "Driving Faster Insights with a Data Mesh". RTInsights. Retrieved 2022-03-01.
- ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
- ^ Bane, Andy. "Council Post: Where Is Industrial Transformation Headed In 2022?". Forbes. Retrieved 2022-03-01.
- ^ ISBN 978-1-09-810849-6.
- ^ Netflix Data Mesh: Composable Data Processing - Justin Cunningham, retrieved 2022-04-29
- ^ a b Baker, Tristan (2021-02-22). "Intuit's Data Mesh Strategy". Intuit Engineering. Retrieved 2022-04-29.
- ^ a b "The next generation of Data Platforms is the Data Mesh". 2022-08-03. Retrieved 2023-02-08.
- ^ "Why We Started Nextdata". 2022-01-16. Retrieved 2023-02-08.
- OCLC 1260236796.)
{{cite book}}: CS1 maint: location missing publisher (link - ^ "Data Mesh defined | James Serra's Blog". 16 February 2021. Retrieved 28 January 2022.
- ^ "Analytics in 2022 Means Mastery of Distributed Data Politics". The New Stack. 2021-12-29. Retrieved 2022-03-03.
- ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
- ^ arXiv:2302.01713v2 [cs.AI].
- ^ ISBN 978-3-031-08168-2.
- ^ Joshi, Divya; Pratik, Sheetal; Rao, Madhu Podila (2021). "Datagovernanceindata mesh infrastructures: The Saxo bank case study". Proceedings of the International Conference on Electronic Business (ICEB). Vol. 21. pp. 599–604.
- ^ Whyte, Martin; Odenkirchen, Andreas; Bautz, Stephan; Heringer, Agnes; Krukow, Oliver (2022). "Data Mesh - Just another buzzword or the next generation data platform?". PwC study 2022: Changing data platforms.
- ^ "The Global Home for Data Mesh". The Global Home for Data Mesh. Retrieved 2022-04-24.