Jump to content

Data mesh

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by 98.60.255.209 (talk) at 17:34, 5 March 2023 (Word choice). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

Data mesh is a sociotechnical approach to building a decentralized data architecture by leveraging a domain-oriented, self-serve design (in a software development perspective), and borrows Eric Evans’ theory of domain-driven design[1] and Manuel Pais’ and Matthew Skelton’s theory of team topologies.[2] Data mesh mainly concerns itself with the data itself, taking the data lake and the pipelines as a secondary concern. [3] The main proposition is scaling analytical data by domain-oriented decentralization.[4] With data mesh, the responsibility for analytical data is shifted from the central data team to the domain teams, supported by a data platform team that provides a domain-agnostic data platform.[5]

History

The term data mesh was first defined by Zhamak Dehghani in 2019[6] while she was working as a principal consultant at the technology company Thoughtworks.[7][8] Dehghani introduced the term in 2019 and then provided greater detail on its principles and logical architecture throughout 2020. The process was predicted to be a “big contender” for companies in 2022.[9][10] Data meshes have been implemented by companies such as Zalando,[11] Netflix,[12] Intuit,[13] VistaPrint, JPMorgan Chase,[14] PayPal[15] and others.

In 2022, Dehghani left Thoughtworks to found NextData Technologies to focus on decentralized data.[16]

Principles

Data mesh is based on four core principles:[17]

In addition to these principles, Dehghani writes that the data products created by each domain team should be discoverable, addressable, trustworthy, possess self-describing semantics and syntax, be interoperable, secure, and governed by global standards and access controls.[19] In other words, the data should be treated as a product that is ready to use and reliable.[20]

See also

References

  1. ^ Evans, Eric (2004). Domain-driven design : tackling complexity in the heart of software. Boston: Addison-Wesley. ISBN 0-321-12521-5. OCLC 52134890.
  2. ^ Skelton, Matthew (2019). Team topologies : organizing business and technology teams for fast flow. Manuel Pais. Portland, OR. ISBN 978-1-942788-84-3. OCLC 1108538721.{{cite book}}: CS1 maint: location missing publisher (link)
  3. ^ Machado, Inês Araújo; Costa, Carlos; Santos, Maribel Yasmina (2022-01-01). "Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures". Procedia Computer Science. International Conference on ENTERprise Information Systems / ProjMAN - International Conference on Project MANagement / HCist - International Conference on Health and Social Care Information Systems and Technologies 2021. 196: 263–271. doi:10.1016/j.procs.2021.12.013. ISSN 1877-0509. S2CID 245864612.
  4. ^ "Data Mesh Architecture". datamesh-architecture.com. Retrieved 2022-06-13.
  5. ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.{{cite book}}: CS1 maint: location missing publisher (link)
  6. ^ "How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh". martinfowler.com. Retrieved 28 January 2022.
  7. ^ Baer (dbInsight), Tony. "Data Mesh: Should you try this at home?". ZDNet. Retrieved 2022-02-10.
  8. ^ Andy Mott (2022-01-12). "Driving Faster Insights with a Data Mesh". RTInsights. Retrieved 2022-03-01.
  9. ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
  10. ^ Bane, Andy. "Council Post: Where Is Industrial Transformation Headed In 2022?". Forbes. Retrieved 2022-03-01.
  11. ^ Schultze, Max; Wider, Arif (2021). Data Mesh in Practice. ISBN 978-1-09-810849-6.
  12. ^ Netflix Data Mesh: Composable Data Processing - Justin Cunningham, retrieved 2022-04-29
  13. ^ Baker, Tristan (2021-02-22). "Intuit's Data Mesh Strategy". Intuit Engineering. Retrieved 2022-04-29.
  14. ^ "How JPMorgan Chase built a data mesh architecture to drive significant value to enhance their enterprise data platform | AWS Big Data Blog". aws.amazon.com. 2021-05-05. Retrieved 2023-01-10.
  15. ^ "The next generation of Data Platforms is the Data Mesh". 2022-08-03. Retrieved 2023-02-08.
  16. ^ "Why We Started Nextdata". 2022-01-16. Retrieved 2023-02-08.
  17. ^ Dehghani, Zhamak (2022). Data Mesh. Sebastopol, CA. ISBN 978-1-4920-9236-0. OCLC 1260236796.{{cite book}}: CS1 maint: location missing publisher (link)
  18. ^ "Data Mesh defined | James Serra's Blog". 16 February 2021. Retrieved 28 January 2022.
  19. ^ "Analytics in 2022 Means Mastery of Distributed Data Politics". The New Stack. 2021-12-29. Retrieved 2022-03-03.
  20. ^ "Developments that will define data governance and operational security in 2022". Help Net Security. 2021-12-28. Retrieved 2022-03-01.
Retrieved from "https://en.wikipedia.org/enwiki/w/index.php?title=Data_mesh&oldid=1143049150"