Data fabric, data lakehouse, and data mesh have recently appeared as viable alternatives to the modern data warehouse. These new architectures have solid benefits, but they're also surrounded by a lot of hyperbole and confusion. This practical book provides a guided tour of these architectures to help data professionals understand the pros and cons of each.

James Serra, big data and data warehousing solution architect at Microsoft, examines common data architecture concepts, including how data warehouses have had to evolve to work with data lake features. You'll learn what data lakehouses can help you achieve, as well as how to distinguish data mesh hype from reality. Best of all, you'll be able to determine the most appropriate data architecture for your needs. With this book, you'll:

• Gain a working understanding of several data architectures
• Learn the strengths and weaknesses of each approach
• Distinguish data architecture theory from reality
• Pick the best architecture for your use case
• Understand the differences between data warehouses and data lakes
• Learn common data architecture concepts to help you build better solutions
• Explore the historical evolution and characteristics of data architectures
• Learn essentials of running an architecture design session, team organization, and project success factors

Table of Contents

Chapter 1. Introduction to Entity Resolution

Chapter 2. Data Standardization

Chapter 3. Text Matching

Chapter 4. Probabilistic Matching

Chapter 5. Record Blocking

Chapter 6. Company Matching

Chapter 7. Clustering

Chapter 8. Scaling Up on Google Cloud

Chapter 9. Cloud Entity Resolution Services

Chapter 10. Privacy-Preserving Record Linkage

Chapter 11. Further Considerations

I’ve been in information technology (IT) for nearly 40 years. I’ve worked at companies of all different sizes, I’ve worked as a consultant, and I’ve owned my own company. For the last 9 years, I have been at Microsoft as a data architect, and for the last 15 years, I have been involved with data warehousing. I’ve spoken about data thousands of times, to customers and groups.

During my career, I have seen many data architectures come and go. I’ve seen too many companies argue over the best approach and end up building the wrong data architecture—a mistake that can cost them millions of dollars and months of time, putting them well behind their competitors.

What’s more, data architectures are complex. I’ve seen firsthand that most people are unclear on the concepts involved, if they’re aware of them at all. Everyone seems to be throwing around terms like data mesh, data warehouse, and data lakehouse—but if you ask 10 people what a data mesh is, you will get 11 different answers.

Where do you even start? Are these just buzzwords with a lot of hype but little substance, or are they viable approaches? They may sound great in theory, but how practical are they? What are the pros and cons of each architecture?

None of the architectures discussed in this book is “wrong.” They all have a place, but only in certain use cases. No one architecture applies to every situation, so this book is not about convincing you to choose one architecture over the others. Instead, you will get honest opinions on the pros and cons of each architecture. Everything has trade-offs, and it’s important to understand what those are and not just go with an architecture that is hyped more than the others. And there is much to learn from each architecture, even if you don’t use it. For example, understanding how a data mesh works will get you thinking about data ownership, a concept that can apply to any architecture.

This book provides a basic grounding in common data architecture concepts. There are so many concepts out there, and figuring out which to choose and how to implement them can be intimidating. I’m here to help you to understand all these concepts and architectures at a high level so you get a sense of the options and can see which one is the most appropriate for your situation. The goal of the book is to allow you to talk intelligently about data concepts and architectures, then dig deeper into any that are relevant to the solution you are building.

There are no standard definitions of data concepts and architectures. If there were, this book would not be needed. My hope is to provide standard definitions that help everyone get onto the same page, to make discussions easier. I’m under no illusion that my definitions will be universally accepted, but I’d like to give us all a starting point for conversations about how to adjust those definitions.

I have written this book for anyone with an interest in getting value out of data, whether you’re a database developer or administrator, a data architect, a CTO or CIO, or even someone in a role outside of IT. You could be early in your career or a seasoned veteran. The only skills you need are a little familiarity with data from your work and a sense of curiosity.

For readers with less experience with these topics, I provide an overview of big data (Chapter 1) and data architectures (Chapter 2), as well as basic data concepts (Part II). If you’ve been in the data game for a while but need to understand new architectures, you might find a lot of value in Part III, which dives into the details of particular data architectures, as well as in reviewing some of the basics. For you, this will be a quick cover-to-cover read; feel free to skip over the sections with material that you already know well. Also note that although the focus is on big data, the concepts and architectures apply even if you have “small” data.

This is a vendor-neutral book. You should be able to apply the architectures and concepts you learn here with any cloud provider. I’ll also note here that I am employed by Microsoft. However, the opinions expressed here are mine alone and do not reflect the views of my employer.

I wrote this book because I have an innate curiosity that drives me to comprehend and then share things in a way that everyone can understand. This book is the culmination of my life’s work. I hope you find it valuable.

Review

"There is no one whose knowledge of data architectures and data processes I trust more than James Serra. This book not only provides a comprehensive and clear description of key architectural principles, approaches, and pitfalls, it also addresses the all-important people, cultural, and organizational issues that too often imperil data projects before they get going. This book is destined to become an industry primer studied by college students and business professionals alike who encounter data for the first time (and maybe the second and third time as well!)"

--Wayne Eckerson, President of Eckerson Group

"James's superpower has always been taking complex subjects and explaining them in a simple way. In this book, he hits all the key points to help you choose the right data architecture and avoid common (and costly!) mistakes."

--Rod Colledge, Senior Technical Specialist (Data & AI), Microsoft

"James has condensed over 30 years of data architecture knowledge and wisdom into this comprehensive and very readable book. For those who must do the hard work of delivering analytics rather than singing its praises, this is a must-read."

--Dr. Barry Devlin, Founder and Principal, 9sight Consulting

"Data management is critical to the success of every business. Deciphering Data Architectures breaks down the buzzwords into simple and understandable concepts and practical solutions to help you get to the right architecture for your dataset. James has an innate curiosity to understand things and then to share that in a way that everyone can understand."

--Matt Usher, Director, Pure Storage

"James' blog has been my go-to resource for demystifying architectural concepts, understanding technical terminology, and navigating the life of a solution architect or data engineer. His ability to transform complex technical concepts into clear, easy-to-grasp explanations is truly remarkable. This book is an invaluable collection of his work, serving as a comprehensive reference guide for designing and comprehending architectures."

--Annie Xu, Senior Data Customer Engineer, Google

"Deciphering Data Architectures is not only thorough and detailed, but it also provides a critical perspective on what works, and perhaps more importantly, what may not work well. Whether discussing older data approaches or newer ones such as Data Mesh, the book offers words of wisdom and lessons learned that will help any data practitioner accelerate their data journey."

--Eric Broda, entrepreneur, data consultant, O'Reilly author of Implementing Data Mesh

"In Deciphering Data Architectures, James Serra does a wonderful job explaining the evolution of leading data architectures and the trade-offs between them. This book should be required reading for current and aspiring data architects."

--Bill Anton, Data Geek, Opifex Solutions

"Marketing buzz and industry thought-leader chatter have sown much confusion about data architecture patterns. With his depth of experience and skill as a communicator, James Serra cuts through the noise and provides clarity on both long-established data architecture patterns and cutting-edge industry methods. that will aid data practitioners and data leaders alike. Put it on your desk-- you'll reference it often."

--Sawyer Nyquist, Owner, Writer, and Consultant, The Data Shop

"Deciphering Data Architectures is an indispensable vendor-neutral guide for today's data professionals. It insightfully compares historical and modern architectures, emphasizing key trade-offs and decision-making nuances in choosing an appropriate architecture for the evolving data-driven landscape."

--Stacia Varga, Author and data analytics consultant, Data Inspirations

"The world of data architectures is complex and full of noise. This book provides a fresh, practical perspective born of decades of experience. Whether you're a beginner or an expert, everyone with an interest in data must read this book!"

--Piethein Strengholt, author of Data Management at Scale

About the Author

James works at Microsoft as a big data and data warehousing solution architect where he has been for most of the last nine years. He is a thought leader in the use and application of Big Data and advanced analytics, including data architectures such as the modern data warehouse, data lakehouse, data fabric, and data mesh. Previously he was an independent consultant working as a Data Warehouse/Business Intelligence architect and developer. He is a prior SQL Server MVP with over 35 years of IT experience. He is a popular blogger (JamesSerra.com) and speaker, having presented at dozens of major events including SQLBits, PASS Summit, Data Summit and the Enterprise Data World conference.