As data management continues to evolve rapidly, managing all of your data in a central place, such as a data warehouse, is no longer scalable. Today's world is about quickly turning data into value. This requires a paradigm shift in the way we federate responsibilities, manage data, and make it available to others. With this practical book, you'll learn how to design a next-gen data architecture that takes into account the scale you need for your organization.

Executives, architects and engineers, analytics teams, and compliance and governance staff will learn how to build a next-gen data landscape. Author Piethein Strengholt provides blueprints, principles, observations, best practices, and patterns to get you up to speed.

• Examine data management trends, including regulatory requirements, privacy concerns, and new developments such as data mesh and data fabric
• Go deep into building a modern data architecture, including cloud data landing zones, domain-driven design, data product design, and more
• Explore data governance and data security, master data management, self-service data marketplaces, and the importance of metadata

Data management is an emerging and disruptive subject. Datafication is everywhere. This transformation is happening all around us: in smartphones, TV devices, ereaders, industrial machines, self-driving cars, robots, and so on. It’s changing our lives at an accelerating speed.

Table of Contents

Chapter 1. The Disruption of Data Management

Chapter 2. Introducing the Scaled Architecture: Organizing Data at Scale

Chapter 3. Managing Vast Amounts of Data: The Read-Only Data Stores Architecture

Chapter 4. Services and API Management: The API Architecture

Chapter 5. Event and Response Management: The Streaming Architecture

Chapter 6. Connecting the Dots

Chapter 7. Sustainable Data Governance and Data Security

Chapter 8. Turning Data into Value

Chapter 9. Mastering Enterprise Data Assets

Chapter 1 0. Democratizing Data with Metadata

Chapter 11 . Conclusion

As the amount of data generated skyrockets, so does its complexity. Disruptive trends like cloudification, API and ecosystem connectivity, microservices, open data, software as a service (SaaS), and new software delivery models have a tremendous effect on data management. In parallel, we see an enormous number of new applications transforming our businesses. All these trends are fragmenting the data landscape. As a result, we are seeing more point-to-point interfaces, endless discussions about data quality and ownership, and plenty of ethical and legal dilemmas regarding privacy, safety, and security. Agility, long-term stability, and clear data governance compete with the need to develop new business cases swiftly. We sorely need a clear vision for the future of data management.

This book’s perspective on data management is informed by my personal experience driving the data architecture agenda for a large enterprise as chief data architect. Executing that role showed me clearly the impact a good data strategy can have on a large organization. After leaving that company, I started working as the chief data officer for Microsoft Netherlands. In this exciting new position, I’ve worked with over 50 large customers discussing and attempting to come up with a perfect data solution. Here are some of the common threads I’ve identified across all enterprises:

• An overarching data strategy is often missing or not connected to the business objectives. The discussion mostly pivots to technology trends and engineering discussions. What is needed is business engagement: a good strategy and well-thought-out data management and analysis plan that includes tangible value in the form of business use cases. To make my point: the focus must be put on usage and turning data into business value.
• Enterprises have difficulties in interpreting new concepts like the data mesh and data fabric, because pragmatic guidance and experiences from the field are missing. In addition to that, the data mesh fully embraces a decentralized approach, which is a transformational change not only so for the data architecture and technology, but even more for organization and processes. This means the transformation cannot only be led by IT; it’s a business transformation as well.
• Enterprises find it difficult to comprehend the latest technology trends. They’re unable to interpret nuances or make pragmatic choices.
• Enterprises struggle to get started: large ambitions often end with limited action; the execution plan and architecture remain too high-level, too conceptual; top-down commitment from leadership is missing.

These experiences and my observations across a range of enterprises inspired me to write this second edition of Data Management at Scale. You may wonder what motivated me and why this book is

worth reading, over the first edition—let’s take a closer look.

Who Is This Book For?

This book is intended for large enterprises, though smaller organizations may find much of value in it. It’s geared toward:

• Executives and architects: Chief data officers, chief technology officers, chief architects, enterprise architects, and lead data architects
• Analytics teams: Data scientists, data engineers, data analysts, and heads of analytics
• Development teams: Data engineers, data scientists, business intelligence engineers, data modelers and designers, and other data professionals
• Compliance and governance teams: Chief information security officers, data protection officers, information security analysts, regulatory compliance heads, data stewards, and business analysts

How to Read or Use This Book

It’s important to say up front that this book touches upon a lot of complex topics that are often interrelated or intertwined with other subjects. So we’ll be hopping between different technologies, business methods, frameworks, and architecture patterns. From time to time I bring in my own operational experience when implementing different architectures, so we’ll be working at different levels of abstraction. To describe the journey through the book, I’ll use the analogy of a helicopter ride.

We’ll start with a zoomed-out view, looking at data management, data strategy, and data architecture at an abstract and higher level. From this helicopter view, we’ll start to zoom in and first explore what data domains and landing zones are. We’ll then fly to the source system side of our landscape, in which applications are managed and data is created, and circle until we have covered most of the areas of data management. Then we’ll fly over to consumer side of the landscape and start learning about the dynamics there. After that, we’ll bring everything we’ve covered together by putting things into practice.

About the Author

Piethein Strengholt works as the chief data officer for Microsoft Netherlands. In this exciting role he acts as a counterpart to CDO-executives for large enterprises and is a driving force in the community and alignment with the product group. Piethein is also a prolific blogger and regularly speaks about the latest trends in data mesh, data governance, and strategy at scale. He lives in the Netherlands with his family.