This book shows how dbt is used to build data transformation pipelines that enable dependency management and allow for version control and automated testing. It explains how dbt is revolutionizing data transformation and the advantages that a command-line tool like dbt provides over and above the use of database stored procedures and other ETL and ELT tools that handle data transformations. You’ll see how to create custom-written transformations through simple SQL SELECT statements, eliminating the need for boilerplate code and making it easy to incorporate dbt as the transformation layer in your data warehouse pipelines. Additionally, you will learn how dbt enables data teams to incorporate software engineering best practices such as code reusability, version control, and automated testing into the data transformation process.

Unlocking dbt walks you through using dbt to establish a project, build and modularize SQL models, and execute jobs in a way that is easy to maintain and scale as your data ecosystem matures. You’ll begin by establishing and configuring a project, a process covered using both dbt Cloud and dbt Core, so that you can confidently stand up a project using either platform. From there, you’ll move into building transformations with peace of mind that your project will scale appropriately as you continue to develop it.

After learning the basics needed to get started, you’ll continue to build on that foundation by looking at the unique ways in which dbt combines SQL with Jinja to take your code beyond what is capable in normal SQL. You will learn about advanced materializations, building lineage in your data flows, the unlimited potential of macros, and so much more. This book also explores supported file types and the building of Python models. Rounding things out, you will learn features of dbt that will assist you in making your transformation layer production ready. These include how to implement automated testing, using dbt to generate documentation, and running CI/CD pipelines.

What You Will Learn

• Understand what dbt is and how it is used in the modern data stack
• Set up a project using both dbt Cloud and dbt Core
• Connect a dbt project to a cloud data warehouse
• Build SQL and Python models that are scalable and maintainable
• Configure development, testing, and production environments
• Capture reusable logic in the form of Jinja macros
• Incorporate version control with your data transformation code

Who This Book Is For

Current and aspiring data professionals, including architects, developers, analysts, engineers, data scientists, and consultants who are beginning the journey of using dbt as part of their data pipeline’s transformation layer. Readers should have a foundational knowledge of writing basic SQL statements, development best practices, and working with data in an analytical context such as a data warehouse.

Table of Contents

Chapter 1: Introduction to dbt

Chapter 2: Setting Up a dbt Project

Chapter 3: Sources and Seeds

Chapter 4: Models

Chapter 5: Snapshots

Chapter 6: Jinja, Macros, and Packages

Chapter 7: Hooks

Chapter 8: Tests

Chapter 9: Documentation

Chapter 10: dbt in Production

About the Authors

Cameron Cyr is a data fanatic who has spent his career developing data systems enabling valuable use cases such as analytics and machine learning. During this time, he has placed a focus on building reliable and scalable data systems with an emphasis on data quality. He is active in the data community and is one of the co-organizers and founders of Nashville's Data Engineering Group. Cameron currently serves as a data engineer for a healthcare tech startup.

Dustin Dorsey is a data leader and architect who has been building and managing data solutions for nearly 15 years. He is currently leading the build out of data infrastructure and analytics environments for a fast-growing healthcare tech startup. Dustin is a well-respected leader in the data community as an international speaker and mentor. He has previously organized several data community events and user groups and currently is one of the founders and organizers of the Nashville Data Engineering group. Dustin is one of the authors of the popular Apress book, Pro Database Migration to Azure.