Bringing a deep-learning project into production at scale is quite challenging. To successfully scale your project, a foundational understanding of full stack deep learning, including the knowledge that lies at the intersection of hardware, software, data, and algorithms, is required.

This book illustrates complex concepts of full stack deep learning and reinforces them through hands-on exercises to arm you with tools and techniques to scale your project. A scaling effort is only beneficial when it's effective and efficient. To that end, this guide explains the intricate concepts and techniques that will help you scale effectively and efficiently.

This book aims to help you develop a deeper knowledge of the deep learning stack—specifically, how deep learning interfaces with hardware, software, and data. It will serve as a valuable resource when you want to scale your deep learning model, either by expanding the hardware resources or by adding larger volumes of data or increasing the capacity of the model itself. Efficiency is a key part of any scaling operation. For this reason, consideration of efficiency is weaved in throughout the book, to provide you with the knowledge and resources you need to scale effectively.

This book is written for machine learning practitioners from all walks of life: engineers, data engineers, MLOps, deep learning scientists, machine learning engineers, and others interested in learning about model development at scale. It assumes that the reader already has a fundamental knowledge of deep learning concepts such as optimizers, learning objectives and loss functions, and model assembly and compilation, as well as some experience with model development. Familiarity with Python and PyTorch is also essential for the practical sections of the book.

Given the complexity and scope, this book primarily focuses on scale-out of model development and training, with an extensive focus on distributed training. While the first few chapters may be useful for deployment and inference use cases, scaling inference is beyond the scope of this book.

The topics we will cover include:

• How your model is decomposed into a computation graph and how your data flows through this graph during the training process.
• The less told but beautiful story of floating-point numbers and how these Higgs bosons of deep learning can be used to achieve memory efficiency.
• How accelerated computing speeds up your training and how you can best utilize the hardware resources at your disposal.
• How to train your model using distributed training paradigms (i.e., data, model, pipeline, and hybrid multidimensional parallelism). You will also learn about federated learning and its challenges.
• How to leverage the PyTorch ecosystem in conjunction with NVIDIA libraries and Triton to scale your model training.
• Debugging, monitoring, and investigating bottlenecks that undesirably slow down the scale-out of model training.
• How to expedite the training lifecycle and streamline your feedback loop to iterate model development and related best practices.
• A set of data tricks and techniques and how to apply them to scale your training over limited resources.
• How to select the right tools and techniques for your deep learning project.
• Options for managing compute infrastructure when running at scale.

Table of Contents

Chapter 1. What Nature and History Have Taught Us About Scale

Part I. Foundational Concepts of Deep Learning

Chapter 2. Deep Learning

Chapter 3. The Computational Side of Deep Learning

Chapter 4. Putting It All Together: Efficient Deep Learning

Part II. Distributed Training

Chapter 5. Distributed Systems and Communications

Chapter 6. Theoretical Foundations of Distributed Deep Learning

Chapter 7. Data Parallelism

Chapter 8. Scaling Beyond Data Parallelism: Model, Pipeline, Tensor, and Hybrid Parallelism

Chapter 9. Gaining Practical Expertise with Scaling Across All Dimensions

Part III. Extreme Scaling

Chapter 10. Data-Centric Scaling

Chapter 11. Scaling Experiments: Effective Planning and Management

Chapter 12. Efficient Fine-Tuning of Large Models

Chapter 13. Foundation Models

About the Author

Suneeta holds a Ph.D. in applied science and has a computer science engineering background. She's worked extensively on distributed and scalable computing and machine learning experiences for IBM Software Labs, Expedita, USyd, and Nearmap. She currently leads the development of Nearmap's AI model system that produces high-quality AI data and sets and builds and manages a system that trains deep learning models efficiently. She is an active community member and speaker and enjoys learning and mentoring. She has presented at several top technical and academic conferences like SPIE, KubeCon, Knowledge Graph Conference, RE-Work, Kafka Summit, AWS Events, and YOW DATA. She has patents granted by USPTO and contributes to peer-reviewing journals besides publishing some papers in deep learning. She also authors for O'Reilly and Towards Data Science blogs and maintains her website at http://suneeta-mall.github.io