Mathematical Engineering of Deep Learning provides a complete and concise overview of deep learning using the language of mathematics. The book provides a self-contained background on machine learning and optimization algorithms and progresses through the key ideas of deep learning. These ideas and architectures include deep neural networks, convolutional models, recurrent models, long/short-term memory, the attention mechanism, transformers, variational auto-encoders, diffusion models, generative adversarial networks, reinforcement learning, and graph neural networks. Concepts are presented using simple mathematical equations together with a concise description of relevant tricks of the trade. The content is the foundation for state-of-the-art artificial intelligence applications, involving images, sound, large language models, and other domains. The focus is on the basic mathematical description of algorithms and methods and does not require computer programming. The presentation is also agnostic to neuroscientific relationships, historical perspectives, and theoretical research. The benefit of such a concise approach is that a mathematically equipped reader can quickly grasp the essence of deep learning.

Key Features:

• A perfect summary of deep learning not tied to any computer language, or computational framework.
• An ideal handbook of deep learning for readers that feel comfortable with mathematical notation.
• An up-to-date description of the most influential deep learning ideas that have made an impact on vision, sound, natural language understanding, and scientific domains.
• The exposition is not tied to the historical development of the field or to neuroscience, allowing the reader to quickly grasp the essentials.

Deep learning is easily described through the language of mathematics at a level accessible to many professionals. Readers from fields such as engineering, statistics, physics, pure mathematics, econometrics, operations research, quantitative management, quantitative biology, applied machine learning, or applied deep learning will quickly gain insights into the key mathematical engineering components of the field.

Table of Contents

1. Introduction

2. Principles of Machine Learning

3. Simple Neural Networks

4. Optimization Algorithms

5. Feedforward Deep Networks

6. Convolutional Neural Networks

7. Sequence Models

8. Specialized Architectures and Paradigms

Epilogue

A. Some Multivariable Calculus

B. Cross Ent ropy and Other Expectations with Logarithms

About the Authors

Benoit Liquet is a Professor of Mathematical and Computational Statistics at Macquarie University currently on detachment from his professor position at Université de Pau et Pays de l’Adour (France). He also holds an adjunct position at The University of Queensland. His research spans the broad spectrum of applied statistics, with a focus on statistical modeling for complex data. He has made significant contributions to methodological developments, exploiting modern statistical and computational cutting-edge methods to tackle a variety of real-world problems from small, designed studies to large-scale high-dimensional data challenges in bioinformatics and biometrics. His research extends to the development of R packages and industrial applications, particularly in the realm of machine learning. Over the years, he has authored numerous articles, book chapters, and books, including the co-authored book “The R Software, Fundamentals of Programming and Statistical Analysis”. He has also co-authored books on dynamical biostatistical models and developed over a dozen R packages, making his methodologies accessible to a wide range of users. He is deeply committed to education and has taught advanced courses in statistics and machine learning at multiple institutions around the globe. Such educational activities reflect his dedication to bridging the gap between theoretical advancements and practical applications.

Sarat Moka is an academic researcher and educator at the School of Mathematics and Statistics at The University of New South Wales (UNSW). His research interests encompass applied probability, computational statistics, machine learning, and deep learning. Dr. Moka has made contributions to optimization methods for efficient model selection in high-dimensional settings. Additionally, he has developed fast unbiased sampling and estimation techniques for spatial point processes and random graphs. Moreover, his research focus extends to efficient pruning methods for deep neural networks. In addition to research, he has been actively teaching advanced statistical and deep learning courses. Prior to joining UNSW in 2023, he was a senior research fellow at the School of Mathematical and Physical Science at Macquarie University and held an ACEMS (ARC Centre of Excellence for Mathematical & Statistical Frontiers) postdoctoral researcher position in the School of Mathematics and Physics at The University of Queensland. He earned a PhD in Applied Probability from the School of Technology and Computer Science at Tata Institute of Fundamental Research, and a Master's and a Bachelor's in Engineering with a focus on electrical, electronics, and communications, at the Indian Institute of Science and Andhra University, respectively. Before pursuing his doctoral studies, he was a scientist at the Indian Space Research Organization (SHAR, Sriharikota), where he worked on Communication Networks that support rocket launch activities.

Yoni Nazarathy is an Associate Professor at the School of Mathematics and Physics at The University of Queensland (UQ). He is also a consultant and co-director of a machine learning consultancy, Accumulation Point. His research spans applied probability, statistics, and machine learning and his industry work includes biostatistical software development, data science training for industry, and large language models. In addition to his many refereed publications in the mathematical sciences, he is a co-author of the book “Statistics with Julia”. Prior to his PhD in the field of queueing theory at the University of Haifa, Israel, he worked as a software engineer, algorithm developer, and team leader in the Israeli tech industry. He then followed with post doc positions in The Netherlands, and academic positions in Melbourne, Australia, before settling at UQ where he has been for over a decade. He is also an avid educator and has taught and created academic and professional courses across the spectrum. He has also contributed to mathematical education via software apps and engagement with pre-university level educators.