As machine learning gains widespread adoption and integration in a variety of applications, including safety- and mission-critical systems, the need for robust evaluation methods grows more urgent. This book compiles scattered information on the topic from research papers and blogs to provide a centralized resource that is accessible to students, practitioners, and researchers across the sciences.

The book examines meaningful metrics for diverse types of learning paradigms and applications, unbiased estimation methods, rigorous statistical analysis, fair training sets, and meaningful explainability, all of which are essential for building robust and reliable machine learning products. In addition to standard classification, the book discusses unsupervised learning, regression, image segmentation, and anomaly detection. It also covers topics such as industry-strength evaluation, fairness, and responsible AI. Implementations using Python and scikit-learn are available on the book’s website.

Table of Contents

Part I Preliminary Considerations

1 Introduction

2 Statistics Overview

3 Machine Learning Preliminaries

4 Traditional Machine Learning Evaluation

Part II Evaluation for Classification

5 Metrics

6 Resampling

7 Statistical Analysis

Part III Evaluation for Other Settings

8 Supervised Settings Other Than Simple Classification

9 Unsupervised Learning

Part IV Evaluation from a Practical Perspective

10 Industrial-Strength Evaluation

11 Responsible Machine Learning

12 Conclusion

About the Authors

Nathalie Japkowicz is Professor and former Chair of the Department of Computer Science at American University, Washington, DC. She previously taught at the University of Ottawa. Her current research focuses on lifelong anomaly detection and hate speech detection. She has also researched one-class learning and the class imbalance problem extensively. She has received numerous awards, including the Test of Time and Distinguished Service awards.

Zois Boukouvalas is an Assistant Professor in the Department of Mathematics and Statistics at American University, Washington, DC. His research focuses on the development of interpretable multimodal machine learning algorithms, and he has been the lead principal investigator of several research grants. Through his research and teaching activities, he is creating environments that encourage and support the success of underrepresented students for entry into machine learning careers.