Get started with Apache Flink, the open source framework that powers some of the world’s largest stream processing applications. With this practical book, you’ll explore the fundamental concepts of parallel stream processing and discover how this technology differs from traditional batch data processing.

Longtime Apache Flink committers Fabian Hueske and Vasia Kalavri show you how to implement scalable streaming applications with Flink’s DataStream API and continuously run and maintain these applications in operational environments. Stream processing is ideal for many use cases, including low-latency ETL, streaming analytics, and real-time dashboards as well as fraud detection, anomaly detection, and alerting. You can process continuous data of any kind, including user interactions, financial transactions, and IoT data, as soon as you generate them.

• Learn concepts and challenges of distributed stateful stream processing
• Explore Flink’s system architecture, including its event-time processing mode and fault-tolerance model
• Understand the fundamentals and building blocks of the DataStream API, including its time-based and statefuloperators
• Read data from and write data to external systems with exactly-once consistency
• Deploy and configure Flink clusters
• Operate continuously running streaming applications

Table of Contents

Chapter 1. Introduction to Stateful Stream Processing

Chapter 2. Stream Processing Fundamentals

Chapter 3. The Architecture of Apache Flink

Chapter 4. Setting Up a Development Environment for Apache Flink

Chapter 5. The DataStream API (v1.7)

Chapter 6. Time-Based and Window Operators

Chapter 7. Stateful Operators and Applications

Chapter 8. Reading from and Writing to External Systems

Chapter 9. Setting Up Flink for Streaming Applications

Chapter 10. Operating Flink and Streaming Applications

Chapter 11. Where to Go from Here?

What You Will Learn in This Book

This book will teach you everything you need to know about stream processing with Apache Flink. It consists of 11 chapters that hopefully tell a coherent story. While some chapters are descriptive and aim to introduce high-level design concepts, others are more hands-on and contain many code examples.

While we intended for the book to be read in chapter order when we were writing it, readers familiar with a chapter’s content might want to skip it. Others more interested in writing Flink code right away might want to read the practical chapters first. In the following, we briefly describe the contents of each chapter, so you can directly jump to those chapters that interest you most.

• Chapter 1 gives an overview of stateful stream processing, data processing application architectures, application designs, and the benefits of stream processing over traditional approaches. It also gives you a brief look at what it is like to run your first streaming application on a local Flink instance.
• 
  
• Chapter 2 discusses the fundamental concepts and challenges of stream processing, independent of Flink.
• Chapter 3 describes Flink’s system architecture and internals. It discusses distributed architecture, time and state handling in streaming applications, and Flink’s fault-tolerance mechanisms.
• Chapter 4 explains how to set up an environment to develop and debug Flink applications.
• Chapter 5 introduces you to the basics of the Flink’s DataStream API. You will learn how to implement a DataStream application and which stream transformations, functions, and data types are supported.
• Chapter 6 discusses the time-based operators of the DataStream API. This includes window operators and time-based joins as well as process functions that provide the most flexibility when dealing with time in streaming applications.
• Chapter 7 explains how to implement stateful functions and discusses everything around this topic, such as the performance, robustness, and evolution of stateful functions. It also shows how to use Flink’s queryable state.
• Chapter 8 presents Flink’s most commonly used source and sink connectors. It discusses Flink’s approach to end-to-end application consistency and how to implement custom connectors to ingest data from and emit data to external systems.
• Chapter 9 discusses how to set up and configure Flink clusters in various environments.
• Chapter 10 covers operation, monitoring, and maintenance of streaming applications that run 24/7.
• Finally, Chapter 11 contains resources you can use to ask questions, attend Flink-related events, and learn how Flink is currently being used.

About the Author

Fabian Hueske is a PMC member of the Apache Flink project and has been contributing to Flink since day one. Fabian is cofounder of data Artisans (now Ververica) and holds a PhD in computer science from TU Berlin.

Vasiliki (Vasia) Kalavri is a postdoctoral fellow in the Systems Group at ETH Zurich. Vasia is a PMC member of the Apache Flink project. An early contributor to Flink, she has worked on its graph processing library, Gelly, and on early versions of the Table API and streaming SQL.