Ready to use statistical and machine learning techniques across large data sets? This practical guide shows you why the Hadoop ecosystem is perfect for the job. Instead of deployment, operations, or software development usually associated with distributed computing, you'll focus on particular analyses you can build, the data warehousing techniques that Hadoop provides, and higher order data workflows this framework can produce.

Data scientists and analysts will learn how to perform a wide range of techniques, from writing MapReduce and Spark applications with Python to using advanced modeling and data management with Spark MLlib, Hive, and HBase. You'll also learn about the analytical processes and data systems available to build and empower data products that can handle- and actually require-huge amounts of data.

• Understand core concepts behind Hadoop and cluster computing
• Use design patterns and parallel analytical algorithms to create distributed data analysis jobs
• Learn about data management, mining, and warehousing in a distributed context using Apache Hive and HBase
• Use Sqoop and Apache Flume to ingest data from relational databases
• Program complex Hadoop and Spark applications with Apache Pig and Spark Data Frames
• Perform machine learning techniques such as classification, clustering, and collaborative filtering with Spark's MLlib

Table of Contents

Chapter 1. The Age of the Data Product

Chapter 2. An Operating System for Big Data

Chapter 3. A Framework for Python and Hadoop Streaming

Chapter 4. In-Memory Computing with Spark

Chapter 5. Distributed Analysis and Patterns

Part II. Workflows and Tools for Big Data Science

Chapter 6. Data Mining and Warehousing Chapter 7. Data Ingestion

Chapter 8. Analytics with Higher-Level APIs

Chapter 9. Machine Learning

Chapter 10. Summary: Doing Distributed Data Science

About the Authors

Benjamin Bengfort is a data scientist who lives inside the Beltway but ignores poli- tics (the normal business of DC), favoring technology instead. He is currently working to finish his PhD at the University of Maryland where he studies machine learning and distributed computing. His lab does have robots (though this field of study is not one he favors) and much to his chagrin, they seem to constantly arm said robots with knives and tools-presumably to pursue culinary accolades. Having seen a robot attempt to slice a tomato, Benjamin prefers his own adventures in the kitchen where he specializes in fusion French and Guyanese cuisine as well as BBQ of all types. A professional programmer by trade and a data scientist by vocation, Benja- min's writing pursues a diverse range of subjects from natural language processing, to data science with Python to analytics with Hadoop and Spark.

Jenny Kim is an experienced big data engineer who works in both commercial soft- ware efforts as well as in academia. She has significant experience working with large scale data, machine learning, and Hadoop implementations in production and research environments. Jenny (with Benjamin Bengfort) previously built a large scale recommender system that used a web crawler to gather ontological information about apparel products and produce recommendations from transactions. Currently, she is working with the Hue team at Cloudera to help build intuitive interfaces for analyzing big data with Hadoop.