Large language models (LLMs) and diffusion models such as ChatGPT and Stable Diffusion have unprecedented potential. Because they have been trained on all the public text and images on the internet, they can make useful contributions to a wide variety of tasks. And with the barrier to entry greatly reduced today, practically any developer can harness LLMs and diffusion models to tackle problems previously unsuitable for automation.

With this book, you'll gain a solid foundation in generative AI, including how to apply these models in practice. When first integrating LLMs and diffusion models into their workflows, most developers struggle to coax reliable enough results from them to use in automated systems. Authors James Phoenix and Mike Taylor show you how a set of principles called prompt engineering can enable you to work effectively with AI.

Learn how to empower AI to work for you. This book explains:

• The structure of the interaction chain of your program's AI model and the fine-grained steps in between
• How AI model requests arise from transforming the application problem into a document completion problem in the model training domain
• The influence of LLM and diffusion model architecture—and how to best interact with it
• How these principles apply in practice in the domains of natural language processing, text and image generation, and code

The rapid pace of innovation in generative AI promises to change how we live and work, but it’s getting increasingly difficult to keep up. The number of AI papers published on arXiv is growing exponentially, Stable Diffusion has been among the fastest growing open source projects in history, and AI art tool Midjourney’s Discord server has tens of millions of members, surpassing even the largest gaming communities. What most captured the public’s imagination was OpenAI’s release of ChatGPT, which reached 100 million users in two months, making it the fastest-growing consumer app in history. Learning to work with AI has quickly become one of the most in-demand skills.

Everyone using AI professionally quickly learns that the quality of the output depends heavily on what you provide as input. The discipline of prompt engineering has arisen as a set of best practices for improving the reliability, efficiency, and accuracy of AI models. “In ten years, half of the world’s jobs will be in prompt engineering,” claims Robin Li, the cofounder and CEO of Chinese tech giant Baidu. However, we expect prompting to be a skill required of many jobs, akin to proficiency in Microsoft Excel, rather than a popular job title in itself. This new wave of disruption is changing everything we thought we knew about computers. We’re used to writing algorithms that return the same result every time—not so for AI, where the responses are non-deterministic. Cost and latency are real factors again, after decades of Moore’s law making us complacent in expecting real-time computation at negligible cost. The biggest hurdle is the tendency of these models to confidently make things up, dubbed hallucination, causing us to rethink the way we evaluate the accuracy of our work.

We’ve been working with generative AI since the GPT-3 beta in 2020, and as we saw the models progress, many early prompting tricks and hacks became no longer necessary. Over time a consistent set of principles emerged that were still useful with the newer models, and worked across both text and image generation. We have written this book based on these timeless principles, helping you learn transferable skills that will continue to be useful no matter what happens with AI over the next five years. The key to working with AI isn’t “figuring out how to hack the prompt by adding one magic word to the end that changes everything else,” as OpenAI cofounder Sam Altman asserts, but what will always matter is the “quality of ideas and the understanding of what you want.” While we don’t know if we’ll call it “prompt engineering” in five years, working effectively with generative AI will only become more important.

"The absolute best book-length resource I've read on prompt engineering. Mike and James are masters of their craft."

—Dan Shipper, cofounder & CEO, Every

"This book is a solid introduction to the fundamentals of prompt engineering and generative AI. The authors cover a wide range of useful techniques for all skill levels from beginner to advanced in a simple, practical, and easy-to-understand way. If you're looking to improve the accuracy and reliability of your AI systems, this book should be on your shelf."

-Mayo Oshin, founder and CEO, Siennai Analytics, early LangChain contributor

"Phoenix and Taylor's guide is a lighthouse amidst the vast ocean of generative AI. Their book became a cornerstone for my team at Phiture AI Labs, as we learned to harness LLMs and diffusion models for creating marketing assets that resonate with the essence of our clients' apps and games. Through prompt engineering, we've been able to generate bespoke, on-brand content at scale. This isn't just theory; it's a practical masterclass in transforming AI's raw potential into tailored solutions, making it an essential read for developers looking to elevate their AI integration to new heights of creativity and efficiency."

—Moritz Daan, Founder/Partner, Phiture Mobile Growth Consultancy

"Prompt Engineering for Generative AI is probably the most future-proof way of future-proofing your tech career. This is without a doubt the best resource for anyone working in practical applications of AI. The rich, refined principles in here will help both new and seasoned AI engineers stay on top of this very competitive game for the foreseeable future."

- Ellis Crosby, CTO and cofounder, Incremento

"This is an essential guide for agency and service professionals. Integrating AI with service and client delivery, using automation management, and speeding up solutions will set new industry standards. You'll find useful, practical information and tactics in the book, allowing you to understand and utilize AI to its full potential."

- Byron Tassoni-Resch, CEO and cofounder, WeDiscover

From the Author

We've been doing prompt engineering since the GPT-3 beta in 2020, and when GPT-4 arrived we found a lot of the tricks and hacks we used were no longer necessary. This motivated us to define a set of future-proof principles that are transferrable across models and modalities, that will still be useful with GPT-5, or whatever model we use in the future. 

The Five Principles of Prompting are:

1. Give Direction: Describe the desired style in detail, or reference a relevant persona.
2. Specify Format: Define what rules to follow, and the required structure of the response.
3. Provide Examples: Insert a diverse set of test cases where the task was done correctly.
4. Evaluate Quality: Identify errors and rate responses, testing what drives performance.
5. Divide Labor: Split tasks into multiple steps, chained together for complex goals.

We first published these principles as a blog post in July 2022, and they have stood the test of time, including mapping quite closely to OpenAI's own Prompt Engineering Guide, which came a year later. Anyone who works closely with generative AI is likely to converge on a similar set of strategies for solving common issues, but this book is designed to get you there quicker.

Throughout this book you'll see hundreds of demonstrative examples of prompting techniques, including both text and image prompting, as well as using Python to build AI automation scripts and products. This isn't a list of prompting hacks to find the right combination of magic words, it's a practical guide for building systems that provide the right context to AI applications, as well as how to test and scale AI systems for production.

The book will be useful for you if:

• Your time is worth more than 40 dollars an hour, and saving a few hours reading this book instead of piecing everything together from multiple sources is worth it to you.
• You're not just using AI casually but you're actually building an AI application or internal template many people will use hundreds or thousands of times a day.
• You want tips for reducing hallucination and improving the reliability of AI, while learning from 100s of real-world examples of how to solve common issues working with AI.
• You'd like to compare the strengths and weaknesses of OpenAI vs other models, as well as common frameworks like LangChain, different vector database options, and AUTOMATIC1111
• You want to see what it looks like to build an end-to-end AI application, from a naive prompt to a full AI agent, including building a basic user interface with Gradio

Table of Contents

Chapter 1. The Five Principles of Prompting

Chapter 2. Introduction to Large Language Models for Text Generation Chapter 3. Standard Practices for Text Generation with ChatGPT

Chapter 4. Advanced Techniques for Text Generation with LangChain

Chapter 5. Vector Databases with FAISS and Pinecone

Chapter 6. Autonomous Agents with Memory and Tools

Chapter 7. Introduction to Diffusion Models for Image Generation

Chapter 8. Standard Practices for Image Generation with Midjourney

Chapter 9. Advanced Techniques for Image Generation with Stable Diffusion

Chapter 10. Building Al-Powered Applications

About the Author

James Phoenix has a background in building reliable data pipelines for marketing teams, including automation of thousands of recurring marketing tasks. He has taught 40+ Data Science bootcamps for General Assembly.

Mike Taylor built and ran a 50-person marketing agency, including working on innovation projects with Unilever, Nestle, and Facebook. Over 300,000 people have taken his marketing courses on LinkedIn Learning.