Trustworthy Online Controlled Experiments

I'm glad I stumbled across this book recently while reading an excellent article on non-significance in A/B testing.

If you want to understand what A/B testing is. Read this book.

If you want to understand why you should conduct A/B testing. Read this book.

If you want to understand how to conduct A/B testing. Read this book.

In over a decade of being interested in and working in online marketing, Trustworthy Online Controlled Experiments is the single most valuable and most comprehensive resource I've ever found on conducting A/B testing.

Kohavi, Tang and Xu have worked at the biggest tech companies in the world (Microsoft, Google, Amazon, LinkedIn, etc) and have the experience to match. They've taken that experience and have used to highlight 23 key principles people should think about when testing.

Trustworthy Online Controlled Experiments distills everything you could want to know about testing, it's free from buzzwords and free from the usual bullshit you find in a lot of similar material, taking a more scholarly approach to the subject.

While the book aims to be useful to people working at Google scale companies, there's a lot to learn even if you're working on significantly smaller programs. From understanding how you should go about collecting data, why you should A/A test^[1], how to properly evaluate whether a test was successful or not, ethical considerations when conducting a test or how to properly feed institutional insights back into the company.

I think this is a book I'll be reading and re-reading for many years to come.