The Misuse of A/B Testing – My Personal Beef

Most people who have been around marketing long enough have had a brush with A/B Testing (also known as “split testing”). If this term isn’t ringing any bells, here’s a quick refresher. A/B Testing was the brainchild of a knighted British polymath named Sir Ronald Fisher, whom some claim to be the father of modern statistical science. Basically, A/B Testing is comparing two different versions of something to see which works best—or in marketing terms, which one consumers prefer.

For Fisher, it all started with a cup of tea in the early 1920s. The issue under examination was whether one should add milk to the cup before or after the tea has been poured. He conducted a test with eight cups of tea—half of which the milk had been added first and in the other half the milk was added after the tea. Surprisingly, his taster could indeed tell the difference, which encouraged him to expand his research.   

Thanks to the internet, AB Testing has exploded in popularity. Now big online players like LinkedIn make conducting these tests as easy as telling your right hand from your left.  Personally, I don’t have a beef with A/B Testing per se. What I do take issue with is how people are using this testing as a shortcut that often leads them down the wrong path. If this sounds like too much to wrap your head around, let me clarify it with the following example.

We all know Eddie Murphy, the comedian. Long before he became the world’s sixth-highest-grossing actor at the box office, he was a young, relatively unknown comic. But he knew he was funny. He enjoyed making people laugh. And he was determined to become the funniest guy on the planet. So he did some research—not unlike what we do for our clients. He looked at his competitive set: Bill Cosby, Richard Pryor, Sam Kinison—whoever happened to be in the comedy marketplace at the time. After assessing his competitors and what style of comedy they were doing, he came to an important conclusion. “If I wanna stand out, I can’t do the same type of comedy as these other guys because if I do, I’ll just be a “look alike.” Instead, Murphy decided to develop his own comedic brand with his own unique voice.

He determined, “Okay, I can be funny in this audience zone in this way. It pulls a little bit from what my competitors are doing, but it’s still my own unique voice.” That’s true “research”—understanding what your audience likes, and who and how you need to be so they prefer you. That’s research that identifies the opportunity.

And now we get to my beef. A/B Testing applied to this example would have Eddie Murphy bypass his research and go immediately to telling different jokes on different nights to see which jokes work best. A lot of businesses want to jump over Eddie’s initial research and go right to testing jokes. They skip the whole part about identifying the opportunity, assessing the marketplace, and considering their unique voice. Without that necessary homework, they may not be in the right zone in the first place. This increases the likelihood of them telling jokes that sound just like Bill Cosby or Richard Pryor.

Too many businesses are rushing right to the bottom of the funnel. “We don’t have to do any of the research!” they proclaim. “We don’t have to have any intent going into this or commit to a direction. We just have to let the world tell us whether it’s A or B!”