The Gollum Guide to A/B Testing: Find the One Precious Click!
They calls it A/B Testing, Master. We calls it… the way to the shiny things! If you just launch the new website… it could be bad! Very bad! They might not click! They might not convert! And we needs the conversions, we needs them!
Oh, Smeagol, don’t be afraid, my love. We’ll be smart about it. We’ll be systematic! Yes, precious, systematic!
Here is our nasty little plan to find the True Precious page design!
The System: Our Nasty Little Plan
Smeagol says: Be organized, precious. We can’t lose the visitors like we lost the fishing spot!
Gollum says: Organized? No! We just want to snatch the good design! Snatch it!
1 – Find the Problem-Spot, Yesss!
Where are they running away, Master? The visitors! They come to the page, they look… and then poof! Gone!
- Look at the Data: Where is the drop-off? Is it the button? Is it the headline? We must find the weakest point. It’s like finding the weak spot in the big ugly castle wall!
- The Big Questions: What do we think will make them click? Change the color of the button? Change the words on the headline? We must only change one thing at a time, or we won’t know which change brought the precious!
2 – Form a “Hypothesis,” My Love (The Guess)
Smeagol says: A hypothesis is an educated guess, my friend. We’ll say, “If we make the button green, conversion will increase by 10%.” Gollum says: Green? Green is slimy! We should make it… red and angry! And they will click out of fear! Heh heh heh.
- The Core Idea: You need two versions: Version A (the original, boring, old thing) and Version B (the new, shiny, potential precious!).
- Be Specific: Don’t just say “make it better.” Say exactly what you expect to happen. Will they scroll more? Will they put the product in the bag? We must know the true goal!
3 – Split the Traffic! (Share the Precious!)
This is the hardest part, Master. Sharing the lovely visitors!
- Equal Parts: We must send half of the people to Version A and half to Version B. Yes, precious, it’s fair! Like when we shared the fish! (Gollum whispers: But we ate the biggest piece, didn’t we? Yesss.)
- Run it Long Enough: We can’t stop early! We must wait until we have enough data—enough nasty little people—to be sure the winning version is truly the winner. If we stop too soon, the test will lie to us, and we HATES liars!
4 – Analyze the Winner! (The Feast!)
Smeagol says: If Version B clearly performed better and the results are “statistically significant,” we declare it the new winner. We deploy it to everyone! Gollum says: Feast! We feast on the delicious results! If Version B wins, we kill Version A! We crush it! We hide it! It will never come back!
- The Magic Number: We look for Statistical Significance. This means we are very, very sure (usually 95% sure) that the win wasn’t just luck or an accident. We don’t like accidents!
- Iterate (Do it Again!): Now that Version B is the new Version A (the control), we start over! We find a new problem-spot and test a new Version B! We never stop chasing the clicks! Never!
Be careful, Master! The data can be tricky. It can whisper lies, like the ring! Follow the system, and the conversion rates will be yours! Ours! Yesss, precious!
All character references are for entertainment purposes only. I own the rights to nothing and don’t make a dime off their use.