Czaq
Active Member
- Joined
- Dec 15, 2025
- Messages
- 57
Most split tests run in this industry are theatre. Someone rotates two landers, checks after 150 clicks, sees lander B "winning" 8 conversions to 5, declares victory, and moves all traffic to B. Then B performs exactly like A did, because the "win" was a coin flip wearing a lab coat.
Here's the thing nobody wants to hear: at small sample sizes, random noise routinely produces differences that look decisive. Flip a coin 20 times and you'll regularly get 13–7. Nobody concludes the coin is biased. But show a media buyer 13 conversions vs 7 and they'll rebuild their whole funnel around it.
This guide covers three things: how to set the split up correctly in your tracker, how much traffic you actually need before the numbers mean anything (with a table, not a lecture), and the specific self-deceptions that ruin tests even when the setup is right. Plus a real test of mine that flipped its result completely between 200 and 2,000 clicks, which cured me of early peeking forever.
Use the tracker's rotation, nothing else.
Here's the thing nobody wants to hear: at small sample sizes, random noise routinely produces differences that look decisive. Flip a coin 20 times and you'll regularly get 13–7. Nobody concludes the coin is biased. But show a media buyer 13 conversions vs 7 and they'll rebuild their whole funnel around it.
This guide covers three things: how to set the split up correctly in your tracker, how much traffic you actually need before the numbers mean anything (with a table, not a lecture), and the specific self-deceptions that ruin tests even when the setup is right. Plus a real test of mine that flipped its result completely between 200 and 2,000 clicks, which cured me of early peeking forever.
Part 1: setting up the split correctly
The mechanics first, because a badly built split invalidates everything downstream.Use the tracker's rotation, nothing else.
