Test Enough Variants and One Wins by Chance Alone
A case study says the winning variant beat the others by 34 percent. The number is specific, the lift looks real, and the write up moves straight from the win to the takeaway. What it does not say is how many variants lost. That missing figure is the difference between a trustworthy result and one that was simply the best of a pile.
The number that is always missing from the winner’s circle
Picture the claim as it usually appears: “our winning variant beat the others by 34 percent.” There is a winner and a margin, and nothing about the field it beat. Out of how many variants was this the best one? Two? Five? Twenty? The write up treats that question as beside the point, when it is the point.
This is different from a single fair A/B test. With two options and one comparison, a meaningful margin is the whole story, because nothing else in the running could have produced it by accident. Once a case study picks a winner out of a wider field and names only the winner, the reader cannot judge the claim without the count.
Why testing more variants makes a fluke more likely, not less
Every additional variant tested is another independent roll of the dice, another chance for ordinary randomness to hand one option a result that looks like a real effect. Statisticians call this the multiple comparisons problem. Test enough things at once and report only the standout, and you will find a standout regularly, whether or not that variant actually performs better.
Keep this separate from a related mistake already covered on this site: a single test called too early, before enough data has accumulated to mean anything. That is a timing problem inside one comparison. This is a counting problem across many comparisons, and the fix for one does not touch the other.
The intuition needs no formula. The more lottery tickets you buy, the more likely one looks like a winner, even if every ticket is identical. A variant that “won” out of twenty earned that title the way a lottery ticket earns a payout: it did not need to be better, only to be the one the dice landed on.
A hypothetical: twenty coin flips, no skill involved
Every number here is hypothetical, invented to illustrate the mechanism. It is not a real study, not a real client, and not a measured result.
Picture 20 independent coin flip tests, run in parallel, each a fair 50/50 flip. No marketing, no variants, no skill, just a coin landing heads or tails. Flip a fair coin ten times and getting seven or more of the same face is not rare, so across 20 separate ten-flip tests, run at once, several should land a lopsided run purely by chance. Nothing caused it. Some of the 20 simply landed a streak, the way some always will.
Now make the substitution the case study never makes for you. Swap the coin for a marketing variant, and the streak of heads for a lift in conversions. The coin with the most lopsided streak becomes “the winning variant,” written up with an impressive margin. The other 19 flips never make the report, because a case study built around one clear winner has no reason to mention them.
What the case study would have to disclose for the win to mean something
A trustworthy multi-variant case study gives the reader enough to judge the win, rather than asking for trust:
- The total number of variants tested, not just the one that won.
- Whether the winning metric was set before the results came in, or chosen after seeing the data. A comparison designed in advance is a test. One assembled after seeing which option looked best is closer to a search for a flattering number.
- Whether the winning margin held up in a separate follow up test, rather than being reported once and left there.
A small number of variants, two or three, tested honestly and disclosed in full, is a stronger basis for a claim than ten or twenty variants where only the winner is shown. A case study naming a winner without stating how many were tested has not given the reader enough to tell a real effect from noise.
How this differs from a test that was simply called too early
Early stopping and the multiple comparisons problem blur together because both can inflate how strong a winning number looks. But they answer different questions. Early stopping asks whether a single test ran long enough before someone declared it a winner. This asks how many separate things were compared at once, and whether that count was disclosed. For an early stopping concern, ask about duration. For this one, ask about the denominator. The case study library is where that denominator question is worth asking on every entry, and if your own agency has a result it can disclose fully, this library exists to hold it to that standard: submit your own.
The one question worth asking about any ‘winning variant’ claim
Strip everything else away and one question does most of the work: out of how many variants was this the winner, and was that count fixed before the results came in or decided afterward? An agency that answers both parts without hesitation is likely reporting something real.
The agency behind a case study may not volunteer that number, and that silence is not neutral. An undisclosed variant count is itself a data point.
FAQ
Does testing more variants always make a case study’s result less trustworthy?
Not automatically. Testing more variants and disclosing the full count, with the comparison decided before anyone saw the results, is a legitimate way to search for a better option. The problem is narrower: testing many variants and reporting only the best one without saying how many losing variants there were.
Is this the same thing as a test being stopped early?
No. Early stopping is about ending one test as soon as it looks favourable, before it has run long enough to mean much. This is about running many separate comparisons at once and only surfacing the single best result. Both can appear in the same case study, but what a case study leaves out when it names the winning variant is a distinct question that deserves its own check.
Sources
- https://socialcasestudies.com/blog/reading-an-ab-test-claim/, fetched 2026-09-09, HTTP 200.
- https://socialcasestudies.com/blog/how-to-read-a-case-study/, fetched 2026-09-09, HTTP 200.
- https://socialcasestudies.com/blog/survivorship-bias-case-study-library/, fetched 2026-09-09, HTTP 200.
- https://socialcasestudies.com/blog/case-studies/, fetched 2026-09-09, HTTP 200.
- https://socialcasestudies.com/submit-your-case-study/, fetched 2026-09-09, HTTP 200.