A Subgroup Win Only Counts If the Cut Was Planned First
A case study says the campaign “worked best” for one audience. There is a bold number and a tidy explanation, and nothing about how everyone else responded. Two questions decide how much weight that claim deserves: was the audience chosen before the campaign ran, and how many other audiences were looked at first?
The case study that opens with the segment
Here is a hypothetical opening line, invented for illustration: “Engagement rose 3x among women 25 to 34.” Now look for what is missing. There is no overall engagement figure beside it, and no word on whether other groups rose, stayed flat or fell. Notice the absence of the whole-campaign number first. When the overall result is flat or unmentioned, the slice carries the entire claim.
That does not make the slice false. A subgroup result can be true and still not be evidence, because the question is how the subgroup was chosen, not whether the arithmetic is right. It is the sibling of a metric picked after the results came in: a different choice, made at the same bad moment.
The arithmetic of finding a winner by slicing
Take a hypothetical campaign with no real effect on anyone. Slice its results into 20 cuts across audience, region and format. A common convention calls a result notable if chance alone would produce it less than 5 percent of the time. With 20 cuts and nothing real anywhere, the expected number clearing that bar by luck is 20 x 0.05 = 1. One cut looks like a win.
If the 20 cuts were independent, the chance that at least one clears the bar is 1 minus 0.95 to the power 20. Since 0.95 to the power 20 is about 0.358, that is about 64 percent. Real cuts overlap, because one woman aged 29 sits in a gender cut, an age cut and a region cut, so read 64 percent as the direction of the effect, not an exact figure.
Twenty is modest. Age bands, gender, region, placement, format and day of week multiply together. Six age bands times two genders times four regions is already 48 cuts, before placement or format. Nobody has to hunt on purpose. Scanning a dashboard for what stands out is enough.
The general problem is called the multiple comparisons problem: test many hypotheses on the same data and false positives become likely. Its close relative, data dredging, presents the significant-looking results as though they had been the hypothesis from the start. One standard fix is the Bonferroni correction, which makes each test stricter in proportion to how many were run. You will not run it yourself, but it shows why the count matters.
Planned cut versus found cut
Statisticians call this the garden of forking paths. Each choice about how to slice, made after seeing the data, is a fork: this age band or that one, include weekends or not. The reported path is the one that ended somewhere interesting, and the dead ends are not in the case study.
| Planned cut | Found cut | |
|---|---|---|
| First appears | The brief or targeting setup, dated before launch | The results section |
| Reason given | Stated in advance: why this audience should respond differently | None until the result arrives |
| Other cuts | Planned cuts that failed are reported too | Not mentioned or counted |
| Overall result | Shown beside the segment | Absent or buried |
A planned cut can be reported honestly even when it fails. The tell is whether the study also reports the planned cuts that did not work. A pre-declared cut does not guarantee a real effect, though. It only removes the selection problem, and sample size still matters.
Why a plausible story does not rescue a found cut
For almost any winning slice, a sensible reason can be written afterwards: heavier sharers, early adopters, the right life stage. Each sentence is believable, and each was written knowing the answer.
Try a thought test. Suppose the best cut had been men 45 to 54 instead. Would the study have found an equally fluent explanation, more disposable income, a preference for plain offers? If yes, the explanation is decoration. It makes the claim feel settled without adding evidence.
Small slices make it worse
Each cut holds only a fraction of the campaign’s volume, so its rate swings more, and the largest swings are the ones that get reported. That is the effect in the post on when the base is small, the percentage is just noise, and slicing creates small bases by design. Also, when age and gender labels are inferred rather than collected, the segment is fuzzier than it looks, as the post on platform-inferred demographics explains. A separate problem, where segments and the headline number disagree, is about aggregation, not timing.
Five questions to put to a subgroup claim
- Was this audience named as a target before launch, and is there a dated document that shows it?
- How many cuts were looked at in total, and are the others reported or at least counted? It is the denominator question from the post on a winner out of how many were tested, applied to audiences.
- What did the overall result do, and is it shown beside the segment?
- How many people or conversions are in the winning cut, not just the percentage?
- Has the effect shown up again in a second campaign or a later period? A repeat is the strongest single answer. A one-off is a lead to test, not a conclusion.
What an honest subgroup finding looks like, and what to do with a found one
A hypothetical honest phrasing: “Overall result was flat. We looked at 12 audience cuts after the fact and one stood out; we treat it as something to test in the next campaign.” The confident version says only “the campaign drove a 3x lift among women 25 to 34.” The first gives you the overall result, the cut count, the timing and the status of the claim. The second gives you a number.
Your stance as a reader should be that you want the claim to be true, and then you check. A found cut is not worthless. It is a good reason to run a planned test with the audience declared in advance.
See how claims are credited
Browse the case study library to see how claims are credited to the agency behind them, or submit your own. Every number in the library belongs to the agency that reported it.
FAQ
Is a post hoc subgroup result always wrong?
No. It can be true. It is weak evidence because it was found after looking, so treat it as a hypothesis for a planned test. That describes the evidence, not the good faith of the people who reported it.
How can I tell if the segment was planned?
Look for dated targeting briefs or setup records, or a case study that states the audience goal in its objective section and reports the segment whether it worked or not. If the audience first appears in the results, assume it was found.
Sources
- Multiple comparisons problem, Wikipedia, fetched 2026-10-09
- Data dredging, Wikipedia, fetched 2026-10-09
- Bonferroni correction, Wikipedia, fetched 2026-10-09