When the Headline Metric Was Picked After the Results Came In
A case study opens by telling you what the campaign was for. A few paragraphs later, a chart shows you what it achieved. Most readers assume those are the same metric, stated twice. Often they are not, and there is a way to check.
The gap between a stated goal and a charted result
Read the opening paragraph of a case study and you will usually find a sentence stating the objective: what the agency was brought in to move. Read the results section and find the metric actually charted, the one attached to the big number. Sometimes these are the same metric in different words. Sometimes they are not, and the case study rarely explains why.
That gap is not automatically dishonest. A campaign built around one objective can still produce its most interesting number somewhere else. But naming one objective and charting a different metric needs an explanation, and most case studies do not offer one; they present the charted number as though it had always been the point.
This is not the same as a campaign having several goals at once. The tell here is narrower: specifically when the metric charted is not the one named as the reason the campaign was run. Five stated goals with a chart showing all five is not this post’s subject; one stated goal with a chart showing a different, unstated metric is.
Why this is a different problem from a mismatched goal
The distinction is worth naming directly: a goal-metric mismatch asks whether this is the metric the agency was hired to move, or whether the result fell short. That compares the result to the goal.
This post asks about timing instead: was this metric chosen before the campaign started, or only after, because it happened to be the number that moved most? That is a claim about sequence, not the size of the result.
Sequence matters on its own. A campaign can hit its stated objective and still ship a headline cherry-picked from several metrics that happened to move. Both can be true at once: the pre-committed goal was met, modestly, while a bigger, unplanned number was promoted to the headline instead. A result can be true and still be the winner of a contest the reader was never told was running.
The general problem this is one case of
This pattern has a name outside marketing: data dredging, testing many outcomes against the same dataset and reporting whichever crossed a threshold, without disclosing how many others were tested and did not. The Wikipedia article on data dredging lays out the concept: test enough outcomes against the same data and something will look significant by chance alone, and a report showing only the winner hides what a reader needs to judge whether that win means anything.
Marketing has no dedicated term for this, so this post applies the statistical concept to one document: the case study, moving the same mechanism from a research dataset to a dashboard of engagement, click, and conversion numbers.
A hypothetical walk-through of the tell
The following is entirely hypothetical. No real agency, campaign, or company is described, and none of the numbers are reported results.
Picture a hypothetical agency whose brief states one pre-campaign objective before launch: increase clicks to the client’s website from social posts. That is the goal the case study’s opening paragraph would eventually describe as the reason the campaign was run.
After the campaign, the agency pulls every number the platform reports: website clicks (the original goal) up a hypothetical 9 percent, comment volume up a hypothetical 40 percent, share count up a hypothetical 22 percent, follower count up a hypothetical 6 percent. Only clicks was named as the goal, and it is the smallest of the four.
The published case study headlines comment volume, “a 40 percent lift in engagement.” The click number, the actual objective, appears nowhere. The headline was chosen after all four numbers existed, not before anyone knew which would win.
Again: the agency, the client, and every percentage above are constructed for illustration, not a real result.
What to check in the case study itself
- Find the sentence, usually near the top, stating what the campaign was supposed to do. Write down that metric.
- Find the metric named in the headline and plotted in the results chart. Check whether it is the same metric.
- Check whether the case study discloses other metrics tracked that did not move as much. Their absence is informative.
- A case study naming its pre-committed goal and showing that goal’s own result, even modest, is stronger evidence than a bigger number on an unstated metric.
What this tell does not prove
A headline metric that differs from the stated goal is evidence of possible post-hoc selection, not proof, and not automatic disqualification. Campaigns legitimately discover wins they did not plan for. A campaign built around clicks can honestly surface a jump in shares nobody predicted, and reporting that is not misconduct. The problem is when the original goal is never mentioned, so the reader cannot tell whether the new number replaced a disappointing one.
The strongest version of this tell is silent about the original goal entirely, removing the reader’s ability to check anything.
Read further
Read a results chart before you read the caption, and see why the best metric on a dashboard is rarely the real story. Browse the case study library or submit your own.
FAQ
Is a mismatch between the stated goal and the headline metric always a red flag?
No, it is a prompt to look closer, not a verdict. The honest version discloses the original goal’s own result alongside the new headline, rather than replacing one with the other.
How is this different from a case where the reported result just misses the hired-for goal?
That is a question about the result versus the goal. This post is about sequence: whether the headline was chosen only after the data existed.
Is picking the best-performing metric after a campaign the same as data dredging?
Yes, in mechanism. Data dredging means testing many outcomes against one dataset and reporting whichever crossed a threshold, without disclosing the rest. Choosing a headline from several metrics that moved applies that logic to a case study instead of a dataset.
What should I look for first when reading a case study with this in mind?
The sentence describing the pre-campaign objective, usually near the top, compared against the metric actually charted in the results. If they are not the same metric, that gap is the thing to interrogate.