The One Follow Up Question That Tests Any Case Study Number
You have just finished reading a case study, the headline number looks good, and your cursor is sitting in a pitch deck waiting for you to type it in. That moment, not the reading you already did, is where most bad numbers get repeated.
The moment this habit belongs to
This is not about how to read a case study. This site already covers how to read a marketing case study without being misled, a slower, earlier skill of working through the whole document and noticing what it discloses and skips. The habit here happens later and takes seconds: the instant after you have decided the number is worth using and are about to type it into your own document. Nothing has gone into your deck yet. This is the last checkpoint before it does.
The distinction matters because the two moments fail differently. You can read a case study carefully, agree it is well put together, and still repeat a number that will not survive the first question anyone asks about it. Close reading catches sloppy documents. This habit catches polished ones.
Why passive reading is not the same test
Reading a case study checks whether the claim holds together on its own terms. Does the number match the chart. Does the language stay consistent from headline to body. A document can pass every one of those checks and still rest on a period, baseline, or exclusion that was never disclosed, because none of those checks require comparing the claim against anything outside the page itself.
A follow up question does the opposite. It asks whether the claim survives contact with information the document chose not to include. Most published case studies are written to survive being read start to finish without friction, a lower bar than surviving one specific question about what is missing. That gap is where an unsafe number lives undetected.
The three questions worth drafting in your head
You do not need a checklist. Draft three questions mentally against any headline number in the seconds before you repeat it.
- What was excluded from this number? A result can be accurate and still describe a narrower slice of activity than the headline implies, for instance one channel out of several.
- Over what period was it measured? A percentage or total means something different across four weeks than across a full year, and a headline rarely states the window next to the number.
- Compared to what baseline or starting point? A lift or multiple is only meaningful next to whatever it started from, and a case study that leads with the multiple and skips the baseline has handed you half a claim.
A hypothetical number and the question that unravels it
Picture a hypothetical case study, invented for illustration only, leading with “engagement rate up 180 percent after the new content strategy launched.” Read alone, the sentence is clean and consistent with the rest of the document. It survives a read.
Now draft the baseline question: compared to what starting point. If the case study never states the account’s prior engagement rate, that question has no answer in the document. An account starting from a very small base can post a 180 percent increase from almost nothing, a different claim from the same increase on an account already performing well. The honest, qualified version would state both figures: hypothetically, “engagement rate rose from an agency-reported 0.4 percent to 1.1 percent,” letting a reader judge the size of the move instead of taking a multiplier with no anchor.
A hypothetical number that survives the question
Now picture a second, contrasting hypothetical case study, again invented for illustration only: “over a 12 month retainer, an agency-reported client saw monthly leads rise from a baseline of roughly 40 to roughly 95, driven by organic posting, with paid spend excluded from this figure.” Every part of the question already sits inside the sentence: period, baseline, exclusion.
Answering the question before anyone asked it is itself worth noticing. An agency stating its own baseline and period up front is behaving as though it expects a skeptical reader, unlike one that leads only with the multiplier and hopes nobody asks what it multiplied. It does not make the result more impressive. It makes the claim easier to trust exactly as stated, because nothing is left for the follow up question to expose.
Why this is a private exercise, not a message to send
None of this involves contacting anyone. Drafting the question in your head is a complete, self-contained habit, and you can run through all three without ever reaching out to the agency that published the case study. The question stays inside your own reasoning, not in an email or a comment.
If the mental question turns up something serious, that is a signal to slow down, not a reason to send a message. Actually requesting the underlying numbers behind a claim is a later, separate move, and it belongs to a different moment than this one.
What it feels like when a number fails
The practical tell is simple: you try to answer your own follow up question using only what the case study gave you, and you cannot. Not because the answer is bad, but because it is not there. That gap, not a bad number, is what makes a claim unsafe to repeat as stated.
This is a different outcome from getting an answer you do not like. A case study that discloses a modest baseline, a short window, or a narrow exclusion has actually passed the test, even if the number looks less dramatic in context. The failure mode this habit catches is silence, not disappointment. An unanswerable question means the claim is incomplete. An unfavorable answer just means the claim is smaller than the headline sounded, a different and far less common problem.
Before you repeat someone else’s headline number, it helps to have seen how a range of them hold up. Browse the case study library for agency-credited examples, or if you have a result of your own that already states its baseline, period, and exclusions clearly, submit your own.
FAQ
Is this the same as asking the agency for their raw numbers?
No. This is a private mental exercise the reader runs alone, with nothing sent to anyone. Actually requesting the underlying data from an agency is a later, separate step, appropriate once the mental question has turned up something worth pursuing.
What if I can’t tell what was excluded, over what period, or compared to what?
Not being able to answer using only the document in front of you is itself the answer. The number is not yet safe to repeat as stated, and the honest move is to qualify it clearly or leave it out of your own deck or report entirely.