A case study number is only as good as what backs it up

A case study number is only as good as what backs it up

A case study is open, a number in bold near the top. Before deciding whether it is impressive, ask what on this page would let you check it. Often nothing does. That is not the same as the number being wrong, but it changes the weight it carries.

What ‘verifiable’ actually means here

Verifiable does not mean true. It means you have been handed something to check against. A dated export with the account handle visible is checkable in a way a sentence is not. It can still be cropped at a flattering edge, or pulled from a window chosen after the fact.

The distinction doing the work is between a source and a citation. “According to our analytics” names where a number came from. It does not show what that place reported. A chart with the tool’s interface around it, a date range in the corner and an account name on top is a different object. Both may describe the same result; only one lets you act on it. It is the first thing to check when you read a marketing case study.

Four levels of evidentiary weight, and what each one actually proves

This orders claims by checkability, not honesty. A level four claim can be accurate; a level one export can mislead.

  1. Dated export or dashboard image with account identity visible. Proves a number came from a real measurement system, for a named account, over a stated span. Does not prove the span was chosen before results were known, or that the account is typical.
  2. Undated or unlabelled screenshot. Proves something was measured somewhere. Does not prove which account, what period, or that the visible portion is the whole series. An image without a date range is a picture of a number, not evidence for it.
  3. A named tool, with nothing to inspect. Proves the writer will say where the figure came from, which often reveals the metric definition in use. Does not prove what it returned.
  4. A bare number in prose. Proves the agency will stand behind the figure in public, and nothing else. You are asked to take it on trust, a reasonable ask and hard to audit.

The same number, shown four ways

What follows is hypothetical, invented for illustration and belonging to no real agency or brand. A garden centre chain’s agency reports a hypothetical 140 percent lift in engagement rate over one quarter.

Level one. An export showing the chain’s handle, a range of 1 April to 30 June, engagement rate plotted weekly against the prior quarter. You can confirm the figure matches the plot and see whether the rise was gradual or a spike. You cannot tell whether the earlier quarter was a weak baseline.

Level two. The same chart, cropped so the axis labels and account name are gone. You can see a line rising. You cannot confirm the account, the period, or that this is not a favourable slice.

Level three. The sentence “engagement rate rose 140 percent, per our social analytics platform.” You learn which system produced the figure, which narrows how the metric was defined, and nothing about what it returned.

Level four. The sentence “engagement rate rose 140 percent.” You know what the agency claims, and no further work is available to you. The hypothetical result is identical in all four versions. Only your ability to interrogate it changed.

What sourcing actually looks like across this library’s own case studies

This argument should be turned on this library too. On 4 September 2026 I opened every case study on the library index page. None of those checked shows a screenshot, dashboard image or export. Every one is a prose summary naming the agency and linking to its original write-up.

Within that, sourcing is not uniform. Some state a clear window, such as eight days or six months; others report a change year-over-year without naming the years. Some give both endpoints of a growth figure; others only the percentage. A few describe the client by category rather than by name, removing the identifying detail level one depends on. None names an analytics platform.

These summaries sit at level four, with one qualification the scale misses: each points at the agency’s published original, a pointer to evidence rather than evidence. Whether that destination shows its work, you have to check.

The gap itself is the useful signal

The temptation is to treat the bottom of the scale as a red flag. Resist it. Prose is the convention of the format, and level four writing is usually just the default. A missing source is not evidence of a false claim, only that you have nothing to check it against.

What it does tell you is how the writer expects the claim to be received. A dated export is written for someone who might go and look; a bare percentage for someone who will take the writer’s word. That is independent of accuracy, and separate from which case studies get published at all.

A short checklist for weighing the next claim you read

  • Is a date range visible? Not “over three months”, but an actual start and end.
  • Is there an identifying detail? An account name, handle or channel title. Without one, an export could belong to anyone.
  • Are numerator and denominator both shown? A finished percentage withholds half the arithmetic. Growth from a tiny base looks identical to growth from a large one.
  • Does the claim name a tool but show nothing from it? That is level three: useful for metric definitions, not evidence.
  • If there is an image, does it survive being looked at? Axis labels, a visible baseline, the quarter before the campaign. Try reading a results chart before the caption.

Try the scale on something real

Browse the case study library and place each headline figure on the four levels, noting what would move it up one. If you publish case studies yourself, submit your own.

FAQ

Does a dashboard screenshot prove a case study result is true?

No. It proves a number was pulled from a real measurement system. It does not prove the window was chosen before anyone saw the results, or that the account is typical of the agency’s work. Verifiable and true are separate claims; a screenshot settles only the first.

Is an unsourced number automatically suspicious?

No, and that reading is worth resisting. Plenty of accurate results are reported in prose because prose is the convention, not because anything is concealed. An unsourced number gives you less to work with, a fact about your position rather than its truth.

Sources

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *