Every Case Study Library Is a Selected Sample

Four case studies in a row, each closing on a number that made you sit up. By the fourth, that outcome starts to feel normal.

Four strong results in a row are not a base rate

The pattern belongs to the shelf, not to the work. A library is not a random sample of the engagements that happened, it is the subset somebody judged worth writing up.

This has a name. Survivorship bias is the error of concentrating on cases that passed a selection process while the cases that did not stay invisible, leaving conclusions drawn from incomplete data. It belongs to the family of selection biases, where cases enter the sample in a way tied to the outcome being measured. So a library is evidence that these outcomes happened under the conditions described, and not of what an agency typically delivers. Holding those apart is the whole job.

What a published case study has already survived

The selection is not one filter but a sequence. First, the engagement had to run to completion. Work that was paused, cancelled when a budget moved, or abandoned in a reorganisation never produces a result to describe, and some of that work was going well when it stopped. Second, the result had to be good enough to justify the hours of writing it up, which removes the outcomes that came out flat.

Third, the client had to agree to be named, or described closely enough for the study to make sense, so anyone under an NDA or a communications policy drops out whatever their result was. Last, the write-up had to survive comparison with the agency’s other published work, which removes the merely fine.

Every one of those decisions can be completely honest. No concealment and no motive is needed for the collection to end up skewed. The shape of the filter does it.

This is also why care does not catch it. You can be excellent at reading a single case study honestly, checking the baseline, the window, the attribution, and still miss this. Close reading examines the study in front of you, and cannot detect a filter on which studies exist.

The same agency, two different pictures

Here is a hypothetical example, with hypothetical figures belonging to no agency and no reported result. Picture an agency that ran 60 engagements over two years and published 5 case studies, each truthful.

The arithmetic is the point. Five out of 60 is roughly one engagement in twelve, and the other 55 are described nowhere. If most were modest, the published 5 sit at the top of the distribution and the impression is wrong. If most were similar, the impression is fair. The 5 read identically either way. These figures are illustrative only and belong to no real agency or result.

What this library is and is not

This library holds 10 published case studies, distributed as 4 in e-commerce, 2 in hospitality, 2 in local business, 1 in healthcare, and 1 in SaaS.

That distribution is a record of who chose to submit. It is not a map of where social media work happens or where it works best. Every number inside those studies belongs to the agency that reported it, never to an industry benchmark.

Questions that interrogate the collection, not the study

• How many engagements happened over the period the studies cover? A good answer is a denominator you can divide by.
• Who decided which work got written up, and on what criterion? A good answer names a stated rule.
• Is there a published example of a modest result? Its absence marks the threshold for publication.
• Do the studies cluster in one industry, budget level, or channel? A good answer claims nothing past that cluster.
• How old is the oldest study, and the newest? A collection that stops years back may document a tactic that stopped working.

Reading a library for its ceiling instead of its average

A library records a ceiling, an outcome reached at least once under conditions that favoured it. No library reports a median. The two support different decisions.

The ceiling is still worth something. It establishes that an outcome is possible, shows the mechanism that connected the work to the number, and exposes the conditions the result depended on, the audience already in place, the timing, the length of the run. A small number of observations supports a claim about what happened, not what typically happens.

So bring a better question to any impressive study. Not whether it works, but what had to be true for it to work, and whether any of that is true for you. Browse the case study library for mechanisms rather than a number to expect, and if you have published work of your own, submit your own.

FAQ

Does survivorship bias mean case studies are dishonest?

No. Every study can be accurate and fairly written while the collection still misleads, because the distortion comes from which work became a study rather than from anything a study says. No motive is required. The same pattern is documented as publication bias, where significant findings are published more often than null ones.

How can I find out how much work an agency did that was never published?

Ask for the denominator, how many engagements started in the same window the studies cover. You are asking for the size of the set, not for anyone’s failures. An unflattering answer is informative because it lets you divide. No answer is informative too.

Is a case study library still worth reading?

Yes, if you read it for mechanism and proof of possibility rather than expected value. The useful extraction is the set of conditions the result depended on, and then a check of which of those you have.

What is the difference between survivorship bias and cherry-picking a date range?

Cherry-picking happens inside one study, choosing the window that flatters a number, and a careful reader can catch it. Survivorship happens across studies, deciding which results become studies at all, which close reading cannot reveal.

Does this apply to award shortlists and best of roundups too?

Yes. The same mechanism operates on any curated collection, because something decided what entered it. A roundup of reported results is a ceiling too, so read it for what became possible once rather than for what a tactic normally returns.

Sources

• Survivorship bias, Wikipedia, fetched 2026-08-25.
• Selection bias, Wikipedia, fetched 2026-08-25.
• Publication bias, Wikipedia, fetched 2026-08-25.
• Sample size determination, Wikipedia, fetched 2026-08-25.