One Store, One Market, One Account: Out of How Many?

A case study names a store: the Elm Street branch, the Manchester market, the account on the second slide. You get a before number, an after number, and the play between them. What you rarely get is how many other stores or accounts ran the same play and were never written up.

The story a single winning unit tells, and the one it doesn’t

A named store, market, or account is not automatically representative of the rollout it came from. It might be. It might also be the best of a group, chosen for publication precisely because it was the best, and from the writeup alone you cannot tell which, because the number that settles it, how many parallel units ran the same play, is missing.

Picking a favorable stretch of time inside one unit’s own history is a reporting window problem, and choosing a favorable window is also a choice. This one selects across space instead: several units run at once, one gets published, and the featured store’s own numbers may be entirely honest. The selection happened a level up, in which store got featured.

Why the same play often runs in more than one place at once

Parallel testing is a common structure, described here in general terms, not as a claim about any specific company, agency, or count.

A brand with several locations has an obvious reason to try a new approach in more than one at a time. Testing in one store and waiting a quarter costs a quarter. Testing in six costs the same quarter and returns six readings. Agencies sit in the same position from the other side: one that develops a playbook applies it across the accounts where it fits, so by the time a case study is written it may have run on several, and only one is the subject.

A hypothetical rollout: what changes between 1 of 3 and 1 of 40

Picture a hypothetical retail group whose featured location reports a hypothetical 62 percent rise in monthly online orders over six months. That hypothetical figure stays fixed below. Only the denominator moves.

Read it first as one of three hypothetical parallel locations. The other two exist, running the same approach, and both would have to be dramatically worse to drag the group to nothing. Top of three hypothetical units means the play produced a strong result out of a handful of tries.

Now read the identical hypothetical 62 percent as one of forty hypothetical parallel locations. Locations differ for reasons unrelated to the play: local competition, a new development nearby, one unusually good manager. Run any approach across forty hypothetical locations and ordinary variation alone will hand you a standout that looks like a result, sitting above thirty-nine hypothetical locations that barely moved. The store recorded the number. What it cannot support is the sentence a reader wants to write next, that the play will do the same elsewhere. The bigger the pool a winner was drawn from, the more of its margin belongs to the draw rather than the play.

Questions that surface the denominator

• How many locations, markets, or client accounts were running this approach during the period covered? Not how many exist. How many ran the play.
• Was the featured unit the first attempt, or was it chosen after results came in from several?
• Does the writeup say how this unit was selected? Stated criteria are a good sign; silence is the default.

An honest disclosure is short. It gives the total number of units the approach ran on, and either says why this one was chosen, the largest site or the only one with clean pre-period data, or that it was the only one run.

Not the same problem as a small sample or a cherry-picked window

A small base inflating a single test’s percentage happens inside one test: forty followers become sixty and the writeup says fifty percent. A window problem happens along one unit’s timeline: twelve months exist, the writeup shows the best ninety days. The denominator problem happens across units at one moment: forty full attempts run, one gets a page.

They are three separate failures and one case study can carry all of them. A store selected as the best of a large rollout, reporting its strongest quarter, on a base of a few hundred followers, has all three, and each needs its own question, which is easier with the general framework for reading a case study in hand.

When a single unit is legitimate evidence anyway

The objection is to the undisclosed denominator, not to publishing one success. Naming a single unit is honest in three situations. It was genuinely the only unit that ran the approach, and the writeup says so. It was an explicit pilot with the scope disclosed at the top, as in one of four pilot markets. Or it adds that outcomes elsewhere varied, ideally with the range, which is rarer than it should be and more persuasive, because a reader shown the losses has reason to believe the win. That applies to collections of published wins too, since the library itself is a selected sample.

What to do with a case study that won’t answer the question

Ask. Send the agency or brand one line: how many locations, markets, or accounts ran this approach at the same time? Until that is answered, hold the published figure as a best-case anecdote rather than an expected outcome. You are not calling it false, only treating it as a number whose selection you cannot see.

Every number in the library is credited to the agency that reported it, never restated as an industry benchmark, so you read each claim as its author made it. Browse the case study library and try the question there, or submit your own work.

FAQ

If a case study won’t say how many units ran the same approach, does that mean the result is fake?

No. A fabricated number and a real number drawn from a selection of unknown size are different things. The unit almost certainly recorded what the writeup says it did. The open question is what it predicts anywhere else, and without the denominator that stays unverifiable rather than disproved.

Should every case study that names one store also state the total number tested?

Yes, or at minimum say whether this was the only unit that ran the approach. An agency may genuinely not track that count across accounts different teams manage. That answer is still useful: the selection was not deliberate, and nobody can reconstruct it now.