The ‘Before’ Picture in a Case Study Is Also a Choice

Suppose a case study reports engagement up 900 percent, without saying which period it was compared against. That is not a missing detail. It is half the claim: the “before” in a before and after comparison is a date somebody chose.

The percentage in the headline has a hidden second number

A percent change is a fraction: after minus before, divided by before. The study prints the answer, not the denominator. Hold the gain still, move the denominator, and one absolute improvement can be reported at almost any size: a low before makes an ordinary gain look dramatic, a typical one makes it look modest. Without the month underneath, you cannot tell which you have. See how percent change is calculated.

A hypothetical worked example: one gain, two baselines

This is invented: no such account or agency, no reported numbers. A hypothetical retail account averages 1,500 saves a month. One month it goes quiet during a rebuild and saves fall to 210. New work starts, and it settles at a hypothetical 2,100 saves.

Against the typical month, that is up 40 percent. Against the quiet month, up 900 percent. Same invented account, same 2,100 saves, same work. Only the prior month it is measured against changed, moving both the gap, 600 saves or 1,890, and the percentage. The improvement did not change. The reporting choice did.

The four kinds of “before” an agency can choose, and what each one implies

Comparison points fall into four rough groups. Spotting which is in play is most of the skill.

• The worst prior month, or a slump. Maximises the percentage without extra skill, because the denominator does the work. Not always cynical: a slump is often why the client went looking for help. The honest version names it, saying which quarter was the low point and why.
• A typical or representative period. A trailing average of the prior six or twelve months, or the same span a year earlier so seasonality cancels. The fairest comparison available and the least flattering to headline, so finding it should raise your confidence in the rest of the page.
• The moment just before a specific intervention. The week before a rebrand shipped, a redesign, a budget increase. The starting number may have been perfectly representative. The issue is that the date bundles the agency’s work with everything else that began then.
• A period chosen for its length rather than its representativeness. One slow week standing in for a quarter. Short windows swing harder, so a low start is far easier to find among weeks than quarters. A week of before against six months of after is not the same measurement on both sides.

Why “the day before the redesign” is a different claim than “a typical month”

The slump baseline is a question about how low the number was. The intervention baseline is about what else was different. When a study anchors to the week before something shipped, everything after lands on one side of the line: a new site, a pricing change, a seasonal upswing, the agency’s work. The result may be excellent, but it is no longer a measurement of one thing.

Sample size rides along as a separate problem: a day or a week is thin enough that ordinary variation supplies part of any difference.

What to ask before you accept the baseline

1. What calendar period does the before figure cover, and was it typical? Not “the previous period” but actual dates, and whether that stretch was normal for the account or a known low point.
2. Was the same before period used for every metric? A study reporting reach, engagement, follower growth and conversions has four chances to pick a starting point, and nothing forces it to use the same one twice. Reach against one month, engagement against a softer one, conversions against a fortnight when checkout was broken: each figure survives its own audit, and the page is a set of best cases stitched together. One stated window across all four is a good sign; four unstated ones are a strong sign the other way.
3. What does the result look like against a trailing average? Six months or twelve, instead of one chosen month. If the percentage collapses when the baseline widens, the figure was mostly a statement about the baseline.

None of this needs the number to be wrong, only for it to arrive with the period it was measured against. See also how to read a case study without being misled.

This is not the same lever as a naturally bad quarter bouncing back, or the attribution window

Two nearby mechanisms get confused with this one. The first is a bad quarter recovering on its own, a property of the data rather than anyone’s decision: an unusually poor period tends to be followed by a more ordinary one, intervention or not. Baseline selection is a choice made afterwards about which period goes in the denominator. The two compound, and need separate scrutiny.

The second is the attribution window, the rule deciding which conversions get credited to a channel and for how long. That is about which events belong to the number; baseline choice is about which period the number is measured against. Underneath both sits what a case study can’t show you.

Read a few side by side

Baselines get easier to spot after reading a few side by side. You can browse the case study library, each entry credited to the agency that published it, or submit your own.

FAQ

Is picking a low before period always dishonest?

No. A genuine turnaround starts from a genuine low point, and describing it against that point is accurate. The problem is the undisclosed baseline, not the low one. A study that names its worst quarter is being straight with you. One reporting a jump that size without naming the period is not.

What is a fairer comparison than a single before month?

A trailing average across several months, or a full prior period the same length as the after period. A single low month is easy to find inside a year, a low twelve month average is not, so widening the window makes cherry-picking much harder.

Does a case study have to disclose its baseline period?

There is no rule requiring it and no format that makes the omission visible, so the burden sits with you to ask. Treat a missing comparison period as an open question rather than bad faith, and ask for the dates. The difficulty of getting them tells you a lot.