Is That Lift Real, or Just the Metric’s Normal Noise?

A reported lift is a before number, an after number, and the percentage between them, offered as the effect of the work. Ask first what the case study almost never answers: how much does this metric move on its own, with nobody touching it?

The comparison a headline lift skips

“Up from X to Y” compares two points. Two points give a direction and a size, but they cannot tell you whether the metric moved more than it usually does. That needs what the sentence leaves out: how the metric behaved before either point was chosen.

Every metric has an ordinary range, the band it wanders around in, week to week, when nothing in particular is happening. So the question is not whether the number is big, but whether it is bigger than what this metric does on a slow week.

Why a metric moves even when nothing changes

Movement without a cause is not mysterious, and three ordinary sources cover most of it. Accounts do not publish identically each week, and a Tuesday post meets a different audience from a Saturday one. Platforms change how they rank and deliver content continually, without announcement. And a metric resting on few people or actions bounces around its own average by small sample arithmetic alone.

None of that describes a specific platform’s mechanics or any measured effect size, only the general reason counts and rates fluctuate.

A hypothetical range, and a lift that falls inside it

The numbers below are invented for illustration, drawn from no published case study and belonging to no agency.

Picture, hypothetically, a regional fitness chain. Its engagement rate drifts a little either way from week to week with no campaign running. Over a quiet quarter, in this hypothetical, the weekly figure sits between roughly 2.8 percent and 3.6 percent.

Now a campaign runs, and the hypothetical case study reports engagement up from 3.0 percent to 3.4 percent, a lift of about 13 percent. Both figures sit inside the band the metric already occupied, so the comparison cannot show the campaign worked. Two quiet weeks from before could have produced the same sentence.

Change that hypothetical after figure to 5.9 percent and it sits well outside the band. Now it is a number that wants an explanation. Still not proof, but it clears a hurdle the modest version never did.

What makes a metric’s normal range wide or narrow

The width of that band is not a constant. Two things drive it.

• The size of the base. A metric built on a small base swings much harder in percentage terms than one built on a large base. A few extra comments can move a small account’s rate several points. The fuller argument is in when the base is small, the percentage is just noise.
• The length of the window. A lift measured across a quarter is steadier than one across a single week, because a longer window lets good days and bad days cancel out.

What the case study would have to show you to make this checkable

Running this comparison needs disclosures most case studies omit.

• The period the baseline was drawn from, as dates, not as “before the campaign”.
• How many weeks or cycles it covers, so you can tell an established pattern from a single reading.
• Whether the before figure is one week or an average of several, and the same for the after figure.

Almost none provide these. The honest conclusion is then not “this is fake” but “this cannot be checked from what was published”.

A different question from percent versus percentage points

This is not the units problem. That separate error states one result two ways that sound very different, a few percentage points reported as a large percentage increase, covered in percent increase and percentage points. This test begins once that one is passed: a correctly stated lift still fails here if it is smaller than the metric’s normal wobble.

When a modest number is still worth believing

None of this makes a big number trustworthy or a small one worthless. A figure outside the ordinary range is not proof either: a run of unusually good weeks can happen for no reason, and the case study chooses which weeks to report. A lift repeated across separate periods beats one large lift seen once.

A neighbouring trap sits close by. When the before period was chosen for being unusually bad, the recovery may be the metric returning to its own average rather than anything the work produced, the argument in regression to the mean. Both read two chosen points as the whole picture.

Try it on a real one

This gets quick with practice. Browse the case study library and, for each reported lift, look for the baseline period and how long it ran. Published one yourself? You can submit your own. The wider checklist is how to read a marketing case study.

FAQ

How would I know how much a metric normally swings if the case study doesn’t say?

Usually you cannot, and that is the honest answer. Treat an undisclosed baseline range as a gap in the evidence rather than something to fill with a guess. A lift you cannot size against ordinary movement counts for less than one where the baseline is shown.

Does this mean small reported lifts are always meaningless?

No, and the distinction matters. “Not distinguishable from noise on this one data point” is a statement about what the evidence can carry, not a finding that nothing happened. The work may well have done something real, and on evidence this thin you simply cannot tell either way. A small lift that repeats across several separate periods is a different and stronger claim than the same lift reported once.

Is this just a different way of saying you need a control group?

Related, but not the same question. A control group answers what would have happened anyway, a comparison against a world you did not observe. This check comes first and is narrower: is the number even far enough from the metric’s ordinary movement to need explaining?

Sources

• When the Base Is Small, the Percentage Is Just Noise, fetched 2026-09-06.
• 100 Percent Better and One Point Better Can Be the Same Result, fetched 2026-09-06.
• Regression to the mean: why the bad quarter flatters the next, fetched 2026-09-06.
• How to Read a Marketing Case Study Without Being Misled, fetched 2026-09-06.