Why an Agency's Average Client Result Changed Year Over Year

Why an Agency’s Average Client Result Changed Year Over Year

An agency’s latest report shows a higher average client result than last year’s did. The obvious reading is that the work got better. But an average isn’t a measurement of one thing that improved. It is a summary of a group, and the group can change even when nothing inside it does. Before crediting the jump to sharper strategy, it’s worth asking a plainer question: is this the same group of clients measured twice, or two different groups wearing the same label?

An average is a claim about two things, not one

When a report says “the average client saw an 18 percent lift,” that sentence is shorthand for the average across the clients currently in the portfolio, over the period the report covers. Both halves of that phrase move independently. A client’s own result can rise or fall. Or the set of clients being averaged can change entirely, through new signings and lost accounts, while every retained client’s number stays put. A reported change in the average could reflect either cause, and most reports don’t say which.

Retail chains hit the identical confound: total revenue can rise because a company opened more stores, which says nothing about whether existing stores are doing better business. The standard response is a same-store or like-for-like figure, restricted to outlets that existed in both periods, because excluding new and closed locations is the only way to keep the comparison like-to-like. Agency portfolios face the same issue every time a roster turns over, but rarely get the same explicit treatment. One outlier account can skew a portfolio average too, a different mechanism worth keeping separate from this one.

A labeled hypothetical: five clients become five clients, but not the same five

The figures below are entirely hypothetical and illustrative, not a reported result from any agency.

Hypothetical Year 1 portfolio, five clients

Client Hypothetical result
Client A 18%
Client B 24%
Client C 12%
Client D 30%
Client E 6%

Hypothetical Year 1 average across these five clients: 18%.

Between Year 1 and Year 2, Client D and Client E leave, and Client F and Client G sign on. Clients A, B, and C stay, each posting the identical hypothetical result it posted the year before.

Hypothetical Year 2 portfolio, five clients

Client Hypothetical result Status
Client A 18% retained, unchanged
Client B 24% retained, unchanged
Client C 12% retained, unchanged
Client F 40% new this year
Client G 26% new this year

Hypothetical Year 2 average across these five clients: 24%.

The average moved from a hypothetical 18 percent to a hypothetical 24 percent. Yet zero retained clients’ individual results changed. The entire movement came from which clients were in the denominator, not from any client doing better work. A reader shown only the two headline averages has no way to tell this apart from a portfolio where every client genuinely improved.

What a fair year over year comparison actually requires

The only version of this comparison that isolates a genuine performance change is a matched-cohort comparison: the set of clients present in both periods, compared against itself across both years, with anyone who joined or left excluded. In the hypothetical above, that means looking only at Clients A, B, and C, which correctly shows no change, because none occurred. That is the comparison a reader wants when the question is “did this agency get better at the work,” not “did its current book of business happen to include some strong results.”

A portfolio average with no stated client count, or a count that differs between the two years’ reports, cannot support a performance claim on its own. A count isn’t proof of a stable roster, but its absence means the comparison hasn’t ruled out composition change. How to read a marketing case study without being misled covers this habit more broadly.

Three things to check in a published year over year average

  • Client counts in both reports. Does each report state how many clients the average covers, and does that count match? If either is silent, or the counts differ, the comparison cannot rule out a roster change.
  • Whether clients are identifiable. Are clients named or otherwise consistently labeled, so a reader could in principle check whether the same accounts appear in both periods? A fully anonymous aggregate gives no way to verify retention.
  • Whether churn is disclosed anywhere. Does either report mention clients gained or lost, even outside the headline number? Disclosed churn, even without a matched-cohort figure, at least tells a reader the roster wasn’t static.

A moved average is not evidence of anything dishonest

Clients leave and new ones arrive for reasons that usually have nothing to do with an agency’s skill: budget cuts, acquisitions, a contract ending on schedule. A composition change in the average is a normal, largely unavoidable feature of that churn, not a sign of manipulation. An agency that publishes its current average without claiming the number proves improved skill has not misrepresented anything.

The risk sits downstream, in how the number gets used afterward. A pitch deck slide treating “our average client result went up this year” as proof the work has gotten better makes a causal claim the number alone doesn’t support, whether or not the agency’s own report made that claim. The same caution applies to a case study library itself: every case study library is a selected sample, shaped by which cases happen to be visible in it.

Read the Numbers Behind the Claim

Matching a claim against what backs it is the reason this library exists. Every entry credits the agency behind it and shows the work, so you can check for yourself whether a reported number holds up. Browse the case study library, or if you have one with the roster and methodology to back it up, submit your own.

FAQ

If an agency’s average client result went up this year, does that mean the agency got better at its job?

Not necessarily. The average could rise because the client roster changed rather than because any client’s own result improved, as the hypothetical roster swap above shows. The only way to know is to check whether the comparison used the same clients, or a matched cohort, in both periods.

How can I check whether an agency’s year over year average is comparing the same clients?

Look for three markers: whether both reports state a matching client count, whether clients are identifiable enough across both reports that retention could be checked, and whether either report discloses churn anywhere. Absence of all three doesn’t prove anything was hidden, but it means the comparison can’t be verified as like-for-like.

Sources

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *