How Old Is the Youngest Cohort Behind That Result?

How Old Is the Youngest Cohort Behind That Result?

A case study says the campaign delivered a strong result “within 30 days of launch”, for a product people usually take weeks to decide on. You want to believe it, but is that figure final, or a count taken while most of the story was still unfolding?

The number is a photograph, not a final score

A reported result is a count taken on a particular date. Look at who is inside it. Someone exposed on day 28 has had two days to act. Someone exposed on day 1 has had four weeks.

Two terms help. A cohort is everyone exposed to the campaign, or acquired, in the same period. Cohort age is the time elapsed since that group was exposed, measured at the reporting date.

The whole post trains one question: how old is the youngest cohort in this number, and how long does this product normally take to convert?

One campaign, two reporting dates (hypothetical)

The table below is HYPOTHETICAL. The figures are invented to show a mechanism. They are not from any case study in the library or from any agency, and they are not a benchmark for how fast conversions arrive. Picture 1,000 people exposed to the campaign and 1,000 exposed in an earlier baseline period, counted at the same ages.

HYPOTHETICAL: cumulative conversions per 1,000 people exposed, by cohort age (invented numbers)

Cohort age Baseline period Campaign
Day 7 40 42
Day 14 65 68
Day 30 90 95
Day 45 100 105

Now the trap. Suppose the baseline is quoted from its final count, 100 at day 45, and the campaign is quoted at day 7, which is 42. The campaign appears to have converted 58 fewer per 1,000, a drop of 58 percent (58 divided by 100). Compare like with like instead. At day 7 it is 42 against 40, about 5 percent higher. At day 45 it is 105 against 100, also 5 percent higher. Same data, opposite story.

A fair comparison needs equal cohort age on both sides.

Why the lag exists, and what the platforms say about it

Google’s help page on conversion windows says that if you set a 7 day window, any conversion that happens more than 7 days after the ad interaction won’t be recorded, and so won’t appear in reports. It also describes a time lag report in attribution reports, which shows how the time between people’s ad interactions and their conversions varies.

Keep two things apart, because they are easy to confuse. A conversion that is delayed but will still be counted once it lands belongs to an immature cohort. A conversion that falls outside the window and will never be counted is a window choice, which is a different problem, covered in the attribution window is a choice.

Some outcomes lag more than others. Purchases with a consideration period come in slowly. Free trials convert to paid only when the trial ends. Signups can become qualified leads long after the signup. This is logic, not measurement, so ask about the actual timing.

Early cut-offs can flatter as well as understate

  • Understating: conversions still on their way mean the numerator is incomplete, so cost per acquisition looks too high and ROAS looks too low at an early date.
  • Flattering: refunds, returns, chargebacks, trial cancellations and early churn arrive after the original conversion. If they have not landed yet, revenue and retained customers look better than they will end up.

As a lookup (direction depends on the business and the exact metric reported):

Metric Way an immature cohort tends to push it, and why
Cost per acquisition Too high, because late conversions are missing from the count
ROAS Too low if purchases are still arriving, too high if refunds have not landed
Retention Too high, because early churn has not happened yet

A single blended number can also mix old and young cohorts: the oldest drags the average toward maturity, the youngest drags it away. It is also worth checking which “conversion” is being counted, since five different funnel stages can all be called that, each with its own lag.

What a careful case study discloses, and what you can ask for

Treat these as questions to put to the case study or to whoever produced the reporting:

  1. What was the report date, and what was the last exposure date? The gap is the age of the youngest cohort.
  2. How long do conversions normally take for this product?
  3. Was the baseline measured at the same cohort age as the campaign?
  4. Were refunds or cancellations netted out, and as of which date?
  5. Is there a later update to the number?

Read vague phrases with care. “In the first 30 days” could mean 30 days of exposure, with every cohort at least a month old once the count is taken. It could equally mean 30 days of results, where the youngest cohort is only hours old.

An honest disclosure sounds like this (hypothetical): “Figures were pulled 45 days after the last exposure date, with refunds netted out to that date. Trials still running at that point are excluded.” A red flag is a precise early figure, down to the decimal, with no mention of when it was pulled.

An early number is not wrong, it is early. A sceptic wants the claim to be true, and so asks for the matured figure rather than dismissing the result.

Where this sits next to the other ways a window misleads

Cohort maturity is about how settled the data is inside the dates shown. It differs from a cherry-picked reporting window, which is a choice of which dates to show, and from a forecast: a projected number is not a reported result, because it is extrapolated rather than counted.

Practise the question

Browse the case study library and ask the cohort age question of each result you read. Or submit your own.

FAQ

What is conversion lag?

It is the gap between someone’s exposure to an ad or post and the outcome being recorded, and any report cut off inside that gap is incomplete. Google’s page on conversion windows explains how windows cap what is recorded.

Does an early result always understate performance?

No. Late conversions make a result look worse, but refunds and cancellations that have not landed yet can make it look better. Which effect wins depends on the metric and the business, so ask which direction applies.

Sources

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *