A modeled result is not a result the agency delivered

A modeled result is not a result the agency delivered

You are halfway down a results page when the grammar starts to bother you. The numbers are specific, the chart is tidy, and no sentence says the campaign ran. Nobody ran it. Somebody checked what it would have done.

The sentence that gives it away

The distinction is one line long. “We ran this and it worked” is a report. “We checked, and it would have worked” is a calculation. Both can be honest. They do not carry equal weight.

Four phrasings do the quiet work. “Would have generated” is a conditional dressed as an outcome, so no campaign produced the number. “Applied retroactively” means the rule was laid over data that already existed. “Modeled on historical data” names the input and leaves the output vague. “Our framework, tested against last year” sounds like validation, except last year already had its answer.

A backtest is a hypothesis run over data that already exists, the general definition being testing a predictive model on historical data, a special case of cross validation over earlier periods. Whoever wrote the hypothesis could see how the story ended, making this the mirror image of a projected number is not a reported result, pointed backwards.

Why a backtest is easy to make look good

Consider what the builder can adjust: the rule, the window, the audience definition, the metric that counts as success. Each can be changed and the calculation rerun as often as patience allows. The finished chart shows the version that survived and records nothing about the discards.

Overfitting is the name for that endpoint, a model fitted so closely to one set of data that it describes that dataset rather than the underlying behaviour, and so predicts new observations poorly. Testing repeatedly and reporting only what looks good is the failure mode catalogued under data dredging.

A backtest also holds information no live operator had: which posts landed, which formats the algorithm favoured, which audiences proved responsive. A strategy written with that knowledge is graded on a period it has already read, the same problem as choices made after seeing the data. None of this needs intent to mislead. The format flatters the result on its own.

A worked example, with hypothetical numbers

Here is a hypothetical, with no real agency, brand, or account behind it. Picture an agency holding twelve months of a client’s past posts in a spreadsheet, asking what a rule of posting the best performing format three times a week would have produced.

The answer is close to guaranteed before the arithmetic starts, because “best performing format” was filled in by reading those twelve months. The rule was written after the winner was known, so the model is marked against the answer sheet it copied. A flattering number proves the sums were done correctly.

For this to become a result, the rule has to run forward, across months the model has never seen, with what happened reported. Until then the hypothetical number describes a decision the agency might make, not one it delivered.

What a backtest cannot produce, and a live campaign can

Work that ran leaves physical traces. Calculated work leaves a different set. If the campaign happened, some of this exists:

  • Dated posts still visible on the account, inside the window described.
  • Platform reporting covering the campaign period, with the date range shown.
  • Invoices, or ad spend records, for the flight.
  • Named start and end dates, stated without being asked for.

A backtest produces a spreadsheet, a chart, and a stated methodology. So ask one question: on what dates did this run, and can I see the account during that window. A model presented honestly answers that it did not run, which is not a scandal, only a different claim. The narrower case, where a platform models part of a campaign that did run, is modeled conversions reported as measured.

The gap between a backtest and a forward result

The remedy is out of sample testing. Hold back a slice of time the model was not built on, run the rule against it, and compare the prediction with what that period did. Cross validation, also called out of sample testing, exists to estimate how well an analysis generalises to unseen data.

A backtest paired with one forward period is a far stronger claim than a backtest alone, and you can ask whether that period exists. Ask too how many backtests preceded this one, because survivorship applies twice. One that looked bad is deleted rather than published, so the models you read are a selected sample. It is a counterfactual presented as a measurement, the gap set out in the counterfactual no case study can show you.

Other fields separate the two claims formally. FINRA Rule 2210, which governs communications by its own member firms, says such communications may not predict or project performance, while permitting a hypothetical illustration of mathematical principles that does not project the performance of an investment or strategy. It does not govern marketing agencies or case studies. The line it draws, between illustrating a method and claiming an outcome, is the line you want.

What a backtest legitimately supports is a decision about what to try next, not a claim about what an agency achieved.

Reading the claim back to yourself

Three checks, in order. Read the verbs, because conditional tense in a results claim is the cheapest tell there is. Look for dates, a start and an end, and treat their absence as the more telling signal. Then ask for the artefact: the account, the reporting, the invoice.

A modeled number can be careful, interesting, well built, and still not evidence that this agency delivered anything. You are entitled to know which of the two claims is on the page before you weigh it.

Reading more of this

Every entry here is credited to the agency that published it, its numbers stated as theirs. Browse the case study library to read claims side by side, or submit your own.

FAQ

What is a backtested result in a marketing case study?

A number produced by applying a strategy to a period that has already happened, rather than by running it and measuring what followed. No campaign took place.

Is backtesting dishonest?

No. It is an ordinary way to develop a hypothesis. The problem is presentation, reporting a modeled outcome in the voice of a delivered one.

Sources

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *