A Media Mix Model's Number Is Modeled, Not Measured

A Media Mix Model’s Number Is Modeled, Not Measured

You are reading a case study and the headline number is not a click count or a dashboard screenshot. It reads like “our model attributed $X in revenue to social.” No tracked conversions, no pixel, no click path, just a number a model produced. The question worth sitting with: does that number deserve the same trust as a measured result, or is it something else wearing the same clothes?

The case study line that sounds precise

A media mix model, MMM for short, is a statistical estimate, fit to aggregate spend and outcome data over a stretch of time, that assigns a portion of the outcome to each channel in the model. Nobody clicked a tagged link logged as coming from social; an analyst ran spend and sales figures through a regression, and the model produced a coefficient translated into a dollar or percentage figure.

This framing shows up more often now partly because click-based, pixel-level attribution has gotten harder to trust. Apple’s App Tracking Transparency framework requires an app to ask permission before it can track a person across other apps and sites, one platform-level mechanism behind that erosion. Your task, reading a case study built on a modeled figure, is to work out whether it is presented with the confidence of a measurement, when it is really the output of choices someone made building the model.

What actually goes into the model

An MMM’s output depends on decisions that could have gone another way. The channel list is a choice: whether social is modeled as its own line item or folded into a broader “digital” bucket changes what gets credited to it. The time window is a choice too: a model fit on six months of data can weight a channel differently than one fit on two years. Control variables are a choice too: seasonality, a price change, or a product launch can be added as an explicit control, or left out and absorbed into channel effects.

Why collinearity is the quiet problem

There is a specific statistical reason two reasonable analysts can look at the same data and land on different numbers, and it has nothing to do with carelessness. When two channels’ spending rises and falls together, a regression-based model struggles to tell which one is doing the work. If social and paid search both rose in the same weeks and fell in the same weeks, the model cannot cleanly separate their effects, and distributes credit between them somewhat arbitrarily. This is a known property of regression-based decomposition in general, not a sign of sloppy analysis, and it is built into the method whenever channels move together, which in a lot of real media plans, they do.

A case study citing an MMM number almost never says whether the channels it modeled moved independently or together, which is exactly what you would need to know to judge the figure assigned to social.

A hypothetical: one media plan, three model specs, three answers

Here is an entirely hypothetical illustration, invented numbers, no real agency, campaign, or model. Picture a company running social, paid search, and email over one hypothetical quarter, spending evenly across the three, with a modest revenue rise by the end.

Spec one treats social as its own channel, fit on twelve months of history, and assigns it a hypothetical $180,000. Spec two bundles social with display as one line, since the two moved together for part of the year, and credits that line $260,000 with no way to split out social’s share. Spec three keeps social separate but fits on only six months before the quarter, a window where social and email happened to rise together, and assigns social a hypothetical $95,000, less than half of spec one. Same media plan, three specifications, three answers, none dishonest. Again: these numbers are constructed purely to illustrate how specification choices move an output, and describe no real company’s actual result.

Questions worth asking before you trust the number

  • Does the case study disclose which channels were included, and which were folded into a broader bucket or left out?
  • Does it disclose the length of the historical window the model was fit on?
  • Does it mention any sensitivity check, whether the number held up under a different reasonable specification?
  • Is the number a single point figure, or a range with stated assumptions?

A case study that answers none of these is not necessarily wrong. It is asking you to take the modeled figure on faith, in a way a measured, counted number does not require.

What a more honest MMM claim looks like

Picture two ways a case study could present the same modeled figure, both illustrative, not real reported results. One says “our model attributed $X in revenue to social,” no channel list, no window, no caveat. The other states a range, say $X to $Y, built on a stated channel list and window, noting that the range widened under a different specification. The second version is more honest: a model disclosing its own uncertainty was tested against alternatives, not reported as the one number one specification produced.

Before accepting a modeled figure as a headline result, ask the agency what channels went into the model, what window it was fit on, whether the number was checked against a different specification, and whether it is a point estimate or a range. The same standard applies to modeled conversions reported as if they were measured, a projected number is not a reported result, and when two metrics rise together with no stated mechanism connecting them.

Read more before you trust the next number

Browse the case study library to see how modeled and measured claims show up side by side across different agencies’ published work, or submit your own if you have a result worth adding to it.

FAQ

Is a media mix model’s output the same as a measured result?

No. It is a statistical estimate sensitive to the channel list, time window, and control variables the analyst chose, and it deserves the scrutiny you would give any model output, not the confidence of a tracked or counted number.

What should I ask an agency that cites an MMM number in a case study?

Ask what channels were included and what got folded into a broader bucket, what window the model was fit on, whether the number was checked against a different specification, and whether it is a point estimate or a range with stated assumptions.

Sources

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *