Survey Says Podcast, Dashboard Says Paid Social: Who Is Right?

Survey Says Podcast, Dashboard Says Paid Social: Who Is Right?

You are reading a case study. One chart, taken from an ad dashboard, says most conversions came from paid social. A line further down says many customers named a podcast when asked how they heard about the brand. They cannot both be the headline. Picking the one that sounds more honest is the wrong move.

Two systems, two different questions

Picture a hypothetical case study for an invented subscription brand. The dashboard export says paid social was credited with most of the new subscribers. The write-up then notes, almost in passing, that a good share of customers wrote “a podcast” in the post-purchase box. Nobody is lying. The sources answer different questions.

The survey asks what the customer remembers and chooses to say. The platform asks which tracked ad interaction came before the conversion, inside its window. One is a human report, filtered by memory and manners. The other is a log, filtered by what the platform can see. Attribution in general assigns credit to touchpoints under a chosen rule, so the rule shapes the winner (see the overview of attribution models).

Neither asks what caused the purchase. A remembered podcast may have changed nothing, and a logged ad may precede a purchase already decided. Everything below builds on that gap.

What the dashboard can see and what it cannot

A platform can only credit interactions it can observe or model. A podcast mention, a conversation at dinner, a search made after hearing a name: none of these leave a trace in an ad platform. The dashboard under-credits anything outside its tracking.

The tilt runs the other way too. Platforms commonly offer configurable click and view windows, and a platform credits itself for conversions that land inside them, including after a mere impression. If the buyer was already on the way, the ad still collects the credit. We cover the mechanics in the attribution window is a choice.

Some platform-reported conversions are estimated rather than observed. That is a separate problem, described in modeled conversions reported as if they were measured.

What the survey can see and what it cannot

A “how did you hear about us” question reaches things the dashboard never will, but it carries its own distortions.

  • Recall. People tend to remember the last or most vivid touch, not the first. Recall bias afflicts any retrospective questionnaire (overview).
  • Answer format. A free-text box and a dropdown produce different answers from the same people; wording and format shape responses (response bias).
  • Non-response. The people who answer are not everyone who bought. Answerers may differ from skippers in ways the survey cannot show.
  • Tidiness and social desirability. People name the respectable-sounding source. A podcast is easier to admit than a retargeting ad.

“I searched for you on Google” and “a friend told me” are answers about the final step, which can sit downstream of the real prompt. And a survey cannot see an ad the customer never consciously registered, which is exactly the situation display and short video impressions create. For the first thing to ask of any survey figure, who answered, read a disclosed sample size can still hide who was surveyed.

Worked example: the same 100 customers, two ledgers (hypothetical)

This section is HYPOTHETICAL. The figures are invented to show arithmetic. They are not drawn from any case study, any agency or any real channel result, and they are not a benchmark.

Channel Dashboard (customers, of 100) Survey (% of the 40 who answered)
Paid social 46 8
Podcast 0 28
Friend or word of mouth 0 22
Search 24 20
Email 10 4
Other or direct 20 18
Total 100 100%

Suppose only 40 of the 100 customers answered the survey. Each respondent is worth 2.5 points, so “podcast, 28%” means roughly 11 people, and nothing is known about the other 60. If they skew toward people who clicked an ad and moved on, the survey column overstates podcasts for the whole customer base.

Two stories can produce both columns:

  • A listener hears a podcast, later sees a retargeting ad and clicks. The dashboard credits the ad, the survey credits the podcast, and both report something true.
  • A friend recommends the brand, and the buyer clicks a paid search result. The dashboard says search, the survey says friend.

Which bias is the case study choosing?

A case study that shows only the dashboard has picked the method that credits trackable paid activity. One that shows only survey answers has picked the method that credits memorable or earned activity. Treat it as a pattern to check, not an accusation.

Rule of thumb: dashboards over-credit paid and trackable channels. Surveys over-credit memorable, recent and socially easy answers. When the agency’s own numbers are the only source, they are the agency’s numbers, and we state them that way, never as ours and never as an industry benchmark.

Questions to put to a case study that reports only one

  1. What was the exact wording and format? A holding-up answer quotes the question and says whether it was free text or a list. A weak one says only “customers told us”.
  2. What share of buyers answered? Good: responses against buyers. Weak: a percentage with no denominator.
  3. Could people give more than one answer? Good: stated, with shares explained. Weak: shares that silently sum past 100.
  4. What window did the dashboard use? Good: named click and view windows. Weak: no window given.
  5. Does the dashboard include view-based credit? Good: shown separately. Weak: one blended number.
  6. Was either figure compared with a holdout or other incrementality evidence? Good: a comparison group that did not get the activity. Weak: before-and-after charts.

If both views exist and the case study shows one, the missing one is the request to make.

Reading both together, and what still stays unknown

Where the two methods agree, confidence rises. Where they disagree, the gap is a pointer, not a verdict: it suggests which channel is mid-funnel or invisible to one of the systems. A channel big in the survey but absent from the dashboard hints at untracked influence. One big on the dashboard but small in the survey may have captured demand rather than created it.

Neither source proves cause. Only a test that withholds the activity from a comparison group can, which is the point of holdout tests.

Browse the case study library and put these questions to each result, or submit your own with both views shown.

FAQ

Is self-reported attribution more accurate than platform attribution?

Neither. Surveys capture what people remember and say, platforms capture tracked interactions in a window. Each has its own bias, and neither establishes cause.

Why do the two methods disagree so often?

Platforms cannot see untracked channels, and surveys cannot see ads the buyer did not register. One customer can honestly appear in both under different channels.

Sources

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *