If the Platform’s Algorithm Changed Mid Campaign, Before and After Are Not Comparable
A before and after case study puts two numbers side by side and credits the gap to the campaign. That only works if nothing else changed in between. One thing that can change without the case study mentioning it is the platform itself.
The comparison assumes the platform stood still
“Before” and “after” are not just two dates. They are two measurements, and the case study implicitly asks you to accept both were taken under one constant distribution system: whatever combination of ranking signals, feed logic, and throttling decided how many of a post’s potential audience actually saw it.
When a platform changes how it ranks, surfaces, or throttles content inside that window, it changes what a view or an impression represents, independent of anything the campaign did. If the change lands between the two measurements, the comparison is measuring the campaign plus whatever the platform did, with no way to see the two apart from outside.
This is not a claim that platform changes are common, or that one happened in any specific case study. It is a claim about what the before and after structure needs in order to mean what it claims.
A hypothetical walkthrough: the change lands in week six of a twelve week campaign
Everything in this section is hypothetical. No real campaign, agency, or platform change is described.
Picture a twelve week campaign. The platform ships a distribution change for a certain post format in week six, the midpoint. The case study reports a hypothetical “before” figure of 1,000 average weekly impressions across weeks one through five, and a hypothetical “after” figure of 1,800 across weeks seven through twelve: an apparent 80 percent lift.
But the change date sits at the seam between the windows. Some of the gap plausibly belongs to the campaign: better creative, a shifted schedule, an audience that had time to build. Some plausibly coincides with the platform change instead, a jump any account posting that format after week six would have seen, campaign or not. No method inside one case study, real or hypothetical, can produce a precise split between those sources.
What actually changes when a platform ships a ranking or distribution update
At the level platforms tend to disclose, a ranking or distribution change can alter what gets shown to whom, how much reach a post format receives relative to others, and how a follower’s feed gets populated. [EVIDENCE NEEDED: a currently live, readable announcement from a platform’s own newsroom, developer blog, or changelog describing a specific ranking or distribution change and what it altered]. No specific platform or date is named here: the platform pages checked in this session returned either an error response or a navigation shell with no article text.
One distinction matters regardless of platform: a platform-wide change affects every account posting the affected format at once, while a narrower change only confounds a case study using it, and is easier to miss.
Where to actually look for the change date
- Public change history, where it exists, lives at the level of a newsroom announcement or a developer changelog entry, not a marketing calendar. Treat the absence of an entry as uninformative, not proof nothing happened.
- Separate an announced change from a quietly shipped one. Platforms sometimes publish a change when it ships; other times a change becomes visible only in aggregate performance data, well after the fact, with no announcement attached to the effective date.
- Before searching any change history, check whether the case study discloses its exact before and after date ranges. Most do not. [EVIDENCE NEEDED: a verifiable source on case study reporting norms for this]. Facts.json carries no count of case studies audited for this. Absent exact dates, there is nothing to check a change history against, often the only finding available.
The tell that a case study has not accounted for this
The most basic tell is silence on dates. A case study with no exact before and after date ranges cannot be checked for this confound at all, because there is nothing to check against. That absence is itself the finding.
A second tell is duration without acknowledgment. A case study spanning a period long enough to plausibly include a platform-wide change, that says nothing about the platform’s conditions during that window, has left out a variable it was in a position to name.
Contrast both with a case study that states its dates precisely enough to check against a platform’s change history. That shape is more trustworthy before a single number gets evaluated, simply because it is checkable.
What this is not the same claim as
This piece is about the comparison being invalid at the moment it was made, because the platform’s distribution mechanism moved during the test itself, contaminating both sides of the split before the case study was written up. That differs from a case study that was accurate when published and has simply aged: a campaign’s numbers can hold up under the platform conditions of the year it ran, yet the platform has since changed enough that the same campaign would not produce the same result today. That is staleness, arriving after the fact. This piece is about a confound arriving during the fact.
Read more from the library
The choice of a starting point matters on its own terms: the ‘before’ picture is also a choice. A platform change is one way the ground can move mid campaign; a calendar effect is another, covered in a seasonal tailwind mistaken for a campaign effect. Both are specific cases of the counterfactual no case study can show you.
Have a result you think holds up under this scrutiny? Browse the case study library, or submit your own with the dates attached.
FAQ
Can you always tell how much of a lift belongs to the platform change instead of the campaign?
No. A single case study, real or hypothetical, cannot cleanly separate the two without a control group that did not run the campaign under the same platform conditions. The honest position is that the comparison is contaminated, not that the contamination can be sized from the outside.
Does this mean every before and after case study is worthless?
No. The confound only applies when a platform-wide or format-wide change actually fell inside the comparison window. A case study that discloses precise dates lets a reader check for that possibility, and most published case studies do not disclose dates precisely enough to allow the check at all.
Sources
- socialcasestudies.com/blog/agency-picks-comparison-baseline/, fetched 2026-09-09
- socialcasestudies.com/blog/seasonal-tailwind-mistaken-for-campaign-effect/, fetched 2026-09-09
- socialcasestudies.com/blog/counterfactual-missing-from-case-studies/, fetched 2026-09-09