A Sentiment Score Is Only as Good as the Model That Assigned It
A case study tells you how many comments a campaign pulled, then what they felt like. The first is a count. The second is a percentage of positive sentiment, printed beside it as though both were the same kind of evidence. One is a judgement, and software made it.
What the sentiment number in a case study is actually reporting
A sentiment score is almost always an automated classifier’s output, pointed at scraped comments, sorting each into a bucket. It is not a human-coded review against a rubric. It is not a survey. It is a model guessing at tone, at speed, across text it has never seen.
That would matter less if the method were disclosed, and it usually is not. The case study rarely names the tool, rarely says what it was trained on, and almost never states which confidence threshold counted as positive, the setting that decides how large the percentage is. This is a case study number being only as good as what backs it up, applied to a metric that sounds qualitative.
Where automated sentiment classifiers reliably get it wrong
- Sarcasm and irony. A weakness of the task, not of one product, which is why the field has run dedicated shared tasks on it. “Great, another surprise fee” pairs a positive word with a negative meaning. Figurative language has its own SemEval sentiment task, irony its own detection task.
- Negation. “Not bad” and “bad” differ by one word that flips the meaning, and models miss the flip. Negation has been surveyed as a sentiment problem in its own right.
- Mixed comments. “The product is lovely, your support team ignored me for a week” praises one thing and attacks another. Splitting them is the premise of aspect based sentiment analysis; a single-label classifier picks one and moves on.
- Community slang and emoji. A model only weighs vocabulary it saw in training. Recently flipped terms, in-jokes and emoji shorthand leave it guessing.
A hypothetical batch of comments, read two ways
Everything below is invented. Not real comments, not a real campaign, not any agency’s numbers. Twenty hypothetical comments under a hypothetical post.
| Hypothetical comment | Plausible label | What a reader sees |
|---|---|---|
| “Well this is just fantastic.” | Positive | Sarcastic in context |
| “Love the design. Took five weeks to arrive.” | Positive | Mixed, complaint is the real half |
| “okay this ate” | Neutral or dropped | Strong approval in that slang |
Say the hypothetical classifier lands on fifteen positive, three neutral, two negative, rounded to seventy five percent. A human reading the same twenty flags several as sarcastic or mixed. Same batch, different number, both figures hypothetical.
Why a confident score is not the same claim as an accurate one
Classifiers usually emit a confidence value with each label. It is not a quality guarantee. Confidence is the model’s own probability estimate for the label it chose, from the same machinery. Machine learning calls that gap calibration, and work on calibration of modern neural networks defines it as producing estimates that represent the true likelihood of being correct, then reports that modern networks are poorly calibrated. A model can be confidently wrong, and reports the confidence either way.
The percentage gets published. Whether anyone read a sample of raw comments against the labels does not, and that check is what turns an output into evidence.
What to ask before you accept a sentiment percentage
- Which tool or method produced this? If the case study cannot say, there is no method to evaluate.
- Did a human review a sample, and how large? A sample read back against its labels is a real check. Zero is a different claim reported the same way.
- Every comment, or only the confident ones? Some pipelines drop comments the model could not label confidently, leaving them out of the denominator.
- How were neutral and mixed comments bucketed? Forcing a three-way read into two buckets changes the number, as does filing a mixed comment by its first clause.
The score is not the comment section
Sentiment scores are not worthless. On a hundred thousand comments an automated read is the only one available. The failure is treating the percentage as a finished conclusion, not a compressed one, the standing of a headline number that hides more than it shows. Ask as much of it as of any flattering figure, because the best metric on a dashboard is rarely the real story. Then open the post it links to and read a few dozen comments yourself. You learn what people responded to, and whether the enthusiasm was about the brand or the giveaway.
Read a few for yourself
The library republishes each case study with credit to the agency behind it, and every figure inside one, sentiment included, is that agency’s, not ours and not a benchmark. Browse the case study library, submit your own, or read how to read a marketing case study without being misled.
FAQ
What tool do case studies usually use to calculate sentiment?
The case study rarely says. A figure can come from a platform’s built-in feature, a third-party listening product, or a model the agency built itself. Those behave differently on the same comments, and the published percentage does not say which produced it.
Does a high sentiment score mean the comments were mostly positive?
It means the classifier labelled most of the comments it scored positive, which is narrower than it looks. Where sarcasm or mixed comments are common, a human read can reach a different split, and the score excludes whatever the tool declined to label.
Why don’t sentiment classifiers just get better at sarcasm?
Because irony lives in context the model lacks: tone of voice, what the person posted an hour earlier, a joke that only works inside one community. The SemEval irony detection task found sorting tweets by kind of irony much harder than detecting it at all, a problem that yields slowly rather than one that gets solved.
Sources
- Wiegand et al., A survey on the role of negation in sentiment analysis, ACL Anthology. Fetched 7 September 2026.
- Ghosh et al., SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter, ACL Anthology. Fetched 7 September 2026.
- Van Hee et al., SemEval-2018 Task 3: Irony Detection in English Tweets, ACL Anthology. Fetched 7 September 2026.
- Pontiki et al., SemEval-2016 Task 5: Aspect Based Sentiment Analysis, ACL Anthology. Fetched 7 September 2026.
- Guo et al., On Calibration of Modern Neural Networks, arXiv. Fetched 7 September 2026.
