Another great analysis from Pollster.com.
Focus is on Presidental Approval Ratings, but the findings are generalizable to other surveys, e.g. customer satisfaction.
* * * * *
From Pollster.com:
Most pollsters offer just two answer categories: “Do you approve or disapprove of the way Barack Obama is handling his job as president?”
Rasmussen’s question prompts for four: “How would you rate the job Barack Obama has been doing as President … do you strongly approve, somewhat approve, somewhat disapprove, or strongly disapprove of the job he’s been doing?”
Rasmussen has long asserted that the additional “somewhat” approve or disapprove options coax some respondents to provide an answer that might otherwise end up in the “don’t know” category.
Rasmussen conducted an experiment to test that argument.
They administered three separate surveys of 800 “likely voters, each involving a different version of the Obama job approval rating: (1) the traditional two category, approve or disapprove choice, (2) the standard Rasmussen four-category version and (3) a variant used by Zogby and Harris, that asks if the president is doing an excellent, good, fair or poor job.
The table below collapses the results into two categories; excellent and good combine to represent “approve,” fair and poor combine to represent “disapprove.”
In general, smaller don’t know percentages tend to translate into larger disapproval percentages. The 4-category Rasmussen version shows a smaller “don’t know” (1% vs. 4%) and a much bigger disapprove percentage (52% vs 46%) compared to the standard 2-category question.
The approve percentage is only three points lower on the Rasmussen version (47%) than the traditional question (50%).
The Rasmussen experiment shows an even bigger discrepancy between the approve percentage on the two-category questions (50%) and the much lower percentage obtained by combining excellent and good (38%).
* * * * *
Generalizing the findings: To increase “data discrimination” when doing customer sat polling, use 4 categories and and focus on the “very” categories at the extremes. That’s where the real info is …
The Homa Files 
December 9, 2009 at 11:22 am |
Definitely agree that the four-category approach is best, and also would suggest that wording itself can be highly influential.
As someone who conducts surveys for clients several times a year, I would say that Zogby’s use of the word “fair” is not precise enough to get good results, or at least similar results to the clearly negative “somewhat disapprove.”
“Fair” has modestly positive connotations and I don’t think is materially different from “good.” I would be interested in seeing whether Zogby lined up with the other surveys if you counted “fair” as approval.
With every survey I draft, I struggle with predicting exactly these types of wordings that will return spongy results. It’s usually after I get the results back that I see them–usually after getting data that doesn’t quite foot with our assumptions and wondering whether it’s truly actionable as a basis for resetting strategy.