Surveys: Could it get worse?

Black Swan

Obtained a customer satisfaction survey from a major international fast food chain restaurant today. The questionnaire item construction was problematic. For example, one item read:

How do you think this restaurant's food, service and cleanliness compares to other fast-food restaurants?
[ ] Better
[ ] About the same
[ ] Worse

This is a triple barrel question in that it asks about 3 different things (food, service, cleanliness), but only allows one answer. What if I thought the food was "Better," but the service and cleanliness were "Worse?" Since 2 answers were "Worse," do I place my one check mark "Worse; or do I check both "Better" and "Worse;" or do I attempt to rank order my reply by checking "About the same;" or perhaps average it by placing a check mark between the "About the same" and "Worse" boxes?

And if I use one or more checks, which check stands for food or service or cleanliness? Will there be confusion or uncertainty between respondents? Will all respondents use the same method to answer the 3 questions with one answer, or different ways between respondents? You really don't know. What an analysis nightmare!

Have you seen any poorly constructed questionnaire items that you can share with us here?

Tree

What drives me crazy in surveys is white boxes. "This short survey of three questions should take you no more than five minutes" and you get an essay style question that should be no less than 200 words. If your survey is mostly open questions, you probably haven't a clue what sort of feedback you're looking to get.

Black Swan

Tree wrote: »

essay style question that should be no less than 200 words. If your survey is mostly open questions, you probably haven't a clue what sort of feedback you're looking to get.

Whenever the survey designers ask for more than checking or circling an answer, they move from quantitative to qualitative data collection, analysis, and interpretation. If the sample size is small, this may be feasible, but large sample sizes become labour-intensive and resource expensive.

If subjects text in their answers online, at least it's readable, but if they hand write in their answers, it becomes problematic attempting to read their writing and understand content. Qualitative researchers claim more depth in answers, but it sometimes get a bit confounding doing so.

But I know what you mean about those surveys that claim only 5 minutes of your time, but then drag you in. :pac:

Sciscitatio

Tree wrote: »

What drives me crazy in surveys is white boxes. "This short survey of three questions should take you no more than five minutes" and you get an essay style question that should be no less than 200 words. If your survey is mostly open questions, you probably haven't a clue what sort of feedback you're looking to get.

Yes I do agree, I see this more and more, where surveys state this will be a "short survey" 10 mins later ive given up (unless they were honest and told me it would take 10+ mins).

On a side yet funny note, I did get a survey yesterday telling me it would take 10 mins! It only had 5 tick box types questions! Shortest 10 min survey ive ever seen!

Black Swan

Forced choice scales with only 4 choices (1, 2, 3, 4; greater than, lesser than; ordinal scale) do not allow for enough variability in measurement. Furthermore, forcing the subject to pick the two choices favouring one side or the other without a middle option appears problematic. If there were more choices than 4, the lack of a middle choice may not add so much to the imprecision, or potential spuriousness.

Example:



Recently we were provided data to analyze from a survey developed and administered by a client. We had no input prior to data collection. It had several questions like the example above. When examining the paper survey questionnaire, we found several where subjects placed a mark between the 2 centre choices, creating a middle position, and confounding the scale data set.