A Good Questionnaire Is Both Art and Science: 7 Question Traps That Quietly Ruin Your Data

There is one thing I have learned from years of running surveys: no analysis, however good, can save a bad questionnaire. We can use the most sophisticated model, but if the questions are confusing, biased, or leading, all we are analyzing is respondents’ confusion in a tidy format.

In statistics there is an old saying that fits here: garbage in, garbage out. And a questionnaire is the first entry point of our data. Here are the seven traps I see most often, with examples and how to fix them.

Trap 1: The double-barreled question

Bad example: “I am satisfied with the salary and benefits I receive.”

What if the respondent is satisfied with the salary but not with the benefits? They have no honest answer. Whatever they pick, you will not know which one they meant.

Fix: split it into two questions. “I am satisfied with the salary I receive.” and “I am satisfied with the benefits I receive.”

An easy sign to spot: the word “and” in the middle of a question. It is not always wrong, but it is always worth checking.

Trap 2: The leading question

Bad example: “How much do you agree that your supervisor has given you very good support?”

That question already implies the expected answer. Respondents tend to follow the direction offered, especially when the question concerns their supervisor or institution.

Fix: use neutral wording: “My supervisor gives me the support I need to get my work done.” With answer choices from strongly disagree to strongly agree, both directions of answer feel equally acceptable.

Trap 3: Vague or overly technical terms

Bad example: “Do you often use our digital services?” How often is often? Every day? Every week? For some people, three times a month is “often”.

Fix: use clear, measurable limits: “In the past month, how many times did you use our app?” with concrete answer choices (0 times, 1 to 3 times, 4 to 10 times, more than 10 times).

Also avoid jargon. If respondents have to open a dictionary to understand your question, you lose data or get a guess.

Trap 4: Absolute words (“always”, “never”, “all”)

Bad example: “My supervisor always listens to my opinion.”

The word “always” forces respondents to reject the statement even if their supervisor listens nine times out of ten. As a result, the score looks lower than reality.

Fix: use an explicit frequency scale (“How often does my supervisor listen to my opinion?” with choices never, rarely, sometimes, often, always), or statements without absolute words.

Trap 5: Answer options that are unbalanced or overlapping

Bad example: age options “20 to 30”, “30 to 40”, “40 to 50”. Someone who is 30 years old will be confused about which to choose.

Fix: make mutually exclusive and exhaustive categories: “20 to 29”, “30 to 39”, “40 to 49”, “50 and above”.

On a Likert scale, make sure it is balanced: an equal number of positive and negative options, with a clear neutral option when needed. Options such as “very good, good, fair, poor” lean toward the positive side and make data pile up at the top.

Trap 6: Order and context that influence answers

Earlier questions can influence answers to later questions (priming). If you ask a series of questions about problems at the office and then ask, “Overall, how satisfied are you with your job?”, respondents have already been conditioned to think about bad things.

Fix: put general questions before specific ones. Place sensitive questions (income, health, personal behavior) at the end, after respondents feel comfortable. Group questions by theme, and consider randomizing the order of items within a scale where possible.

Trap 7: A questionnaire that is too long and tiring

Tired respondents answer carelessly: choosing the middle answer over and over or following a zig-zag pattern (straight-lining). Toward the end of a long questionnaire, data quality usually drops.

Fix: ask only what you need. For every question, ask: “What analysis will use this answer?” If there is no answer, drop the question. Estimate the completion time and pilot it. Most surveys distributed to the public should take only a few minutes.

Bonus: two habits that set professional questionnaires apart

Pilot test. Give the questionnaire to ten to twenty people similar to the real respondents. Ask: which parts are confusing? Which questions feel odd? How long did it take? Two hours of piloting can save two months of wasted data collection.

Validity and reliability testing. Once the pilot data is collected, check item-total correlations, factor structure, and reliability (for example Cronbach’s alpha or composite reliability). Items that behave strangely need to be fixed before the main survey runs.

Ethical considerations

Explain the purpose of the survey, who can access the data, and how confidentiality is protected. Offer the option to skip sensitive questions. And do not ask for identifying data you do not truly need. Respondents who trust confidentiality answer more honestly, and honest answers are the only thing that makes a survey valuable.

Summary: a checklist of the seven traps

Before distributing a questionnaire, check every question:

1. Does it ask about only one thing?
2. Is it neutral, not leading?
3. Are the terms clear and measurable?
4. Is it free of forcing absolute words?
5. Are the answer options balanced and non-overlapping?
6. Does the order avoid conditioning the answers?
7. Is the length reasonable and does every question have a purpose?

A good questionnaire is invisible. Respondents fill it in smoothly, and you get data you can trust. That is the sign that art and science have worked together.

Three more examples of fixes

Example A. ‘What do you think of the quality and price of our products?’ (double-barreled). Fix: split into ‘What do you think of the quality of our products?’ and ‘What do you think of the price of our products?’.

Example B. ‘Don’t you agree that our service has been very fast?’ (leading, and a double negative). Fix: ‘Our service is fast.’ with an agree-disagree scale.

Example C. ‘How often do you use our service?’ with choices ‘rarely, sometimes, often’. Fix: give a time frame and clear ranges, for example ‘In the last 30 days, how many times did you use our service?’ with numeric choices.

An odd or even scale?

A scale with an odd number of options (for example five) gives a neutral midpoint. An even scale (for example four or six) forces respondents to choose a direction. Neither is absolutely right. An odd scale suits cases where neutral is a legitimate stance. An even scale suits cases where you want to reduce the tendency to pick the middle answer to play it safe. What matters is to be consistent within one instrument and to understand the consequences when analyzing.

Translating a questionnaire from another language

Translating an instrument from English is not enough with machine translation. A good practice: translation by two independent translators, merging by the researcher, back-translation by a third translator who has not seen the original, then comparison with the original version, and a pilot with respondents. After that, retest validity and reliability on your sample, because the meaning of words can shift across cultures.

---