ID|EN
Chat di WhatsApp

Tag: survey

  • A Good Questionnaire Is Both Art and Science: 7 Question Traps That Quietly Ruin Your Data

    There is one thing I have learned from years of running surveys: no analysis, however good, can save a bad questionnaire. We can use the most sophisticated model, but if the questions are confusing, biased, or leading, all we are analyzing is respondents’ confusion in a tidy format.

    In statistics there is an old saying that fits here: garbage in, garbage out. And a questionnaire is the first entry point of our data. Here are the seven traps I see most often, with examples and how to fix them.

    Trap 1: The double-barreled question

    Bad example: “I am satisfied with the salary and benefits I receive.”

    What if the respondent is satisfied with the salary but not with the benefits? They have no honest answer. Whatever they pick, you will not know which one they meant.

    Fix: split it into two questions. “I am satisfied with the salary I receive.” and “I am satisfied with the benefits I receive.”

    An easy sign to spot: the word “and” in the middle of a question. It is not always wrong, but it is always worth checking.

    Trap 2: The leading question

    Bad example: “How much do you agree that your supervisor has given you very good support?”

    That question already implies the expected answer. Respondents tend to follow the direction offered, especially when the question concerns their supervisor or institution.

    Fix: use neutral wording: “My supervisor gives me the support I need to get my work done.” With answer choices from strongly disagree to strongly agree, both directions of answer feel equally acceptable.

    Trap 3: Vague or overly technical terms

    Bad example: “Do you often use our digital services?” How often is often? Every day? Every week? For some people, three times a month is “often”.

    Fix: use clear, measurable limits: “In the past month, how many times did you use our app?” with concrete answer choices (0 times, 1 to 3 times, 4 to 10 times, more than 10 times).

    Also avoid jargon. If respondents have to open a dictionary to understand your question, you lose data or get a guess.

    Trap 4: Absolute words (“always”, “never”, “all”)

    Bad example: “My supervisor always listens to my opinion.”

    The word “always” forces respondents to reject the statement even if their supervisor listens nine times out of ten. As a result, the score looks lower than reality.

    Fix: use an explicit frequency scale (“How often does my supervisor listen to my opinion?” with choices never, rarely, sometimes, often, always), or statements without absolute words.

    Trap 5: Answer options that are unbalanced or overlapping

    Bad example: age options “20 to 30”, “30 to 40”, “40 to 50”. Someone who is 30 years old will be confused about which to choose.

    Fix: make mutually exclusive and exhaustive categories: “20 to 29”, “30 to 39”, “40 to 49”, “50 and above”.

    On a Likert scale, make sure it is balanced: an equal number of positive and negative options, with a clear neutral option when needed. Options such as “very good, good, fair, poor” lean toward the positive side and make data pile up at the top.

    Trap 6: Order and context that influence answers

    Earlier questions can influence answers to later questions (priming). If you ask a series of questions about problems at the office and then ask, “Overall, how satisfied are you with your job?”, respondents have already been conditioned to think about bad things.

    Fix: put general questions before specific ones. Place sensitive questions (income, health, personal behavior) at the end, after respondents feel comfortable. Group questions by theme, and consider randomizing the order of items within a scale where possible.

    Trap 7: A questionnaire that is too long and tiring

    Tired respondents answer carelessly: choosing the middle answer over and over or following a zig-zag pattern (straight-lining). Toward the end of a long questionnaire, data quality usually drops.

    Fix: ask only what you need. For every question, ask: “What analysis will use this answer?” If there is no answer, drop the question. Estimate the completion time and pilot it. Most surveys distributed to the public should take only a few minutes.

    Bonus: two habits that set professional questionnaires apart

    Pilot test. Give the questionnaire to ten to twenty people similar to the real respondents. Ask: which parts are confusing? Which questions feel odd? How long did it take? Two hours of piloting can save two months of wasted data collection.

    Validity and reliability testing. Once the pilot data is collected, check item-total correlations, factor structure, and reliability (for example Cronbach’s alpha or composite reliability). Items that behave strangely need to be fixed before the main survey runs.

    Ethical considerations

    Explain the purpose of the survey, who can access the data, and how confidentiality is protected. Offer the option to skip sensitive questions. And do not ask for identifying data you do not truly need. Respondents who trust confidentiality answer more honestly, and honest answers are the only thing that makes a survey valuable.

    Summary: a checklist of the seven traps

    Before distributing a questionnaire, check every question:

    1. Does it ask about only one thing?
    2. Is it neutral, not leading?
    3. Are the terms clear and measurable?
    4. Is it free of forcing absolute words?
    5. Are the answer options balanced and non-overlapping?
    6. Does the order avoid conditioning the answers?
    7. Is the length reasonable and does every question have a purpose?

    A good questionnaire is invisible. Respondents fill it in smoothly, and you get data you can trust. That is the sign that art and science have worked together.

    Three more examples of fixes

    Example A. ‘What do you think of the quality and price of our products?’ (double-barreled). Fix: split into ‘What do you think of the quality of our products?’ and ‘What do you think of the price of our products?’.

    Example B. ‘Don’t you agree that our service has been very fast?’ (leading, and a double negative). Fix: ‘Our service is fast.’ with an agree-disagree scale.

    Example C. ‘How often do you use our service?’ with choices ‘rarely, sometimes, often’. Fix: give a time frame and clear ranges, for example ‘In the last 30 days, how many times did you use our service?’ with numeric choices.

    An odd or even scale?

    A scale with an odd number of options (for example five) gives a neutral midpoint. An even scale (for example four or six) forces respondents to choose a direction. Neither is absolutely right. An odd scale suits cases where neutral is a legitimate stance. An even scale suits cases where you want to reduce the tendency to pick the middle answer to play it safe. What matters is to be consistent within one instrument and to understand the consequences when analyzing.

    Translating a questionnaire from another language

    Translating an instrument from English is not enough with machine translation. A good practice: translation by two independent translators, merging by the researcher, back-translation by a third translator who has not seen the original, then comparison with the original version, and a pilot with respondents. After that, retest validity and reliability on your sample, because the meaning of words can shift across cultures.

    Building a questionnaire for a thesis, dissertation, or institutional survey? The datalytics.indo team can help with everything from formulating the concept and question items, piloting and validity-reliability testing, sampling, analysis (regression, CB-SEM and PLS-SEM, path analysis, factor analysis, clustering, and more), through to interpretation. Private tutoring is also available. Let’s discuss your needs first. We’ll provide a tailored quote based on your project.

    Chat on WhatsApp

  • Employee Satisfaction Surveys That Often Go in the Wrong Direction: 6 Mistakes That Leave the Results Unused by Anyone

    Almost every company has done it: an employee satisfaction or engagement survey. The questionnaire is distributed, the results are compiled, a neat presentation is made, and then it is saved in a shared folder. A year later, the same survey is repeated. The score is up 0.1. Everyone smiles. Nothing changes.

    I have been involved in several surveys like this, including an engagement survey for a retail company with hundreds of employees, whose results were presented directly to the board of directors and the board of commissioners. From that experience, I learned that failed surveys rarely fail because of the statistical formulas. They fail because the direction was wrong from the start. Here are the six mistakes I run into most often.

    1. Chasing a score, not a decision

    The first question I ask clients is always the same: “Once these survey results are out, what decision will you make?” Often the answer is a few seconds of silence. A survey designed without a decision behind it will produce a single number (for example an engagement score of 4.12) that gives no clue about anything. Is that good? Is that bad? What should be done?

    Design the survey backwards. Start from the decisions that might be taken (for example a change in leave policy, better communication from supervisors, or a career development program), and only then derive the questions relevant to those decisions.

    2. Reporting only averages

    An average score of 4.12 out of 5 sounds good. But an average can hide two very different groups: some employees are very satisfied, others very dissatisfied. Their average looks “decent”, while the company is facing a serious problem in one group.

    Always look at the distribution and comparisons between groups: by division, tenure, job level, location. And do not forget to read the percentage of “dissatisfied” or “very dissatisfied” answers, not just the average. Organizational problems usually hide in the tail of the distribution.

    3. Not looking for drivers, only measuring levels

    Engagement surveys often stop at the question “how engaged are our employees?”. But the more valuable question is: “what influences engagement the most in our company?”

    This is where statistical analysis comes in. With regression or path analysis, we can see which factor has the strongest relationship with engagement: direct leadership, compensation, workload, growth opportunities, or communication. This is what separates a survey that merely “takes a snapshot” from one that “points the way”. If you only have the budget to fix two things this year, you need to know which two have the biggest impact.

    4. Questions that are biased, double-barreled, or too sweet

    Some examples that often slip through:

    • Double-barreled questions: “I am satisfied with my salary and benefits.” What if someone is satisfied with the salary but not the benefits? They have no honest answer.
    • Leading questions: “How much do you agree that your supervisor has supported you well?” That wording already implies the answer.
    • Vague terms: “I feel valued in this company.” Valued by whom? In what form?

    A good question is short, specific, one idea, and does not hint at the answer. Always pilot the questionnaire on ten to twenty people before wide distribution, and ask them which parts are confusing.

    5. Ignoring confidentiality and the fear effect

    An employee satisfaction survey is only useful if employees dare to be honest. Employees will only be honest if they are sure their answers cannot be traced back to them. Two bad habits that often damage this trust:

    • Reporting results for groups that are too small. If a team has only four people and results are reported per team, a supervisor can easily guess who answered what. Set a minimum number of respondents for every report.
    • Asking for excessive identifying data. The combination of division, position, age, and tenure can identify someone almost for certain, especially in small units.

    If employees are suspicious, they will answer safely, meaning high scores without meaning. This is where a survey produces beautiful but empty data.

    6. No follow-up, so the next survey gets quieter

    This is the most expensive mistake. Employees fill in the survey hoping something will change. When no visible follow-up appears, they learn one thing: this survey is a formality. Response rates fall, honesty falls, and the next survey produces data that is even less useful.

    A far healthier practice: announce the results (honestly, including the unpleasant ones), set a few realistic priorities, name an owner for each, and report progress before the next survey. When employees see one small thing change because of the survey, their willingness to answer honestly next time rises sharply.

    So what does a good survey look like?

    In short, a useful engagement survey has five traits:

    1. Designed from the decisions that will be made.
    2. Uses a tested questionnaire (valid and reliable), not just a list of questions that sound good.
    3. Analyzed by looking at distribution, group comparisons, and drivers, not just averages.
    4. Confidentiality is protected with a minimum reporting threshold.
    5. Followed by visible, communicated follow-up.

    When those five are in place, that 4.12 stops being just a number. It becomes the starting point of a real conversation between management and employees, and that conversation is what actually changes an organization.

    An example one-page report framework for the board

    Good survey results can be summarized on one page. Here is the framework I recommend: (1) three headline numbers with comparisons (overall score, participation rate, change from the previous survey), (2) the three biggest strengths and three areas for improvement, with numbers, (3) two to three main engagement drivers obtained from statistical analysis, (4) prioritized recommendations with owners and deadlines, and (5) one sentence on the limitations of the data. The following pages are appendices for those who want to dig deeper. A board that reads one clear page is more likely to act than a board that reads fifty slides.

    Do not forget open-ended questions

    Numbers give a picture of ‘how much’, but free comments give the ‘why’. One or two open-ended questions, for example ‘What is the one thing that most needs to improve in this company?’, often produce unexpected insights. Those comments can be grouped by theme (content analysis), then counted by frequency. A combination of numbers and stories is far more convincing to management than numbers alone. Make sure the comments reported are anonymized so they cannot be traced to individuals.

    Short Q&A

    How often should an engagement survey be run? Many companies run a comprehensive survey once a year, supplemented by a short pulse survey of a few questions each quarter to monitor specific issues. Too often makes employees weary, too rarely means problems are detected late. Most importantly, do not run a new survey before follow-up on the previous one is visible.

    Is a high score always good? Not always. A very high and uniform score across all divisions sometimes signals that employees are afraid to be honest or that the questions are too sweet. Compare it with behavioral data such as resignation rates, absenteeism, and training participation as a cross-check.

    Planning a customer satisfaction or employee engagement survey for your company or institution? The datalytics.indo team can help with everything from survey design, instrument testing, and sampling, through analysis (engagement drivers with regression or path analysis), to presenting recommendations to management. In-house training is also available. Let’s discuss your needs first. We’ll provide a tailored quote based on your project.

    Chat on WhatsApp