ID|EN
Chat di WhatsApp

Tag: research

  • A Good Questionnaire Is Both Art and Science: 7 Question Traps That Quietly Ruin Your Data

    There is one thing I have learned from years of running surveys: no analysis, however good, can save a bad questionnaire. We can use the most sophisticated model, but if the questions are confusing, biased, or leading, all we are analyzing is respondents’ confusion in a tidy format.

    In statistics there is an old saying that fits here: garbage in, garbage out. And a questionnaire is the first entry point of our data. Here are the seven traps I see most often, with examples and how to fix them.

    Trap 1: The double-barreled question

    Bad example: “I am satisfied with the salary and benefits I receive.”

    What if the respondent is satisfied with the salary but not with the benefits? They have no honest answer. Whatever they pick, you will not know which one they meant.

    Fix: split it into two questions. “I am satisfied with the salary I receive.” and “I am satisfied with the benefits I receive.”

    An easy sign to spot: the word “and” in the middle of a question. It is not always wrong, but it is always worth checking.

    Trap 2: The leading question

    Bad example: “How much do you agree that your supervisor has given you very good support?”

    That question already implies the expected answer. Respondents tend to follow the direction offered, especially when the question concerns their supervisor or institution.

    Fix: use neutral wording: “My supervisor gives me the support I need to get my work done.” With answer choices from strongly disagree to strongly agree, both directions of answer feel equally acceptable.

    Trap 3: Vague or overly technical terms

    Bad example: “Do you often use our digital services?” How often is often? Every day? Every week? For some people, three times a month is “often”.

    Fix: use clear, measurable limits: “In the past month, how many times did you use our app?” with concrete answer choices (0 times, 1 to 3 times, 4 to 10 times, more than 10 times).

    Also avoid jargon. If respondents have to open a dictionary to understand your question, you lose data or get a guess.

    Trap 4: Absolute words (“always”, “never”, “all”)

    Bad example: “My supervisor always listens to my opinion.”

    The word “always” forces respondents to reject the statement even if their supervisor listens nine times out of ten. As a result, the score looks lower than reality.

    Fix: use an explicit frequency scale (“How often does my supervisor listen to my opinion?” with choices never, rarely, sometimes, often, always), or statements without absolute words.

    Trap 5: Answer options that are unbalanced or overlapping

    Bad example: age options “20 to 30”, “30 to 40”, “40 to 50”. Someone who is 30 years old will be confused about which to choose.

    Fix: make mutually exclusive and exhaustive categories: “20 to 29”, “30 to 39”, “40 to 49”, “50 and above”.

    On a Likert scale, make sure it is balanced: an equal number of positive and negative options, with a clear neutral option when needed. Options such as “very good, good, fair, poor” lean toward the positive side and make data pile up at the top.

    Trap 6: Order and context that influence answers

    Earlier questions can influence answers to later questions (priming). If you ask a series of questions about problems at the office and then ask, “Overall, how satisfied are you with your job?”, respondents have already been conditioned to think about bad things.

    Fix: put general questions before specific ones. Place sensitive questions (income, health, personal behavior) at the end, after respondents feel comfortable. Group questions by theme, and consider randomizing the order of items within a scale where possible.

    Trap 7: A questionnaire that is too long and tiring

    Tired respondents answer carelessly: choosing the middle answer over and over or following a zig-zag pattern (straight-lining). Toward the end of a long questionnaire, data quality usually drops.

    Fix: ask only what you need. For every question, ask: “What analysis will use this answer?” If there is no answer, drop the question. Estimate the completion time and pilot it. Most surveys distributed to the public should take only a few minutes.

    Bonus: two habits that set professional questionnaires apart

    Pilot test. Give the questionnaire to ten to twenty people similar to the real respondents. Ask: which parts are confusing? Which questions feel odd? How long did it take? Two hours of piloting can save two months of wasted data collection.

    Validity and reliability testing. Once the pilot data is collected, check item-total correlations, factor structure, and reliability (for example Cronbach’s alpha or composite reliability). Items that behave strangely need to be fixed before the main survey runs.

    Ethical considerations

    Explain the purpose of the survey, who can access the data, and how confidentiality is protected. Offer the option to skip sensitive questions. And do not ask for identifying data you do not truly need. Respondents who trust confidentiality answer more honestly, and honest answers are the only thing that makes a survey valuable.

    Summary: a checklist of the seven traps

    Before distributing a questionnaire, check every question:

    1. Does it ask about only one thing?
    2. Is it neutral, not leading?
    3. Are the terms clear and measurable?
    4. Is it free of forcing absolute words?
    5. Are the answer options balanced and non-overlapping?
    6. Does the order avoid conditioning the answers?
    7. Is the length reasonable and does every question have a purpose?

    A good questionnaire is invisible. Respondents fill it in smoothly, and you get data you can trust. That is the sign that art and science have worked together.

    Three more examples of fixes

    Example A. ‘What do you think of the quality and price of our products?’ (double-barreled). Fix: split into ‘What do you think of the quality of our products?’ and ‘What do you think of the price of our products?’.

    Example B. ‘Don’t you agree that our service has been very fast?’ (leading, and a double negative). Fix: ‘Our service is fast.’ with an agree-disagree scale.

    Example C. ‘How often do you use our service?’ with choices ‘rarely, sometimes, often’. Fix: give a time frame and clear ranges, for example ‘In the last 30 days, how many times did you use our service?’ with numeric choices.

    An odd or even scale?

    A scale with an odd number of options (for example five) gives a neutral midpoint. An even scale (for example four or six) forces respondents to choose a direction. Neither is absolutely right. An odd scale suits cases where neutral is a legitimate stance. An even scale suits cases where you want to reduce the tendency to pick the middle answer to play it safe. What matters is to be consistent within one instrument and to understand the consequences when analyzing.

    Translating a questionnaire from another language

    Translating an instrument from English is not enough with machine translation. A good practice: translation by two independent translators, merging by the researcher, back-translation by a third translator who has not seen the original, then comparison with the original version, and a pilot with respondents. After that, retest validity and reliability on your sample, because the meaning of words can shift across cultures.

    Building a questionnaire for a thesis, dissertation, or institutional survey? The datalytics.indo team can help with everything from formulating the concept and question items, piloting and validity-reliability testing, sampling, analysis (regression, CB-SEM and PLS-SEM, path analysis, factor analysis, clustering, and more), through to interpretation. Private tutoring is also available. Let’s discuss your needs first. We’ll provide a tailored quote based on your project.

    Chat on WhatsApp

  • What Does a P-Value Actually Say? 6 Misconceptions Still Taught in Many Classrooms

    If there is one number that is most often worshipped and most often misunderstood in research, it is the p-value. Students memorize it: “if p is less than 0.05, the hypothesis is accepted; if more, it is rejected.” Lecturers ask about it at the defense. Journals use it as a yardstick. And in the middle of all that, almost nobody can truly explain what the number means.

    I am not blaming anyone. The definition really is not intuitive. But understanding it correctly can save you from wrong conclusions, in a thesis and in the workplace. Let us start with the definition, then take apart the six most common misconceptions.

    The correct definition, in human language

    The p-value is the probability of getting a result as extreme as the one we observed (or more extreme), assuming the null hypothesis is true. The null hypothesis usually says “there is no effect” or “there is no difference”.

    Imagine you suspect a coin is unfair. You flip it ten times and get nine heads. The question: if this coin were actually fair, how likely would we be to get a result this extreme? The probability of getting nine or ten heads out of ten flips of a fair coin is about 1.1%. That is the p-value (for a one-sided test): about 0.011. A small number means your result is surprising if the coin is fair, so you begin to doubt the assumption that the coin is fair.

    Notice what it does not say: the p-value does not tell you the probability that the coin is fair. It only tells you how odd the data is if the coin is fair. Two very different things, and that is the root of the misconceptions that follow.

    Misconception 1: “p = 0.03 means the null hypothesis has a 3% chance of being true”

    Wrong. The p-value is the probability of the data (or more extreme) assuming the null hypothesis is true, not the probability that the null hypothesis is true given the data. Flipping that sentence is like equating “the probability that someone carries an umbrella if it is raining” with “the probability that it is raining if someone carries an umbrella”. The two are clearly different. To get the probability of the hypothesis itself, we need a different approach (for example Bayesian) and prior information about how plausible the hypothesis was before seeing the data.

    Misconception 2: “p < 0.05 means the effect is important”

    No. The p-value does not measure the size of an effect. With a very large sample, a trivial difference (for example a satisfaction score difference of 0.02 on a scale of 1 to 5) can produce a very small p. Conversely, with a small sample, a practically important effect can produce a large p because the data is not strong enough to establish it.

    To judge the importance of an effect, report the effect size (for example a difference in means, a regression coefficient, Cohen’s d, or R²) along with a confidence interval. A confidence interval is far more informative: it shows the range of plausible effect values, not just yes or no.

    Misconception 3: “p > 0.05 means there is no effect”

    Also wrong. A non-significant result only means the data is not strong enough to reject the null hypothesis. Perhaps the effect truly does not exist. Perhaps the effect exists, but the sample is too small to detect it (low power). “No evidence of an effect” is not the same as “evidence of no effect”. In a report, write “insufficient evidence was found” and show the confidence interval to make clear whether a meaningful effect is still possible.

    Misconception 4: “0.05 is a sacred number”

    The 0.05 threshold is a convention, not a law of nature. It became popular in the early 20th century because it was easy to use with statistical tables, not because there is any scientific reason that 0.049 is fundamentally different from 0.051. Treating p = 0.049 as a “finding” and p = 0.051 as “not a finding” is a fragile way of thinking. In its 2016 statement, the American Statistical Association warned that scientific conclusions and policy decisions should not be based only on whether a p-value passes a specific threshold.

    Misconception 5: “If I try many analyses and one is significant, that is my finding”

    This is a subtle trap, and often unnoticed by those who fall into it. If you test 20 relationships that do not actually exist, with a 0.05 threshold, you should expect about one “significant” result by chance. The probability that at least one of 20 independent tests produces p < 0.05 when there is no effect at all is about 64%. If you report only the significant one, you are presenting coincidence as discovery.

    The practice of trying various analyses, picking subgroups, or deleting data until a significant result appears is called p-hacking. How to guard against it: set your hypotheses and analysis plan before looking at the data, report all the analyses you ran (not just the ones that worked), and use corrections for multiple testing when testing many hypotheses at once.

    Misconception 6: “A very small p-value means the result is certainly true”

    A small p-value does not guarantee that the study design is good, the measurement valid, the sample representative, or that there is no bias. A perfect analysis of flawed data still produces flawed conclusions. The p-value answers only one narrow question about random variation, and does not touch questions about data quality or research logic.

    So how should you use the p-value?

    The view I share with students and researchers:

    1. Understand what is being tested. Write the null hypothesis in plain words.
    2. Report the actual p-value (for example p = 0.031), not just “p < 0.05”.
    3. Always include the effect size and confidence interval.
    4. Consider power. Is your sample sufficient to detect a meaningful effect?
    5. Tell the whole story of your analysis. Do not hide analyses that did not work.
    6. Combine it with theory and common sense. A significant result that contradicts all existing theory deserves suspicion and replication.

    The p-value is not the enemy. It is a useful tool, as long as it is used to answer the right question. What is dangerous is turning it into a traffic light that decides “true or false” without thinking.

    A numerical example: the same effect, different conclusions

    Imagine a training program raises the average competence score by 2 points on a scale with a standard deviation of 10 (a small effect, Cohen’s d = 0.2). If we test it on 30 people, that effect will almost certainly not be detected (large p), and we might conclude ‘the training has no effect’. If we test it on 3,000 people, the same effect will produce a very small p, and we conclude ‘the training has a significant effect’. The effect is exactly the same: two points. Only the sample size changed. The question that really matters is: is two points meaningful in practice, and is the cost of the training proportionate to the benefit? The p-value does not answer that.

    Alternatives and complements

    Several increasingly recommended approaches: confidence intervals to show the range of plausible effects, effect sizes to state magnitude, power analysis before a study to make sure the sample is sufficient, pre-registration of analysis plans to prevent choosing analyses after seeing results, and replication to check whether findings hold up. Bayesian approaches also offer a way to state how strongly the evidence supports a hypothesis, though they require additional understanding. The point is not to throw away the p-value, but to place it as one piece of evidence among several.

    Processing data for a thesis, dissertation, or journal manuscript and want your interpretation to hold up in front of examiners and reviewers? The datalytics.indo team can help with everything from research design and sampling, through analysis (regression, CB-SEM and PLS-SEM, path analysis, factor analysis, clustering, and more), to interpretation and writing up results. Private tutoring in SPSS and SmartPLS is also available. Let’s discuss your needs first. We’ll provide a tailored quote based on your project.

    Chat on WhatsApp