ID|EN
Chat di WhatsApp

Tag: questionnaire

  • The Rice Scale and the Gold Scale: Understanding Validity and Reliability Without the Headache

    If there is one topic that confuses students and early-career researchers the most, it is probably validity and reliability. Those two words appear in every research proposal, are demanded by every examiner, and are often answered with numbers copied from SPSS output without being truly understood.

    Let us use one simple analogy. Imagine you want to weigh yourself. The right instrument is a bathroom scale. If you use a rice scale, it may show a number, but the result cannot be trusted for that purpose. If you use a very sensitive gold scale, the result may be extremely precise, but the instrument was not designed for the weight of a human body.

    That is the essence of validity and reliability. Reliability asks: is this instrument consistent? Validity asks: does this instrument measure what it is supposed to measure? Both are needed, and they are not the same.

    Reliability: the consistency of a measuring instrument

    A reliable instrument produces consistent results when used repeatedly under the same conditions. A scale that shows 60 kg today, 72 kg two minutes later, and then 55 kg is clearly not reliable. In questionnaire research, there are several ways to assess reliability:

    • Test-retest. The same instrument is given to the same respondents at two different times. If the results differ greatly even though conditions are the same, the instrument is unstable. The condition is that what is being measured is assumed not to change much over that time.
    • Parallel forms. Two equivalent versions of the instrument are given, and the results are compared.
    • Internal consistency. This is the most commonly used. We check whether the items meant to measure the same concept actually agree with each other. The most popular tool is Cronbach’s alpha. In PLS-SEM, the recommended measure is composite reliability.

    A common benchmark: an alpha or composite reliability of around 0.70 and above is considered adequate for general research, and higher for riskier decisions. But a benchmark is only a benchmark. Alpha is affected by the number of items, so long instruments tend to get higher alpha simply because they are long.

    Validity: are we measuring the right thing?

    Validity is harder to prove than reliability, because it concerns meaning, not just numbers. There are several kinds of validity evidence:

    • Content validity. Do the items cover all aspects of the concept to be measured? It is usually judged by experts or through a literature review. A job satisfaction instrument that only asks about salary has poor content validity, because job satisfaction is not only about salary.
    • Construct validity. Do the items truly form the intended concept? This is where factor analysis (exploratory or confirmatory) comes in. Items that measure the same concept should cluster on the same factor. In PLS-SEM, we check outer loadings, average variance extracted (AVE), and discriminant validity, which is evidence that different concepts really are different.
    • Criterion validity. Does the instrument’s score relate to other relevant measures? For example, does a selection test score predict employee performance several months later?

    Why the two are not the same

    This is the most often misunderstood part. Reliable does not mean valid. Imagine a broken rice scale that always shows exactly the same number every time you step on it. It is very consistent, yet it does not measure your weight correctly. Consistent, but wrong.

    Conversely, valid is reliable (to a degree). If an instrument truly measures what is intended, its results cannot be random.

    That is why an alpha of 0.90 proves nothing about validity. Many students write “this instrument is valid and reliable because Cronbach’s alpha = 0.92”. That sentence mixes up two things. Alpha only speaks about internal consistency. In fact, a very high alpha sometimes signals items that are too similar (redundant).

    Item analysis: checking one by one

    Besides assessing the instrument as a whole, we examine each item. A commonly used measure is the corrected item-total correlation: how strongly an item’s score correlates with the sum of the other items. An item with a low correlation (for example below about 0.30) is suspect: it may measure something else, be ambiguously written, or be answered carelessly. We also check how alpha changes if the item is deleted. But do not delete items mechanically. Consider theory first: an item that is important in content should be improved, not simply dropped.

    Practical steps to prepare a good instrument

    1. Start from theory. Define your concept clearly, and derive its dimensions and indicators from the literature review.
    2. Write clear items. One idea per item, simple language, no double-barreled or leading wording.
    3. Ask experts to review for content validity.
    4. Pilot test. Give it to a small group of respondents similar to the final sample, and ask what is confusing.
    5. Analyze the pilot results. Check item-total correlations, reliability, and factor structure.
    6. Revise, then retest on the main sample. Validity and reliability are assessed again on the actual research data, because an instrument that works well in one sample is not necessarily good in another.
    7. Report honestly. State the measures used, their values, and their interpretation, including any weaknesses found.

    The mistakes that happen most often

    • Taking a questionnaire from another study and claiming it is “already valid”. An instrument that is valid in population A is not necessarily valid in population B, especially if it has been translated.
    • Deleting items until the numbers look good. The instrument ends up with three highly consistent items, but it no longer covers the concept.
    • Ignoring content validity and relying only on statistics. Numbers cannot replace reasoning about meaning.
    • Mixing scales. An instrument with several dimensions has its alpha computed as one scale. Compute it per dimension.

    Validity and reliability are not an administrative ritual before the analysis. They are the guarantee that every number you analyze afterwards actually means something. If your scale is wrong, however sophisticated the analysis on top of it, the result is still wrong.

    A mini example: an eight-item job satisfaction instrument

    Imagine you build a job satisfaction instrument with eight items, then test it on 60 employees. The SPSS output gives two important things. First, the overall Cronbach’s alpha, say 0.84, which indicates good internal consistency. Second, the item-total statistics table. Suppose item 6 has a corrected item-total correlation of only 0.12 and alpha rises to 0.87 if that item is deleted. That item is suspect. But before deleting it, reread its wording. Maybe it is a negatively worded statement whose score was not reversed (reverse-coded), maybe it asks about something different (for example distance to the office rather than satisfaction), or maybe the sentence is ambiguous. Improving an item is often more appropriate than discarding it.

    After reliability, run a factor analysis to see whether the eight items really form one dimension, or two different dimensions (for example satisfaction with the work itself and satisfaction with compensation). If the analysis shows two dimensions, compute reliability for each dimension, not as one scale. That is the honest way to work: the data tells you the structure, and you adjust the analysis to that structure.

    Need help building an instrument, testing it, or interpreting the results of validity and reliability testing? The datalytics.indo team can help with everything from research design and questionnaires, through analysis (factor analysis, CB-SEM and PLS-SEM, regression, path analysis, and more), to interpretation. Private tutoring in SPSS and SmartPLS is also available. Let’s discuss your needs first. We’ll provide a tailored quote based on your project.

    Chat on WhatsApp

  • A Good Questionnaire Is Both Art and Science: 7 Question Traps That Quietly Ruin Your Data

    There is one thing I have learned from years of running surveys: no analysis, however good, can save a bad questionnaire. We can use the most sophisticated model, but if the questions are confusing, biased, or leading, all we are analyzing is respondents’ confusion in a tidy format.

    In statistics there is an old saying that fits here: garbage in, garbage out. And a questionnaire is the first entry point of our data. Here are the seven traps I see most often, with examples and how to fix them.

    Trap 1: The double-barreled question

    Bad example: “I am satisfied with the salary and benefits I receive.”

    What if the respondent is satisfied with the salary but not with the benefits? They have no honest answer. Whatever they pick, you will not know which one they meant.

    Fix: split it into two questions. “I am satisfied with the salary I receive.” and “I am satisfied with the benefits I receive.”

    An easy sign to spot: the word “and” in the middle of a question. It is not always wrong, but it is always worth checking.

    Trap 2: The leading question

    Bad example: “How much do you agree that your supervisor has given you very good support?”

    That question already implies the expected answer. Respondents tend to follow the direction offered, especially when the question concerns their supervisor or institution.

    Fix: use neutral wording: “My supervisor gives me the support I need to get my work done.” With answer choices from strongly disagree to strongly agree, both directions of answer feel equally acceptable.

    Trap 3: Vague or overly technical terms

    Bad example: “Do you often use our digital services?” How often is often? Every day? Every week? For some people, three times a month is “often”.

    Fix: use clear, measurable limits: “In the past month, how many times did you use our app?” with concrete answer choices (0 times, 1 to 3 times, 4 to 10 times, more than 10 times).

    Also avoid jargon. If respondents have to open a dictionary to understand your question, you lose data or get a guess.

    Trap 4: Absolute words (“always”, “never”, “all”)

    Bad example: “My supervisor always listens to my opinion.”

    The word “always” forces respondents to reject the statement even if their supervisor listens nine times out of ten. As a result, the score looks lower than reality.

    Fix: use an explicit frequency scale (“How often does my supervisor listen to my opinion?” with choices never, rarely, sometimes, often, always), or statements without absolute words.

    Trap 5: Answer options that are unbalanced or overlapping

    Bad example: age options “20 to 30”, “30 to 40”, “40 to 50”. Someone who is 30 years old will be confused about which to choose.

    Fix: make mutually exclusive and exhaustive categories: “20 to 29”, “30 to 39”, “40 to 49”, “50 and above”.

    On a Likert scale, make sure it is balanced: an equal number of positive and negative options, with a clear neutral option when needed. Options such as “very good, good, fair, poor” lean toward the positive side and make data pile up at the top.

    Trap 6: Order and context that influence answers

    Earlier questions can influence answers to later questions (priming). If you ask a series of questions about problems at the office and then ask, “Overall, how satisfied are you with your job?”, respondents have already been conditioned to think about bad things.

    Fix: put general questions before specific ones. Place sensitive questions (income, health, personal behavior) at the end, after respondents feel comfortable. Group questions by theme, and consider randomizing the order of items within a scale where possible.

    Trap 7: A questionnaire that is too long and tiring

    Tired respondents answer carelessly: choosing the middle answer over and over or following a zig-zag pattern (straight-lining). Toward the end of a long questionnaire, data quality usually drops.

    Fix: ask only what you need. For every question, ask: “What analysis will use this answer?” If there is no answer, drop the question. Estimate the completion time and pilot it. Most surveys distributed to the public should take only a few minutes.

    Bonus: two habits that set professional questionnaires apart

    Pilot test. Give the questionnaire to ten to twenty people similar to the real respondents. Ask: which parts are confusing? Which questions feel odd? How long did it take? Two hours of piloting can save two months of wasted data collection.

    Validity and reliability testing. Once the pilot data is collected, check item-total correlations, factor structure, and reliability (for example Cronbach’s alpha or composite reliability). Items that behave strangely need to be fixed before the main survey runs.

    Ethical considerations

    Explain the purpose of the survey, who can access the data, and how confidentiality is protected. Offer the option to skip sensitive questions. And do not ask for identifying data you do not truly need. Respondents who trust confidentiality answer more honestly, and honest answers are the only thing that makes a survey valuable.

    Summary: a checklist of the seven traps

    Before distributing a questionnaire, check every question:

    1. Does it ask about only one thing?
    2. Is it neutral, not leading?
    3. Are the terms clear and measurable?
    4. Is it free of forcing absolute words?
    5. Are the answer options balanced and non-overlapping?
    6. Does the order avoid conditioning the answers?
    7. Is the length reasonable and does every question have a purpose?

    A good questionnaire is invisible. Respondents fill it in smoothly, and you get data you can trust. That is the sign that art and science have worked together.

    Three more examples of fixes

    Example A. ‘What do you think of the quality and price of our products?’ (double-barreled). Fix: split into ‘What do you think of the quality of our products?’ and ‘What do you think of the price of our products?’.

    Example B. ‘Don’t you agree that our service has been very fast?’ (leading, and a double negative). Fix: ‘Our service is fast.’ with an agree-disagree scale.

    Example C. ‘How often do you use our service?’ with choices ‘rarely, sometimes, often’. Fix: give a time frame and clear ranges, for example ‘In the last 30 days, how many times did you use our service?’ with numeric choices.

    An odd or even scale?

    A scale with an odd number of options (for example five) gives a neutral midpoint. An even scale (for example four or six) forces respondents to choose a direction. Neither is absolutely right. An odd scale suits cases where neutral is a legitimate stance. An even scale suits cases where you want to reduce the tendency to pick the middle answer to play it safe. What matters is to be consistent within one instrument and to understand the consequences when analyzing.

    Translating a questionnaire from another language

    Translating an instrument from English is not enough with machine translation. A good practice: translation by two independent translators, merging by the researcher, back-translation by a third translator who has not seen the original, then comparison with the original version, and a pilot with respondents. After that, retest validity and reliability on your sample, because the meaning of words can shift across cultures.

    Building a questionnaire for a thesis, dissertation, or institutional survey? The datalytics.indo team can help with everything from formulating the concept and question items, piloting and validity-reliability testing, sampling, analysis (regression, CB-SEM and PLS-SEM, path analysis, factor analysis, clustering, and more), through to interpretation. Private tutoring is also available. Let’s discuss your needs first. We’ll provide a tailored quote based on your project.

    Chat on WhatsApp