Tag: data analysis

I will not pretend that the thesis defense room is friendly. A good examiner’s job is to keep asking until you cannot answer. The good news is that the questions that make students nervous almost always come from the same list. After years of working with students, lecturers, and researchers, and having taught at several universities before that, I have seen the same mistakes appear again and again, semester after semester.

Here are the five data analysis mistakes I run into most often, along with how to fix them before the examiner finds them first.

Mistake 1: Testing normality on the wrong object

This is number one because almost everyone has done it. Students run a normality test (Kolmogorov-Smirnov or Shapiro-Wilk) on every variable, then conclude “the data is not normal, so I must use a non-parametric test.” But for linear regression, the normality assumption applies to the model’s residuals, not to the raw variables. And for a two-group comparison, what you check is the distribution within each group, not all the data pooled together.

Two more notes: with large samples, formal normality tests can reject normality because of trivial deviations. With small samples, the tests are too weak to detect problems. So do not rely on the p-value alone. Look at histograms and Q-Q plots too, and consider the central limit theorem. For Likert scales, many researchers treat the combined score as interval data. That is common practice, but it should be explained and defended, not quietly done.

Mistake 2: Not being able to justify your sample

“My sample is 100 people because I calculated it with the Slovin formula.” Then the examiner asks: “From what population? How did you choose them? Do they represent the population?” And the answer: “I sent the questionnaire to my friends…”

A sample size without a clear selection method only satisfies a numerical requirement, not a scientific one. If your sample is not random, do not disguise it. Say honestly that it is purposive or convenience sampling, explain why, and state its limitations: your results must not be generalized to the whole population with confidence. Examiners respect honesty about limitations far more than a fragile claim of generalization.

Mistake 3: Misunderstanding validity and reliability

There are two versions of this mistake.

Version one: skipping instrument testing entirely, or simply taking a questionnaire from another study without retesting it on your sample.

Version two: assuming a high Cronbach’s alpha means the instrument is valid. Alpha measures internal consistency (reliability), not validity. An instrument can be very consistent and still measure the wrong thing. Like a rice scale that always shows the same number very consistently, but is not suitable for weighing gold. Reliable, but not valid for your purpose.

For construct validity, use factor analysis (exploratory or confirmatory) or, in PLS-SEM, outer loadings, AVE, and discriminant validity. For reliability, report Cronbach’s alpha or composite reliability, and explain the interpretation. And remember, alpha tends to rise as the number of items increases, so a high value on a long instrument is not always impressive.

Mistake 4: Chasing “significant” and forgetting to ask “how big”

Many results and discussion chapters turn into a list of p-values: “significant”, “not significant”, “significant”, “significant”. No effect sizes, no confidence intervals, and no substantive explanation.

A p-value answers only one narrow question: how surprising is this data if there were actually no effect? It does not say how large or how important the effect is. With a large sample, a very small effect can be “significant”. With a small sample, a meaningful effect can be “not significant”. Also report coefficients, effect sizes (for example R², Cohen’s d, or path coefficients with their intervals), and explain their practical meaning in language a layperson can understand.

A related temptation: trying various analyses, removing “troublesome” respondents, or constantly changing the model until the result is significant. In the literature this practice is called p-hacking, and an experienced examiner can smell it. Decide the analysis based on theory before seeing the results, and record every change you make along with the reasons.

Mistake 5: Conclusions that go beyond the data

This last mistake is subtle. Your study is a one-time survey of 150 students at one campus, and the conclusion reads: “Leadership causes performance improvement among all employees in Indonesia.”

There are two leaps there. The first leap: from correlation to cause and effect. One-time survey data can only show a relationship, not direction and cause. The word “causes” requires a stronger design. The second leap: from sample to population. A sample from one campus does not automatically represent the whole country.

Use language proportionate to the strength of your data: “is associated with”, “is related to”, “supports the idea that”, and state the limits of generalization openly. The limitations section of a study is not a sign of weakness, but a sign of scientific maturity.

A note on integrity

In every engagement, I always emphasize one principle: the purpose of helping with analysis is not to let students “just receive the finished result”, but to make sure they understand what they are doing and can defend it in front of examiners. An answer you do not understand is a trap waiting in the defense room. That is why good tutoring always comes with explanation, not just a finished result.

Quick checklist

Before submitting your results chapter, ask yourself:

1. Have I run my assumption tests on the right object?
2. Can I explain how the sample was selected and where its generalization ends?
3. Have I reported validity and reliability with the correct interpretation?
4. Have I reported effect sizes and a substantive explanation, not just p-values?
5. Is my conclusion proportionate to the strength of the data and the research design?

If all five answers are “yes”, you are already better prepared than most people at a defense.

Examples of improved sentences in the results chapter

A few sentences I often come across, and healthier versions:

• Before: ‘The data is not normal, so a non-parametric test was used.’ After: ‘The model residuals met the normality assumption based on a Q-Q plot and the Shapiro-Wilk test, so linear regression was used.’
• Before: ‘The sample is 100 according to the Slovin formula.’ After: ‘The sample of 100 respondents was selected purposively from employees of the marketing division. Results are not generalized to the whole company.’
• Before: ‘The instrument is valid and reliable because alpha is 0.91.’ After: ‘The instrument has good internal consistency (alpha 0.91), and construct validity is supported by factor analysis with loadings above 0.60.’
• Before: ‘Leadership has a significant effect (p < 0.05).’ After: ‘Leadership is positively associated with performance (coefficient 0.34; 95% confidence interval 0.18 to 0.50), with a medium effect size.’

The differences look small, but to an examiner, the sentences in the second version show that you understand what you did.

Short Q&A

Is it acceptable to use a parametric test even if the data is not normal? In many cases, yes, especially with sufficiently large samples because of the central limit theorem, or when the chosen test is robust to mild violations. What matters is that you check the assumptions properly, consider alternatives, and explain your choice.

What if my supervisor suggests something different? Discuss it with data and methodological reasons, not defensively. Often the difference comes from conventions in a particular field. Explaining your choice clearly shows that you understand, and that works in your favor at the defense.

---