Tag: statistical tests

The question I receive most often from students and early-career researchers is not “how do I run this test in SPSS?” but “which test should I use?” And the answer is almost always: “it depends.” Depends on what? On a few things that can actually be answered with five simple questions.

If you answer these five questions honestly before opening any software, you will avoid picking an unsuitable test, and the defense room becomes a much friendlier place.

Question 1: What is the goal of the analysis?

There are four main goals, and each points to a different family of tests:

• Comparing (is there a difference between groups or over time?)
• Relating (are two variables associated?)
• Predicting or explaining (which variables influence another variable?)
• Grouping or summarizing (is there a hidden structure in the data?)

Write your goal in one ordinary sentence before choosing a test. “I want to know whether average satisfaction differs between new and long-standing customers.” That sentence already shows the direction: a comparison of two groups.

Question 2: What is the measurement scale of your variables?

The measurement scale determines which mathematical operations make sense.

• Nominal: categories without order (gender, city, brand).
• Ordinal: ordered categories (education level, satisfaction ranking).
• Interval/ratio: numbers with meaningful distances (income, age, test score, body weight).

Likert variables are often treated specially. A single Likert item is strictly ordinal, while a combined score from many items is often treated as interval. That practice is common, but it should be stated and justified.

Question 3: How many groups are compared, and are they independent or paired?

Two questions in one. How many groups? Two, or more than two. Independent or paired? Independent means the groups consist of different subjects (for example men and women). Paired or repeated means the same subjects are measured more than once (for example scores before and after training) or matched pairs.

Question 4: Are the assumptions of parametric tests met?

Parametric tests (for example the t-test and ANOVA) assume certain things, such as a roughly normal distribution and comparable variances across groups. If those assumptions are clearly violated (for example a small sample with very skewed data, or ordinal data), a non-parametric alternative that does not rely on distributional assumptions is usually safer. But remember: non-parametric tests also have assumptions and are generally less powerful when the parametric assumptions are actually met.

Question 5: What is the size and nature of your sample?

A very small sample limits the choice of tests and makes assumption checks unreliable. A clustered sample (for example students within classes) violates the independence assumption and calls for special approaches (for example multilevel models).

Quick selection tables

Here is the short guide I use as a starting point. It does not replace judgment, but it saves time.

Comparing groups on a numeric/ordinal variable

Situation	If parametric assumptions are met	Non-parametric alternative
Two independent groups	Independent samples t-test	Mann-Whitney test
Two paired groups	Paired t-test	Wilcoxon test
Three or more independent groups	One-way ANOVA	Kruskal-Wallis test
Three or more measurements on the same subjects	Repeated-measures ANOVA	Friedman test

Categorical variables

Situation	Test
Relationship between two categorical variables (independent groups)	Chi-square test (or Fisher’s test if expected frequencies are small)
Dichotomous change in the same subjects (before-after)	McNemar’s test
Three or more dichotomous measurements on the same subjects	Cochran’s Q test

Relating variables

Situation	Measure
Two numeric variables, linear relationship	Pearson correlation
Ordinal data or assumptions not met	Spearman or Kendall correlation
Two categorical variables	Phi, Cramer’s V

Predicting or explaining

Situation	Technique
One numeric dependent variable, one predictor	Simple linear regression
One numeric dependent variable, many predictors	Multiple linear regression
Binary dependent variable (yes/no)	Logistic regression
Many relationships at once among latent variables	SEM or PLS-SEM
Direct and indirect effects among observed variables	Path analysis

Three good habits when choosing a test

1. Decide on the test before seeing the results. Choosing a test after seeing which one gives a significant result is a recipe for false findings. Write your analysis plan at the start.

2. Always report effect sizes and confidence intervals, not just p-values. Every test has a suitable effect size: Cohen’s d for a difference between two means, eta squared for ANOVA, regression coefficients for regression, and so on.

3. Check assumptions on the right object. Normality in a t-test is checked within each group, and in regression on the residuals, not on the pooled raw variable.

One complete example

A training manager wants to know whether a new training program improves employees’ competence scores. She measures the same score on 40 employees before and after the training.

• Goal: compare two measurements.
• Scale: competence score (interval).
• Groups: one group, two measurements, paired.
• Assumptions: what is checked is the normality of the difference between the after and before scores.
• Choice: paired t-test (if the differences are roughly normal) or Wilcoxon test (if not).
• Reporting: the mean difference, its confidence interval, the effect size, and a practical explanation.

The same five questions guide almost every test choice, from the simplest to the most complex. Mastering the logic is far more valuable than memorizing software menus.

Three more examples to build intuition

Example 1. A researcher wants to know whether education level (high school, bachelor’s, master’s) is related to the choice of digital payment method (three types of e-wallet). Both variables are categorical, with independent groups. The choice: Chi-square test, with Cramer’s V as a measure of association to show the strength of the relationship.

Example 2. A team wants to compare task completion time across four interface designs, each tried by different people. Numeric variable, four independent groups. If the residuals are roughly normal and variances comparable: one-way ANOVA with a post hoc test. If the data is highly skewed and the sample small: Kruskal-Wallis.

Example 3. A manager wants to predict whether a customer will cancel a subscription (yes or no) from subscription length, number of complaints, and usage frequency. Binary dependent variable, mixed predictors. The choice: logistic regression, evaluated with measures such as recall, precision, and AUC.

A short assumptions checklist

Before trusting the result of a t-test or ANOVA, check: independence of observations, normality (within groups or of residuals), and homogeneity of variance (for example Levene’s test). For regression: linearity, independence of errors, homoscedasticity, normality of residuals, and absence of severe multicollinearity. If an assumption is clearly violated, do not stop there: consider a transformation, a test that is more robust to violations, or a more suitable model.

Short Q&A

What if I am comparing two groups but the sample sizes are very unbalanced? A t-test with a correction for unequal variances (Welch’s test) is usually safer than the standard t-test. Also note that a very small group limits precision, whatever the test.

Is it acceptable to run many tests on the same data? Yes, as long as you report all of them and account for multiple testing. The more tests you run, the greater the chance of finding a ‘significant’ result by chance, so the significance threshold needs to be adjusted or the results interpreted more carefully.

---