Ask a Statistician: 3 Questions I Get Asked the Most (and My Honest Answers)

Every time I run a training session or supervise a thesis, the same three questions come up. The wording varies, but the core is identical. This Saturday, I’m answering all three in one place, in language you can use right away.

The questions: how large a sample is enough, does non-normal data require a non-parametric test, and does a non-significant result mean your research failed. Let’s start with the one I hear most often.

Question 1: “How large a sample is enough?”

My honest answer: it depends on what you want the sample to tell you. There is no magic number.

If your goal is estimating a proportion (for example, “what percentage of customers are satisfied?”), sample size determines precision, measured by the margin of error. At a 95% confidence level and the most conservative assumption (proportion of 50%), the required random sample is roughly:

Margin of error	Random sample needed
±10%	about 97 people
±5%	about 385 people
±3%	about 1,068 people
±2%	about 2,401 people

Notice the pattern: to cut the margin from ±5% to ±2.5%, you don’t just double the sample—you need roughly four times as many respondents. That’s why tight precision is expensive.

Two important caveats. First, these numbers assume a random sample. A convenience sample of 10,000 from a group chat poll isn’t automatically more accurate than 400 randomly selected respondents, because what randomization guarantees is the ability to quantify the error, not the sheer count. Second, for small populations the requirement drops. With a finite-population correction, a population of 2,000 with a ±5% margin needs only about 323 respondents, and a population of 500 about 218.

If your goal isn’t estimating a proportion but testing relationships or building a model (regression, SEM, PLS-SEM), the logic changes: what matters is statistical power, the number of variables or paths in the model, and the effect size you want to detect. So don’t copy “100 respondents” from someone else’s thesis without checking whether their objective matches yours.

Question 2: “My data isn’t normal. I have to use a non-parametric test, right?”

This is the question that sends students into a panic two weeks before their defense. My short answer: not necessarily, and you may be testing normality on the wrong thing.

In many analyses—regression in particular—the normality assumption applies to the residuals (the difference between observed and predicted values), not to each raw variable individually. Raw variables can be skewed, yet the residuals may still satisfy the assumption.

A few things to check before switching to a non-parametric test:

• What does your chosen test actually assume? The t-test and ANOVA, for instance, require normality within each group or of the residuals, and both are reasonably robust against mild violations when the sample isn’t tiny.
• Look at the visuals. Histograms and Q-Q plots are often more informative than a single p-value from a normality test. Formal tests like Shapiro-Wilk can “reject” normality in large samples due to deviations that are practically trivial.
• Know your alternatives. If the assumption truly fails, non-parametric tests aren’t the only option. You can also transform the data or use methods that are robust to violations.

The wording in your results section matters too. Instead of writing “the data were not normal so a non-parametric test was used,” examiners appreciate a sentence that states which assumption was tested, with which tool, on which object, and what the consequence was.

Question 3: “My result isn’t significant. Does that mean my research failed?”

No. And this is probably the most damaging misconception in research.

A p-value answers a narrow question: if there really were no effect, how rarely would I see data as extreme as this? It does not answer “is the effect important,” and a large p is not evidence that the effect doesn’t exist. It could simply mean your sample was too small to detect it.

Imagine a training program that raises the average competency score by 2 points on a scale with a standard deviation of 10 (a small effect). Tested on 30 people, that effect will almost certainly go undetected, and you might conclude “the training had no impact.” Tested on 3,000 people, the exact same effect produces a tiny p-value and you conclude “significant impact.” The effect is still two points. Only the sample size changed.

That’s why I recommend the following when reporting results:

1. Report the effect size (e.g., the coefficient or the mean difference), not just p.
2. Include a confidence interval so readers can see the plausible range.
3. Discuss the sample’s power to detect the effect, especially when the result is non-significant.
4. Ask about practical relevance: does a difference that size matter for decisions or costs?

A non-significant result, reported honestly, is a legitimate finding. What’s dangerous is the habit of trying different analyses until something is significant, then reporting only that one.

Bonus: three sentences you can adapt for your methods and results sections

Sometimes what trips students up isn’t the analysis but the writing. Here are template sentences you can tailor to your own case. Fill in the numbers with your own data and decisions.

On sample size: “The sample size was determined to achieve a maximum margin of error of ±5% at a 95% confidence level, requiring at least approximately 385 respondents for a large population. The sample was selected using [state the sampling technique].”

On assumptions: “The normality assumption was assessed on the model residuals using a Q-Q plot and the [state the test], and the results [met or did not meet the assumption]. Accordingly, the analysis proceeded with [state the method].”

On a non-significant result: “The effect of [variable A] on [variable B] was not statistically significant (coefficient [value]; 95% confidence interval [lower] to [upper]). With a sample size of [n], this study had limited power to detect a small effect, so this result should be interpreted as [an honest interpretation] rather than as evidence of no effect.”

Note that all three sentences state the reason, not just the decision. That’s what examiners look for: proof that you know why you did something.

The common thread

All three answers share one principle: don’t apply rules of thumb without understanding the rationale behind them. Sample-size thresholds, normality tests, and the p < 0.05 cutoff are all tools. Tools used without understanding their purpose produce conclusions that look neat but are hard to defend in a thesis defense or a boardroom.

If you have another question, leave it in the comments or send a direct message. Good questions will be featured in the next Saturday edition.

Read also: [Data Analysis & Statistical Services](https://datalyticsindo.com/en/data-analysis-statistical-services/) – statistical analysis, data processing, and methodology support from the Datalytics team.

---