The ritual, dismantled — then rebuilt so you can defend every step.

Estimation comes before testing

Inference has two jobs, and the teaching order of most curricula gets them backwards. The first job is estimation: what is the size of the thing, and how precisely do we know it? The second is testing: can we rule out a specific value, usually zero? Estimation is the more informative, and testing is a special case of it.

A point estimate is a single best guess — the sample mean for μ, the sample correlation for ρ. Good estimators are unbiased (right on average over repeated samples), efficient (small sampling variance), and consistent (converging on the parameter as n grows). The sample mean is all three, which is why it is everywhere.

But a point estimate alone is dishonest, because it conceals how much it would have moved with a different sample. An interval estimate attaches that uncertainty. Reporting M = 14.2 tells the reader the answer; reporting M = 14.2, 95% CI [11.9, 16.5] tells them how much to trust it, and whether the study was even capable of answering the question.

Confidence intervals, and the four things they do not mean

A 95% confidence interval is built so that, over infinitely many repetitions of the study, 95 per cent of the intervals constructed this way would contain the true parameter. The confidence attaches to the procedure. Your particular interval either contains μ or does not, and you will never know which.

Press run 500. Roughly five per cent of the intervals will be drawn in terracotta because they missed μ entirely — and notice that they look exactly like the others from the inside. That is the whole lesson: nothing about your single interval tells you whether it is one of the misses. Now drop n to 8 and watch the intervals widen and wobble.

The logic of null hypothesis testing

Null hypothesis significance testing is a proof by contradiction run on probabilities. You assume the thing you doubt, work out what data that assumption predicts, and see whether what you observed is surprising under it. Nothing more mystical is going on, and being able to state the chain in five steps answers most exam questions on the topic.

The deeper criticism, which you should be able to voice: H₀ as usually stated — that the effect is exactly zero to infinite decimal places — is almost never plausible in psychology. Any two groups differ by something. With a large enough n, everything is significant. This is why the field's centre of gravity has moved toward estimation, effect sizes and equivalence testing, and why you are being taught the logic rather than the ritual.

What a p-value is, and the five things it is not

A p-value is the probability of obtaining a result at least as extreme as the one observed, assuming the null hypothesis is exactly true. Every part of that sentence is doing work: 'at least as extreme' (not 'exactly this'), and the conditional 'assuming H₀ is true' (which is the assumption, not the conclusion).

The constructive reading: a p-value is a rough index of surprise under a specific model. Small p means the model — which includes H₀ and all your auxiliary assumptions about independence, distribution and measurement — sits poorly with the data. Notice that the null is not the only thing that can be wrong. A tiny p can be produced by a violated independence assumption just as well as by a real effect.

Type I and Type II error, and the trade-off between them

Any decision rule under uncertainty can be wrong in two directions. You can claim an effect that is not there — a Type I error, false positive, rate α. Or you can miss an effect that is there — a Type II error, false negative, rate β. Power is 1 − β: the probability of catching a real effect of a given size.

The trade-off is structural: with everything else fixed, lowering α to reduce false positives moves the decision threshold outward, which increases β and reduces power. You cannot minimise both by choosing a number. The only way to reduce both simultaneously is to get more information — more participants, less measurement error, a stronger manipulation, a within-subjects design.

There is a third error worth knowing for vivas. A Type III error is getting the right answer to the wrong question — or, in its common form, correctly rejecting the null but misidentifying the direction or the source of the effect. A significant interaction misinterpreted as a main effect is a Type III error, and no amount of statistical rigour protects against it.

One-tailed and two-tailed tests

A two-tailed test splits α between both ends of the null distribution: you will reject if the effect is surprisingly large in either direction. A one-tailed test puts the whole of α in one tail, which makes it easier to reach significance in that direction and impossible in the other.

The arithmetic consequence is direct. At α = .05, the two-tailed z critical value is 1.96; one-tailed it is 1.645. So a one-tailed test buys you power — roughly the power you would gain from adding a fair number of participants — in exchange for a commitment: if the effect runs the other way, however strongly, you must report a null result.

Practical note for reading the literature: many older papers report one-tailed tests without saying so. If a paper's t and df imply a two-tailed p of .09 but reports p < .05, that is what happened. Being able to reconstruct a p from t and df is a genuinely useful reviewing skill.

Effect sizes: the number that actually answers the question

A test tells you whether an effect is distinguishable from zero. An effect size tells you how big it is — which is the question anyone outside statistics was actually asking. Every journal that matters now requires one, every meta-analysis is built from them, and every power calculation needs one as input.

Power analysis: deciding n before you collect anything

Power is the probability of detecting an effect of a specified size if it truly exists. It is determined by four quantities, and fixing any three determines the fourth: effect size, sample size, α, and power itself. That is the whole of power analysis, and the simulator below is the whole of it made visible.

Set d = .50 and n = 30 per group: power lands near 48 per cent — a coin flip, for one of the most common designs in psychology. Set d = .30, the more realistic value for many social and clinical effects, and power collapses toward 20 per cent. Then switch to the power curve and look at the n you would need. That gap between typical practice and adequate design is the mechanical explanation for the replication crisis.

One structural point about underpowered research that is worth carrying into every design meeting. Low power does not merely mean you might miss things. It also means that any effect you do detect must have been unusually large in your sample to clear the threshold — so the published estimate is inflated, sometimes by a factor of two. Low power corrupts the literature in both directions at once.

Multiple comparisons and the inflation of error

Run one test at α = .05 and the chance of a false positive is 5 per cent. Run twenty independent tests with no real effects anywhere and the chance that at least one comes out significant is 1 − .95²⁰ = 64 per cent. Nothing has gone wrong in any individual test; the problem is the family of tests taken together.

Assumptions, ranked by how much the violation actually costs

Every test makes assumptions, and textbooks list them as equally important. They are not. Ranked by the damage a violation does, the order is roughly: independence, then correct model specification, then homogeneity of variance with unequal n, then normality. Most students spend their anxiety in exactly the reverse order.

p-hacking, forking paths, and how honest people produce false findings

Researcher degrees of freedom are the many small, defensible-looking decisions made during analysis: which outliers to exclude, whether to transform, which covariates to include, when to stop collecting, which of three measures to treat as primary. Each choice is arguable. Made after seeing the data, in the direction that helps, they collectively drive the false-positive rate far above .05 — simulations put it above 60 per cent with only a handful of such freedoms.

Beyond the p-value: estimation, equivalence, and Bayes

Nothing in the previous eleven days argues against inference. It argues against a single dichotomous decision standing in for a result. Three alternatives are now standard enough to be expected in good work, and all three are examinable as 'current developments'.

You now have the reasoning that the rest of the course applies. Volumes 4 and 5 are, in a real sense, the same twelve days repeated with different test statistics: state the model, check what it assumes, compute the signal-to-noise ratio, locate it in the appropriate null distribution, report the estimate and its uncertainty, and name the mistake you avoided.

The volume 3 reference sheet

The quantities and definitions from this volume that are examined directly, with the precise wording that earns the mark.

Reporting templates you can paste

APA-conformant sentences with the slots marked. Fill the numbers, keep the structure — these are the patterns that stop a results section from being rewritten by a reviewer.

Ten questions before you move on

Several of these are the exact wordings that appear in NET papers and in viva questions. If you score below seven, reread before volume 4 — everything there assumes this reasoning is automatic.