Volume 6 · fourteen days
Every analysis so far assumed your scale works.
A mean of seven items treats them as interchangeable, equally related to one construct, and equally interpreted by every group you compare. Those are testable claims, and testing them is psychometrics. This volume runs from item analysis and reliability beyond alpha, through factor structure and confirmatory models, to measurement invariance, latent regressions, longitudinal SEM and networks.
14
days, items to latent structure
4
assumptions Cronbach's alpha needs
3
invariance steps before any group comparison
Day 53
45 minutes · what a scale score assumes
What a mean scale score silently assumes
When you write rowMeans(items) you assert four things: the items measure one thing, they measure it about equally well, the response options mean the same distances, and they mean the same thing to everyone you compare. Each is testable. If they fail, every model in volumes 4 and 5 inherits the failure — attenuated correlations, confounded group differences, and a construct label that does not match what you measured.
| The assumption | If it fails | Tested on day |
|---|---|---|
| The items measure one construct | Your scale score is a blend of two things; correlations with it are uninterpretable. | 57, 58, 60 |
| Items relate to it about equally | Alpha understates reliability; unweighted means misrepresent the construct. | 55 |
| Items are not redundant or dead | Wasted length, inflated alpha, participant fatigue. | 54, 55 |
| It means the same across groups | A "group difference" may be a measurement difference. Fatal for cross-cultural claims. | 63 |
| It means the same across time | "Change" may be change in interpretation, not in the construct. | 63, 65 |
The sentence that fails review
"Depression was measured with a nine-item scale (α = .87)." Alpha is not evidence of unidimensionality, it is not the reliability of your composite unless the items are essentially tau-equivalent, and it says nothing about whether your groups used the scale the same way. The stronger version names the factor structure you confirmed, omega, and the invariance level you established — three extra clauses that take this volume to produce and remove an entire category of reviewer objection.
Do this now · 15 minutes
For each scale in your dataset, write the four assumptions as four sentences about your own items, and mark which you have evidence for. That list is your day-53-to-63 work plan.
Day 54
50 minutes · which items earn their place
Item analysis: the boring day that saves the scale
Before factor analysis, look at each item's distribution and its relationship to the rest. Three diagnostics find almost every problem: a corrected item-total correlation below about .30 (the item is not measuring the same thing), extreme skew with a floor or ceiling (no variance to work with), and a near-perfect correlation with another item (redundancy that inflates alpha without adding information).
Do not maximise alpha
Dropping every item that raises alpha when removed converges on a scale of near-identical items — high alpha, narrow construct, poor validity. This is the attenuation paradox: reliability and validity stop moving together. Drop an item when it is substantively off-construct, badly worded, or empirically dead (r.drop below .20, no variance), and keep breadth otherwise. Justify each removal in the paper.
Do this now · 20 minutes
Run
alpha() on each of your scales and tabulate n, r.drop, mean, SD and alpha-if-dropped per item. Flag any item with r.drop under .30, and write a substantive justification before removing anything.
Day 55
55 minutes · the four assumptions
Alpha, omega, and when alpha is simply wrong
Cronbach's alpha estimates the reliability of a composite only if the items are unidimensional, essentially tau-equivalent (equal loadings), have uncorrelated errors, and are continuous. Real scales rarely satisfy the second, often violate the third, and never the fourth for Likert items. Omega relaxes the equal-loadings assumption and is the better default — a two-minute change with a real effect on the number you report.
| Coefficient | Assumes | Use when |
|---|---|---|
| Alpha | Unidimensional, equal loadings, uncorrelated errors | Reporting convention; always alongside omega. |
| Omega total | Unidimensional; loadings may differ | The sensible default for a scale score. |
| Omega hierarchical | A general factor plus group factors | Multidimensional scales: how much of the composite is the general factor? |
| Ordinal alpha / omega | Polychoric correlations for ordered categories | Fewer than five response options. |
| Test-retest (ICC) | Stable trait over the interval | Trait claims; report the interval length. |
| Split-half / Spearman-Brown | Two parallel halves | Two-item scales, where alpha is meaningless. |
Three real-world rules
Two-item scales: alpha is not interpretable — report the Spearman-Brown coefficient or just the inter-item correlation. Few response options: with three or four categories, use polychoric-based ordinal omega (
psych::omega(..., poly = TRUE)) or alpha is biased downwards. Multidimensional scales: a high alpha on a two-factor scale is a warning, not a reassurance — alpha rises with the number of items regardless of structure. Establish dimensionality first (days 57–61), then compute reliability.Do this now · 25 minutes
Recompute reliability for every scale in your dataset as omega total with a bootstrapped interval, next to your existing alpha. Where they differ by more than .02, your loadings are unequal — note it and report omega.
Day 56
50 minutes · coding, observation, diagnosis
Agreement: choosing the right ICC form, and kappa
If humans coded your data — behavioural observations, open-text responses, clinical ratings — reliability means agreement between raters, and the statistic depends on the design. The common error is reporting "the ICC" without saying which of the six forms, because they can differ by a lot and they answer different questions.
| Your situation | Report |
|---|---|
| Each target rated by different raters drawn from a pool | ICC(1) — one-way random |
| All targets rated by the same raters, who represent a population | ICC(2) — two-way random, absolute agreement |
| All targets rated by the same raters, who are the only raters of interest | ICC(3) — two-way mixed, consistency |
| Your analysis uses the MEAN of k raters | The k-variant (ICC2k / ICC3k) — higher, and correct for a composite |
| Nominal categories, two raters | Cohen's kappa (irr::kappa2), weighted if ordinal |
| Nominal categories, three or more raters | Fleiss' kappa (irr::kappam.fleiss) |
Reporting agreement
Name the form, the number of raters, the number of targets, whether raters were crossed or nested, and the interval — "ICC(2,1) = .64, 95% CI [.44, .80], two raters crossed over 30 targets". Report agreement on the same data your analysis uses (a subset double-coded for reliability is standard: say what proportion). And decide your disagreement-resolution rule in advance: consensus discussion, third rater, or averaging.
Do this now · 20 minutes
If any of your variables were human-coded, run the right ICC form or kappa and write the full reporting sentence. If none were, run
ICC() on the example data and read the six rows until you can say what each one asks.
Day 57
55 minutes · the decision that shapes everything
Dimensionality: parallel analysis, not eigenvalues over one
The Kaiser criterion (eigenvalues greater than one) systematically overextracts and should not be used. Parallel analysis compares your eigenvalues against those from random data of the same size — keeping only factors that beat noise — and it is the best-performing simple method available. Run it, look at the scree plot, and then let theory arbitrate, because the statistical answer is often ambiguous by one factor.
| Method | Verdict |
|---|---|
| Parallel analysis | Best simple default. Beats noise, only mildly over-extracts. |
| Velicer's MAP | Good, slightly conservative. Useful as a second opinion. |
| Scree plot | Read it — the elbow carries information no criterion captures — but it is not decisive alone. |
| Eigenvalue > 1 (Kaiser) | Do not use. Overextracts badly; retained only by habit and SPSS defaults. |
| % variance explained | Not a criterion for the number of factors. There is no threshold. |
| EGA (day 66) | Network-based dimensionality; strong performance, increasingly reported. |
| Theory and interpretability | The tie-breaker. A sixth factor with two items and no name is not a factor. |
How to report the decision
Never report only the number you kept. Report what each criterion suggested, that they disagreed if they did, which solutions you examined, and why you chose the one you did — interpretability and theoretical fit are legitimate reasons when stated openly. A reviewer objects to a hidden decision, not to a reasoned one.
Do this now · 25 minutes
Run
fa.parallel() and nfactors() on your item pool and write down every suggested number. Then extract the two most plausible solutions and keep both files — tomorrow you decide between them on interpretability.
Day 58
55 minutes · letting the data speak
EFA: extraction, oblique rotation, and naming factors honestly
Exploratory factor analysis with maximum likelihood extraction and oblique rotation is the default for psychological data, because psychological constructs correlate — orthogonal rotation forces a structure your theory does not claim. Read the loading matrix for simple structure: each item loading clearly on one factor, cross-loadings small, every factor with at least three good items.
| Choice | What to pick |
|---|---|
| Extraction | fm = "ml" (maximum likelihood) or "minres". Both fine; ML gives fit statistics. |
| Rotation | "oblimin" or "promax" (oblique). Use "varimax" only if you truly claim uncorrelated factors. |
| Correlation type | Polychoric (cor = "poly") with fewer than five response options; Pearson otherwise. |
| Loading cutoff for reporting | Report all loadings; suppress below .30 for readability and say you did. |
| Cross-loadings | An item loading above .30 on two factors is ambiguous — flag it, consider dropping, never hide it. |
| Minimum items per factor | Three well-loading items. Two is unstable; one is not a factor. |
Naming factors, honestly
A factor's name is a hypothesis about what its items share, and it is the most over-claimed object in psychometrics. Name it from the items that actually load — not from the questionnaire's marketing, not from the construct you hoped to find. If the three highest loadings are all reverse-keyed items, you may have found a method factor rather than a construct. Write the name, then show a colleague the items without it and see what they call it.
Do this now · 25 minutes
Run your two candidate solutions with oblique rotation, print loadings with a .30 cutoff, and name every factor from its items. Count cross-loadings and items with communalities under .20. Choose the solution you can defend in a sentence, and record the reason.
Day 59
40 minutes · a short, important day
Why reviewers catch "PCA to identify factors"
Principal components analysis and factor analysis answer different questions. PCA finds weighted composites that maximise explained variance — a data-reduction technique with no measurement model and no error term. EFA posits latent causes producing the items, separating common variance from item-specific error. If your claim is about a construct, PCA is the wrong tool, and the difference shows up in the loadings.
| PCA | EFA | |
|---|---|---|
| Question | Can I summarise these variables in fewer? | What latent variables produce these items? |
| Error term | None. Components are exact composites. | Item-specific variance is modelled and removed. |
| Direction | Items → components | Factors → items |
| Loadings | Inflated relative to EFA | Common-variance only |
| Legitimate use | Data reduction, collinearity, image compression, control variables | Measurement, scale development, construct claims |
| Reports as | "We reduced 40 physiological channels to 5 components" | "A five-factor structure fitted the item responses" |
The one-line rule
If the word "construct", "latent" or "measures" appears in your sentence, use EFA or CFA. If the sentence is "we needed fewer columns", PCA is fine and honest. The failure mode reviewers flag is a paper that runs
principal(), calls the output "factors", and then reports alpha for each — three incompatible steps in one paragraph.Do this now · 15 minutes
Run
principal() and fa() on the same items and compare the loading magnitudes and the variance accounted for. Then check your own draft or supervisor's previous papers for the word "factor" applied to PCA output.
Day 60
60 minutes · testing a structure you predicted
Confirmatory factor analysis: the syntax and the standardised solution
EFA asks what the structure is; CFA tests a structure you specify in advance. lavaan's model syntax is three operators — =~ for "is measured by", ~~ for a covariance, ~ for a regression — and one convention: the first indicator's loading is fixed to 1 to set the latent scale unless you standardise.
| Syntax | Meaning |
|---|---|
f =~ x1 + x2 + x3 | Latent factor f is measured by these indicators. |
f1 ~~ f2 | Estimate the covariance (correlation, when standardised) between two latents. |
x1 ~~ x2 | Correlated residuals — only with a substantive justification (day 62). |
y ~ f1 + f2 | Structural regression — latent predictors (day 64). |
f =~ NA*x1 + x2 | Free the first loading; combine with std.lv = TRUE to fix the factor variance to 1 instead. |
estimator = "MLR" | Robust ML: correct SEs and a scaled chi-square under non-normality. A sensible default. |
missing = "fiml" | Full-information ML — uses all available data instead of deleting cases. |
Identification, in practice
A factor needs at least three indicators to be identified on its own (two if it correlates with another factor). Latent scale must be set somehow — by fixing the first loading to 1 (default, so the loading table is in raw units) or by fixing the factor variance to 1 (
std.lv = TRUE, so loadings are standardised directly). Read the Std.all column for interpretation: those are the loadings you report, and anything under about .40 deserves a comment.Do this now · 30 minutes
Specify the factor structure you concluded on day 58 as a lavaan model, fit it with MLR and FIML, and report the standardised loadings. Flag every indicator loading below .40 and decide — in writing — whether to keep it.
Day 61
55 minutes · thresholds, and their limits
Fit: what each index measures, and reporting it without spin
Chi-square tests exact fit and rejects essentially every real model at large n, so approximate fit indices developed. The conventional set is CFI and TLI (comparative, higher is better), RMSEA (parsimony-adjusted misfit, lower is better, with a confidence interval) and SRMR (average residual correlation). Thresholds are rules of thumb from simulations with specific conditions — not laws.
| Index | Conventional | What it actually tells you |
|---|---|---|
| Chi-square | p > .05 | Exact fit. Rejected by trivial misfit at n > 400. Report it; do not rely on it. |
| CFI | ≥ .95 good, ≥ .90 acceptable | Improvement over a model with no relationships. Sensitive to weak overall correlations. |
| TLI | ≥ .95 | Like CFI, penalised for complexity. Usually slightly lower. |
| RMSEA | ≤ .06 good, ≤ .08 acceptable | Misfit per degree of freedom. Unreliable with small df and small n — report its CI. |
| SRMR | ≤ .08 | Average standardised residual. The most directly interpretable: how far off are the correlations? |
| AIC / BIC | Lower | Only for comparing models on the same data. Not absolute fit. |
Reporting without spin
Three failure modes. Cherry-picking: reporting only CFI because RMSEA failed — report all four, always. Threshold theatre: "acceptable fit (CFI = .901)" for a model whose RMSEA is .11 and whose residuals show a clear second factor. Silence about respecification: fitting six models and reporting the last. State how many models you fitted, in what order, and why. A mediocre fit honestly described and diagnosed is publishable; a polished fit with a hidden history is a retraction risk.
Do this now · 25 minutes
Report all six fit measures for your CFA, then print the residual correlation matrix and find the three largest residuals. Write one sentence explaining what the misfit is — that sentence is what day 62 acts on.
Day 62
50 minutes · the overfitting trap
Respecification: defensible and indefensible
Modification indices tell you how much chi-square would drop if you freed a fixed parameter. Following them mechanically is the fastest route to a model that fits this sample and nothing else: simulation work shows MI-driven respecification recovers the true model rarely, and capitalises on sampling noise reliably. The index is a diagnostic tool, not an instruction.
| Respecification | Defensible? |
|---|---|
| Correlated residual between two items with near-identical wording | Yes — state the wording overlap as the justification. |
| Correlated residuals among reverse-keyed items | Often yes — a documented method effect. Say so. |
| Correlated residuals between adjacent items in a long questionnaire | Sometimes — proximity and carry-over effects are real. Justify. |
| A cross-loading that the MI suggests and your theory does not | No. This is fitting noise; it changes what the factor means. |
| Freeing parameters one at a time until CFI passes .95 | No. This is the textbook overfitting trap. |
| Dropping an indicator because it lowers fit | Only with a content reason, and report the model both ways. |
The publishable framing
"The initial model showed marginal fit (RMSEA = .091). Inspection of residuals and modification indices indicated substantial shared variance between N1 and N2, whose wording is near-identical. We freed that residual covariance — the single modification we made — which improved fit to RMSEA = .066, CFI = .958. Both models are reported in Table 2; the substantive conclusions are unchanged." That paragraph is transparent, justified and reviewable.
Do this now · 20 minutes
Inspect your modification indices, and for each of the top three write the content justification you would put in the paper — or admit there is none. Make at most one change, and report both models with
compareFit().
Day 63
60 minutes · before any group comparison
Invariance: the test your cross-group claim depends on
Comparing latent means or correlations across groups — gender, culture, clinical status, or time points — assumes the measurement works the same way in each. That is tested in a nested sequence: configural (same structure), metric (same loadings), scalar (same intercepts). Without at least metric invariance, a group difference may be a difference in how the scale is understood, not in the construct.
| Level | Constrains | Licenses |
|---|---|---|
| Configural | Same items on same factors | Saying the construct has the same structure in both groups. Nothing quantitative. |
| Metric (weak) | Loadings equal | Comparing correlations, regressions and covariances across groups. |
| Scalar (strong) | Loadings and intercepts equal | Comparing observed or latent MEANS. This is what a t-test between groups needs. |
| Strict | Also residual variances equal | Rarely required; comparing observed variances. |
| Partial | One or two constraints relaxed | Comparisons are still defensible if most indicators are invariant — report which are not and why. |
The decision criterion, and what failure means
Judge by change in fit, not by the chi-square difference test, which rejects trivial non-invariance at large n: ΔCFI > .01 or ΔRMSEA > .015 indicates non-invariance. Here scalar invariance fails, so a raw mean comparison between genders is not licensed. The productive response is not to abandon the comparison but to locate it —
partialInvariance() finds the offending intercept — and then either report a partial-invariance model or reframe the finding as differential item functioning, which is itself interesting.Do this now · 30 minutes
Run the configural-metric-scalar sequence for the grouping variable your paper compares. Report ΔCFI at each step. If scalar invariance fails, locate the item and write the sentence you would put in the limitations — or reframe the claim.
Day 64
60 minutes · latent variables in regressions
SEM: regressions between latent variables, without measurement error
The payoff of measurement modelling is that structural coefficients between latent variables are corrected for unreliability — the attenuation from volume 4 day 35 disappears, because error is estimated rather than absorbed into the estimate. The cost is sample size, complexity and a fit that must hold for both parts of the model at once.
Three practical limits
Sample size. Rules of thumb (5–10 cases per estimated parameter; n ≥ 200 for modest models) are crude — better to run a Monte Carlo power analysis for your specific model (volume 8 day 79 gives the recipe,
simsem automates it). Two-step thinking. Establish acceptable measurement fit before adding structural paths; otherwise you cannot tell which part is misfitting. Factor scores. Exporting lavPredict() output into a regression discards the measurement-error correction you just paid for — keep the model latent if you can.Do this now · 30 minutes
Convert your main volume-4 regression into a latent model: measure each construct with its items, fit the measurement model alone first, then add the structural paths. Compare the latent structural coefficient with your earlier observed-variable estimate — the difference is attenuation.
Day 65
60 minutes · the trait-state question
Cross-lagged panels, and why RI-CLPM replaced them
The classic cross-lagged panel model regresses each variable on both variables at the previous wave and calls the cross-paths evidence of directional influence. The problem: with three or more waves it conflates stable between-person differences with within-person change, so a cross-lagged path can be large and significant when nothing changes within anybody. The random-intercept version separates them, and it has become the expected analysis.
| Model | Use when |
|---|---|
| CLPM | Two waves only, or when between-person prediction is genuinely the question. Say which. |
| RI-CLPM | Three or more waves and a within-person hypothesis. The current default. |
| Latent growth model | The question is the shape of change, not reciprocal influence. Volume 5 day 50, in SEM form. |
| Latent change score | Change between adjacent waves as an explicit latent variable. |
| Multilevel (lmer) | Many unequally spaced occasions — ESM. Volume 5. |
Continuous-time (ctsem) | Intervals differ a lot between people; lagged effects depend on interval length. |
The interval problem, briefly
A cross-lagged coefficient is not a property of the process, it is a property of the process and your measurement interval. The same underlying dynamic yields different lagged coefficients at one week, one month and one year, and effects can even change sign as intervals lengthen. So never compare your lagged coefficients with another study's unless the intervals match, and state the interval whenever you report one.
Do this now · 30 minutes
If you have three or more waves, fit both the CLPM and the RI-CLPM and compare the cross-paths. Write the sentence that describes what changed. If you have two waves, write the limitation paragraph explaining what a two-wave design cannot separate.
Day 66
55 minutes · an alternative to latent causes
Networks: symptoms as an interacting system
Network psychometrics replaces the latent-cause assumption with a different one: that symptoms directly influence each other. Statistically it estimates a regularised partial correlation network — each edge is the association between two variables conditional on all the rest. It is genuinely useful and routinely over-interpreted; stability analysis is what separates the two.
Four honest caveats
Cross-sectional networks are not causal, and an edge is not a mechanism — it is a conditional association in one sample. Centrality is unstable at typical sample sizes, especially closeness and betweenness; report the CS-coefficient and do not rank nodes without it. Edges are shrunk by regularisation, so absent edges are not evidence of conditional independence. Networks and factor models are often statistically equivalent — choosing one is a theoretical commitment, so argue for it rather than presenting the network as assumption-free.
Do this now · 25 minutes
Estimate a network on any multi-item dataset, then run both bootstraps. Write down the CS-coefficient for strength and state plainly which centrality indices you are entitled to discuss. Compare EGA's dimension count with your day-57 answer.
Reference
The order these steps must happen in
Every step assumes the previous one passed. Running them out of order is how scales get published that do not measure one thing.
1 · items
Distributions, floor and ceiling, item-total correlations, redundancy. Day 54.
2 · dimensionality
Parallel analysis plus a second criterion, arbitrated by theory. Day 57.
3 · structure
EFA for discovery, CFA for confirmation — ideally in different samples. Days 58, 60.
4 · fit
All four indices, plus the residual matrix. Day 61.
5 · reliability
Omega with an interval, given the structure you established. Day 55.
6 · invariance
Configural, metric, scalar — before any group or time comparison. Day 63.
7 · structure between constructs
Latent regressions, error-corrected. Day 64.
Reference
Sentences you can paste and adapt
Reliability
Internal consistency was adequate, omega total = .84, 95% CI [.83, .85] (alpha = .81). Corrected item-total correlations ranged from .44 to .71.
EFA
Parallel analysis suggested five factors and Velicer's MAP five; a six-factor solution added no interpretable factor. Maximum-likelihood extraction with oblimin rotation yielded five factors with loadings from .43 to .83 and two cross-loadings above .30 (A4, O4).
CFA
The five-factor model fitted acceptably, chi-square(265) = 812.11, p < .001, CFI = .958, TLI = .942, RMSEA = .066 [.062, .070], SRMR = .041, with one residual covariance freed between two near-identically worded items.
Invariance
Metric invariance across gender held (ΔCFI = -.001, ΔRMSEA = -.006), but scalar invariance did not (ΔCFI = -.013). Partial scalar invariance, freeing the N4 intercept, was tenable, so latent means are compared under that model.
Agreement
Two raters independently coded 25% of transcripts; agreement was ICC(2,1) = .64, 95% CI [.44, .80], and disagreements were resolved by discussion.
SEM
The structural model fitted well, CFI = .961, RMSEA = .052 [.045, .060]. Stress predicted depression, beta = .41, and the indirect effect via support was .11, bootstrapped 95% CI [.06, .16] from 2,000 resamples.
Reference
Install these once
Checkpoint
Seven questions before volume 7
{{ quizCounter }}
{{ quizScore }}
{{ quizQ }}
{{ quizFb }}
Next: volume 7
Ten days on posteriors and synthesis: brms with the formula syntax you already know, priors you can defend, convergence diagnostics, Bayes factors and evidence for the null — then a complete metafor meta-analysis from effect-size coding to a publishable forest plot.
Start day 67 →