Experiments
Designing and reading tests so you know what actually caused a change.
Data Analyst track
Junior
Write correct SQL, build trusted dashboards, ask good questions.
Nothing here yet.
Mid-level
Own an analysis end to end, from vague question to recommendation.
Core: start here
- Decision RulesAgreeing before the test what result ships it, what kills it and what is inconclusive.
- PeekingChecking a running test and stopping the moment it looks significant.
- The A/A TestRunning a test where both groups are identical, to prove your experiment machinery is unbiased.
6 more mid-level concepts
- Guardrail MetricsThe numbers that must not get worse while you chase a win somewhere else.
- Holdback GroupA slice of users deliberately left out of a change so its long-term effect stays measurable.
- Novelty EffectA temporary lift that comes from something being new, not from being better.
- Randomised Controlled TrialRandomly splitting subjects to compare an outcome, the design every A/B test comes from.
- Test DurationRunning long enough to cover weekly cycles and novelty, and no longer than you can defend.
- Treatment EffectThe measured difference the change made, and the many reasons it may not be that difference.
Senior
Own experimentation and metrics design; call out bad numbers.
Core: start here
- Sample Ratio MismatchWhen the groups are not the size the design promised, meaning the assignment itself is broken.
- Sequential TestingTesting rules designed so you can look repeatedly without inflating the false-positive rate.
8 more senior concepts
- Bayesian vs Frequentist TestingTwo readings of probability that give different answers to the same experiment.
- Difference-in-DifferencesEstimating an effect by comparing a treated group's change with an untreated group's change.
- Minimum Detectable EffectThe smallest change worth finding, which decides how much traffic the test needs.
- Multi-Armed BanditsLearning the best option while serving, instead of testing then shipping.
- Multivariate TestingChanging several things at once and using the design to separate their effects.
- Pre-registrationWriting down the analysis plan before seeing the data.
- Quasi-ExperimentMeasuring an effect without randomisation, using groups that were naturally separated.
- Randomization UnitWhether you randomize by user, session or device, and how much it changes what you can conclude.
Staff
Shape how the organization measures and decides.
Nothing here yet.
Principal
Set measurement strategy across the company.
Nothing here yet.