Illustrative setup
Suppose a product is testing a new registration entry. This is a hypothetical work-note scenario, not a real project. Traffic is assigned 50/50, the test is planned for 14 days to cover 2 full weekly cycles, and a 24-hour A/A observation happens before launch.
Write the decision before the measure
State the product decision the test may support: keep the new entry, revise it, or restore the previous version. The primary measure should answer that decision; guardrails observe whether errors, latency, or downstream steps deteriorate.
If the team cannot say what a result would change, even a statistically clear outcome may have no defined use.
Confirm each item before launch
- Randomisation unitChoose user, device, or session, and document cross-device and anonymous-to-signed-in handling.
- Mutual exclusionCheck whether concurrent tests alter the same page, population, or primary measure.
- Measure definitionFreeze numerator, denominator, deduplication, window, time zone, and outlier rules.
- Sample and durationEstimate the sample from the detectable effect and baseline; do not stop for ordinary mid-test movement.
- Quality monitoringPrepare daily checks for allocation, missing events, version coverage, and error rate.
What the A/A observation checks
An A/A observation does not prove that groups will always match. It checks random assignment, event firing, log delay, and report calculations. A clear allocation anomaly within 24 hours should trigger implementation review before the formal test.
Set stopping rules before seeing results
If the plan says 14 days, define which safety issues allow an early stop and which ordinary fluctuations do not. Repeatedly checking the result and stopping at its most favorable point changes the false-positive risk.