Study first
Review the ideas behind the questions
Review what makes an experiment readable before it runs. A useful test has evidence, a clear hypothesis, a primary metric, checks, a fair split, and a careful interpretation.
Write The Test Before Running It
A test idea becomes an experiment only when the change, expected result, reason, and metric are clear.
- A/B testing compares two or more complete versions of a webpage by splitting web traffic between versions.
- A clear hypothesis states what is changed, what outcome is expected, and why.
- The primary metric should be decided during test design because it is used to prove or disprove the hypothesis.
In Practice
A Change Alone Is Not A Hypothesis
Changing a headline is only a test idea until the expected outcome and reason are written down.
The Main Metric Comes First
Choosing the primary metric after seeing results makes the experiment harder to trust because the test was not designed around that decision.
Common mistakes
Launching a test because a stakeholder likes the new design.
Write the hypothesis and primary metric so the team knows what the test is trying to learn.
Q&A
What should a beginner A/B test hypothesis include?
It should state the change, the expected outcome, and the reason the team expects that outcome.
Why choose a primary metric before the test runs?
It gives the experiment a clear basis for deciding whether the hypothesis was supported.
Check The Test And Read Results Carefully
A test needs a fair setup before results can be trusted. Even then, not every result gives a clear winner.
- A test should be quality assured, including across devices and browsers or sign-in state where appropriate.
- Sample groups should be split equally and randomly when the test is live.
- A properly designed test can still be useful when the result is inconclusive or no clear winning variant appears.
In Practice
Random Split Protects The Comparison
If one version gets a different kind of visitor, the result may reflect traffic assignment instead of the page change.
Small Samples Have Limits
Small usability rounds can explain problems, but surveys, A/B tests, and benchmarking need hundreds of participants for clear findings.
Common mistakes
Calling a test a failure because there was no clear winning variant.
Read whether the design was sound and what the result teaches before deciding the next iteration.
Q&A
What should happen before trusting experiment results?
Confirm the test was checked, metrics recorded correctly, and sample groups were split fairly.
Does an inconclusive test always mean the work was useless?
No. A properly designed test can still give useful insight and guide the next iteration.
Separate Sample Clues From Test Proof
Small studies can find usability clues. A/B tests and benchmarking need much larger samples before the team claims clear findings.
- Usability testing rounds usually need 4 to 8 participants, while A/B testing needs hundreds for clear findings.
- A small launch percentage can be used only if variant proportions remain equal.
- Other metrics and user feedback can be analyzed after the test, but the primary metric remains the main hypothesis check.
In Practice
Tiny Test Results Need Humility
A very small difference on a very small sample is not enough to call a clean A/B winner.
Secondary Metrics Add Context
Secondary signals can explain behavior, but the primary metric is the one chosen to judge the hypothesis.
Common mistakes
Treating a tiny A/B sample as proof while calling a small usability study useless.
Use small usability studies for qualitative clues and use larger samples for clear A/B findings.
Q&A
Can a 12-visitor A/B test safely prove a winner?
No. It is far below the scale usually needed for clear A/B findings.
What should happen to secondary metrics after the test?
They can be reviewed as context, but they should not replace the pre-chosen primary metric as the main win condition.