Study first
Review the ideas behind the questions
Check the scorecard before trusting a lift. Focus on whether the metrics cover user value and side effects, whether the data is complete enough, and whether the analysis slice matches the people who could be affected.
Read More Than The Lift
A conversion lift can be real and still be a poor rollout decision if the scorecard misses user harm, product health, or data-quality problems.
- A complete experiment scorecard should include the main success measure, diagnostic metrics, guardrails, and data-quality checks.
- Guardrail metrics track outcomes the team does not want to degrade, such as load time, crashes, or abandonment.
- A rollout decision should trade off improvements against regressions instead of reading one local metric alone.
In Practice
Use Guardrails Before Rollout
A landing-page variant that gets more submits but slows the page or raises support contacts needs a tradeoff review, not an automatic rollout.
Separate Diagnostics From The Main Goal
Feature clicks or form starts can help explain a result, but the rollout decision still needs the main success measure and guardrails.
Common mistakes
Calling a winner because one local click metric rose.
Review the main success metric, diagnostics, guardrails, and data-quality checks together before deciding.
Q&A
Why are guardrails part of CRO scorecard checks?
They show whether the test hurt important parts of the experience while improving the target metric.
Can a diagnostic metric explain a lift?
Yes, but it should not replace the main success measure or the guardrail review.
Check Whether The Data Can Be Trusted
Bad assignment, missing fields, and timing problems can make a clean-looking scorecard unsafe for a conversion decision.
- An SRM means the observed treatment and control counts differ from the configured allocation, which can make analysis untrustworthy.
- Treatment assignment information must be present and consistent so treatment and control can be compared.
- Data-quality checks should cover missing rates, invalid values, join rates, uniqueness, and data delays.
In Practice
Treat SRM As A Stop Sign
If the split is not what the test configured, diagnose the cause before reading the conversion lift.
Verify The Assignment Field
A user should not appear in both treatment and control when the test is meant to keep one assignment per user.
Common mistakes
Treating missing rows as harmless because both variants still have some conversions.
Check whether missing data is patterned or unbalanced before trusting the result.
Q&A
What does SRM check?
It checks whether observed treatment and control counts match the allocation the test configured.
Why check timestamp quality?
Bad timestamps can affect event filtering and the order used in triggered analysis.
Use Alerts And Slices Carefully
Early checks, alerts, segments, and triggered analysis can protect users, but the setup must preserve the experiment's decision frame.
- Early metric checks can catch bugs and regressions, but weak early movement should not replace the planned test duration.
- Stable segments are safer for slicing than segments that the treatment can change during the test.
- Triggered analysis needs an inclusive trigger condition and a clean complement check before the triggered lift is trusted.
In Practice
Alert On Serious Harm
Set alerts for important guardrails or data-quality failures so the team can investigate before more users are affected.
Check Date Effects
A one-day spike during a sale, holiday, outage, or tracking change may not generalize to normal traffic.
Common mistakes
Using a triggered readout without checking whether the trigger missed affected users.
Check SRM and the triggered complement before using the triggered result for a rollout decision.
Q&A
When should an alert change the test plan?
When it points to serious user harm, a major guardrail move, or a data-quality issue that needs investigation.