Define the conflict
The starting scenario is deliberately specific: the primary metric improves while conversion rate declines.
Product analytics · Interactive framework
A/B Test Conflict Decision Assistant
An interactive decision tree that helps teams evaluate experiments where the primary metric improves but conversion rate declines.
Overview
Turning a static framework into a guided decision tool
A/B testing decisions are not always straightforward. A variant can improve the primary success metric while simultaneously reducing conversion rate, creating a difficult trade-off between local improvement and overall business performance.
I created this project to make that discussion more structured. Instead of reviewing the full framework at once, users can follow one branch at a time and receive a clear recommendation based on the answers they select.
Process
How the project developed
The final version combines experimentation principles with a simple interface that reveals only the next relevant decision.
Translate the framework
I converted each question and recommendation into ordered nodes with explicit answer paths.
Build the interaction
JavaScript renders the selected branch, hides irrelevant paths, and lets users restart to explore another scenario.
Decision logic
The framework evaluates more than one metric
Each branch adds context before a recommendation is made, preventing the launch decision from being based on a single result in isolation.
Statistical significance
The first check separates a meaningful conversion decline from normal test noise.
Magnitude of impact
The logic treats a small decline differently from a drop greater than five percent.
Customer context
Users are prompted to examine affected segments, friction, confusing copy, and possible trust loss.
Business value
The gain in the primary metric is assessed against overall conversion and longer-term product goals.
Interface
A focused path through a complex framework
The interface grows downward as each answer is selected. Unchosen paths are de-emphasised, while the chosen branch remains visually connected to the next question.
Outcome
Clear recommendations for conflicting experiment results
The assistant supports several practical outcomes, depending on the evidence gathered along the path.
Proceed and monitor
When the decline is not statistically significant, the framework recommends rollout with post-launch monitoring.
Fix and re-test
When the issue is identifiable and fixable, the recommended next step is to amend the variant and run the experiment again.
Do not launch
When conversion risk outweighs the benefit, the framework prioritises overall conversion and captures the learning for a new approach.
Live project
Explore the decision tree yourself
Follow different branches, compare recommendations, and restart the tree to test another path.