Experiments
Run A/B tests on onboarding flows, read the results, and ship a winner with confidence.
What experiments are for
An experiment splits traffic on a channel across two or more variants — each variant is a published flow version. Rheo keeps each person on the same variant across sessions so results stay comparable.
Use experiments when copy, layout, or branching needs evidence instead of opinion.
Experiment lifecycle
| Status | What it means |
|---|---|
| Draft | You are configuring variants, traffic split, and end date. No users are bucketed yet. |
| Running | The channel serves variants according to your weights. Data accumulates in analytics. |
| Pending decision | The scheduled end time has passed. Users still see the same variants — traffic does not stop. You must promote a winner or extend the experiment. |
| Stopped | The experiment is closed. The channel is pinned to the flow version you chose. |
Pending decision — what to do
When an experiment reaches pending decision, Rheo pauses the clock but does not yank users back to a single flow. Bucketing stays frozen so late events still land in the right arm.
You will get a notification. Open the experiment detail page and choose one of:
- Promote a winner — pick the variant (or another published version) that should become the channel's direct assignment going forward.
- Extend — push the end date out to gather more data. The experiment returns to Running with the same assignment.
Rheo may show a data readout at pending decision: for example, suggesting you extend if too few people have started the flow, or highlighting a variant that beats the control on completion. Treat this as guidance — product judgment still matters.
Set up an experiment
- Publish each variant as its own flow version. Duplicating a baseline flow in the builder is the usual starting point.
- Draft the experiment on the channel you want to test. Add variant arms, set traffic weights, and pick a required end date (rounded to the top of the hour, UTC).
- Name variants in prose for your team if helpful — in the product, each arm is labeled automatically from the flow name and version (for example "Welcome v3"). Duplicate pins get a numeric suffix so lists stay unique.
- Plan sample size while still in draft. See Experiment sample planning.
- Start when ready. Only Grow plans and above include experiments; concurrent experiment limits depend on your plan.
Keep a separate test channel on a direct pin if you want internal QA outside the split.
Read results on the dashboard
- The first variant in the list is the control. Lift, p-values, and "significant" badges for every other arm are pairwise comparisons vs that control at roughly 5% significance (p below 0.05).
- If you run several treatment arms against one control, the chance that at least one arm looks significant by chance is higher than 5%. Treat multi-arm results as exploratory unless you apply stricter statistical rules outside Rheo.
- Sample ratio mismatch means actual traffic split does not match your configured weights. If Rheo flags this, verify assignment before trusting completion gaps between arms.
For population-level funnels and cohort charts, use Customers & analytics. For one person's journey, use Customers.
Extend or stop early
- Extend when you need more calendar time or traffic — common at pending decision when sample size targets were not met.
- Stop early only with a documented reason (for example a broken variant or ethical/product constraint). Pick which published version the channel should serve afterward.
Plan limits
| Plan | Experiments |
|---|---|
| Indie | Not included |
| Grow | Up to 2 concurrent (draft + running + pending decision count toward the cap) |
| Scale | Up to 5 concurrent |
| Enterprise | Unlimited |
Troubleshooting
| Symptom | What to check |
|---|---|
| Uneven exposure between arms | Traffic weights, and whether another channel pin is bypassing the experiment |
| Missing metrics for a variant | SDK events and that the variant id appears in timelines |
| No clear winner | Widen the observation window via Extend rather than changing weights mid-flight |