Experiments

What experiments are for

An experiment splits traffic on a channel across two or more variants. Each variant is a published flow version. Rheo keeps each person on the same variant across sessions so results stay comparable.

Use experiments when copy, layout, branching, or monetization needs evidence instead of opinion.

Experiment lifecycle

Pending decision: what to do

When an experiment reaches pending decision, Rheo pauses the clock but does not yank users back to a single flow. Bucketing stays frozen so late events still land in the right arm.

You will get a notification. Open the experiment detail page and choose one of:

1. Promote a winner: pick the variant (or another published version) that should become the channel's direct assignment going forward.
2. Extend: push the end date out to gather more data. The experiment returns to Running with the same assignment.

Rheo may show a data readout at pending decision: for example, suggesting you extend if too few people have started the flow, or highlighting a variant that beats the control on your primary metric. Treat this as guidance. Product judgment still matters.

Primary metric

Every experiment has a primary metric: the conversion Rheo uses for rates, lift, significance, and the winning-variant call. You set it while the experiment is in draft. It locks when the experiment starts so results stay comparable.

The denominator for rates is always distinct users who started the flow (flow_started). The numerator is the success event for your primary metric.

Custom events need a target

Many flows fire the same event on more than one screen or input. When you choose a custom event, Rheo asks you to pin where success happens:

Create the experiment with the event first if variants are not pinned yet. After you add variants with published flow versions, open the draft Primary metric card and pick the layer or step from those manifests. You cannot start the experiment until required targets are set.

Flow-level custom events that do not need a layer or step (for example flow_abandoned) only need the event name.

Winner logic

Rheo compares each treatment to the control (first variant) on the primary metric. A variant can be called the winner when:

1. The difference vs control is statistically significant after a Bonferroni adjustment for multiple treatments.
2. The lift confidence interval stays positive (treatment is better, not just different).
3. Among arms that pass those checks, Rheo prefers the one with the highest relative lift.

If no arm clears the bar, the readout may suggest extending or keeping the control. You still choose what to promote.

Set up an experiment

1. Publish each variant as its own flow version. Duplicating a baseline flow in the builder is the usual starting point.
2. Draft the experiment on the channel you want to test. Choose a primary metric, add variant arms, set traffic weights, and pick a required end date (rounded to the top of the hour, UTC).
3. Name variants in prose for your team if helpful. In the product, each arm is labeled automatically from the flow name and version (for example "Welcome v3"). Duplicate pins get a numeric suffix so lists stay unique.
4. If the primary metric is a scoped custom event, pin variants first, then pick the input layer or screen / surface on the draft Primary metric card.
5. Plan sample size while still in draft. See Experiment sample planning.
6. Start when ready. Only Grow plans and above include experiments; concurrent experiment limits depend on your plan.

Keep a separate test channel on a direct pin if you want internal QA outside the split.

Read results on the dashboard

• The first variant in the list is the control. Lift, p-values, and "significant" badges for every other arm are pairwise comparisons vs that control on the primary metric.
• Column labels follow the metric (for example completion rate vs purchase rate vs a named custom event rate).
• If you run several treatment arms against one control, Rheo applies a Bonferroni adjustment so overall false-positive risk stays near the planned family rate. See Experiment sample planning.
• Sample ratio mismatch means actual traffic split does not match your configured weights. If Rheo flags this, verify assignment before trusting gaps between arms.

For population-level flow funnels and cohort charts, use Customers & analytics. For one person's journey, use Customers.

Extend or stop early

• Extend when you need more calendar time or traffic. Common at pending decision when sample size targets were not met.
• Stop early only with a documented reason (for example a broken variant or ethical/product constraint). Pick which published version the channel should serve afterward.

Plan limits

Troubleshooting

Related

• Experiment sample planning
• Customers & analytics
• Channels & rollout
• Concept: Experiment
• Event catalog: SDK event names used as custom metrics