Experiments
Run a flow, banner, or code experiment on a channel, read starters or exposures, and pin a winner or clear the channel.
What experiments are for
An experiment splits traffic on a channel across two or more variants. The first choice is the content kind. A flow experiment serves a published flow version on each variant. A banner experiment serves a published banner version on each variant (a holdout control can resolve with control: true and render nothing while still logging exposure). A code experiment serves a variant key and parameters, and the host reads them with get. The channel and content kind stay fixed after create. Rheo keeps each person on the same variant across sessions.
Use experiments when copy, layout, branching, or monetization needs evidence instead of opinion.

Experiment lifecycle
| Status | What it means |
|---|---|
| Draft | You are configuring variants, traffic split, primary metric, and end date. No users are bucketed yet. |
| Running | The experiment can be attached to its channel. Once the channel is serving it, users are bucketed by your weights and data accumulates. |
| Pending decision | The scheduled end time has passed. If the channel is serving this experiment, users still see the same variants. Traffic does not stop. Promote a winner or extend. |
| Stopped | The experiment is closed. Pin a variant or clear the channel when it was serving this experiment. End leaves a different assignment in place. |
Pending decision: what to do
When an experiment reaches pending decision, Rheo pauses the clock. If the channel is serving this experiment, bucketing stays frozen so late events still land in the right arm.
You will get a notification. Open the experiment detail page and choose one of:
- Promote a winner: pin that variant on the channel, or clear the channel. A flow or banner winner pins that variant's published version. A code winner pins that variant's key and parameters. If the channel is serving something else, End marks the experiment stopped and leaves that assignment in place.
- Extend: push the end date out to gather more data. The experiment returns to Running with the same variants. Extend does not change the channel.

Rheo may show a data readout at pending decision: for example, suggesting you extend if too few people have started the flow, or highlighting a variant that beats the control on your primary metric. Treat this as guidance. Product judgment still matters.
Primary metric
Every experiment has a primary metric: the conversion Rheo uses for rates, lift, significance, and the winning-variant call. You set it while the experiment is in draft. It locks when the experiment starts so results stay comparable.
| Metric | What counts as success |
|---|---|
| Flow completion | Distinct users with a flow_completed event (default) |
| Paywall purchase | Distinct users with surface_outcome and outcome purchase_completed |
| Custom event | Distinct users with the SDK event you pick (see scoping below) |
Flow experiments use starters (flow_started) as the denominator. Banner and code experiments use exposures (experiment_exposed / banner impressions) as the denominator. Code experiments also require a success event name you type while drafting; that name freezes when the experiment starts.
Custom events need a target
Many flows fire the same event on more than one screen or input. When you choose a custom event, Rheo asks you to pin where success happens:
| Custom event | What you must pick |
|---|---|
| Choice selected | The choice input (field key) that should count. Required when the flow has more than one choice layer. |
| Text submitted | The capture layer (field key) that should count — text, scale, wheel picker, date/time, number stepper, phone, address, and similar field-keyed inputs. Required when the flow has more than one capture layer. |
| Step viewed / completed / skipped, surface presented / outcome, decision evaluated, app review prompts, external link opened | The screen or surface (step id) that should count. |
Create the experiment with the event first if variants are not pinned yet. After you add variants with published flow versions, open the draft Primary metric card and pick the layer or step from those manifests. You cannot start the experiment until required targets are set.
Flow-level custom events that do not need a layer or step (for example flow_abandoned) only need the event name.
Winner logic
Rheo compares each treatment to the control variant on the primary metric. A variant can be called the winner when:
- The difference vs control is statistically significant after a Bonferroni adjustment for multiple treatments.
- The lift confidence interval stays positive (treatment is better, not just different).
- Among arms that pass those checks, Rheo prefers the one with the highest relative lift.
If no arm clears the bar, the readout may suggest extending or keeping the control. You still choose what to promote.
Set up an experiment
- Flow experiments: publish each variant as its own flow version. Duplicating a baseline flow in the builder is the usual starting point. Banner experiments: publish each variant as its own banner version. Code experiments: add a key, a weight, parameter rows, and one success event name. There is no flow or banner version to publish.
- Draft the experiment on the channel you want to test. The channel and content kind stay fixed. Choose a success metric, add variant arms, set traffic weights, mark the control, and pick a required end date (rounded to the top of the hour, UTC).
- Each arm's key is the name the dashboard and the host both use. Flow arms also show the flow name and version.
- If a flow experiment's primary metric is a scoped custom event, pin variants first, then pick the input layer or screen / surface on the draft Primary metric card.
- Plan sample size while still in draft. See Experiment sample planning.
- Start when ready. Starting does not put the experiment on the channel. Assign the running experiment from the channel when it should serve. Only Grow plans and above include experiments; concurrent experiment limits depend on your plan.
Keep a separate test channel on a pin if you want internal QA outside the split.
Read results on the dashboard
- The variant marked control is the baseline. Lift, p-values, and "significant" badges for every other arm are pairwise comparisons vs that control on the primary metric. The first variant you add is the control until you pick another.
- Column labels follow the metric (for example completion rate vs purchase rate vs a named custom event rate).
- If you run several treatment arms against one control, Rheo applies a Bonferroni adjustment so overall false-positive risk stays near the planned family rate. See Experiment sample planning.
- Sample ratio mismatch means actual traffic split does not match your configured weights. If Rheo flags this, verify assignment before trusting gaps between arms.
For population-level flow funnels and cohort charts, use Customers & analytics. For one person's journey, use Customers.
Extend or stop early
- Extend when you need more calendar time or traffic. Common at pending decision when sample size targets were not met.
- Stop early only with a documented reason (for example a broken variant or ethical/product constraint). If the channel is serving this experiment, pin a variant or clear it. If the channel is serving something else, end the experiment and leave that assignment in place.
Plan limits
| Plan | Experiments |
|---|---|
| Indie | Not included |
| Grow | Up to 2 concurrent (draft + running + pending decision count toward the cap) |
| Scale | Up to 5 concurrent |
| Enterprise | Unlimited |
Troubleshooting
| Symptom | What to check |
|---|---|
| Uneven exposure between arms | Traffic weights, and whether another channel pin is bypassing the experiment |
| Missing metrics for a variant | SDK events and that the variant id appears in timelines |
| Custom metric shows zero success | The correct layer or step is selected, and that layer/step exists on every variant |
| Paywall purchase stays flat | The flow presents a paywall integration and the SDK emits surface_outcome with purchase_completed. Dollar lift can arrive later from the billing webhook. |
| No clear winner | Widen the observation window via Extend rather than changing weights mid-flight |
Related
- Experiment sample planning
- Customers & analytics
- Channels & rollout
- Concept: Experiment
- Event catalog: SDK event names used as custom metrics