A/B Tests in Flows

Flows let you run A/B tests to split your customer base into groups with a defined probability and validate hypotheses about how different interactions affect your metrics. You can add up to ten A/B tests to a single flow.

​

Setting up an A/B test

When you add an A/B test block to a flow, you’ll configure the following parameters: Hypothesis — what you plan to prove or disprove with the test. Traffic — how participants are distributed across variants, expressed as percentages. You can give each variant a custom name, which makes the test report easier to read and helps you find specific customers later. The maximum number of variants is five, including the control group. Keep in mind that a test with three or more branches will run longer, because it takes more time to accumulate participants in each branch. Analytics — the metrics used to decide which variant wins.

​

Primary success metric

The primary success metric determines the winning variant. The available options are:

• Order conversion rate — the percentage of customers who placed at least one order after being assigned as test participants.
• Average order value (AOV) — revenue generated during the test divided by the number of orders placed.
• Average revenue per user (ARPU) — revenue generated during the test divided by the number of participants in the variant.

​

Secondary success metric

The secondary success metric lets you track other important metrics in parallel with the primary one. For example, if you’re optimizing for order conversion rate, you may also want to make sure average order value doesn’t drop. The same metrics available for the primary success metric are available here.

​

Advanced settings

Advanced settings include expected lift, statistical power, and confidence level. These are set to optimal defaults. Only change them if you fully understand how the change will affect your test results.

​

Control group

You can add a control-group branch to the test. Customers in this branch receive no interactions inside the flow: You can pull customers out of the control group later using a filter.

​

A/B test participants

A customer counts as an A/B test participant only if they:

• Passed through the A/B test block, and
• Successfully completed the group of steps after the A/B test block, or landed in the control-group branch.

If a customer enters the flow and is assigned to one of the A/B test branches, they’ll be assigned to the same branch on any subsequent entry into that flow.

​

Filtering by test branch

You can pull the participants of each variant using a filter, which is useful for follow-up segments, exports, or further analysis.

​

Setup recommendations

Below are common pitfalls to avoid when designing an A/B test. The flow interface also surfaces hints for each of these.

​

Unequal wait times between branches

If one variant has a shorter wait than another, that branch will accumulate participants — and metrics — faster. It can even appear as a false winner while the other branches are still collecting participants and orders. The size of the gap matters: differences of up to 24 hours between branches are acceptable. For example, if one branch has a much shorter wait, by the time the second branch reaches the minimum required number of participants, the first branch may already have seven times more. Those participants also have more time to take the target action, since their orders start counting one day after the branch split.

​

Long wait before the A/B test

A long wait before the A/B test block postpones participant collection by the configured wait time. This doesn’t affect the correctness of the result, but it does extend the test duration. For example, a flow with a 30-day wait before the test will only start collecting participants 30 days after launch.

​

Long wait after the A/B test

A long wait after the A/B test block lengthens the time customers spend in the branch, which dilutes the effect of the tested interaction. The longer a customer is in the branch, the more other communications they receive — which can cause false conversions to be attributed to the test variant. For example, in a flow where users become participants the moment they enter the test but only finish the branch a week later, the effect of the branch should appear after seven days — but conversions begin counting immediately.

​

A “Condition” block between the test and the steps

A “Condition” block placed after the A/B test block but before the step group can distort branch distribution. The percentage split happens inside the A/B test block, but customers only become participants once they complete a step in the assigned branch. If a “Condition” block drops a lot of customers, the actual ratio between variants will diverge from the configured one. Configure these blocks so every customer who enters the condition continues into a step group. For example, if customers in the second branch who don’t have the mobile app are filtered out, those customers will exit the test before reaching the steps — so the real distribution won’t match the configured 50/50. In cases like this, move the check to before the A/B test block, so every customer can become a participant regardless of which variant they’re assigned to. Route the customers who fail the check into branches outside the test.

​

Merging branches after the test

If the interactions after the split are mostly identical between branches, the chance of detecting a difference in metrics — and finding a winner — drops. For example, if two branches differ only in their first steps and then share two identical interactions over the next six hours, the effect of the initial tested communications gets washed out by the time customers reach the last step group. This makes it less likely the test will identify a winner.

​

Editing a flow with an A/B test

Once the flow is live, the test appears under Analytics → A/B Tests.

• The test name cannot be edited — it matches the flow name and version.
• You can only edit the A/B test from the flow page.
• Editing a flow that has an A/B test starts a new test on the current version and ends the previous one, even if the change was made in other blocks.
• Deleting the flow or the A/B test block ends the test but does not delete it. You can delete it manually.
• Stopping the flow ends the test.
• To start a new A/B test, create a new version of the flow using the Edit button.

​

A/B test report

You can open the report from the A/B test block inside the flow or from the test page itself. The report is generated within 24 hours after the test launches. If the test ran for less than 24 hours, no report will be generated. Inside the report, you’ll see charts and a table with values for the primary and secondary success metrics. Variant statistics include orders placed:

• From the moment the customer became a test participant — that is, after they completed the first step in the branch or landed in the control group.
• With no time cap between joining the test and placing the order, as long as the test is still running when the purchase happens.

​

Why the report shows fewer participants than flow entries

A participant is a customer who successfully completed the step group after the branch split. If the steps weren’t completed, the total participant count will be lower than the number of flow entries. Common reasons a customer doesn’t complete the step group:

• Step-group recency window — the steps fell outside their configured recency window.
• Frequency cap — the flow has a limit on how often it can apply to a given customer.