Website A/B Tests

Website A/B tests help you measure how the Website Personalization module impacts your customers’ average order value and conversion to order. By splitting your site traffic into variants, you can see exactly which forms, popups, and recommendation widgets move the needle — and which ones don’t.

​

How to create an A/B test

A/B tests live under Analytics → A/B Tests. To set up a new test, click Create test and choose Website.

​

How to configure an A/B test

​

Hypothesis

What you’re trying to prove or disprove with the test. Write it down before you launch — it keeps the test honest and makes results easier to interpret later.

​

Participants

The website whose visitors will take part in the test. The lower the traffic on the site, the longer the test will need to run before it reaches a conclusion.

​

Traffic

How site visitors are distributed across test variants. Distribution is entered in shares and is automatically converted into percentages. In this block, you also assign which campaigns are shown in which variant. A campaign assigned to one variant will not be shown to visitors in any other variant.

​

How customers are distributed across branches

In website and personalization tests, participants are distributed by device, not by customer. What happens to a customer with multiple devices?

• If all of the customer’s devices land in the same branch, the customer is counted once in that branch.
• If the devices land in different branches, the customer is counted as a participant in each of those branches. Orders are attributed per device — assigned to the device used for the most recent visit to the site.

​

Analytics

The metrics used to evaluate the success of the tested variants. Primary target metric — the metric used to decide the winning variant. Available primary target metrics:

• Conversion to order — the percentage of customers who placed at least one order after entering the test.
• Average order value — revenue generated during the test divided by the number of orders placed.
• Average revenue per user (ARPU) — revenue generated during the test divided by the number of participants in the variant.

Secondary target metric — a metric you track alongside the primary one to watch for trade-offs. For example, if conversion to order goes up, it’s useful to confirm that average order value didn’t drop in the process.

​

Additional settings

Expected lift, statistical power, and confidence level. These are set to optimal defaults.

​

Launching the test

Once every field is filled in, launch the A/B test by clicking the corresponding button. As soon as you do, the segment’s audience will be split randomly into variants according to your traffic settings.

​

Traffic

​

Global website personalization A/B test

A global website A/B test shows you how website personalization as a whole affects average order value and conversion to order. The report makes it clear how visitor behavior changes when personalization is layered on top of the site, and it helps you calculate the payback of the module. How to set it up:

1. Keep a single variant and set Forms to display to All. Users in this variant will see every active form.
2. Click Add control group. The control group will not see any campaigns — except for those explicitly configured to be shown to everyone.

​

A/B test of a form or set of forms with a control group

A/B tests can show how several mechanics perform together. For example, your site might have a welcome sequence made up of three popups that fire one after another, and you want to check how the sequence affects conversion to order. How to set it up:

1. Keep a single variant and list all popups from the welcome sequence in it.
2. Click Add control group. The control group will not see the welcome sequence assigned to the other variant, but they will still see every other mechanic running on the site.

​

A/B test comparing forms against each other

Use this setup to check which form delivers the most value for a given task. Example: Find out which set of widgets on the product page has the bigger impact on average order value — popular products in the category plus cross-sell items, or popular products plus personal recommendations. How to set it up:

1. In the first variant, specify the forms from the first set.
2. In the second variant, specify the forms from the second set.
3. Click Add control group. Visitors in the control group will not see the product-page widgets specified in the other variants, but they will still see every other mechanic running on the site.

​

How to show a campaign to every site visitor — even those in the control group

Every form has a setting that lets you exclude it from the global website A/B test. This is useful when you have forms that need to reach every visitor regardless of group — for example, a popup announcing that the site is under maintenance.

​

Recommendations

​

A/B test report

Data for every variant appears in the report as early as the day after launch. Inside the report you’ll find graphs for each test metric, plus configuration details — segment, participant count, hypothesis, and additional settings. You can also pick a date range and break the data down by day, week, or month. Reading the results:

• If the test shows that variants are not statistically different, the metric will be labeled as having no winner.
• If there isn’t enough data to call a winner yet, the metric will be labeled as still collecting data.

Results table: Below the graphs is a table with conversion rates and average order values, along with the difference in percent and in dollars compared to Variant 0.