Skip to main content
A/B tests in site personalization let you split your customer base into groups according to a chosen probability so you can test hypotheses and compare metrics across variants. Use them to validate ideas before rolling a change out to everyone, and to measure the real impact a personalization mechanic has on your business. A/B tests are available for:
  • Information forms and contact collection forms (pop-ups, embedded blocks, and other on-site forms)
  • Recommendation widgets
A/B testing for personalization is now also available for contact collection forms, not only recommendation widgets.

​
How to add an A/B test

To add an A/B test to a personalization mechanic:
1

Create the mechanic

Create a form (pop-up, embedded block, or other on-site form) or a recommendation widget.
2

Customize the design

Configure the appearance, content, targeting, and display conditions as usual.
3

Add the A/B test

Click the Add A/B test button in the mechanic’s settings.
You can also designate a control group that does not see the form. This is the cleanest way to measure how the mechanic itself affects order conversion and average order value — the control group lets you compare customers who saw the form against customers who did not.

​
Test configuration

​
Hypothesis

State what you are trying to prove or disprove with the test. A clear hypothesis keeps the test focused and makes it easier to interpret the result afterwards.

​
Traffic distribution

Traffic is split across variants using proportional values that are automatically converted to percentages. You can add as many variants as you need. Keep the following in mind when choosing how to split traffic:
  • The more variants you add, the larger the sample size you need, and the longer the test will run.
  • Tests with multiple branches can end without a clear winner.
  • Uneven splits (for example, 75% / 25%) take longer to reach significance than an even 50% / 50% split.
Traffic is distributed at the device level. If the same customer visits your site from multiple devices and the devices land in different branches, the customer will participate in both branches. Orders are attributed to the device used during the most recent site visit.

​
Analytics metrics

Choose the metrics you want to track. The primary metrics available are:
  • Order conversion — the percentage of participants who placed an order.
  • Average order value — the average amount of a single order.
  • Average revenue per customer (ARPU) — total revenue divided by the number of participants.
  • Click conversion — the percentage of participants who clicked inside the form or widget.
You can also pick secondary metrics to compare how related indicators move together with the primary one.

​
Additional settings

Under the additional settings you can adjust:
  • Expected uplift — the minimum effect size you want the test to be able to detect.
  • Statistical power — the probability of detecting a real effect if one exists.
  • Confidence level — the probability that the result is not due to random chance.
These values come preset with sensible defaults, so you only need to change them if you have a specific reason to.
Use an A/B test sample size calculator to estimate how many participants you need and how long the test should run before you launch it.

​
How the test runs

​
Launch limitations

A form or widget with an A/B test can only be launched once. After you stop the test, you cannot restart the same mechanic. If you need to change anything and run a new test, clone the form or widget and launch the clone with a fresh A/B test attached.

​
Who counts as a participant

Every customer who matches the targeting and visibility conditions and has actually seen the form or widget is counted as a participant. If the same customer is shown the mechanic multiple times, they always stay in the same branch — assignment is sticky.

​
When reports become available

Reports appear the day after the form or widget is launched. Open them from the mechanic’s settings by clicking View report inside the A/B test configuration block.
Attribution in the A/B test report differs from the standard form or widget report. In the A/B test report, orders are attributed to impressions, not clicks. Because of this, numbers in the A/B test report can differ from numbers in the regular mechanic report — that is expected, not a bug.

​
Best practices

  • Write the hypothesis down before launching so you know exactly what “winning” looks like.
  • Start with a simple 50% / 50% split between two variants when you can — it gives you the fastest, cleanest read.
  • Add a control group whenever you want to measure the absolute impact of a mechanic, not just compare variants to each other.
  • Plan the test duration in advance using a sample size calculator. Stopping a test early because one variant looks ahead is one of the most common ways to draw the wrong conclusion.
  • Run only one A/B test per mechanic at a time, and avoid changing targeting or content while a test is live.

​
FAQ

No. Once a test has been stopped it cannot be restarted on the same form or widget. Clone the mechanic and start a new test on the clone.
The A/B test report attributes orders to impressions of the form or widget, while the regular report attributes orders to clicks. The two views answer different questions, so some difference between them is normal.
Distribution into branches happens per device. If a customer ends up in different branches on different devices, they participate in both. Their orders are attributed to whichever device they used most recently.
Long enough to reach the sample size your expected uplift, power, and confidence level require. Use a sample size calculator before launching to get an estimate, and avoid calling the test before that target is reached.