Mobile App A/B Tests

Mobile app A/B tests let you measure the value of individual in-apps and of the channel as a whole, and pick the best in-app for a specific use case. To create a mobile app A/B test, go to Analytics → A/B tests, click Create test, and choose Mobile app.

​

Test setup

Hypothesis — the statement you want to prove or disprove with the test. Participants — the apps whose users will take part in the test. Variants — how app users are split across the test. Distribution is set as shares and automatically recalculated into percentages. For each variant, you can define which in-apps are shown:

• Show only specific in-apps.
• Show all in-apps.
• Show the in-apps that are not selected in the other variants (control group).

How are customers distributed across branches?

In website and personalization tests, participants are split by device.What happens to a customer with multiple devices?

• If all devices land in the same branch, the customer counts as one participant in that branch.
• If devices land in different branches, the customer counts as a participant in each of those branches. Orders are attributed by device — to whichever device was last used to visit the site.

The ways to configure variants depending on your goal are described below. Analytics — the metrics used to evaluate the performance of each variant.

• Primary target metric — the metric that determines the winning variant.
  Returns and cancellations are not counted in any of the metrics.
  • Order conversion — the percentage of customers who placed at least one order after entering the test.
  • Average order value — revenue during the test period divided by the number of orders placed.
  • Average revenue per user (ARPU) — revenue during the test period divided by the number of participants in the variant.
• Secondary target metric — a comparison metric for tracking trends. For example, if order conversion goes up, you want to make sure average order value did not drop. The same metrics are available as for the primary target.
• Advanced settings — expected uplift, statistical power, and confidence level. Default values are optimal.

Once all fields are filled in, you can start the A/B test by clicking the corresponding button. Before launching, make sure the in-apps you selected are in the Running status. As soon as the test starts, the segment audience is split into variants at random, according to the traffic settings. After the test is launched, only its name can be edited.

​

Variants

​

Global app test

A global A/B test shows how in-apps affect average order value and order conversion overall. The report tells you how user behavior shifts with in-apps and helps you calculate the channel’s payback. Create two variants:

• One that shows all running in-apps.
• A control group.

​

Testing a specific in-app

Use an A/B test to measure how well an individual mechanic performs. Create two variants:

• One that shows the in-app(s) you want to evaluate.
• A control group that sees all in-apps except the one being tested.

​

Comparing in-apps against each other

Use an A/B test to find out which set of in-apps better solves a given task. Create one variant per set of in-apps.

​

Report

Data for every test variant appears in the report the day after the A/B test is launched. Inside the report you can see charts for each test metric along with the configuration details — segment, number of participants, hypothesis, and advanced settings. You can also pick a time range and break the data down by day, week, or month. If the test shows that variants do not differ in a statistically significant way, the metric will be labeled as having no winner. If there is not enough data to determine a winner, the metric will indicate that data is still being collected.