Concurrent MVT

Overview

Analytics and CS can create tests that can run in parallel with the merchant-visible tests without affecting any of the merchant's traffic. 100% of the site's traffic is available for a test.

This is a way to run a test that will not be directly visible to the client, and it will not affect the client's ability to run tests on as much of their available traffic as they desire.

It can also help with running overlapping tests. For example, say the client is running a Recs Vs No-Recs test, and you need to optimize a placement configuration. You could set up a Concurrent placement test at the same time to decide between the placement options.

Caution: Because concurrent tests do not have any restrictions, CS/Analytics need to design their tests so that they do not conflict with any other tests. If there are conflicting tests, problems ranging from undesired customer experience to subtle distortions of test results can occur.

There are two types of MVT:

Exclusive: At any given point in time, a visitor to the site can belong to only one exclusive test at a time. Traffic to the site is divided among the given tests through the MVT controls in the dashboard.

Concurrent: A visitor to the site can also belong to any number of concurrent tests.

Note: Concurrent tests are always set up to use 100% of traffic (any traffic not allocated to treatments is automatically allocated to the control)

Only exclusive tests are visible to the client. Behind the scenes, though, Algonomy can set up concurrent tests as needed.

Example 

Say two exclusive tests (A and B) are taking 100% of a site's traffic. Additional concurrent tests (X and Y) can run in parallel: 

  • Exclusive Test A : 20% A1 / 20% A2 / 20% Acontrol

  • Exclusive Test B : 20% B1 / 20% Bcontrol

  • Concurrent Test X : 40% X1 / 60% Xcontrol

  • Concurrent Test Y : 50% Y1 / 50% Ycontrol

Possible example session assignments for a single visitor to the site:

  • YES : Session in A1, X1, Ycontrol

  • YES : Session in Bcontrol, Xcontrol, Y1

  • NO : Session in A1, Bcontrol (The customer would either be part of Test A or Test B, but not both.)

Adding a Concurrent Test

Internal Omnichannel Personalization users can see the Concurrent checkbox in the MVT UI. When you add a treatment to a concurrent test, you have access to 100% of the site's traffic.

Graphical user interface, text, application Description automatically generated

Reporting

MVT dashboard: Only exclusive tests are visible.

Conflicting Tests

Concurrent tests can be dangerous.

One way this can happen is if different treatments on tests all are including injection of JavaScript code. Unless every combination of JavaScript from each possible treatment has been tested together, it can be difficult to be certain that the customer experience will be as desired in all cases.

More often, interference between tests is most likely to happen when either they affect the same functionality (for example, two tests that both affect the same placement), or when one of the tests may affect the traffic flow in a way that could impact other tests.

For example, imagine you have a test on category pages, and a concurrent test on home pages. The treatment in the concurrent test promotes recommendations and moves category navigation further down the page.

  • The treatment assignment for the concurrent test on the home page affects traffic going into the category page test (treatment = less traffic to category, control = usual traffic to category).

  • The impact of the concurrent test will now affect the category page test. Some of the best traffic has been diverted away from the category page, which is going to affect the results and make it difficult to project what the eventual impact will be.

When you design concurrent tests, you need to be aware of all other tests and make sure your test will not interfere with them.