MVT FAQ

Frequently Asked Questions

What is MVT?

MVT (Multivariate testing) is a reliable and scientific method that helps retailers optimize their merchandising decisions by testing personalization changes against customer traffic. It is a feature within the Omnichannel Personalization framework—designed to deliver actionable insights that will enhance the performance of your recommendations. It quickly and easily identifies which variation has the most positive effect on retailers’ KPIs and displays the results within hours of the test going live.

Will the way I use Algonomy Omnichannel Personalization change?

The MVT tool is accessed through the Algonomy Omnichannel Personalization dashboard and will help you optimize and justify your merchandising decisions. It is an added feature that compliments and enhances your recommendations and will give you actionable insights into the performance of your KPIs.

What kind of tests can I run?

MVT gives you the ability to test a number of different things, including:

  1. Test merchandising rules within recommendations

  2. Test the location of one or more Omnichannel Personalization recommendation carousels on one or more pages

  3. Test the number of placements on a page

  4. Test all Omnichannel Personalization recommendations against no recommendations

  5. In coordination with the Algonomy team, additional page elements outside of personalization can also be tested

For more information see Choosing Which Tests to Run.

Can the same customer be in multiple tests?

No. If a customer is randomly selected to be part of a test group, which will be the only test they participate in.

What happens when a return customer revisits the site?

Our MVT system can identify return customers. If the customer is assigned to a test, the system will keep them in the same test (as long as they do not delete their cookies).

How do I set up an MVT?

Please work with your Algonomy Client Services Engineer to define your treatments and see How to Create Multivariate Tests for step-by-step instructions for setting up a test.

Understanding Lift & Confidence Level

When you see a statistic reported with a confidence level, such as "95% confidence interval," it means that you can be 95% confident that the true value lies within that range. In other words, if you were to repeat the sampling process many times, 95% of the time, the average metric/measure of your sample would fall within that range. Statistical confidence is a crucial concept in understanding and interpreting statistical data. This helps us to assess the reliability of our findings and make informed decisions based on evidence.

Think of it like throwing a dart at a target. If you hit the bullseye every time, you are very confident that your aim is accurate. But if your darts are scattered all over the target, you are less confident that your aim is true. In statistics, the "bullseye" is the true value of the population, and the "scatter" is the variation in your sample. The larger the scatter, the less confident you can be that your sample's average represents the true population average. This is why sample size is important. A larger sample will give you a smaller scatter, making you more confident in your estimate.

  • What does it mean "lift is 3% with a 95% confidence level"?

The "3%" indicates an observed lift, suggesting that the desired outcome is increased by 3% due to the intervention or treatment being studied.

The "95% confidence level" is a measure of the uncertainty or margin of error associated with this estimate. In statistical terms, it means that if you run the same analysis multiple times, you would expect the true lift to fall within a range around the observed 3% in 95% of those analyses. Confidence level is often used to express the reliability of an estimate.

In simpler terms, this is a way of saying that you are reasonably confident (95% confident, to be specific) that the true lift is somewhere in the neighborhood of 3%, but there is a 5% chance that it could be outside of that range due to random variation or other factors.

  • Calculating the range (Confidence Interval): (For a population of 10000):

The range, often referred to as a confidence interval, is a statistical measure that provides an estimated range of values which is likely to include an unknown parameter, in this case, the true lift. The width of this interval gives you an idea of the precision or uncertainty associated with the point estimate (in this case, the 3% lift).

When it is stated that there is a 3% lift with a 95% confidence level, it implies that the 95% confidence interval for the true lift is likely to be centered around the observed 3%. The width of the interval depends on the specific statistical method used and the characteristics of the data.

Confidence Interval = Lift ± Margin of Error

The formula for the margin of error (E) in a confidence interval for a proportion is given by:

  • Z is the Z-score associated with the desired confidence level (for a 95% confidence level, Z is approximately 1.96),

  • p is the observed proportion (in this case, the 3% lift or 0.03),

  • n is the sample size (in this case, the population size, which is 10,000).

Let's plug in the values: E 0.00334

Confidence Interval = 0.03 ± 0.00334 = (0.0267,0.0333) = (2.67%,3.33%)

This means when lift is 3% with a 95% confidence level (for a 10000 population), if you were to conduct the same analysis multiple times, you would expect the true lift to fall within 2.67% to 3.33% in 95% of the cases.