Multivariate Testing Glossary

Here are some common terms used in setup and analysis of Multivariate Tests:

Metric/Term Name	Description
A/B Test	A common, widely used name for website testing. It specifically refers to testing version A of a website versus version B of a website, but the term is often used in other cases (multiple treatments, email tests, etc.).
Confidence Level	Refers to the margin of error on our assertion of significance. If you decide to act on a test when it reaches a confidence level of 95%, then in roughly 5% of the tests you do, you will be acting on a false positive result.
Control	The Control is the current state of your recommendations environment that all treatments are compared against.
Flippers or "Flip Floppers"	Flippers or "Flip Floppers" Only applies if you are using the legacy, cookie-based treatment assignment and tracking approach. Current best practice recommendations are to only use Deterministic Treatment Assignment, see above. One major advantage of Deterministic Treatment Assignment is that it cannot produce flippers. Because some users may configure their web browsers to either reject cookies or use firewalls, they do not interact with the test in a consistent way. These users will "flip" between treatments. Ideally, the test should include users that have a consistent view of the experiment. The Omnichannel Personalization detects session "flip floppers" during the test. Due to the unpredictability of this user behavior, these sessions are excluded from test. This ensures more refined and accurate data.
Lift	Lift refers to the change created by a treatment, whether positive or negative. Within the UI, we compare each metric (such as revenue per session or average order size) for a treatment against the control to determine lift for the treatment.
MVT	Refers to the Algonomy testing tool, as well as the general concept of "MultiVariate Testing." The broader notion of multivariate testing applies to situations where there are multiple factors with different values, and the test is going to evaluate different combinations of those factors. For example, one factor might be where on the page to put a new placement, and another factor would be which strategies to prefer in the placement. The Algonomy MVT tool can support that type of test, but requires the user to define each treatment to be the combination of factors.
Outliers	An outlier is considered a visit with an abnormally large purchase. When performing an MVT with the Omnichannel Personalization, the purpose is to detect the impact of a treatment. Outliers in the data set can distort the results of your test. Within the UI, you have the option to view results with or without outliers. An outlier is any session that contains an order that is more than three standard deviations from the mean in a log-normalized distribution of order values.
Test Totals	The test totals feature in the Reporting page gives you an overall view of results and key metrics (such as sessions, revenue, revenue per session, average order size, click-through rate, conversion rate).
Treatment	A treatment refers to the condition you are imposing on the experiment. This treatment is administered on a subset of traffic.
Unqualified Users	Web site visitors that are assigned to either the control or the treatment groups, but are not exposed to the actual test. An example would be a test that is comparing treatments on the home page and the user never navigates to the home page to see the treatment or control (i.e., they enter via third party search engines).
Visits	A visit is determined as a subset of a session, marked by a 30-minute (or more) gap in activity before continuing. As an example: A user logs on and looks at various products, but then leaves the computer without closing the website. When the user returns 30 minutes later and continues shopping, they are starting a second visit.