MVT Reporting

Overview

Once tests have been created, you can measure the results of the test across various KPIs. The MVT report link can be found within the Omnichannel Personalization dashboard under Optimization > MVT. This page will give you visibility into test details, such as:

  • Daily totals for raw metrics (KPIs, lift)

  • Multiple graphs per page for easy comparisons

  • Confidence levels per metric

  • Adjustable date ranges

  • Test duration

  • Test totals

  • Eligible traffic

  • Audit log

The reporting page provides insight into the KPIs that matter to you with regard to tests being conducted on the site

You can create one or more tests. Each test will at a minimum have one treatment and one control. You designate a desired percentage of traffic for each treatment. Customers who are placed in your test are assigned to the control or one of the treatments according to these desired percentages.

Multivariate Tests Reports

Navigation

To view MVT Tests, in the Omnichannel Personalization Cloud dashboard, go to Optimization > MVT.

Graphical user interface, text, application Description automatically generated

On the Multivariate Tests page, all currently running tests will be shown. To see expired and canceled tests, click the box beside Show Expired. Choose the test you wish to view reports for and click Report. The reports will be available the day after the test starts.

Graphical user interface, application, Teams Description automatically generated

Attributes Section

Date Range

The first block of the page addresses some of the options for the reporting. Users can choose a date range, allowing them to look at various slices of time rather than simply the entire test range.

A picture containing diagram Description automatically generated

The date range defaults to the entire duration of the test. To change these dates, click on either of the dates. This opens the date picker with calendar. Select the desired dates and click the green check mark.

Add dimension filter

MVT reports enable users to filter the test results using various dimensions. All the results will be contingent on the dates selected in the previous section, and large date ranges can cause a delay in the results returned.

Filter by Single Dimension

Users can filter the MVT reports by single dimension with multiple values. Once the user applies a filter, the result will contain those visits that has the dimension value as part of the visits (all other visits which does not include the dimension value will be excluded from the filtered report).

Filter by Multiple Dimensions

Users can also filter the MVT reports by multiple dimensions to get insight into how visitors perform against different variations for better experience. Multiple values can be selected for each of the dimensions, however the limit for selection of dimension combinations is restricted to 10.

For example, if a user selects one dimension as Category with two values, Shoes & Cloths, and Page Type as other dimension with two values, Item Page & Cart Page. Then the user can only filter the results by one of the following 4 possible combinations:

  • Item Page – Shoes

  • Cart Page – Shoes

  • Item Page – Clothes

  • Cart Page - Clothes

The data gets filtered for the visits which have Item Page as part of the Page Type and Shoes as part of the Category so that the MVT report shows visits that correspond to the selected dimension values.

Combinations of dimensions

Currently, multi dimension filtering supports the following dimension combinations:

  • Category - Page Type

  • Category – Placement

  • Channel – Page Type

  • Channel – Segment

  • Page Type – Segment

  • Placement – Segment

Uses cases for multi-dimension filters

Uses cases for multi-dimension filters can include:

  • Filtering to the right visits that experience the test variants or control. For example, if the visit contains both dimensions as a part of the same view, then the visit will be considered in the report when filtering by those two dimensions. If both dimensions are individual part of the visit but of different views, then the visit will not be considered in the report.

    • For example, When a user selects one dimension as Page Type: Item Page and other dimension as Category: Shoes, then all the visits where the user visited shoes on Item page will be a part of the reporting. Whereas the visit where the user viewed Item Page, but not shoes (or) viewed Shoes but not on the same Item Page but on other page types. Then that visit will not be a part of the report when filtered by both the dimensions (Page Type and Category).

  • Filtering by Segments or other attributes to determine how different segments perform. For example, the test may show the winners are different for different segments.

 

Adding Single or Multi-Dimension Filters

To add a dimension filter:

  1. Click Add dimension filter.

  1. Select the desired dimension type from the drop-down menu.

Graphical user interface, text, application Description automatically generated

  1. Select a value for the selected dimension.

  1. (Optional) For multi-dimension filtering, select the additional dimension you want and any options desired.

    Note: Based on the dimension you selected, the other dimensions are made available as second filter. For example, if you select Page Type as the first filter, then you will see the other dimensions (Category, Channel, and Segments) as part of the second filter based upon the supported dimension combinations.

  2. Click Confirm.

Note: To return to the default results, select Overall Performance from the drop-down menu.

Dimensions

The following dimensions can be used to filter the test reports:

  • Browser: Allows analysis by the browser type used by customers to access the site. Choices include “Any,” which will update the chart and Daily Data with a line for each browser type. Single choice only.

  • Category Tier 1: Allows analysis of data for visitors based on their exposure to at least one page within a top-level category. Multiple choices are allowed to compare categories.

  • Category Tier 1 and 2: Allows analysis of data for visitors based on their exposure to at least one page within a top-level category and secondary category. Multiple choices are allowed for comparison.

  • Category Tier 1, 2 and 3: Allows analysis of data for visitors based on their exposure to at least one page within a top-level category as well as secondary and tertiary category. Multiple choices are allowed for comparison.

  • Channel: Allows analysis of data for customers based on the Channel (Email, Tablet, Desktop, etc.) used to access the site. Multiple choices are allowed for comparison.

  • Landing Page: Allows analysis based on the page the user landed on upon entry to the site. Multiple choices are allowed for comparison.

  • Page Type: Allows analysis of data presented for customers that viewed a specific Page Type. Multiple choices are allowed for comparison.

  • Placement: Allows analysis of data for customers that viewed a specific placement. Single choice only.

  • Platform: Allows analysis based on the OS used by the user. Single choice only.

  • Referrer: Allows analysis based on the referring site. No choices available.

  • Segment: Allows analysis based on segment. Requires manual entry of segment names.

Note: Contact your account manager or Algonomy Support team if there any questions or concerns regarding the data presented in these reports or for assistance on interpreting the results.

Outliers

The next item that can be configured to customize the report output is the manner of including Outliers. As defined above, an outlier is an order that exceeds a threshold that is based on a statistical analysis that includes roll up sales for the previous thirty days from which we look at the mean, and then three standard deviations from the mean to determine the outlier threshold.

For example, if the Outlier threshold is $1000, and the order value is $1500, the order will be included in the reporting (it will count as a conversion), but the value will be adjusted to $1000.

By default, all outlier orders are included in the test results, but their values are truncated at the threshold.

The option allows the user to choose to include the outliers in their entirety. To do so, click the checkbox.

Unqualified Visits

Algonomy differs from other testing tools in that we assign a user to either the control or the test treatment as soon as they land on the site, while others do not assign users to groups until they reach the page where the testing is occurring. Thus, if the user does not actually reach the test page, we then filter them out of the test results. If the user eventually does reach the test page, that Visit will be marked as qualified. This is evaluated on a Visit-to-Visit basis.

By default, unqualified Visits, those Visits of users who were assigned to a group, but not exposed to the actual test, are not included in the test results reported. To include them (uncommon), check the box.

Flip Floppers

By default, flip floppers, those users who are exposed to more than one treatment of the test, are not included in the report.

Summary Section

In the next section of the page there is a summary for the requested date range for both the control group and the test treatments. The summary figures shown reflect site wide metrics and while they will reflect changes made to the date range, Outliers and Unqualified Users, they will not reflect selected dimensions.

The column headers that represent statistics that can be viewed in the charts, have an arrow beside them. Clicking on the arrow will change add a chart to reflect that statistic.

Columns

The column headers indicates data range for the following measurements:

  • Treatment: For each row, this column indicates which treatment is reflected in the summary

  • Traffic Percentage: For each treatment, indicates the percentage of the traffic that the treatment received.

  • Revenue Per Visit: Indicates the revenue per visit for each treatment ($Sales/#Visits).

  • Conversion Rate: Indicates the conversion rate for each treatment (#Visits-with-Orders/#Visits).

  • Average Order Value: Indicates the average order value for each treatment ($Sales/#Orders).

  • Visits: Indicates the total number of visits that are a part of the test for each treatment. As defined above, a visit is any time spent on the site without significant (30 minute) gap in activity.

  • Orders: Indicates the total number of orders that are a part of the test for each treatment.

  • Converted Visits: Indicates the total number of visits that resulted in an order.

  • Bounce Visits: Indicates the total number of visits that end without the user continuing past the first page landed on.

  • Bounce Rate: Indicates the percentage of visits that end without the user continuing past the first page landed on.

  • Click Through Rate: Indicates the number of users who click on a Algonomy placement, as percentage of views (#clicks/#views).

  • Average Views: Indicates the average number of views per treatment

  • Average Visit Time: Indicates the average number of minutes a user spends per visit.

  • Outlier Visits: Indicates the number of visits where the overall sales across the orders in a visit is an outlier.

  • Revenue: Indicates the total value of sales attributed to each treatment.

  • Units: Indicates the total quantity of units sold per treatment.

  • Units Per Order: Indicates the average number of units per order per treatment (#Units/#Orders).

Below each of these measurements, there is a Lift value, which is the difference between the treatment and the control. There is also a Confidence percentage under some of the measurements, indicating the level of confidence that the lift is an accurate representation of the test. Confidence increases as the number of users/visits increases.

Charts

Below the Summary section there are two charts. These charts can be changed to reflect any of the statistics in the drop-down menu at the upper left of the chart, including: Average Order Value, Click Through Rate, Conversion Rate, Orders, Revenue Per Visit, Revenue and Visits.

Beside the drop-down menu is a small icon of a graph. This indicates that the graph itself can be changed from the default line chart to a column chart.

Chart, line chart Description automatically generated

If the statistic type is one for which Lift can be calculated, you are also presented with a radio button to choose between using the data or the lift as the basis of the graph.

In the top right-hand corner of each graph are buttons for maximizing or deleting the graphs. At the bottom right of each graph is a button that says “Image.” Clicking on this button will give you a static image of the file that can be saved as a picture, downloaded, or copied and pasted into documents.

Chart, line chart Description automatically generated

Daily Data

The next section of the page is the Daily Data section. This is expandable and collapsible, using the small blue arrow to the left of the words Daily Data and is also downloadable if you click the Daily Data link to the right after the date range. The date range will update to reflect any changes made in the section above to limit the results to a range within the overall test.

For each date in the date range, the table shows a line for the Control and a line for the Treatment(s).

The first two columns indicate the date and whether the line is for the Control or the Treatment. The third column identifies any dimension defined in the Attributes section above.

A screenshot of a computer Description automatically generated

The remaining columns are filled with data, as follows:

Visits (#)

Sales Per Visit ($)

Average Order Value ($)

Sales ($)

Conversion Rate (%)

Orders (#)

Clicks (#)

Views (#)

Click Through Rate (%)

Outliers (#)

Table Description automatically generated

MVT Visit Data for Conversion Metrics

The next section of the page is the MVT Visit Data for Conversion Metrics section. You can download the data by clicking the Visit Data link to the right of the date range. The date range will update to reflect any changes made in the section above, limiting the results to a specific range within the overall test.

For each date in the date range, the downloaded CSV file shows the test name, treatment ID, treatment name, eligibility, and data for the following attributes:

  • Number of Orders

  • Sales Amount

  • Number of Items

  • Outliers

Note: This report is currently available only for non-concurrent tests.

Audit Log

The very last section of the page shows any alterations that have been made to the testing parameters after the test has started.