Choosing Which Tests to Run

Overview

This document will guide you with which MVT tests help optimize your site and test your hypotheses. Below are some examples of tests we have found to be useful.

Setting up tests should be done with a clear idea of what is being tested and a statistical understanding of significance.

You can create one or more tests. Each test will at a minimum have one treatment and one control. You designate a desired percentage of traffic for each treatment. Customers who are placed in your test are assigned to the control or one of the treatments according to these desired percentages.

Primary Tests

These are some of the most common and important MVT tests. They may be useful for implementation, site re-design, or just moving-the-needle. Please refer to your Algonomy team for best practices and let them know your testing goals.

Strategies

  • Preferred strategies based on best practices, current results on site, etc. vs no strategies preferred

  • Different combinations of preferred strategies

Placement positioning/Styles

  • Vertical vs. Horizontal

  • Location of placement on page

  • Center below fold vs. gutter above fold

  • Carousel vs. No Carousel

  • Number of products shown (4 - 5 is typically the best number)

Number of Placements

  • Incremental placements on pages with existing placements.

  • New placements on pages without existing placements.

  • New placements on interstitials and Quick Views

SEO Placements on Item Page

  • The goal of these placements is to decrease bounce rates and increase engagement.

Secondary Tests

These are typical needle-movers and may identify lift opportunities. These MVTs will help you better understand user behavior.

Merchandising Rules in regard to sales items

  • This will test customers' sensitivity to price by seeing how they react when confronted with placements that only show sale price vs. placements that do not show sale price.

Layout of Placements

  • Show/hide Add to cart button

    • Call to actions tend to drive more clicks and more purchases

  • Show/hide price, sales price, strikethrough price, etc.

    • Sales price listing, price show or no show are both big things to test and can drive significant lift – it depends on the site in question.

  • Show/hide ratings

  • Change Coloring

    • Sales price

    • Add to cart buttons

    • Placement Messaging

  • Change Strategy Messaging

  • Placement positioning

  • Image Size

    • Best practices recommend image sizes 120x120 or 150x150, according to the esthetics of your layout.

Experimental Tests

These are examples of novel tests for which you may have special application. Reach out to the Algonomy Client Excellence team when pioneering new applications.

Quick View vs. No Quick View

  • This test compares the use of recommendations that use the Quick View modal popup. There are three options to test: Recommendations that include Quick View, Recommendations without Quick View, and Quick View with Recommendations.

Lifestyles images vs. Product Images

  • This test will compare images that show a product being used (such as a model wearing shoes) to just the image of a product.

Helpful Tips

  • A well thought out test plan with strong hypothesis is key to a meaningful test. Be sure to take into consideration the use case that you are testing for and that it would have a material impact on your site’s key metrics.

  • Tests should be evenly split with each treatment randomly receiving an equal amount of traffic in order to best avoid any unwanted sampling size bias.

  • In order to get accurate and meaningful results, it is critical all instrumentation issues are resolved.

  • Factors to consider when running a test are traffic volume, relevance of feature to be tested, and value of the proposed test’s success metric. For example, if the click-through rate is the success metric then the test should not go longer than two weeks, as clicks are generally easy to come by and you will get a large enough sample size to reach statistical significance and minimize inconsistency.

  • Make sure you are using reasonable sample sizes. Setting aside less than 10% of traffic per treatment will prolong your test and potentially create more daily volatility than optimal. This, in turn, would slow down turnaround on roll out of the winning treatment across all traffic and capitalize on your learnings.

  • Avoid tests of site features with very low user engagement. For example, if a link has very low user engagement, then it shouldn't be tested, unless dramatically increasing visibility. Running a test that will impact a small portion of your customers is unlikely to yield a material impact.

    • Examples: "Search Not Found” Page, recommendations on Out-of-stock product and other error pages, Purchase Complete page

  • Keep test length manageable. The purpose of tests should be to learn from the experiment and leverage those findings as quickly as possible to get the most out of your site. Running tests for several months or in perpetuity minimizes your ROI on the test and limits your ability to capitalize on changes.

  • If you have any doubts on if what you are testing will provide you value, do not hesitate to reach out to your Account Manager. The Algonomy Client Excellence group or Client Support group will work with you to ensure you are getting the most possible value from your experiments. Their expertise will ensure that costs of testing will be rewarded with useful information.

  •