Bot Handling

Overview

A minimally viable implementation consists of recommendations on the Item page and at least one of the following: Add-to-Cart and Cart. Certain page types are more conducive to supporting lift via recommendations, as shown below. Following the guidelines on the pages that follow helps safeguard performance of Recommend on a retailer's site.

All sites are different, and what works best for one is not guaranteed to translate into gains for another. As such, treat this guidance as general best practices, a foundation for optimization from which site-specific tuning should occur. Understand the logic behind our optimization methodology and the questions to ask, so that you can adapt and apply as appropriate.

Details

Bot Requests

In general: We log all requests through our KnownRobotsFilter and tag the records as originating from a bot if it matches those values.

Within the Site Configuration for a given site, there is a "Put known robots in listen mode" configuration.

  • When this is checked:

    • When a bot self-identifies, Recommend treats the page load as though the site is in listen mode. This prevents rudimentary crawlers from spending too much time on PDPs rather than category pages, or getting stuck in loops which use up their crawler budget.

    • Underlying cost: the googlebot does not get the same user experience as a real user, which would ding the site in terms of SEO.

  • When this config box is left unchecked:

    • Recommend returns recommendations to bots

    • We do not log the views from bots - "The view log is blocked for a request if the user agent header contains a known bot string" no matter the site config setting

  • The default for this site configuration is unchecked.

FAQ

Q: What is Algonomy's position with regards to GoogleBot and other known bots?

A: GoogleBot is one of the user-agents we use in the KnownRobotsFilter. We treat all known bots like this as a user-agent.

Q: Are recommendations returned when bots call Algonomy's API?

A: When "put known robots in listen mode" is checked: no, bots see the page as if recommendations are in listen mode. When it is left unchecked, recommendations are returned to bots.

Q: Do the known bot requests get added to our dashboard analytics?

A: This is agnostic of the site configuration. Known Bot requests never get added to dashboard analytics. If the bot is recognized/marked as a bot, we don't log it.