Google’s Multi-Armed Bandit Approach to A/B Experimentation

By: Adam Dennis


Most retail owners targeting online sales via business websites want the option of testing out different versions of a web page to see which one has a higher conversion rate.  Most dealers in the automotive industry have heard of, or currently utilize, split A/B testing.  In split A/B testing, traffic is evenly divided between two variations of the same web page until a traffic cap is reached, and a statistically significant victor is chosen.  A slightly different approach, known as multi-armed bandit testing, adjusts traffic as the test progresses.  This method works extremely well with responsive web design, to ensure that dealerships are truly serving consumers website pages with the best user interface for them, on every device.  Google has recently been integrating this type of testing within its Analytics API, along with new analytics features and report data.

Classical A/B Testing and Multi-Armed Bandit Testing

To illustrate the difference between split A/B testing and the multi-armed bandit approach, we can imagine that a dealership has opened up showroom space for new vehicles.  The owner wants to try out two different brands of vehicles to see which ones are preferred.  He performs an experiment on the first 2,000 customers that walk in.  The sales rep directs half of the customers into one area of the room with vehicle A, and the other half into another area of the showroom with vehicle B.  Then, only after all 2,000 customers have come through does he decide which machines to keep.  This is, in a nutshell, a “classical” A/B split test.

If the owner used a multi-armed bandit approach, however, he would tell the sales rep to analyze his data twice a day, checking on which vehicle was preferred, and gradually adjust incoming traffic to favor the more popular vehicle.  By the end of the experiment, most of the traffic would be flowing to the higher performing vehicle, thus saving on conversions that might otherwise be lost.  Hypothetically, due to the dynamic traffic adjustments, the sales rep could have directed 600 people into the area with the lower conversion rate, while directing 1,400 into the room with the higher conversion rate.

Clearly this example has become slightly muddled by internet marketing terminology, but this example should still serve to clarify the difference between the two approaches.  The logic appeals to many marketers who wish to maintain higher conversion rates during testing.  Google has several articles on the subject of multi-armed bandit testing, and claims that it is just as statistically valid, more efficient, faster, cheaper, and more appropriate for website testing, since it optimizes throughout the testing process.

Google’s Content Experiments API

It is no surprise that Google now implements multi-armed bandit testing in Google Analytics, which has become a fully functional A/B experimentation platform.  Multiple variations can be tested simultaneously.  Google adjusts traffic flow twice a day based on the user’s testing criteria, until the traffic cap is reached, with a three-month maximum timeframe.  Google Analytics’ Content Experiments API rolled out earlier this year, and is designed to facilitate analytics of the website content and content experimentation.  In-page tracking, in-site search tracking, site speed analysis, and AdSense data tracking are a few of the features offered by the Content Experiments API.

With multi-armed bandit testing integrated into the Analytics API, developers and marketers now have a new toolbox to test multiple web page variations and view the results as they occur in their Analytics reports.  This new, highly programmable, Analytics interface provides new A/B experimentation solutions that are flexible, responsive, and customizable.  As time goes on, the API will certainly continue to evolve and offer new functions for website owners.

Further Reading

Posted in: