Artificial Intelligence powered A/B Testing // Jente De Ridder
How to create an A/B test that tests a couple of thousand variations? If this would be tested manually you’ll be working on it for months.
Jente gave us a very interesting case about AI-powered A/B Testing. At a client, Humix has used a tool called Sentient.
Sentient Ascend is an A/B testing tool that runs on an Artificial Intelligence (A.I) algorithm.
- Sentient Ascend promises to increase the number of variants that you can test simultaneously while maintaining your normal amount of traffic. -> Test more in less time.
- Thanks to the A.I. algorithm, winning variants are found much faster, without losing statistical significance.
We ran a test across 3 different pages, 8 changes were applied. This resulted in 256 potential designs of the sales funnel.
Results:
- The winning variant resulted in a sales uplift of 29,2%, compared to the control version.
- Based on the Bayesian statistics reported by Sentient the relative sales uplift, if this variant gets implemented, will be somewhere between 8,8 and 43,4% (with a confidence level of 95%).
- On average, there was an uplift of 17% during the test period (across all variants).
Key Learnings:
- Sentient Ascend helped us identify a combination of design changes that we would not likely have been arrived at through traditional A/B testing.
- The learnings from this experiment provide substantial insights that can be applied to other future experiments.
- Conversion optimization maturity within the organization is needed to come up with test hypothesizes and interpret the results.
- A certain volume (+3.000 monthly conversions) is needed to perform this kind of multivariate testing.
- A conversion can be as you set it. For example a download, a click, …