Test, Learn, Optimize
Your resource to smarter customer experiences
Author
How often is A/B testing reduced to the following question: ‘what sample size do I need to reach statistical significance for my AB Test?’ On the face of it, this question sounds reasonable. However, unless you know why you want…
I was talking with Kelly Wortham during her excellent AB Testing webinar series. During the conversation, one of the attendees asked if they just wanted to pick between A and B, did they really need to run standard significance tests…
Multi-armed bandits, Bayesian statistics, machine learning, AI, predictive targeting blah blah blah. So many technical terms, morphing into buzz words, that it gets confusing to understand what is going on when using these methods for digital optimization. Hopefully this post…
In this post we are going to introduce an optimization approach from artificial intelligence: Reinforcement Learning (RL). Hopefully we will convince you that it is both a powerful conceptual framework to organize how to think about digital optimization, as well…
The key idea behind Conductrics is that marketing optimization is really a reinforcement learning problem, a class of machine learning problem, rather than an AB testing problem. Framing optimization as a reinforcement learning problem allowed us to provide, from the very beginning, not just AB and…
In our last post, we introduced the idea of shrinkage. In this post we are going to extend that idea to improve our results when we segment our data by customer. Often what we really want is to discover what digital…
As some of you may have noticed, there are often little skirmishes that occasionally break out in digital testing and optimization. There are the AB test vs multi-armed bandits debate (both are good, depending on task), standard vs multivariate testing…
UPDATED: 5/20/2021 A version of this post was originally published over at Conversion XL For all of the talk about how awesome (and big, don’t forget big) Big data is, one of the favorite tools in the conversion optimization toolkit,…
Personalization, one to one, predictive targeting, whatever you call it. Serving the optimal digital experience for each customer is often touted as the pinnacle of digital marketing efficacy. But if predictive targeting is so great, why isn’t everyone doing it…
Normally, when we talk about AB Tests (standard or Bandit style), we tend to focus on things like the different test options, the reporting, the significance levels, etc. However, once we start implementing tests, especially at scale, it becomes clear…