SparkPost recently released a new API feature for A/B testing transactional email and notifications. (Dig into the A/B Testing API documentation for usage details.)

A/B testing is a common practice for determining if a variation on certain aspects of an email campaignâ€”such as different subject line, call to action, images, and so onâ€”will positively affect engagement rates.

### Simple A/B Testing vs. Bayesian A/B Testing

Table of Contents

As we noted in the announcement blog post, SparkPost’s implementation of A/B testing uses a statistical algorithm for picking the winning variant of the message. Specifically, we built it using a Bayesian decision model. But what does that mean, exactly?

First, itâ€™s in contrast to the most common approach used by email marketers. Most A/B testing relies on a simplistic, â€œfrequentistâ€� method. In this method, one must simply compare the engagement rates of the emails sent on the variants, and the variant with the higher engagement rate is the winner of the test.

One of the main reasons to not use a frequentist approach is the problem known as p-hacking while trying to determine a winner. Another problem of A/B testing is determining an appropriate window of time to run a test for. The volume of data sent can have a significant impact on how meaningful test results are. On the other hand, sending too much email to a â€œtestâ€� variant with bad engagement is something to be avoided. This is where Bayesian A/B testing comes into play.

### Our Bayesian Decision Model

Given two email campaigns A and B with A_{d} and B_{d} as the count of delivered emails to each, and A_{e} and B_{e} as the count of â€œengagementsâ€� for each campaign the probability that campaign B will lead to better engagement rates than A is given by this NodeJS function (using the

mathfn

Â library).

const { logBeta } = require('mathfn'); const probabilityBbeatsA = (A_d, A_e, B_d, B_e) => { const [ alphaA, alphaB ] = [ A_e + 1, A_d - A_e + 1]; const [ betaA, betaB ] = [ B_e + 1, B_d - B_e + 1]; let total = 0; for (let j = 0; j < betaA; ++j) { total += Math.exp(logBeta(alphaA + j, alphaB + betaB) - Math.log(betaB + j) - logBeta(1 + j, betaB) - logBeta(alphaA, alphaB)); } return total; };

If youâ€™d like to learn more about of the logic used in this code sample, you can read an additionalÂ detailed mathematical explanation of of the statistical model we use.

### Example Results

% Engaged A | # Delivered A | % Engaged B | # Delivered B | p(B>A) |

10 | 10,000 | 11 | 100 | 68.94 |

25 | 100,000 | 26 | 1,000 | 77.24 |

10 | 10,000 | 11 | 1,000 | 84.98 |

10 | 100,000 | 11 | 1,000 | 86.15 |

20 | 100,000 | 25 | 100 | 90.36 |

10 | 1,000 | 15 | 100 | 94.58 |

10 | 10,000 | 15 | 100 | 95.58 |

25 | 100,000 | 26 | 10,000 | 98.60 |

10 | 1,000 | 20 | 100 | 99.82 |

10 | 100,000 | 11 | 10,000 | 99.92 |

10 | 100,000 | 15 | 1,000 | 99.99 |

20 | 100,000 | 25 | 1,000 | 99.99 |

As shown in the above table, the Bayesian method of testing can give more information than a pure frequentist approach. We now have confidence that the new variant will be better in the long term. Using a 95% confidence is industry standard in several applications, although anywhere between 90â€“99% may desired.

Using a Bayesian model for testing, we can also stop an A/B test as soon as we find a variant which beats the default template by the desired confidence threshold.

Why did we choose this particular Bayesian model? It is widely used across the industry for Bayesian A/B testing. Follow up can be found on these references and more!

- http://www.evanmiller.org/bayesian-ab-testing.html
- https://www.chrisstucchio.com/blog/2014/bayesian_ab_decision_rule.html
- https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf (the modelÂ model used by VWO)
- https://www.peakconversion.com/2012/02/ab-split-test-graphical-calculator/
- http://varianceexplained.org/r/bayesian_ab_baseball/
- https://yanirseroussi.com/2016/06/19/making-bayesian-ab-testing-more-accessible/
- Our model also matches the results from https://abtestguide.com/bayesian/

â€”Jason Sorensen

Lead Data Scientist

The post The Bayesian Logic in SparkPostâ€™s A/B Testing API appeared first on SparkPost.

Source: Emails

{$excerpt:n}