System Overview

Understanding A/B Split Run Testing

An A/B split run is a standard method of testing the response rates to two different options, A or B. Testing messages in this way provides an inexpensive method for measuring real-world behavior to different subject lines, different offers, or different graphics. Measuring small performance improvements in a sample can lead to significant improvements when reaching the entire audience. You also can use A/B testing to resolve subjective differences of opinion on which copy to use, or what method of incentive or promotion is likely to generate a better response.

This system allows you split and test an entire list, or test a sample of the total list. It also allows you test both subject line and message content, automatically sending the test with the subject options paired with each content option; AA, AB, BA, BB. Note the A/B subjects entered here will overwrite any subject entered in the message or template(s). To test message content, you must save each option as its own template.

See also:

How do I compare the effectiveness of different subject lines?

How do I compare the effectiveness of different message content?

Keys to a Successful A/B Test

When testing message content, it is recommended to change only one element of the message for a particular test. For example, change just the offer, the link name, or the graphic between option A and option B.

A/B testing is most useful when you have a large enough sample to produce a statistically valid result. A statistically valid result can be relied upon, with some degree of certainty, to predict the results when mailing to the full list.

If you’re working with lists of 10,000 to 20,000 or more, you will be able to test many valid samples easily. When testing small samples of one or several hundred from a smaller list the margin of error may be higher, but you may still learn something quite valuable.

Figuring Out the Math

Determining a statistically significant sample size and confidence intervals can be mathematically complex, so if you don’t have a good statistician on staff, try these sites to learn more:

http://www.surveysystem.com/sscalc.htm

http://www.isixsigma.com/offsite.asp?A=Fr&Url=http://www.surveyguy.com/SGcalc.htm

http://www.isixsigma.com/library/content/c040607a.asp

http://www.dmreview.com/article_sub.cfm?articleId=7230

To keep things simple, we've put together this table of sample sizes with margin of error using the standard convention of a 95% confidence level (95% of the time, your results will be plus or minus the margin of error).