Kinds Of Split Testing Software
Split testing works. Use it well, and you will improve your conversion rates, and, thus, your profits. And, if you are advertising, or have significant product costs, raising your conversion rate just a little can raise your profits a lot. But there are many different kinds of split testing software out there.
I can’t discuss every feature of every testing tool out there, but I can give you a major overview of the different kinds of split testing software.
The first main division is between simple split testing (single variable) and multivariate testing.
Single variable split testing is very simple. You just try stuff and measure the results. And all you need is a tool that rotates and keeps score.
Paul Hancox has a simple split testing tool. And Google has one that’s free, though it takes a bit to set up all your accounts to use Google’s tool. And the reporting is not real-time.
But simple split testing can be discouraging. When you test a new idea against your control version, sometimes the control version wins. In fact, it often wins. You can get a string of disappointing tests all in a row. What would it be like if you could test 10 things at once? Well, you’d be bound to find at least one or two ideas that led to improvement. And what if you could test all those ideas simultaneously with the same amount of traffic it takes to test one idea with simple split testing? Well, it sounds like magic, but that’s exactly what multivariate testing can do for you.
Among multiple variable software solutions, there are hosted plans and self-install plans. If you want to install software on your server (which can be faster and cheaper) you will have more control over your data. And you can usually find someone to install it for you (often the vendor provides this service for a nominal fee). If you want to go with a hosted plan expect to pay more.
The Split Test Accelerator is a good choice if you want to install the software on your server. Kaizentrack is another optoin. If you want a hosted solution, you can look at Sitespect and Vertster.
Another 3-way distinction among multivariate split testing tools is between Taguchi-based systems, Full factorial systems, and Random independent rotation of factors. With a Taguchi system, you run traffic to only a small, carefully-selected subset of the possible combinations of your factor levels. This allows you to cycle through every combination every 12, 18, or 36 pages, whichever the case may be for the test design you’re using. This allows you to find the best combination very quickly. Some care does have to be taken to avoid interaction effects among the factors. Taguchi-based tools include the Split Test Accelerator and Vertster.
When you run full factorial tests, as you would with Google’s Website Optimizer, you create a few factors, and a few options for each factor, and the software cycles through each combination. So, if you have 5 factors with 2 options each, you would cycle through 32 different pages. FFT can be used like a Taguchi test for small test designs. Or, if you run a lot more traffic to it, you can deal even more effectively with interaction effects by just letting all the interactions (good and bad) play out. Note that you can’t run as many combinations with full factorial tools as you can with a Taguchi too.
Some multivariate tools randomly serve the options for each factor independently from what is served for the other factors. This can lead to more flexible test design, but also to more interaction effects and other statistical noise, leading to longer tests.
There are other distinctions and features worth noting. And each solution has a different mix of pros and cons. But this should get your oriented a bit before you make your decision.
Dominic DeLong has used multivariate testing software for years to improve landing pages.
