A/B testing (a.k.a. split testing) is now a well-know, popular practice in online marketing. The idea is simple: create multiple versions of a website and see which version results in a higher conversion rate. It is scientific, its results are quantifiable, and its ROI is thus measurable. Like most other web software businesses, we have practiced A/B testing consistently, and our experience has revealed to us some costs that were not so apparent to us before we started doing A/B testing:

1. A/B testing requires sacrificing some conversions as part of the experiment. This point is most obvious, as we have in the past tested multiple variations that not only performed worse than the control but also inadvertently led to a sacrificing of a number of sign-ups that we would otherwise get if we had just stuck to the control. Test multiple worse-performing variations and, over time, the sacrificed conversions would add up to a lot.

2. A/B testing measures only quantity, not quality. A/B testing measures conversions based on the occurrence of events, a binary value. Just because one version leads to more sign-ups or even sales does not mean that the other version has less value. The seemingly-successful variation may contain a change that specifically attracts the wrong kind of users or customers in larger numbers. We ourselves noticed that, by reducing the barriers to sign-up (e.g. by not requiring ownership of a domain name), we increased the number of sign-ups but saw a decrease in the average quality of customers.

3. A/B testing can be slow and expensive. The cost of A/B testing far exceeds the cost of A/B testing software (we have used Visual Website Optimizer in the past, which we unreservedly recommend despite its high price point), because there is a cost involved in 1) deciding on which elements to test and what changes to make and 2) actually creating those alternate versions, especially if they are site-wide overhauls. Also, consider the opportunity cost of creating several worse-performing versions. Depending on your traffic level, it might also take a while to obtain statistically-significant results, prolonging the cycle of optimization.

4. Things change. Just because version A performed better over version B one year ago does not mean that it will still perform better now. We ourselves have seen different outcomes when we conducted the same test twice during different times, the latter of which coincided with a certain marketing campaign that we were running.

The point of this post is to remind us not to forget about qualitative testing, i.e. usability testing. In fact, during the earlier stages of a web-based business, it may be more prudent to rely primarily on qualitative testing, which is far more actionable than A/B testing, in that A/B testing merely tells you which is better given two versions, but qualitative testing tells us precisely how to make it better given just one version. The way we do it with Zuupy CrowdDeals is simple: 1) get ideas from users, then 2) test and confirm using A/B testing.