Tyranny of numbers — checking for statistical significance
Sunday, March 30th, 2008Its become fashionable to run AB and multivariate tests for just about anything on the internet. Of course, direct marketers have been practicing tests vs. control groups for a long time before the rise of the world wide web. It would be a shame to spend so much time testing and yet not know when you’ve found something meaningful. Knowing when and how to check for statistical significant belongs in every marketer’s toolbox (or at least, you better get know a friendly web analyst who will run the numbers for you). Here are two of my all-time favorite posts on this subject:
RKG on finding statistical significance in two Adwords tests Interesting commentary on the value (or lack-of-value) in copy testing in PPC
Avinash Kaushik on separating signal from noise with statistical significance