PrintStatistical Significance
Using Analytics Appropriately
It has probably been some time since some of you have had a Statistics class, which is fine. I would like to cover a few topics here that are not so much concerned with calculation as much as creating–and following–statistically sound goals.
As you begin microtesting, you may begin anxiously watching results as they come in. You might find yourself leaving AdWords open in a small window on your monitor, or otherwise "checking in" frequently. While it certainly is exciting, that excitement and tension can lead you to choose the wrong test winner, and ultimately, the wrong proposition for your offering.
What I would like us to avoid is the all-too-common situation of choosing the winning variation based on some arbitrary goal you had in your head, be it, "First to 200 clicks" or, "First concept with 10 conversions" or, "Whichever looks the best at the end of two weeks." In a more passive form, this is actually quite common, even in the professional PPC world, where a PPC consultant will ask you A) "How long do you want to run the ads?" or B) "How much you want to spend?"
If you happen to keep a spray bottle of water on your desk for your ficus, feel free to use it on whoever asks you this question.
Your correct answer would be C: "Until we have statistically significant results," followed by, "Call me when we spend $X."
I can share from experience that PPC results can take odd and inexplicable "runs" in volume and preference... a week of one-ad-click days followed by a ten-click day, for example. A keyword running as hot as lava for three days and then returning to norms. While it could be tied to social shares, PR, the day of the week, or other factors, many of which you can see in Analytics, sometimes, it is a truly random occurrence.
To prevent our human emotion/anxiety/excitement from getting in the way, we can simply drop a handful of our click statistics into one of a few sites devoted to finding PPC statistical significance and they will tell us if we have reached significance, and if so, what level... or, if we have not, how many more clicks we will require. Some analytics packages have the statistical significance tool built right in.
So, if you've always been dying to apply some of those learnings from Stat into your professional life, using it for the validation of microtesting results is a perfect place to do so.
This may seem like a rather straightforward consideration, which makes sense because it is. But it is also a consideration that is very commonly overlooked.
