Google Optimizer: Multi Variant Combination Results
Note that these results reflect an Experiment that has only been running about a day, so all the info is not available yet. Most notably, the "red" and "green" bars that will show me clear cut winners and losers. For those statistcal people out there, that will be determined when the sample number is large enough to determine a statistically significant outcome.
So far, "Combination 3" is the front runner:
This combination is the "Table with bonus items and no links" discussed in previous posts, along with the "crossed out old higher price", and the orginal version of the third variant, which simply did not have the text under the price reading "limited time sale".
When I click on any of the combination links, it shows me the actual sales page (my Experiment Page) with thes Variants installed.
So far so good, I'm getting good data that is very interesting to analyze.





2 Comments:
Just a thought in passing, but I've never been totally convinced that waiting for a test result to be statistically significant is really worthwhile.
Instead of asking the question "Is the current leader really better", ask "Is it likely to be worse?".
It's true that you'll sometimes select the worse of the two options, but in these cases, it's unlikely to be MUCH worse.
On the other hand, you can halve the length of time an A:B test takes, allowing you to run twice as many experiments in the same time.
I believe that this advantage outweighs the potential downside, if you are running a never-ending set of a:b tests, though that's just an opinion, and not based on any mathematical proof.
Steve
I agree with you Steve. Although I have a lot of background and training in business statistics, I can bend the rules from time to time. You bring up a good point: wasting time if you have another experiment in the queue - even worse, think about the fact that only a portion of your traffic is getting the better converting combination of factors. The tests can definatley linger on befor estatistically significant levels are reached - I'd say if it's obvious and you've pumped thousands of visitors through the site - go for it. Mae the changes permanant.
If we were testing a process that could improve the survival rate of open heart surgery I might have a different opinion:)
Post a Comment
Links to this post:
Create a Link
<< Home