Cleaning up a Variant Test on the Full Site Experiment - Plus Tips & a New Experiment

The Google Website Optimizer test I put together that basically created a whole new navigation system (see HERE) was daunting to say the least. I wasn't happy with the way the test variant looked, it was too crammed and busy - just messy looking. The goal was to cut down on the options of places to go and combine the sales page into the landing page. I have since cleaned it up a bit, added a new eCover and removed a lot of the other graphics. This taints the test a bit but it is slow going anyway given the small niche the book is geared toward.

One of the most important things when doing a large experiment like this is organizing your pages and code. I basically made copies of the pages involved, and had the old and the new open in my development window at the same time. Since I was redesigning the whole page, I placed the test section tags immediately within the body tags - this made it easy to spot and cut down on possible problems.

The Google Optimizer preview window (and the live site) played some games with the styles - although I was doing a whole body cut and paste, some styles showed differently in the test variant then the original - even though the code was the same for that given part (i.e. some text that was unrelated to the changes I made).

I'm still not sure why this happened. I am a bit guilty of sloppy coding - I'm not a programmer or developer - I just sort of wing that with the help of Dreamweaver.

I have my next experiment planned already and it is something totally different. I'm about to launch a new product and what I'm going to do from day one is test 3 different prices. I'll basically set up three products in Clickbank, which will all be the same except for the price. I'll change the text and the link info in each variant to reflect the different prices. There should be a lot of traffic for this campaign so I'll see results right away. I'll have to look at returns also since the price people pay will affect their decision to return a product.

I'm looking forward to that one.

Optimizer Reports: The Red & The Green

Be sure to read my last post on the traffic that is used for Goggle Website Optimizer as well as the method used - clearing up some myths floating around out there: here.

So I'm getting some "Color" in my reports. What does this mean? It means that Google Optimizer is getting closer to determining ultimate winners and losers. For now it is only on the individual test sections report, lets have a look:



note: click on the images to enlarge


"old sale picture" and "Limited time Sale" both have some color. I believe the threshold is 75% for a winning Green variant (ie it has a 75% chance of beating the others - see column headings) and 25% for the Red losers(25% or less chance of beating the others.) When everything is at 100%, this experiment will be over and I will change my site around permanently, based on the results.

This is again based on statistical "confidence" metrics...at some point there will be enough samples taken, and conversions from those samples, to say "ok your done - this variant will beat the others most of the time".

To review what these two were: the "old sale picture" used an higher price crossed out above the actual price. the "limited time sale" just added text to that effect under the price.

The combinations haven't changed a lot from this post.

As far as the other other larger test goes, I'm still waiting on a few more conversions to post some detail.

I also have some "lessons learned" form that experiment and from doing a full-site variant test with Google Optimizer...problem is the site is old and really want to re-vamp the full front page so that would mean a major experiment overhaul...oh well we'll see.

Website Optimizer: What Traffic Does it Track? What Kind of Test is it?

I've seen these questions come up on various boards and such:

What traffic does Google Optimizer track? Does it only track AdWords?

Google Optimizer tracks and does the multivariant testing on all incoming traffic, regardless of source. You need an AdWords account because all the tracking and G analytics horsepower is housed within AdWords - Google Optimizer leverages off of this...why reinvent the wheel?


What Type of Testing Does it Do? (Taguchi, A/B, etc)

I've seen a lot of misinformation on this topic. Particularly around Taguchi - it seems that a lot of folks in the Internet Marketing world are a bit confused here.

Here is quick explanation:

Factorial Experiments allow you to test multiple 'factors' with multiple 'choices'. For example, a factor could be an image on a landing page and a choice would be various images being swapped.

Now, outside of automated efficient electronic type of testing, such as testing physical operations like machines or chemicals, the tests become very expensive with the more factors and choices you add. Remember permutations? The possible number of combinations expands greatly with each new factor and choice of factor.

So, we have "Fractional Factorials" which in a nut shell limit the amount of total combinations via statistics needed to get a meaningful result. Taguchi is a form of fractional factorial experiments.

So back to the web - It costs nothing more to do 10 factors with 20 choices then it does to do 2 factors with 2 choices....so there is no need for a fractional method like Taguchi.

Google Optimizer uses a full factorial method, where the factors are the "test sections" and the choices are the "variants" on a given experiment page.


Hope this helps,

Meegwell

The Combinations of Variants Tell a Different Story

The train I caught was absolute hell and I would like to say I don't know why Amtrak is still in business, but I do know why - there is no choice and they are subsidized to stay alive....but that's another story.

I'm settled in my hotel now and I see there were at least one if not more conversions since my earlier post so the results of the combinations won't match those of the individual pages posted earlier...

I seem to have a leader breaking out. The exciting thing about this is I can look back to just a week or so ago and say "damn...it looks like I just started making x amount more per week, just because I tried different combos of sales page crap"

How else would I have figured out that potential customers became customers because they liked the bonus setup, but they only like it when it was presented with a crossed out old price and NOT with some text about a limited sale????

Ahhh...getting into the mind of the web shopper.

So here is where we are so far as far as combos goes:



As you can see, combo 3 is leading the pack with 4/84...a conversion rate of 4.7% !!!

I'm so ready to "release the hounds" and let this combo go and sit back and enjoy a 4.7% conversion rate....but I cant, because there is more data to be collected.

But let me tell you, the history of this site's conversion rate is more like less than 1%.





meegwell

More Google Optimzer Sales Page Results

The full site test at prowebvalue is still collecting data - I have yet to report on the issues I had there but will shortly. In the meantime, Im seeing some more clear results from the original sales page test. Here is the indivual page results:



As you can see, 2 of the 3 changes are beating the original on a stand alone basis. The top row shows an estimated conversion rate of .85 to 2.85. This is figured by factoring in the stated margin of error (see the +/- 1%?). The second one is similar, and the third has the original beating the new, with a similar estimated conversion.

I am going to make a request to G to clearly show an "improvement" percentage for this report.

The combo results tell a different story but I've got to catch a train and will report those tonight.

Landing Page Experiment: A Full Site Test!

My first test is still running fine and I'll get back to that one when more data has come in. Unlike that first test, which was simply swapping out some variations of different sales page elements, my second test was much more involved!

I just finished launching the second test, and quite frankly I'm a little nervous given all the changes that were involved. Here's the scenario:

This test became so involved I'm going to post the links to the new and old so you can get a feel for yourselves.

I wanted to change a the landing page of an old site of mine. The site sells a book that teaches professional valuation techniques, applied to websites. It was one of my first projects and is a small technical niche - kind of a "near the heart" thing and not so much for the sales.

The plan was to incorporate the sales page into the landing page, and eliminate some of the paths that needed to be taken before a visitor is presented with the option to buy the book. The problem is those paths were many, and in order to keep a good amount of information (like table of contents samples), many pages had to be duplicated and re-designed since the sales page was now the index (those other pages had links to the old sales page location). Can you picture this mess?

So I did it. I created the new landing pages, cut out a bunch of the duplicate links and paths, and just left the table of cocntents samples. For each sampl, I built a new page that sent the visitor back to the index, so they dont end up at what was the old sales page.

This basically creates a whole new site - the content being the same, but the whole navigation system is different...Im sure I screwed something up I'll catch tomorrow.

Problems and tips: these were o' plenty - but it's a bit late now so Im going to post details tomorrow. For now, I've setup the 'old' and the 'new' in a separate directory and kept all the linking for each as if Google Website Optimizer brought you to each page.

The old, original index was this: Original



The new index, and site navigatio and sample pages is here: New Variant




note: In order to put this "original" out there, I had to remove all Google Optimizer coding.

Like I said, I'll dive into the details of actually creating this test tomorrow...

Google Optimizer: Multi Variant Combination Results

Note that these results reflect an Experiment that has only been running about a day, so all the info is not available yet. Most notably, the "red" and "green" bars that will show me clear cut winners and losers. For those statistcal people out there, that will be determined when the sample number is large enough to determine a statistically significant outcome.

So far, "Combination 3" is the front runner:



This combination is the "Table with bonus items and no links" discussed in previous posts, along with the "crossed out old higher price", and the orginal version of the third variant, which simply did not have the text under the price reading "limited time sale".

When I click on any of the combination links, it shows me the actual sales page (my Experiment Page) with thes Variants installed.

So far so good, I'm getting good data that is very interesting to analyze.

Website Optimizer: Some of My First Results

Now that I have enough variations and a decent Experiment setup, we can take a look at some results.

As mentioned previously, I have three Test Sections with two Variations each setup on my Test Page:

1.) Full Table (makes the 3 book package look like one book w/ 2 Free bonus books)
2.) Old Sale Price (shows an original, higher price crossed out above real price - not in original page))
3.) "Limited Time Price" (mentions this text - not in original)

So this gives me 2 x 2 x 2 = 8 variations ('2' being the original + the new variation)

The reports give you two primary views:

Page Sections: shows each page section w/ impressions and conversions for each variation of each section.

Combinations: Showing the best combination of variations.

After about a day or so, my Page Sections looks like this:



What this is showing me, on the right, is that the conversions are all over the board for this time period...no clear cut single, stand-alone Test Section Variation really stands out.

The best performer on a stand alone basis was the "Bonus Table" which was changing a large table to display the 3 book package as a primary book w/ 2 free bonus books (basically testing the idea of a 'free bonus')

The combination results are what is really important...out of the 8 possible combinations, which did the best? That is how I will change my page when the Experiment is done.

I'll post the combination results this afternoon...

Editing Experiments in Google Website Optimizer

When I went to edit my original Experiment so I could add more Test Sections, the first thing I did was Stop the test. Don't do this. The Experiment is then considered "done". I had to recreate the experiment and replace much of the tracking code. I thought at first, "no problem, all the tracking code and scripts are in place"...but as discussed in this previous post, that didn't matter.

So I re-pasted all the tracking code and scripts. There is a 'pause' feature which is probably the correct option for doing this - if that is even really necessary.

At first I was having a hard time figuring out how to go back and add more Test Sections to an exsisting Experiment (without "stopping" it!) - then I noticed the "edit" link on the right hand side of the checck list telling you what steps have been completed when you click on the Experiment name.

Take note of this: when previewing your variations, hit the save button first - after you paaste your variation code in the box - otherwise it wont show up and you'll probably think error first.

What I added: I put a variation on a blank space below the discounted price that read "limited sale price", and also replaced a large amount of data in a table. The table contained three books that are part of the package and also had many links to see samples of the books and other related pages. I thought I would test out getting rid of all the links (possibly distracting the potential customer) and re-arranged the books so the most popular selling book was up top, and the second two were listed as 'Free Bonuses'.

The way I handeled the variation code was pinpointing the TR and /TR for the set of three rows containing the info on the three books. I made the changes in Dreamwaever using the test editing page duplicate (discussed in this post) and then just cut and pasted into the variation box in Google Website Optimizer, taking note to cut from the same exact TR and /TR that I placed the Test Section Code in.

So I feel like I've come a long way in the past few days...Ive got a lot more experiment with and look forward to posting on the reports.

Ironing Out More Wrinkles Setting Up Google Optimizer Split Tests

The problem with the delay in conversion tracking I had was that I left the product password on the (off domain) download page - the one I disccovered earlier I couldn't use as a Google Optimizer converion page becasue it was on a different domain. My resolution to that problem was to tell the customer they needed to click "here" to get the password (and land on an on-domian conversion page) - which didn't work since they saw the password I left on the download page and went about their merry way without ever hitting my conversion page. I fixed that by deleting the "the password to open the .pdf is..." text from the download page.

More Things I've Learned & Updates:

OK - I've now tried to modify an experiment unsuccesfully, and then successfully...adding a significantly more amount of Test Section combinations. I learned a lot and have updated the Clarifying Terms post to better reflect the relationship between Google Website Optimizer Experiments, Test Pages, Test Sections, Variations, and Conversion Pages.

I've also discovered that you can't use the same Test Page script on all pages, originally thinking I could add it to a Dreamweaver Template in the header. I updated a previous post about that here.

My original Experiment, which simply added a graphic showing an "old price" (higher) crossed out and none such thing as variants above the actual price produced results of a .62% conversion with the crossed out higher price, 0% without (only about a day of testing).

Next I wanted to add some more Test Sections and Variations to this experiment, but ran into a fatal flaw I'll discuss in my next post...

Google Optimizer Terms Clarifications

I wanted to take a moment to clarify the terms I'm using here since it can get a bit confusing. I'm trying to stick with capitalizing official Google Optimizer terms, which I'll define below, but in earlier posts I wasn't doing this and probably using multiple terms for one meaning.

The most confusing is the term 'test' since I and many others use it a lot when talking about this type of subject, becasue hey, we are testing stuff and we do trial and error related tests just to complete our test!

In Google Optimizer and this blog, a Test Section is something distinct (notice the capital T?):

note: these definitions are not official, they are just my interpretation.


Test Section: a snippet of code within a page that is being swapped. There could be multiple Test Sections on a single Test Page. Exaples may be a graphic or text headline (html). Each Test Section can have multiple Variations.

Variations: Different versions of Test Sections. For Example, if you are testing a header that originally said "Flying Swamp Chickens Here!", you would section off that html for the text as a Test Section within your Test Page, then add one or more Variations such as "Big Flying Swamp Chickens: Gettum' Here!" and "Fat Flying Swamp Chickens Found Here!".

Test Page: The web page you are performing Tests on. Usually a sales page or a landing page, the Test Page will contain one or more Test Sections where images, text, prices, graphics, etc are 'swapped' for different visitors and those visitor's actions are tracked to see if they make it to a Conversion Page.

Conversion Page: The page that measures the success (by way of a visit) of the Tests on the Test Page. As visitors visit the Test Page, and different visitors see a different Test or combination of Tests, they may or may not end up at the Conversion Page. If they do, that information is tracked, or credited, to the Test ot Test combination that visitor was exposed to when they visited the Test Page. The Conversion Page is usually a "goal" like a visitor signing up for something, or purchasing something...i.e., a "thank you" page.

Experiment: A combination of a Test Page with one or more Test Sections, Variations of Test Sections, and a Conversion Page.

Test Page Tracking Code: The code placed in the header of the Test Page to track visitors and create the Tests.

Test Section Code: The code used to 'wrap' the sections of code on the Test Page that will be swapped with two or more variations. The sections are all given names within the Test Section Code.

Conversion Page Code: Code placed in the body of the conversion page that tracks when a visitor visits, and helps to associate that visitor with which Test Sections they were exposed to, given that particular combination of Test Section Variations credit for the conversion in the reports.

In a previous post here, I mentioned a delay in G optimizer conversion data posting...turns out it was a bit of user error...you know, a problem between the mouse and the chair...that may have caused it. I'll explain in my next post.

Multiple Test to Single Conversion Page

One of the issues I ran into while setting up Google Optimizer was whether or not I needed to add more tracking code to the conversion page since it already had tracking code from another experiment. I ran into an error when I didn't do it, so I pasted another set of conversion page experiment scripts below the first set on the page.

The first experiment was an actual test, as I mentioned in a previous post. It was a fake Test Page, but a real conversion page. Now I've stopped that experiment and Google Optimizer directs you to remove the code scripts from the Test Page and Conversion Page.

This prompted me to go to the Conversion Page and compare the two tracking scripts from the two different experiments. One line seems to identify the experiment, for it is the only line that is distinctly unique with each experiment:

urchinTracker("/2964349575/goal");

(note: I changed the number slightly because at this point I'm not sure whether I should publish this number....probably doesn't matter but I'd rather be safe.

The script from the other experiment was identical except only for the number (all text the same)

In conclusion, with something like a sales purchase thank you page on a large site, you have the potential to have many different experiments all pointing to the same Conversion page. This will create a long stretch of multiple tracking scripts...unless there is a cleaner way to do it?

Delay in Google Optimizer Data Posting

Assuming I got everything set up correctly - there seems to be a significant delay in conversion data posting. I had some conversions overnight, the last one at 2:30am EST, and its now 8am and Google Optimizer is still showing zero.

Not a big deal - Google Analytics is the same way. Im just in this state of "did I set it up correctly or not". G Optimizer did check for my code and said ok...we'll see.

Helpful Tip: Setup a "Testing" Page to Grab Code

In G Optimizer you need to replace sections of code within a given Test Page (Experiment)with alternative code. I found it easiest to copy the page in my development environment, renaming it from sales.php to test_sales.php or whatever, with no intent to ever upload it.

From there you can tweak the various aspects you want to test, such as headings, sales pitch, price or price font size, whatever...all on the test_sales.php without worrying about screwing up your original. Then, you can copy and paste the code section into G optimizer more easily: the start/stop sections are easy to loacte since they are surrounded by the same code as the actual Test Page - this makes it easier for placing your Test Section code.

meegwell

Templates and G Optimizer Tracking Code

I use Dreamweaver for development and typically have a single "main" template for each site. G Optimizer requires you to put a script in the header of the page you will be testing for a given experiment, along with the tracking code on that page and your conversion page, as well as the start/stop code snippets to ID the sections of html to be tested, or 'swapped', on your test page.

I immediately ran into the problem of code having to be embedded in non-editable template regions of the page. What I did was place the tracking script in the header of the template, so all pages have that now. I've only experimented so far with testing sections within editable regions so the test section code hasn't been a problem....and then the tracking code can go anywhere in the body so that shouldn't be a problem with anyone.

UPDATE: You can't get away with posting the tracking code in the template for it is unique for each experiment. You'll need to put an 'editable attribute' section in the template head and add the script there for each test page.

I ran my first test of Google Optimizer on a pure test test page (that is , a page i made just for the test) but used an actual conversion page (the new thank you page I mentioned earlier with the DLGuard problem).

When I was satisfied I knew enough to be dangerous, I implemented on a real test page with a new experiment I planned on taking live (the sales page). I figured since the conversion page (thank you) was still the same, I would just leave the tracking code there and put tacking code, test snippets, and script on the new, actual, test page.

No can do. The "test that you didn't screw up your code" section of a G Optimizer experiment setup told me it didn't see tracking code on the conversion page...So then I wondered "what if I'm running multiple experiments with the same goal? No problem apparently, I've got two different tracking codes pasted into the same conversion page and although I don't have data posted to reports yet, it passed the smell test.

I havent tested multiple experiments to the same Test Page yet. After thinking about this, I dont think you should have to do that anyway. The way I see it, an experiment is page based. You experiment on a given page, with multiple sections tested within that page. One page is defined per experiment. I might test it out anyway.

Some Initial Thoughts From the First Google Optimizer Test

What I had in my head the whole time I searched for an A/B tester was that I would create multiple versions of the pages I was testing. In Google Optimizer, you actually define sections of code you want to swap within a page. I haven't tried it yet but I assume you can just define the whole page or just the body etc.

So I already had my first test in mind which basically took a sales page, where I was selling 3 eBooks, and swapped out the way I presented the books. The default setup was to show all three eBooks as a "package" for a single price. The 'test' was to show the primary eBook, which is the one most of my traffic is driven by, as the eBook for sale, and the other two as Free Bonuses for buying the first. In addition, I was going to put in the ole' regular higher price crossed out graphic above the actual price.

With this in mind, when I got in to G optimizer for the first time last night (moments after finding my beta invite email in the spam filter!), I struggled with whether I should do such a test for my first try.

See, the way G optimizer is setup, having you define sections of code, required me to pinpoint the exact start and stop points within the larger page - I guess I could have done the whole body - but I was still a little anxious. Another thought was if I do the whole body, what if I want to test something smaller, within the body test section tags? This would be "nested testing" as I see it...and I thought about it a lot while lying in bed a few hours ago. I think I'll test nested testing today, and make it the subject of My next post. In any case, the way its setup I think will ultimately be fantastic for it will give incredible flexibility to testing - and that's the point...test 2 headlines on a page in combination with 4 different sales pitches and just let G optimizer tell you what the best combination of the 6 variables is - which combo makes the most conversions? For those of you who have taken the gmat or other similar tests - think permutations - then think about how much time this "section of code" testing is saving you over doing it a full page at a time. In the 2 headlines and 4 sales pitches, you would have to make 8 pages if you used a straight full page a/b tester.

Another thing I stumbled on and pondered a bit was the "percent of test" setting - I wasn't quite sure if this meant "% we will swap the pages" i.e. your test page will be shared 50% and 50% between the original and the optimized - or whatever - in any case it cant be this with all the different variations - it's just how much you want the test running vs. how much you want your standard pre- google optimizer pages running.

What is an Experiment? Another stumbling point - do I setup an experiment for a site, a page, a part of a page....I set up a G Optimizer "experiment" for a page, with multiple tests under that experiment. Sort of like a "Campaign and Ad Group" in AdWords. So, my experiment is the sales page (the "test" page in the setup) - I named the experiment "Sales Page Old Price and Bonus" to correspond with the two tests I mentioned earlier. Within that Experiment, I have sections of code tagged and some alternate html setup to swap (this is easy to setup in "new experiment' process) for each section. For example, with the "old price" graphic, which is just the higher price crossed out above the current price, I defined a section of blank space above the price, and the alternate or test code was simply the html to place and center the graphic there.

I was about to continue with another issue of tracking code and templates, but I think it deserves it's own post...

G Optimizer Split Test Setup & DLGuard

The first problem I ran into when setting up my first G optimizer test had to do with the way I have DLGuard setup. At the advice from Sam at DLGuard, I set the download security software up so that the "thank you" pages are all on the same domain as the DLGuard software itself - this is necessary - and not an issue if you are dealing with one website. My multiple sites that sell products all direct the transactions through a single centralized domain w/ DLguard on it. This works fine.

G optimizer requires, at this point, that the "test" page (the page you are running a/b tests on) & the Conversion page (the page that measures the success of your a/b test) be on the same domain.

In my case, and many others I'm sure, the conversion page was my 'thank you' page for product downloads - on the centralized domain.

How I resolved this: this issue came up just last night as I setup my first Gooogle Optimizer test, so maybe I'll eventually figure out a better way to resolve it. For now, what I did was add some text to the download page (the page itself and the customize area of the DLGuard "box") telling the customer that they NEED TO CLICK HERE after downloading in order to get the password for the eBooks they just downloaded. I normally just gave them the password right there.

That clicks bring them to a "thanks - the password is blah blah" page back on the domain I'm running G Optimizer tests on. So now Im able to use that new page as my Conversion Page in Optimizer. It will be a day or so before the Gooogle optimizer data kicks in so I'll see how that goes.

Why Are We Here?

About a week ago I was actually shopping around for A/B testers when I first heard about Google Website Optimizer. It was definately time for me to stop jumping from one new marketing project to the next, and begin fine tuning what I already had out there. I'm a big fan of stats, analytics, and such so I knew what I needed to do, I just didn't have the right tool.

Over at the Warrior Forum I made a post asking what A/B test tools folks used and someone said Google Optimizer. I searched and soon discovered it was an invite-only beta so I signed up to be a tester. I read some info. on it and knew that G Optimizer was the tool I needed. I just needed to be patient and wait for the invite.

The more I thought about it the more excited I got about split testing every little aspect of my landing and sales pages...I checked my spam filter twice as much as usual in case the invite got sucked up there. Sure enough, it did. I found it last night and spent the next couple hours setting up my first test. Note: this would have taken much less time if I was not using DLGuard on a separate domain from the site I was testing...and since that was my first of a few stumbling blocks, I'll focus on that issue for my next post.

I'll make separate posts for each major issue I run into, resolved or not, as I jump into the world of G Optimizer...