Where to Marketing online?

TL;DR

• Optimizely’s most successful landing page A/B test was a three-word change that made our Pay Per Click landing page “symmetric” with our SEM ads.
• Visitors who saw the symmetric version of our PPC landing page converted to sales leads more frequently than users who saw the non-symmetric version, with high confidence.
• The observed improvement in conversions from visitors to sales leads was more than 39%.
• This symmetry experiment saved Optimizely hundreds of thousands of dollars and led to tens of thousands of additional sales leads.

Background

You may have heard that keeping your ad consistent with your landing page is good. But is that a myth? And if it isn’t, how can you quantify the impact of this symmetry? In early 2013, we performed an experiment on our Pay Per Click landing page aimed at answering these questions. Our goal was more to answer an interesting question than to find a conversion lift, but we were pleasantly surprised to discover that making our landing page mirror our ads led to a tremendous lift in conversions.

The experiment

We send traffic to our PPC Landing Page from lots of advertising campaigns, including search marketing campaigns. In some of our SEM ads we call ourselves an “A/B testing tool” and in others we call ourselves a “Split testing tool” and so on. These are all valid ways to describe Optimizely, but some people think about our space differently.

In the control, the headline of the page was always the same. It always said “Test it out for free, ” no matter which ad the user had clicked on.

In the variation, the headline of the page mirrored the search ad. For example, if the search ad said “A/B testing tool” then the headline on the landing page would say “A/B testing tool, ” and if the search ad said “Split testing tool” then the headline would say “Split Testing Tool.”

Original vs. Variation

Those are just a few examples, but conceptually this is how we split the traffic between control and variation.

Experiment results

We saw a 39.1% increase in conversions from visitors to leads in the variation with symmetric messaging. In the control, which had non-symmetric messaging, the conversion rate was 12.21%. But when the headline on the landing page was symmetric with the ad copy, that conversion rate jumped to 16.99%.

Funnel impact

We get a sales lead whenever someone submits a form on our PPC landing page, so the 39.1% increase in form submissions translated to a proportional increase in sales leads. We didn’t pay a penny more for the 39.1% increase in sales leads. In fact we paid slightly less because the experiment reduced the bounce rate on the landing page, thereby improving our Quality Score in AdWords and giving us a discount on a CPA basis.