The Paid Search Impact of Google Instant – Some Initial Data

The arrival of Google Instant created great excitement in the search community.  There has been a lot of speculation about the impact that Google Instant will have on paid and natural search.   Will it help or hurt the long tail? What will its impact be on conversion rates? How will click-through dynamics change on the page? Will big brands benefit from an increased traffic share? Some initial data started surfacing at SMX East and some blogs.  The consensus seems to be that there are no dramatic changes as yet.  Here we will present our own data on the paid search impact we have observed so far.

IMPRESSIONS AND AVERAGE CPC

We looked at data for a set of our US retail clients and compared metrics after the launch of Google Instant on September 8th, up to October 6th, against a similar length period prior to the launch.  In figure 1 we show the distribution of percentage changes in overall impressions and average CPCs before and after the launch of Google Instant at a campaign level.  We focus on a reasonably large set of campaigns where the overall spend has stayed fairly stable over the period.   In each boxplot the horizontal black line represents the median change for each metric.   An earlier post explain how to interpret the boxplots in figure 2.  For impressions there was a median 2.9% decrease before and after the launch of Google Instant, and for CPCs a median 3.2% increase.  Given the variability in the data, it would be very premature to get too excited about the changes we see below.  It really suggests that nothing has changed significantly in terms of impression volumes and CPCs.  A change may come when people will change their search behavior over a longer period of time and get familiar with Google Instant.

Changes in impressions and average CPCs before and after the launch of Google Instant.

Figure 1: Changes in impressions and average CPCs before and after the launch of Google Instant.

THE EFFECT ON THE KEYWORD TAIL

Many pundits argued that the arrival of Google Instant would spell the end for the long tail.  A searcher may start a search with the intention of using a long-tail search term but may end up getting good enough results when they have only typed a part of the original intended query.  Others argue that most people would finish typing their original query anyway.  We analyzed a large set of over 20,000 paid search keywords across several US advertisers to see if we see any drop in the revenue contribution from tail terms.    We compare the 3 weeks before Google Instant to the 3 weeks thereafter.  There were 22,031 keywords with an impression in the 3 weeks before Google Instant.  This dropped to 20,911 keywords after Google Instant.   This is not significant, as this level of variation was not unexpected for such a large keyword set even before Google Instant.  What we are more interested in, is to see if there has been a shift in the contribution to total revenue between head and tail terms.  In figure 2 we rank all keywords into percentiles according to their traffic contribution, starting with the highest traffic keywords on the left.  As an example, we see that the top 10% of keywords in terms of traffic account for about 80% of total revenue.  We can consider these as head terms.  Total revenue refers to click date revenue only, as there will be very little data with full cookie revenue.  If there has been a substantial shift in traffic towards the head keywords after Google Instant, we would have expected the red line in figure 2 to have shifted above the black curve on the left of the plot; however, it is clear that there has been very little change in the relative contribution of head terms and tail terms.  This suggests that we are not seeing a dramatic decrease in the contribution of tail terms to total revenue yet.

Figure 2: Revenue contribution by keyword click percentiles.

Figure 2: Revenue contribution by keyword click percentiles.

CHANGING CLICK-THROUGH DYNAMICS

An interesting potential impact of Google Instant is how it will affect the click-through dynamics on the first and subsequent pages of search results. There are some theories suggesting that relatively more clicks may occur on higher positions than before.  The data for a group of our US clients in figure 3 suggest that there may have been a subtle shift in click-through rates before and after Google Instant. We plot the square root of the click-through rate in figure 3, as it results in better visualization due to the heavily skewed click-through rate distributions. We focus on a subset of keywords with at least 1000 impressions in a 4 week periods before and after the launch of Google Instant. The median click-through rate (represented by the black dots) seem to have shifted slightly higher at the top positions and slightly lower at the lower positions.

Subsequently, we fit a logistic regression model that enable us to model the click-through rate as a function of position and match type before and after Google Instant.  The regression fits are shown in figure 4 for 2 match types. They also suggest that there has been a subtle increase in click-through rates after Google Instant at the higher positions and a subtle decrease at lower positions.  In order to evaluate the statistical significance of these changes in the light of the inherent variability in the data, we fitted another regression model for three position ranges: top (positions 1-2), middle (positions 3-6) and bottom (positions 7-12) instead of individual positions.  We then compute click-through odds ratios (and their 95% confidence intervals) for comparing the odds of a click before and after Google Instant for a specific position range. For a refresher on click-through odds refer to this earlier post.  The results are summarized in table 1 below.  If there has not been a significant change in the odds of a click we would expect the confidence interval to include 1.  Table 1 suggest that the odds for a click on the top positions has increased by about 6.6% after Google Instant, while the odds for a click has decreased by about  13.1% after Google Instant at the lower positions.  The change for the middle positions is much smaller and barely significant, reflected by the fact that the corresponding confidence interval almost includes 1.  The lower positions here also include some page 2 positions, which may suggest that Google Instant is reducing the importance of page 2 results as well. The relatively thin data on second page positions prevent confident inference about the effect on page 2 at this stage.

oCTR Top vs BottomFigure 3: A comparison of click-through rates for top and bottom positions before and after Google Instant.

Logistic Fits Pre and Post GInstant

Figure 4: Modeled click-through rates by position pre- and post-Google Instant

Table 1: Click-through odds ratios pre and post Google Instant from logistic regression fits

Table 1: Click-through odds ratios pre- and post-Google Instant from logistic regression fits

SUMMARY

Below we summarize our findings:

  • There does not seem to be a significant shift in overall impressions and CPCs at this stage.
  • Our data does not provide any evidence at this stage that we should start preparing for the funeral of the long tail of paid search just yet.
  • A subtle shift seems to be happening in terms of click-through dynamics on the first page (and possibly the second page).  There seems to be a slight increase in click-through for higher positions and slight decrease for lower positions, resulting in relatively more traffic from higher positions than before.  If this results in higher competition amongst advertisers for higher positions in order to maintain volume, it is certainly not something Google will be too unhappy about.  The best strategy remains to bid traffic to its estimated value rather than focusing too much on position.
  • More time is needed to see how Google Instant affects longer terms search behaviour, as people become more familiar with it.
  • Google Instant could also affect conversion rates, which was not investigated here.  We will investigate this once we have gathered some more conversion data.

About Wiesner Vos
A chartered statistician with a passion for quantitative marketing.

Comments

9 Responses to “The Paid Search Impact of Google Instant – Some Initial Data”
  1. I think I managed to understand most of this :) It’s really great to see some hard data informing opinion about Google Instant, instead of just speculation. Interesting to see that not much has changed in terms of the long tail yet, but I guess it would be expected that the user learning/behaviour change cycle will take a while to have a significant effect. I wait upon more findings with bated breath

  2. Wiesner Vos says:

    Hi Joe, Always nice to hear from you. I also think the behavioural changes will take a while to become clear. I did not think Google would stir the money maker too much and release something that will result in immediate radical changes. I think we will see a more subtle change over time. Quite keen to see what happens with conversion rates over the longer term.

  3. Tal Galili says:

    Great post, and topic.

    If ever you’d wish to add the R code/ data of things you do, then you’d be welcome to add your blog to:

    http://www.r-bloggers.com/

    Cheers,
    Tal

  4. Wiesner Vos says:

    Hi Tal, Thanks for your positive comment. In fact, I am a keen reader of R-blogger and I am happy to share my code and data (where I have permission). I will definitely consider posting on R-Blogger in future. Wiesner

  5. Good stuff, Wiesner.

    As Kevin Lee of Didit pointed out: Google has never released a major change to AdWords that has been revenue neutral for Google, they’ve always been revenue positive. The effects are subtle at this point, but a small percentage increase in revenues per search leads to a huge chunk of change for them. It will be interesting to see how this plays out.

  6. Wiesner Vos says:

    Thanks for the feedback George. I am sure the subtle changes will reflect in the Google income statement down the line. Btw, I really enjoy reading your blog posts.

  7. Wiesner Vos says:

    I neglected to mention in the post that we did think about the proportion of actual US Googlers exposed to Instant and how that affects results. I suspect that Google is limiting exposure initially, which is also mentioned in a post by EConsultancy http://econsultancy.com/us/blog/6701-google-instant-has-changed-the-way-we-search

    This does mean that we need to be cautious about making very strong claims about the impact of Google Instant at this stage. It will be a longer term project to understand the real impact of Google Instant.

  8. tsanko says:

    Wonderful…thanks a lot for posting a good informative blog

Trackbacks

Check out what others are saying about this post...
  1. [...] This post was mentioned on Twitter by Graeme Lipschitz, Adriaan Strydom. Adriaan Strydom said: The Paid Search Impact of Google Instant – Some Initial Data : Clicks2Customers http://bit.ly/cY6uME [...]



Speak Your Mind

Tell us what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!