19th October 2011
Analytics Under Attack: Google's Evil, Unethical Move To Remove Referrer Data
Google has announced that it is to cease providing referrer information in some instances. In the official blog post, Google's Evelyn Kao writes:
When you search from https://www.google.com, websites you visit from our organic search listings will still know that you came from Google, but won't receive information about each individual query.
Initially this change affects people logged in to Google accounts and using Google.com which, Google claims, is a very small percentage of searchers (although still a large number of people). But it's likely this will change as, according to Google's own blog entry:
As we continue to add more support for SSL across our products and services, we hope to see similar action from other websites.
To give an example of what Google have actually done, I have searched today for "car insurance" both logged in to my Google account and searching on https://www.google.com, and not logged in to my Google account and searching on http://www.google.com/. In each case I have clicked through to the same landing page. Here are the referrers of that landing page in both cases:
Referrer When Not Logged In, Clicking On A Natural Link: http://www.google.com/#hl=en&sugexp=kjrmc&cp=5&gs_id=l&xhr=t&q=car+insurance&qe=Y2FyIGk&qesig=Eeu3hebYxgo0in9YDLhtAA&pkc=AFgZ2tkKH3Xw88yrwvHzg5MkB-5vAi8dBrAzxf3se4-a7_BaiiecMyYZt0D_3TtcaX8K2jJgbEC3Yw7qMsDB65pNgSjYWjDjlA&pf=p&sclient=psy-ab&source=hp&pbx=1&oq=car+i&aq=0p&aqi=p-p1g3&aql=f&gs_sm=&gs_upl=&bav=on.2,or.r_gc.r_pw.r_cp.,cf.osb&fp=8e7fa2636e8b849&biw=1680&bih=947
Referrer When Logged In, Clicking On A Natural Link: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CHoQFjAA&url=http%3A%2F%2Fwww.moneysupermarket.com%2Fcar-insurance%2F&ei=RoaeTuS6IY_D8QOP8fixCQ&us...
I have highlighted the key difference in bold red above. When not logged in, my query "car insurance" is available in the referrer for Analytics to pick up and use to provide the site owner with information about what I was looking for. When logged in, my query "car insurance" has been stripped, so the site owner is completely clueless about why Google sent me to that page on their website. Note, then, that when logged in, that referrer is a lie - the page I was visiting before was not the one in the referrer at all. For example, I was actually on https://www.google.com/, not http://www.google.com/.
This small change has some very large consequences for site owners. For example, no matter what analytics package you use, any reports that show keywords will become less useful (and, at the extreme, useless). Check out this short interview from Google Analytics Evangelist Avinash Kaushik, following his keynote at 2010's Search Engine Strategies (which I attended with interest):
If Google removes keywords from referrer data then all of the great keyword ideas, keyword techniques and keyword attribution models that Avinash shares are no longer possible. Evangelise that, Avinash!
Joking aside, a lot of the great work SilverDisc and others do in making sites better for users will be made more difficult and less effective by this move.
Google's move upsets the ethical balance that exists between searchers, search engines and site owners. This is the very principle that ethical SEO is based upon - the three stakeholders to be considered are
- Site owners who produce great content designed to meet their visitors' needs.
- Search engines who are allowed to crawl and index that content as long as it provides benefit to the site owner.
- Searchers who get to find the information they need in order to satisfy their enquiry.
From my original ethical SEO paper, the most ethical technique
- produces the most good and does the least harm
- respects the rights and dignity of all stakeholders and treats all stakeholders fairly
- promotes the common good
- helps all participate more fully in the goods we share as a community and a society
- enables the deepening or development of those virtues or character traits that we value as individuals, professions and members of a society
How does Google removing referrer information produce an unethical result? Let's break it down:
- produces the most good and does the least harm?
- site owners can no longer optimise their sites to better match the searcher needs, so they will struggle to produce the best possible websites
- respects the rights and dignity of all stakeholders and treats all stakeholders fairly?
- site owners, rather than being treated with dignity, are treated as being "not trustworthy" and are denied a piece of information that the other two stakeholders (Google and the searcher) both have - the search query that resulted in that searcher visiting their site.
- promotes the common good?
- the common good is Google working with site owners to produce a better Web, which to be fair does happen a lot in other ways. This move, however, does not promote the common good - Google gains and the site owner loses.
- helps all participate more fully in the goods we share as a community and a society?
- clearly this move prevents full participation of site owners in something they have had available to them since the earliest days of the Web and something upon which the Web was built - referrer data was provided in the HTTP 0.9 specification and has been there ever since
- enables the deepening or development of those virtues or character traits that we value as individuals, professions and members of a society?
- again, this move alienates site owners and does not engender a spirit of cooperation and teamwork among site owners and Google, whose entire service is built on the content that site owners freely provide
So this move is unethical. But is it evil? (Note I deliberately use the word "evil", of course, since Google's corporate mantra is "Don't be evil").
What's really evil about Google's announcement is the patronising spin they've put on it. Google's headline, even on its Analytics blog which is aimed at site owners rather than searchers, is not "We're removing site owners' ability to pull keywords from the referrer"; it is "Making search more secure: Accessing search query data in Google Analytics". This fails to treat site owners with the respect they deserve. The whole piece is positioned as making search more secure, for example when using insecure Wifi hotspots, yet at least a couple of things don't stack up if this is the objective:
- If the user is visiting a secure Web site then Google still strips the referrer (thanks Danny Sullivan at Search Engine Land for this info), even though this is not necessary and, given they don't do this on their Encrypted Search, Google clearly knows it's not necessary
- Searchers' referrers still contain keywords if searchers click on an ad, rather than a natural result.
That last point really shows where Google's mind is at. To juxtapose a couple of points from their blog post:
we recognize the growing importance of protecting the personalized search results we deliver. As a result, we’re enhancing our default search experience for signed-in users ... [but] ... if you choose to click on an ad appearing on our search results page, your browser will continue to send the relevant query over the network to enable advertisers to measure the effectiveness of their campaigns and to improve the ads and offers they present to you
So advertisers who pay Google money get treated one way, site owners who pay Google by providing the content the whole Google service is built on get treated a different way, and searchers' privacy is not really protected. Nice. To complete the example I gave earlier, the third link below is the referrer I received on the same website as result 2, but this time clicking on a paid ad rather than a natural result:
- Referrer When Not Logged In, Clicking On A Natural Link: http://www.google.com/#hl=en&sugexp=kjrmc&cp=5&gs_id=l&xhr=t&q=car+insurance&qe=Y2FyIGk&qesig=Eeu3hebYxgo0in9YDLhtAA&pkc=AFgZ2tkKH3Xw88yrwvHzg5MkB-5vAi8dBrAzxf3se4-a7_BaiiecMyYZt0D_3TtcaX8K2jJgbEC3Yw7qMsDB65pNgSjYWjDjlA&pf=p&sclient=psy-ab&source=hp&pbx=1&oq=car+i&aq=0p&aqi=p-p1g3&aql=f&gs_sm=&gs_upl=&bav=on.2,or.r_gc.r_pw.r_cp.,cf.osb&fp=8e7fa2636e8b849&biw=1680&bih=947
- Referrer When Logged In, Clicking On A Natural Link: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&sqi=2&ved=0CHoQFjAA&url=http%3A%2F%2Fwww.moneysupermarket.com%2Fcar-insurance%2F&ei=RoaeTuS6IY_D8QOP8fixCQ&us...
- Referrer When Logged In, Clicking On A Paid Link: http://www.google.com/url?http://www.google.com/aclk?sa=L&ai=ClrXYRoaeTt...q=car+insurance
What can site owners do about this? Individually, not a lot. Promoting and using other search engines, such as Microsoft Bing, would be a start. This strikes me as a great opportunity for Microsoft to build and foster better relationships with site owners, for example by promising never to remove referrer data from its search results.
If they were able to operate as a collective, site owners could do Google serious damage. In my 2007 post "Bringing Down Google With Two Simple Lines of Code" I showed how this could be done.
The permission can be taken away with two simple lines of code placed in a site's robots.txt file:
Sure, every site owner in the world would need to publish this file to their sites. But if they did such a thing, the Google search engine could no longer crawl or index any of the Web's content. It would be defunct.
So, fellow site owners, Google's future is in our hands. If you want to go "on strike" and stop Google profiting from the fruits of your labours, simply publish the code. Be warned that your site will eventually be removed from Google's index if you do so. As a unilateral step, this may do you more harm than good. But if we all do it en masse, then beware Google!
That post was written four years ago. Now, with social media so prevalent that it can lead to regime change in countries, maybe it can lead to regime change among search engines too. Microsoft, are you listening?