An interesting essay on search neutrality

(Just as a reminder: while I am a Google employee, the following post is my personal opinion.)

Recently I read a fascinating essay that I wanted to comment on. I found it via Ars Technica and it discusses “search neutrality” (PDF link, but I promise it’s worth it). It’s written by James Grimmelmann, an associate professor at New York Law School. The New York Times called Grimmelmann “one of the most vocal critics” of the proposed Google Books agreement, so I was curious to read what he had to say about search neutrality.

What I discovered was a clear, cogent essay that calmly dissects the idea of “search neutrality” that was proposed in a New York Times editorial. If you’re at all interested in search policies, how search engines should work, or what “search neutrality” means when people ask search engines for information, advice, and answers–I highly recommend it. Grimmelmann considers eight potential meanings for search neutrality throughout the article. As Grimmelmann says midway through the essay, “Search engines compete to give users relevant results; they exist at all only because they do. Telling a search engine to be more relevant is like telling a boxer to punch harder.” (emphasis mine)

On the notion of building a completely transparent search engine, Grimmelmann says

A fully public algorithm is one that the search engine’s competitors can copy wholesale. Worse, it is one that websites can use to create highly optimized search-engine spam. Writing in 2000, long before the full extent of search-engine spam was as clear as it is today, Introna and Nissenbaum thought that the “impact of these unethical practices would be severely dampened if both seekers and those wishing to be found were aware of the particular biases inherent in any given
search engine.” That underestimates the scale of the problem. Imagine instead your inbox without a spam filter. You would doubtless be “aware of the particular biases” of the people trying to sell you fancy watches and penis pills–but that will do you little good if your inbox contains a thousand pieces of spam for every email you want to read. That is what will happen to search results if search algorithms are fully public; the spammers will win.

And Grimmelmann independently hits on the reason that Google is willing to take manual action on webspam:

Search-engine-optimization is an endless game of loopholing. …. Prohibiting local manipulation altogether would keep the search engine from closing loopholes quickly and punishing the loopholers–giving them a substantial leg up in the SEO wars. Search results pages would fill up with spam, and users would be the real losers.

I don’t believe all search engine optimization (SEO) is spam. Plenty of SEOs do a great job making their clients’ websites more accessible, relevant, useful, and fast. Of course, there are some bad apples in the SEO industry too.

Grimmelmann concludes

The web is a place where site owners compete fiercely, sometimes viciously, for viewers and users turn to intermediaries to defend them from the sometimes-abusive tactics of information providers. Taking the search engine out of the equation leaves users vulnerable to precisely the sorts of manipulation search neutrality aims to protect them from.

Really though, you owe it to yourself to read the entire essay. The title is “Some Skepticism About Search Neutrality.”

46 Responses to An interesting essay on search neutrality (Leave a comment)

  1. SEO help many webiste to show to people not cheat, If the website cheat people, It’s not seo’s fault, it’s people’s fault, Google sholud forbird cheat, I think Google is doing this.

  2. Matt

    Thank you. Well said!

    “I don’t believe all search engine optimization (SEO) is spam. Plenty of SEOs do a great job making their clients’ websites more accessible, relevant, useful, and fast. Of course, there are some bad apples in the SEO industry too.”

  3. @Harith, I liked that part as well. In fact, I often use SEO as an excuse to get my client’s websites up to standards. It’s ironic that so many people don’t care about accessibility, valid code and relevant content but do want to score in Google, when in the end half (or more) of SEO is in fact those things they otherwise would never have invested in.

  4. Hi Matt, and thanks for the thoughtful article, and for signposting to the Grimmelmann piece.

    The net neutrality debate, of course, in part arises from Google’s dominant position in the search market.

    Could you share your thoughts on Google’s strong market share, and the implications arising from users trusting a single search engine for all the answers?

    I often think it is strange how we all use one search engine: it’s like going to the library, and the librarian handing you a single book for any question you have.

    Do you think the net neutrality argument extends further than just sites competing for visiblility and spam; how about the impact on intellectual endeavours, and even plain old intellectual curiousity and serendipity?

  5. “critics base their case for regulation on the immense power of the search engines”

    Are we not all aware of that power when we start? If we build a business to be wholly reliant on traffic from successful ranking in Google, should we not acknowledge the risk that also comes with that ranking? However relevant a site is for a term today, there may be more relevant site tomorrow – the risk I take from building a site reliant on Google is mine, the fact that the rules and the ranking factors may change tomorrow is a part of that risk. It’s not something I complain about, I just keep working to keep relevant, fresh and on topic.

    Google is not a government institution, we have to right to ranking in Google or even any right to relevant results – that is something that Google is aiming for to remain competitive, something that Google wants too. We seem to forget that Google hates spam too.

    Thanks for the link by the way Matt, it’s a great read.

  6. Should have been “*no* right to ranking in Google”

  7. Hi Matt,

    There’s alwz a saying a bad apple spoils all good ones. This same as happened to we SEO people. By wrong doing most SEO people are spamming & spoiling the name & fame 😉 for good SEO’s. Thx for your nice article.

  8. I don’t think the search neutrality debate is one about webspam – I think we can all agree that spammy websites are bad and need to be ousted from the SERPs. Nor do I think the debate is about making the search algos public – they’re Google’s intellectual property and thus deserve full protection.

    The search neutrality debate is about Google and other search engines giving preference to their own properties over those of their rivals, and I think it expands beyond the listing of organic results and should encompass other elements on the SERPs, such as the OneBox and the paid ads.

  9. I don’t believe all search engine optimization (SEO) is spam. Plenty of SEOs do a great job making their clients’ websites more accessible, relevant, useful, and fast.

    Thanks!

  10. I think SEO is a powerful tool and as every tool it can be used either for good or bad. Optimizing a useful website can help other people find the quality content easier and there is nothing wrong with that. Optimizing low quality websites, trying to cheat by using blackhat and spammy techniques is bad both for search engines and for users. Nevertheless it is Google’s and Bing’s job to ensure that their results are accurate and I think they both do good job.

    BTW Matt what you think of blekko? They are quite open, transparent and they explain their results by providing a detailed analysis (/rank operator). How do you think that this will affect SERP spam? Do you believe that Google is going to implement something like that on the future?

  11. Its nice to know you do have respect for SEOs who apply best practises. When I first started out it was easier to carry out practises that would have been considered to be in the Gray area of SEO however, I was taught to follow best practises and guide lines. These practices are the winners in SEO as Google has now grown in terms of finding and devaluing the Gray areas of SEO. I will be sure to read the Some Skepticism About Search Neutrality essay.

  12. Follow-up comment: Having read the essay now I have to admit I’m not terribly impressed by it. The author starts off with his 8 principles of search neutrality, which I think should be labeled ‘elements’ instead – search neutrality encompasses several of those principles (though not all 8 imho) and by defining them each individually I think he makes it much easier to subsequently shoot them all down, having narrowed each down to easily falsifiable premises.

    Additionally the author erects straw men for some of the definitions. For example for the Objectivity principle the author states: “The unvoiced assumption here is that search queries can have objectively right and wrong answers.” This is a weasel phrase and is a misrepresentation of the Objectivity principle.

    Also, by ignoring the interplay of the eight principles – and by including principles which are at best circumstantially applicable to search neutrality but should not form part of a serious debate (Equality and Transparency for example) – the author distorts the actual issue at the core of the matter.

    Lastly, the author seems to focus exclusively – though never explicitly stated – on the organic search results, which again is not an accurate representation of the search neutrality debate. Paid ads, universal search and the OneBox are as much part of this issue as the organic SERPs, and ignoring them in order to prove a point (which is already flawed) does a grave disservice to the public debate on the matter.

  13. Don’t take this the wrong way. This is a defense of Search Engines and it is well done. I like it and downloaded the PDF. I’ve never heard of SE Neutrality until now. Sounds like a principle I should be aware of as the web continues to evolve.
    BTW – I think it is up to the SEO community, not the search engines, to clearly define what an SEO does and what he shouldn’t do. There should be a third party certification that’s universally accepted. Just a thought.

  14. @”That is what will happen to search results if search algorithms are fully public; the spammers will win.” I like this is true.. 🙂

    Happy to hear that Google can easily catch “Bad Apples”

  15. A valid question by Grimmelmann: What will happen to search results if search algorithms are fully public? The spammers will win. Moreover competition will ensure that search engines keep on evolving to cut spam.
    Also liked this one: Telling a search engine to be more relevant is like telling a boxer to punch harder.

  16. I once thought you could have a fully public algorithm combined with supplying users spam filters however the spammers would quickly find cheats that by-pass spam filters. Ultimately the search engines cannot defeat spam by themselves they need a strong, enforceable code or law that removes the legality of spam.

  17. Great article, but there is one thought missing in the self-interest section. Let’s assume that Google doesn’t have any rules built in to favor one of it’s offerings that compete with other businesses. That would be neutral. Now, you are a Googler that is responsible for growing said business. Being a part of Google, you have access to the entire algorithm that determines results. You are in the position to use that insider knowledge to construct your pages such that they are guaranteed to rank highly. Is there a conflict here?

  18. @jeff hall – I don’t see how a law could prevent webspam. Unlike e-mail, SMS, or fax spam where they’re sending us their unwanted junk, webspammers are simply hosting their web sites. They’re not forcing search engines to spider their sites.

    That said, I’m against web spam 110% (times infinity) so I applaud Google’s efforts to prevent it.

  19. Well search engines have it a tough job to do, and the bottom line is you can’t keep everyone happy all of the time. No matter what your algorithms are it won’t suit everyone, but I think Google is doing a good job so far.
    I would like Google to have a more easily accessible guide to building websites for beginners, so they can at least get a fair shot. Most beginners do not even know what SEO is or any of the basics, so these are the people who need help the most.

  20. Since New York Times editorials, as a rule, reflect no independent thought but only a
    kind of prevailing conventional wisdom, it is clear that search neutrality has truly arrived on the policy scene.

    That’s mainstream media in general, not the NYT specifically. I love how certain major dailies refer to themselves as “the paper of the people” now.

    Seriously, the only issue I have with this piece is that it’s a tad condescending. What he’s saying makes sense, but it gives off a certain air of superiority that makes it hard to read.

  21. SEO is something great, it makes the internet more user friendly. The biggest problem is that 50% of the SEO experts use these techniques for there own “money making websites”. Adsense is a fine tool but websites “made for adsense” built by SEO experts gives the google search results a bad reputation. In google.be you can find them in many of the search results. I love google and really hope it’s possible to fix this.

  22. While I never agreed on the fact that Google latest “algo” update was favoring sites displaying Google products or ads, I totally agree on all the posts/articles stating that Google is loosing the war against spam. Just search for any keyword related to highly competitive affiliate niches and 99% of the top 100 results are ridiculous!

    I know that 90% of sites in these industries have duplicated content and spam links like crazy, but there are also many legit sites with unique high quality content. If you want to learn, fight and fix the latest techniques used by SEO Spammers it is as simple as following these industries and you’ll see how easy it is for Black Hat Seos to manipulate your rankings!

  23. Good stuff, especially since they wrote it in 2000 and had to forecast a lot of it.

  24. I think google has grown too big by now, what we need to do is start using other search engines so google feels some heat.

    It is obvious that google tweaks its search results, because i have had issues trying to set up “My Client Center” and had issues logging in because i already have an account… cant accept an invitation from MCC to an adwords user because i have an adwords account?????

    So i called the 1800 number, and that is only for setting up an account not for help, so they are ok taking adwords money but not providing support or giving out a number.

    so… I googled:
    google phone numbers, google adwords numbers, google support numbers………..
    Nothing Google doesnt even know its own numbers?

    25000 employees with no phone numbers? i wonder if they have email? gmail is after all still in beta, sounds like a dream place to work since no-one will ever bother you with requests 🙂

    And when you go through the support system and do not find the answer it tells you that phone support is available in a 4 hours window and the time is mentioned GMT + 11 (solomon islands) and when you wait then “AdWords phone support is temporarily unavailable” – they must be chilling on the beach on the solomon islands rather than work I understand I would too since googlers have no phone numbers according to google.

    I guess this is ultimate proof that google is getting to big, if they dont even care about paying customers(adwords), then how much do they care the general public?

  25. A very well researched article.

    For me, though, the dominance of Google in the search market is a source of concern – even more so in Europe, where its market share is vastly higher than the US.

    It imposes a heavy responsibility on Google, and this quote from the essay captures the reason why:

    “..Perhaps just as importantly, structural features of the search market can make it hard for users to discipline search engines by switching..”

    That, plus the point Grimmelman makes elsewhere in the article – that the capital requirement makes it all but prohibitive for a new search engine to be launched – not only gives Google extraordinary power over what searchers can find now, but probably entrenches their position well into the future.

    To mangle another quote from the essay: that gives them a de facto property right in the search market.

    We’re reliant, therefore, on Google continuing to ‘do no evil’. But, as we saw with the Google Map cars and the collection of WiFi data, Google does make mistakes.

    And where they have a 90% share of the search market those mistakes would be potentially hard to spot (by uers), but could have a major impact – affecting people’s businesses and, potentially, lives.

    Search neutrality in the sense of a search engine not scewing results to further its own agenda is not easy to assure, but I’m uncomfortable with the power that currently rests in the hands of one institution and where, in a sense, it’s regulating itself.

  26. Hi Matt

    you said
    I don’t believe all search engine optimization (SEO) is spam. Plenty of SEOs do a great job making their clients’ websites more accessible, relevant, useful, and fast.

    Means that you also agree to it that some SEOs can scam google. What we can conclude from this is that they are more techie having sharp skills than Google employees as ther are able to fail Google algos. They cheat Google seo techniques.

    Anu thoughts on this…? Will google continue to be a prey to such predators ?

    ATUL

  27. While I do agree this is a very interesting article, you could just as well argue that, in general, we all do know the algorithms of the search engines. Sure, the specifics are a little foggy, but webmasters and spammers alike are narrowing the gap.

    More or less, spam is simply an economic failure in which the various search engines try to find the right balance between determining spam and useful content. Personally, I think Google is one of the best and does an outstanding job of finding this balance. As you mention, there are “bad apples” but there always will be in an economically efficient society.

  28. Oh yaa, this book is very intresting.

    Matt, I have already read it. 🙂

  29. Hey Matt

    On the subject of neutrality. Our experience is that, independent of the methods by which Google ranks results, people don’t see Google as neutral. The customers we speak to see a site being top on Google as virtually a recommendation.

    We get called by a lot of disgruntled customers who have bought from Chinese sites which are top of the SERPs for “prom dresses” (via spam techniques). We ask these customers why they trusted these sites when they had never heard of them before. They always say ‘because they were top of Google’. Because trusted sites are usually top of the SERPs many consumers now believe that top sites can automaticaly be trusted.

    These sites conduct no PPC, no PR, no print…no other marketing. Google is the only way these companies are able to reach, and rip off UK customers. Every time a customer opens up a shoddy dress sent from one of these Chinese sites, it reflects on Google. Google has, in effect, been the medium through which these sites were put forward. The customer’s trust in Google is diminished.

    I really wish Google would address this.

    Thanks

    Vic

  30. Matt, I’m glad there is competition and challenge; without which there is a danger of complacency. Competition (or in this case oposition) can only help Google re-shape and improve.

    The main thing that moved me to Google from msn was the non-flash/image based advertisements.

  31. Interesting concept, search neutrality. I does seem a little redundant in the current state of the web because anyone with time and money can set up a search engine. I would also guess that if the web became more proprietary, developers including Google would find a way around any content restrictions. If there were any bias in the system, users would just start moving away from those methods. Just look at something like Cuil, it never gave expected results, so people just ignored it.

    Also good job on “not all seo is spam”. SEO is just good web design practice, I wish people could see that…

  32. Matt,
    There is also the issue of paid links. Although it can be considered either as link spam or “bad” SEO but I think is separate.
    For example, a blogger who was impressed by our service decided to give a big endorsement putting a big banner at the top (Nothing was paid or exchange in return). This can look as an advertising to an indexing algorithm.
    On the other hand a competitor who ranks fairly highly in SERP, have 99% of their links paid, their links are just scattered across different websites.

    I see this to be very challenging for Search indexing because how can you distinguish between a link that looks like paid, and 100s of those that are just hidden in the web. The search engine cannot be neutral in this case, you either have to punish the website with paid links or accept these links (which seems like Google does based on the competitors ranking) and in either case neutrality was removed either by the cheaters or by Google who is trying to make the result neutral.

  33. @Susan Hallam

    soory net nutrality is about unreconstructed PTTs in particular the botched deregulation of the USA still thinking in the “screw it their only subs” mindset – and wanting to charge for favoring one content provider over another

    Trust me on this I spent 15 years inside the wire in BT and used to work in the bad old days when you got charged 20p plus a data charge for every email – I know what some of the more dodgy marketing/sales types will try and get up to to game the system to their own ands and who cares about customers or the long term viability of the company MCI back in the day (i am sure Vint has a view on this) and more recently the Global Services disaster that BT had

  34. thank you matt

  35. As Grimmelmann says midway through the essay, “Search engines compete to give users relevant results; they exist at all only because they do. Telling a search engine to be more relevant is like telling a boxer to punch harder.” (emphasis mine)

    I guess that’s why we want to trust “Google” to return users whatever query they ask for – Well with all new Google updates I guess we might see new changes come 2011..

    Jerry

  36. Matt

    You and I haven’t always seen eye to eye, whether that be about life the universe and everything or whether one of my sites deserves to rank well or not 🙂

    But….. on this I agree with you completely,

    You may (or not) remember in 2006 during a session at Pubcon while I was in the audience I was handed the mic by Rand, he asked me to say a few words. I turned to the audience and you were standing (taking notes I may add) at the back of the packed room.

    Anyway, some of those words were “On Google’s website they can do what they like as it is THEIR website”

    I believe that statement still holds true and that anyone can do what they like on their website too. For me that is the true neutrality of search.

    Search shouldn’t be regulated and when competition in search becomes a reality and from whoever and wherever that competition comes I look forward to seeing it and the quality of the product, as decided by users and their choices, not regulation to be the definition of success.

    Search Neutrality? – Yes please, as long as that means everyone (Google too) can do what they like on their own websites.

    If search neutrality means an open algo then we may as well go back to Gopher as it will deliver the same quality of results as any web based search engine.

  37. Is anything ever truly neutral? I don’t think so (even Switzerland). Its an interesting article. In life and everything else, every thing has its own denotation. One, or one’s company can only work to be as fair as possible. Google appears to do an ok business at it. They can never make everyone happy after all. Only a $100 bill appears capable of that….

  38. “calmly dissects the idea of “search neutrality” that was proposed in a New York Times editorial.”

    The New York Times piece mentioned is not an “editorial.” As that word implies, an editorial by definition is written by an editor or a member of a paper’s editorial board. This piece is what’s commonly called an “op-ed,” an opinion piece written by someone without connection to the paper.

    It’s a very common distinction, especially given Grimmelmann’s assertion (groundless though it is) that Times generally “reflect no independent thought.” I’d dispute that in another forum but here it’s irrelevant. This piece is nothing but an expression of an opinion by a website operator advocating government action that he sees as helping to make his business more profitable.

    That is, for anyone who hasn’t looked at it, it’s written by someone who claims that his website was targeted by Google with “penalties” that prevented it from ranking well. The absurd stance that started this discussion — that the government should regulate search engine algorithms — came from a man with a clear agenda and self-interest, not from anyone connected with the “major media.”

  39. Sorry, just re-read my prior comment and found a confusing error. My second paragraph should begin “It’s a very important distinction,” not “common distinction.”

  40. Hello Matt,

    Thank you for this excellent information. I learned a lot reading this article and I really enjoyed it.

    Also I am glad to see Google doing more to fight SPAM and sites with low levels of original content.

    However you still have a ton of work to do and I have been waiting for over a year now for you to fix some Google mistakes. I wish my articles received more attention and here it is again:

    http://gototom2.blogspot.com/2009/09/warning-google-continues-to-penalize.html

    Please fix these Google mistakes documented on this page.

    Thank you.

  41. Interesting….and I’m glad there is some manual oversight. People can make decisions that automated systems can’t. I was directed to this blog by someone and so glad I found it….I’m just getting a new site started and I hope to learn a lot.

  42. I’m relieved to hear your view that plenty of SEOs do a good job because there is a vital job to be done and I get tired of hearing SEOs all tarred with the spam brush. Almost any tool can be put to good or bad use.

  43. Search neutrality cannot be assured as long as SEO companies are operating. There are numerous cases where the websites or keywords without much content or no content are in the top and pages which contain real information or relevant information are not even in hundred. Google is misusing its monopoly over search engines. A page of mine on May December love was on top for the search but instantly IDK what happened the page is out of search engine.

  44. Grimmelmann’s points in the snippets you included are very well written, I like the subtleness in he uses in making a straight point! 🙂 …off to read the full article now! Thanks.

  45. We frequently receive requests like: “We want to pay to have a link published on your web site”. What this means? Wel, from my point of view is the way Google is handling ranking that has generated this. As the major difference Google has added to search engine results are links then it is there we all play this SEO game. If you want to rank higher you must have more links. So your main tools is to have links: how? Well one way is to pay for them. It is not a surprise! These paid links hurt the bsic concept: if a site has many links that point to it then this site is better and should rank higher. This start to become no more true! At least not true as it was some years ago. While previously the SEO game was ALL based on contents (and we have seen there a ton of word spamming in the past), now the war is based on links. It is so simple to understand. So as soon as someone will come with a big new way to search thing we will see all sort of fake links published in link directory site and the like.

  46. A lot of spam is not meant for people but for search engines. Owners of high quality website have no other options than to use sneaky link spam.

    For instance I used to own a free informational sites that my users loved, however it didn’t get to the first page which was taken up by shallow sales pages. The issue is they can pay people $1 an hour to make thousands of back links but an actual user cannot backlink the content they like unless they have a website – this means the right to vote is restricted only to webmasters. Web viewers – the people who use the search engines cannot even mention a site in a forum or blog without it being no-followed.

    So how do regular web users let search engines know what sites they value? The backlink metric is completely floored and google forces webmasters to use grey hat tactics

css.php