How 404 pages work in Google Toolbar Beta 5

I thought I’d play hooky from a meeting and talk about how the newest version of the Toolbar handles 404 pages for users, because I see some people writing about it this morning.

We tried to give a heads-up in a couple places. The Toolbar beta 5 announcement on the Google blog mentioned “You’ll get suggestions instead of error pages: If you mistype a URL or a page is down, now the Toolbar will give you that familiar “Did you mean” with alternatives, like when you do a Google search.” And the John Mueller did an excellent run-down for webmasters when he talked about the Google toolbar beta on Google’s official webmaster blog. Here’s the part of John’s post that probably interests you:

404 errors with default error pages
When a visitor tries to reach your content with an invalid URL and your server returns a short, default error message (less than 512 bytes), the Toolbar will suggest an alternate URL to the visitor. If this is a general problem in your website, you will see these URLs also listed in the crawl errors section of your Webmaster Tools account.

If you choose to set up a custom error page, make sure it returns result code 404. The content of the 404 page can help your visitors to understand that they tried to reach a missing page and provides suggestions regarding how to find the content they were looking for. When a site displays a custom error page the Toolbar will no longer provide suggestions for that site. You can check the behavior of the Toolbar by visiting an invalid URL on your site with the Google Toolbar installed.

So if you’re a webmaster and want users to see your custom 404 page, just make your page be more than 512 bytes long. I do think that this feature is really handy for most users. Let me give some screenshots to demonstrate what it looks like.

I installed the Toolbar Beta 5 for Internet Explorer and surfed to a 404 page on mattcutts.com, and I see this:

My 404 page is more than 512 bytes

My 404 page, while not that useful, is more than 512 bytes long, so the toolbar doesn’t change the page.

I had to look around a little bit to find a default 404 page. My former grad school has one, so surfing to a 404 page like http://www.cs.unc.edu/~sadasdf normally looks like this (in Firefox):

A default 404 page

With the toolbar installed, I get this page:

The toolbar version of the 404 page

There’s a few things I would point out:

- The first several links all provide ways to navigate or search unc.edu. I’m offered the option to go to www.unc.edu, or www.cs.unc.edu, or to search on www.cs.unc.edu for some words.
- Note that the toolbar took my nonsense phrase “sadasdf” and segmented that phrase into a more useful phrase “sad asdf” to search for. For “mattcutts” it suggested “matt cutts” and for “mygoodpage” it suggested “my good page”. That’s really helpful for a non-savvy user because it offers a search which may uncover the information that the user is looking for.
- There is a “Why am I seeing this page?” link.

If you click on the “Why am I seeing this page?” link, you get a page with more info, including how to turn the feature off:

Instructions to disable the 404 page

I counted and it was three mouse clicks (click on a picture of a wrench, click to uncheck a box, click to save) to turn off the feature. Try to load a non-existent page, and I’m back to the standard 404 page that IE gives:

Instructions to disable the 404 page

So my short summary is:
- If you’re a user and you don’t want help with 404 pages, it’s very easy to turn off just this feature (or don’t install the Google toolbar).
- If you’re a webmaster, customized 404 pages should continue work fine. If you want to be sure that users see your 404 page, make it 512 bytes or longer.

Bonus tip: Most of the people that read my blog use Firefox instead of Internet Explorer. If you want some similar functionality on Firefox, I like to use the ErrorZilla extension. It’s a handy little plug-in that gives you error pages like this:

Example ErrorZilla page

I find the ErrorZilla plug-in really useful, even as a power user.

103 Responses to How 404 pages work in Google Toolbar Beta 5 (Leave a comment)

  1. J. Zygnerski

    So, when will the new Google Toolbar be ready for Firefox users? So many articles are popping up about all of its features that I’m tempted to open IE and check it out.

    Unfortunately, IE will never replace FF as my default browser, so I’m stuck with the regular Google toolbar (the one that people don’t write articles about) for every day use.

  2. I see the usefulness of the feature.

    Some people may view it as a form of hijacking.

    I don’t mind it as all of my sites have custom 404 pages that are above the requirements.

  3. Hi Matt;

    Thanks for writing about this today. Other web sites that wrote about this story today made me think that the google toolbar was “hijacking” custom 404 pages. I see now that isn’t the case.

    Your clarification was very helpful. Thank you!

    Mike

  4. I downloaded the toolbar beta a couple of weeks ago when I got my new laptop and was very disappointed to discover they removed my favorite feature, with no way (that I can figure out at least) to bring it back. It is the “word find” option where the keywords you search for display on the right side of the toolbar and you can then click it to find the words on the page. The new toolbar beta still has it, but has changed it into a drop down menu feature instead, which is kind of a pain when you want to quickly click to find all 8 instances of a certain keyword on a page. It used to be a nice one-click feature as opposed to a multi-click thing it is now.

    I did notice the 404 new addition as well as a couple of other things. But the word find change means I am uninstalling the beta and go back to the oldie but goodie toolbar instead. Was interesting though, I didn’t realize just how much I used that feature until it was changed.

  5. 404 errors have plagued the web since the beginning. Bottom Line: It is a ‘piss poor’ user experience. Whether I”m typing a URL or navigating a page the user experience needs to be improved.

    Thanks you for doing something here. People take a fit because it was DNS that normally handled this. As long as Google provides a good user experience then I think that you’re fine with doing it. Hell it’s better than a 404.

  6. People still use the google toolbar? I am serious, when I started using firefox I ditched the toolbar for the built-in search. Now that IE7 also has the same ability I don’t see the point in this type of toolbar.

    I see the benefits of google, money. I just don’t quite understand the need for the toolbar for an end user. I admit that I am a power user (geek at that) and I understand how all of this stuff works. But this also means that I deal with end users. Once they install a decent browser they almost all no longer look for the google toolbar. At least once the search is pointed out to them.

  7. Think i’m gona stick to the existing toolbar. :-)

  8. You had me nervous until I noticed as long as the custom 404 page “is more than 512 bytes long” it will still appear. Hate to say it but the google 404 version gives more options than my custom 404, time to add a search field.

  9. What does google suggest if you go to unc.edu/expertsexchange? or /~mapornothing? :D

  10. Incredible, Google finally found a way to out-evil Microsoft! Congrats on beating them to the punch.

  11. I find toolbars obnoxious. I’ve irreversibly equated them with spyware/adware, and for the most part this isn’t such a stretch for the Google toolbar since so many freeware apps try to sneak it onto my computer.

    This 404 redirect feature sounds really helpful though. Can’t Google make an official Firefox extension?

  12. Good Day,
    This was all helpful BUT I would like to know does Google penalize a site’s position in search if it has 404 errors not fixed? I ask this because while I am the webmaster and SEO for some sites I have to work with the webmaster for other sites that I only do the SEO and SEM.

    One webmaster I work with continually causes 404 errors because he is too lazy or forgets to do the 301 redirects in the .htaccess file.-despite a zillion e-mail reminders from me. sigh. Well, I am frustrated, I am sure visitors are annoyed which can only lead me to think that google bot’s patience must also be wearing thin! If these unattended regularly occurring 404 errors could cause a negative impact to overall rankings for a site I would like to see it in writing so I can shove it down his pretty little throat in front of the clients. I seem to recall reading that this was the case quite awhile ago but I cannot find the reference now
    I am not publishing the url of the site in question but it enjoys front page top 3 ranking positions for over 30 of the search terms in Mr. Google across several pages. I have worked incredibly hard as have the clients to achieve and maintain this. I don’t want some totally avoidable 404 errors wrecking this.
    Thank you for reading my thoughts and hopefully answering my question.
    Best Regards

  13. Thanks for the detailed clarification, Matt. My 404 page is 299 bytes in size and that toolbar behavior threw me off when I tested this. With your feedback I can at least adjust this now. Thanks.

  14. Ironic indeed. I found this long time back in a web page source regarding a similar practice by Microsoft :)
    —————

  15. —————-
    Unfortunately, Microsoft has added a clever new
    “feature” to Internet Explorer. If the text of
    an error’s message is “too small”, specifically
    less than 512 bytes, Internet Explorer returns
    its own error message. You can turn that off,
    but it’s pretty tricky to find switch called
    “smart error messages”. That means, of course,
    that short error messages are censored by default.
    IIS always returns error messages that are long
    enough to make Internet Explorer happy. The
    workaround is pretty simple: pad the error
    message with a big comment like this to push it
    over the five hundred and twelve bytes minimum.
    Of course, that’s exactly what you’re reading
    right now.
    —————-

  16. Gavin

    I got a good chuckle at all the Google bashing posts about this. Still more to come I suspect.

  17. Feydakin

    Yup, plenty more to come..

    This feels an awful lot like hijacking a website in an effort to provide a “better user experience”..

    It seems to me to be a seriously dangerous path to embark upon.. At what point does a toolbar get to say, the page you requested wasn’t good enough in our judgment, so here’s this one instead..

    Made me think of a sig I saw recently, Don’t be Google.. They came for my 404 and I said nothing.. .. .. ..

  18. Michael D, I would like it if they’d add even more useful options (like ErrorZilla does).

    Zac Garrett, I don’t view this as being about money (there are no ads on that page, for example). What I see over and over with less savvy users is that when they land on a 404 page, they’re flummoxed. You and I have the notion of “okay, I’ll march up the directory tree to find the closest url and then look for a new version of the page” but lots of people don’t.

    Jenstar, I’ll pass on that feedback.

    Nice try, Gab. Here’s what I saw:
    “Search within the site http://www.cs.unc.edu for the terms experts exchange”
    So no expert sex change suggestion. ;)

    “This 404 redirect feature sounds really helpful though. Can’t Google make an official Firefox extension?”

    Brian, there is a Firefox Google Toolbar, but I don’t think it currently includes the functionality from this IE beta.

    Feydakin, I think the fact that users can turn it off (or not install the toolbar) and webmasters can prevent it — but it can still help regular users — is the big litmus test for me. I do think a regular user can benefit from better 404 handling (and that’s why I use ErrorZilla in my Firefox, for example).

  19. Gomer

    Matt, whenever an issue comes up like this, it is really pointless to read what you have to say as you just try and spin it some way to make Google look good (or at least okay).

    Feydakin, Don’t be Google, I like that.

  20. Feydakin

    I appreciate that it can be avoided Matt.. But this is another tool that forces the webmaster to take action to avoid it (create their own 404 page which I really think they should anyway – you should see mine) http://largeorangepop.com/bert-404.php [yes it's a brand new site] or forces the toolbar user to deactivate it.. Both of these action are contrary to the entire “opt-in” mentality of the internet..

    It specifically requires that the user not only “op-out” but know about it it to begin with.. If I did this with email I’d be going to jail..

  21. Keri Morgret

    I agree with @Jenstar regarding the word find. The word find and word highlight options are two of the main reasons I use the Google Toolbar, and I also would not upgrade if the new toolbar did not have those features.

    I would also love to see some upgrades to the word search features — the search technology has changed, but the word search technology has not. If I search for auto-pilot, the first result is spam, then next several have autopilot or auto pilot, and only a couple have auto-pilot. Only the exact match of auto-pilot will trigger the highlighting or word find feature. If I search for a singular and find a plural I’m OK, but if I search for a plural and find a singular form of the word, I also can’t word find/highlight.

  22. Interesting updating with the 404 addons. I’ve already updating my IE, not sure if I have this update or not.. I’ll have to go check it out.

  23. Erik

    Feydakin has the best argument against this intrusion. Specifically, webmasters having to opt-out instead of choosing to opt-in.

    Bad move Google. Not cool.

  24. Scott

    Matt,

    This seems kind of similar to your other post about the ISP adding it’s 2 bits to web pages: http://www.mattcutts.com/blog/confirmed-isp-modifies-google-home-page/

    Do you think there will be a point where Google just adds it’s own info to the top of a page like the ISP, if you have the tool bar installed? I.E. I visit http://netscape.aol.com/ then you find a bit up top saying … you might also like… news.google.com, etc.?

  25. Gomer, if you don’t think I add any value to the conversation, you’re welcome to ignore me. I won’t wrestle you to the ground and force you to read my blog.

    Feydakin, Internet Explorer has used the 512 byte threshold since 1999, so there’s not much new here. If you return a tiny/default/empty page, IE will already display a different version of the 404 page instead of the default page.

    Keri, I also saw another person say that about the find feature (in addition to Jenstar), so I’ll mention it to the toolbar folks. I don’t know what their current thinking is on the word find feature though.

  26. What I like about Matt Cutts is he does not always promote Google products alone. I like the idea of showing possible alternative content in case I visit a 404. But since Matt shows what are the nice things to do. I’d install ErrorZilla and not install Google Toolbar 5. :D

  27. Dave (original)

    While I would prefer to use FireFox over IE7, I find that FireFox is FAR less forgiving on pages with erronous code. I also find it locks up on many vBullentin forums.

    IMO, FireFox has a bit of catching up to do.

  28. Gomer, he won’t wrestly you to the ground. When you get to be head of G’s webspam team, you have internet marketing ninjas to do your dirty work. Keep an eye out for a bearded guy whose last name rhymes with joysin.

    Matt, to cite the brilliant Sean Connery on SNL’s celebrity jeapordy: “You’re sitting on a gold mine!”
    http://www.youtube.com/watch?v=etmHaeaNsCw

  29. (In regard to the expert sex change suggestions ;D.)

  30. I won’t wrestle you to the ground and force you to read my blog.

    You just talked yourself into your next Hallowe’en costume, Matt….LUCHADOR! El Googlero MagNIfico!

    I only have one question on this: how come your screen caps reflect IE6 and not IE7?

    Matt, whenever an issue comes up like this, it is really pointless to read what you have to say as you just try and spin it some way to make Google look good (or at least okay).

    What did you think he was going to say about it? “So yeah, this idea really sucks ass. No one should use the Google Toolbar…ever…for any reason. The department that makes it also uses 12-year-old slave labor to assemble code.”

    As far as his spin goes, there’s a slight, yet fully understandable spin toward Google. Why wouldn’t there be? He likes those semi-monthly or biweekly direct deposits, and probably wants to ensure they keep coming for the foreseeable future. But look at the amount of Google references he makes vs. the amount of non-Google references he makes (as in comparing Google services to other services). He’s actually pretty even-handed about it, and probably even goes a bit too far the other way (e.g. recommending the ultra-crappy Wikia search).

    Matt’s about 129,386,512th on the list of biased people in the SEO community.

  31. Cuttlets (including me) sitting ringside wearing our matching Back Hat’s rule custom Matt Cutts T-shirts.

    it is really pointless to read what you have to say as you just try and spin it some way to make Google look good (or at least okay).

    Cuttlets jump to our feat in horror!

    I won’t wrestle you to the ground and force you to read my blog.

    Cuttletts cheer, jumping up and down throwing popcorn into the ring, with blood thirsty throats, shouting “Gouge his PR, Gouge his PR!!!”
    ;)
    dk

  32. so no plans for Google to put Adwords on the 404 pages, then?

  33. Matt, this is taking things to be a monopoly now on the web which totally contradicts what Google was saying about Microsoft. The fact is that many users will ignore adjusting the toolbar settings which they are craftily forced upon when downloading software with Google toolbar attached.

    I don’t want this as I am confident that my custom 404 page is more highly usable than Google’s. Actually I remember at one SES where you was promoting Apple’s custom 404 page saying it was the best practice.

  34. Nico

    Hi Matt,

    I don’t really understand why Google try to make webmaster’s jobs.

    Manage web publication (and so define the graphical render of an inaccessible document) must be done by the website owner, not by another actor even if that provides a “better user experience”.

    What will Google say/do if we try to modify its search engine to have our own better user experience ?

  35. Harith

    Matt,

    Talking about your former grad school.

    So you werea “Graduate School Honors Fellowship Winner”!

    Honestly Matt, don’t you miss working as a researcher or a teacher in a university following the footsteps fo your Dad?

  36. mike

    Hmm – when do the PPC ads start showing on this magical page?

  37. Matt, you are missing a few points here:-
    You said “So if you’re a webmaster and want users to see your custom 404 page, just make your page be more than 512 bytes long.”
    Now, as a webmaster, it is my right to serve a customized error page as long or as short as I want. Google should not dictate on me, the size of the error page and Google has no right to modify a custom error page served by me.

    Also, the way it is presently working is a opt-out fashion. If the webmasters don’t want it for their site, serve a big page.

    I would have respected this feature had this been an opt-in feature, where the webmasters specifically give permission for Google Toolbar to display such a page.

    We all know how much all the opt-out spam sucks. I’d say, this is one additional things many webmasters will have to opt-out for each of their sites. This isn’t good. If at all, it should have been an opt-in feature.

  38. There may not be any ads on the page BUT this is still about money! This is about seeing the Google logo as many times as possible so that it would retain on the users’ mindset, that it’s Google here, and Google everywhere. It’s about money…

  39. Matt, thanks for mentioning the Firefox Google toolbar, but it’s more the concept (and dirty history) of toolbars that irks me than the functionality it provides. 9 times out of 10 my first destination after getting a 404 is google.ca so I would genuinely be interested in this functionality, but only if it could be untethered from a bulky toolbar.

  40. Rico

    Hello,

    Some questions :

    - What will hapen if Yahoo / Live or any other search engine actor develop the same feature, if i use more than one search engine toolbar?

    - Will it be possible, for webmasters, to not allow google to do this ? will we be obliged to put a personnal google tag to not allow it on our website ?

    - What will happen if i have a 404 error on an intranet page / private access page / robot exclusion directory ? will google overstep my server configuration to suggest me a new search on its search engine ?

    I don’t really like this new feature, because i think your job is to deliver relevant results to users, and the only thing you have to take care about 404 pages, is to maintain your index as fresh as possible to not deliver links towards non-existent documents, in order to deliver a better user experience… but you should not take the wheels on server configurations to try deliver it better.

    As french people say : “Maurice, tu pousses le bouchon un peu trop loin!” ;)

    Thank you

  41. Don’t see why so many people are complaining about it… Most designers / developers should have custom 404 pages up, thus bypassing the smaller file size limit

  42. I’m kind of liking some of the new features like fixing misspelled URL’s. I do it often :)

  43. I hope that the toolbar doesn’t intercept 404 errors sent in response to XmlHttpRequests?

    RESTful interfaces often return 404 to javascript AJAX frameworks which are then handled by the javascript, not a custom error page.

    What happens if the missing URL was loaded in an iframe? Does the google page get squashed into the frame or does it take over the whole window? Either way, it’s not too useful for web surfers.

    I should do some experiments, but that would mean using IE. Bleurgh.

  44. Yes, Microsoft has been doing this for ages (did they ask you to opt in?) and as a developer, if you are not already aware of the 512 byte issue, then the majority of your users may have not been seeing your 404 message for a long time. At least Google’s implementation is more useful and follows the same precedent. Sounds fine to me.

  45. Hey Matt, while Google is now doing the 404 project, could you poke the webteam for the Google.com site for me?

    Although it is rare to get a 404 on Google itself, when you do you see this:
    http://www.google.com/404

    Hardly helpful is it? No link to Google main page,
    no search or link suggestions. Or a Report Broken Link button.

    So it would be nice if Google stood out as a good 404 page rolemodel as well :P

  46. “Feydakin, Internet Explorer has used the 512 byte threshold since 1999, so there’s not much new here.”

    Matt I know my mom never let me get away with “all the other kids are doing it” line and I bet yours didn’t either so don’t think it’s going to fly now.

  47. Dan

    Bu default, are the clicks of the hyperlinks provided in the google toolbar 404 page tracked by Google? I certainly hope not.

  48. “Matt I know my mom never let me get away with “all the other kids are doing it” line and I bet yours didn’t either so don’t think it’s going to fly now.”

    LOL.. But seriously, for most folks outside of “this ring” it is helpful.

  49. Hi matt,

    Is it not like Hostile Take over of 404 error page. I am not getting the exact concept behind this because you say, there is no money (no ads u mean). In some of google’s netowrk (specially orkut ) there was no ads first, but now there is ads inserted.

    What if Google automatically installs Google Ads (adwords) into the 404 page. This will bring more money to google but a less amount for webmasters.

    Is tehre is a gaurentee for that not going to happen

  50. Dave (original)

    Forget your medicaton again, Keniki?

  51. Why don’t you add in the Google Cache & Wayback links to the toolbar?

    Here’s a suggestion. I was just discussing yesterday with one of the big genealogy websites how they might handle their load balancing and the feasibility of utilising Googles own cache to serve pages. A feature of the toolbar could be that if a 404 was found and you had a cache of the prior good page, the toolbar could be set to reroute the user to the cache. Alternatively you could actually have an enterprise product whereby high traffic sites could opt in via Webmaster Central to have their first clicks default direct to the cache to keep the load on their servers down, obviously you guys have infrastructure most webmaster could only dream of :-)

  52. I definitely think this is best left to a webmaster on the intended web site.

    I would argue that ‘Normal’ end users don’t know what a 404 page is and therefore will not turn the feature off meaning those times a missing character in a url etc will mean you as a web site are at the mercy of Google’s cache of you site rather than your custom 404 page. This is very bad in my opinion as I know the only times I see 404 hits are when I have rushed a new page and therefore the fact this content is new it will not be in Google’s cache resutling in Google toolbar users never finding my intended page.

    If the visitor has hit my catch all 404 page with a search box linked to MY own database the would find this new page within a few seconds.

  53. I am very dissapointed with this. This means that if I don’t have a custom 404 page on all my sites and I have a broken link, the google search box provides an escape clause to search which will might go to competitors sites…

  54. I agree on the word find feature – I use it a lot and don’t like the sound of the dropdown.

  55. Hello Matt,

    I do think it is a good new function, but I use FF, so it’s no good for me right now.
    Sorry for raising this question here (but as you’re an open-minded person I do hope you doen’t mind), but I’ve found that http://www.google.com/support/webmasters/bin/answer.py?answer=35769&topic=8521&hl=en states in “Design and content guidelines” that “The Google crawler doesn’t recognize text contained in images.” but I’ve heard he does it now. Are the guidlines up to date? I do hope they are. I read them like a mantra twice per month.

    Regards,
    David

  56. Feydakin

    Graywolf, I agree, besides, using the excuse, “Microsoft does it” doesn’t exactly inspire a warm fuzzy feeling inside..

  57. JM

    Matt, seriously:
    Google Don’t have any rights to modify the result of sites.
    Now is the 404 error, tomorrow the sidebar to include “similar sites”
    and next include ads in all sites that you visit.

  58. dan

    Another alternative for typos in domains is to use OpenDNS. I’ve been using it for over a year and am pleased with the reduction in DNS lookup times too.

    Of course it doesn’t help with 404 errors…

  59. Oh noz, if I don’t want Google to ‘hijack’ my 404 I have to create a page longer than 512 bytes, holy crap! It is the end of teh worldz. Lets all go rob the nearest Best Buy and grab some TVs while we can!

  60. I ran into that dang 404 replacement screen a day or two ago. The first thing I did was to find out how to turn it OFF!

    A toolbar should be just that. A toolbar. Don’t be messing with how my browser is supposed to operate. Just give me useful tools that I don’t have built into my browser and all will be good.

    BTW, I have never liked Firefox or its predecessors.

  61. Ooh, I’ve been spending all day taking care of 404′s as I just switched all my pages from a subdomain to the root domain (and hoping this helps my site overall SEO-wise). Right now I’m redirecting all old pages using 301 method using SiteDirector.

  62. JLH

    Matt said, “if you don’t think I add any value to the conversation”

    I think you do, if that matters.

  63. It’d be nice if turning off Google’s error pages and disabling browse by name were two different options. I don’t want to see Google’s error pages instead, but I want to be able to type in (for example) “microsoft” into my address bar and be sent to microsoft.com, instead of the Google search page…

  64. Dave (original)

    Matt, I must say If find it highly contrary for you to call out a certain ISP for appending/splicing into Google pages, yet Google is now *replacing* custom 404 pages from *other* sites with what IT THINKS is best and you are condoning it.

    Is Google finally turning into yet another BIG company where the left hand rarely knows what the right hand is doing?

  65. Dave (original)

    So if you’re a webmaster and want users to see your custom 404 page, just make your page be more than 512 bytes long

    That’s a rather arrogant thought process on behalf of Google. That is, why would ANYONE set-up a CUSTOM error page IF they didn’t want THEIR visitors to see it?

    I believe the right (do no evil) thing to do is, IF a site has a custom error page set-up, do NOT overide it.

  66. My God…I can’t believe some of the answers on here. A custom 404 page is not that difficult to set up, and at 512 bytes, it’s not like you have to put that much into it anyway. Hell, you could throw up a blank page with a long title and say “404 Not Found” and it would suffice. 512 bytes is half a kilobyte. People complaining about this are telling me they can’t set up a custom 404 that spits out half a kilobyte output?!?!?!? Get freakin’ REAL already.

    Besides, if done right, a custom 404 can be user-friendly. You can have a page that matches the look and feel of the rest of the site, redirect users if they misspelled things, etc.

    The one thing I would change on this is the search function. If the search function only searched the affected site and didn’t include Adwords anywhere, then Google at least provides a way for users to find alternatives within that site and the customer doesn’t lose to competitors.

    That way, those who cannot or do not wish to install a custom 404 don’t get negatively affected, and may see a positive result from it. Webmasters who don’t get it still have something to complain about (as we all know, complaining about piddling crap is the lifeblood of the web design and SEO industries). Users still get the chance to find what they want without seeing an ugly, demi-cryptic 404 error page. And Google gains some more market share. Everyone wins.

  67. Dave (original)

    MWA, so you are making suggestions about this, so why can’t anyone else? Oh, I get it, IF you don’t agree they aren’t “freakin’ REAL”.

    Seems to me your post is more of “complaining about piddling crap” than anyone else’s.

    Perhaps take some of Keniki’s medication to calm yourself :)

  68. Hiya guys. Agreed that a toolbar should be just that, a toolbar. Been faithfully running firefox for the past few years and it generally has been a huge improvement over IE. Will definitely not load IE7 on my machine, nor the new Google toolbar. Thanks for the errorzilla exttention tip, am going to load it straightaway.

    One question Matt, surely the feedback you get from your blog gets integrated into identifying trends / patterns / possibilities?

  69. Dave, I’m quite calm. The only reason you reacted is because you did what most people did and misread the post completely.

    Matt never said that any custom 404 page would get overwritten. He said anything over 512 bytes would. Take a blank HTML page with “Untitled” as the title. That’s 189 bytes. No copy, no formatting, no nothing. 189. So you’re already a quarter of the way there, and you haven’t done anything yet.

    Even if you don’t add any additional code to format the page to make it look like the rest of the site, as any good custom 404 page should do, you should be able to write enough of an explanation of the 404 to generate the extra 389 bytes. With no additional HTML tags whatsoever, that would be 65 words. Do you think you can put 65 words on a page? Does that seem like such a daunting task?

    Now…what isn’t clear in this is whether or not the 512 bytes includes the code on the HTML page itself, or any other supporting materials such as CSS, Javascript, or images. If the latter are included, a 512-byte page could be accomplished without any difficulty whatsoever.

    So…what does all of this mean? The only people affected by this aren’t the ones who have a custom 404 page (except for a very select few isolated cases at BEST…and we’re talking 0.000000001%), but the ones that don’t have a custom 404 page.

    This is why I suggested(not complained, like most of the rest of the vested-interest self-absorbed politically minded crowd around here) that any searches performed be performed on the domain and include no advertising. There’s no competitor issues, users still have a chance at being able to find something they want, and webmasters would actually benefit because Google would send some of the lost traffic caused by the lack of implementation of a custom 404 back to the webmaster’s site.

    Dave: I never said anyone couldn’t make suggestions. I also never said anyone couldn’t make a complaint. There’s a difference between a suggestion and a complaint, though. A complaint is nothing more than the gainsaying of an idea without any suggestions for an alternative, which if you look closely almost no one (including you) has given. You just flew off the handle because you misread something and didn’t like the net result. You can say whatever you want, but at least base it on fact and not the twisted interpretation of it that everyone else has. This is what screwed up SEO as an industry in the first place, and all you’re doing is contributing to the mess.

    To the rest of you who misread the post, go back and reread it and keep the 512-byte number in mind. That’s the key part here.

  70. The first time Google’s 404 page appeared I thought my server was down. For non techies they might think the web site they are looking no longer exists.

    Many times Google’s 404 page suggestions point to other sites. This is most probably the case for sites that are new that have not yet been indexed or still are still sandboxed.

  71. Matt–I’m all for it. Makes logical sense. It’s user friendly and it also is Google friendly. When I first started reading, the underlying thought was about custom 404 pages (of course), but when I saw that ultimately Google was just trying to provide a more friendly surfing environment (who likes the default 404 pages anyway?), I was quickly sold. Great choice … and smart, too.

  72. Harith

    Dave (original)

    “Perhaps take some of Keniki’s medication to calm yourself”

    Pls. don’t talk to our kind Canadian friend MWA like that. In fact MWA is a senior commenter on Matt’s blog. As such…RESPEK ;)

  73. I totally agree with Jayant Gandhi what u say matt?

    Damn you Google! Who are you to tell me what should be the size of my custom error page if at all I show one? It is my right to show a error page which is less than 512 bytes. Your toolbar manipulating that error page and showing something else that favors your search box is simply not acceptable.

    http://www.jkg.in/147-google-goes-evil.htm

  74. Wait a minute…when did I ever become kind? I’m kind of an ass, but that’s as close as I get. ;)

    Respek back, Harith.

  75. Dave (original)

    Pls. don’t talk to our kind Canadian friend MWA like that. In fact MWA is a senior commenter on Matt’s blog. As such…RESPEK

    I always show “RESPEK” for those who show “RESPEK” for others and thus qualify. In other words, if you cannot take it, don’t dish it out as I do not suffer a fool :)

    LOL! @ Keniki, grow up and take your medication.

  76. I think you actually have to do it, rather than thinking up the idea.

  77. Hmmm

    While I see how this can be spun to seem helpful, any time a website is hijacked for profit by another site (you won’t be sharing any AdSense for those searches performed on 404′s), it’s very much not cool.

    Google is following RODGER’S internet here, and hijacking others’ pages for personal gain. Depending on one’s love, or not, for Google, or one’s comfort with other’s dictating what one does on one’s own site (such as MWA, and a handful of others above) one may actually swallow the line that “It’s for your own good.” After all, there are plenty of suckers out there — we elected Bush twice for goodness sake.

    But really…. Come on….

    You seem a decent guy Matt, and 90+% of the time I agree with Google’s moves.
    But this isn’t one of those times…

    It’s very much a double-standard forming:
    1) Google accuses Rodgers of wrongdoing for hijacking it’s pages
    2) A few weeks later, Google starts doing the same thing to everyone else

    Nice one.

  78. Dave (original)

    Matt never said that any custom 404 page would get overwritten. He said anything over 512 bytes would.

    Yes I know and said as much, so did most others. It should be ANY size custom 404 error page though. Some Webmasters might just show a relative img tag that KEEPS the custom 404 error page below 512 bytes, or whatever.

    So…what does all of this mean? The only people affected by this aren’t the ones who have a custom 404 page (except for a very select few isolated cases at BEST…and we’re talking 0.000000001%), but the ones that don’t have a custom 404 page.

    No idea where you pulled that number from, but I can have a good idea :) Anyway, it should be nil, zip, zero, nada, nothing.

    Matt called out a certain ISP a while back for splicing into Google’s home page. It is NOT the fact of how much % they are splicing, it’s the fact they they ARE doing so. Google’s homepage is 0.000000001% of the Web yet here Matt is condoning Google for doing pretty much the same thing.

    It’s hypocrisy at *best*.

  79. Dave (original)

    Take a blank HTML page with “Untitled” as the title. That’s 189 bytes. No copy, no formatting, no nothing. 189. So you’re already a quarter of the way there, and you haven’t done anything yet.

    IF the bytes of a blank page is 189 bytes, don’t you think Google should be using 190 bytes rather than 512 bytes? Or, check for ANY text/code at all.

    Seems like a no-brainer to me.

    (not complained, like most of the rest of the vested-interest self-absorbed politically minded crowd around here)

    That IS “complaining”.

    BTW. You were “complaining” about others “complaining” before that and were quite abrasive in your manner. I.e

    My God…….Ger freakin’ REAL……complaining about piddling crap….Webmasters who don’t get it still have something to complain about (as we all know, complaining about piddling crap is the lifeblood of the web design and SEO industries).

  80. Hey Matt

    I’m not sure I like the simple 512 byte limit. I don’t think Google has the balance between Webmaster and User quite right there. E.g. Google would over-ride this simple 404 handler (tags changed to allow viewing here):

    {HTML}
    {HEAD}
    {TITLE}Not Found{/TITLE}
    {META HTTP-EQUIV=”refresh” content=”3;URL=http://www.mysite.com/search.php”}
    {/HEAD}
    {BODY}
    Sorry, that page was not found. Please wait and we’ll redirect you to our site search engine.
    {/BODY}
    {/HTML}

    Having said that, I can see where Google would have every right to insert its own 404 handler – and that would be when the referrer to the 404 page was a Google SERP. ;)

    In that case Google could do one or more of any number of things:

    1) What it does now – a simple 404 handler
    2) Auto-remove the URL from the Google index
    3) A more complex 404 handler. e.g. redirect the searcher back to their search results, with the 404 page missing/greyed out and some text explaining what happened.
    4) An extra option: enable/disable/enable for SERP errors only

  81. Nick

    Matt,

    I ran across your blog, and since you seem to read and respond to comments, wanted to share some suggestions for the Google Toolbar. I participated in a discussion some time ago when the then-new v4 Toolbar was released via auto-update.

    http://groups.google.com/group/IEToolbar-Group-GettingStarted/browse_thread/thread/2896c784af85fb5c

    Absolutely _NONE_ of the issues brought up here were resolved in any fashion. I still cannot stand the newer versions of the toolbar. I continue to use v3 via filesystem permission changes and have no plans on changing. The new drop-down search history box is a UI joke and the simplistic, compact layout which I’ve always relied on Google for was killed. The two biggest features that I use the toolbar for (which are not present in the built-in search functionality of IE7) is spell checking and the “find next occurrence” feature. If these go away what’s the point?

    I realize this is a rant and that it’s likely too late to change anything, but I hope Google knows they pissed off and lost a lot of users over the ridiculous UI changes. With more changes like the one you describe in this entry you’ll probably lose even more. It wouldn’t be so bad if the hijacked 404 page contained the original content as well as Google’s additions. As others have pointed out, just because a message is less than 512 bytes doesn’t mean it isn’t important.

    Verisign has tried (and tries) to pull stunts like this. Verisign is pure evil. Don’t be evil.

  82. Dave (original)

    Thereyago, Alan. I knew sooner or later we would agree on something :)

  83. Hi Dave

    So you agree it would be OK for the Google Toolbar to intercept a 404 on any site to which it sent a visitor, and issue a redirect back to the Google SERP which referred the visitor to the 404 page? (Option three above)

    I think on the whole that would be a good thing for Google to do, but it may court a fair bit of controversy. In this case, though, I think Google would hold the high ground since they were providing the visitor and, really, 404 pages should not be in their index.

  84. Dave, I have absolutely no clue how you managed to turn my social commentary on the complaints of others into me complaining. Talk about your great quantum leaps. I wasn’t complaining…I was making fun of the dopes who didn’t read things properly. That’s all that it was. When I complain, you’ll know. In the meantime, Johnny can’t read.

    Alan, your example is a bad one for four reasons:

    1) Meta refresh can be disabled within a browser, thus making your method useless to those users.

    2) Anyone who knows PHP enough to build a site search engine should also know enough to either code a 301 redirect to the site search engine or return a custom 404 that matched the look and feel of the rest of the site. This saves 3 seconds of users’ time, and since every nanosecond seems to count these days, that’s huge.

    3) Even if the meta refresh wasn’t included, and even if the PHP script wasn’t written by the developer, the developer could at least code a form that sent parameters to the search bar.

    4) The HTML code is invalid.

    In that case, Google’s actually doing the webmaster a favor by giving the users some way to search directly as opposed to either having to wait or not be able to at all.

    Dave: IIS’s default 404 page is larger than 190 bytes. I don’t know exactly how large, but it’s likely one of the reasons for the 512-byte limit.

  85. wheel

    Google’s not running ads on 404 pages on your site? Really?

    Then I guess Matt Cutts won’t mind if I slap a permanent vinyl sticky on the back window of his car window that shows my company logo then. Because that’s not an ad right?

    Google slapping their logo on a page of my website, any page, is deplorable. If visitors want a Google logo or any Google page, they have a website for it. It’s called Google, and it’s not my website.

    And claiming an opt out feature that really won’t be found by the average user is the same technique and the same morals as scumware and adware.

    But I’m sure they’ll just stick to displaying the logo on your site as their method of advertising, right? Don’t count on it. We’ve already seen nofollow feature-creap.

    I honestly can’t believe that Google is standing here trying to justify displaying their logo when a visitor comes to my site, on *any* page.

  86. Dave (original)

    Dave, I have absolutely no clue how you managed to turn my social commentary on the complaints of others into me complaining.

    Semantics. Let me give you a hint, there was NOT turning, only quoting. But don’t worry, you just don’t ‘get it’ going by your reply to Alan and others.

    Alan, I agree with you that Google should NOT replace *another* sites custom 404 page (no matter what size) with theirs. In light of what Matt stated on the ISP (Rogers) splicing into Google’s page, I think is very hypocritical for Matt to now praise/condone a very similar action by Google.

  87. Semantics. Let me give you a hint, there was NOT turning, only quoting.

    No, “semantics” is your stock answer that you give when someone explains to you why whatever bonehead thing that comes out of your mouth is stupid. It’s also the reason why you got pissy over at IHY after Bill made you his personal bitch and took what passes for your ball. But if you want to throw that stock phrase out, go ahead.

    You quoted me completely out of context to suit your own needs, but just so that it’s abundantly clear even to someone as towheaded as you, let me state it bluntly: I was not complaining about webmasters and SEOs making idiotic and asinine remarks about this policy. To be perfectly honest, I find it and found it amusing, much as I do every single time this happens in the web design industry, and to take it a step further the IT industry. Large company announces something relatively trivial that affects a small portion of people; geek community misreads, distorts, and blows it way out of proportion; comments that could only come from people making tangental leaps best reserved for the programming on The Space Channel ensue. Normally, you’re on the other side of the equation but for some reason you’ve decided to participate in the idiot charade for reasons known only to you.

    In other words, not complaining, commenting on something I find funny. I’m hoping Google does more things like this at least every 3-4 months just so webmasters can bitch, moan and complain like whining schoolchildren because they didn’t see it coming. Bring on the stupidity!

    That’s not “semantics”, Dave. Like I said before, and like I’ll say again, Johnny…can’t…read. Don’t twist my words around until you understand the full meaning behind them (in this case, you don’t).

    But don’t worry, you just don’t ‘get it’ going by your reply to Alan and others.

    No, I disagree with the bizarre and illogical ranting that most people have used here. The only one who has brought ANYTHING to the table to support the point of view was Alan, and as stated before, his example was a badly coded piece of work that Google would actually be doing any webmaster that used it a favor by summarily replacing it.

    The real hypocrisy in the equation here is in the stance that everyone else seems to be taking. Google has put something in place that primarily benefits the small-scale webmaster, the one who may not even know what a custom 404 is, never mind how to implement one and be able to provide a search alternative. It also positively impacts webmasters who, for whatever reason, cannot implement a custom 404 (e.g. those on free hosts or on low-budget paid hosts). I believe they call this person “the little guy”…you know, the one big corporations screw over all the time. This actually helps those people. Anyone who can sit here and say with a straight face that the number of people with a custom 404 that is less than 512 bytes (assuming there is anyone in that class of people that has a 404 page that is in any way useful and less than 512 bytes in the first place) is greater than the number of webmasters who cannot implement a custom 404 in the first place needs to put the crash helmet on before they suffer any more brain damage.

    Feel free to reply, Dave. Hurry up though…I’m making some popcorn and whatever entertainment you’re going to provide next would really go well with it.

  88. Dave (original)

    …I was not complaining..

    Matter of opinion but when you accuse others of complaining and promptly complain, moan & bitch about it in subsequent posts, you become a hypocrite.

    MWA, sorry, that as far as I got as I’m not the slightest bit interested in your long unread rant. stick with your domain parking complaint, at least that makes some sense, although you offer NO solutions other that “make a rule” LOL!

    BTW, getting personal and name calling is not very becoming for you. Also, you don’t have not the slightest idea why I don’t post at IHY anymore and your precious “incrediBills” words to *ME* (hint) have zip to do with it. But, keep guessing as you always do :)

  89. Let’s see…you disagreed with Bill, he laid the smackdown on you as only he can, you ran away like a child who wasn’t smart enough to argue back. No amount of “semantics” on your part, as you would put it, will change that. Billy ran you out of town. I never took sides on that issue, either, but I’m starting to see his point.

    Again, Dave…I wasn’t complaining. Like I said, I find it amusing as I always do…although eventually, as I’m getting with you, I get bored with it. Call me what you want…your opinion doesn’t change the reality of the situation. All it does is clouds it a bit.

    The reason you can’t be bothered to read what I said is because you know it’s right. You can’t handle logic when it disagrees with whatever opinion you have, so you throw out stock answers. It is what it is.

  90. Dave (original)

    LOL! You’re still complaining and get quite worked up over it.

    MWA, again your name calling and childish rhetoric are not becoming of you and make you look as foolish as your guess work on why I no longer post to IHY. Incredibill is only a legend in your mind and his.

    The reason you can’t be bothered to read what I said is because you know it’s right

    LOL! Perhaps (highly unlikely though) you are “right”, I’ll never know though as I wont read it :)

  91. Sounds like a pretty handy add on. Thanks for the heads up and info. I’ll have to check it out on IE. It might not be a bad idea to incorporate the site links you guys develop for a site into this handler (kinda seems like that what it’s doing now?).

    Wow – I can’t believe how many people talk trash to you for no reason on your site!

  92. When I first heard about it, I thought it would be a bit invasive (i.e. -redirecting users to a unique type of search results page)

    This sounds like a good innovation though.

    Does the Toolbar send 404 data back to Google to help with indexing? Does it send page visit data to Google to help with indexing too?

  93. As a webmaster I’m not too concerned since my website is well-designed and user hardly ever hits 404. Although it doesn’t protect from the case when user makes a typo. I guess I’ll just have to 301 redirect all typos to an error page that doesn’t return 404.

    However as a user I’m royally pissed off because I can’t turn this functionality off and I don’t want it. The only reason I have toolbar installed are PR and favorites.

    This just puts Google in the same bunch as Zango and other internet page-hijacking scum… They must be really distressed over their stock price to try things like this…

  94. Bu default, are the clicks of the hyperlinks provided in the google toolbar 404 page tracked by Google? I certainly hope not.

  95. That 404 looks so much better with the toolbar, that’s something i had no idea of! Great Stuff.

  96. Очень нужная вещь.. эта страница 404

  97. The first time Google’s 404 page appeared I thought my server was down. For non techies they might think the web site they are looking no longer exists.

  98. Since the original error page is so short (less than 512 bytes), you should display that error message AND your suggestions, in case it has important information in it. My preference would be the original error from the server first, then your suggestions, or you could do the opposite. But to totally hijack the 404 is not right. At the very least display the original 404 error, then add your suggestions.

  99. I do think it is a good new function, but I use FF, so it’s no good for me right now.
    Sorry for raising this question here (but as you’re an open-minded person I do hope you doen’t mind)

  100. Bu default, are the clicks of the hyperlinks provided in the google toolbar 404 page tracked by Google? I certainly hope not.

  101. The first time Google’s 404 page appeared I thought my server was down. For non techies they might think the web site they are looking no longer exists.

  102. Im for sure going to use the same toolbar and not change. Thanks

  103. Hi,

    I read all above comments but not found answer…I am doing SEO for a website and handling Webmaster tool. I have received many crawl errors like ‘Not found’ ‘Soft 404s’ , ‘Restricted by Robots.txt’ & ‘Unreachable’ in daily basis.

    As per above discussions I have custom 404 error page and it looks like my website & I understand any searches performed be performed on the domain. There’s no competitor issues, users still have a chance at being able to find something they want.

    Few Questions:
    1. Does it impact on search engine rankings?
    2. How can we resolve crawl errors on Webmaster Tools?
    3. I want to know what does the crawl errors is it just inform us for our site errors or we should resolve this?

    Please help me. I am very eager to know all about Crawl Errors.

Leave a Comment

Your email address will not be published. Required fields are marked *

*

If you have a question about your site specifically or a general question about search, your best bet is to post in our Webmaster Help Forum linked from http://google.com/webmasters

If you comment, please use your personal name, not your business name. Business names can sound salesy or spammy, and I would like to try people leaving their actual name instead.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

css.php