Free links to your site

I can’t believe a new feature from Google isn’t getting more notice, because it converts already-existing links to your site into much higher quality links, for free. The Google webmaster blog just announced that you can find the pages that link to 404 pages on your site.

Let me back up and give you a little history. When someone comes to your site’s webserver and asks for a page that doesn’t exist, like http://www.mattcutts.com/asdfasdfasdf , most web servers are configured to return an HTTP status code of 404, which means that the page was “Not Found.” If someone links to a page on your site that doesn’t exist, most webservers give a pretty sucky experience: visitors usually land on a pretty useless page, and search engines might not give you full credit for those 404 errors.

Now Google’s webmaster portal lets you see who is linking to your 404 pages. Once you register your site, click on Diagnostics, then Web crawl, and select “Not found”. You’ll see something like this:

My 404 errors

See that red oval? If I click on it, I’ll see all the urls that link to that 404 page:

My 404 errors

Why would you care about this? The simple reason is that if someone is linking to a non-existent page on your site, it can be a bad experience for users (not to mention that you might not be getting credit for that link with search engines unless you’re doing extra work). Some of the easiest links you’ll ever get are when people tried to link to you and just messed up.

It would be trivial to mail some of these people and say “Hey, I noticed you linked to my site (thank you!) but the link is broken, so users will get a 404 page. Would you mind changing your link on page A to point to the right page, which is url B?” When the other site fixes their link, their visitors find your site directly, plus all search engines can follow those links and give you credit for them. Converting 404 links to links to the right pages converts sucky links to free direct text links for all.

You can also download this data as a convenient CSV file. Just click on Diagnostics, then Web crawl, and then click on “Download all sources of errors on this site.”

P.S. Here’s a bonus tip: if your webserver does cruddy things on 404 pages, Google’s webmaster portal will help you create better 404 pages with about 14 lines of JavaScript that you can copy/paste into your 404 page template. Because it’s just some code to add to the HTML template for your 404 page, it works for Apache or IIS.

163 Responses to Free links to your site (Leave a comment)

  1. Would setting up 301 redirects to take users who click on bad links to the right page achieve the same (or similar) effect?

  2. An excellent tool – does make life easier. However, does it catch ALL 404 link errors? Its just that Google isnt great at revealing all backlinks…

  3. Can we modify the code so it will looks more nice ?

    And how we can modify so it will not hurt Google and we can see the same error results in webmaster central ?

    Thank you.

  4. Matt, if I redirect all 404 to the homepage, will I get penalized for this?

  5. This additional information in web master tools is excellent – now its much easier to resolve the problem of “missing pages”.

    I have all my clients read a timeless article from the folks at A List Apart: “The Perfect 404″
    http://www.alistapart.com/articles/perfect404/

  6. patrick

    I don’t know about other people, but I was pretty damn excited about it! Unfortunately, I haven’t created a blog to write about it yet, ha. I still need to look into it more, but it’s definitely a welcome addition to webmaster tools.

    One question Matt, and maybe I missed this, but under the “Linked From” column where it lists the # of pages, what exactly does it mean when it says “unavailable” as it does in your screenshot?

  7. Like the tool, but wouldn’t it be easier to simply redirect all non assigned urls to the home page?

  8. Dave (original)

    Matt, IF these dead links are turned into “much higher quality links”, wont it skew Google SERPS towards the select few who know about it? I would guess 90% of site owners would NEVER know about this.

    Also, what is point of a custom 404 page with PageRank?

  9. DR

    Hi Matt, I’ve heard two sides to this story. Some people in the industry promote allowing the search engines to find 404 and kick the pages out of the index and I’ve heard others say that you should 301 each page – even if there’s not an updated page to 301 it too and just 301 it to the root or the root category page of the original.

    My question is how much will 404 hurt you in the serps – reason I’m asking is I have a site that literal dropped almost all rankings the other day which were anywhere from the first page of Google to the 4th page and the only thing I could think of is in webmaster portal it should I had 404 pages from a site we 301 to our site.

    All comments welcome and appreciated!

  10. Tom

    Nice work Google – we built a tool in-house at Distilled to do this a while back and have been using it for great success. Nice to see Google following suit :-)

  11. rishil, it’s the vast majority of the backlinks you’d care about, at least as far as Google is concerned.

    Brent Hodgson, I purposefully left out the “301-to-the-home-page” issue from my post, because there are pros and cons. The pro is that you don’t have to ask other people to fix their links (and as long as you’re doing this in a normal way, you shouldn’t run into problems with Google), but the downside is that it’s a really weird user experience for your visitors.

    patrick, I’ll have to ask. My guess would be either really new or really old links that we saw.

  12. @ some wondering if they could turn back broken links to their homepage: a404 custom page helps users be aware of what actions they just take on our site. If you directly take them to your homepage, they may feel confused, keep asking why. And you may lose most potential users!

    Also a good 404 page (which keeps the same template with other pages, and lists some important pages + search box) with a 301 redirect (that Google loves) helps both User Experience and SE crawling, thus ranking.

  13. Thanks for covering this Matt, I’m off to give it a hammering now and see what it thinks of my sites.

  14. Hey Matt – wonderful news. Given appropriate link love on SPN: http://tinyurl.com/4jbj6a

    cheers
    Kal

  15. The lists of backlinks are hard to go through one-by-one, as whenever you close the “pages” popup, you’ll end up at the top of the page.

    Now, if only you could sort those links by PageRank of linking site, that would save some time in finding only the important false links. Not even in terms of SEO (though as a consequence that too), but just because it seems unrealistic that you’d try contacting every one in the list.

    On another note, some of these functionalities that are currently split up between Google Analytics and Webmaster Tools might well fit in either tool, from a user perspective.

  16. Philipp Lenssen, download all the 404 pages at once:

    You can also download this data as a convenient CSV file. Just click on Diagnostics, then Web crawl, and then click on “Download all sources of errors on this site.”

    That’s much easier to scan to find links that you want to focus on.

  17. Wow, I was just poking around the webmaster tools just now and must have totally missed this. Thanks Matt.

  18. g1smd

    As for the “redirect all errors to the homepage” thoughts.

    Just *don’t* do it.

    You need URLs that are completely wrong to return a 404 status code in the HTTP header.

    Every webserver makes it possible to have a custom 404 page. You can make that page look a little bit like the home page, but you can also customise it and present information related to the search the person made. You need to say “page not found” or “item not found” or similar so the user gets a sense of what is going on. Make sure the Error page is at least 512 bytes long otherwise users of IE get an invitation to use Microsoft’s search site instead.

    Finally, what do you think your average search engine bot might think of a site with infinite name space – nearly all of which sends a redirect out. I would think that it isn’t much different to a site that is a bot trap with infinite URLs in an unbounded calendaring system or that encountered on a site using session IDs in URLs.

    What about when bots ask for your robots.txt file – and you return your home page for that request? Ya think some might be confused? It isn’t worth the risk to meddle with this stuff.

    So, I would redirect only “very close” URL errors to the correct page, and send a 404 for stuff that is just plain wrong.

    Google does test your 404 response. You’ll see several accesses to a “noexist” URL per month. I assume their bot makes many decisions based on the result of that request. It is wise not to tamper with it or go against the HTTP specs.

  19. Is it better to use 301 redirect to internal search or to use custom 404 (410) page (example of such page is here http://zvirata.bazos.cz/inzerat/249184/Francouzsky-buldocek.php)? For user it is nearly same, because on internal search after redirect I can write message that ad was deleted (but what will happen with Pagerank?). I have a lot of links to 410 pages (ads), it’s great tool to find them in webmaster tools, but in this case it doesn’t help.

  20. RogerW

    @ patrick & Matt Cutts: when it says “unavailable”, it can also be an error in the Sitemap uou submitted to Google, rather than an external link. It would be nice if it said so though :-)

  21. This is great, I’ve been waiting on this feature for quite some time.

    Unfortunately though when I download the report it doesn’t include this useful information, so it’s much harder to spot patterns and figure things out when there are a lot of 404′s as I have to step through each line on the page and make notes.

    Another useful feature would be the ability to see separate lists of the internal and external broken links.

    Overall though, definitely a step in the right direction.

  22. Maybe Google could use this to fix their own 404 that’s ranked very well for a while now. Search for “google earth” and notice result ~#5:

    http://www.googleearth.com/?GXHC_gx_session_id_=7936eae2f3e09120&

    I’d love to see that get resolved.

    That being said, this will be a great tool for my sites.

  23. Clive

    Matt

    Great feature.

    Just wondering if you were planning on doing any more Q&A videos any time soon? Or are you trying to back off that type of thing and steer people to the google webmaster blog etc.

    There’s a couple of things I’d love to ask you!!

    cheers

    clive

  24. Matt the screenshot is breaking your css. hangs about 150px or so over the sidebar, or under as it were.

    Nice tidbit for us though, thnx

    Peace

    Mich D aka @MichDdot

  25. Thanks Matt, I appreciate your feedback.

    Presumably if the user was redirected to a logical and expected page it would be a Positive user experience.

    Misspelled URL -> Correct Page

    For example, it would be reasonable to redirect “/blog/)” links [above] to “/blog/”

  26. One other question — is there any way to remove items from the list? For example, I’ve fixed a few (resolved the inbound link that fired the 404), but the 404 is still there if you re-crawl. Will it go away because the link is fixed, or will it remain on the list because that URL is still a 404?

  27. Hello Matt,

    I think this new feature is a very good addition for the Google Webmaster Tools. I always saw lot of URLs in Webmaster Tools and wondered where in my site are those links posted.

    I wasted so much time in finding the URLs that does not existed in my site. Now finally I found that the not found links shown in my site are actually placed on some other sites.

    This is one of the best additions in Google Webmaster Tools that I have ever seen.

    Thank You Google
    Regards

  28. Great post Matt, thanks. I did sort of read through the Google post, but didn’t realize this.

    Thanks

  29. I’ll have to check this out. Thanks for the heads up on this tool. People do mistype URLs quite often, so this is a great way to backtrack and try to fix those errors. Thank you!

  30. I’m totally calling foul on this headline for being misleading and completely over the bounds of white hat puffery!

    How to get free links implies that you are getting something you didnt have before you started. What you’re really doing is find links that you already but lost Allow me to make an analogy …

    Perhaps at some point in your life you’ve lost your car keys. I mean really lost your cars not just misplaced them for a few hours. Like go to the dealership and spend $$ to get new keys. Then one day four years later you find that your cats/children moved those keys to a safe hiding spot in the laundry room under the dryer. Did you just get free keys? Nope you just found the old ones you already owned but had lost.

    But then again you probably don’t want this post to rank for “find links you had but lost” instead you want it to rank for “free text links” or “free links”.

    Just a guess though looking at your post title and URL permalink … :-)

  31. I have to agree with graywolf. links to 404 are definitely not free links. And tools for finding links to 404 pages exist for quite some time as far as I know. Even simply looking at your server’s access log gives some clues.

  32. Thanks for the tip. For some reason a lot of the links on the Google Webmaster Tools seem to be giving 404 errors :-(

  33. g1smd

    Ah, quit yer whinging already. :-) This is a great feature, and maybe “Linkbait of the Year” headline too.

    I regularly increase the “effective” inbound links by 5 to 10% by looking for errors on incoming links and either redirecting them, or contacting the owner to correct what they typed. This feature helps me find them a lot more easily, and allows me to find some types of error that couldn’t be found elsewhere.

  34. g1smd

    @greywolf: Ah, quit yer whinging already. :-) This is a great feature, and maybe “Linkbait of the Year” headline too.

    I regularly increase the “effective” inbound links by 5 to 10% by looking for errors on incoming links and either redirecting them, or contacting the owner to correct what they typed. This feature helps me find them a lot more easily, and allows me to find some types of error that couldn’t be found elsewhere.

  35. graywolf: agreed… heck, if you have a decent 404 then you haven’t even lost them… just not making the most of them. Kind of like having a Porsche parked in the garage and never using it.

  36. Matt, does any PageRank or anchor text flow to a non-existent URL where the server is sending a 404 error code? Many of us have seen the “URL-only” listings through the years, where Google shows us a destination it knows about but hasn’t crawled. Do such destinations accrue PageRank and anchor text?

    If not, that would certainly justify your headline in everyone’s eye, wouldn’t it?

  37. daniel

    I’d love to use this tool, but mine is showing hundreds of false positives – they seem to be caused by HBX tracking

  38. I may be missing something obvious here, but this seems like something that could easily be exploited.

    1) Create splogs and other useless sites that link to dummy page (or no page at all).

    2) If necessary, delete dummy page from server.

    3) Visit WT, see 404 report that indicates the 10,514 links to non-existent page to verify that Google knows about the incoming links and that they’re going to work once you put something up in the hole.

    4) Put something up in the hole, either by reactivating the dummy page or by redirecting to another page.

    This would do next to nothing from a direct traffic point of view, but that doesn’t mean much to the LJOCD (Link Juice Obsessive Compulsive Disorder) crowd.

    I’ve also noticed that some of the “links” appear to be one self-referencing link. For example, I see one on one of my sites where the only link to domain.com/this-is-a-dead-page is domain.com/this-is-a-dead-page. If it’s just a self-referencing link, maybe you could highlight that row in a different color or, better yet, just remove the whole link outright?

  39. Do we need to disable “Provide suggestions for incorrect or unavailable URLs” option if we are using Google toolbar ?

    If enabled that option then it says “Oops! This link appears broken.
    HTTP 404 – File not found.”

  40. JB

    Hi Matt,
    Great information and very useful feature…before yesterday, it took hours to accomplish what is now available in seconds.

    The best feature is knowing when the page containing the link was indexed. But, we’re seeing backlinks coming from sites that haven’t existed in months—in once case, a year (May 2007). There’s no way to fix these links and we feel that it could be damaging our overall trust.
    1) How long does Google hold onto these links?
    2) Is there, or will there be, a way to remove these from the index? This is important because a 301 isn’t possible on many links with odd characters or variables—.htaccess can’t handle these properly, and IIS is almost useless.

    Thanks

  41. Here is another suggestion Matt.

    Google Webmasters tool can also tell us if there are any links on our site pointing to a 404 page on another site.

    This will make it simple and easy for every webmaster to fix links.

  42. Carly

    No need for any custom in house tools, or doing anything special most stats programs like Awstats in cPanel shows you this. Just scroll to the bottom, click 404 and it will show your 404 URL’s and the referrer beside it.

    It also has the advantage of showing you any 404 images, CSS, JS etc and not just the URL’s so you can fix those pages.

    This also shows the “number” of hits to the 404 URL/Element, so you can see how much traffic is going to waste or how much the missing design element is called. Oh yeah, if someone is monkeying with your URL’s trying to append external files or funky query strings to your pages in attempt to exploit your site, it will show that too.

  43. I don’t know why people feel that the title is misleading…I got to this post here from weblogtoolscollection and the guy there seem to be echoing what graywolf said here…but i disagree with either of them…

    It could even be that the person who linked to you never did it the right way from the begining…he would have wanted to link to you but he made some typos..in that case this new feature in google lets you get all those links free..all you have to do is to just let him know about it…also ask him to find all such broken links using my broken links remover plugin , if he is on wordpress… :)

    moreover, most of the times links that were orignially good, never get lost due to linker’s mistakes…it could be that you changed the location and didn’t do a proper 301 redirect…in that scenario this tool actually helps you to recover those links freely…

    Thus in either case google webmaster tool is helping you to get the links without you spending any money or effort (FREE)

    “Free links to your site” is a good title…and very apt here…

  44. If the links are broken then you need to correct them…even if they are self referencing links…may be you don’t get any juice out of it but broken links are still bad…

  45. the webmaster tools are really useful… i have used the useful script provided by google for my customised error page: http://www.redesignyourbiz.com/wallpapers.html

    The above page does not exist on my server hence google will recommend you a different page containing a wallpaper from my site.

    using 301 redirect i am then redirecting the user to a page on webshots. so my ranking for that page is not getting wasted.

  46. Abe

    A little sensationalist, but still very handy to know!

    I have started using this code on several websites I run and it seems to be working.

    Thanks for sharing this little gem :)

  47. For the Linked From column, a descending sort would help me prioritize the 404 links I need to fix first.

  48. You want to hear something ironic…. I have just tried to log into my webmaster tools account and I received an error 404. What’s up with this??

    PLEASE HELP.

  49. With me GMC is reporting 404 errors, but these URL’s are still live!

  50. It seems that some people are misunderstanding the circumstances in which it might be appropriate to send a 301 Moved Permanently redirect, as opposed to a 404 Not Found error. HTTP was designed the way it was for a reason, and it gives best results when used correctly :-)

    If you have a page residing at a certain URL and then move that page to a different URL – for example, http://example.com/orang is moved to http://example.com/orangutan – then it is entirely appropriate to issue a 301 Moved Permanently response to requests for the first URL. This way existing links on other sites, or a user’s personal bookmarks, which point to the old location will still find the correct resource.

    The tool Matt describes above offers excellent support for dealing with this scenario; I would recommend using 301 as the primary method for dealing with these situations, with the “contact the other webmaster” approach as a secondary method. (The other webmaster might not bother updating their link, and you might still have users with personal bookmarks who should be redirected.)

    However, if a user types http://example.com/orange (and there is nothing on the site at that URL) then it is incorrect to issue anything other than a 404 Not Found – the client has made an error, and should be informed of the fact. Simply redirecting them to your home page is as confusing as a desktop application encountering an error and, instead of telling you about the problem, just resetting itself to its startup state. It also allows malicious people to create lots of spammy links to your site – do you really want people to follow http://example.com/animalporn and get your homepage, with the implication that it’s on there somewhere if they look around for it ;-)

    A custom 404 page, with appropriate search options, and maybe “Were you looking for…” suggestions, can help; but pretending that the user didn’t make a mistake is simply wrong.

    Of course, if you once had a page concerning citrus fruit but switched the emphasis of the site to simians, then http://example.com/orange might be more appropriately served with a 410 Gone response; but that’s a different issue :-)

  51. That could be done before in WMT, though more difficult.

    When will I be able to tell Google when to crawl our site?
    Sometimes Google crawls when there are many users and the server goes down

  52. Ummmmm … I periodically run the Xenu link checker on my site and know that it is pretty clean for internal links. However, when I just checked WebMaster Tools, I found several 404 broken internal links flagged … but these were actually OK – all long-time pages that hadn’t changed in a LONG time.

    The common theme seemed to be that I was using relative links – i.e. href=”../../PATHNAME” … and yes, I confirmed there was nothing wrong by not only clicking on the source page, but even using the command line and do a cd/ls from the directory of the linking page.

    I use a LOT of relative links (easier than using absolute if you ever want to move things) and this Webmaster boo-boo was only a handful of cases … so make me wonder if there is an occasional burp with GoogleBot on relative links (?)

  53. That is a pretty nifty tool.

  54. Stephen

    What would be great is to close the loop between this new feature and the embeddable Javascript code for the 404 pages:

    1. As a webmaster, you find out about links that point to a 404 page from Webmaster Tools. Emailing everyone who’s linked to the page is a lot of work, so instead you…
    2. Tell Google what page the broken link should point to, either as part of the sitemap or directly from Webmaster Tools. Then…
    3. The Google Javascript code on the 404 page redirects the user to the correct page.

    This way the user never needs to know about the broken link, search engines still get the 404, and you don’t have to spend a lot of time asking people to fix their links.

  55. Mark

    Good idea. But, i’d like to know how to remove pages which shouldn’t be indexed, like pages with Google Analytics tracking strings on the end.

  56. Matt,

    Thanks for the headsup on this. Gonna have fun playing about with it to see if we can convert some of those dead links.

  57. It’s a great tool, and I have been using it for a lot of sites when moving pages or implementing a new platform. It’s been great to be able to catch pages that other sites links to.

    However, since yesterday, I have been getting 404 crawl errors from Google Webmastertools, especially from the 404 report. Feels like an ironic loop.

  58. We have over the last few months been going through our logfiles looking for 404 links and 301 redirecting then to the “intended page” so do not have any 404 links now showing. Should we remove the 301′s so that the natural order is maintained or keep things as they are?

  59. kshu

    Hi. It would be great if the “Enhance 404 pages” feature from Webmaster Tools could use the search from AdSense (or Google Coop monetized), instead of a regular search from Google with only the site parameter set, like it does now.

  60. Soon we will all have more useful 404s…

    I haven’t had the time yet to implement, but I soon will.

  61. Matt – thanks for pointing out this excellent component of webmaster tools. This has always been a little tricky to find using analytics tools. We have always used customized 404 pages with php code that sent email to webmasters indicating what triggered a given 404. This of course inundated inboxes with individual 404 notification instances. Which then led us to put the emails into a “digest” format, which let us to build a few modules and plugins for the functionality for the Drupals and WordPresses of the world. The product ended up being http://www.errorlytics.com and basically does what this feature of webmaster tools does, but beyond just the Google’s-eye view of the web. It’s installed on the client website and allows the user to quickly and easily write the redirect (301 of course) or ignore the 404 error. The product is still in beta (of course) and will eventually have a pay model layered on top of it, but for now if anyone who is interested in this subject matter (handling 404s in a user/spider/ad revenue friendly way) wants to check it out and let us know what ya’ll think we’d appreciate it. Currently account sign up is free.

  62. Thanks Matt to let us know about this new feature. Like the tool. Truly useful.

  63. Now, if I can just get it to appear that Google.com referred those links, we’ll be all SET.

  64. Dave (original)

    Matt, can you define what you mean by “higher quality links” in detail for myself and others?

    IMO, the link spam is a big enough Monster myself, without the head of the Google Spam team feeding the frenzy.

    On a brighter note. Great job from Google with the 404 JavaScript.

  65. Google Site for Google Apps

    How do you configure 404′s with Google Sites for Google Apps? When I try to create a Google Site page with say index.html Google Sites converts the filename to index-html (index “dash” html) so people arriving on the Google Site receive the error

    “Page not found

    We’re sorry, but we were unable to locate the page you requested.

    Go to Home

  66. CG

    It would be great if you could filter out links from internal pages — tedious to go through all these.

    Thanks for the tip.

  67. What we really should do is worry about that everyone hitting a 404 has an easy way to find on our sites what they intended to when they clicked on the link.

  68. Dave (original)

    Mike Dammann, that is what a custom 404 page is for. The least you have on your custom 404 page is a home page link.

    Matt, is putting Adsense on a custom 404 page allowed?

  69. Two other features would help,

    1. We should be able to sort the “Discovery Date” column. I would always like to attend to errors that were most recently found

    2. Need a search option. I may want to check “Is http://www.xyz.in/abc.html” in the not found list?

    This feature is very useful and we all welcome it. Thank you.

  70. It’s a great tip even though we’ve got a solid 404 system in place (it catches pretty much everything). We’re also testing the 404 code that Google provides to make 404 pages more useful, it seems to be working really well. Most of the broken links are pretty easy to fix, either they’re on old pages that we control or on the site of an associate company. I’ll try to contact the others but if i have no luck we’ll just 301 them to relevant pages on the site. Thanks Matt.

  71. That’s a really useful tool for our webmasters (and editors) to discover internal and external links that don’t work any longer. Now the clean up process is 10 times faster than before. Thanks to the team members at Google Webmaster Tools who worked on this ! If we now can get a small button to move a weired URL to the “Delete URL Tool” would be even greater – now we have to copy the URL from one Google Tool into another.

    Thomas

  72. Hi Matt this is a great tip and i have used it for one of our clients and found loads of dead links : )

  73. Great tool Matt. I’ve seen this not found feature some days ago because i have 300 links. Do u think that the feature “pages” can be extend so i can contact the people that link my 404 directly from GWT? ;)

  74. I really feel sites like BBC should start making use of these tools, I see way too many broken links which they don’t seem to care about. Not just BBC most of County Websites or Government Agency Website.

    Very interesting feature Matt! Thanks for explaining it up here.

    Cheers!

  75. Matt, thanks a lot for this! I’d just like to add that beyond the great JS code Google supplies it’s fairly easy for people to add PHP based redirects to their 404.php page (in WordPress) to redirect incoming errors.

  76. That is a useful tool for our webmasters to discover internal and external links that don’t work any longer.

    Thanks a lot.

    Andrea Pilotti – Italia

  77. Oh Matt, I like you and hate you all at the same time. But yet again, you chip away at small pieces of my USPs. All would be good if my boss didn’t RSS your blog, but he does, so meh! Suppose…. Suppose I should just get good at my job instead. [:

  78. Oh and as a side note Matt; I put this across on the Google blog, but I’m not sure if those comments get looked over by anyone at the big G or not.

    Anyhooo, my suggestion – Just like how you can set up Google Analytics to send out a daily email etc with a break down of data that you request, I was wondering if one day, they could implement a similar thing on Webmaster Tools where I can get email alerts or daily breakdown of site/page errors, crawl rate, yudda, yudda so I can spoon feed to our dev team to make amends where required, because going through a list of XX clients every so often is quite the proverbial ball ache!

    Yep, that’s right Matt, I’d love G to make my working day more efficient (or at least to the point where I can leave at 5pm sharp every once in a while) ^_^

  79. Matt – good grief I just reclaimed a PR7 .edu inbound from a large business school site, thanks to this new feature. It took one email and a half hour, and they fixed it.

    Not that I care about such things…

    -ew

  80. Great Matt, always helpful. This is a great feature, and maybe “Linkbait of the Year” headline too.

    I regularly increase the “effective” inbound links by 5 to 10% by looking for errors on incoming links and either redirecting them, or contacting the owner to correct what they typed. This feature helps me find them a lot more easily, and allows me to find some types of error that couldn’t be found elsewhere

  81. This extra feature does help – I am interested to know if and how long it would be before links pointing in to a domain which previously 404′d actually then start to get recalculated? It stresses the importance of 100% up time and a good server…..

  82. Will these all be categorized at internal links?

  83. Cool idea, but looks like a pain in the arse to implement.

  84. Excellent article Matt,

    My current 404 page has always had a number of links to the main resources of my site; and it has always worked well to divert users from dead end pages. I will definately try that widget though, the closest match function is a much better way to regurgetate traffic into the normal flow of your website: obviously because the links are more relevant.

    Thanks

  85. could all this be fixed with a 301?

    Regards

  86. In my GWT i see 3 links from bad, malware infested splogs who have deliberatley created broken links. I have left these to return a 404 as i am fearful of creating any association with these sites and the porno keywords they sometimes use.
    is this the right or wrong thing to do?
    In the meantime i think its great tool and really appreciate it.

  87. Shawn

    Wow I just found 48 links pointing to:

    http://mysite.com/index.htm when I changed it a while ago to http://mysite.com/index.php

    what’s the proper why to address this, 301 index.htm or 301 the 404?

    Thanks Shawn

  88. Free links? Nice. Here is one possibly worthy of a free link

    After last nights “debate” I decided to buy http://www.joetheseo.com in the hope of making more than $25,000 this year ;)

    Cheers

    Joe The SEO

  89. all I’ve got to say is at long last , without these great tools from google we would be spending half our time running round like headless chickens trying to figure out problems with our sites. thanks google !

  90. Great addition to Webmaster Tools and a nicely laid out tutorial. Thanks Matt.

  91. “Joe the SEO” eh?

    Brilliant David. As an OHIO small business owner, why didn’t I think of that?!? I hope you get rich off it, but you’ve probably got 3 weeks and counting to make it happen. :)

  92. g1smd

    ***Wow I just found 48 links pointing to:
    *** http://mysite.com/index.htm when I changed it a while ago to http://mysite.com/index.php
    *** What’s the proper why to address this, 301 index.htm or 301 the 404?

    You should 301 redirect all index.html and htm and php URL requests (for both www and non-www) to strip off the index filename. Within that same redirect force the www on to the URL so that another rule hasn’t got to be invoked (and hence cause an unwanted redirection chain) to do a second redirect to fix these index URLs.

    Follow that up with your non-www to www redirect for all remaining URLs not caught by the first rule.

    [code]
    #
    # Force all remaining requests for named index files to drop the index file filename, and force www.
    #
    RewriteCond %{THE_REQUEST} ^[A-Z]{3,9} /([^/]*/)*index.(s?html?|php[45]?|[aj]spx?|cfm)(?[^ ]*)? HTTP/
    RewriteRule ^(([^/]*/)*)index.(s?html?|php[45]?|[aj]spx?|cfm)$ http://www.example.org.uk/$1 [R=301,L]
    #
    #
    # General rule to force all non-www URLs to be www URLs. This rule must be the last of the redirects:
    #
    RewriteCond %{HTTP_HOST} ^example.org.uk [NC]
    RewriteRule (.*) http://www.example.org.uk/$1 [R=301,L]
    #[/code]

  93. Oh, very very cool! I hate that so many people seem to think big bad Google is out to get them, when clearly they do so much to help website owners! Thanks for pointing this out here Matt!

    Lisa

  94. Dave (original)

    @ David S,

    Hmmm, a “SEO” site with a grey bar, not a good look. Are you sure haven’t purchased trouble?

  95. I have found this tool really useful. Where possible i have managed to get the incorrect link changed to what it should point to & for the other i 301′d.

    Great tool because it gave me and interesting insight into what baited links i’d got

  96. Shawn

    Thanks g1smd,

    “301 redirect all index.html and htm and php URL requests (for both www and non-www) to strip off the index filename.”

    Worked great and Matt I assume this is how google would like to see the changes made too?

    Shawn

  97. Hey, that’s pretty awesome! I never knew about this tool, should come in handy. Although I don’t much care if people come to the wrong link of my site as they’ll have a chance to register even on a 404 error page.

  98. g1smd

    I reposted the code because the first version was screwed up, but unfortunately the corrected code was deleted, and only the junk copy still shows! Hopefully Matt can swap it back.

  99. g1smd

    (This comment replaces the corrupted comment above)

    ***Wow I just found 48 links pointing to: ***
    *** ht tp://mysite.com/index.htm when I changed it a while ago to ht tp://mysite.com/index.php ***
    *** What’s the proper why to address this, 301 index.htm or 301 the 404? ***

    You should 301 redirect all index.html and htm and php URL requests (for both www and non-www) to strip off the index filename. Within that same redirect force the www on to the URL so that another rule hasn’t got to be invoked (and hence cause an unwanted redirection chain) to do a second redirect to fix these index URLs.

    Follow that up with your non-www to www redirect for all remaining URLs not caught by the first rule.

    [quote]
    #
    # Force all remaining requests for named index files to drop the index file filename, and force w ww on to the URL
    #
    RewriteCond %{THE_REQUEST} ^[A-Z]{3,9} /([^/]*/)*index.(s?html?|php[45]?|[aj]spx?|cfm)(?[^ ]*)? HTTP/
    RewriteRule ^(([^/]*/)*)index.(s?html?|php[45]?|[aj]spx?|cfm) $ ht tp://w ww.example.org.uk/ $ 1 [R=301,L]
    #
    #
    # General rule to force all non-www URLs to be www URLs. This rule must be the last of the redirects:
    #
    RewriteCond %{HTTP_HOST} ^example.org.uk [NC]
    RewriteRule (.*) ht tp://w ww.example.org.uk/ $ 1 [R=301,L]
    #[/quote]

    Hopefully this code won’t break too much when posting it here.

    Join ht tp and w ww back together (broke them mid-word to stop them auto-linking here).

    Remove the space before the dollar sign, and if there is a 1 just after the dollar sign, remove the space before the 1 as well.

  100. This was an awesome suggestion. Thanks for the free links and for providing all the data on how to capture them. Very cool!

    Michael

  101. After a quick flick through the comments I don’t think this has been mentioned yet, so I’ll add my 2 cents.

    The webmaster tools javascript that’s mentioned, while being useful for 404′s can also be useful on the search results page (of a wordpress blog at least).

    I’ve not looked into the details of it, but I’m pretty sure the wordpress search functionality is ‘limited’ to the content of posts and pages.

    This is great for most cases but when you have a plugin that modifies content and stores it elsewhere (e.g. global-translator) or just outputs stuff on the fly from some other source (e.g. related posts from other blogs) that should be useful, and would be indexed by a search engine, then the blog’s internal search mechanism presumably won’t pick this up. This makes the ‘second chance’ option to search again via a ‘real’ search engine more useful.

    To add the code to your search results simply edit the search.php template file and past the javascript in the section that deals with no matches.

  102. Thanks for the great tip Matt. I just checked out my blog and caught a bunch of links that pointed to the wrong address. I’ve made all the corrections, and if everything goes well, hopefully it will make a difference. If nothing else, it only took a few minutes!

  103. thnak you very much Mr. Matt Cutt, We learn another feature that ishigh valuable. linking is always a concerned subject to every webmaster but caring to “not found pages” and its linking from various sites is need to concern first…..

  104. It’s great, and has helped me fix a lot of problems – but one minor gripe:

    The download doesn’t include the source pages. To get a csv with the sources, you have to visit each reference individually and download the csv for just that error.

    Minor complaint, but being able to get the complete info quickly would make it that much better.

  105. Great post…I’ve never used those tools before. Thank you!

  106. Ahhh! This is the best feature ever! I discovered 12 incoming links that were all leading to a 404 page. I have now contacted all of them and they have changed the links. Great new feature and thanks for the help!

  107. thanks matt – nice heads up on the 404 thing.

    i think wordpress has a plugin that converts all your 404 errors into links to your home page – might be useful in regard to this.

  108. Thanks for the great post. I gotta check out webmaster tools one of these days.

  109. Thanks for the info. And thanks for that last comment Charles Lowe. I’m going to have to look for that wp plugin. This goes to show I do not look at the Google Webmaster tools nearly enough. There are probably a lot of little things I’m missing out on there.

  110. Great post. Funny how you can use the tools regularly and miss such a useful feature.
    When is the “No1 on page 1″ optimiser tool set for release Matt? ;-)

  111. I get 404 errors because google crawls my URL/post titles, however, my XML is set to %categories/posts%. Therefore the 404 errors are correct as there are no pages on URL/Posts. Would anyone know what I can do to correct this so I do not get these 404 errors. Thanks

  112. Totally agree with carl. AWSTAT if installed on your server can show you this 404 errors and where they are coming from

  113. Hi Matt, thanks for the overview! btw, your graphics in this piece are running under your sidebar. If you add max-width: 740px to your img style sheet it’ll clean it up.

  114. Great write-up! There is so-much information available now that it’s hard to keep up. I satrted using the webmaster’s tool and found a few problems and fixed them. Thanks again

  115. Awesome past Matt, it’s one of those things that you don’t realize even though it’s right under your nose. I will be sure to check for the 404 errors from now on.

  116. Alessandra M

    Hey Matt, Thanks for the suggesting me!! It’s really helpful for me.
    I used this code 404 for home page like
    http://www.rvsjunction.com this is my site. If some one put like this http://www.rvsjunction.com/asdasdasdasd URL and hit enter then automatically redirect at my home page. It’s superb!!!
    Thanks really thanks!!!!
    Keep Going!!!!

  117. nice idea , I will check my log for 404

  118. you could change your 404 so that it goes to your home page like mine does.
    http://www.deerdays.com/adf;kajds;fjakds;jga;
    go to that and my it will go to default.aspx

    IIS makes it easy to configure.

    you can be tricky like what I do for http://www.probattlemma.com/. It’s a 404 page that has a javascript so that i can send html links to it like this
    http://www.probattlemma.com/NewsArtical172.html that will redirect to the appriate news.aspx? page query displaying the article.

    You might ask why would you want to do that. Becaues XML Feeds do not allow the “&” chr as part of a link. So if you want your link to be in a feed it can’t be a link with a query following it.

  119. An excellent tool – does make life easier. However, does it catch ALL 404 link errors? Its just that Google isnt great at revealing all backlinks

  120. A friend once told me that whenever he sees the 404 thing he gets the impression that the site is not ‘good enough’ to visit otherwise it would not be having this problem! This might be the view of a person who is not from the field I am in.But he did tell me one thing which is worth considering.A serious look into this would help keep the google presence intact all times.

  121. A little late to the party here, but I just went and check and we have over 900 page not found errors on one of the sites. Yikes. Turns out a fix needed to be done on our programming end, something had gotten messed up, pulling individual home listings went bonkers. Thank goodness I didn’t have to start fixing 900+ errors.

  122. A good method I use on some of my sites is to redirect my 404 pages to the homepage. ;)
    Works great!

  123. Matt you never stop amazing me! I found your blog by accident after reading your Wikipedia page. The 404 error has been a frustration for some time and the feature works great! Thanks again to you and Google for the amazing free webmaster tools!

  124. I just checked my 404 back links and added a couple of pages with names coming from the bad links. Now I don’t have 404. Easy and simple, alghtough not sophisticated.

  125. Wow, I’m embarrassed to be finding this tool so long after it became available. Thanks Matt!

  126. Hello, Matt,

    301 Moved the URL http://www.mmorpgrealm.com/2008/06/28/ragnabot-errr-i-mean-ragnarok/ to http://www.mmorpgrealm.com/ragnabot-errr-i-mean-ragnarok/

    GG Webmaster showed me the page has been 404.

    Have no idea about that.
    Thanks!

  127. Something is wrong with google 404 repoorts or with me :)
    In my case google reports some 404 errors on pages that do exist. I can happily view them in my browser and no 404 is returned. Just a note.

  128. wow. What a great tip Matt.
    Heading to webmaster tool right now.
    same question mentioned above though, if I redirect all 404 to the homepage, will I get penalized for this?

    Thanks

  129. I’ve never look at my 404 page, because there is so much to do to maintain my sites. Is it really important to fix your 404 page?

  130. tamaras,

    From what I gather, it’s not that you need to “fix” anything, it’s that you’re getting link love and then it’s not counting for your site because you don’t have a page there. You have an error showing up, which is NOT the same thing as a page on your site. So if you want an easy way to “get more links” (because you already have them they are just not counting) you can redirect the 404 errors to where they should go, or to your home page in the very least.

  131. Thanks for the tip, I will check my site thoroughly for errors.

    Paul

  132. That is a great idea. I actually never even thought of that, but it makes perfect sense. Now that i am looking into it I see about 15 good links I could restore.

    Thanks Matt

  133. Thank you so much Matt, for this great info as well as your other informative writings, and I am going to webmaster tools right after I finish this comment. So now, I can just redirect all the 404 pages to my homepage. Thanks again!

  134. I have to say that I have used this now in a new way. We recently changed our shopping cart software from Amazon to a new system. The new system allowed us to create identical URL’s for the site EXCEPT for products that were once in stock on old site, but have since gone out of stock and would not be created on the new site. We had no way to know what those pages were as some were 5-6-7 years ago. So we have been “mining” the “not found” for many months now and each month Google gives us about 500 of those URL’s to redirect. We cannot always tell the product exactly, but some times we can. After all, we only get the URL, no sku, no title, nothing. First we check the referring page and try to determine the context of the link and try to redirect to an appropriate spot. So if a URL indicates it was a ring and the referring page was about sapphires, then we redirect to http://www.gemaffair.com/category/1034389421/1/Sapphire-Ring.htm but maybe we only know it is a ring then we redirect to a higher top level cat like http://www.gemaffair.com/category/644850461/1/Rings.htm and then maybe we know nothing and no clues, then we just redirect to the home page. While this is tedious and time consuming work, it has boosted our links by 2x and some of the links are very old, very on topic and very valuable. This is by far one of the most useful insights for webmasters that Google has been transparent about. Thanks to Google and to Matt, again.

  135. These are great tools, cheers Matt.

  136. Just found this blog and love it, had to comment here as checking for 404s is one of the most useful aspects of Webmaster Tools for me. Love the new keyword features too, thanks Matt!

  137. filip

    Matt,
    I think being able to see links to 404s is a fantastic feature, as well as general monitoring of inbound links. Which is why i’m a little disappointed and scared by the recent change in webmaster tools to only allow you to see/download a fraction of the links.
    I almost didn’t believe it as this has been one of my favorite features!

    Everything looks OK on my small sites, but for larger ones I can only download a fraction of the total!! I really feel being able to monitor ALL links is an important process in staying compliant as well as gaining incite on what others think of your site.
    Please urge the team to rollback!

  138. I found 46 links to one of my 404 pages! Great tool, very effective.

  139. Broken links are effectively lost. When I find something in the attic that I didn’t even know I had mislaid, I definitely consider it a freeby. So this post’s title works for me!

  140. Hi Matt,
    Its really helpful to maintain 404 page and that too with a positive message.

    The second screenshot is too wide to fit in left colum. Its overlapping the archives list on the right. I’m sure you use Chrome, SO may be image resize helps.

  141. Well, I see it the other way as well. It means you can find those pages who are trying to get you site degraded in serps due to bad user experience.

    This would come really handy. I think redirecting to main page would not do much good, because it is not a good visitor experience.

  142. Cheers for the tool. It’s nice to be able to hear from someone who actually works for google, reassures me that i’m not an insane SEO and there is life at google!

  143. No need for any custom in house tools, or doing anything special most stats programs like Awstats in cPanel shows you this. Just scroll to the bottom, click 404 and it will show your 404 URL’s and the referrer beside it.

    It also has the advantage of showing you any 404 images, CSS, JS etc and not just the URL’s so you can fix those pages.
    Thanks

  144. FrontPage (I know I’m old school) has a tool to find all your 404 pages. Can see links to site on Yahoo explorer and ask people to correct them.

  145. ken

    Thanks for the information, I just corrected most of my 404, and will do the rest, there were about 20, thanks .

  146. Bob

    I’m a little confused don’t most servers have a way for set error pages to automatically go to your home page?
    Thansk Bob

  147. I agree with Brian, Awstats does the trick. You can’t export it to a csv like you can with the webmaster tools, but, cut and paste works fine.

  148. vergi sorgulama

    I’ve never look at my 404 page, because there is so much to do to maintain my sites. Is it really important to fix your 404 page?

  149. hye
    i am the owner of coolguruji.com, its a search engine. i want to make backlinks for my website but i dont know how counts backlinks. so any one can help me to improve my backlinks and show me the way to counts backlinks, one thing is more, is there any list of free backlinks postings website, because it is very difficult for me to find free backlink posting site on web.
    thanks in advance

  150. Great tool, very effective.

  151. Thanks for the information, helpful indeed.

  152. How long does Google hold onto these links?

  153. Wow I never realized the potential of turning a crawl error into a positive impact. Thank you!

  154. This was very timely. I was repairing all the bad links on my website to other sites. Thanks for the information and I will look at it from the other direction into my site.

  155. Thanks, its good! I Have 99 Page 404 not found

  156. Hi Matt, thanks for the information.
    I just corrected most of my 404, and will do the rest, there were too many, thanks!

  157. great site, I still am new to the internet link game and all the info helps out.

  158. I have redirected all my 404 pages to the correct pages and I have disallowed them in the robots.txt file, still picking it up as errors why is that.

  159. Jim

    How can I remove some 404 pages from the index? or should i just do a 301 ?

  160. Thanks for the tip. Now im removing those 404 pages from index.

  161. hello matt,
    so, what should we do best to resolve about content 404 not found ? Redirect or using robots? still confuse,,
    thank you

Leave a Comment

Your email address will not be published. Required fields are marked *

*

If you have a question about your site specifically or a general question about search, your best bet is to post in our Webmaster Help Forum linked from http://google.com/webmasters

If you comment, please use your personal name, not your business name. Business names can sound salesy or spammy, and I would like to try people leaving their actual name instead.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

css.php