Url removal: yah!
For now, I’m just going to say “hot damn.” The smart folks on the webmaster console team have migrated Google’s url removal tool into the webmaster console. Along the way, it’s picking up a *lot* of nice new functionality. I’ll talk about it more pretty soon, because I have a fun story to tell, but in the mean time you can read more about it from the official webmaster blog or on Search Engine Land.
Yah!
gices Said,
April 18, 2007 @ 1:19 am
Yeh, i was just reading that on the Google Webmaster Central and your post popped up on google reader. I’m about to test this new functionality as i have some URLs on my site which i dont want included in the index and although i’ve blocked it with the relevant meta tags, it’s taking too long to be removed from Google index. So i think i’ll love this new tool
Dr. David Klein Said,
April 18, 2007 @ 1:26 am
O.K. Now I am feeling uber powerful! First I leave the comment on your blog at
http://www.mattcutts.com/blog/robotstxt-analysis-tool/
about what to do about the 404 errors I find on google tools for URL’s that I don’t have.
And now this happens! Boy is google being attentive to my needs and filling my requests!
So while I am at it can I also request a slider bar on the google tools to adjust my PR, and also check boxes to click for requests of which search terms I want to be number one for?
Thanks Google for being so quick to handle my personal requests!
Search EnginesWeb Said,
April 18, 2007 @ 1:31 am
This is so-o obvious that is it sad it was not implemented years ago.
BUT —- One suggestion…
A URL REPLACEMENT option….
Perfect for frustrated
Webmasters moving to a new URL or domain name or moving many things to a new directory - (perhaps those moving to a new URL - once verfied - can keep their newer backlinks and pagerank)
If this idea is implemented, please acknowlege SearchEnginesWeb
Errioxa Said,
April 18, 2007 @ 2:07 am
It would be good that you could put the URL with some type of parameter. For example http://www.mydomain.com/*.jpg or http://www.mydomain.com/*.mdb
Johan Said,
April 18, 2007 @ 2:08 am
Search EnginesWeb what does Google do with the new domains old back links?
daniel Said,
April 18, 2007 @ 2:14 am
fantastic - what a great idea. i love what webmaster central is becoming.
tuf Said,
April 18, 2007 @ 2:24 am
url replacement tool is useless.
just use 301 redirections
Aaron Shear Said,
April 18, 2007 @ 2:55 am
I love all of these cool new features. But it would be great to assign user accounts to your webmaster tools page. If there are different users, some may have less of an understanding of what a simple tool like remove may do to the wrong page.
With my account I have only shared the login with a few people and made sure they knew exactly what each part does.
Cheers!
TheSEOGuru Said,
April 18, 2007 @ 3:32 am
Hi Matt,
Thanks for this, I just go through one document related to user behavior.
One suggestion related to SERPs, shuffle sponsored list on the left side then u will get good ad clicks
cheers
paul Said,
April 18, 2007 @ 5:02 am
This came out of a great session at SES last week in NY, you should have been there Matt, not only was it fun, it was productive!
Jon Cram Said,
April 18, 2007 @ 5:45 am
I’d like to sneak in a question on a related subject (relating to ‘removal’ and ‘webmaster console’).
In the webmaster console, when I go to Statistics > Page Analysis and look at the Keywords list, I see many words that are not present in my site but are present in an unrelated subdomain.
Let’s say that my site is at http://example.com and an unrelated site, run by someone else and covering a completely different matters, is at http://subdomain.example.com.
I normally find the keywords list useful, however with the recent inclusion of a substantial amount of other terms this tool is becoming much less useful - about 60% of the keywords listed are not mine but from the unrelated subdomain.
Is there anything that can be done about such issues? I can’t find any existing options within the webmaster console to deal with it.
Could changes be made to the webmaster console such that, for a given site, one could explicitly specify subdomains that are part of the site, therefore automatically excluding all other subdomains?
For example. if I add a site for example.com, I would specify that support.example.com and secure.example.com are part of the same site, causing information for unrelated1.example.com and unrelated2.example.com to not be displayed in the webmaster console for the given site.
BobR Said,
April 18, 2007 @ 6:03 am
Hey Matt,
This is great and thanks for this tool. I would also like to see something in there that shows 100% of the pages Google knows about.
I run a yahoo store and it’s not hard for us to get orphaned pages that we honestly don’t know about. (I still think this is part of the reason for my bottom of the results drop) As far as I know my site has about 1,900 pages but Google always shows more. I can only dig so deep and every time I do I find orphaned pages. I remove them and wait a few more days and look again in hopes of finding more to get them removed.
It would be an easier task if we could somehow see 100% of the pages Google knows about someplace.
I know if we had FTP access like most webmasters do it would be easy but we don’t with yahoo stores.
Maybe you guys should start Google stores for us…..
Thanks for reading this,
Bob
netmeg Said,
April 18, 2007 @ 6:25 am
woo hoo… hey, maybe it was my fault - i suggested this on a ’suggestions for webmaster tools’ thread on webmaster world last year, and either adam or vanessa thought it was a good idea and said they’d pass it on to the gwt people. it was a pain having to search through h and highwater to find the url removal console every time i wanted to remove a bunch of stuff.
something wrong with my keyboard - i’m shiftless today
Scott Hendison Said,
April 18, 2007 @ 6:29 am
Thanks for the heads up, and it’s about time! And as SearchEnginesWeb said, how obvious. (His other suggestion too is long overdue - we want to change domains or URL’s without all the pain)
Putting more of the tools we need all in one place is much appreciated, and it’s very cool to watch the evolution of the console. It keeps getting better every month.
I also want to second Aaron’s request for mulitiple user accounts/logins. He’s braver than I am, giving logins to other people…
German Said,
April 18, 2007 @ 6:56 am
SUPER!!!!!!
How do you do when you can’t verify your own website due to some 404 file tests. Should we all enter all our sites in the webmaster console?
Seriously, I do prefer the way with the tobots.txt.
Michael VanDeMar Said,
April 18, 2007 @ 8:06 am
Yeah, Matt, does that mean that it is now only available to those of us with Google accounts? In other words (don’t mind the tinfoil hat here please
) paranoid people won’t get the benefit of the tool?
JLH Said,
April 18, 2007 @ 8:13 am
They’ve also expanded Page Analysis statistics that now include common phrases in external links. You can check them out and see how they actually look on the page. Vanessa must have used those extra two days to get her taxes done by rolling out some new products…what’s next? One can only speculate..me thinks it ain’t gunna be live PR.
German Said,
April 18, 2007 @ 9:50 am
Hi Michael,
Don’t mind me being paranoid
I think most people in my branch are.
Maybe it has something with our business peeking in other’s lives everyday and we can’t stand the thought of another big brother peeking in ours the way we are doing it (unwillingly).
Matts knows from all my sites. I’m not hiding them and they are interlinked or linked in someway. I am not hiding them and I do have an account.
I am just spending my life on the Internet,seeing what trying to go though my firewall, from every software I am using to a Google toolbar notifyer where I am sure I didn’t download the toolbar. My windows computer is since a week corrupted because an automatic update that I didn’t ask for. Yes I am paranoid and I can’t way to try linux to stop all these check in check out update here and there.
I will sure prefer avoiding any tool that mean i have to enter the names of my website and changing their configuration because of Google.
recep Said,
April 18, 2007 @ 10:12 am
Thanks for lovely article , l just burnt ur feedz
Heather Paquinas Said,
April 18, 2007 @ 10:42 am
Hey Search EnginesWeb, I think you should use a 301 redirect.
JohnD. Said,
April 18, 2007 @ 10:45 am
Now if only Webmaster Tools would update correctly. It’s still saying we have 3100 incorrect links on one of our sites. We did do some restructuring, but made sure to update all of our links and even updated the sitemap.xml. In the process our site that has been #1 for targeted terms for many years on Google has dropped completely off of all results. Still shows up #1 on Yahoo and MSN, just not Google.
The good news is our traffic hasn’t dropped terribly bad since being dropped from Google. Just shows when you have a useful site people will get to it one way or another, Google Analytics just shows traffic coming from other places, and not Google now.
feedthebot.com Said,
April 18, 2007 @ 10:46 am
good stuff, Vanessa and Rand should talk more often
Sidenote: I am confused why the Google webmaster guideline tell us to avoid helping the elderly and infirm - http://feedthebot.blogspot.com/2007/04/why-google-has-to-work-very-hard.html
Charles Stankovich Said,
April 18, 2007 @ 10:57 am
It would be nice to see it tied into the Web crawl errors page, or at least the Not found section. That way you could select the URL you wanted to remove if that’s the appropriate action.
ted sullivan Said,
April 18, 2007 @ 5:44 pm
Great. I have used that tool a lot to fix strange problems over the years.
Only thing is it is kind of deadly. Maybe there should be some speed bumps if you are removing more then a single page. Like a email and a confirmation required.
LanzaroteTourist.com Said,
April 18, 2007 @ 7:17 pm
I have to agree with Search EnginesWeb, I’ve not long change a site to a new URL and I’m having problems with the old URL still being indexed and showing up for the results. Although I’ve got a 301 redirect in place it does not seem to have the affect I was hoping for.
Having a option to change URL’s would help a lot of web designers.
Leslie Said,
April 18, 2007 @ 7:18 pm
I know this way off topic matt - but nowhere else to go and ask or complain. what is the story with google apps referals? Why US only? I live in europe - but most of the visitors to my main site are from the US.
Give us all a break, please…
Harith Said,
April 18, 2007 @ 8:31 pm
Matt
How to subscribe to your mattcutts twitter. Don’t you need all the friends you can get? well I do
Deb Said,
April 18, 2007 @ 10:00 pm
Hi Matt
I have already use it and works fine…………
Kevin Said,
April 19, 2007 @ 12:27 am
Matt - I’m impressed! I added the removal request for some old pages as soon as I read this entry of yours, and a day later they’ve been removed! Brilliant! It’s about time
Seth Aldridge Said,
April 19, 2007 @ 12:45 am
This is a really useful tool. I have a couple things right out of the gate that it would work perfectly for. I also am in love with the Google Walking Maps you posted a few days ago. Really interesting stuff
Fargham Said,
April 19, 2007 @ 1:33 am
It is a useful tool, once my site which contains less then 20 pages strated showing indexed pages almost 2000. At that time i wasn’t using sitemaps, But after adding sitemap, those pages were automatically removed over period of time.
Matt I have a question, Though its irrelevant to this post, But was curious about it.
Its seems to me that a major back link export is underway, I have noticed a few baclinks added to just newly launched sits just now. Is it?
Cherez Said,
April 19, 2007 @ 2:22 am
The fact is that the completed url removals that I have in the older removal page does not show up in the google webmaster console. This is important for me, because I have about 2 weeks left for reinclusion. And because of this, I need to see if any of the “completed” removals have timed out.
Fargham Said,
April 19, 2007 @ 2:23 am
one thing more added in webmaster console is under Page analysis!
First it was only kewords, then only phrases and now both. Also you can view phrases with variations.
Very handy addition.
Adam Said,
April 19, 2007 @ 3:46 am
Excellent stuff! It’s a handy tool that Yahoo have had for a while, so I’m glad Google have followed suit.
Serp Said,
April 19, 2007 @ 5:54 am
Very good. It works fine
Jhet Said,
April 19, 2007 @ 7:54 am
Well since we’re on the topic of webmaster tools, is there a tool or technique that will allow you to update the title and description in the Google search results quickly?
Dyce Said,
April 19, 2007 @ 8:35 am
I like it…. will have to see how much I use it over time… but already been having a go
Ian McAnerin Said,
April 19, 2007 @ 10:28 am
Let’s say I have a client that is susceptible to the whole ” a little knowledge is a dangerous thing” brain-bomb and got over-enthusiastic.
So they decided that since it sometimes takes months for a redirected link to come back up to it’s original position, that they would cleverly *remove* the original URL and then put in the redirect.
I don’t suppose there is an “undelete” button somewhere? Or maybe a dunce cap with a Google logo I can hand to the offending party as a reminder to listen to the nice SEO they have hired rather than doing it themselves?
I’m assuming now it will need to be respidered again and that this would take much longer than a simple redirect would have. I’d be thrilled to find out this is not the case, but I’m not holding my breath.
Ian
PS Now that I’m thinking about it, I REALLY would like a Google logoed Dunce Cap. I can think of a ton of opportunities for it.
KJW Said,
April 19, 2007 @ 10:44 am
Wow - that is great! I already used it…
Vermut Said,
April 19, 2007 @ 11:51 pm
Mmmm… It does not works fine. I’ve delete one entire directory from my server and the Http resonse header of my apache web server say “404 Not Found”, but the report says: “Negato” (It’s italian for “Denied”)
Fargham Said,
April 20, 2007 @ 2:56 am
some insight about Google web history!
g1smd Said,
April 20, 2007 @ 3:51 am
There is an oddity in Webmastertools that is probably very easy to fix.
Steps to reproduce:
1. Register a domain, and get a site online.
2. Add a robots.txt file of some sort.
3. Sign up for webmastertools and ask for the status of robots.txt for the site. At this point Google has not crawled the site.
4. Webmastertools will give a date and time that it says the robots file was crawled, and it will say that it is “404 - not found“.
The date and time given is that of when you signed up for webmastertools, and the “404″ is incorrect. The file is actually there, Google hasn’t actually looked at it.
The message should say: “not yet crawled”.
Dr. David Klein Said,
April 20, 2007 @ 6:17 am
I really like Ian’s suggestion. I would actually wear one from time to time.
ec Said,
April 20, 2007 @ 8:12 am
I have recently changed the navigation for my entire site, from dynamic to static, ie page.php?categoryid=3&pageindex=22 to /cat/3/22.html, how do I tell google to reset my entire site and start from scratch?
Georgia Said,
April 20, 2007 @ 10:48 am
It’s good to finally see the engines doing more to work with SEO types. I think the SEs desire to keep as much info from SEOs contributed to just about every SEO playing the black hat game - even if only a little. The more we know about how to rank the more time we can spend doing it right.
HDR Photography Said,
April 20, 2007 @ 4:15 pm
Definitely should have been implemented a long time ago!
Charles Stankovich Said,
April 20, 2007 @ 5:01 pm
Dr. Dave, Ian, we have the ability to make our own fabrics; wouldn’t you rather have a shirt with Matt’s likeness accompanied by a few Google logos and some Hibiscus flowers? Just think Hawaiian Cuttlette shirts, how about it Matt?
Psychic Said,
April 21, 2007 @ 4:27 am
I wonder if it means that I want to change a page and title so the new url reflects the changes. By using the tool I can delete the original url without penalty. As we know doing this in the past caused a 404 and harmed our rankings.
Harith Said,
April 21, 2007 @ 5:05 am
Matt
Long time no post. Haven’t you earned enoughhusband bonus points
avşa Said,
April 21, 2007 @ 6:53 am
thanx
John Said,
April 21, 2007 @ 5:04 pm
I like it, its working fine
Titanas Said,
April 22, 2007 @ 1:04 pm
It took a while but it’s good to know that Google listens. Keep up the good work.
Archna Sajwan Said,
April 22, 2007 @ 9:41 pm
I would say instead of going for url removal go for redirecting via .htaccess atleast this way, one can retain a url and inbound link. Same as you do it for dynamic url rewriting.
Will Howard Said,
April 23, 2007 @ 8:15 am
Question? If I remove an subdomain URL form the main URL will this affect my main URL in a negitive way. Or what affect will it have?
Ken Barbalace Said,
April 23, 2007 @ 9:43 am
I made use of the new URL removal tool this past weekend and it worked great. The pages in question were out of Google’s search index very quickly. For those very rare times when it is needed, this is a great tool.
SEO Practices Said,
April 23, 2007 @ 2:49 pm
Matt I just used the function to try to remove 12 urls, but when I went back to check for it, I got I message that said that the urls removal were denied. I’m not sure what happend ???
Hampstead Said,
April 24, 2007 @ 1:58 am
Here’s a feature suggestion for you:
Somehow there are 7,500 404 errors being reported in my Webmaster Tools. I have run many link checkers over the site and I can’t find these pages.
I’m able to download a list of the URLs and it would be very useful to be able to upload said file to the URL removal tool.
Could these 404s be external links?
Erwin Said,
April 24, 2007 @ 6:09 am
Very nice addition for Google.
I always used the 301 and I will keep using them, but this seems a nice improvement. Guess that pages that aren’t spidered often can be removed faster this way.
Tom Harrison Said,
April 24, 2007 @ 9:17 pm
Brilliant!! (And yes, long awaited, but still brilliant).
Thanks for posting this, Matt, and thanks to Vanessa and crew for a set of tools that just get better and better. The Webmaster console can totally change the relationship between sites and engines from combative to cooperative. Those of us willing to play by the rules (and I’ll admit to having been on both sides in the past) are now given a rich set of tools that allow us to *declare* our intentions, rather than to have Google have to try to figure them out. The tinfoil hat types will find life increasingly difficult.
The timing of the Delete tool is perfect. Over several years, we had tried to make a large and useful site based on content that was not original but which to a lesser or greater degree engendered the creation of useful and original content. It didn’t work. So we recently have killed tens of thousands of pages so that we can focus our resources and efforts on a few really remarkable and outstanding pages. Within a few days, we should be able to clear the decks of cruft, letting Googlebot spend only the time needed to find what’s still actually there. Efficiency is good.
In the absence of the webmaster console, and this new tool, the changes we have made would be hard to understand, and even downright suspicious looking. Now presumably everything is out in the open.
Hooray!
Anatoly Lubarsky Said,
April 29, 2007 @ 5:46 pm
Url removal tool does not work when robots.txt exceeds 5000 bytes.
If some url is blocked in such a file - removal request is denied.
The Dog Clothing Company Said,
July 16, 2007 @ 10:43 am
I just submitted a few URL removal requests but they are now pending. Does anyone know how long it takes before they are actually removed?
reggy Said,
July 18, 2007 @ 7:28 am
Hi Matt,
am finding this google webmaster tool error section very confusing:
I have errors under “URLs restricted by robots” , “Not Found” & “HTTP Errors” - the errors under not found and http are 410,404 and 403 - i have tried using the removal tool for some of the urls i want removed
what i would like to know is if 410,404 and 403 errors are displayed does this mean that google will remove the urls and if so how long does it take
Thank You!