Pages blocked by my robots.txt file are slowly being listed on the “Restricted by robots.txt” page. The number was slowly going up til it got to 23.
It should have continued to go up as Google continued to index my site, since there are a lot more blocked pages.

However, now the number is going down. It is at 22.
So pages that are blocked are disappearing from the list.

Is this normal? I don’t see how it can be.

I have checkd the robots.txt file and url’s in Google and everything seems to be working properly.

I also wish Google woudn’t take so long to index my entire site. It is not that big. Very frustrating.

Hopefully you can assist me, we are dumbstruck by what has happend with our site we are developing currently we launced our indexing to google in July and we` got a very high ranking first page well a few weeks in an suddenly we have lost our indexing and placement but still in some search engines we are on the first page, the other strange thing that has happend is the page we were shown on is showing other companies with LINUXOS or Linux Os Solutions in there titles etc…

What we have done is we have looked in our webmaster tools and we can see our rankings were high through google but now we are struggling to have our description etc, through google indexed, I have been looking at our robot txts etc but would appreciate if someone could explain what had happened..

Many thanks for your time

My webmaster tool was indicating the following error with respect to my sitemap.xml. “URL timeout: robots.txt timeout
We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.”

My sitemap was generated by following the guidelines and it was definitely in the right location.

Can you please help me to resolve this problem?
Many thanks

After I created my robots.txt, using the one on wordpress as a template, I noticed none of my new pages were being indexed anymore. They use to be indexed immediately and then NOTHING.

I had used these recommended lines:

Disallow: /*?*
Disallow: /*?

I did this particulary because a survey form I use creates multiple pages with the ? that don’t need to be indexed.

However, for some reason (I don’t know why), the googlebot sees many of my pages with this type of permalink: http://www.thisishowyoudoit.com/blog/?p=57

However, my permalink structure looks like this: http://www.thisishowyoudoit.com/blog/10-reasons-why-not-to-host-your-wordpress-blog-on-a-windowsiis-platform/

Thus, after implementing my robots.txt, many of my pages were not longer indexed.

Just wanted to warn people about these particular lines in the robots.txt file.

Thanks,
Richard

In the two weeks since the redesign, I had been seeing URLs only with no snippets in the SERPs.

Last night, I finally figured out what the problem was. Not taking any chances, I updated the WordPress settings AND manually uploaded a robots.txt file (the WordPress one is somehow invisible!).

This morning, still no change. In webmaster tools, Google is still showing the old robots.txt. How long do I have to wait?

I’ve noticed over a few weeks that it’s not working properly.
It displays the correct robots.txt and you can check URL’s against that.
However if you make any edits in the tool, running a check appears to use the cached version of the file rather than the locally edited version. Kind of defeats the purpose of the tool really.

I hope Google fix this soon as when it works, it’s a God send!

I know what changes to make to the text, but not how to upload them to the root directory. I tried inserting the text in the template, but i doubt that will work.

I am not sure Blogger allows us to upload HTML to blogger templates. Anyone have any advice?