It seems as if every 5-6 weeks, the Google webmaster console team rolls out more requested features into Google’s webmaster tools. You can read Vanessa’s post about the latest good stuff. Danny covers it pretty well over here, but I thought I’d show snapshots from my site.
Here’s what my site looks like:
As you can see, my site is pretty small (I’ve only written a few hundred posts). On average, Google pulls down 438 of my pages each day. And my domain, while not super-speedy, normally returns pages in under 3/4ths of a second.
The other thing to notice is the crawl rate. Google limits how hard it hits web servers with something called “hostload,” which is a measure of how many bots can be simultaneously fetching pages from a web server. Notice how “Faster” is grayed out for my site? That means that hostload and crawl-rate isn’t anywhere near a limiting factor for my site. Heck, even a single Googlebot could fetch pages at a leisurely pace and still crawl most of my site every day. 🙂
But suppose I ran a large domain such as wordpress.com or geocities.com, or a site with thousands of pages. Then hostload could potentially be a factor. Even if N bots are allowed to fetch pages simultaneously, those bots might not be able to fully crawl a site in the amount of time that we crawl before beginning indexing. If hostload is a factor for your site, the “Faster” option will be available to you. In that case, if you’re willing for Google to crawl your site harder, we should be able to fetch and index more pages from your site.
Of course, you can always opt for slower crawling as well. If you’d like less load on your webserver but don’t want to block Google completely (e.g. with robots.txt), requesting a slower crawl is a good idea.
Let’s see, what else is in this release? Oh, you can opt-in to having your images labeled in the Google Image Labeler game. If you want people to provide free labels for your images, that’s a great reason to try out the webmaster console right there.
The final feature is just a count of urls that we found in a Sitemaps file. As Vanessa said in the official post:
Recently at SES San Jose, a webmaster asked me if we could show the number of URLs we find in a Sitemap. He said that he generates his Sitemaps automatically and he’d like confirmation that the number he thinks he generated is the same number we received. We thought this was a great idea.
This is the perfect example of how things should work to me. The webmaster console folks have their own ideas on what webmasters will find useful. But talking to webmasters is the best way to hear what people really want (this particular idea came from Tim Jackson at Plumber Surplus, for example).
So what do you want to see from future versions of the webmaster console? Philipp put out a call a month ago and got over 60 comments. I know the webmaster console team will read the feedback here as well. So what should we do next? And if you haven’t tried the Google webmaster console yet, please give it a test drive. It may help you find problems with your site, and it will offer more and more information over time.