Matt Cutts: Gadgets, Google, and SEO

I communicate with people in lots of ways: face-to-face, email, via my blog, leaving comments in the blogosphere, conferences, etc. At SMX West a couple people asked “I sent you a friend invite on service X but you haven’t responded. Do you not like me?” Please don’t feel bad, because it’s not that. I’m letting a lot of requests drop on the floor — even requests from other Googlers to chat on Google Talk. I did a quick check of various social services and here’s what I found:

LinkedIn: 176 invitations to connect
Twitter: 671 requests 1060 requests
Google Talk: 27 chat requests
Facebook: 190 friend requests
MySpace: 35 friends, and it’s a fake account that someone else set up in my name (I’m not 42 years old, thank you very much ).

At this point, managing friend invitations feels more like work than fun. Many of these services have really poor interfaces for mass approving, and a while ago I discovered that if I stopped responding to friend requests, very few people got angry with me. So if I haven’t responded to a friend request from you, please don’t take it personally — I’m just a little socially exhausted.

By the way, I have a precise measurement of being Calicanissed. He told his twitter following to add me, and I got almost exactly 400 additional twitter requests. Jason didn’t know it, but I had my twitter set to the private mode that requires each twitterer to be approved. Thanks, Jason.

Popular Mechanics asks 20 questions of Udi Manber, who is a VP of Engineering at Google on core search quality. My favorite:

There have been a lot of fads in search of late, such as Human Assisted Search and contextual search. Do those get folded into search as a whole? What are real trends in search and what are fluff?

So let me first tell you about Google. At Google we do not manually change results. For example, if we find for a particular query that result No. 4 should be result No. 1, we do not have the capability to manually change it. We made that decision not to put that capability in the algorithm—we have to go and actually change the algorithm. That is, we have to find what weakness in the algorithm caused that result and find a general solution to that, evaluate whether a general solution really works and if it’s better, and then launch a general solution. That makes the process slower, but it puts a lot more discipline on us and makes it more unbiased.

That’s the right answer for a general/Popular Mechanics audience. For the nitpicking search junkies that read here, I’ll just add that we are willing to take manual action on a small number of issues like webspam and removals for legal reasons. The rest of the interview was also interesting.

One other interview: A few months ago I talked with John Jantsch for the Duct Tape Marketing blog, but I don’t think I pointed the interview out to folks here. The MP3 is up on the site, if you want your search engine optimization (SEO) interview fix. I think this is a pretty good interview if you’re a small business owner or a little newer to SEO.

Speaking of MP3s, I’m caught up on my Daily SearchCast and I’m looking for maybe one other podcast to add to my listening rotation. What podcast would you recommend that I try? I don’t mind something outside of search — in fact, a non-search podcast might be nice to listen to something different.

Sometimes I feel like the technology space moves slowly. Cool new devices appear every few months, but I want neat new things every day! When I feel like this, it’s tough to remember that technology moves quite quickly compared to most industries. I was recently at a book sale and picked up a techno-thriller from 1996 called Back Slash. As pulpy books go, it wasn’t half bad. Until I arrived at this passage about twenty pages into the book:

To the right of the desk, in an oak cabinet custom-built by Crane, were three midtower computer cases. Each housed a Pentium-based computer system capable of 166 MHz processor speed. Each had 128 megabytes of Random Access Memory (RAM) and a 1.6-gigabyte hard drive. The video card of each held two megabytes of memory, and he could channel the output from the three machines to either of his two monitors. He could also link them in parallel for greater computer power. Twenty thousand bucks, right there. ….

Directly above the desk, the shelves held a variety of easily accessible accessories: a 5-1/4-inch floppy-disk drive–in case he ever needed it, two 3-1/2-inch disk drives, two one-gigabyte tape-backup drives, three multidisc CD-ROM players, two 28.8-kilobytes-per-second fax modems, and on one shelf, ten 4.3-gigabyte hard-disk drives.

Crane figured he could store much of the Pentagon’s data here if he wanted to have their crap on hand.

I had to put the book down and leave it. The description of a “cutting edge system” was so jarring that I could no longer suspend my disbelief. A videocard with two megabytes of memory? Geez. It makes 1996 feel like this:

[Image CC-licensed by Steve Jurvetson.]

It makes me want to rev up my grumpy-old-man voice:

“Back in my day, we had 300 baud modems and we were grateful! Sometimes you’d type too fast and you’d have to wait for the modem to catch up.”

“You know, in our high school typing class we had to use mechanical typewriters. No joke.”

“We had to type programs into our Commodore 64 from magazines. And in those days, the magazines didn’t even have checksums!”

What old timey technology story would you tell?

The title pretty much says it all. A while ago, someone saw my call for good summer vacation reading and the resulting pile of Amazon books that I bought, and they sent me a couple free books, maybe to get a review or a mention. I appreciate the creativity, but please don’t send me any books or other free stuff. If you’ve got a new book coming out, I’m happy to hear about it, but if I decide to read or review it I’ll buy my own copy.

A while ago, someone sent a big cookie with a “No spam” message like this:

I appreciate the thought, but please don’t send me any free stuff. Google has a gift policy to avoid potential conflicts of interest. Even if Google didn’t have such a policy, I wouldn’t want to accept any gifts of value, because it’s important to avoid even the appearance of impropriety. Usually I just give away any unsolicited stuff that gets sent my way. Thanks.

Okay, go read this post on the Google webmaster blog. In fact, if you read my site, you really should add the Official Google webmaster blog feed to your list of subscriptions, because that blog is almost 100% SEO/webmaster-related posts, and it is official. Done reading? Okay, I’ll give you my personal take on why I like this idea.

I’ve done a lot of site reviews in my time. Many of them go like this:

Webmaster: Matt, can I get a site review for ExampleCo??
Me: Hey, I’ve heard of Example. I really like your red widgets.
Webmaster: Thanks! We’re rolling out a new line of blue widgets this fall. The site is example.com.
Me: Okay, let’s take a quick look.

(small chat about blue widgets until the site loads.)

Me: Huh.
Webmaster: What? What does “Huh” mean?
Me: Well, when I visit www.example.com I get map of the world and then at the bottom of the page there’s a dropdown to select which country version of Example to go to next.
Webmaster: Right. Example is a big business with lots of different country-level domains, so we have to ask the user where they want to go. Why, is that a problem?
Me: It sort of is. Dropdown boxes and forms are kind of like a dead end for search engine spiders. Historically we haven’t crawled through them.
Webmaster: But it’s just a dropdown box with ten countries listed. You can’t just crawl that?
Me: Not really. Think of search engine spiders much like small children. They go around the web clicking on links. Unless there’s a link to a page, it can be hard for a search engine to find out about that page.
Webmaster: But it’s just ten countries. Couldn’t the search engine just pick one of those values and keep going?
Me: In theory you could do that, but in practice the major search engines don’t usually do that.
Webmaster: That sucks. I like how clean the page looks. Is there a way around that?
Me: Sure. You could put the list of countries at the bottom of the page and make them hyperlinks so that Googlebot can crawl through to the other urls. A good rule of thumb is to take a look at your site in a text browser like Links or an ancient browser with JavaScript/CSS/Flash turned off. If you can reach all your pages just by clicking regular links, your site should be pretty crawlable.

I’ve had this conversation a lot over the years. Savvy webmasters and SEOs know how to make a site crawlable, e.g. making sure that someone can reach every page on a site via normal HTML links. But the web is filled with sites that have a dropdown box or some other form that search engines typically didn’t know how to handle.

Now Google is finding ways to crawl through forms and drop-down boxes. We only do this for a small number of high-quality sites right now, and we’re very cautious and careful to do the crawling politely and abide by robots.txt. If you’d prefer that Google not crawl urls like this, you can use robots.txt to block the urls that would be discovered by crawling through a form. But I hope that the dialog above is a pretty good example of why this new discovery method can be helpful to webmasters.

Danny asks a good question: if Google doesn’t like search results in our search results, why would Google fill in forms like this? Again, the dialog above gives the best clue: it’s less about crawling search results and more about discovering new links. A form can provide a way to discover different parts of a site to crawl. The team that worked on this did a really good job of finding new urls while still being polite to webservers.

By the way, I wanted to send out props to a couple people outside Google who noticed this. Michael VanDeMar emailed me a little while ago to ask about this, and Gabriel “Gab” Goldenberg recently noticed this behavior as well. I appreciate them discussing this because it encouraged Google to talk about this a little more.