And Juliane Stiller from Google’s German Webmaster blog stopped by the Googleplex for a more fun interview. Read the intro in English or German or just watch the video below:

Thanks for setting this up, Juliane! Note to self: wear a different shirt for my next SEO video interview. I happened to wear the same polo shirt for both interviews.

The reason for this [supposedly unlisted urls getting crawled --Matt], explained Ken Simpson, CEO of anti-spam company MailChannels, is that one’s Google Toolbar may be configured to pass URLs that one visits to Google for indexing. “If you run Google Toolbar, it knows pages you visit,” he said.

Sorry, but if Ken Simpson is implying that the Google Toolbar led to these urls being crawled, then he’s mistaken. Let’s take the first result from the [inurl:mms2legacy] query given in the article. The first url in that result set that I saw was http://mediamessaging.o2.co.uk/mms2legacy/showMessage2.do?encMmsId=F1ABCF6D326A3F65 . Well, if you take the string F1ABCF6D326A3F65 from that url and search for that then you’ll find multiple references to that url. In the cases I looked into, we found these pages via someone publishing a link on http://my.opera.com or other places around the web. I can definitively say that all the urls I looked into were discovered via crawling regular old links.

Folks with great memories may remember that I’ve talked about this before. Back in 2006, both Philipp Lenssen and Google OS did controlled experiments by visiting unlinked deep pages with the toolbar, and both concluded that the toolbar did not lead to those urls being indexed.

It’s good to reiterate this every couple years though, especially as Google has gotten better at finding new pages as it crawls. We get questions like this often enough that we have an FAQ answer about it:

Why is Googlebot downloading information from our “secret” web server?

It’s almost impossible to keep a web server secret by not publishing any links to it. As soon as someone follows a link from your “secret” server to another web server, your “secret” URL may appear in the referrer tag and can be stored and published by the other web server in its referrer log. So, if there’s a link to your “secret” web server or page on the web anywhere, it’s likely that Googlebot and other web crawlers will find it.

Security through obscurity is not a great way to keep a url from being crawled. If you don’t want your content in Google’s web index then we provide a ton of advice on how to prevent that content from getting into Google.

I’ve checked out a quite a few “we don’t have any malware” reports at this point, and I’ve yet to see a false positive — the sites in question have each had some malware on them.

Would you believe that a year and a half later, that’s still true for me? It may be possible that our malware flagging system has false positives, but I can’t recall a single case that I’ve seen where there wasn’t some security hole or malware that was a true issue for the website owner. If you want to know why, read Google’s white paper about how we detect such stuff -- it’s called The Ghost In The Browser Analysis of Web-based Malware and it was written by Niels Provos and several other Googlers.

In fact, just last week I handled a very similar case where Google proactively reached out to a website that had a scripting flaw security. The deja vu from my January 2007 post plus the situation last week made me want to write a generic malware debunking post. Are you ready? Here we go:

$ACCUSER = Brett Glass
$FORUM = Dave Farber’s Interesting People mailing list, specifically this email.
$LONG_ACCUSATION = (I’m going to quote Brett’s whole email here, just for context)

Everyone:

Google has been a strong supporter of the agenda of Free Press, an
inside-the-Beltway lobbying group which has spent hundreds of
thousands of dollars lobbying for regulation of the Internet under
regime known as “network neutrality.” While some of the tenets
included in this agenda are not reasonable, one of those that IS
reasonable is the notion that large corporations such as Comcast
should not block content with which they disagree.

However, Google -- itself a large corporation -- appears to be
blocking a site which expresses opinions with which it does not
agree on this very issue. When one does a search for the terms
“neutrality” and “site:pff.org” (the link

http://www.google.com/search?hl=en&q=neutrality+site%3Apff.org&btnG=Google+Search

will perform this search for you), many of the pages and documents
on the site -- in particular, white papers expressing views with
which Google disagrees -- are tagged with a warning that “This site
may harm your computer.” One cannot click through to the documents
and pages in Google’s search results without cutting the URL from
the page and manually pasting it into one’s browser.

The Web site, operated by a group known as the “Progress and
Freedom Foundation,” does not appear to contain any malware. When
one queries Google as to why the site was blacklisted, it claims
that “Part of this site was listed for suspicious activity 1
time(s) over the past 90 days.” Yet, we could find no malware or
other exploits in the blacklisted PDF files, some of which contain
very well presented and cogent arguments against the agenda which
Google has been actively supporting.

Could it be that Google (whose motto is, reportedly, “Don’t be
evil,”) saying, “Do as I say, not as I do?”

--Brett Glass

P.S. -- What’s especially interesting is that if one queries Google
using just the term, “site:pff.org” (you can use the link

http://www.google.com/search?hl=en&q=site%3Apff.org&btnG=Search

to do this query), one can see that the majority of the supposedly
dangerous site is not blocked. But most or all of the documents
expressing viewpoints on “network neutrality” are.

$SHORT_ACCUSATION = “Google blocked a site with opinions that it disagrees with. Worse, the query [site:pff.org] seems to show that only urls under pff.org/issues-pubs/ are labeled as potentially harmful, and that is the directory where many of the documents that disagree with Google are.”

Given what we have so far, my generic debunking would begin like “Dear $ACCUSER, I saw on $FORUM where you mentioned that Google is flagging a website as malware. You said that $SHORT_ACCUSATION. I wanted to give you a little more background and context to let you know that Google did see an actual malware attack via a real security hole. The other thing you need to know is that Google flagged the site because of the security hole, not because Google agrees or disagrees with any particular content on the site.”

Then I’d give a little background history on all the different ways that Google helps users and webmasters avoid malware. Most of the background would come from this overview post. Since that post was published in mid-2007, Google has done even more to protect users:

- Niels Provos and his colleagues published another technical report with more details about the malware detection framework and what it discovered (more info here).

- Google launched a Safe Browsing API so that third party applications can benefit from Google’s list of malware and phishing urls. If you appreciate that Firefox 3 has better security, one of the reasons is that Firefox 3 utilizes the Safe Browsing API.

- More recently, the anti-malware folks at Google launched a Safe Browsing Diagnostic page where you can enter a url and get a ton of really useful information.

The last one is especially impressive. For example, check out the Safe Browsing Diagnostic page for pff.org:

That page gives a ton of helpful info to site owners and anyone else who is interested in why a particular site or url was flagged as potentially harmful.

All that would go quite far to reply to people that had questions about their site being flagged for malware. But this post is getting quite long, so let’s get back to this specific report in this case. The original person who reported this situation had already noticed that not all of pff.org was flagged. If you do a site: query on Google, you only see warnings for pff.org/issues-pubs/ .

If you visit pff.org/issues-pubs/, you’ll see that it’s a web form. It looks like pff.org stored their data in a SQL database but didn’t correctly sanitize/escape input from users, which led to a SQL injection attack where regular users got exposed to malicious code. As a result, normal users appear to have loaded urls like hxxp://www.ausbnr .com/ngg.js and hxxp://www.westpacsecuresite .com/b.js <--- Don’t go to urls like this unless you are 1) a security researcher or 2) want to infect your machine. Notice that even in this case, Google didn’t flag the entire pff.org site, just the one directory on the site that appeared to be dangerous for users.

I never like it when people accuse Google of flagging a site as malware just because we don’t like it for some reason. The bright side of this incident is that pff.org will find out about a security hole on their site that was hurting their users (it looks like pff.org has disabled the search on the vulnerable page in the last few hours, so it appears that they’re responding quickly to this issue). Flagging malware on the web doesn’t earn any money for Google, but it’s clearly a Good Thing for users and for the web. I’m glad we do it, even if it means that sometimes we have to write a generic malware post to debunk misconceptions.

You may want to read this paper about the Google cluster architecture if you haven’t already, because I’m going to remind you of two things about Google that are pretty obvious in retrospect. You can think of the Google cluster architecture as a bunch of moderately powerful personal computers connected by ethernet. That’s not quite correct, but it’s a pretty good abstraction. In that model, you have pretty good disk/RAM/computational throughput, but network communication is much more limited. That leads to the first nice thing about Protocol Buffers: they’re very compact going over-the-wire via network.

To understand the other nice thing about Protocol Buffers, bear in mind that in the Google cluster architecture, there are many different types of servers that talk to each other. Question: how do you upgrade servers when you need to pass new information between them? It’s a fool’s game to try to upgrade both servers at the same time. So you need a communication protocol that is not only backward compatible (a new server can speak the old protocol) but also forward compatible (an old server can speak the new protocol). Protocol Buffers provide that because new additions to the protocol can be ignored by the old server. That lets you upgrade different servers at different times (check out the “A bit of history” section in that overview). Protocol Buffers are especially appropriate to represent requests and replies between a client and a server.

(By the way, congrats also to the folks that worked to release this code outside of Google. Making open-source code available to the outside world is a great way to build goodwill with developers.)

There are over 10,000 .proto files in use at Google, and Protocol Buffers are a vital part of Google. If you’re a programmer, why not try Protocol Buffers out for yourself?

July 1st is also a good time to sit down and ask the question “What do I want to accomplish during the rest of this year?” I’ve been talking to various people on my team about which projects to tackle next, and I wanted to ask for your feedback too.

In the comments, feel free to suggest projects that you think Google should work on next in webspam. I have a comment policy and I’ll reserve the right to prune comments that don’t contribute to the discussion. But if you have a constructive, polite suggestion then I’d be interested to hear it.

The one other thing I would ask is to please think about your suggestion before reading the other comments. If people read the other feedback first, the suggestions won’t be as independent.

I’m a fan of this change, and I’m a fan of Adobe in general. They get a lot of credit in my book for opening up the specifications for PostScript, Adobe’s font standards, and their Acrobat/PDF format. Very few companies have been able to open up their specs and still compete successfully against powerful opponents. I respect Adobe for that -- not to mention providing one of the first widely-known “plug-in” mechanisms (in Photoshop). The idea of a plugin or extension has greatly affected how people view software from Firefox to WordPress. I’m glad that this most recent change by Adobe will make it easier for search engines to index content in Flash files.

Fun trivia: This video was taped in the lobby of building 43 on the Googleplex campus. Also, I made sure to wear my “We ♥ webmasters” shirt from our webmaster portal team up in Kirkland.

I think we’ll try to get an official recording up soon with answers from the Q&A, but Barry Schwartz has been working hard the last couple days. He was smart and recorded the chat audio as an MP3. He also copy/pasted a raw dump of the Q&A. You can hear/read both over on his post.

If we get a cleaned up version of the Q&A where we have a chance to correct any typos or similar stuff, I’ll add a pointer here. In the mean time though, there’s a huge amount of good information in the Q&A section. It was wild because we had maybe 15 people in one room in Mountain View (and more in Kirkland and Zurich), and a bunch of Googlers were just answering questions as fast as they could.

I thought the chat went really well, so I’ll probably press for us to continue this tradition. In the mean time, lots of people have been talking in Google’s webmaster discussion group.