This “hairball” post about Cuil isn’t really snarky, so I’ll post it. Cuil is no longer around, but it did spawn a funny post on Reddit about Cuil Theory.

Cuil launched this week. For a search engineer, a new search engine is like a Christmas present: you can’t wait to play with it. Most search engineers can get a good feel for the strengths/weaknesses of a new engine within 10-15 queries. And I’d like to think that with another 5-10 queries, I can usually figure out how I’d spam a search engine. It’s my job to protect Google’s index from spam, so naturally I’m intimately familiar with different webspam techniques.

What’s also fun is to figure out the how a search engine provides various features. For example, for a Cuil search like [matt cutts] you’ll see the following categories:

Where do those categories come from? Most people didn’t drill down that far, but it’s quite doable to figure out. If you want, take a few minutes to see if you can puzzle out how the categories are generated before reading on.

Google OS figured it out, for example: “Another interesting idea is the explorative category section that shows related Wikipedia categories and topics.” With a little work, it’s easy to verify that the right-hand box comes from Wikipedia category pages. For example, the string “matt cutts” occurs on the Wikipedia page for search engine optimization, and that page also includes a link to a search engine optimization consultants page. Sure enough, one of the categories listed for [matt cutts] is “Search Engine Optimization Consultants” and the entries under that category are from Wikipedia. Likewise, I think the Wikipedia page for Traffic Power and its link to a category page for black hat SEO probably accounts for why the category “Black_hat_seo” appears for my name.

There’s nothing wrong with surfacing Wikipedia category pages, of course, but sometimes that can lead to some drift in topicality. For example, p2pnet wrote about a search for their name: “[The search query] p2pnet.net, however, gave Canadian copyright law, Project Gotham Racing Series, file sharing networks, Wired magazine people, and filesharing programs.” You can see the categories for the search [p2pnet.net] below:

And this Wikipedia page has the string “p2pnet.net” and also has a category page for “Project Gotham Racing series”. The idea of surfacing Wikipedia category pages will have advantages and disadvantages depending on the user and the query.

Don’t conclude from my previous post that I dislike journalism. All through middle and high school I woke up early to read the local newspaper each morning. I was the editor of our newspaper in high school. My mother wanted me to be a journalist. I’ve been thinking of the issues confronting journalism for a few years now.

Back in early 2007, a journalist friend in the Midwest emailed me. He saw the impact of the web, changes in the newspaper industry, and he was worried about his newspaper’s–and his own–future. He asked my opinion on all of this. With his permission, this is what I wrote to him back in 2007 with a few minor edits.

Definitely take me with a large grain of salt–I got lucky in joining Google, but I wouldn’t give my opinions any more weight than an average person’s. My personal hunch is that newspapers will have some issues in the years to come. If you think about the fraction of revenue that comes from classified ads, it does seem that revenue will eventually migrate online, and sites like craigslist.org are more likely to capture a big fraction of that traffic compared to individual newspapers or even newspaper syndicates. If a newspaper loses ~30% of classified ad revenue over 5-10 years, that’s really hard to adjust to without structural change.

It’s funny because my Dad basically took a job out of grad school and stayed in the same post until he retired. It seems like the odds of that happening for people like you and me are a lot lower. There’s just not as many companies that are doing things like taking care of their workers for 30 years at a time.

So the first thing I’d recommend is to grab a domain name and work on burnishing your personal reputation online. It’s definitely not the case that everyone needs a blog, but having one place that acts as a face to the world can really help. There’s room for a resume/CV, but also for some writing samples that show off your abilities.

Which takes me to how open-minded [Midwest newspaper] is. I’ve heard newspaper policies ranging from “If you blog, we’ll fire you” to “If you don’t blog, we’ll fire you.” I hope that the paper is pretty open-minded. But they shouldn’t be able to stop you from building up your reputation online in your own time, and even if there’s copyright issues with putting full articles up on your personal site, you could no doubt quote a few excerpts of choice stuff as a part of fair use.

So making the switch to a mental model where you are more like a consultant for any company that you work for, but you look for ways to improve your reputation and learn new skills as you go–that’s a good way to make sure that you’re protected if you unexpectedly end up as a free-agent.

You’ve got a good sense of humor and you’re well-spoken, so the biggest questions to me would be
- what do you love or what are you interested in?
- where do you want to be in 5-10 years?

For example, it would be interesting to know a little bit about your interests. Things like games, gadgets, politics, or technology make great subjects if you want to try some active blogging + something like AdSense to make a little bit of money on the side. But some of the larger issues are things like
- how introverted you are vs. how much you enjoy talking to people.
- what ties do you have to [Midwest city], and do you want to stay around there or are you willing to move?

I’ve noticed that networking and getting to know a few people in an industry can make a big difference. If the tech-field is interesting to you, [Midwest city] is going to be a more limited pool of opportunities compared to something like Silicon Valley/San Francisco or New York. If you like to travel and like meeting new people, it turns out that becoming an expert in a niche and then getting on a conference speaker circuit can be good. You might start out on panels about journalism or media or ethics, but that could quickly lead to consulting gigs, for example.

I do think that the tech industry will be a leading one for the next 10-20 years, and probably biotechnology will start to emerge after that. But I think the service economy will remain strong throughout. Starting to get on a conference speaking circuit is really a way to rebrand yourself as an expert on some topics. That role would let you expand and offer your services/advice in different ways.

I guess the larger issue is that working for a salary is great, but if you can find ways to participate more directly in the success of a company, that can be a faster way to make money. The whole dot com boom demonstrated that there are a lot of dumb start-ups out there, but at the same time, when you’re young is exactly the right time to take risks like that.

Re-reading your email, I guess the smallest step forward would be to find out what you can legally do online that wouldn’t conflict with your employer’s guidelines. Then I’d just try a few experiments. It doesn’t cost much to buy a domain name, so you might consider starting a blog about [Midwest city] or a news site (probably the blog is a little easier to start). Set aside $100 or so (or ask for someone to register a domain for you as a birthday present) and try a few things. Sign up for AdSense and see what sort of articles do well on places like techmeme.com or reddit.com or digg.com. In general, I’ve found that starting with a small niche and building your way up is great practice and teaches you a ton about what sort of things attract attention and good discussion.

Some of that advice has aged well, some less so. I still think that grabbing a domain and experimenting is invaluable. I believe that the entire world is being digitized–from businesses and places to people–and it’s better to be involved in that process than to stand passively and let other people define you online. I believe that participating in the upside of a company is better than only drawing a salary. And I think that most of the time, no one cares about your career ladder or skills development as much as you do. No good company opposes the development of its employees, but ultimately you have to take the initiative and drive your career in the direction you want.

By the way, the title of this post is an allusion to Letters to a Young Journalist, an excellent book by Samuel G. Freedman.