Google has a pretty good culture of doing postmortems. When something fails, someone close to the failure tries to document what happened and why. A good postmortem document should also point the way to avoid similar mistakes in the future. Mistakes happen, but you don’t want to make the same mistakes over and over again. Instead, it’s important to try to get to the root of a problem and fix it there. Failure can be a good thing if you learn valuable lessons along the way.

But it’s also a rule of thumb of software engineering that it’s 10x harder to catch a problem at each new level of deployment. That’s why solid tests (e.g. unit tests) are so helpful; it’s much easier to fix a problem when you catch it earlier. Likewise, requiring code reviews before submitting code changes can avoid lots of stupid mistakes. Once buggy or poor code is checked in, debugging a problem can get much harder. And if a problem makes it out into production, it’s typically even more difficult to fix.

When you take a solid practice like postmortems and think about going further upstream, you land on the idea of a “premortem.” An example of a premortem would be going to each member of a team before a product launches and asking them: “What’s going to break or fail? What’s the mostly likely thing to go wrong?” After all, the people who have been working hard on a project have been steeped in it and are intimately familiar with potential weaknesses and failures.


I first heard the idea of a premortem from a Freakonomics podcast, but the idea is so simple that it practically explains itself. However, there are a couple subtleties. First, don’t bring everyone into one big room and ask “Hey, what’s going to go wrong?” That’s a recipe for groupthink unless there are a small number of glaring problems.

Instead, you want to collect the initial feedback independently and privately so that people won’t be biased by hearing what others are saying. With private feedback you might end up hearing opinions that people are afraid to express in public. If you do have to settle for a big group meeting, ask people to channel their personal voice of doom on their own before opening up the public discussion.

The second subtlety is that “dogfooding,” the process of testing your own product internally before introducing them to the world, is almost like a premortem if you can get good feedback internally. The touchy issue here is authenticity: people want their feedback to be taken seriously, but internal feedback might be biased or skewed for various reasons. Even if you disagree with internal feedback, it makes sense to take a clear-eyed look at what people are saying. And if you do disagree, it helps to explain your reasons for adjusting to feedback or not.

Could a pre-mortem help your next project avoid a massive failure? Why not give it a shot to find out? Premortems can be an easy and fast exercise, and you might get some really useful insights. Just ask the people close to the launch to brainstorm “What’s most likely to go wrong?” before the project launches.

Watching Anita Sarkeesian at XOXO

I had one more experience at the XOXO Festival that I wanted to mention. I really enjoyed Anita Sarkeesian’s talk. You can watch it here:

Sarkeesian explained her experience with humor and grace, and that really resonated with me. I don’t want to join the tone police–passionate voices have a role in this discussion too, and passion may work well for others. But I know it can be hard to take abuse while making your case with civility, and I admired Sarkeesian’s ability to rise above the fray.

As part of my job, I’ve unfortunately become somewhat of a connoisseur of vitriol and threats. My first death threat was over a decade ago in a situation involving the DMCA and the Church of Scientology (before you jump to assumptions, the death threat came from the anti-Scientology side). I got a threat at a 2002 search conference that I considered credible enough that I started carrying a cell phone with me after that. I got an open-ended threat against my family just a couple weeks or so ago, even though I haven’t been working on webspam for months.

But here’s the thing: I’ve never received threats as pointed, menacing, or explicit as Anita Sarkeesian, Zoe Quinn, and others discussing GamerGate have. No one should face threats of physical harm for expressing their opinions. No one should be doxxed or have their personal information posted just for expressing their opinions. That should be the starting point and the bare minimum for any discussion. If you disagree with someone, win them over with your ideas, not with threats.

I should mention that I’m a big fan of clear disclosure of potential conflicts of interest, and I’ve posted my own disclosure page at the top of my blog for over five years. I’ve also been playing computer games since Pong in the 1970s. As a kid, I wrote a script to solve Colossal Cave Adventure on a local university’s PRIMOS system. I subscribed to Electronic Games magazine back when people called them “coin-op” games. By the way, check out that Electronic Games link to see how Nintendo tried to avoid sexist language in games back in 1993. Hell, I feel bad for people who never got to play Raiders of the Lost Ark on an Atari 2600, or Infocom games on a Commodore 64, or marvel the first time they saw the parallax effect in Moon Patrol:

Moon Patrol!

The gaming world is changing, and in my opinion for the better. We’ve got browser-based games like Kingdom of Loathing or Candybox2. We’ve got absurdist wonders like Progress Quest and games you play outside like Ingress. Playing Depression Quest was important for me, because I have friends who are deeply affected by depression. I can’t wait to see where gaming goes next–how about we make virtual and augmented reality work this time around! I hope that gaming can be even more welcoming to new ideas and experiences than it was when I was a kid. I also hope everyone can agree that doxxing and threats aren’t ever welcome.

Fostering open source services

Open source is really good at creating products. Almost any commercial software package or product like Word, Excel, Windows, or Photoshop has a great open source equivalent. However, open source has been less successful at creating services. Where’s the open source version of Google, or Facebook, or Twitter, or Gmail, or Craigslist?

You could sum it up with this drawing:

Where are the open source services?

Now to be fair, the bottom-right box isn’t completely empty. There’s Wikipedia, which is a phenomenal service/website supported by donations. There’s Tor, where many companies and people volunteer to run relays and bridges. There’s BOINC, which is the open source software used by volunteers for SETI@home and Folding@home. There’s also OpenStreetMap, which is a wonderful resource.

But why aren’t there more open source services? Let’s run down some differences between products and services.

“One and done” vs. ongoing support

With open source products, it can take a lot of work to create something great like Linux or Firefox, but then everyone can download that product and use it immediately–there’s no extra cost for the producer or the consumer, other than maybe a bit of bandwidth for downloading.

Once a product is done, it’s often done–frozen until the next major update. A product might take a year or more to reach a milestone, but it can often be used for years after that. In contrast, services may change from week to week, which implies strong product leadership to determine priorities.


If you download a copy of LibreOffice, you might write some unpleasant things or even hate speech, but that doesn’t hurt LibreOffice itself. However, if someone sets up a “free as in beer” translation API or geocoding API, you often see multiple levels of abuse. For example, some people might use a service so much that it overloads the service provider. Or people might scrape the translation API in an attempt to generate spammy text in lots of different languages. When you offer a product, potential abuse is usually less of an issue.

User Experience and Speed

Products don’t have to be perfect; often “free as in beer” is enough of a feature that someone will use GIMP as opposed to paying for Photoshop. But user experience and speed do matter, and commercial services have a strong incentive to nail both of those issues. It takes a ton of work to be fast, for example. Commercial services are often “free as in beer” as well as fast and pleasant to use.

Funding models

Thanks for staying with me so far, because I think this is the most important difference. I believe what might be missing is a good funding model for open source services. With a finished product, if you can find someone to donate bandwidth for downloading and maybe a simple website, you’re close to done. But with a service, there’s typically an ongoing cost involved with every API call. For something like web search, there can be a lot of processing work going on behind the scenes.

So what are the major funding models that might support an open source service? Right now, I can think of ads, occasional pledge drives, grants, subscriptions, or micropayments. From that list, my guess is that ads are the least appropriate. If ads are easily separated or can be blocked, then you might get a “free rider” problem where someone could take your service, remove all the ads, and offer it up as their free service. Personally I think advertising can be incredibly useful and responsive to a user’s needs, but some other individuals dislike ads. Ads can be the foundation of a freemium or hybrid approach; for example, I think Automattic offers free blogging on and funds itself partly through ads.

Regarding pledge drives, Wikipedia is a notable success, but it’s a lot of work on both the producer side and the user side, much like public radio (by the way, you can donate to Wikipedia here). Grants can work well, but grants tend to end after a few years, so they aren’t a complete solution to sustainability.

That leads me toward subscriptions or micropayments. I’m excited to see some movement in this area. Patreon lets you support your favorite creators and does at least a couple smart things. First, they only take 5% of donations. That puts them in the “doing it for the love” category. Patreon can be beloved while still making some money ($1M in donations each month * 5% cut * 12 months means >$600K/year). Second, they attempt to minimize payment fees by charging only once a month for all the people you support. So if you’re supporting four creators, then the credit card charges are split four ways. The first move is brilliant, and the second is very smart.

Bitcoin is another possibility for micropayments, although it’s still early days for that. I’m also excited to see Google Contributor launch. The idea is that a user contributes a certain donation each month. As the user surfs around a participating site, they don’t see ads on that site, but the site still gets paid from the user’s contribution.

Ultimately, I don’t know how to foster more open source services. I just know that I want them. In the same way that Firefox pushed Internet Explorer to improve or Apache pushed IIS, I personally would like having an open source search engine to push Google as well. Wikia Search was an attempt at that, but it didn’t get much traction.

Maybe the answer isn’t funding. In a recent talk, Melody Kramer floated the idea that people could support public radio in *tons* of different ways like volunteering their time or experience, not just with money. Maybe we need better ways for companies or regular people to volunteer their CPU, storage, or bandwidth. If we all kicked in 10% of our free disk space, could we come up with open source versions of Dropbox, Box, or Google Drive?

So I don’t have the answer. I just think it’s an interesting and perhaps an important problem. Do you agree? How would you foster more open source services?

Powerful USB chargers

If you’re a geek like me, there’s probably a bank or cluster of micro USB chargers somewhere in your house for recharging phones, tablets, Kindles, headphones, etc. Lately I’ve been playing with a couple USB chargers that I really like.

One is a USB charger with 3.5 amp (!) output. Just for context, a typical micro USB charger might be one amp. So this adapter has the potential to charge USB devices much faster than a conventional charger.

The other USB charger is 4A, but with dual micro USB plugs. So each micro USB plug puts out 2 amps–which is still quite a lot. I especially like this charger because it only takes one power outlet, but provides two very capable outputs.

If you haven’t levelled up your USB chargers recently, it might be time to take a fresh look. Or this could be a good gift or stocking stuffer for any geeks on your holiday shopping list.

Come to think of it, what other geek gifts would you recommend?

Improving your account security

Every year or so, it’s worthwhile doing an audit of your online security. The most important accounts to protect are your bank accounts and your email accounts. Here are some things to consider doing:

– Choose strong passwords. Just as important: don’t re-use the same password across web services. Consider using a password manager like LastPass or 1Password to generate strong, secure passwords and keep them safe.

– Add two-factor authentication to your important accounts. Certainly your Gmail account, but also your Twitter account, domain registrar, etc.

– Put a PIN or unlock code on whichever phone has Google Authenticator or would receive two-factor SMS texts. Consider enrolling your phone in Find my iPhone or Android Device Manager.

– Let’s get specific on your Gmail/Google account now. Click into your account’s security settings. For Google, print out backup codes for your 2-step verification and put them somewhere safe. Add a recovery email account and phone number to your account. Check to make sure that everything looks locked down tight, e.g. no app passwords that you don’t remember.

– Make sure you put a PIN on your phone number or cell phone voicemail. Why? If Google or another service leaves a recovery code in your voicemail, you don’t want hackers to access your voicemail easily by spoofing caller ID.

– In Gmail, check for any unexplained filters or forwarding rules where a hacker could be forwarding your email to a different email address.

Advanced techniques

If you’re a CEO, high-profile individual, or at much greater risk of being hacked, consider these additional steps:
– If you already enabled two-factor authentication, consider getting a Security Key. Why? Because a Security Key should stop almost all phishing, even extremely targeted “spearfishing.” Security Keys are still new, but the protection they provide against phishing is extremely good.

– You might actually want to remove your phone number from Google or other account recovery systems. Why? Humans and customer service are usually the weakest link in a security system. Hackers may use social engineering to convince your cell phone provider to add a forwarding number, then attempt to hack your account by sending a recovery code to your phone number and listening on new/additional number.

To be clear, the vast majority of users will be more protected by adding a recovery phone number to their account. I would only remove the recovery phone number if 1) you are tech-savvy and 2) you believe that someone is likely to attempt to hack or stalk you.

Those are my major tips. What am I forgetting, or what advice would you give to protect your online accounts?