When I was in grad school, I would have loved to have a device like the Kinect. So I decided to run my own contest:

The first $1000 prize goes to the person or team that writes the coolest open-source app, demo, or program using the Kinect. The second prize goes to the person or team that does the most to make it easy to write programs that use the Kinect on Linux.

It’s time to announce the prize winners. There’s been so many cool things going on with the Kinect that instead of two winners, I ended up declaring seven $1000 winners.

Open-source Application or Demo

I picked two winners in this category.

• Tomoto Washio for Kinect Ultra Seven. This program lets you transform into Ultra Seven with lots of different powers. It’s a really fun demo–check out the videos:
  
• Tiago Serra and the SenseBloom team for XBox Kinect OSCeleton. The application sends 3D tracked body skeletons through the OSC protocol. Essentially, it’s a do-it-yourself motion capture system:
  

People that have made it easier to write programs for the Kinect

A ton of people have made the Kinect more accessible on Linux or helped the Kinect community. I ended up picking five winners.

• Hector Martin
• Arne Bernin
• Kyle Machulis
• Brandyn White
• Joshua Blake

All of these individuals pushed things forward so others can develop great programs on the Kinect more easily. Congratulations to all the winners, and to everyone doing neat things with their Kinect!

Before I joined Google, I was a grad student interested in topics like computer vision, motion self-tracking, laser scanners–basically any neat or unusual sensing device. That’s why I was so excited to hear about the Kinect, which is a low-cost ($150) peripheral for the Xbox. The output from a Kinect includes:
- a 640×480 color video stream.
- a 320×240 depth stream. Depth is recovered by projecting invisible infrared (IR) dots into a room. You should watch this cool video to see how the Kinect projects IR dots across a room. Here’s a single frame from the video:

What’s even better is that people have figured out how to access data from the Kinect without requiring an Xbox to go with it. In fact, open drivers for the Kinect have now been released. The always-cool Adafruit Industries, which offers all sorts of excellent do-it-yourself electronics kits, sponsored a contest to produce open-source drivers for the Kinect:

First person / group to get RGB out with distance values being used wins, you’re smart – you know what would be useful for the community out there. All the code needs to be open source and/or public domain.

Sure enough, within a few days, the contest was won by Héctor Martín Cantero, who is actually rolling his reward into tools and devices for fellow white-hat hackers and reverse engineers that he works with, which is a great gesture. Okay, so where are we now? If I were still in grad school, I’d be incredibly excited–there’s now a $150 off-the-shelf device that provides depth + stereo and a lot more.

It’s time for a new contest

I want to kickstart neat projects, so I’m starting my own contest with $2000 in prizes. There are two $1000 prizes. The first $1000 prize goes to the person or team that writes the coolest open-source app, demo, or program using the Kinect. The second prize goes to the person or team that does the most to make it easy to write programs that use the Kinect on Linux.

Enter the contests by leaving a comment on this blog post with a link to your project, along with a very-short description of what your project does or your contribution to Kinect hacking. The contest runs until the end of the year: that’s Dec. 31st, 2010 at midnight Pacific time. I may ask for outside input on who should be the winner, but I’ll make the final call on who wins.

To get your ideas flowing, I’ll offer a few suggestions. Let’s start with the second contest: making the Kinect more accessible. In my ideal world, would-be hackers would type a single command-line, e.g. “sudo apt-get install openkinect” and after that command finishes, several tools for the Kinect would be installed. Maybe a “Kinect snapshot” program that dumps a picture, a depth map, and the accelerometer values to a few files. Probably some sort of openkinect library plus header files so that people can write their own Kinect programs. I would *love* some bindings to a high-level language like Python so that would-be hobbyists can write 3-4 lines of python (“import openkinect”) and start trying ideas with minimal fuss. To win the second contest, you could write any of these libraries, utilities, bindings or simplify installing them on recent versions of Linux/Ubuntu (let’s say 10.04 or greater).

Okay, how about some ideas for cool things to do with a Kinect? I’ll throw out a few to get you thinking.

Idea 1: A Minority Report-style user interface where you can open, move, and close windows with your movements.

Idea 2: What if you move the Kinect around or mount it to something that moves? The Kinect has an accelerometer plus depth sensing plus video. That might be enough to reconstruct the position and pose of the Kinect as you move it around. As a side benefit, you might end up reconstructing a 3D model of your surroundings as a byproduct. The folks at UNC-Chapel Hill where I went to grad school built a wide-area self-tracker that relied on a Kalman filter to estimate a person’s position and pose. See this PDF paper for example.

Idea 3: Augmented reality. Given a video stream plus depth, look for discontinuities in depth to get a sort of 2.5 dimensional representation of a scene with layers. Then add new features into the video stream, e.g. a bouncing ball that goes between you and the couch, or behind the couch. The pictures at the end of this PDF paper should get you thinking.

Idea 4: Space carver. Like the previous idea, but instead of learning the 2.5D layers of a scene from a singe depth map, use the depth map over time. For example, think about a person walking behind a couch. When you can see the whole person, you can estimate how big they are. When they walk behind the couch, they’re still just as big, so you can guess that the couch is occluding that person and therefore the couch is in front of the person. Over time, you could build up much more accurate discontinuities and layers for a scene by watching who walks behind or in front of what.

Idea 5: A 3D Hough transform. A vanilla Hough transform takes a 2D image, looks for edges in the image, and then runs some computation to determine lines in the image. A 3D Hough transform finds planes in range data. I’ve done this with laser rangefinder data and it works. So you could take a depth data from a Kinect and reconstruct planes for the ground or walls in a room.

Idea 6: What if you had two or more Kinects? You’d have depth or range data from the viewpoint of each Kinect and you could combine or intersect that data. If you put two Kinects at right angles (or three or four Kinects around a room, all pointing into the room), could you reconstruct a true 3D scene or 3D object from intersecting the range data from each Kinect?

I hope a few of these ideas get you thinking about all the fun things you could do with a Kinect. I’m looking forward to seeing what cool ideas, applications, and projects people come up with!

On the childlike-sense-of-wonder-scale (as fake Steve Jobs would say), the iPad is better than the Macbook Air but not as stunning as the iPhone when the iPhone first came out. I played with my wife’s iPhone for just a few minutes before I knew I had to have an iPhone. But I never really cared about the Macbook Air, mainly because the screen resolution was worse than my current laptop. The iPad fits between those two products in the spectrum of desirability for me.

The form factor is… weird. You’re going to feel strange carrying one of these into the grocery store, in the same way you felt weird using your cell phone in the grocery store at first. Leave it to Apple to blaze a trail of coolness though; the iPad will make this form factor acceptable, so you won’t feel quite as strange carrying a tablet into a meeting in a few months. The form factor fundamentally is awkward though: the iPad is book-sized, but much more delicate than a book. A screen this big with no protection will get scratched or scuffed. I’d expect to see plenty of articles about dropped iPads like you did about Wiimotes getting thrown into TVs and windows.

The gadget lover in me wants one, but the part of me that cares about open source and tinkering is stronger. I’m with Cory Doctorow on this one. The iPad is gorgeous, but it’s still not worth it for me. Yesterday, I also bought two books at the bookstore to read on a trip. Walking back to my car with “paper media” felt a bit dorky–why am I buying books on paper in 2010? If I could buy a book digitally and really own it (not just obtain a license to read a book, where the license could be revoked), I’d quickly switch to buying my books digitally. But the success of the Kindle shows that a lot of people care more about the convenience than completely owning what they’re buying digitally.

I think the iPad will be a huge hit. Non-tech-savvy consumers will love it because of the user experience, the simplicity, and the lack of viruses/malware/trojans. It’s like a computer without all the hassles of a typical computer (pre-installed crapware, anti-virus software, inconvenient software upgrades). Lots of tech-savvy consumers will love the iPad for the same reasons, and especially for the polish and user experience. The current iPad lacks a few things (such as a camera), which ensures that future generations of the iPad will also be a huge hit.

But the iPad isn’t for me. I want the ability to run arbitrary programs without paying extra money or getting permission from the computer manufacturer. Almost the only thing you give up when buying an iPad is a degree of openness, and tons of people could care less about that if they get a better user experience in return. I think that the iPad is a magical device built for consumers, but less for makers or tinkerers. I think the world needs more makers, which is why I don’t intend to buy an iPad. That said, I think the typical consumer will love the iPad.

At any point, I could go to a web page to view a map of where I’d been. The page would show a “heat map” of signal strength for each carrier or frequency band. Maybe I could also slice/dice by time or see the total number of readings in each location. I’m pretty sure you could rig this up out of 2-3 cell phones running Android in the worst case.

So far, I’ve found:

Android

- RF Signal Tracker is a nice app to collect and map signal strength data. It looks like it can upload to OpenCellID, which is a project to create an open database of cell IDs (numbers that correspond to cells).
- Antennas is a pretty cool free app to show you nearby antennas and signal strength. It can even export some data in KML for use with Google Maps/Earth, but it doesn’t seem to make a heat map that could be easily grokked.
- Sensorly has a free Android app, but they seem to want you to pay to zoom in closer than city level. I’m willing to do that, but didn’t see the for-pay addon in the Android Market.

iPhone

- I also found an iPhone app called Signals that will continuously collect signal data and upload it.
- AT&T offers an iPhone app called Mark the Spot to report dropped calls, no coverage, etc. I have to admit that I don’t understand why this is manual though. Personally, I’d want my phone to ping my carrier with its location every time the phone dropped a call.

Web

- SignalMap is a website to (manually!) submit the number of bars for a location. It doesn’t appear to have any mobile app to back it up. Likewise, Dead Cell Zones and Got Reception? appear to rely on manual reports. I don’t think manual reports is the best way to tackle cell phone coverage maps though — you really want an app for this.
- http://www.cellreception.com/ has the standard manual reports data, but also will map the location of cell phone towers based on the location of cell phone towers registered with the FCC.
- Root Wireless powers the cell phone signal strength maps that CNET uses, but I didn’t see any apps I could download or install on a phone. I registered to be a beta tester a long time ago, but no one ever contacted me.

That’s what I could find. Do you know of any good Android (or iPhone) programs to collect, map, or upload cell phone strength measurements? If so, let me know in the comments.

This month is really busy with some internal Google projects–don’t worry, not related to webspam–so I’m not planning to do a new 30 day challenge this month. I have kept biking in to work and I’m enjoying it more lately. I think I’ll enjoy biking even more after I bling my bike out with the full-color LED lights I bought from MonkeyLectric at Maker Faire. Here’s an image from MonkeyLectric’s gallery to show you what they look like:

I have to say, they’re a big step up from my Tireflys, which are just LEDs that stick on the stem valve of your bike tire.

One of my favorite hidden Google gems is a program for Android phones called My Tracks. I like it enough that we made a short video about it. Enjoy!

As always, you can watch more videos on the official webmaster video channel on YouTube.

I picked up the new iPhone 3G S this Friday and I thought I’d jot down a few thoughts:

The Good:
- The iPhone 3GS is considerably faster than the iPhone 3G. Especially in the browser, you’ll notice pages render faster. JavaScript-heavy pages (such as the mobile web version of Gmail) execute much more smoothly. The iPhone 3GS feels less like an underpowered mobile phone and more like a powerful pocket computer that can keep up with its owner.
- The built-in video camera is very cool. I expect a corresponding spike in home videos. For example, here’s my cat Ozzie playing with a toy:

More seriously, within about five years, for any group of 10 or more people, at least a few will have a video camera built into their phone. That’s a very powerful trend in a lot of ways.
- I tend to agree with Michael Arrington that no one with an iPhone 3GS needs a Flip video camera now. Cisco bought Pure Digital Technologies, the makers of the Flip, for almost $600 million dollars about three months ago. That might prove to be good timing on Pure Digital’s part.
- Lots of small changes in OS 3.0 are quite nice, such as showing outgoing vs. incoming calls in the “Recents” list.

The Bad:
- Apple’s iPhone philosophy has always seemed to me to be about simplicity. The single button forced a constrained elegance on the iPhone’s interface. In providing some newer features, the iPhone 3G S feels less like an iPhone and more like someone shoveled in a lot of features. I didn’t really need copy/paste, and it seems to pop up at random inconvenient times: double-tap a word if you’re not in the browser; in the browser, hold your finger on some text. Except the copy handles don’t seem to show up on the web pages I want, and sometimes unwanted copy handles appear when I’m just scrolling with my finger.
- The iPhone 3GS is not the huge leap that the iPhone or the iPhone->iPhone 3G was. I do think that leaves some opportunities for Android, Palm, and other competitors.
- Battery life has been worse so far for me. I’ve been using the phone more and it’s only been a few days, so I’m not going to jump to conclusions on this yet. Apple also recommends that you let the phone run down completely at least once a month, and I haven’t done that yet. I expect that battery life will be better for most people.
- Not a great name; the “GS” part makes me think of Ghostscript. A few days ago, I would have said that the “iPhone Video” is a much better name, but it’s true that the speed bump is more noticeable than the video. I still think Apple could have come up with a better name than “iPhone 3G S” though. I’m sure someone who knows about Mercedes Benz cars knows the difference between the E class, the SLK class, or the GL 420 CDI, but most normal people don’t know what a bunch of letters and numbers mean.

The Ugly:
- On my previous iPhone (the 3G), the metal band around the front matched smoothly with the black plastic back. On the new iPhone 3GS, I can feel the seam where the band meets the plastic. On the front of the phone, when I flick my finger off the glass, I can feel the seam of the metal band there too.
- In my personal opinion, someone miscalculated in charging iPhone 3G owners $200 extra to upgrade. New iPhone 3G S customers pay $199 (16GB) or $299 (32GB); many early adopters would have to pay $399 (16GB) or $499 (32GB) to upgrade. The CPU speed bump and video abilities aren’t enough to counteract what many early adopters will perceive as a bit of a slap in the face. Last year, the line for the iPhone 3G at Valley Fair stretched outside the building most of the day. When I went to get my iPhone 3G S on the release day at Valley Fair this year, there was no line at all.

And remember that early adopters often give their previous phones to family members. In my case, two other relatives are taking our older iPhone 3G phones and moving from a different carrier to AT&T. By charging early adopters more, AT&T ensures that more people will hang on to their old phones instead giving them to other people, many of whom would then become AT&T/iPhone customers. By limiting the “trickle down” effect as older iPhones go to family members, AT&T is missing a chance to gain more marketshare by acquiring additional new customers.

I’ll be interested to see how Apple and AT&T react. AT&T has already allowed some (but not all) iPhone 3G owners to upgrade without paying an “early adopter penalty.” And Apple can move quickly and decisively when needed–remember the $200 iPhone price drop in 2007 just a short time after the iPhone was released? Of course, it’s possible that penalizing early adopters is all part of some four-dimensional chess game that Apple is playing. If Apple decides to terminate its exclusive U.S. deal with AT&T in a year or so, maybe it didn’t want a bunch of people signing up for two-year contracts this time around? Right now I’m puzzled by what appears to me to be a misstep, but the folks at Apple are smart, so I’d be willing to believe that Apple has good reasons for what they’re doing.

Should you upgrade? That’s something only you can answer. If you still have a non-smartphone or an original iPhone, it’s probably worth it. If you have an iPhone 3G (especially if you’re not eligible for the discounted upgrade yet), you might try OS 3.0 and see if that’s enough. I decided to get the 3G S and I’m glad that I did. I fill all sorts of idle moments with surfing, tweeting, and checking my mail. The iPhone 3G S makes all those activities much faster and more pleasurable. Overall I’m quite happy with my iPhone 3G S.

Just as a proof-of-concept, here’s a barcode scanner written in six lines of Python code:

import android
droid = android.Android()
code = droid.scanBarcode()
isbn = int(code['result']['SCAN_RESULT'])
url = “http://books.google.com?q=%d” % isbn
droid.startActivity(‘android.intent.action.VIEW’, url)

Thanks to fellow Googler Vijayakrishna Griddaluru for sending me this sample code. Visiting the resulting url offers the option to add that book to your library:

Pretty easy, huh? You can read all about the new scripting environment. Not only can you scan bar codes, you can use text-to-speech, make phone calls, send text messages, read sensor data, and find your location–all from easy scripts. One person wrote a script to go into silent mode when the phone is placed screen-down on the table. It took less than 20 lines of code, and that’s including comments!

The Android Scripting Environment should make fun projects even easier. Brad Fitzpatrick wrote about using his Android phone to open his garage door automatically when his motorcycle gets close to home. Now those sorts of projects are even easier to write.

Last year I suggested potential Summer of Code projects and one of my favorite suggestions was “How about a good open-source program to manage your book library? Something like the Delicious Library program, but that works with Linux?” In the blog comments, Colin Colehour left an excellent comment: “Matt, Can’t you use Google Books to keep track of your book library at home? You can add books that you own to the ‘my library’ list and then export that as an xml file and they have RSS feeds.”

The suggestion was so obvious that I smacked my head. Why install software at all when a website will store the data for you? The only problem was how to tell Google which books I own. Well, there’s a neat hack for this too: Amazon carries the Adesso NuScan 1000 bar code scanner for $65.44 with free shipping. I’m sure you can get barcode scanners for cheaper (anyone remember the CueCat scanner that was free?), but the Adesso had good reviews.

With that, adding your books to Google’s My Library feature is simplicity itself–the Google Books team has tweaked the workflow so that you can barcode scan and add lots of books very quickly. Here’s the video to demonstrate:

Why would you record which books you own in the first place? The immediate reason is that you can run full-text searches against the books in your library. That’s right: just by scanning bar codes, you can search over the text of books you own. Down the road, I can easily imagine other uses. Wouldn’t it be great if you could upload your list of books to Amazon, and it would automatically suggest other books you should read? Or avoid suggesting books that you already own? Josh Lowensohn mentions another great reason to do this: it creates a record for insurance purposes.

Once you have your book list, there are social networks for book lovers such as Goodreads and LibraryThing. And please note: this isn’t the only way to scan your books. Delicious Library 2 is $40 commercial software for the Mac that can use your Mac’s built-in webcam.

Special thanks to Michael ‘Wysz’ Wyszomierski for recording and producing this video. I love that he showed the computer’s screen and showed an “action shot” of scanning the books.

This post assumes that you can already communicate with a Wiimote on Linux. See my earlier post if you haven’t done that.

You’ll need Subversion and autoconf installed if you haven’t already installed it. I think the official/full list of packages to install is

sudo apt-get install autoconf autogen automake gcc bluetooth libbluetooth2-dev libgtk2.0-dev pkg-config python2.5-dev flex bison subversion

You must also install the development package for BlueZ, which is the official Linux Bluetooth protocol stack:

sudo apt-get install libbluetooth-dev

Make a directory to hold your code:

mkdir ~/wiibalance
cd ~/wiibalance

Check out the CWiid (get it? CWiid? Seaweed?) library using Subversion:

svn co http://abstrakraft.org/cwiid/svn/

When I checked CWiid out, it was up to version 183.

Compile the library, e.g.

cd ~/wiibalance/svn/trunk
autoconf
./configure --libdir=/usr/lib (according to this page, you need the libdir parameter on Ubuntu)
make
sudo make install

Now apply the patch to CWiid to add the Balance board. Go to http://abstrakraft.org/cwiid/ticket/63, click on balance.diff, then at the bottom of the resulting page, click on “Original Format” to download the diff as a raw text file. Do the same to download the “weighdemo.py” file. Now apply the balance.diff patch:

patch --dry-run -b -p0 < balance.diff (this command tests that everything would apply without errors)
patch -b -p0 < balance.diff

Finally, compile the code and install it:

make
sudo make install

Then you can play with the weighdemo.py script. You’ll want to change the line “sys.path.insert(0, ‘/home/tbble/code/cwiid/svn/cwiid/python/build/lib.linux-x86
_64-2.5/’)” to point to the right place on your system, e.g. /home/youraccountname/wiibalance/svn/trunk/python/build/lib.linux-i686-2.5/ . And you may need to disable the “if wiimote.state['ext_type'] != cwiid.EXT_BALANCE” if statement. I also added a few lines to print out the values and calibration numbers for the four sensors. Here’s what I get when I run my program:

$ ./weighdemo.py
Put Wiimote in discoverable mode now (press 1+2)…
Type q to quit, or anything else to report your weight

right_top 2249 [2293, 4004, 5725]
right_bottom 1467 [1449, 3155, 4874]
left_top 12471 [12476, 14238, 16015]
left_bottom 6822 [6848, 8581, 10325]
-0.58kg
Type q to quit, or anything else to report your weight

right_top 3618 [2293, 4004, 5725]
right_bottom 3983 [1449, 3155, 4874]
left_top 14208 [12476, 14238, 16015]
left_bottom 9621 [6848, 8581, 10325]
82.18kg

In this run, the first example is weighing with nothing on the scale. The second example is with me standing on the scale.

Here’s what you need to know. First, absorb this info from the Wikipedia entry:

Although the Japanese packaging states that it is designed to support people weighing up to 136 kilograms (300 pounds) and the “Western” Balance Board up to 150 kg (330 pounds), they are actually the same board. The packaging differs due to laws in Japan and other nations regarding weights and measures. While the board only displays weight readings up to what is printed on the packaging, the actual physical structure of the board can withstand much greater force equivalent to around 300 kg (660 pounds).

Okay, so the Wii Balance Board is certified for 300 pounds (136 kg) in Japan and 330 pounds (150 kg) in the U.S. The Wii Balance Board has four sensors, so each sensor is certified for up to 136 kg / 4 = 34 kg per sensor in Japan or 150 kg / 4 = 37.5kg per sensor in the United States. Now that you’ve been schooled on that, the following Wii Balance Board calibration information from WiiBrew will make more sense.

Each sensor returns 2 bytes of data and also has six bytes of calibration data. Think of it as a 16 bit number and three calibration numbers, that are also 16 bits apiece. The three calibration numbers correspond to the sensor reading for 0 kg, 17 kg, and 34 kg (those numbers should look familiar if you look at the previous paragraph). So if sensor #1 gives a value of 5725 and the 34 kg number is also 5725, then that sensor is reporting exactly 34 kg of weight on it.

The WiiBrew page says “Calculating the weight on each sensor simply involves interpolating between the two calibration values your reading falls between (or using the higher two values if your reading exceeds the highest calibration value), and the total weight on the board is the sum of these [four sensor] values.”

With that info, let’s go back to one of my sensor readings and convert it to actual kg. Take the reading “right_top 3618 [2293, 4004, 5725]“. The sensor value is 3618, which is between the 0 kg calibration number of 2293 and the 17 kg calibration number of 4004. So the weight on the right_top sensor is 17kg * (3618-2293)/(4004-2293) or 17kg * 0.7744 or 13.2 kg. I tend to put my weight on the back of my feet, which is why 13.2 kg is a little lower than my total weight (82 kg) divided by the four sensors, which would be 20.5 kg per sensor.

If your eyes haven’t completely glazed over, the upshot is that the Balance Board has four independent weight sensors, each calibrated to kilograms, and you can read those sensors in real-time over Bluetooth. Translating the sensor values into weight only takes four lines of Python code. From there, you can do anything you want with that data. Play a game, hook your Wii Balance Board up to Google Earth, or you can even automatically upload your weight to a Google Spreadsheet.

For example, you can run the command “sudo apt-get install python-pygame” to install pygame. Then go back to this page and download the scalesgui.py and system.ini files as you did before (click on them, then click on “Original Format” at the bottom of the resulting page). Make scalesgui.py executable with “chmod a+x scalesgui.py” and comment out the “EXT_BALANCE” if statement, then run “./scalesgui.py” and this is what you’ll see:

This is a snapshot, but the program runs just as fast as any video game. You can lean and move the circle around really fast. And in the bottom right, you can see that I weigh 81.74 kg, which is about 180 pounds.

I know this is a long post, but the upshot is that with 10-15 minutes of work, you can use a Wii Balance Board as a real-time sensor with Linux. I hope you have fun!