Linux USB device driver info

What, *another* half-finished blog post about Linux USB drivers? Yup.

Suppose you have a device and want a Linux device driver for it. There are a few steps you’ll need to take. One of the heroes in this area is Greg Kroah-Hartman. Greg wrote USBView, which is a Linux tool to enumerate a list of USB devices. He’s also done a massive amount of documentation as we’ll see below. One of his more eye-catching tricks is to walk a classroom through the process of writing a Linux driver for a USB thermometer live and in real-time. In addition to all the work he does for Linux in general, he recently announced a program to work with manufacturers and provide Linux drivers for new devices for free. That’s right, manufacturers get a free driver. From the original announcement:

All that is needed is some kind of specification that describes how your device works, or the email address of an engineer that is willing to answer questions every once in a while. A few sample devices might be good to have so that debugging doesn’t have to be done by email, but if necessary, that can be done.

In return, you will receive a complete and working Linux driver that is added to the main Linux kernel source tree. The driver will be written by some of the members of the Linux kernel developer community (over 1500 strong and growing).

That is majorly good karma for Linux and Greg. But if you’re not a manufacturer, here are the steps that you’d look into.

1. Get documentation of the USB protocol, or reverse engineer the protocol. It’s far easier if you can get documentation of the protocol. If you do need to reverse engineer the USB protocol, here are some tools that might help:

Windows tools: USB Snoopy let you do actions with your device and log the stream of USB information going downstream/outbound to the device, or upstream/inbound back to your computer. Snoopy Pro is a variant of the same code that evidently has some improvements. It appears that the preferred location for Snoopy Pro is here.

Linux tools:
– The previously mentioned USBView will show you devices that are currently plugged in.
– The usbutils package includes a bunch of handy console tools for USB, including lsusb, which shows you the USB devices that are currently plugged in. The output of lsusb looks like this:

Bus 001 Device 006: ID 2222:3061 MacAlly
Bus 001 Device 002: ID 0557:7000 ATEN International Co., Ltd
Bus 001 Device 003: ID 045e:00db Microsoft Corp.

2. Write the driver.

– It seems that the process of writing drivers in Linux is getting easier over time. The Linux Journal has documented this well. Compare this 2001 article by Greg Kroah-Hartman to Greg’s 2004 article on controlling a simple USB lamp device. Then see Greg’s follow-up article on writing a linux driver in user space. It turns out that you can use the libusb library to read/write with USB devices without ever mucking around in the kernel. This is possible because Linux provides a USB filesystem (called USBFS) that automatically mounts USB devices into the Linux directory tree. Note that libusb also works on BSDs, Mac/OSX computers, and that there is a Windows libusb port.

If you really want to delve into this deeply, there’s an O’Reilly book on Linux device drivers that you can buy as well.

Reverse engineering a Windows USB driver

For a while, I was really into reverse-engineering USB drivers. Don’t ask why. The heart wants what the heart wants. I didn’t finish this “hairball” post, but it has some info in it that still might be good.

I recently stumbled across this post and it inspired me. I decided to try to reverse engineer the USB protocol for my Omron pedometer, which can upload your step data, but only to a Windows computer.

There are two parts to writing a Linux driver for a new USB device: reverse-engineering the USB protocol, and writing the Linux program.

Reverse-engineering the USB protocol

Typically your problem is that a device only runs under Windows. Like it or not, that means that you’re going to need something that runs Windows. It can be a Windows computer, or you can get fancy and run Windows as a “guest” operating system using something like VMWare to do virtualization. That is, you’d install Linux, then install VMWare, then install Windows to run under VMWare. But let’s start simple.

Step 0. Find the Vendor ID and Product ID of your device

Every USB device should have a Vendor ID plus a Product ID (sometimes called a device ID) that identifies it. You’ll need to discover this information before you can talk to the device. I plugged my Omron pedometer into a linux machine and typed “lsusb”. You’ll get a lot of information back. I saw a line like

Bus 002 Device 018: ID 0590:0028 Omron Corp.

That tells me that the vendorid is hexadecimal value 0x0590 (which is 1424 in decimal) and the productid is hex value 0x0028 (which is 40 in decimal). For other operating systems, this post tells you how to find your vendor id and product id under Mac and Windows. For Windows XP, it looks like you can run “msinfo32.exe” and then look under “Components” and then “USB” and look for “VID_” (vendor id) and “PID_” (product id).

1) The simple approach: a dedicated Windows computer

In the beginning, it’s easiest to just use a Windows computer and run some software to sniff on the USB packets as they go back and forth. The wild part is that the best open-source/free program I found is five years old (SnoopyPro). It still worked fine on Windows XP though. SnoopyPro is the program you want. There’s a whole long history of how it forked from USBSnoopy (evidently also called “sniff-bin“), and there’s another program called sniffusb which is related but different (I think both sniffusb and SnoopyPro are forks off of the original USBSnoopy/sniff-bin program). It’s all very confusing. I went with SnoopyPro and it worked fine for me.

Further reading on SnoopyPro and related USB sniffer programs:
Some documentation on how to use SnoopyPro
If you’re willing to shell out for a book, it looks like USB Complete, now in its third edition, is one of the best. – mentions all the different sniffers – talks about how to convert SnoopyPro (and SniffUsb) logs/traces into hexadecimal data.

Are there other options? Sure. USB Monitor from HHD Software is $85 and runs on Windows. Or you could spend $850-950 to buy a hardware USB protocol analyzer. Since I have only a casual interest, that’s a bit steep for me.

One last option is to run Windows as a virtual “guest” on a Linux system running something like VMWare. VMWare can let programs interact with USB devices. As the virtual version Windows interacts with the USB device, the Linux computer gets to see everything that happens, because it sits between Windows and the USB device. In fact, Eric Preston presented a method that could log all the the output of the Linux usbmon program as binary data. Eric changed usbmon to use relayfs so that large amounts of data could be quickly relayed from kernel space to user space, then wrote a user space program to dump that binary data to disk. Eric also wrote a dissector for ethereal so that he could view the USB data in real-time. Unfortunately the PDF of his slide presentation have disappeared from where they used to be. In fact, all of appears to be gone now. 🙁

By the way, Ethereal is now known as Wireshark, and it is a protocol analyzer that runs on many platforms and apparently supports USB traces. It looks like Wireshark has supported USB since version 0.99.4:

Wireshark now supports USB as a media type. If you’re running a Linux distribution with version 2.6.11 of the kernel or greater and you have the usbmon module enabled and you have a recent CVS version of libpcap (post-0.9.5) installed you can also do live captures. More details can be found at the USB capture setup page on the wiki.

Follow the link in the quote to find Wireshark’s USB wiki page.

On Ubuntu 7.10 (Gutsy Gibbon), I was able to do these commands:

sudo mount -t debugfs none_debugs /sys/kernel/debug
sudo modprobe usbmon
ls /sys/kernel/debug/usbmon
0s 0t 0u 1s 1t 1u 2s 2t 2u

General USB Reading:
USB in a NutShell is a pretty good overview of how USB communication goes.
This Java and USB tutorial starts with a good overview of USB.
This USB and Linux tutorial starts to get into the nitty gritty of USB on Linux.

Cheap internet-connected scale: Wii Balance Board + Linux

You can ignore this ancient “hairball” blog post. Gather round, kids, and witness this blog post from a time *before internet-connected scales*. That’s right. Back then, we had to hack our Wii balance boards to connect them to the internet. Of course now you can buy wifi-connected scales from Fitbit and Withings. But in a olden days, you had to hack something up or even write it down on paper!

You can easily make an internet-connected scale out of a Wii Balance Board and a Linux machine:

First, find a Bluetooth dongle and configure your Linux machine to talk to the Wiimote.

Next, apply a few extra patches so that your Linux machine can talk to a Wii Balance Board.

Finally, use some Python code to upload your weight to a Google Spreadsheet.

If you’d like to hear me describe how to hook it everything together, you can watch me give a 7-8 minute talk about it (more info in that post), or you can watch it here:

Special thanks to Kevin Kelly and Gary Wolf for kickstarting the Quantified Self movement and encouraging me to talk about this project.

Wanted: bookmarks.html merging program

You can ignore this “hairball” blog post. This post dates back to a time when people actually curated, saved, and managed their bookmarks.html file. Then Google Chrome introduced the ability to save and sync all your bookmarks, extensions, etc. in the cloud. Now I sign in to Chrome and everything is synced in the cloud.

Over the years, I’ve accumulated lots of bookmarks.html files. I’d love someone to write an App Engine program that would let you upload bookmarks.html files and would merge them all into one master file. After that, you could prune/remove useless bookmarks, especially any bookmark items that are installed by default on a new browser but are useless.

Why do it on the Google App Engine?

Because it would be an easy way to get started. Essentially you want to upload a small set of files to one web location from several different computers, and then do something interesting with that data. App Engine is perfect for that kind of thing.

Can App Engine’s version of Python parse bookmarks.html files?

The Mozilla/Firefox bookmarks.html file format is a little strange, but not too strange. I found a few programs to parse bookmarks.html files. For example, one fellow wrote a Python program to merge bookmarks using sgmllib, which I’m guessing would work on App Engine.

Digging into it more, it looks like several people like Beautiful Soup as a parser. First off, you can download it as a single Python file to work in App Engine. It also looks pretty easy to use. I like this short example of extracting favicons to .ico files from a bookmarks.html file using Beautiful Soup. At least one other person has released tools to manipulate bookmarks.html files with Beautiful Soup.

Can you upload files to Google App Engine?

Yes! There’s evidently a limit of 10MB on uploaded files, but my biggest bookmark file was about 500K, and I suspect most people have much smaller bookmark files. Stack Overflow has a good example of file uploading in Google App Engine, plus there’s official examples as well as people helping other people to the point of showing live examples.

Plus browsers are getting better about uploading files to the web easily. Google Chrome supports really easy drag-and-drop file upload. I think Safari supports drag-and-drop file upload as well? And I know Firefox has the dragdropupload extension that eases uploading files to the web.

What about uploading Google Chrome bookmarks files?

Ah, a person after my own heart. The short answer is that Google Chrome can export bookmarks in a format that looks like Firefox to me. Click on the Wrench, then “Bookmark manager,” then Tools->Export Bookmarks… to get a bookmarks.html file. The more fun answer is that “C:Documents and Settings{$USER}Local SettingsApplication DataGoogleChromeUser DataDefault” appears to have a “Bookmarks” file, and it appears to be in JSON format. Can Python parse JSON? It can; Yahoo mentions that simplejson is a great library to use, and it turns out that Google App Engine supports simplejson very easily. Just say “from django.utils import simplejson” to use simplejson. So it wouldn’t be hard to upload raw Chrome bookmark files either.

Aren’t there existing websites to do this?

Maybe, but I don’t know of them. I thought that Foxmarks might be able to do this. Foxmarks (like the now-defunct Google Browser Sync) can synchronize bookmarks across multiple computers. And Foxmarks provides a web interface that lets you manipulate and export your bookmarks, but you can’t upload a raw bookmarks.html file to Foxmarks; instead, you have to upload/sync bookmarks via a browser extension. If Foxmarks added the ability to upload bookmarks.html files (vote for that idea here), that would be pretty sweet.

TimeTrax gone, SXRecorder lives

What’s that? You’ve never heard of an XMPCR? Don’t worry, the rest of the world hasn’t either. You can ignore this “hairball” post as I do spring cleaning on my blog.

TimeTrax was a program that allowed XMPCR owners to listen to XM Radio on their computer. Even nicer, the program would “time shift” the recording by taking the XM meta data and recording the raw audio of a channel to an MP3 with the artist/title of the song.

TimeTrax is not really viable anymore (see the Wikipedia page), but another program called SXRecorder will do much of the same thing.

If you’re using an XMPCR 100, you’ll need to install USB drivers for your device:

From the page: “Virtual COM port (VCP) drivers cause the USB device to appear as an additional COM port available to the PC.” So if you have the FTDI chip in a device (FTDI = Future Technology Devices International) like the XMPCR, it makes that USB device look like a serial port device, so that you can talk to COM ports.

Searching on Google for SXRecorder finds . The software is free, but you can donate $25 or buy extra plugins for SXRecorder for $35.

You’ll need to activate your XMPCR receiver, but you can refresh your radio receiver at