Review: Yahoo pipes

I’ll chime in that I think Yahoo pipes is a really neat idea. As every decent UNIXhead knows, pipes let you combine small command-line tools easily by routing the output of one tool into the input of another tool. For example “cat census-names | cut -d’,’ -f2 | sort | uniq -c | sort -rg” might take a list of peoples’ names, extract just the last names, sort the list, unique-ify the list and produce a count of how many times each name occurred, then sort by the biggest number. VoilΓ , from a raw list of names you’ve now got a list of the most popular last names. Pipes are one of the things that makes Linux/Unix roxor.

The idea of Yahoo Pipes, as far as I can tell from a quick look, is to allow that same pipe behavior on RSS feeds. The system also outputs RSS urls. There are operators like sorting, counting, truncating, etc. If you wanted to make a mash-up of different feeds, this would be a neat way to prototype it.

What’s an example use case? Well, I’ve got one right here in my back pocket. πŸ™‚ Suppose you had multiple RSS calendar feeds, and wanted to combine those multiple feeds into one feed. Yahoo Pipes has a “union” operator, so I’m assuming you could pipe in (say) four feeds and get back one feed that was a superset of all the items that you fed in. A couple months ago, I went searching for a package that did set operations on RSS feeds (specifically looking to combine multiple different calendar feeds), and didn’t really see anything great at that point. This tool would solve that problem.

I took it for a test drive right after Jeremy posted about it and hit an error message partway through, but I’m sure they’ll get it smoothed over pretty quickly. I was able to save a “module” (which appears to be a little chunk of pipe processing which is connected to an RSS output). Then you can click publish and you get an obfuscated url back. I tested the obfuscated url and it generates RSS just fine; the test module I made is safely tucked into Google Reader now. This is the closest to fun that RSS has ever been for me, but my eyes glaze over at the sight of XML. πŸ˜‰

It isn’t terribly hard to do operations on RSS feeds, but Yahoo Pipes
– has a fun UI for playing with slicing and dicing feeds
– doesn’t require a ton of work to save/publish modules (you do need a Yahoo ID though)
– produces easy-peasy urls that output RSS, so those urls could in turn be used by other people or in larger modules. Bonus geek points for staying true to the Unix pipe idea in that way.

O’Reilly has a good write-up as well. Congrats to the Yahoo folks that built Pipes. Nice stuff. πŸ™‚

40 Responses to Review: Yahoo pipes (Leave a comment)

  1. Thanks for the reivew, Matt.

    Ed dropped a response on my site… it looks like they found the bug you hit already (and fixed it, of course).

    Have fun making pipes. πŸ™‚

  2. Did Matt say roxor πŸ˜‰

    Awesome stuff!

  3. Good deal–thanks for the update, Jeremy.

  4. Thanks for the info Jeremy, I am sure that I will think of a use for this.

  5. This thing rocks…I have multiple calendar feeds because i work in the entertainment arena so I need multiple feeds from different cities. I’ve been looking for a good solution to this for a long time. I did hit a couple of server errors along the way, but it seemed to come through fine. I’ll try it again on a different account and send in any errors I see. Thanks for the heads-up Matt!

  6. Hi Matt. This is Pasha Sadri, from the Pipes team. Thanks for writing about pipes. I am glad that you liked pipes and the concept behind them. Protocols like GData can specially make pipes or similar systems very useful. I wanted to point out that since individual Pipes output RSS, they can be used as sources in the editor. If you create pipes with user input parameters, a module user interface is built for them so that they can be easily used just like any other module in the editor.

  7. Was this a paid review Matt? πŸ˜‰

  8. The developers who constructed this were just too self-absorbed to allow this to be immediately, fully utilized by the largest amounts of people.

    Which is unfortunate.

    There are several bugs – one of them being the disappearence of the PUBLISH buttom >:-(

    SearchEnginesWeb is going to have to help them. >:-)

  9. Any idea if it works with Atom as well as RSS?

  10. I’m kind of jealous that I didn’t think of this. Here I’ve been doing things like this in code for various projects, but it never occured to me to do something like this.

    what a great idea.

  11. I think the hype around Yahoo Pipes has caused the “pipes server” to take nosedive. I’ll try again later.

    I agree that if XML is to become the independent interface between systems, tools to pummel the code into shape are long overdue.

    Oh and it does ATOM too, Ian.

  12. Was this a paid review Matt?

    Had same thought. πŸ˜‰

  13. Please Google Feedfetcher report the number of feeds’ subscribers within GG READER & GG IG?

  14. Chris Winfield, none of my posts are paid. πŸ™‚ I had a pet peeve where I wanted to do “RSS arithmetic” (for lack of a better word) a while ago, so this touched a chord with me. I don’t mind giving Yahoo props when they do something well.

    Ryan, I imagine that there’s still time to be a fast follower. If you didn’t worry about the GUI, it wouldn’t be hard to make a web service that processed RSS feeds, performed similar operations, and generated RSS as output.

  15. I dont understand. There doesnt seem to be a subdomain called pipes at yahoo.com
    Or has this been restricted country-wise ?

  16. Matt-

    πŸ˜‰ = joke in this case. Good review BTW.

  17. “Was this a paid review Matt?”

    Of course not.

    And I see neither you nor Aaron Pratt have discovered the aim of that review.

    Matt is The King of Linkbaiting and he wished to show us how to get backlinks naturally.

    Right Matt πŸ™‚

  18. this looks to me to be very similar to google’s custom search http://www.google.com/coop/cse/overview
    you can create your own search engine including all the url’s you want.

  19. “Our Pipes are clogged! We’ve called the plumbers!”

  20. This Yahoo pipes does sound like a good idea. Unfortunately, when I went to pipes.yahoo.com, the servers seemed to be down since I got this message:
    “Our Pipes are clogged! We’ve called the plumbers!”

    It’s cool to see a well-known googler giving a positive review of a rival’s new product! Keep up the good work Matt.

  21. #OFF-Topic

    Matt would you mind to de-duplicate local (in box) and regular organic results, it is very common to see the same sites in local and in organic for example:

    http://www.google.com/search?q=toyota+chicago

  22. Yeah… but following is no fun Matt.

    3 of my sites I was the first do do a concept (and of course, didn’t patent it…. stupid me)

    I’ve been looking for something that will take feedbutton.com to that next level. RSS feeds are fun, but unfortunately unprofitable.

    speaking of….hey Yahoo… wanna pay me millions for an RSS related site with over 3000 active users that uses 100G of bandwith but makes $5/month? It’d be the perfect compliment to your other RSS stuff…. you know you’re tempted…

    switching topics, I don’t think Matt did this as linkbaiting or as a paid review. I think he did it because he’s a Geek, and we geeks love playing with new stuff…. especially ones that are all linuxy or codey.

  23. It’s really a good thing, I was wondering how to bind more feeds into one, but I was also thinking about rss binded and posted to wordpress, or any other blog software automatically. Guess I haven’t found a plugin yet.. you can mail me though if you do.

  24. In all honesty this is a great example of what a “review” is all about. Matt is passionate about this subject and it shows.

  25. I just tried checking it out. It’s down due to high traffic. It said:

    Our Pipes are Clogged!

    Dear Users…

    Due to overwhelming demand we’ve had to significantly increase available capacity. We’ll be back up shortly.

    If there is what we call “slashdotted,” is there something like “mattcutted?”

  26. Benj… only in this case, the message went up after it did get slashdotted πŸ™‚

    you left out the best part… where it said “we’ve called the plumbers”. (in keeping with the unix pipe spirit) seeing it almost made me c|n>k

  27. Matt,

    Have you seen this? It hasn’t been released publicly yet.

    I’ve been having fun with it all day.

    http://www.msdewey.com/

    ps. Hope you’ll be around for the SEM Tools seminar, I’ve got a suprise for you……lol

    GaryTheScubaGuy

  28. I can’t wait for Google’s version, the Internets Tubes

  29. Matt wrote: “A couple months ago, I went searching for a package that did set operations on RSS feeds (specifically looking to combine multiple different calendar feeds), and didn’t really see anything great at that point. This tool would solve that problem.”

    Hey Matt, did you not find RSSBus? (www.rssbus.com). It can do what Pipes does but uses scripting instead of a GUI to define the flows. It’s a commercial software product, but the desktop version is free. It’s from one of the leading companies selling Internet components to software developers /n software. How do I know them? For 12 years (94-2006) I ran a reseller of software components (Xtras.Net) and they were one of my strongest vendors. I also did some marketing consulting for them in late 2006 but have no vested interest in them anymore. At this point, I’m just a fan of RSSBus. It’s really cool technology; you should check it out.

  30. Great Article. Thanks.

  31. It’s cool to see a well-known googler giving a positive review of a rival’s new product! Keep up the good work Matt.

  32. Any idea if it works with Atom as well as RSS?

  33. Why should matt stop himself from writing if yahoo does something that deserves to be praised.Matt is not just a google guy.

  34. Are someone using this on their websites? just looking for some samples.

    And is it stable engough to use it on a production environment?

  35. Forgot to ask..

    Does someone know a place where I can find sample of how I can use it on my website?

  36. I have tried this out! And it work excellent!! πŸ™‚

  37. I have tried this out! And it work excellent…

  38. Awesome article I love Yahoo Pipes it is a great tool for web masters. Thanks

  39. Hi,
    i wanted to make a single feed like you said. I was able to combine 4 feeds and make it one, but the problem i’m facing is that it doesn’t shows the latest feeds, it shows the feeds in the pattern in which i make pipes
    e.g softwares union games union poetry
    it always shows poetry on top as it is the last in union

    but i want to get the latest on top from all of the feeds.

  40. I’ve been dabbling with pipes for the last 3 or 4 weeks. I just seems like for one reason or another, it is hard to accomplish much, at least for me. Pipes seems a little clunky to me, though I must confess to being a little rusty πŸ™‚

    It would be reassuring to know whether Yahoo plans on continuing pipes before committing more time and effort – I’ve read remarks on other blogs to the effect that at times, pipes doesn’t seem to be working all the time.

css.php