Review: Yahoo pipes
I’ll chime in that I think Yahoo pipes is a really neat idea. As every decent UNIXhead knows, pipes let you combine small command-line tools easily by routing the output of one tool into the input of another tool. For example “cat census-names | cut -d’,’ -f2 | sort | uniq -c | sort -rg” might take a list of peoples’ names, extract just the last names, sort the list, unique-ify the list and produce a count of how many times each name occurred, then sort by the biggest number. Voilà, from a raw list of names you’ve now got a list of the most popular last names. Pipes are one of the things that makes Linux/Unix roxor.
The idea of Yahoo Pipes, as far as I can tell from a quick look, is to allow that same pipe behavior on RSS feeds. The system also outputs RSS urls. There are operators like sorting, counting, truncating, etc. If you wanted to make a mash-up of different feeds, this would be a neat way to prototype it.
What’s an example use case? Well, I’ve got one right here in my back pocket. 
Suppose you had multiple RSS calendar feeds, and wanted to combine those multiple feeds into one feed. Yahoo Pipes has a “union” operator, so I’m assuming you could pipe in (say) four feeds and get back one feed that was a superset of all the items that you fed in. A couple months ago, I went searching for a package that did set operations on RSS feeds (specifically looking to combine multiple different calendar feeds), and didn’t really see anything great at that point. This tool would solve that problem.
I took it for a test drive right after Jeremy posted about it and hit an error message partway through, but I’m sure they’ll get it smoothed over pretty quickly. I was able to save a “module” (which appears to be a little chunk of pipe processing which is connected to an RSS output). Then you can click publish and you get an obfuscated url back. I tested the obfuscated url and it generates RSS just fine; the test module I made is safely tucked into Google Reader now. This is the closest to fun that RSS has ever been for me, but my eyes glaze over at the sight of XML. 
It isn’t terribly hard to do operations on RSS feeds, but Yahoo Pipes
- has a fun UI for playing with slicing and dicing feeds
- doesn’t require a ton of work to save/publish modules (you do need a Yahoo ID though)
- produces easy-peasy urls that output RSS, so those urls could in turn be used by other people or in larger modules. Bonus geek points for staying true to the Unix pipe idea in that way.
O’Reilly has a good write-up as well. Congrats to the Yahoo folks that built Pipes. Nice stuff.
Jeremy Zawodny Said,
February 7, 2007 @ 9:44 pm
Thanks for the reivew, Matt.
Ed dropped a response on my site… it looks like they found the bug you hit already (and fixed it, of course).
Have fun making pipes.
jeremy schoemaker Said,
February 7, 2007 @ 10:26 pm
Did Matt say roxor
Awesome stuff!
Matt Cutts Said,
February 7, 2007 @ 10:46 pm
Good deal–thanks for the update, Jeremy.
Nenad Said,
February 7, 2007 @ 11:13 pm
Thanks for the info Jeremy, I am sure that I will think of a use for this.
TheMadHat Said,
February 7, 2007 @ 11:53 pm
This thing rocks…I have multiple calendar feeds because i work in the entertainment arena so I need multiple feeds from different cities. I’ve been looking for a good solution to this for a long time. I did hit a couple of server errors along the way, but it seemed to come through fine. I’ll try it again on a different account and send in any errors I see. Thanks for the heads-up Matt!
pasha sadri Said,
February 8, 2007 @ 1:40 am
Hi Matt. This is Pasha Sadri, from the Pipes team. Thanks for writing about pipes. I am glad that you liked pipes and the concept behind them. Protocols like GData can specially make pipes or similar systems very useful. I wanted to point out that since individual Pipes output RSS, they can be used as sources in the editor. If you create pipes with user input parameters, a module user interface is built for them so that they can be easily used just like any other module in the editor.
Chris Winfield Said,
February 8, 2007 @ 3:35 am
Was this a paid review Matt?
SearcH EngineS WeB Said,
February 8, 2007 @ 3:41 am
The developers who constructed this were just too self-absorbed to allow this to be immediately, fully utilized by the largest amounts of people.
Which is unfortunate.
There are several bugs - one of them being the disappearence of the PUBLISH buttom >:-(
SearchEnginesWeb is going to have to help them. >:-)
Ian Said,
February 8, 2007 @ 3:57 am
Any idea if it works with Atom as well as RSS?
Ryan Said,
February 8, 2007 @ 5:52 am
I’m kind of jealous that I didn’t think of this. Here I’ve been doing things like this in code for various projects, but it never occured to me to do something like this.
what a great idea.
Glynn Bird Said,
February 8, 2007 @ 5:53 am
I think the hype around Yahoo Pipes has caused the “pipes server” to take nosedive. I’ll try again later.
I agree that if XML is to become the independent interface between systems, tools to pummel the code into shape are long overdue.
Oh and it does ATOM too, Ian.
Aaron Pratt Said,
February 8, 2007 @ 6:27 am
Had same thought.
Cloudream Said,
February 8, 2007 @ 7:24 am
Please Google Feedfetcher report the number of feeds’ subscribers within GG READER & GG IG?
Matt Cutts Said,
February 8, 2007 @ 7:25 am
Chris Winfield, none of my posts are paid.
I had a pet peeve where I wanted to do “RSS arithmetic” (for lack of a better word) a while ago, so this touched a chord with me. I don’t mind giving Yahoo props when they do something well.
Ryan, I imagine that there’s still time to be a fast follower. If you didn’t worry about the GUI, it wouldn’t be hard to make a web service that processed RSS feeds, performed similar operations, and generated RSS as output.
Anjanesh Said,
February 8, 2007 @ 7:36 am
I dont understand. There doesnt seem to be a subdomain called pipes at yahoo.com
Or has this been restricted country-wise ?
Chris Winfield Said,
February 8, 2007 @ 7:36 am
Matt-
Harith Said,
February 8, 2007 @ 8:39 am
“Was this a paid review Matt?”
Of course not.
And I see neither you nor Aaron Pratt have discovered the aim of that review.
Matt is The King of Linkbaiting and he wished to show us how to get backlinks naturally.
Right Matt
travel genius Said,
February 8, 2007 @ 9:49 am
this looks to me to be very similar to google’s custom search http://www.google.com/coop/cse/overview
you can create your own search engine including all the url’s you want.
Jack Hynes Said,
February 8, 2007 @ 11:34 am
“Our Pipes are clogged! We’ve called the plumbers!”
Dabo Said,
February 8, 2007 @ 11:38 am
This Yahoo pipes does sound like a good idea. Unfortunately, when I went to pipes.yahoo.com, the servers seemed to be down since I got this message:
“Our Pipes are clogged! We’ve called the plumbers!”
It’s cool to see a well-known googler giving a positive review of a rival’s new product! Keep up the good work Matt.
PageRank Said,
February 8, 2007 @ 12:12 pm
#OFF-Topic
Matt would you mind to de-duplicate local (in box) and regular organic results, it is very common to see the same sites in local and in organic for example:
http://www.google.com/search?q=toyota+chicago
Ryan Said,
February 8, 2007 @ 12:31 pm
Yeah… but following is no fun Matt.
3 of my sites I was the first do do a concept (and of course, didn’t patent it…. stupid me)
I’ve been looking for something that will take feedbutton.com to that next level. RSS feeds are fun, but unfortunately unprofitable.
speaking of….hey Yahoo… wanna pay me millions for an RSS related site with over 3000 active users that uses 100G of bandwith but makes $5/month? It’d be the perfect compliment to your other RSS stuff…. you know you’re tempted…
switching topics, I don’t think Matt did this as linkbaiting or as a paid review. I think he did it because he’s a Geek, and we geeks love playing with new stuff…. especially ones that are all linuxy or codey.
Scorpiono Said,
February 8, 2007 @ 3:17 pm
It’s really a good thing, I was wondering how to bind more feeds into one, but I was also thinking about rss binded and posted to wordpress, or any other blog software automatically. Guess I haven’t found a plugin yet.. you can mail me though if you do.
Aaron Pratt Said,
February 8, 2007 @ 6:41 pm
In all honesty this is a great example of what a “review” is all about. Matt is passionate about this subject and it shows.
Benj Arriola Said,
February 8, 2007 @ 6:47 pm
I just tried checking it out. It’s down due to high traffic. It said:
Our Pipes are Clogged!
Dear Users…
Due to overwhelming demand we’ve had to significantly increase available capacity. We’ll be back up shortly.
If there is what we call “slashdotted,” is there something like “mattcutted?”
Ryan Said,
February 9, 2007 @ 6:08 am
Benj… only in this case, the message went up after it did get slashdotted
you left out the best part… where it said “we’ve called the plumbers”. (in keeping with the unix pipe spirit) seeing it almost made me c|n>k
Gary Beal Said,
February 9, 2007 @ 8:11 am
Matt,
Have you seen this? It hasn’t been released publicly yet.
I’ve been having fun with it all day.
http://www.msdewey.com/
ps. Hope you’ll be around for the SEM Tools seminar, I’ve got a suprise for you……lol
GaryTheScubaGuy
Heather Paquinas Said,
February 10, 2007 @ 7:59 pm
I can’t wait for Google’s version, the Internets Tubes
Mike Schinkel Said,
February 19, 2007 @ 5:24 pm
Matt wrote: “A couple months ago, I went searching for a package that did set operations on RSS feeds (specifically looking to combine multiple different calendar feeds), and didn’t really see anything great at that point. This tool would solve that problem.”
Hey Matt, did you not find RSSBus? (www.rssbus.com). It can do what Pipes does but uses scripting instead of a GUI to define the flows. It’s a commercial software product, but the desktop version is free. It’s from one of the leading companies selling Internet components to software developers /n software. How do I know them? For 12 years (94-2006) I ran a reseller of software components (Xtras.Net) and they were one of my strongest vendors. I also did some marketing consulting for them in late 2006 but have no vested interest in them anymore. At this point, I’m just a fan of RSSBus. It’s really cool technology; you should check it out.
Michael Said,
February 21, 2007 @ 11:40 pm
Great Article. Thanks.
DJ Said,
June 27, 2007 @ 1:35 pm
It’s cool to see a well-known googler giving a positive review of a rival’s new product! Keep up the good work Matt.
sheds Said,
June 28, 2007 @ 11:26 pm
Any idea if it works with Atom as well as RSS?
Lokesh Said,
July 20, 2007 @ 7:19 am
Why should matt stop himself from writing if yahoo does something that deserves to be praised.Matt is not just a google guy.
Mujra Sheez Said,
January 16, 2008 @ 1:23 am
Are someone using this on their websites? just looking for some samples.
And is it stable engough to use it on a production environment?
Mujra Sheeza Said,
January 16, 2008 @ 1:25 am
Forgot to ask..
Does someone know a place where I can find sample of how I can use it on my website?
Saima Khan Mujra Said,
January 25, 2008 @ 3:10 am
I have tried this out! And it work excellent!!
zjian Said,
March 4, 2008 @ 7:26 pm
I have tried this out! And it work excellent…