Exploring Yahoo! Pipes

By Adena Schutzberg

Author's Note: This article explores a new technology, one with potential uses for geospatial professionals. At this time it's not fully geo-enabled. We feel it's our responsibility to highlight emerging technologies that will touch our community. We welcome your feedback.

My name is Adena and I'm not a programmer…
Most of us in the geospatial arena fall somewhere between two extremes; we are neither "programmers" nor "non-programmers." For those who fall closer to the "non-programmer" side, whenever a software vendor says, "With some customization you can add features you need," we squirm. Can we do that customization on our own? Do we need to call in a coder? Will we need to learn something new? Yikes!

Vendors have realized that a great many of their users are not programmers, but experienced users who could be even better (and perhaps more loyal, meaning more software sales) with some ability to customize programs and build their own tools. That vision, thankfully, has driven "more accessible" customization environments and the use of visual tools for building models and small programs.

Not long ago Yahoo! stepped into this space with a Web 2.0-type tools set that allows customization and manipulation of "feeds" without programming. It's called Pipes. Here's how Yahoo! describes Pipes: "Pipes is an interactive feed aggregator and manipulator. Using Pipes, you can create feeds that are more powerful, useful and relevant." If the term "feed" is still a bit fuzzy in your mind, try this: insert the term "Web data" instead of "feed" in the two sentences above. Now, that sounds valuable, doesn't it?

Pipes is in beta and it's not yet fully geo-enabled, though those requests were at the top of the list for enhancements. Sources note: (1) "Pipes will output GeoRSS. They plan to release a map viewer soon." (2) "Yahoo will open up access to the Pipes engine to programmers and add support for the KML data source, which is used to display geographic data in Google’s popular Google Earth mapping application and Google Maps website." These tidbits, among others, suggest that this environment has significant potential. Furthermore, this is another case where "doing some homework" will likely get you ahead either in your life (creating a pipe that does something personal for you!) or on the job (creating a pipe that speeds up, simplifies or automates a task). And, no, you don't have to be a programmer… honest.

Finding, Running and Looking at a Pipe
I'm a big manual reader, but, alas, the "official" tutorial for Pipes is not yet available, so I had to bumble around a bit. The good news: I didn't bumble long at all. (There are some intro articles by non-Yahoos listed on the Pipes blog.)

As with many tech endeavors, before jumping into building my own pipe, I looked at other people’s pipes. In fact, on the main page you can search for pipes that address topics of interest. I chose to search on "GIS." (Figure 1) I got back three pages of pipes!

Figure 1. Results of a query for existing pipes with "GIS" in the them. (Click for larger image)

I chose to explore "Free Software GIS News" (Figure 2) since I know MarkusN (he's the GRASS guy). His pipe, I learn by clicking on it, "fetches the latest Free Software GIS news for you!" Clicking on "Run this pipe" runs the program and does indeed provide the latest Free Software GIS news (Figure 3). What can you do with that information? Well, you could come to the pipe’s website and run this pipe now and again. But you can also, using the Subscribe pulldown, subscribe to a feed with an RSS reader, get the results via e-mail each day, or a few other things. You can also use the output of this pipe (the list of news items) as the input for another pipe.

Figure 2. Marcus's pipe collects posts from many sources about free GIS. Some of the sources are listed on the left and a "map" of the pipe is on the right. (Click for larger image)

Figure 3. Some of the results when you run Marcus' pipe. (Click for larger image)

Now, Marcus is a programmer, but this pipe is not complex at all. How do I know? I clicked on the View/Edit link near “How This Pipe Was Made.” That brings up the pipe editor development environment. (Since Pipes is a Yahoo! tool, to access the editor or create your own pipe you'll need a free Yahoo! user ID. You may have one if you use Yahoo! Mail or Groups.) Marcus' pipe has a bunch of boxes hooked together with octopus-like links. (Figure 4) All pipes look like that. Some have more boxes and connections, some have fewer. This one is simple, but has many inputs. Let's start with something simpler.
Figure 4. Some of the linked box and lines that make up Marcus' pipe. (Click for larger image)

Creating a Pipe
MyFirstPipe is much simpler than Marcus' attempt. It has two inputs that get tossed into a "source" (which is basically a place to "get information," like Flickr or in this case Yahoo! Maps) to create the output. The inputs are a location and a text value. The output is a list of results that fall near the place related to the text (Figure 5). It's basically a local search, one you could already do via Yahoo! Maps.

Figure 5. My first pipe, called MyFirstPipe, includes two user input boxes which fuel a process that spits out the answer. It takes advantage of three Yahoo!-offered goodies, including Yahoo! Maps. (Click for larger image)

Now, let's go inside the editing environment and see what I did to build the pipe (Figure 6). First, I decided what my goal was: to have users input where and what they were interested in and have the pipe output results. I had options in the input for what prompt was issued. I used "Where?" and "What?" I could include a default, which I did for both items, so if the user didn't input anything, they'd get a search on "kite" around "Boston." The most fun part of building the pipe? Linking up the components! You click on the "dot" of an input form and drag it to the "dot" that it will "feed" in the "source." So I attached the "Location Input" box to the "Location" field in the Yahoo! Local source. Then I took the "Text Input" and linked it to the "Find" box in the source. The result of this operation is fed into the “Pipe Output.”
Figure 6. The layout of my pipe. I dragged dialog boxes from the list at left and hooked them together to build the pipe. (Click for larger image)

Before we leave the edit environment and run the pipe, let's take a look at some of the options in each category. Sources are slim just now and include many Google properties like Yahoo! Local and Flickr, but also GoogleBase. You can also use "Fetch" to search any RSS feed you like (for example, one from eBay or our All Points Blog). User inputs are what someone would key in. They come in many flavors (like database elements or cell types in Excel): text, number, date, location, etc. I like the fact that that there's location and I imagine that'll be more useful to geospatial professionals in time. The list of Operators parallels those you might know from a database or Excel. There are tools to sort, filter and merge data, for example. And, you can even translate data to another language!

Running the Pipe and Beyond
Now, let's get down to brass tacks: running the pipe and thinking about what to do with the output (Figure 7). I chose not to key in any text and took the defaults (kite and Boston). The result was six items (Figure 8). Clicking on one, Danehy Park, fires up Yahoo! Maps. It shows the location of interest, noting in the text that a kite festival is held there (Figure 9). The next step is to take the output of a pipe and use it. I could, for example, put my pipe output as RSS on my Web page (Figure 10).

Figure 7. The input box presented when my pipe is run. (Click for larger image)

Figure 8. The six results of running my pipe with default inputs. Note how the results can be used: RSS, e-mail, etc. (Click for larger image)

Figure 9. A result, Danehy Park, shown in Yahoo! Local. (Click for larger image)

Figure 10. The output of MyFirstPipe, as an RSS feed, shown on my custom Google home page.

This is a simple example which shows how easy it is to play with pipes. Yahoo! promises more sources and operators in time. Pipe authors can publish pipes for others to use, like Marcus did. You'll want to check out the published ones (search on "map" for some of interest), some of which are very clever in their own right and others which power map mashups elsewhere. The real power for us in geospatial will come as Pipes expands to support more sources and more operators. One of the top requests is further integration with Yahoo! Maps, so that the output is a map. By the way, if you feel like you could do something like what I did in my example, consider yourself a step closer to the "programmer" end of the continuum!

Further Pipes Resources for the Geospatial Community

Playing the Pipes
Mikel Maron

Yahoo!'s Pipes — The universal mash-up engine
Stefan Geens

Pipes: Something working, RSS MetaCarta search output
Christopher Schmidt

Yahoo! Pipes: Some Tips
Brady Forrest

Yahoo Pipes - What It Means for Google Earth/Maps - See Earthify
Frank Taylor


Published Friday, March 30th, 2007

Written by Adena Schutzberg



If you liked this article subscribe to our newsletter...stay informed on the latest geospatial technology

© 2016 Directions Media. All Rights Reserved.