New Dimensions in Spatial ETL: Safe Software’s FME User Conference 2008

By Adena Schutzberg

While the theme of Safe Software's FME User Conference, held March 6-7 in Vancouver, was New Dimensions in Spatial ETL (extract, transform, load), I'd say it was a GIS conference that was not so much about GIS. The challenges are geographic, to be sure, but the conference was really about integration - integration of data, platforms, services, levels of government, etc. While GIS and related technologies were in the picture, the focus was the "glue" that enables integration. Below are the key announcements and takeaways from the two-day event.

FME 2008
As is always true in an update of Feature Manipulation Engine (FME), Safe Software's workhorse for translating data between formats and restructuring them to meet specific needs, there's a long list of enhancements. Among them is a new interface tool that allows drag and drop of transformers. That tool alone, said co-founders Don Murray and Dale Lutz, will save users "1000s of clicks." The pair tag teamed each day's product-focused plenary sessions with an amusing banter that reminded me of twin brothers trying to entertain the family during the holidays. (They are a riot.)

Some of the new or enhanced formats supported: PDF (3D only), AutoCAD Map 3D (with object data), CityGML, Golden Software's Surfer, GeoJSON, IFC (for BIM), LandXML (from a 1998 request!), SQL Server Spatial 2008 (not yet available) and read/write support for GeoConcept. If you are keeping score, the number of formats FME supports has grown from nine in 1997 to 220 in 2008. One subtlety of FME that should be emphasized and may not be widely understood: FME can be consumed as a Feature Data Object (FDO), the open source "connection" interface pioneered by Autodesk. Said another way, if your software supports FDO, you can, with an FME license, gain access to those 220 formats. Which software supports FDO? I asked around and confirmed just three products: AutoCAD Map 3D, MapGuide (Enterprise/Open Source) and 1Spatial's Radius Studio. I hope to see more in the future; FDO is still a relatively new option for developers.

New transformers (software calls that transform data) tie back to key customer demands such as 3D (an extruder), geometry management (triangulation of complex polygons), simplifying data for systems that don't understand complex types (a donut bridge builder and constructive solid geometry "flattener"), attribute maintenance (the ability to retain attributes of simple geometry when they are combined into more complex features), and Web portal support (tile creation for Microsoft Virtual Earth and Google Earth). The flashiest demos of the new transformers and Web services support tapped into existing Web services. One demo grabbed local data and put them into KML for visualization in Google Earth. Clicking on a location on the map provided a link that called a Web Service to populate a pie chart of the fraction of conference goers from that location. A second demo called a pay service from White Star (the oil/gas data providers) that provides locations of wells, and popped them up on a Web-based map.

FME Desktop should be available for download this week and available in packaged form a few weeks later.

FME Server
If you take the power of FME Desktop and put it on a server you get FME Server. That simple statement doesn't reveal the power of this new product. In simple terms, you can put a workspace (a saved workflow of transformers and conversions) up on server for use. How do you do that? You develop the workspace in FME's desktop graphic development tool, Workbench, and publish it to the Web with the push of a button. ("That's easy!" was the running in-joke during that presentation.)

How do you use the saved workspace once it's up there? You can call the workspace to run (1) via a URL with appropriate parameters, (2) via its own API (soon to be SOAP enabled), or (3) from another application such as another ETL solution. The resulting data, the output of the workspace, can be (1) streamed via a MIME type (a PDF for example), (2) zipped and delivered, (3) saved out to a file to be used later, or (4) in a future release, served via OGC's Web Map Service or Web Feature Service specification servlets.

FME Server is aimed at dynamic data, data that changes and needs to be served, reformatted or re-checked regularly. FME Server provides the infrastructure to ensure that those who need the data receive them in exactly the format needed, without necessarily needing to have, or know how to run, FME.

Safe offers three key use cases for FME Server:
  1. Distributing dynamic data - allowing users to get at data formatted as they need, whenever they need, without a copy of FME or knowledge of how to use it
  2. Off-loading computationally complex and time-consuming tasks - multiple servers can help break up and speed up big jobs
  3. Integration with workflow and other systems - for example, Informatica or IBM WebSphere or ArcGIS server
 One of the most powerful demonstrations of how FME Server might be used had a user key in a city of interest, use that to find traffic data from the Web (perhaps in GeoRSS), symbolize the data, and output them in KML to be visualized in Google Earth. The demo made me think of a "geosavvy version of Yahoo! Pipes (I wrote about it last year), Yahoo's tool for non-programmers to link together Web services for their specific needs.
URL to Pipes Article

That, to me, is the power here. While there have been discussions of standards enabling all sorts of mashups of geodata, non-programmers have had limited access to any definitive tool for their creation. Now, with FME Server to do the hard work of understanding the different formats, querying, symbolizing and combining data in the appropriate output form, that all changes. One big plus: all of the various formats are treated the same.

FME Server should be shipping in late March or early April. Beta 3 should be out this week. FME Server does not include a copy of Desktop, though most users will want to use it as the preferred "authoring tool." Further, FME Server users will need to have a Web server and database up and running to use FME Server. Finally, SpatialDirect, Safe's "clip, zip and ship" solution for data distribution in many formats, has been retired as a standalone product. It's now an optional add-on to FME Server. Existing SpatialDirect users will be upgraded to that version of Server without extra fees. Several SpatialDirect users noted their excitement about this change. They will finally be able to use Workbench as an authoring tool, something not previously possible.
Questions, Questions
To my surprise, at this conference about FME, the majority of the questions to presenters were not about FME! The Log-In-Project, a joint effort of several ESRI European offices and con terra, presented by Christian Heisig of con terra, aims to provide a secure Web service interface to geospatial data, including monitoring of services' uptime, and uploading and downloading of data of different types. The questions were mostly about the security and monitoring parts.

Wesley Hardin and Jamie Katz from Burns and McDonnell described an in-house developed Geospatial Dashboard for a central Connecticut transmission line project. It uses FME to standardize data and make them available to all stakeholders and contractors via Google Earth Pro (soon Google Earth Enterprise). It costs about $150,000 and took three people three months to develop. The questions were about data quality and linking to other key systems (Primavera Expedition among others) and how to keep the datasets small (some tricks - remove symbology information from the streamed data and lop off a few unneeded decimal points).

Frank Orr from CH2M Hill described a Homeland Security-funded data integration project for the Colorado North Central Homeland Security Region. The project goes live in May and requires 10 counties to be able to upload and download eight layers of data for hazard planning needs. That meant "matching" data models from each county on the way in to the server and mapping the "other" counties' data to that same model on the way down. The questions were about metadata, security and data management issues such as edge matching.

What does this tell me? The technology most of these users interact with day in and day out is sort of invisible to them. They worry more about what's around it. I think that's how software is supposed to work.

FME as the Backend to the App
If there's a complaint about the new visualization-focused Web tools (Google Earth, OpenLayers, MapServer and others) or even some of the lightweight desktop GIS visualization packages (QGIS comes to mind), it's that they need more data crunching and analytical power. Some put ArcGIS Server or GRASS in the back end to provide that functionality. Many at the FME User Conference put FME back there. The City of Calgary did. Staffers from that city told me why: "It's easy and fast." They had taken one of the three advanced FME training courses the day before. Jason Birch of Nanaimo, British Colombia explained that when he joined the city's GIS group, it didn't have ArcGIS. So he had to use the tool at hand, FME, to do some light analytics and data prep. He and his colleague Matthew Dunstan ran down a long list of projects where FME is their "leatherman tool" for GIS. Now with FME Server, I expect we'll see even more FME use in this vein. I'll offer that from what I saw in user presentations, FME is used maybe 20 percent of the time as a data format converter and 80 percent of the time as an analytical toolbox/data restructuring tool.  

FME in All Sorts of Shops
As the "Switzerland" of the GIS world, FME pops up in all kinds of implementations. CH2M Hill, one of three Google Earth partners, uses it to enable clients to see their own data on Virtual Earth, Google Earth and Google Fusion. Specifically, users want to overlay their own large raster imagery and ArcSDE and Oracle Spatial data on these platforms and FME paves the way. Jubal Harpster of CH2M Hill brought a few servers and clients to show how it's done. The City of Surrey, British Columbia uses ArcIMS and GeoCortex IMF to power its Web mapping. FME provided tools to set up a "pay for data" portal. While that's a technical challenge, it seems determining the pricing was at least as difficult!

Live Demos
After seeing some "video recorded" demos recently, I was very happy to see lots of Safe's demos run live and, frankly, not work! Sometimes it was Vista weirdness, sometimes it was human error, sometimes it was running with the code that was compiled the night before... No one in the audience seemed to care. In fact, one attendee from Michael Baker noted these "failures" as one of the best parts of the event. Another attendee commented to me on the same topic, "They are in this with us; that gives me great confidence things will get fixed!"

Whatever happened to...
Recall all those diagrams from recent years with a centralized database in the center and many different thin and thick clients around the outside? I used to draw them myself to highlight how Oracle Spatial might be in the center, with AutoCAD, ArcGIS and other clients around the periphery, for example. I didn't see those diagrams at FME UC. What I did see were diagrams with a centralized data store (or stores) in the center, then a layer of FME, and then those same clients. Why? The pure database vision does not take into account the different data structures or the different data needs of, say, a planner, an engineer or the assessor. Those end users want - no, insist - on seeing data in the way they understand them, on the client with which they are familiar. While it's possible to do that with many databases and clients, it's quite a lot of work. FME provides a single environment that can keep everyone happy. I expect to see FME popping up in the stack, perhaps on a server, more and more often, even as (perhaps because?) new spatial databases appear, such as Microsoft SQL Server 2008.
The sessions took place in a round room at Simon Fraser University. Photo courtesy of Safe Software.

The Joy of a 130-Person Conference
When we met in a round room at Simon Fraser University on the opening morning of the event, I knew it was going to be a good few days. You could see everyone in the room. When Murray or Lutz alluded to, or asked a question of, a colleague, they were in the room. If you are investigating, or even vaguely interested in, a technology that has a conference with fewer than 500 attendees, spend the money to attend. You'll learn more than at any larger event and not have to work half as hard to do so.

Users Attend User Conference
It was great that so many users attended the event. They presented all of the papers in all the sessions, while Safe staff acted as "doctors in a Doctors Office and instructors in typically filled-to-overflowing, hands-on workshops. Some conferences seem overloaded with vendor/partner/reseller presentations. That was not a problem here. Who was missing? Some of Safe's software partners. I was pleased to speak with, and be amused and enlightened by, staff from Microsoft and MapInfo. I know that ESRI staffers were on hand, too, as was a local rep standing in for 1Spatial. I was disappointed not to see representatives from other players.

The "Idol" session. Photo courtesy of Safe Software.

Fun and Games
After the buzz last year about "FME Idol, and not having seen the TV show on which it was based, I was certainly excited to see it live. Alas, watching a handful of top coders try to decipher a challenge live (when you don't know the challenge) is not exciting. I did appreciate that Safe staffers had put together some fun "audition videos" to fill the time and that some demos missed in a morning session could be revisited. The coolest part from my perspective, Safe staffers were competing alongside the players and several conquered the challenge! Alas, none of the potential Idols fully completed it within the allotted time. Don't get me wrong, I'm in awe of users with the guts to get up there and show off their skills. The closest to completion took home a topographically correct "bowl" featuring the local geography, while the runners-up pocketed iPhone Touches. After a festive evening meal overlooking the water, attendees were treated to an improv show. The actors had clearly been briefed on what Safe does. (Imagine trying to explain that to actors!) To my surprise, they did a great job injecting "extract, transform and load" and other subtleties into their stories. I'd not laughed so hard about geospatial technology in a long time!
All materials from the general sessions, hands-on workshops and breakout sessions are now available at: Video footage of the general sessions, keynote presentation by Peter Batty, as well as a few of the breakout sessions will be posted on this site within the next two weeks.

[Disclosure: Safe Software covered my transportation and lodging for the conference. I received, as did other attendees, a Safe Software jacket and "tuque (winter hat).]

Published Tuesday, March 18th, 2008

Written by Adena Schutzberg

If you liked this article subscribe to our newsletter...stay informed on the latest geospatial technology

© 2016 Directions Media. All Rights Reserved.