Directions Magazine (DM): Each release adds support for new formats; this time there are 16, including CityGML, several related to point clouds, and others related to simulation and visualization. Where are we with geospatial data formats? Is the industry chasing the long tail or are there likely to be new ones at the head?
Don Murray and Dale Lutz: (DM and DL): Things continue to change at an ever-increasing pace, and the tail is definitely lengthening for certain. We notice that there are more varieties of formats out there, now more than ever. In the early days, our list of supported formats included about 10 formats. Now this list includes over 275 formats sub-divided into over 10 data categories and we expect this list to continue to grow.
Having said this, some formats definitely rise to the top of the pack usage-wise. Given the deep penetration of Esri technology, it’s no surprise that Geodatabase consistently rates as an important format for our users. The new File Geodatabase API that Esri released has created great excitement as it enables the use of this file format without an ArcObjects license, and its inclusion in FME 2012 will open that format to a wider range of customers and workflows.
One new area that is really exciting is the whole new category of Web-based data stores. On this front we are very excited about Google Fusion Tables. Google has done a good job here and with FME 2012 it is trivial to get data into and out of Google Fusion Tables. The ease with which data can be loaded into them and then visualized through any browser is truly a thing of beauty. This format is definitely another “head” liner in our release.
DM: Do you see existing customers moving from primarily 2D to 3D? Where do you think geospatial users are on that continuum?
DM & DL: As we travel around and talk with our users, we do encounter examples of customers taking 2D assets and materializing the Z to make 3D datasets out of them. But honestly, they are the early adopters at this stage. Products like Autodesk Infrastructure Modeler and Esri’s recently acquired Procedural technology moving into the mainstream will play a key role in speeding the adoption and expectation of 3D to wider and wider audiences, and will push geospatial users further down the path.
In terms of cross-disciplinary collaboration, we are seeing some interest in moving data from GIS to BIM, to create a starting point. The BIM challenge is a hard one and with each release we are in an even better position to answer the challenge of moving data between GIS and BIM systems. With FME 2012, users can take BIM-like data at a high level of detail in CityGML and move them to a GIS system, or go back in the other direction as well.
Where the most activity on the 3D front seems to be is in LiDAR (Point Clouds). There we are seeing a great deal of interest and users have lots of data. One of the big stories of FME 2012 is how good we have become at moving LiDAR data. FME can routinely work with files in excess of 1 billion points: translating, reprojecting, clipping, thinning, transforming, and even loading very quickly into Oracle Spatial Point Cloud.
DM: You refer to FME Server as a spatial CEP (complex event processing) platform. Can you explain what a CEP is and the challenges of creating a spatial version?
DM & DL: Essentially, one can think of CEP as a system that responds to real-time data notifications, performs a task and then generates further actions downstream. With FME Server 2012 users can now consume more and more real-time data feeds. These real-time data feeds can be anything from sensor output, to output from an application, to user input. The point is that there is no explicit action a user or program has to take to cause the data transformation actions to take place – it is effectively “hands off” once the initial set up is done. The FME Server architecture allows any number of these "feeds" to trigger actions by associating the feed with a “topic name.” Once a “topic” has had data pushed to it (by a user, sensor or application), it in turn will initiate and then pass the data on to whatever transformations are registered to be interested in that topic. These transformations can easily involve a spatial aspect, because the FME transformation engine is inherently spatial. Because data coming from so many of these sensors, (arguably almost all of them) inherently have a location, it is logical that the spatial aspect can play a key role in determining the outcome of the event.
A good metaphor here is the "Bat Signal.” When Commissioner Gordon activates the Bat Signal he doesn't know precisely what happens after it is triggered. The stimulus is decoupled from the response. Robin may see the signal and break off a pursuit to rush to the Police Dispatch Center. Bruce Wayne may notice it and excuse himself from an uptown party to rush to the Bat Cave to begin an exhaustive Internet research session. Alfred may see it and decide it’s time to top up the Batmobile with fuel. In a similar way, the response(s) to any real-time activation are independent of each other and the original message.
The stimuli that FME Server can respond to include a POST request from a Web-connected initiator, the arrival of a file in a directory, or even the arrival of an email message. The ability of any of these types of actions to initiate transformation actions is quite powerful, especially when combined with the "real-time" data and sensors that are continuing to grow in importance. We will be using this to demo a number of scenarios, from data collection, data validation, sensor integration, to augmented reality. Check out the two video links:
FME Server 2012, Real-Time and Sensor data
FME Server and Sensor Data
DM: The new release includes more transformations to manipulate XML, "without requiring knowledge of scripting languages." Codecademy's Code Year has prompted renewed discussions as to whether everyone should know how to code. How do you feel about that? Do you think working with FME actually teaches some key ideas about coding?
DM & DL: It would be great if everyone knew how to write code! Finding really good programmers is always a challenge and one for which we are constantly on the lookout.
While it is an interesting question as to whether everyone should know how to code, this is unlikely to happen in the real world in my lifetime and so we need to build applications that address the world of today. We would argue strongly against this view. We would say that first everyone should know how to swim! Once everyone knows how to swim then we can talk. ;-) There are so many more important things that everyone should know how to do before we get to programming. Everyone being able to speak a common language would probably do more for the success of the human species than everyone knowing how to program.
Learning to write code and be good at it is very difficult and takes significant intellectual investment. We actually believe in more of a multi-disciplinary approach where people of different strengths work together to create. Also people only have so much time to learn and we would hate to think that some world class doctors are learning to program when they could be spending the same amount of energy doing other tasks.
Having said this, FME offers a graphical programming environment. As such, it provides the power that might otherwise be associated with coding, say in Python or another language, but at a much higher level of abstraction. As such we would say that working with FME does teach some coding concepts. To be effective with it you have to think logically and systematically. So it builds this ability, while alleviating the need to engage in the rules of syntax which can be time wasting sources of frustration for beginning coders – the graphical environment is the syntax and plain out prevents you from doing things that won’t run. (Whether you get the result you desire, well, that is a totally different matter.)
FME also provides mechanisms for encapsulation and abstraction, and even reusability via the custom transformer concept. Many users also use custom transformers and thus build libraries that they reuse. The new FME Store is also enabling users to share FME "software components" with each other to help them be more productive. More and more we are seeing an FME "eco-system" develop as the number of FME users continues to balloon.
DM: What do you think FME's role might be in the new push to better use "big data"?
DM & DL: FME is a data moving engine. That is all it is. With FME 2012 we have worked hard to make it faster and have now moved it into the realm of "real-time" data. With "big data" we see more and more organizations needing FME to access these data. Also loading data into the databases initially requires significant processing. Big data is another one of those topics that is just out there and will take off. The amount of data being collected by sensors is truly massive and again FME is posed to play a role in addressing this.
DM: Can you share a story about how a user/organization is taking advantage of new tools in FME 2012 to solve their spatial data challenges?
DM & DL: While it is still early days for FME 2012, we just today got an email from Robert White from Whitestar, a long-time user, informing us that they have started using FME as part of their daily workflows to publish data to Google Fusion Tables, and they are thrilled with the result.
I’m confident that as we head out on the FME World Tour road show in April, we’ll have many more success stories to share from these user meetings.