Safe Software launched FME 2013 in January (press release). Each release is basically a milestone for the company and for geospatial software users. The data formats supported and features added reflect “what we are up to” in the use of our favorite products and data sources. Directions Magazine asked Safe President Don Murray and Vice President of Software Development Dale Lutz to address data format and industry trends reflected in the new release.
Directions Magazine(DM): FME Desktop and Server now support more than 300 data formats with 28 new ones added in this release. Which three would you identify as “up and coming,” that is, formats that GIS users might get their hands on for the first time in 2013? Why are these up and coming? What do they bring to the table?
Safe Software (SS): Of the new formats, these three stand out to us:
- SpatiaLite -- SQLite has become an extremely popular database format because it features high performance, is easily portable because it is completely contained in a single file, and is easily embedded into applications (arguably most iPhone and Android apps make use of it, for example). SpatiaLite extends SQLite with spatial data types, in the same way as PostGIS extends PostgreSQL to do spatial. We’ve been told by several customers that the only thing holding them back from using SpatiaLite in their organization was FME’s lack of support for it. With the release of FME 2013, they’ll need to find another excuse! [Learn more: SpatialLite video - Ed.]
- Salesforce -- Though Salesforce data is not explicitly spatial, there is a great deal of implied spatial content in the Salesforce data holdings of its subscriber organizations. The ability to get this data out and integrated with the other on-premises data holdings of an organization, potentially geocoding along the way, has already opened up possibilities for new and better understanding of CRM [customer relationship management] data within some of our early adopter organizations. [Learn more: Salesforce video - Ed.]
- E57 -- This emerging standard for point cloud data is beginning to see uptake from both scanner vendors as well as organizations looking to make long-term archives of their scans. FME’s support removes the friction that might otherwise be holding back wider adoption and use of this format.
DM: The FME 2013 products include 3D and LiDAR tool enhancements. Are these datasets mostly in use for visualization or are they, and the tools to manipulate them, ready for actual design and construction work?
SS: In the past, visualization was overwhelmingly the primary usage for these datasets. However, we are seeing more and more maturing of the toolsets which can use these types of data for analytic applications. These types of applications require high levels of quality in 3D datasets, and if this is present, then significant and valuable work can be done today.
Figure 1. The impact of flooding is simulated by using FME 2013 to tint blue any points with elevation values that can be specified by the user. (Click for larger image; Image courtesy Safe Software)
DM: There’s a new GeometryValidator transformer tool "which provides a seamless way for users to validate and repair erroneous 2D and 3D data.” Can you offer some insight into the options for this “clean up” process?
SS: As hinted at above, we have received many requests to provide value-add as we move 3D data from one format to another to ensure that it is of high-enough quality for the consuming application. In particular, objects that claim to be closed solids need to be checked to be sure that they are, and if they are not, repaired so that they become closed solids. Another issue that frequently causes problems is the incorrect orientation of the faces which make up the object, the result being they are incorrect as to which side is inside and which is outside of the object. So our GeometryValidator offers options to detect these kinds of problems and, where possible, repair, all in an automated way.
We have implemented a wide variety of checks and corrections for both 2D and 3D, including detecting problems such as self intersections, duplicated points, and bad orientations. Wherever possible, we not only flag the problems, but also attempt to repair them, where repair is able to be done. Early feedback is that this set of options has been very well received by our user community. [Learn more: GeometryValidator video - Ed.]
DM: A very popular target format is the Esri shapefile. You note with the 2013 release these files are created with spatial indexes that load more quickly into ArcGIS. How did you do that? Did you build a better index?
SS: Ah, this one is a small step for us but a big one for our Esri users. The format of the spatial index (or .sbn/.sbx file pairs) used by ArcGIS for shapefiles is not published, but ArcObjects offers a call to make a spatial index on a shapefile for later use. So we’ve built in a connection from FME to ArcObjects so that if a license of ArcGIS is present, we ask ArcObjects to pre-build these indexes. Net result -- shapefiles that are very fast to open and pan and zoom around inside ArcGIS.
DM: FME Server 2013 supports more protocols than ever, allowing it to ingest, process and then provide notifications based on large amounts of real-time incoming data. Can you give an example of how an organization is using (or could use) that kind of power in today’s world of sensors and social media?
Figure 2. This diagram of the FME Server Notification Service illustrates how reporters and subscribers can use FME Server to send and receive notifications through various protocols. (Click for larger image; Image courtesy Safe Software)
SS: Great question. These new protocols were driven by user requests. The TCP/IP protocol is part of the new “real-time message streaming” capability which enables high bandwidth data streams to be ingested and processed by FME Server. In one case, this is being used by a client that has a sensor network. It is passing all the data coming from this entire sensor network to FME Server. FME Server then sends out “notifications” when something of interest is discovered based on rules defined in a workspace. The Java Message Service (JMS) protocol was added so that FME Server could be plugged into an enterprise level data stream adding spatial capabilities. With FME Server 2013 we can handle both high-throughput JMS situations through our “message streaming” or low -throughput workflows through our event processing capabilities.