Standards That Control Standards

The handling of geographical data, unfortunately, is no different.

Recently, I have been on a quest to understand what is happening with geographical data.It all started when I attended an event with an associate who was looking to acquire a GIS solution for a large business with an extensive property portfolio spread across the UK.They were having problems getting the geographic data at a reasonable price because the agencies that sell it did not seem able to cope with private companies who wanted small areas of coverage.However, the issue that quickly arose with me was the way that commercial GIS solutions were selling compatibility.

ESRI .SHP Files
Time after time, as we discussed the various merits of GIS solutions with their sales teams, one of the key selling factors was compatibility with an ESRI Shapefile.Only one solution talked to us about GML and open standards before the ESRI format.

I am familiar with ESRI's strong position in the market but I am also fully aware of the standards that exist to enable geographical data to be shared between different solutions.After a while the whole ESRI dominance thing got me angry and I set out to find out what is causing this situation.

One thing that came out very quickly is that ESRI is one of those businesses you either love or hate.It's the Microsoft of the geographical world in that it defines its market and there are those that find that situation difficult to handle.It was one of those companies in the right place at the right time when the demand for geographical solutions exploded.The question is whether it exploits its dominance in a way that is bad for the user community in general.

It doesn't take long to find out that ESRI is a significant player in the development of standards.It does a great deal of work with the OpenGIS Consortium (OGC) and it has input at the FGDC level.What is interesting is that ESRI has employed some of the people that had independent input to standards.This implies that it has sought out the best minds to implement standards in its own products.

Certainly, a quick scan of the corporate marketing brochures and web site indicates that ESRI is strongly supportive of standards.There is much talk of the ISO standards and technical committee 211 (ISO TC211) that is doing much of the work in this area.There is support in the product for the National Spatial Data Infrastructure (NSDI) and OGC Simple Features Specification.However, it is this last standard that starts to give away the problem.

The current standards are too basic for many GIS applications.Therefore ESRI adds 'extras', including z values, annotation and support for survey and raster data.A Shapefile is just more useful than a standards-limited data format.

Tangled net of standards
Understanding the workings of standards really ought to be very easy. We have the International Standards Organization (ISO) that works things out on behalf of us all.It works very slowly but it gets there in the end.The speed of operation is a problem for businesses because, inevitably, they end up using draft standards because they can't wait for the final approval.

In the geographic space ISO has TC211 that has come up with a pretty good set of standards and is working on more.The most important standards as far as data sharing are concerned are ISO 19115 and ISO 19107.ISO 19115 is a metadata standard.It defines what should be in data catalogues and ensures that we all call things with the same names.ISO 19107 provides a schema for spatial data so that the way in which data is stored is consistent across compliant solutions.
We have already established that not everybody feels that TC211 has taken the standards to a really useful level.In these situations, the IT industry inevitably kicks off an alternative movement.

The OpenGIS Consortium was set up to create XML standards for data exchange. It was recognition that this was a technology that could be used to create interoperable infrastructures and, at that time, GIS solutions had not exploited it successfully.Unfortunately, it seems that OGC didn't feel the need to align itself with the activities of TC211.Instead, it repeated much of that work and came up with something different.The highest profile standard is the Geographic Markup Language (GML), an XML dialogue designed to enable applications to exchange geographic data.It is only recently that we have started to see a convergence of OGC standards with their ISO equivalents.

Even when OGC and ISO get their acts together there is still the problem of international politics.When it comes to governments, national standards bodies provide the driving force.Therefore, in the USA, if ANSI doesn't like an ISO standard it doesn't get adopted.The British Standards Institute (BSI) carries the same power in the UK.

The USA has an extra level of administration.Thanks to Bill Clinton, the Federal Geographic Data Committee (FGDC) was created to pull together the different formats being used throughout the US and to ensure that there was a proper national standard for spatial data.In itself, this has to be seen as federal recognition that the standards weren't working.The result of this has been the National Spatial Data Infrastructure (NSDI). However, there are still regional variations (RSDI) and there are still states that don't comply with any standard at all.NSDI works only for the USA but work is progressing on the development of the Global Spatial Data Infrastructure (GSDI) - surely just another repeat of the work already carried out by ISO and OGC.

Elsewhere work is being done to create metadata profiles for key regions, including Europe and Australia.

Self interest
It is clear that the development of standards for sharing and handling geographic data has to take into account much of the self-interest that comes from the companies that are involved and the government agencies that are tasked with finding the best solutions for their own locations. Vendor self-interest is a feature of all IT standards.Rather than starting from a blank sheet of paper, it makes sense to take a practical view of existing technology and then to bend it to suit wider requirements.Having your technology accepted as a foundation for a standard can be lucrative since IPR and patent rights are nearly always maintained.

Similarly, vendor self-interest is usually the cause of multiple standards for the same purpose.Failure to agree with standards or exclusion from the process often leads to an alternative development followed by a marketing war.Neither side will ever back down.The end result is great for competition but no fun for the user community.Eventually, the need to compete reduces the value of the standards.Added extras become the norm and a new market develops, as products appear to unify the different standards.

This is beginning to happen with geographic data.Some of the NSDI work can be seen in this light.In the UK, the GIgateway is the result of several years' work with both private and government funding.In Canada, we have seen the development of the JUMP unified mapping environment, originally an open source project that was designed to be able to connect to geographic data from a variety of sources.Unfortunately, so far, it seems to have been restricted to GML and Shapefile formats - thus taking us full circle back to the dominance of ESRI's data structures.

Finally...
The IT industry has never been good at managing the development of standards.There is too much self-interest and the process of development is too slow for users that need a rapid solution to their technology problems. The situation is made worse with geographic data because political interest is added to the commercial interests to be served.

The standards being developed by ISO, FGDC, OGC and others are creating confusion without actually serving up a practical solution.This is why ESRI has added extensions to its Shapefile format.

ESRI is a dominant force in the GIS market.This is why so many third parties adopt its format.ESRI continues to work with standards bodies but it cannot be blamed if others choose to use the Shapefile because of its practicality.Other major players in the market - MapInfo, Cadcorp or Intergraph - would be just as happy to have their formats used in the same way.None of them offer totally standards-based solutions.

In the end, the standard has to be defined by the products that the market buys.The best standards are those that evolve through day-to-day use of practical and useful products.Microsoft has proved the point with Windows and now, perhaps, we can to look to dominant technology suppliers to set "de facto" definitions for geographical data.

Web sites: