Map Data Controversy in the U.S.: Analysis and Opinion

The USGS has fixed errors in the old DEMs, which had been freely available over the Web at USGS sites.Many people have grown to rely on this source of data and will need to replace their old datasets with the new ones.

At the same time, however, the USGS made several additional changes: it used a new format for the data and it contracted with a private vendor to serve the data on the Web.

Now we are in a situation where the only free online versions of these important datasets are in a format that most people cannot use; those that can use them require hours to download them when before minutes would suffice.

In hindsight, it is obvious that there would be problems.Withdrawing the old data created a sudden surge in demand for the new DEMs.Changing the format surprised many users (despite extensive efforts by the USGS to inform people of the impending change).This has caused a scramble to find ways to get older software to read the new format.Moving to a completely different delivery model was bound to create additional problems.

Doing everything at once was like yanking the tablecloth from the proverbial table: if it is not done just right, your dinner will be all over the room.

Over the years, people who have made major changes in software systems have let us know of the lessons they learned.Some of these are commonplace: Don't take your old system offline until you know the new one is working.Roll the new system out gradually.Provide mechanisms for "reverse compatibility." Benchmark the old system so you can accurately specify requirements for the new system.Provide system redundancy.

None of this happened.

Our intention is not to cast blame.We think the USGS generally has done a spectacular job in generating and making available truly useful, high-quality GIS data.It is the best in the world.Everyone we have contacted at the USGS has been genuinely helpful and concerned.Directions Magazine competes with the private vendor for your attention on the Web and for advertiser dollars, yet we have the deepest respect and regard for their efforts and the contributions they have made to the GIS community.

Rather, our intention is to help people understand what is happening now, so we can all cope with the situation.And it is to suggest how things could go better next time.

What could have been done better? Let's take the lessons one at a time.

• Don't take the old system offline. It was not necessary to eliminate the old DEMs from the USGS sites all at once.The errors had been there; most people could live with them for a few more months.

• Roll the new system out. The new DEMs could have been made available in stages: maybe provide the new DEMs for just a few states first, then a few more, and so on.Measure the quality of the new data delivery process.Respond to user complaints.In this scenario, much less is invested by the time problems are found and the fix is easier.

• Provide reverse compatibility. The USGS states it does not want to be in the data converter business.That is a strategic error.The USGS is in the business of providing data.Whenever it fundamentally changes the format, it must provide mechanisms to mitigate the effects of that change.Providing no converters is tantamount to letting the private sector do it-and charging whatever the market will bear.This is not consistent with the USGS' mandate to make public domain data publicly available at little or no charge.

• Benchmark. Apparently, the USGS had little or no idea of the actual demand for its online data products.It is easy to anticipate that the bottleneck would be bandwidth.Indeed, the private vendor several months ago announced its "premium" policy whereby it would provide faster downloads for a fee.It appears the vendor anticipated there would be problems, but even it did not anticipate the full extent of the problems.A little measurement in advance would have gone a long way to alerting everyone of the problems that lay ahead.

• Provide redundancy. It would have been easy enough to contract with two or more vendors to provide the new DEMs and other USGS data.At the very least, any problems with the new system would have been shared among the vendors, vitiating any allegations of favoritism, monopoly, or unethical dealings that can arise when just one vendor is involved.

The problem is that in order to reserve enough bandwidth for premium users, the vendor has to limit bandwidth for all other users.In our view, this is a glaring violation of the spirit of the agreement with the USGS.It is tantamount to restricting access in the pursuit of commercial gain.It does not matter whether the mechanism of restriction is overt bandwidth reduction (our tests suggest it is) or whether it is over-demand for the "non-premium" slice of the bandwidth.The effect is the same: the public loses.

Simple temporary solutions are available.First, eliminate this two-tiered system.Let all users share the bandwidth on a first come, first served basis.Do this until adequate facilities emerge to serve people fairly and expeditiously.

Second, resume the USGS distribution of DEMs.Now that these data are available on the vendor's site, it would be fast and easy to transfer them to USGS servers for free distribution to GIS users.Clearly it was premature for the USGS to withdraw from the business of providing these data over the Web.

There are other aspects of the current situation that need to be addressed. For example, data can be downloaded at the vendor's site only by clicking through ten layers of ad-laden screens.Datasets cannot be bookmarked because they are assigned random URLs at the time they are selected.These are additional, arbitrary obstacles to data access imposed by the vendor.

These data are an important U.S.asset.Their availability should not be artificially limited, as it currently is.Availability should not be subject to the profit-making pressures that assail any private entity.As a test of a new public-private partnership, the delivery of the new DEMs is only the beginning, but for now it has to be considered a glaring failure.

Things are likely to improve.The USGS and the vendor are very capable and will see to that.But we believe this month's bad experience in unintended consequences and commercialization of a public service has set the stage for ongoing problems.

Bill Thoen, Contributing Editor for Directions Magazine and founder of the popular MapInfo-L discussion list, had a good idea when he recently wrote, "Perhaps we (the GIS Internet community) ought to set up a non-profit clearinghouse to provide these data as an additional alternative.Are there enough sites with extra capacity so that we could carry this load safely and economically across a distributed network?"

Directions Magazine has some capacity.We are willing to help serve data fairly and freely to everyone.If you would like to join us in exploring this idea of a not-for-profit GIS data clearinghouse, then please contact me directly.

I invite you to share your thoughts and suggestions with us.We will post as many of your letters as we can.

--William Huber, Ph.D., Editor