The Technology Behind the New Geodata.gov and the Non-Technology Challenges Ahead

On January 31, the Department of Interior announced that ESRI had won the contract to update the Geospatial One-Stop (GOS) portal, also known as geodata.gov.There were several bidders and the top three demonstrated their solutions as part of the competition.Attendees at ESRI's Federal User Group Meeting in February got a sneak peek at the prototype of the selected solution.And, so did two of the Directions Magazine editors.

The Metadata Catalog: Many Ways In
First, it's valuable to understand the underlying architecture, which is significantly different from that of the existing portal, which ESRI also built.At the core of the new solution is a metadata catalog.Remember that the geodata.gov website does not "hold" any data, but rather acts as keeper of metadata pointing users to data of interest.When you choose to look at or download some data, it's done from the source location, usually managed by the data's steward.That metadata catalog "exposes" itself in different ways for developers and end-users.

Geodata.gov V2 Architecture Diagram (Click for larger image)

The catalog service is a tool for developers.It allows them to "tap into" the catalog using a variety of standard methods including the OpenGIS Catalog Service for the Web (CS-W), via a text search, via a portlet (more on those later) and in a few other ways.These different ways to access the catalog programmatically mean that Web developers, even "non-geo" developers, can build specialized portals or websites for federal agencies or local or state governments that tap into the "big catalog" of geospatial metadata.

Basic Search Box

The end-user interface, which most GIS professionals and likely "regular people" will use, is a webpage powered by some new underlying technology.It looks like any other search engine with a single white box and search button, but that's misleading.ESRI has "enhanced" the standard Google Search Appliance (a standard Google product) which sits behind that search box to be "geosavvy." The enhanced appliance is tuned into geographic names (via a gazetteer, an ArcWeb service) so it knows about locations.For example, it's smart enough to know that Denver (Colorado) and Flood (Virginia) are both place names."Regular" Google, doesn't know that.The enhanced search tool is "smart enough" to ask about which geography is of interest.If in fact you were interested in floods in Denver, the system can get the coordinates of Denver and look for data "in the area." The Google appliance ties into ArcIMS which provides the online viewing of the data found and some aspects of the data presentation - say listing records found from largest area to smallest (or vice versa).

Geospatial "smarts" enhance the Google search tools (Click for larger image)

The Google searches are fast, just like "regular" Google searches.And some of the "goodies" from regular Google are still available.So, if you misspell Denver, "Danver" you might see a "Did you mean Denver?" question.Google will also sort results by relevance or date.And, like regular Google there's an Advanced search, which offers data type/format, data category, spatial filtering and other options.The hope is that most users can be successful with just the "simple" search.

An advanced search tool provides more filtering and sorting options.

Portlets
Underlying all this is the IBM Websphere Internet development environment.(That's one of the government's preferred environments.) Websphere and other solutions (SAP, BAE's Weblogic and others) all support JSR-168, a Java standard for "portlets." Portlets are little applications that can be "plugged into" other websites which use JSR-168 supporting Web development environments.Geodata.gov V2 will allow users to choose which of many portlets they might want to have available when they visit once they log in.(Think of My Yahoo! or the equivalent.) We saw a weather portlet, for example, provided by Pinnacor, as one customization a user might make.But GOS V2 is not just a consumer of portlets; it also "publishes" its portlets for external developers to use.We saw the search portlet accessed via an SAP implementation.Support for this standard means that elements of the GOS portal can be reused in other portals, hopefully saving time and money for others who want to provide access to geodata.gov's metadata or other services.

A weather portlet (from an outside provider) might be one of the portlets a user selects for a customized geodata.gov experience.(Click for larger image)

Standards
The mapping side of the portal will support a host of Open Geospatial Consortium OpenGIS specifications including Web Map Service (WMS), Web Feature Service (WFS), Web Coverage Service (WCS), Web Map Context, as well as ESRI's ArcIMS interfaces.What does that mean? It means that data made accessible to geodata.gov can be delivered in standard ways to end-users.

For example, pretty much any data "published to" geodata.gov can be delivered out as a Web Map Service.That means if you have a WMS client (a thin, or thick desktop client or even one on a handheld) you can access WMS data from geodata.gov.If a data provider delivers its data via Web Feature Service (WFS)(live vector data) to geodata.gov, a WFS client can tap into it.Or, perhaps equally exciting, data delivered to geodata.gov via WFS can be processed via tools from Safe Software to be converted to a selected vector format, clipped, reprojected and zipped for download.

The metadata catalog will support data harvesting as it has in the past via Z3950 (Isite), ArcIMS metadata, Open Archives Initiative (OAI), Web Access Folder and the OpenGIS Catalog Service for the Web (CS-W).Others methods, including manual uploads from data custodians are also supported.We learned that more than half the metadata currently indexed in geodata.gov is automatically harvested, an encouraging sign.GOS administrators are enlisting "ambassadors" to help bring new data custodians into the fold.

Data Channels Become Communities of Interest

Communities of Interest will have their own discussion areas.(Click for larger image)

The data categories, which in version one were organized into "channels," are likely to upgraded into "communities of interest." These communities might revolve around a particular geography (the Great Lakes area) or a topic (wildfires) or a data type (HVRR data).Each community will have its own forum and "chat room." And, of course, new communities can be added as demand increases.The current vision is that only registered users of the portal can be members of communities and there is ongoing discussion about whether all communities will be open to all interesting parties or if some will require the "OK" of a moderator of sorts.Other questions are still open including this one: should the federal government open geodata.gov such that Google can index the geodata.gov metadata and make it searchable by "regular" Google?

Geodata.gov Marketplace

Information about upcoming data buys is not just listed, but mapped.(Click for larger image)

The geodata.gov Marketplace area of the website will be enhanced.Recall that the Marketplace is a place to list planned data acquisitions and help those who might be interested in participating in such acquisitions (or at least not spending money to acquire the same data!).Instead of a simple list in version 1, that information will be mapped, so it will be easier to see if neighboring counties or states are planning buys in the future or are looking for data now.Further, there are tools to communicate with those involved in acquisitions to form partnerships.While there are some legal requirements for some federal agencies to disclose data acquisition plans, there may need to be some enticements to further engage states, counties and local governments to use this area of the website.

Alerts

Users can set up alerts at geodata.gov.When conditions are met, information is posted on the portal or delivered via e-mail.

Users can set up alerts to be posted to the website when they return or to send e-mail when certain conditions are met.A user interested in data about a particular county in Colorado can set an alert to indicate when new data is registered in that geography.During an emergency situation, such as a hurricane, users might have e-mail sent each time a new predictive map is available.

Expectations and Goals
The portal is slated to go live toward early summer.The stakeholders along with the public, have provided lots of feedback on version 1 which the version 2 team is using as a key resource to guide development.The team lists four key goals for version 2: enhanced usability for non-geo users as well as GIS professionals, enhanced performance (GOS currently holds some 90,000 records, with the expectation it'll hit 300,000 in the coming months), enhanced interoperability (via OGC and other specifications) and enhanced functionality (keeping what works and updating it).And, the version 2 team hopes to provide a single authoritative place to start a query for U.S.spatial data.From what we saw, they are well on their way.

Concerns
The second coming of GOS presents both opportunity and challenges.There are two key concerns that come to into play.The first concern is in regard to how to engage stakeholders to contribute to the "greater GOS." To make the new portal an effective, relevant resource, it must support local and state agencies and their respective online data offerings and GIS clearinghouses.So a basic question goes beyond the technology to harvest data and make the system interoperable: "How do we incentivize participation of governments at all levels both in and outside the Department of Interior to participate and share their data?" The information from GOS is comprehensive but vastly incomplete if it does not include contributors at the state and local level.

Portals, in all contexts, rely on frequent, if not constant, updating.It is one thing to put the onus on federal agencies; it is quite another to rely on the constrained resources at the local level.And yet, the most current geospatial data often reside in a county assessor's office or, if you are lucky, with the local government's GIS analyst.

An ancillary concern revolves around the question of "Who knows and who cares about GOS?" A survey last fall by Geospatial Solutions revealed that 40% of respondents had never heard of Geospatial One-Stop.That publication, and to our knowledge virtually every geo-related publication in the U.S.and several abroad, covered not only the concept but the launch of the portal.

How will geodata.gov be promoted? That's not ESRI's job and it's not the job of the U.S.Federal Government to hire a PR agency to spread the gospel according to GOS.So, does it lie fallow after this second release? Will the word filter down to local and state authorities that they can participate among the GOS "Communities" to develop their own "GOS-like" portals to draw from, or contribute to, the metadata catalog?

GOS is obviously a tremendous resource already and could be an even better one if as much emphasis is placed on delivering the message on how and why to participate in GOS as there is on the feature-rich resources that have been built into the portal.

Share