Combining GIS and Semantic Technology to Create a Cultural Visualizer

In northern Spain, the cities of Pamplona and San Sebastian both border the westernmost edge of the Pyrenees Mountains and southwestern France. Both are working on integrating advanced semantic technology with a novel GIS augmented reality (AR) device. The project, dubbed CINeSPACE, promises to immerse tourists in the rich cultural heritage of the area by literally morphing the user into the past through the use of multimedia archives. (YouTube has an example posted.)

Users visiting these cities will be able to rent the one kilogram (2.2 lb) portable device at tourism offices. The novel design, shown in the image below, includes both a PDA type of device with a GIS interface displayed on a touch screen to help the user navigate and select multimedia content, and video binoculars to create the augmented reality effects.

Image of user with the CINeSPACE device in San Sebastian.

The navigation system, developed by the Fraunhofer IGD group in Darmstadt, Germany, uses four different technologies to allow maximum positioning accuracy. GPS is used to establish the position within approximately 10 meters. Then markerless optical tracking is used to establish a more defined location. Markerless tracking uses algorithms which compare reference images of real objects with images captured in real time to produce the current location and orientation compared with the location and orientation where a reference image was captured. This technique uses real images as opposed to artificially placed bar code-like images placed in the field view, as is the case with optical tracking with markers. Markerless tracking is the same type of technology used in Microsoft's Photosynth software to position the location of the camera when a photo is taken.

Then, inertial tracking is used by mathematically integrating the equations of motion to find the current position from acceleration data provided by integrated circuit accelerometers. Finally, additional orientation information is found from solid state accelerometers which measure the direction of the force created by the earth's gravitational field on microscopic cantilevers. An electronic 3D compass is used to provide the orientation of the device relative to the earth's magnetic field.

All of these methods allow the device to provide position and orientation information so accurate that historical film clips can be placed above the real image and then morphed into the real scene being viewed, creating a sensation something like daydreaming.

The particular challenge presented by the device is the need to allow the user to access a wide variety of geo-referenced information in a small and portable device, and at the same time allow the user to annotate information in an efficient manner. To address these needs a novel semantic GIS technology has been developed at Tracasa SA, called GeoConcepts ontology.

The GeoConcepts ontology provides a way to capture in a GIS system the spatial dimension of concepts, defining whether they are traditional geographic features (street, building or town) or vague concepts (events, daily activities, historical milestones or personalities) common to geospatial thinking.

In order to initially populate this GeoConcepts ontology, data are collected from social websites such as Panoramia or Flickr, where user multimedia content is normally stored in a geo-encoded format. Polygons are calculated that include clustered data points and reject data points which are outside of chosen statistical tolerance limits to eliminate erroneous or badly tagged photos. The resulting ontology is shown in the figure below for a set of GeoConcepts for the city of San Sebastian. Both normal GIS concepts like City Hall (Ayuntamiento in Spanish) and conceptual objects like fireworks (fuegos in Spanish) are shown. Also of interest are those concepts with an implied interrelationship to other polygons, such as "nice view of Santa Clara."

A Google Earth image of San Sebastian. (Click for larger image)

With the GeoConcepts ontology loaded into the CINeSPACE device, the simple act of looking in a direction through the video binoculars provides the user with feedback as to the GeoConcepts at which he is looking. This is very useful for video and image annotation and for content selection.

The CINeSPACE device is a novel combination of GIS and semantic technologies providing users with an intuitive and novel interaction with the cultural heritage of a city. The device will be available during field trials in the cities of Glasgow, Scotland; San Sebastian, Spain; and Venice, Italy in the beginning of 2009.