Evaluating the Usability of Aggregated Datasets in the GIS4EU Project

August 3, 2010

Sharing is Caring

The use of spatial data within the European Union (EU) has been hampered by differences in data structures, geographic scale and language used among the public sector organizations that share and distribute these datasets. Fragmentation of datasets, gaps in availability and duplication of information have occurred because of a lack of guiding principles to harmonize spatial data infrastructures within the EU. In order to resolve this, the EU published the INSPIRE directive (Infrastructure for Spatial Information in the European Community, directive 2007/2/EC), which establishes an infrastructure for sharing geographic data between public sector organizations and facilitates public access to spatial data.

The GIS4EU project (initiated in November 2007) comprised over 20 organizations, including academics, researchers, data providers, technical partners and users (see Figure 1). The project aimed to provide base cartographic datasets for the European community on the following themes from the INSPIRE directive - administrative units, hydrography, transportation networks and elevation. It has developed common data models based on the requirements of the INSPIRE data specifications for the selected themes. Datasets supplied to the GIS4EU project were harmonized by remodeling them into a common data model and aggregating them into unified datasets. The GIS4EU project has evaluated the usability of harmonized, INSPIRE-compliant datasets, in order that the needs of a potentially diverse range of users can be met. This article discusses issues of usability in geographic datasets, and describes the usability testing approach employed within the GIS4EU project. The results of usability testing within the GIS4EU project are described briefly, followed by a discussion of the implications of our findings for organizations that are mandated to adopt the INSPIRE directive.

Figure 1: Organizations contributing to the GIS4EU project were based in a number of EU countries, shown here in orange. For a complete list of participating organizations, please see the acknowledgements. (Click for larger view.)

Usability testing is fundamental for ensuring that users can interact with computer hardware and applications accurately and efficiently. Guidelines for some aspects of computer usability are well established. The International Organization for Standardization (ISO) offers guidelines for Visual Display Terminal (VDT) ergonomics (ISO 9241-11) and for software development (ISO/IEC 9126). Definitions of usability vary depending on the aspect of computing in question. ISO 9241-11 for VDTs defines usability as a combination of "effectiveness, efficiency and satisfaction." ISO/IEC 9126 suggests "the capability of... [software] to be understood, learned, used and attractive..." as a definition of usability. Few researchers have attempted to define data usability. A paper by Monica Wachowicz and Gary Hunter (see Data Science Journal [Spatial Data Usability Special Section], 2: 75-78) suggests five elements of data usability: marketing (e.g. dataset value), quality, software and tools (human-computer interaction), human perception, and cognition and applications (e.g. aggregation, modeling). The relative importance of these elements is likely to vary between users. Additionally, the elements are also likely to overlap depending on the intended use of the data. The authors also highlight the time-dependence of data usability: out-of-date data are seldom useful.

Spatial data have additional, specific usability problems related to the inherent complexity of geographic information. Geographic information is complex for a number of reasons. For example, geographic entities are spatially and temporally dynamic. Representations of many geographic features are scale dependent. Features may be classified in a context dependent way, and any given geographic feature may ultimately be visualized in a number of different ways. Users must often give meaning to representations of geographic features within a dataset, based on textual metadata, which is both vague and context dependent. Geographic data are used in many different ways by a wide range of users, such as public sector organizations, universities and private companies. As usability is primarily determined by the needs of individual users, defining the usability of spatial datasets is likely to become increasingly challenging, particularly as efforts to increase public access to geographic data continue (e.g. through public-participation GIS). Until a robust definition of spatial data usability and an appropriate testing approach are established, assessments of the usability of geographic datasets will be difficult.

The lack of established protocol for assessing the usability of spatial datasets required the development of a testing approach to be employed in the GIS4EU project. Initially, four stages were used to define spatial data usability and consider how these definitions would be used to assess the usability of INSPIRE compliant datasets.
  1. Identification of elements of usability. Members of the GIS4EU project produced spider diagrams in small groups. Elements taken from the literature were used as examples and to stimulate debate.
  2. Defining elements of usability. Groups were asked to define each of the elements identified on their spider diagrams, again with examples to prompt discussion (see Table 1).
  3. Designing tests. Tests were designed within each group, which could be used to decide if a dataset complied with the usability elements defined in stage two.
  4. Refining spider diagrams and definitions. The groups were brought together, and added elements from their spider diagram to a common spider diagram, to combine the ideas of all the groups. The result was one spider diagram showing the elements of usability, an associated list of definitions and a logical test for each element. The final spider diagram is presented in Figure 2.
Table 1: An example of a testing procedure for an element of usability, using a logical answer approach. (Click for larger view.)
Figure 2: The final spider diagram produced by the GIS4EU data providers to identify elements of data usability. (Click for larger view.)

The usability elements identified by the spider diagram approach were then formulated into a questionnaire to be given to the data providers involved in the GIS4EU project. The complete questionnaire is available.The questions mostly required "yes" or "no" answers, with some also including an "almost" category, defined as meaning more than 80%.
Example: Do you think the remodeled dataset covers the needs of the users of the original dataset? Yes / no / almost (at least 80%).
Respondents were encouraged to provide further details if they felt that usability requirements were not met.

Eight of the 10 data providers for the GIS4EU project returned completed questionnaires (a complete list of GIS4EU data providers is given in the acknowledgements). Qualitative analysis suggested that the data providers were largely satisfied that the aggregated and harmonized datasets were accurate and complete. The majority of data providers also agreed that the data and their associated metadata were compliant with INSPIRE and existing international standards for metadata (e.g. ISO 19115, ISO 19139). The questionnaire results also revealed some concerns of the data providers. These were both specific to harmonization processes used in the GIS4EU project and general issues relating to data harmonization. The encoding of metadata using XML (Extensible Markup Language) was not considered to be easily understandable by non-expert users. A solution advocated for this would be to transform the XML-coded metadata to a more user-friendly document using XML Stylesheet Language Transformations (XSLT). Some respondents also suggested that the quality of original datasets provided varied among data providers. This may present a usability problem if some geographic areas of the EU are represented by poorer quality data, particularly if the quality of such data is insufficient for the intended use. General data harmonization issues highlighted by usability testing include the loss of information during feature or attribute matching. For example, features may be present in the original dataset but not in the INSPIRE model and vice versa. Another general issue identified concerns about the way in which information about the original dataset is contained within metadata. Important caveats or information regarding the original dataset could be lost when a common and rigid structure for metadata is used to describe harmonized datasets. The end-user must be able to understand the original context of the dataset’s production and the subsequent harmonization and aggregation processes.

We believe that our experiences of evaluating usability in the GIS4EU project have implications for understanding spatial data usability and are valuable to organizations that are mandated to adopt INSPIRE. The combined iterative spider diagram and questionnaire approach helped to elucidate common themes of usability, and this approach may be of use to organizations implementing spatial data harmonization processes. Many of the usability elements identified within GIS4EU are consistent with those identified in the theoretical approach of Gary Hunter and colleagues (see Data Science Journal [Spatial Data Usability Special Section], 2: 79-89). We contend that the identified usability elements (shown in Figure 2) could potentially develop a common model for representing spatial data usability. This should be developed by future research. It is also likely that countries implementing INSPIRE data specifications will face a similar challenge of how to evaluate usability of their harmonized datasets. However, we must reiterate that the concept of usability is poorly defined, both as a general term and specifically in the context of spatial data. Resolving a tightly defined concept of spatial data usability will significantly enhance the outcome of spatial data harmonization projects. As the exchange of spatial data among public and private organizations, and between expert and non-expert users is likely to increase, defining the "users" and identifying their needs will become increasingly difficult. Increasing exchange and harmonization of spatial data, and increased use of geographic information by non-expert users, suggests that spatial data usability will remain a significant challenge for geographic information professionals.

Acknowledgements: The GIS4EU project was funded through the EU Contentplus program. Dr. Gobe Hobona kindly produced Figure 1 and provided insightful comments during the writing of this article. Staff in the Centre for Geospatial Science engaged in many discussions on the nature of usability and the GIS4EU project. We also thank the organizations that participated in the GIS4EU project: CORILA (Consortium for Coordination of Research Activities concerning the Venice Lagoon System, Italy), GERIN (Intergraph Deutschland, Germany), VUGK (Research Institute of Geodesy and Cartography in Bratislava, Slovakia), UDG (University of Girona, GIS Centre, Spain), ICC (Institut Cartogràfic de Catalunya, Spain), GISIG (Geographical Information Systems International Group, Italy), FOMI (The Institute of Geodesy, Cartography and Remote Sensing, Hungary), RLIG (Regione Liguria, Italy), RPIE (Regione Piemonte, Italy), UNOTT (Centre for Geospatial Science, University of Nottingham, UK), UK (Comenius University in Bratislava, Slovakia), CGE (Comune di Genova, Italy), UNISAP (University of Rome "LA SAPIENZA", Italy), POLIN (Intergraph Polska Sp.z o.o., Poland), IGP (Instituto Geográfico Português, Portugal), INSA (National Institute for Applied Sciences, France), INSIEL (Italy), CSI-Piemonte - Consortium for Information Systems (Italy), INTIT (Intergraph Italia LLC, Italy), RVEN (Regione del Veneto, Italy), MAV (Venice Water Authority - Information Service, Italy), UAV (Università IUAV di Venezia, Italy) and RLODZ (Lodz Region, Poland).


Sharing is Caring

Geospatial Newsletters

Keep up to date with the latest geospatial trends!

Sign up

Search DM

Get Directions Magazine delivered to you
Please enter a valid email address
Please let us know that you're not a robot by using reCAPTCHA.
Sorry, there was a problem submitting your sign up request. Please try again or email editors@directionsmag.com

Thank You! We'll email you to verify your address.

In order to complete the subscription process, simply check your inbox and click on the link in the email we have just sent you. If it is not there, please check your junk mail folder.

Thank you!

It looks like you're already subscribed.

If you still experience difficulties subscribing to our newsletters, please contact us at editors@directionsmag.com