Palantir: An Open Source Development Success Story

“Along with the improved scaling properties, PostGIS [and open source] offers simplified licensing, meaning we can deploy PostGIS indexes on as many servers as necessary for the customer's data. These reasons, among others, are why we decided to engage with OpenGeo to develop on PostGIS and migrate to it as our sole platform”

—Eli Bingham, Technical Lead, Palantir Technologies

Introduction

In late 2007 Palantir launched Gotham, a new geospatial and comprehensive analytics platform designed to meet the challenges of their vast array of customers. Gotham enabled data integration, search and discovery, knowledge management, secure collaboration, and algorithmic analysis across a wide variety of data sources. This data could include many attributes, geospatial properties among them. The enterprise software company aimed to provide a way to perform geospatial searches—defined by radius, route, polygon, or map layer—and then deliver results as layers of points on top of a tiled map visualization. Palantir built a map application (Gotham Map) on top of the Gotham platform to provide this information.

Figure 1. Extensive customer research indicated that meter-level accuracy was a critical factor for adoption. For some customers (such as those in the defense or humanitarian aid industries), this could literally be a matter of life or death. Customers in this space often use Gotham to find the safest options available for those in dangerous situations. For instance, if they had to plan a convoy route in an area that was often under fire, they could search for incidents that had occured within the last two weeks within 200m of the desired transportation route.

The above image shows how Gotham Map is being used to aid in Hurricane Sandy relief efforts. You can read more about this project on Palantir's website.

Building a Platform

When evaluating databases, Palantir kept these requirements in mind and opted to build the platform on top of Oracle Spatial. One key reason for this decision was the fact that Oracle Spatial supported the WGS84 coordinate system and geodetic coordinates. Projected coordinates would simply not be flexible and accurate enough for a global spatial query tool. Palantir also had ample internal expertise and understanding of the database itself, making it a natural fit.

After Gotham’s successful launch, Palantir expanded the platform to support additional geospatial features broadly used in intelligence, federal and local law enforcement, regulation and oversight, cyber defense, fraud prevention, and other areas within the public and private sectors. With so many new users and supported industries, the continues to invest in the system to keep it up to date and effective. In this vein, Palantir's Eli Bingham looked to expand support to other databases.

While not tied to any particular technology, Eli had previous positive experiences with open source technologies and looked to PostGIS. Unfortunately, at that time PostGIS was at version 1.4 and lacked the geodetic coordinate model that could provide the global flexibility and accuracy required. At the same time, Palantir lacked the in-house expertise to extend PostGIS, but decided to make the investment in hiring a developer to extend PostGIS to meet their needs. Eli reached out to his network and was introduced to Paul Ramsey and OpenGeo who were keen to find out how and in what way Palantir was interested in extending PostGIS. After probing the issues, Paul reported back to Palantir explaining what work would need to be completed to add this important functionality to PostGIS—functionality that Paul and others already wanted to build into the project.

Extending PostGIS

PostGIS has long supported a planar “geometry” type, which models geometry edges as straights lines on a plane. For small areas, planar geometry is sufficient, but for global data that crosses the date lines and poles planar geometry fails; polygons that circle the pole can’t even be represented and objects that cross the dateline get badly misinterpreted. Palantir required a new type, a “geography” type, that could model edges as great circles on a sphere. Along with the geography type, Palantir needed calculations that worked against a spheroid and provided meter-level accuracy in its results.

Figure 2. Above are examples of where the geometry type fails. On the first line is a polygon covering a pole. On the second line a polygon that crosses the dateline, which is projected across the entire world. For more information on these projections, please see Paul Ramsey's Introduction to PostGIS slide deck from FOSS4G-NA 2012; the slides specific to these projections are at the end of the presentation.

Adding spheroidal geography support to PostGIS was especially interesting to OpenGeo, as it affects and benefits such a wide range of geospatial developers and users—most notably those developers who are new to geospatial and may find geography useful because they aren’t very familiar with GIS or map projections. Geography allows them to work with latitude and longitude data without having to worry about specific projections. It also benefits large organizations like Palantir who have truly global data, which covers the poles, datelines, and everything in between, allowing them to model their data simply and correctly.

To make the geography development happen, OpenGeo engaged with Palantir to work on a few major enhancements. The first enhancement set was released with PostGIS 1.5 and included:

the new "geography" type;
new spheroidal algorithms for indexes, distance calculations and point-in-polygon;
a new disk storage format; and,
a new system-based column metadata storage.
Supporting Open Source Development

While the 1.5 release satisfied Palantir's needs, they were interested to hear Paul’s opinion on how they could leverage PostGIS in the future. After analyzing their use-cases, Paul suggested that high speed "nearest-neighbor" searching could help them build more useful applications. In a second enhancement set, Palantir also funded the development of index-assisted nearest-neighbor searching for PostgreSQL, which debuted in PostgreSQL 9.1 and PostGIS 2.0.

Eli and the other Gotham developers were happy to fund this open source development and were especially impressed with the network effects of community bug testing and further feature development. To them it seemed only fair that others would be able to benefit from their investment, since they had benefited so greatly from what was already built by others and would similarly benefit from what others built in the future. Their assumption has proved correct; since Palantir’s original investment, many users have funded or developed new functions and performance enhancements for geography calculations.

Palantir’s early intention was to use PostGIS integration along with a starter pack on top of the Gotham platform, but after validation and performance testing they found the PostGIS solution exceeded their other offering's scaling properties. When asked about the benefits of using PostGIS for Gotham, Eli Bingham said:

“Along with the improved scaling properties, PostGIS [and open source] offers the lack of license issues, meaning we can deploy PostGIS indexes on as many servers as necessary for the customer's data. These reasons, among others, are why we’ve decided to migrate to PostGIS as our sole platform [for Gotham]...”

This freedom means they are no longer concerned with how many servers they deploy on, as is so often the case with proprietary closed-source solutions. Palantir not only discovered that open source can offer a superior solution, they also found additional benefits in engaging with the open source community, both directly and through a commercial open source company. Using the work they did with OpenGeo as a model, Palantir has gone on to support open source in a variety of ways, and they’ve even open sourced some of their projects, on their GitHub page.