Lars Rasmussen on the startup of Google Maps

Here's some interesting background to where Google Maps came from.  Must feel pretty good to have created a revolution (slippy maps) - and maybe to have just missed creating another one (Google Wave as an alternative to email).

Data model diagrams for GTFS

Recently I've been doing some work with the General Transit Feed Specification (GTFS).  This is a simple data model initiated by Google, in order to allow transit organizations to publish their schedules for routing and visualization purposes.

As usual, Google has nice, succinct documentation for the GTFS format (which is a set of CSV files embodying an implicit data model).  But I was surprised to find that they don't provide any kind of data model diagram showing the relationships between the various entities (files) in the model.  I have a feeling that they think this would be "too technical" for users - or maybe they just don't believe in using a long-established body of theory?

So, to help my understanding of GTFS I put together a simple diagram of the data model.  I used Google Diagrams, which is why this looks a little primitive.   (Hmmm... maybe another reason they don't do diagrams is that they are religious about dog-fooding?)

After doing this I found the paper A Transmodel based XML schema for the Google Transit Feed Specification, which has a much more detailed UML model of GTFS:

The end goal of doing all this was to be able to transform a GTFS data feed into shapefile format.  For this I used a simple JEQL script.  But that's another blog post.

Paul Ramsey's Keynote at FOSS4G 2011

As usual, Paul delivered a highly amusing and thought-provoking keynote at FOSS4G 2011.  For the benefit of the hard-core twitterless, the slides are here.

Everything you wanted to know about UK Coordinate Systems

Those helpful folks at the UK Ordnance Survey have provided a fine reference Guide to Coordinate Systems in Great Britain.  It's well worth reading if you're interested in the intricacies of coordinate systems, datums, surveying and GPS.  It's a bit technical, but if you can handle coding in Javascript you can probably get something out of it...

An interesting extract gives the general flavour:

• The WGS84 Cartesian axes and ellipsoid are geocentric; that is, their origin is the centre of mass of the whole Earth including oceans and atmosphere.
• The scale of the axes is that of the local Earth frame, in the sense of the relativistic theory of gravitation.
• Their orientation (that is, the directions of the axes, and hence the orientation of the ellipsoid equator and prime meridian of zero longitude) coincided with the equator and prime meridian of the Bureau Internationale de l’Heure at the moment in time 1984.0 (that is, midnight on New Year’s Eve 1983).
• Since 1984.0, the orientation of the axes and ellipsoid has changed such that the average motion of the crustal plates relative to the ellipsoid is zero. This ensures that the Z-axis of the WGS84 datum coincides with the International Reference Pole, and that the prime meridian of the ellipsoid (that is, the plane containing the Z and X Cartesian axes) coincides with the International Reference Meridian.
• The shape and size of the WGS84 biaxial ellipsoid is defined by the semi-major axis length a  6378137.0 metres, and the reciprocal of flattening 1/f  298.257223563. This ellipsoid is the same shape and size as the GRS80 ellipsoid.
• Conventional values are also adopted for the standard angular velocity of the Earth, and for the Earth gravitational constant. The first is needed for time measurement, and the second to define the scale of the system in a relativistic sense.

Now then, if you find yourself getting lost on the way from Foyle's to the nearest Lyon's tea shop, you can't say you haven't been warned!

Interview with Directions Magazine

The Twittersphere is buzzing with my latest (well, only) interview with Directions Magazine on the State of the JTS Topology Suite.

Props to Rob Rutherford and Amber Weber at Clover Point for setting this up!

Would you be lost without it?

Nowadays GPS is ubiquitous in electronic gadgets (no electric toothbrush is complete without it!) GPS is a classic example of iceberg technology ¹. An output which in one sense verges on trivial - two numbers! - in fact relies on a huge unseen infrastructure of technology, physics, organizations, and yes, government funding.

The UK Telegraph has an interesting article on GPS ² covering some of the technical, scientific, historic and political background to the system.



On a side note, I'm happy to observe that statements like this aren't as common as they used to be:

Slightly worryingly, when I was invited to watch a “contact” in another control room earlier in the day, the software crashed and a familiar, blue Microsoft Windows screen appeared on the wall

Notes:

1. yes I made this up. My entry for Internet memedom!
   
2. don't worry, it's still safe to use your cellphone after visiting this site
   

JTS 1.12 released

JTS 1.12 has been released, and is now available for download at SourceForge.

Some highlights of this release include:

• Single-Sided Buffers
• MinimumClearance function
• nearestNeighbours methods for STRtree
• Vector2D class for vector operations
• DD (DoubleDouble) enhanced precision arithmetic
  
• improved robustness for Delaunay and Voronoi computation
• enhanced geometry digitization in the TestBuilder, along with a dynamic grid

... along with many other features, enhancements and fixes.

Single-Sided Buffer

Apple walls getting higher

ZDNet has an interesting long view on the Apple iCloud announcement. Money quote:

The mobile world is moving away from the chock-full-o’-content web and increasingly becoming a world of closed pipes of content, with Apple, Google, HP and Microsoft regulating the flow. Tech fiefdoms scattered across a vast open land have expanded into warring nation-states with adjacent borders. The Mac vs. PC vs. Linux argument from the early days of consumer computing has lost a great deal of its luster in recent years with the development of cloud computing on the open web, but the concept of platform wars is quickly making up for lost ground with the development of cloud computing in the closed mobile space.

I've always found the Apple walled garden a bit scary in its restrictions. Now the walls are getting higher and thicker - and other vendors will be following suit as fast as they dare.

I don't find Google's garden as threatening, for some reason. Perhaps because they have a different business model, based on page views rather than selling devices or software. For now that encourages them to not put any limits on where and how you access their services. But will that change if the Chromebook takes off? Is the fact that Honeycomb is not open source a tiny black cloud on the horizon?

Java gets Reheated at OSCON

It's good to see that O'Reilly is adding a Java flavour to the OSCON conference. They have a good blog post highlighting why they think Java (and the JVM) is still one of the most important programming platforms around, especially for open source development.



Some of the interesting Java projects that will be covered at the conference are:

• Incanter, a Clojure-based statistics library (this project is particularly interesting to me, since it overlaps with some of the target space of JEQL)
  
• Gradle, a Groovy DSL for building, testing, and deploying software
• Jenkins, a continuous integration platform
• Neo4J, a graph database

Some may think that Java is entering the Red Giant phase of language evolution, but it still has the capacity to throw off some powerful solar flares. The longer the coffee pot sits on the stove, the stronger it gets...

JTS arrives in JavaScript

As I predicted back in 2008, JTS has now colonized the JavaScript ecosystem, thanks to the work of Bjorn Hartell on JSTS. At least, the project has been launched - it's a bit unclear as to how much functionality is actually there.

I hope to see some slick browser-based UI clients pop up, using Canvas or perhaps SVG?

The language scorecard for JTS now looks like:

• JTS - Java (and Groovy, Scala, Jython, etc)
• GEOS - C/C++
• NET Topology Suite - MS .NET
• Shapely - Python (via GEOS)
• Ruby - via GEOS
• PHP - via GEOS
• R - via GEOS
• JSTS - JavaScript

[Update: Jan 31 2013] - Added R to the list

FOSS4G is the place to be!

FOSS4G 2011 is happening in Denver this September.

I've submitted two abstracts for presentations:


Spatial Processing using JEQL

JEQL is a simple yet powerful language designed for expressing spatial (and non-spatial) processes. It follows the Table-Oriented Programming paradigm and provides a SQL-like query language. This talk gives an overview of JEQL and show some examples of its use.


What's New in JTS

The JTS Topology Suite has had numerous improvements in the past few versions. These include performance improvements, bug fixes and new functionality such as Delaunay Triangulation, Single-Sided Buffers, and Hausdorff Distance. This talk will discuss the new features, and demonstrate them using the JTS TestBuilder utility. Potential further features will be presented for discussion.


Vote early, vote often - and vote for me!

See you in Denver....

Polygon Triangulation via Ear-Clipping with Delaunay Refinement

After this thread on the JTS list Michael Bedward was inspired to create a polygon triangulation algorithm using the Ear-Clipping approach. Standard Ear-Clipping algorithms don't handle polygons with holes, but Michael made a nice extension to do this. (David Eberly has a good write-up on this subject here, and of course there's always Wikipedia). One problem with ear-clipping is that it produces sub-optimal triangulations (in the the sense that it creates lots of very skinny triangles, which are visually and computationally unappealing). I suggested that he add a refinement step based on "flipping triangles" to improve the quality of the output triangle mesh. This is similar to the approach used in Delaunay triangulation algorithms, and in fact it turns out that a good flipping criteria is to test for the Delaunay condition. (Two adjacent triangles which form a convex quadrilateral are Delaunay if neither lies in the circumcircle of the other. Wikipedia has a nice visual explanation). This code will get added to JTS in the next release. In the meantime, here's some cool pictures showing how it works. A gnarly test polygon:

Before refinement

After refinement

A country that has been in the news recently:

Before refinement

After refinement

Update (Nov 1, 2021): JTS now has Polygon Triangulation code.  See this post for details.

Dynamic Views in Google Blogs

Check out the cool new Dynamic Views feature for Google Blogs. I'm not sure how useful this is actually going to be for readers (especially since you have to do URL editing to invoke them - WTB!? (Where's The Button 8^)

But they sure look purty. Here's a shot of the Flipcard view:

Slope/Aspect/Elevation using JTS

David Skea is a longtime colleague, JTS contributor, and all-around geospatial guru. He's developing a Slope/Aspect/Elevation service to be used in forestry-related applications here in British Columbia. His work is a nice example of using JTS to perform real-world spatial processing.

The key to computing slope, aspect and elevation is to have a Digital Elevation Model (DEM) available for the terrain in the area of interest. In this case, the terrain data is provided by the TRIM irregular DEM, which is a dataset of over 500 million mass points covering all of British Columbia. To compute the required values for a given polygon, the first step is to extract only the mass points in the immediate region of the polygon (i.e. using the polygon envelope with a suitable buffer distance. The JTS Delaunay Triangulation API is used to compute a TIN triangulation from the TRIM mass points.


Using a JTS PointOnGeometryLocator, the triangles whose centroids lie in the polygon are selected. Since each mass point has an elevation value, each TIN triangle is located in 3D space. The normal vector can be computed for each triangle, which provides the slope and aspect values. The elevation is the height of the triangle centroid.

The SEA values for the triangles are averaged to compute the overall slope, aspect and elevation value for the polygon.

Here's a screenshot showing the results, using Google Earth as a convenient 3D visualization tool (click for full-size image).

GeoGeeks presentation on Geometry Libraries

Paul Ramsey has been doing his usual excellent work of promoting geo-geekery by setting up a Victoria GeoGeeks Meetup. Tonight I gave a presentation on Geometry Libraries for Fun and Profit.


I was intending to give it back-to-back with a presentation on JTS, to make the whole subject more concrete - but I rambled on for so long I was yanked from the stage before I could start on the second one! That's actually good, since it will give me time to polish up the presentation a bit more. In the meantime, the current slide deck is here: JTS - A Libary for Geometry Processing.

Charting with JEQL

SQL has an elegant syntax for grouping and aggregating data. Java has powerful libraries for charting and graphing (like JFreeChart). JEQL has easy handling of tabular data. These go together like chocolate and peanut butter - and, um, more chocolate. Put them all together and you get this code:

CSVReader troute hasColNames: file: "routeLine2.csv";

tCityCount = select count(*) cnt, fromCity city
 from troute
 group by fromCity
 order by cnt desc;

tdata = select city key, cnt value from tCityCount limit 20;

Chart type: "bar"
 data: tdata
 extrude:
 color: "00c0f0"
 showItemLabels:
 xAxisTitle: "Top Cities by Originating Air Routes"
 xAxisLabelRotation: -0.5
 width: 1000
 file: "cityRoutes.png";

producing this graph:


This uses the same air routes dataset created in a previous example, with a simple grouped summary to show the 20 cities with the most air routes originating in them.

JFreeChart is a bit notorious for being hard to use, but the JEQL command hides all the gory details behind a much simpler (but still powerful) Chart command interface. Currently Bar, XY, and Pie charts are supported. It will be easy to add other chart types as required.

Also on the ToDo list is to provide an option to emit Google Chart URLs, using the same set of options (at least, the ones that are applicable - and maybe some new ones).

Note: this is an experimental extension, which is not in a final state in the current JEQL release.

JEQL 0.9 released

Good news for those who wanted to try out the geodetic air routes JEQL script. The script requires JEQL version 0.9 to run. This version is now available for download.

Goodbye LAMP, hello SMAQ

A great article on the emerging SMAQ stack for big data.

I wonder if PIG does spatial?

Diving deep into Computational Geometry

For a deep dive into the bracing waters of cutting-edge computational geometry, check out the online proceedings of the Canadian Conference of Computational Geometry.


Do NOT expect to see these algorithms appear in JTS anytime soon!

Visualizing geodetic information with JEQL

The geo-blogosphere has been buzzing about the global Facebook friends visualization. This was done by Paul Butler using an R script and some clever techniques for working with geodetic data.



This kind of lightweight spatial analysis and visualization is squarely in the target zone for JEQL, so I thought I'd try something similar.

Of course I don't have access to the Facebook friends dataset, so I needed some other suitable dataset of global-scale links. An obvious candidate is airline routes. Luckily there is an excellent open data repository called OpenFlights. It has datasets which are tables of airport locations and air routes between airports:

airports.dat:

1,"Goroka","Goroka","Papua New Guinea","GKA","AYGA",-6.081689,145.391881,5282,10,"U"
2,"Madang","Madang","Papua New Guinea","MAG","AYMD",-5.207083,145.7887,20,10,"U"
3,"Mount Hagen","Mount Hagen","Papua New Guinea","HGU","AYMH",-5.826789,144.295861,5388,10,"U"
...

routes.dat:

0B,1542,AGP,1230,BBU,1650,,0,738
0B,1542,ARW,1647,BBU,1650,,0,340
0B,1542,BBU,1650,AGP,1230,,0,738
...

The first step is to prepare a suitable dataset for rendering. To get a table of FROM/TO locations, the routes table needs to be joined to the airport table. This is trivial to do in JEQL. After a bit of cleanup (such as removing missing data and duplicate routes), the final result is a dataset of links between airport locations using Lat/Long coordinates:

28,"Bagotville","Canada",48.330555,-70.996391,146,"Montreal","Canada",45.470556,-73.740833
29,"Baker Lake","Canada",64.298889,-96.077778,132,"Rankin Inlet","Canada",62.81139,-92.115833
30,"Campbell River","Canada",49.950832,-125.270833,119,"Comox","Canada",49.710833,-124.886667
30,"Campbell River","Canada",49.950832,-125.270833,156,"Vancouver","Canada",49.193889,-123.184444
...

As Paul found, a few other steps are needed to produce a visually appealing map:

• Densify the route links to produce approximations to great-circle arcs
• Break the arcs at the International Date Line to allow them to render correctly
  
• Colour-theme the routes from longest to shortest using lighter colours for shorter routes
  
• Render the lines with longer ones further back in the Z-order

For good measure I also added a background rendering of world land areas, as well as a country boundary layer on top. To avoid swamping the map, only routes with more than one flight listed are displayed.

The rendering is done using the following JEQL script:

CSVReader troute hasColNames: file: "routeLine2.csv";

trte = select  fromCity, toCity,
  Val.toDouble(fromLon) fromLon, Val.toDouble(fromLat) fromLat,
  Val.toDouble(toLon)   toLon,   Val.toDouble(toLat)   toLat
 from troute;
 
tlines = select fromCity, toCity, line, len
 with {
   line = Geodetic.split180(Geodetic.arc(fromLon, fromLat, toLon, toLat, 2));
   len = Geom.length(line);
 }
 from trte order by len desc;
 
tplot = select line,
 Color.interpolate("f0fff0", "00c077", "004020", len / 200.0 ) lineColor,
 0.4 lineWidth
 from tlines;
 
//----- Plot world landmasses for context
ShapefileReader tworld file: "world.shp";
tworldLine = select GEOMETRY, "22222277" lineColor from tworld;
tworldFill = select GEOMETRY, "333333"   fillColor from tworld;

width = 2000;
Plot width: width height: width / 2
 extent: LINESTRING(-180 -90, 180 90)
 
 data: tworldFill
 data: tplot
 data: tworldLine

 file: "routes.png";

That's about 40 lines of code, with only 9 statements. Not a bad LOC score... Of course the heavy lifting of handling geodetic data is done by Java functions, but that's part of the point. JEQL makes it easy to link to Java code, since that's a more appropriate technology for creating performant, resusable code.

Here's the final output:



The continent outlines aren't revealed as crisply as the Facebook friends map. That's because friend links tend to be more spatially coherent than airline routes (unless your acquaintances all have private jets). But it certainly shows where the world hotspots are for air traffic. Boy, those Europeans love to fly!

Next up: the KML version...