Sunday, December 26, 2010

Visualizing geodetic information with JEQL

The geo-blogosphere has been buzzing about the global Facebook friends visualization. This was done by Paul Butler using an R script and some clever techniques for working with geodetic data.



This kind of lightweight spatial analysis and visualization is squarely in the target zone for JEQL, so I thought I'd try something similar.

Of course I don't have access to the Facebook friends dataset, so I needed some other suitable dataset of global-scale links. An obvious candidate is airline routes. Luckily there is an excellent open data repository called OpenFlights. It has datasets which are tables of airport locations and air routes between airports:

airports.dat:

1,"Goroka","Goroka","Papua New Guinea","GKA","AYGA",-6.081689,145.391881,5282,10,"U"
2,"Madang","Madang","Papua New Guinea","MAG","AYMD",-5.207083,145.7887,20,10,"U"
3,"Mount Hagen","Mount Hagen","Papua New Guinea","HGU","AYMH",-5.826789,144.295861,5388,10,"U"
...

routes.dat:

0B,1542,AGP,1230,BBU,1650,,0,738
0B,1542,ARW,1647,BBU,1650,,0,340
0B,1542,BBU,1650,AGP,1230,,0,738
...

The first step is to prepare a suitable dataset for rendering. To get a table of FROM/TO locations, the routes table needs to be joined to the airport table. This is trivial to do in JEQL. After a bit of cleanup (such as removing missing data and duplicate routes), the final result is a dataset of links between airport locations using Lat/Long coordinates:

28,"Bagotville","Canada",48.330555,-70.996391,146,"Montreal","Canada",45.470556,-73.740833
29,"Baker Lake","Canada",64.298889,-96.077778,132,"Rankin Inlet","Canada",62.81139,-92.115833
30,"Campbell River","Canada",49.950832,-125.270833,119,"Comox","Canada",49.710833,-124.886667
30,"Campbell River","Canada",49.950832,-125.270833,156,"Vancouver","Canada",49.193889,-123.184444
...

As Paul found, a few other steps are needed to produce a visually appealing map:
  • Densify the route links to produce approximations to great-circle arcs
  • Break the arcs at the International Date Line to allow them to render correctly
  • Colour-theme the routes from longest to shortest using lighter colours for shorter routes
  • Render the lines with longer ones further back in the Z-order
For good measure I also added a background rendering of world land areas, as well as a country boundary layer on top. To avoid swamping the map, only routes with more than one flight listed are displayed.

The rendering is done using the following JEQL script:

CSVReader troute hasColNames: file: "routeLine2.csv";

trte = select fromCity, toCity,
Val.toDouble(fromLon) fromLon, Val.toDouble(fromLat) fromLat,
Val.toDouble(toLon) toLon, Val.toDouble(toLat) toLat
from troute;

tlines = select fromCity, toCity, line, len
with {
line = Geodetic.split180(Geodetic.arc(fromLon, fromLat, toLon, toLat, 2));
len = Geom.length(line);
}
from trte order by len desc;

tplot = select line,
Color.interpolate("f0fff0", "00c077", "004020", len / 200.0 ) lineColor,
0.4 lineWidth
from tlines;

//----- Plot world landmasses for context
ShapefileReader tworld file: "world.shp";
tworldLine = select GEOMETRY, "22222277" lineColor from tworld;
tworldFill = select GEOMETRY, "333333" fillColor from tworld;

width = 2000;
Plot width: width height: width / 2
extent: LINESTRING(-180 -90, 180 90)

data: tworldFill
data: tplot
data: tworldLine

file: "routes.png";


That's about 40 lines of code, with only 9 statements. Not a bad LOC score... Of course the heavy lifting of handling geodetic data is done by Java functions, but that's part of the point. JEQL makes it easy to link to Java code, since that's a more appropriate technology for creating performant, resusable code.

Here's the final output:



The continent outlines aren't revealed as crisply as the Facebook friends map. That's because friend links tend to be more spatially coherent than airline routes (unless your acquaintances all have private jets). But it certainly shows where the world hotspots are for air traffic. Boy, those Europeans love to fly!

Next up: the KML version...

19 comments:

Eugene Antimirov said...

Thanks! It's easier to do than I thought.

darkblueB said...

I, too, was curious about this graphic..
I have to say though you might have missed one subtlety.. In old fashioned raster graphics, there is an additive op, color blending mode, that sums up to a limit. So you dont have to "color theme" the lines, they naturally add in areas where they are dense, to a limit of white. At any rate, looking forward to a KML ! (for which yes, you would need the color ramp)

darkblueB said...

oh, another thing..
I imagined that you could write the lines in a projected srs, then warp the whole thing once, instead of having to do great circle math each time..

Dr JTS said...

@darkblueB:

Interesting idea about the additive blending mode. I'll look and see whether Java2D supports that.

As for using a projected CRS, no matter what the CRS, you still have to densify long lines for them to follow great circles correctly. And you still have to deal with wrapping long lines correctly in planar CRSes. So I don't think there's much advantage over using raw lat/long.

And also one of my goals is to provide good geodetic support, in JEQL and ultimately in JTS as well. So this was a good learning exercise.

studiob1 said...

Very elegant. The JEQL code is even simpler for KML output since the Tesselate element in LineString will give you densification.

Dr JTS said...

Good point about Tesselate providing densification.

I was thinking of computing a Z value for each densified vertex, to provide a nice-looking arc in the vertical dimension. This will require explicit densification.

Thanks for "elegant" - always #1 on my list!

Juan Lucas Domínguez Rubio said...

Very nice!

G.A. said...

Very cool Martin.
A question I wanted to ask you since a while: will JEQL ever have a public API to integrate it in code?

Dr JTS said...

@GA -

Yes, absolutely. This is in fact already there in embryo, but it needs to be improved and documented!

In fact there are two APIs - one to allow running scripts and interacting with the engine, and one to support extending JEQL with external functions and commands.

Stay tuned for some near-term developments here - I have some good ideas about how to proceed with exposing an API.

T M said...

Any idea how to apply this to trade data?

I'm trying to generate a map of imports/exports. This style of map is good for point to point, but what about if I want to add a weight/value to the connecting lines? It would effectively represent how much of a good is passing along that line.

Figured I would throw the question your way, you seem to have a significant handle on this...

Cheers,

Dr JTS said...

@T M:

I can think of a few ways to visualize weights on the lines - e.g. use a colour map, and/or line thickness. I guess you might also be able to use symbols, although that could get cluttered. There might be other clever visualization methods too.

Colour and line weight are easy to generate in JEQL, using appropriate values in each table record. If you send me some day I can whip up a prototype.

T M said...

I tried to find contact info on your profile, nothing there. Please email me (address is in my profile), I will send you an excel sheet of some initial data. Would really like to see how this could work.

gene said...
This comment has been removed by the author.
gene said...
This comment has been removed by the author.
gene said...

JEQL works very well in Mac OS X:
$ export JEQL_HOME=/Users/.../jeql-0.10/
$ export LIB=$JEQL_HOME/lib
$ CLASSPATH=$(JARS=("$LIB"/*.jar); IFS=:; echo "${JARS[*]}")
$ java -cp $CLASSPATH jeql.JeqlCmd -help
Jeql 0.10 ---- (c) 2007-2011 Martin Davis ---- Java 1.6.0_24
jeql [ options ] [ ...]
Options:
-help - prints this message
-man - prints a listing of functions and commands
-mon - display monitor during script execution
-stdin - reads script from standard input
-script - reads script from argument
-verbose - runs in verbose mode
You can create a bash script:
$ ./jeql file.jql

Dr JTS said...

Good to hear, Gene!

zedzero said...

Hey Dr JTS,

I found your site while doing research into creating just such a map.

I'm a motion graphics animator using Adobe After Effects, struggling to recreate something like this for a project.

Is there anyway to animate this and output it as something like a Quicktime movie?

I don't even know what application you'd use to create this image from the dataset!

best regards

Pete

Dr JTS said...

Animate it how?

Anyway, there's nothing in JEQL currently to create video, but that's an interesting idea for sure. Could probably be done just by generating a sequence of images for the frames, and then combining them using some other tool.

Romee Vergies said...

Great post. I was checking constantly this blog and I'm impressed! Extremely useful info.

Airline Flight Paths - Get airline route maps information