For the following queries, we will make use of trajectories for aggregation and creating effective splits in our data based on parameters that change in time.
This step is completed once, only on the ingestion of data. It is shown below to provide an understanding of how to do it. With temporal data types and MobilityDB functionality, we can see the queries are very intuitive to create.
We first create a geometry point. This treats each latitude and longitude as a point in space. 4326 is the SRID.
ALTER TABLE flights ADD COLUMN geom geometry(Point, 4326); UPDATE flights SET geom = ST_SetSRID( ST_MakePoint( lon, lat ), 4326);
Now we are ready to construct airframe or airplane trajectories out of their individual observations. Each icao24
in our dataset represents a single airplane.
We can create a composite index on icao24 (unique to each plane) and et_ts (timestamps of observations) to help improve the performance of trajectory generation.
CREATE INDEX icao24_time_index ON flights (icao24, et_ts);
We create trajectories for a single airframe because:
this query serves as a simple example of how to use MobilityDB to create trajectories
these kind of trajectories can be very important for plane manufacturer, as they are interested in the airplane's analysis.
we are creating the building blocks for future queries. Each row would represent a single flight, where flight is identified by icao24
and callsign
.
CREATE TABLE airframe_traj(icao24, trip, velocity, heading, vertrate, callsign, squawk, geoaltitude) AS SELECT icao24, tgeompointSeq(array_agg(tgeompoint(geom, et_ts) ORDER BY et_ts) FILTER (WHERE geom IS NOT NULL)), tfloatSeq(array_agg(tfloat(velocity, et_ts) ORDER BY et_ts) FILTER (WHERE velocity IS NOT NULL)), tfloatSeq(array_agg(tfloat(heading, et_ts) ORDER BY et_ts) FILTER (WHERE heading IS NOT NULL)), tfloatSeq(array_agg(tfloat(vertrate, et_ts) ORDER BY et_ts) FILTER (WHERE vertrate IS NOT NULL)), ttextSeq(array_agg(ttext(callsign, et_ts) ORDER BY et_ts) FILTER (WHERE callsign IS NOT NULL)), tintSeq(array_agg(tint(squawk, et_ts) ORDER BY et_ts) FILTER (WHERE squawk IS NOT NULL)), tfloatSeq(array_agg(tfloat(geoaltitude, et_ts) ORDER BY et_ts) FILTER (WHERE geoaltitude IS NOT NULL)) FROM flights GROUP BY icao24;
Here we create a new table for all the trajectories. We select all the attributes of interest that change over time. We can follow the transformation from the inner call to the outer call:
tgeompoint
: combines each geometry point(lat, long) with the timestamp where that point existed
array_agg
: aggregates all the instants together into a single array for each item in the GROUP BY
. In this case, it will create an array for each icao24
tgeompointSeq
: constructs the array as a sequence which can be manipulated with MobilityDB functionality. The same approach is used for each trajectory, with the function used changing depending on the datatype.
Right now we have, in a single row, an airframe's (where an airframe is a single physical airplane) entire day's trip information. We would like to segment that information per flight (an airframe flying under a specific callsign). This query segments the airframe trajectories (in temporal columns) based on the time period of the callsign. Below we explain the query and the reason behind segmenting the data this way.
-- Each row from airframe will create a new row in flight_traj depending on when the -- callsign changes, regardless of whether a plane repeats the same flight multiple -- times in any period -- Airplane123 (airframe_traj) |-------------------------| -- Flightpath1 (flight_traj) |-----| -- Flightpath2 (flight_traj) |--------| -- Flightpath1 (flight_traj) |-------| -- Flightpath3 (flight_traj) |--|
CREATE TABLE flight_traj(icao24, callsign, flight_period, trip, velocity, heading, vertrate, squawk, geoaltitude) AS -- callsign sequence unpacked into rows to split all other temporal sequences. WITH airframe_traj_with_unpacked_callsign AS ( SELECT icao24, trip, velocity, heading, vertrate, squawk, geoaltitude, startValue(unnest(segments(callsign))) AS start_value_callsign, unnest(segments(callsign))::tstzspan AS callsignSegmentPeriod FROM airframe_traj ) SELECT icao24 AS icao24, start_value_callsign AS callsign, callsignSegmentPeriod AS flight_period, atTime(trip, callsignSegmentPeriod) AS trip, atTime(velocity, callsignSegmentPeriod) AS velocity, atTime(heading, callsignSegmentPeriod) AS heading, atTime(vertrate, callsignSegmentPeriod) AS vertrate, atTime(squawk, callsignSegmentPeriod) AS squawk, atTime(geoaltitude, callsignSegmentPeriod) AS geoaltitude FROM airframe_traj_with_unpacked_callsign;
Note:
We could have tried to create the above (table flight_traj
) per flight trajectories by simply including callsign
in the GROUP BY
statement in the query used to create the previous airframe_traj
table (GROUP BY icao24, callsign;
).
The problem with this solution: This approach would put the trajectory data of two distinct flights where that airplane and flight number are the same in a single row, which is not correct.
MobilityDB functions helped us avoid the use of several hardcoded conditions that depend on user knowledge of the data. This approach is very generic and can be applied anytime we want to split a trajectory by the inflection points in time of some other trajectory.
We can now use our trajectories to pull flight specific statistics very easily.
In Format as, we have
Table
-- Average flight speeds during flight SELECT callsign,twavg(velocity) AS average_velocity FROM flight_traj WHERE twavg(velocity)IS NOT NULL -- drop rows without velocity data AND twavg(velocity) < 1500 -- removes erroneous data ORDER BY twavg(velocity) desc;
Change the visualization type to Bar gauge
.
The options (visualization settings - on the right side of the screen) should be as follows
Panel Options
Title → Average Flight Speed Show → All values
Bar gauge
Orientation → Horizontal
Standard Options
Unit → meters/second (m/s)
Min → 200
The settings we adjust improve the visualization by cutting the bar graph values of 0-200, improving the resolution at higher ranges to see differences.
We can easily combine results from multiple queries in the same visualization in Grafana, simplifying the queries themselves. Here we apply some domain knowledge of sport pilot aircraft license limits for altitude and speed to provide an estimated count of each.
In Format as, we have Table
-- Flights completed by private pilots (estimate) SELECT COUNT(callsign) AS private_flight FROM flight_traj WHERE (maxValue(velocity) IS NOT NULL -- remove flights without velocity AND maxValue(velocity) <= 65) -- sport aircraft max is 140mph (65m/s) AND (maxValue(geoaltitude) IS NOT NULL -- remove flights without altitude AND maxValue(geoaltitude) <= 5500); --18,000ft (5,500m) max for private pilot -- Count of commercial flights (estimate) SELECT COUNT(callsign) AS commercial_flight FROM flight_traj WHERE (maxValue(velocity) IS NOT NULL AND maxValue(velocity) > 65) AND (maxValue(geoaltitude) IS NOT NULL AND maxValue(geoaltitude) > 5500);
In Grafana, when we are in the query editor we can click on + Query
at the bottom to add multiple queries that provide different results.
Change the visualization type to Stat
.
To label the data for each result separately, choose Overrides
at the top of the options panel on the right. Here you can override global panel settings for specific attributes as shown below.
The final statistics visualization will look like this:
Note: This query makes used of a sample set of data that has 200 flights to return results. flight_traj_sample
is just a sampled version of flight_traj
. As of the writing of this workshop, Grafana does not support display of vectors, and so individual latitude and longitude points are used as a proxy.
In order to make the query use Grafana global time range panel replace the hard-coded timestamps with [${__from:date}, ${__to:date})
.
WITH timePeriod(Period) AS ( SELECT tstzspan '[2020-06-01 02:35:00, 2020-06-01 02:55:00)' ), flightTrajTimeSlice (icao24, callsign, time_slice_trip, timeSliceGeoaltitude, timeSliceVertrate) AS ( SELECT icao24, callsign, atTime(trip, Period), atTime(geoaltitude, Period), atTime(vertrate, Period) FROM flight_traj_sample TABLESAMPLE SYSTEM (20), timePeriod ), flightTrajTimeSliceAscent(icao24, callsign, ascendingTrip, ascendingGeoaltitude, ascendingVertrate) AS ( SELECT icao24, callsign, atTime(time_slice_trip, sequenceN(atValues(timeSliceVertrate, floatspan '[1,200]'), 1)::tstzspan), atTime(timeSliceGeoaltitude, sequenceN(atValues(timeSliceVertrate, floatspan '[1,20]'),1)::tstzspan), atTime(timeSliceVertrate, sequenceN(atValues(timeSliceVertrate, floatspan '[1,20]'),1)::tstzspan) FROM flightTrajTimeSlice ), finalOutput AS ( SELECT icao24, callsign, getValue(unnest(instants(ascendingGeoaltitude))) AS geoaltitude, getValue(unnest(instants(ascendingVertrate))) AS vertrate, ST_X(getValue(unnest(instants(ascendingTrip)))) AS lon, ST_Y(getValue(unnest(instants(ascendingTrip)))) AS lat FROM flightTrajTimeSliceAscent ) SELECT * FROM finalOutput WHERE vertrate IS NOT NULL AND geoaltitude IS NOT NULL;
We explain next the above query. Table timePeriod
specifies a user-specified time period. Table flightTrajTimeSlice
clips all the temporal columns to the given time period. Then, in the flightTrajTimeSliceAscent
, atSpan
clips the temporal data to create ranges where the vertrate
was between '[1, 20]'. This vertrate
means an aircraft was ascending. Then, sequenceN
selects the first sequence from the generated sequences. This first sequence is takeoff and eliminates mid-flight ascents. Finally atTime
returns the period of the first sequence. The final table finalOutput
uses unnest
to unpack the temporal data into rows for visualization in Grafana. Each row will contain a latitude
, longitude
, altitude
, and vertrate
at those locations.
Tips for QGIS visualization: QGIS uses geometry points for visualization, so for that in the third CTE you can use trajectory function on ascendingTrip and unnest the result.
We will modify make the follow adjustments for the visualization.
Change the visualization type to Geomap
.
The options (visualization settings - on the right side of the screen) should be as follows:
Panel Options
Title → Flight Ascent in Time Window
Data Layer:
Layer type: Markers
Location: Coords
Latitude field: lat
Longitude field: lon
Styles
Size: geoaltitude
Min: 1
Max: 5
Color: vertrate
Fill opacity: 0.5
Standard Options:
Unit: meters/second (m/s)
Color scheme: Green-Yellow-Red (by value)
We will also add a manual override (top right of panel options, beside "All") to limit the minimum value of vertrate. This will make all values below the minimum the same color, making larger values more obvious. This can be used to quickly pinpoint locations where a large rate of ascent existed.
Overrides
Add field override > Fields with name > vertrate
Min: 5
Max: 20
Here is a zoomed in version of how each individual flight ascent will look, as well as a view of multiple flights at the same time. The marker size is increasing with altitude, and the color is showing more aggressive vertical ascent rates. We can see towards the end of the visualized ascent period, there is a short increased vertical ascent rate.
The final visualization will look like the below.