PDAL integration: readers.tpcpatch and writers.tpcpatch

MobilityDB ships native PDAL plugins under contrib/pdal/ that read and write tpcpatch values directly without staging through readers.pgpointcloud + tpcpatchSeq(). Build them with:

cd contrib/pdal
cmake -B build && cmake --build build -j
sudo cmake --install build

PDAL discovers the plugins via PDAL_DRIVER_PATH or, after install, on its default search path. Verify with pdal --drivers | grep tpcpatch.

readers.tpcpatch

Reads any SQL query returning rows of (timestamp, pcpatch, pcid) and emits a PDAL PointView with all schema dimensions plus a time_t dimension carrying microseconds since epoch. Decode is delegated to pgPointCloud's pc_patch_from_wkb, so all three compression types (PC_NONE, PC_DIMENSIONAL, PC_LAZPERF) are handled.

To unpack a stored tpcpatch into per-instant rows the reader can consume:

{
  "type": "readers.tpcpatch",
  "connection": "host=/tmp dbname=mydb",
  "query": "SELECT (EXTRACT(EPOCH FROM timestampN(traj, n))*1000000)::bigint AS t,
                   valueN(traj, n)              AS pcp,
                   PC_PCId(valueN(traj, n))     AS pcid
            FROM trajectories,
                 generate_series(1, numInstants(traj)) n
            WHERE id = 42
            ORDER BY n"
}

writers.tpcpatch

Receives a PointView with a configurable time dimension, groups points by timestamp into per-instant pcpatch values, then dispatches an INSERT, UPDATE, append-merge, or upsert against a tpcpatch column using MobilityDB's tpcpatch(pcpatch, timestamptz) + tpcpatchSeq(pcpatch[]) constructors. Encode is delegated to pgPointCloud's pc_patch_to_wkb with optional pc_patch_compress. Schema scale / offset is applied symmetrically — values flow as their semantic units (metres, degrees, …) through PDAL even when the schema stores them as scaled int32.

Options:

  • connection (positional) — libpq connection string.

  • table (positional) — destination table name.

  • columntpcpatch column name (default traj).

  • pcid (required) — entry in pointcloud_formats whose schema the input dimensions must match.

  • id_column, id_value — identify the target row for any non-insert mode.

  • modeinsert (default — new row per pipeline run), update (overwrite the row identified by id_column = id_value), append (concatenate via merge() onto the existing row), or upsert (insert if absent, append if present). append / update raise a clear "matched no row" error if id_value is not found, so silent zero-row writes cannot happen.

  • time_dim — name of the source time dimension (default time_t; set to GpsTime to consume LAS timestamps directly without a filters.assign stage).

  • time_format — interpretation of time_dim: unix_microseconds (default), unix_seconds, or gps_adjusted (LAS-1.4 Adjusted GPS Standard Time — the writer applies the GPS-Unix epoch shift and 2024 leap-second offset).

  • compressionnone (default), dimensional, or laz. MobilityDB normalises tpcpatch on-disk storage to PC_NONE, so the option only affects libpq transport size — typical reduction is ~3x for dimensional on dim-redundant data.

  • flush_threshold (default 1024) — number of completed per-instant patches buffered in memory before issuing an INSERT/append. Honoured for mode=append/upsert; mode=insert/update defer to a single tail-flush since each pipeline run produces a single row.

The writer emits periodic Info-level progress lines via PDAL's logger and a final summary in done(): total points / patches written, destination column, pcid, compression, and mode. Visible at pdal pipeline -v 5.

Supported compression schemes

The reader and writer support all three pgPointCloud compression types: PC_NONE, PC_DIMENSIONAL, and PC_LAZPERF. The PC_LAZPERF regression at contrib/pdal/examples/test_lazperf.sql requires the running pgPointCloud install to be built with --with-lazperf=DIR; the SQL fixture aborts with a clear message otherwise. The plugins are kept under contrib/pdal/ rather than in the main MobilityDB build tree so they can be built and updated independently.

Limitations

  • WKB endianness. readers.tpcpatch accepts only little-endian (NDR) pcpatch WKB. A big-endian patch (first byte 0x00) is rejected with a clear error rather than silently mis-decoded. pgPointCloud emits NDR on every common platform; XDR support would be additive.

  • Compression of stored tpcpatch. The writers.tpcpatch compression option controls only the libpq transport form. MobilityDB's tpcpatch normalises its on-disk pcpatch storage to PC_NONE.

  • Streaming write. writers.tpcpatch implements PDAL's Streamable::processOne interface, so a readers.las → writers.tpcpatch pipeline runs in PDAL streaming mode automatically and never holds the full point cloud in memory. The completed-patch queue is bounded by the flush_threshold writer option (default 1024); on mode=append / mode=upsert each threshold-triggered flush issues an incremental SQL batch via merge(), while mode=insert / mode=update defer to a single tail-flush in done() (single-row semantics). This is the path that makes multi-GB LAS ingest tractable.