Navigation Statistics

The statistics system turns the per-image metadata JSON files written by the navigation pipeline into a local SQLite database and a deterministic report (Markdown text plus PNG charts). It is the standing quality check on a production run: success and failure rates, which techniques and models carry the load, offset distributions, how well the techniques agree with one another, and whether the confidence tiers behave as designed.

Two commands cooperate over the database and are cron-friendly (each is a single non-interactive invocation):

# Ingest one or more navigation-results roots into ./nav_stats.sqlite3
sd_stats_ingest /data/nav-offset-results --db nav_stats.sqlite3

# Generate report.md + charts for any slice of the database
sd_stats_report --db nav_stats.sqlite3 --output-dir stats_report
sd_stats_report --db nav_stats.sqlite3 --instrument coiss \
    --start-date 2005-03-01 --end-date 2005-03-01 --output-dir day_report

Ingestion

sd_stats_ingest ROOT [ROOT ...] [--db PATH] scans each root recursively for *_metadata.json files (the documents navigate_image_files() writes under nav_results_root) and loads them into the database. Roots may be local directories or any URL the project’s filecache layer accepts, so cloud-hosted results ingest the same way as local ones.

Ingestion is idempotent: the database holds one row per image (keyed by image name), and re-ingesting the same or an updated metadata file replaces that image’s row and its child rows rather than duplicating them. Malformed files are logged and skipped; the exit status is nonzero only when nothing at all was ingested.

Every ingestible document must carry observation.image_name and observation.instrument (the pipeline records both in every metadata document it writes). A document missing either field is logged, skipped, and counted as an error, exactly like a file that fails to parse.

The database is disposable, and there is no schema migration: opening a database whose images table does not match the current column set raises an error naming the mismatched columns. The remedy is always the same – delete the database file and re-run sd_stats_ingest (the source of truth is the metadata documents, so nothing is lost).

Database schema

The database is plain SQLite; opening it directly (sqlite3 command-line shell, Python’s sqlite3 module, pandas, a GUI browser) is a supported way to answer questions the standard report does not. Three tables hold the data; techniques and feature_sources reference images by image_name with ON DELETE CASCADE, and re-ingesting an image replaces its row and children.

images — one row per image, keyed by image_name:

Column	Type	Meaning
`image_name`	TEXT	Image filename (primary key), e.g. `N1454725799_1_CALIB.IMG`.
`instrument`	TEXT	`coiss` / `vgiss` / `gossi` / `nhlorri` / `sim`, from the metadata document’s `observation.instrument` field (required; a document without it is skipped as an ingest error).
`camera`	TEXT	The camera that took the image (`NAC` / `WAC` / `SSI` / `LORRI`), from the metadata document’s `observation.camera` field. Offset statistics group by this: pointing error belongs to the camera, not the spacecraft. NULL when the document carries no camera (results navigated before the field existed); it is never inferred from the image name.
`image_path`	TEXT	Absolute path of the source image at navigate time.
`image_et`	REAL	Observation midtime, TDB seconds past J2000. Taken from the navigation provenance, or — for an image that never loaded, which has no provenance — from the `observation.image_et` the navigator read out of the PDS3 index. An image whose navigation died for want of a SPICE kernel is therefore still placed in time.
`image_date`	TEXT	UTC calendar date `YYYY-MM-DD` derived from `image_et`; drives the `--start-date` / `--end-date` report filters.
`status`	TEXT	Navigation outcome: `success`, `failed`, `conflicted`, or `error` (the last from image-load failures); `unknown` when the metadata document carries no status at all.
`status_reason`	TEXT	Outcome reason: successes hold `ok` or `rank_1_only`; failures hold the failure reason (e.g. `no_features_extracted`, `missing_spice_data`). NULL only when the metadata carries neither `status_reason` nor `status_error`.
`offset_dv`, `offset_du`	REAL	Fused pointing offset in pixels (V then U); NULL when navigation found no offset.
`sigma_dv`, `sigma_du`	REAL	Per-axis 1-sigma uncertainty of the fused offset, pixels.
`confidence`	REAL	Fused confidence in `[0, 1]`.
`confidence_rank`	TEXT	Confidence tier label assigned by the ensemble.
`n_techniques`	INTEGER	Number of per-technique results recorded for the image.
`excluded_from_consensus`	TEXT	JSON list of technique names the ensemble excluded as outliers (`[]` when none).
`image_class`	TEXT	Image-classifier verdict (e.g. `clean`).
`noise_sigma`	REAL	Image-classifier noise estimate.
`image_shape_v`, `image_shape_u`	INTEGER	Pixel dimensions of the image data (V then U), from the metadata document’s `observation.image_shape` field; NULL when the image never loaded (e.g. a read error).
`run_start`, `run_end`	TEXT	UTC ISO8601 run start and end of the navigation of this image, from the metadata document’s `timing` section; start is captured before the image load and end after navigation (or at error time).
`elapsed_s`	REAL	Wall-clock seconds between `run_start` and `run_end`.
`config_hash`	TEXT	sha256 of the fully-resolved configuration used for the run.
`git_sha`	TEXT	Short git SHA of the navigating code (`-dirty` suffix when the tree had uncommitted changes).
`pipeline_run`	TEXT	UTC ISO8601 timestamp of the navigation run.
`source_file`	TEXT	Path or URL of the ingested metadata document.

techniques — one row per technique result per image:

Column	Type	Meaning
`image_name`	TEXT	Foreign key into `images`.
`technique_name`	TEXT	Technique class name (e.g. `BodyLimbNav`, `StarUniqueMatchNav`).
`offset_dv`, `offset_du`	REAL	The technique’s own offset estimate, pixels.
`sigma_dv`, `sigma_du`	REAL	Per-axis 1-sigma from the technique’s covariance.
`confidence`	REAL	The technique’s calibrated confidence in `[0, 1]`.
`spurious`	INTEGER	1 when the technique flagged its own result as spurious.
`at_edge`	INTEGER	1 when the fit landed at the edge of its search space.
`source_names`	TEXT	JSON list of body / ring / catalog names the technique used.
`diagnostics`	TEXT	The technique’s full diagnostics dataclass as a JSON object.

feature_sources — per image, feature counts grouped by source:

Column	Type	Meaning
`image_name`	TEXT	Foreign key into `images`.
`feature_type`	TEXT	Feature type (e.g. `BODY_DISC`, `STAR`, `RING_EDGE`).
`source_model`	TEXT	NavModel family that produced the features (`body`, `rings`, `stars`).
`source_name`	TEXT	Body, ring, or catalog name (e.g. `IAPETUS`, `UCAC4`).
`n_features`	INTEGER	Features of this type/source extracted for the image.
`n_gated`	INTEGER	How many of them the reliability gate removed.

Indexes exist on images(image_date), images(instrument), and the image_name foreign keys.

Querying the database directly

Success rate per instrument, from the sqlite3 shell:

SELECT instrument,
       COUNT(*) AS images,
       AVG(status = 'success') AS success_rate
FROM images
GROUP BY instrument;

The ten largest fused offsets, with their confidence tier:

SELECT image_name, offset_dv, offset_du, confidence_rank
FROM images
WHERE status = 'success'
ORDER BY MAX(ABS(offset_dv), ABS(offset_du)) DESC
LIMIT 10;

JSON columns unpack with SQLite’s built-in JSON functions – for example, counting how often each technique was excluded from the consensus:

SELECT excluded.value AS technique, COUNT(*) AS images
FROM images, json_each(images.excluded_from_consensus) AS excluded
GROUP BY excluded.value
ORDER BY COUNT(*) DESC;

The same database loads straight into pandas:

import pandas as pd
import sqlite3

conn = sqlite3.connect('nav_stats.sqlite3')
images = pd.read_sql_query('SELECT * FROM images', conn)
per_technique = pd.read_sql_query(
    'SELECT t.*, i.instrument, i.image_date '
    'FROM techniques t JOIN images i USING (image_name)',
    conn,
)
# Median per-technique confidence by instrument:
print(per_technique.groupby(['instrument', 'technique_name'])['confidence'].median())

Reporting

sd_stats_report [--db PATH] [--output-dir DIR] [--instrument NAME] [--start-date YYYY-MM-DD] [--end-date YYYY-MM-DD] [--min-image NAME] [--max-image NAME] [--top-n N] [--filelists] [--suspect-fraction F] [--csv] writes report.md and its charts into the output directory. All filters combine and apply to every section; dates are inclusive UTC image dates, so a single day’s run is --start-date D --end-date D. --min-image / --max-image bound the numeric portion of the image name (the first digit run in the basename, so --min-image N1454725799 and --min-image 1454725799 are equivalent); both bounds are inclusive and either may be given alone. The same inputs always produce the same numbers and the same charts.

Three options control drill-down output:

--top-n N makes each categorical section (failure reasons, failure taxonomy, ensemble exclusions, suspect offsets) list up to N example image names per category and instrument, caps the suspect-offset and worst-BOTSIM-pair tables at N rows, and lists the N slowest images.
--filelists writes one plain-text file per category and instrument (one image name per line, the full list rather than the top N) into the filelists/ subdirectory of the output directory, ready to feed back into re-runs and triage scripts.
--csv writes images.csv next to report.md: one row per image with every images column plus per-image technique and feature-source counts, for pandas or spreadsheet analysis.

The first two write image names rather than file names — N1454725799 rather than N1454725799_1_CALIB.IMG — because that is the token the datasets’ --image-filelist option selects on. The filelists are directly consumable by it: one name per line, with a leading # comment naming the category.

Every image count in the report carries its percentage — 5 (3.2%). Counts are broken down by instrument: a table of counts gets one column per instrument plus a total column, where an instrument column’s percentage is of that instrument’s images and the total column’s is of all selected images, so each column sums to 100% on its own. Tables of statistics rather than counts (offsets, run time, per-body shares, cross-technique agreement) carry an instrument column instead, a total being meaningless for a mean or a standard deviation. Bar charts are stacked, one segment per instrument, with a fixed color per instrument across every chart.

The report contains:

Images selected — per instrument: how many images, the first and last image, and the first and last available date. Image numbers only compare within one instrument, so the bounds are never pooled across instruments. The date bounds are found independently of the image ordering, so a single image with no recorded epoch at either end of the number range cannot hide the instrument’s real time span.
Success / failure counts with a breakdown of failure reasons. The reason table carries each reason’s status, so errors (SPICE-related or not) are visible alongside outright navigation failures.
Failure taxonomy by image content — failed images classified from their recorded feature inventory (stars-only, single-body, multi-body, rings-only, body+rings, no-features), with a per-category failure-reason breakdown and a per-body table of how often each named body appears in failed versus successful images (a body with a high failure share points at a modeling problem for that body).
Technique usage — the images each technique ran on, plus a per- technique, per-instrument detail table of non-spurious runs and mean confidence.
Model and source usage — which bodies, rings, and star catalogs appeared, in how many images, and how many of their features survived the reliability gate.
Offset statistics — mean, median, standard deviation, minimum, and maximum of the fused V and U offsets over successful images, grouped by camera, with one histogram per camera, plus the same statistics grouped by (instrument, camera, image size). Distributions are never pooled across cameras: one Cassini WAC pixel is ten NAC pixels, so a pooled distribution would describe neither camera.
Suspect offsets — successful images whose fused offset reaches at least --suspect-fraction (default 0.9) of the instrument’s per-axis maximum expected pointing offset (the configured extfov_margin_vu search margin; for Cassini ISS the NAC/WAC margin chosen from the image name) on either axis. An offset pinned near the search boundary may be a correlation artifact, so these images deserve operator review. When a limit cannot be resolved for an image (unknown instrument, no recorded image shape), the report says so rather than silently skipping it.
BOTSIM pair consistency (Cassini ISS) — BOTSIM observations shutter the NAC and WAC simultaneously and the image names share one spacecraft-clock count. One WAC pixel is ten NAC pixels, so a consistent pair satisfies NAC offset = 10 x WAC offset per axis; the section reports the count, median, and 95th percentile of the NAC - 10 x WAC residuals over pairs where both frames navigated successfully, and (with --top-n) the worst pairs. This is an end-to-end accuracy check that needs no ground truth.
Cross-technique agreement — for every technique pair, the median and 95th-percentile Euclidean distance between their offsets on images where both produced non-spurious results.
Confidence calibration — per confidence tier, the distribution of each image’s maximum cross-technique disagreement. The tiers always read high / medium / low / failed / conflicted, so a tier with no images reads as an explicit zero rather than a missing row. Without ground truth, agreement between independent techniques is the production proxy for accuracy (the calibrated anchor is the simulation campaign; see Confidence Calibration (Shared Sigmoid-of-Linear Combination)): a healthy pipeline shows high-tier images agreeing tightly and disagreement growing toward the low tier.
Ensemble outlier exclusions — how often the ensemble excluded a technique from the consensus, and which techniques.
Run-time statistics — per instrument (and pooled, when more than one instrument is selected): minimum, maximum, mean, median, standard deviation, and total of the per-image wall-clock run times, a run-time histogram, and (with --top-n) the slowest images. The section is omitted when no ingested document carries timing data.