spindoctor.dataset
- class DataSet(*, config: Config | None = None)[source]
-
- abstractmethod static add_selection_arguments(cmdparser: ArgumentParser, group: _ArgumentGroup | None = None) None[source]
Adds dataset-specific command-line arguments for image selection.
- Parameters:
cmdparser – The argument parser to add arguments to.
group – Optional argument group to add arguments to. If None, creates a new group.
- pds4_bundle_name() str[source]
Returns bundle name for PDS4 bundle generation.
Checks config section pds4.{dataset_name}.bundle_name first, then allows override.
- Returns:
Bundle name (e.g., “cassini_iss_saturn_backplanes_rsfrench2027”).
- static pds4_bundle_path_for_image(image_name: str) str[source]
Maps image name to bundle directory path.
- Parameters:
image_name – The image name to map.
- Returns:
Bundle directory path relative to bundle root (e.g., “1234xxxxxx/123456xxxx”).
- pds4_bundle_template_dir() str[source]
Returns absolute path to template directory for PDS4 bundle generation.
Checks config section pds4.{dataset_name}.template_dir first, then allows override. If just a name is given, it is relative to the pds4/templates directory. If a full path is given, it will be left as absolute.
- Returns:
Absolute path to template directory (e.g., “/path/to/pds4/templates/cassini_iss_saturn_1.0”).
- pds4_image_name_to_browse_lid(image_name: str) str[source]
Returns the browse LID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LID.
- Returns:
The browse LID.
- pds4_image_name_to_browse_lidvid(image_name: str) str[source]
Returns the browse LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LID.
- Returns:
The browse LID.
- pds4_image_name_to_data_lid(image_name: str) str[source]
Returns the data LID for the given image name.
- Parameters:
image_name – The image name to convert to a data LID.
- Returns:
The data LID.
- pds4_image_name_to_data_lidvid(image_name: str) str[source]
Returns the data LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a data LID.
- Returns:
The data LID.
- pds4_lid_part_to_image_name(lid_part: str) str[source]
Returns the image name for the given LID part.
Inverse of the image-name transformation baked into
pds4_path_stub()and thepds4_image_name_to_*builders. The bundle stores each product under a filename whose stem is the LID part (e.g.1234567890w); recovering the original image name lets bundle scanners round-trip that stem back through the canonical LID builders instead of re-applying the transform.- Parameters:
lid_part – The LID part (an on-disk product filename stem).
- Returns:
The image name that produced the given LID part.
- pds4_path_stub(image_file: ImageFile) str[source]
Returns PDS4 path stub for bundle directory structure.
- Parameters:
image_file – The image file to generate path stub for.
- Returns:
Path stub relative to bundle root (e.g., “1234xxxxxx/123456xxxx/1234567890w”).
- pds4_template_variables(*, image_file: ImageFile, nav_metadata: dict[str, Any], backplane_metadata: dict[str, Any]) dict[str, Any][source]
Returns template variables for PDS4 label generation.
- Parameters:
image_file – The image file being processed.
nav_metadata – Navigation metadata dictionary (as read from offset_metadata JSON file).
backplane_metadata – Backplane metadata dictionary (created from backplane FITS file).
- Returns:
Dictionary mapping variable names to values for template substitution.
- abstractmethod static supported_grouping() list[str][source]
Returns the list of supported grouping types.
- Returns:
The list of supported grouping types.
- abstractmethod yield_image_files_from_arguments(arguments: Namespace) Iterator[ImageFiles][source]
Yields image filenames based on provided command-line arguments.
- Parameters:
arguments – The parsed arguments structure.
- Yields:
Information about the selected files in groups as ImageFiles objects.
- abstractmethod yield_image_files_index(**kwargs: Any) Iterator[ImageFiles][source]
Yields image filenames based on index information.
- Parameters:
**kwargs – Arbitrary keyword arguments, usually used to restrict the search.
- Yields:
Information about the selected files in groups as ImageFiles objects.
- class ImageFile(image_file_url: ~filecache.file_cache_path.FCPath, label_file_url: ~filecache.file_cache_path.FCPath, results_path_stub: str, index_file_row: dict[str, ~typing.Any] = <factory>, image_et: float | None = None, camera: str | None = None, extra_params: dict[str, ~typing.Any] = <factory>, image_url_resolver: ~collections.abc.Callable[[~filecache.file_cache_path.FCPath, ~pathlib.Path], ~filecache.file_cache_path.FCPath | None] | None = None, _image_file_path: ~pathlib.Path | None = None, _label_file_path: ~pathlib.Path | None = None)[source]
Bases:
objectRepresents a single image file with its metadata and lazy-loaded paths.
- image_file_url
Remote URL for the image file.
- label_file_url
Remote URL for the label file.
- results_path_stub
Local path stub for storing results.
- Type:
- image_et
Optional observation epoch (TDB seconds past J2000) read from the index row when the file was enumerated. Known without SPICE and without opening the image, so it is still available for an image whose load fails. None when the image was not enumerated from an index, or its index row carries no readable time.
- Type:
float | None
- camera
Optional name of the camera that took the image, read from the index row when the file was enumerated. Available on the same terms as
image_et, and uses the same names asObsInst.camera. None when the image was not enumerated from an index, or its index row names no recognized camera.- Type:
str | None
- extra_params
Optional extra parameters that will be passed to the observation class’s from_file method when the file is read.
- image_url_resolver
Optional callable
(image_file_url, label_file_path) -> FCPath | Nonethat determines the definitive image URL from the label contents. When set, it is invoked (at most once) before the image file is first retrieved; a non-None return value replacesimage_file_url.image_file_urlis then a provisional guess untilresolve_image_url()has run.- Type:
collections.abc.Callable[[filecache.file_cache_path.FCPath, pathlib.Path], filecache.file_cache_path.FCPath | None] | None
- property image_file_name: str
- property image_file_path: Path
Local path to the image file, downloading and memoizing on first use.
Not thread-safe: a single
ImageFilemust be used by one thread at a time. The lazy memoization is unsynchronized, so concurrent first access from two threads can race and download twice. Enumerate-then- dispatch oneImageFileper worker thread.
- image_file_url: FCPath
- property label_file_name: str
- property label_file_path: Path
Local path to the label file, downloading and memoizing on first use.
Not thread-safe; see
image_file_pathfor the single-thread-per-ImageFileexpectation.
- label_file_url: FCPath
- resolve_image_url() FCPath[source]
The definitive image URL, consulting the label when a resolver is set.
When
image_url_resolveris set, the label file is retrieved and the resolver maps the label contents to the correct image URL, which replacesimage_file_url. The resolver runs at most once; subsequent calls return the memoized URL.A resolver failure (typically an unretrievable label) falls back to the current
image_file_urlguess with a logged warning: resolution runs before the pipeline’s per-image error boundary is open, and the guess is usable whenever the image file itself is retrievable.- Returns:
The image file URL, corrected from the label contents when a resolver is set and reports a different filename.
- results_path_stub: str
- class ImageFiles(image_files: list[ImageFile])[source]
Bases:
objectA collection of ImageFile objects that behaves like a sequence.
Supports iteration, indexing, and length operations on the wrapped image files.
- image_files: list[ImageFile]
- class DataSetPDS3(pds3_holdings_root: str | Path | FCPath | None = None, *, index_filecache: FileCache | None = None, pds3_holdings_filecache: FileCache | None = None, config: Config | None = None)[source]
Bases:
DataSetParent class for PDS3 datasets.
This class provides functionality common to all PDS3 datasets.
- static add_selection_arguments(cmdparser: ArgumentParser, group: _ArgumentGroup | None = None) None[source]
Adds PDS3-specific command-line arguments for image selection.
- Parameters:
cmdparser – The argument parser to add arguments to.
group – Optional argument group to add arguments to. If None, creates a new group.
- classmethod camera_from_index_row(index_row: dict[str, Any]) str | None[source]
The camera that took the image, read from its PDS3 index row.
Like
image_et_from_index_row(), this needs neither SPICE nor the image itself, so an image whose navigation died for want of a SPICE kernel still names its camera. The result uses the same names asObsInst.camera, so an observation’s camera and this one are interchangeable.- Parameters:
index_row – An
ImageFile.index_file_row; an empty dict (an image not enumerated from an index) yields None.- Returns:
The camera name, or None when the row carries no recognized camera value. An unrecognized value yields None rather than itself: a name the rest of the system does not know would silently split the statistics it feeds.
- classmethod image_et_from_index_row(index_row: dict[str, Any]) float | None[source]
The image’s observation epoch, read from its PDS3 index row.
The index carries every image’s time, so this is the one source of an epoch that needs neither SPICE nor the image itself – which is what lets an image whose navigation died for want of a SPICE kernel still be placed in time.
julianparses every format the index tables use directly, including day-of-year (1999-009T08:13:58.687) and a trailingZ.- Parameters:
index_row – An
ImageFile.index_file_row; an empty dict (an image not enumerated from an index) yields None.- Returns:
TDB seconds past J2000, or None when the row carries no readable time (no time column, a masked/null value, or an unparsable one).
- static supported_grouping() list[str][source]
Returns the list of supported grouping types.
- Returns:
The list of supported grouping types.
- yield_image_files_from_arguments(arguments: Namespace) Iterator[ImageFiles][source]
Given parsed arguments, yield all selected filenames.
- Parameters:
arguments – The parsed arguments structure.
- Yields:
ImageFiles objects containing information about groups of selected image files.
- yield_image_files_index(**kwargs: Any) Iterator[ImageFiles][source]
Yield filenames given search criteria using index files. Overridden by subclasses.
- Parameters:
**kwargs – Arbitrary keyword arguments, usually used to restrict the search.
- Yields:
ImageFiles objects containing information about groups of selected image files.
Filter image selections against existing navigation result files.
Implements the --has-offset-file / --has-no-offset-file /
--has-png-file / --has-no-png-file / --has-offset-error /
--has-offset-spice-error / --has-offset-nonspice-error image
selection options shared by the PDS3 datasets.
The navigation pipeline writes {nav_results_root}/{results_path_stub}_metadata.json
and {results_path_stub}_summary.png (see
spindoctor.navigate_image_files.navigate_image_files()). Presence
filters are answered by walking the results tree once per selected volume and
collecting the existing result files into sets, so each candidate image costs
no additional cloud round trip. Absence filters are answered with batched
FCPath.exists() calls (or from the walked sets when a walk already
happened), and the error filters retrieve the metadata JSON files in batches
and inspect their status / status_error fields.
- METADATA_SUFFIX = '_metadata.json'
Suffix of the per-image offset metadata file under the results root.
- RESULTS_FILTER_BATCH_SIZE = 64
Number of images checked per batched
exists()/retrieve()call.
- class ResultsFilter(volumes: Iterable[str], nav_results_root: str | Path | FCPath, *, has_offset_file: bool = False, has_no_offset_file: bool = False, has_png_file: bool = False, has_no_png_file: bool = False, has_offset_error: bool = False, has_offset_spice_error: bool = False, has_offset_nonspice_error: bool = False, logger: PdsLogger)[source]
Bases:
objectFilters candidate images against their navigation result files.
Constructed once per enumeration when any of the results-based selection flags is active. Construction validates the flag combination (the flags AND together; directly contradictory pairs raise) and, when a presence or error filter is active, walks the results tree under each selected volume to collect the existing result files.
The filter is applied in two stages:
passes_presence()is a cheap per-row set-membership test used while scanning index rows.filter_batch()applies the absence and metadata-content filters to a batch of already-accepted images with one batchedexists()and/orretrieve()call, preserving input order.
- filter_batch(image_files: list[ImageFile]) list[ImageFile][source]
Applies the absence and metadata-content filters to a batch.
Input order is preserved. Absence filters (when the results tree was not walked) are answered with one batched
exists()call covering every active absence suffix. The error filters retrieve all metadata files in one batched call and inspect theirstatus/status_errorfields.- Parameters:
image_files – Batch of images that already passed the cheap filters.
- Returns:
The images that also pass the absence and error filters, in input order.
- property needs_batch_filtering: bool
True when
filter_batch()performs any work.The caller uses this to decide whether to buffer accepted images into batches (amortizing the batched
exists()/retrieve()round trips) or to yield them immediately. When the results tree was walked, the absence filters are answered from the walked sets inpasses_presence()instead and cost nothing here.
- passes_presence(results_path_stub: str) bool[source]
True if the image passes the filters answerable from the walked sets.
Covers the presence filters and, when the results tree was walked anyway, the absence filters too (a set lookup instead of a per-file
exists()round trip).- Parameters:
results_path_stub – The image’s results path stub (relative to the results root, no suffix).
- Returns:
True if every active filter answerable from the walked sets is satisfied.
- SUMMARY_PNG_SUFFIX = '_summary.png'
Suffix of the per-image summary PNG file under the results root.
- class DataSetPDS4(*, config: Config | None = None)[source]
Bases:
DataSet,ABCParent class for PDS4 datasets.
This class provides functionality common to all PDS4 datasets.
- static add_selection_arguments(cmdparser: ArgumentParser, group: _ArgumentGroup | None = None) None[source]
Adds dataset-specific command-line arguments for image selection.
- Parameters:
cmdparser – The argument parser to add arguments to.
group – Optional argument group to add arguments to. If None, creates a new group.
- static supported_grouping() list[str][source]
Returns the list of supported grouping types.
- Returns:
The list of supported grouping types.
- class DataSetPDS3CassiniISS(pds3_holdings_root: str | Path | FCPath | None = None, *, index_filecache: FileCache | None = None, pds3_holdings_filecache: FileCache | None = None, config: Config | None = None)[source]
Bases:
DataSetPDS3Implements dataset access for PDS3 Cassini ISS (Imaging Science Subsystem) data.
This class provides specialized functionality for accessing and parsing Cassini ISS image data stored in PDS3 format. Includes all volumes (1001-1009 and 2001-2116).
- static add_selection_arguments(cmdparser: ArgumentParser, group: _ArgumentGroup | None = None) None[source]
Adds Cassini ISS-specific command-line arguments for image selection.
- Parameters:
cmdparser – The argument parser to add arguments to.
group – Optional argument group to add arguments to. If None, creates a new group.
- pds4_bundle_name() str[source]
Returns bundle name for PDS4 bundle generation.
- Returns:
Bundle name.
- static pds4_bundle_path_for_image(image_name: str) str[source]
Maps image name to bundle directory path.
- Parameters:
image_name – The image name to map (e.g., “N1234567890”).
- Returns:
Bundle directory path relative to bundle root (e.g., “1234xxxxxx/123456xxxx”).
- pds4_bundle_template_dir() str[source]
Returns absolute path to template directory for PDS4 bundle generation.
- Returns:
Absolute path to template directory.
- pds4_image_name_to_browse_lid(image_name: str) str[source]
Returns the browse LID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LID.
- Returns:
The browse LID.
- pds4_image_name_to_browse_lidvid(image_name: str) str[source]
Returns the browse LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LID.
- Returns:
The browse LIDVID.
- pds4_image_name_to_data_lid(image_name: str) str[source]
Returns the data LID for the given image name.
- Parameters:
image_name – The image name to convert to a data LID.
- Returns:
The data LID.
- pds4_image_name_to_data_lidvid(image_name: str) str[source]
Returns the data LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a data LID.
- Returns:
The data LIDVID.
- pds4_lid_part_to_image_name(lid_part: str) str[source]
Returns the image name for the given LID part.
Inverts the transform applied by
pds4_path_stub()and thepds4_image_name_to_*builders, which move the leading camera letter (N/W) to a lowercase suffix (N1454725799->1454725799n). The inverse moves the trailing lowercase letter back to the front as an uppercase prefix (1454725799n->N1454725799).- Parameters:
lid_part – The LID part (an on-disk product filename stem).
- Returns:
The image name that produced the given LID part.
- pds4_path_stub(image_file: ImageFile) str[source]
Returns PDS4 path stub for bundle directory structure.
- Parameters:
image_file – The image file to generate path stub for.
- Returns:
Path stub relative to bundle root (e.g., “1234xxxxxx/123456xxxx/1234567890w”).
- pds4_template_variables(*, image_file: ImageFile, nav_metadata: dict[str, Any], backplane_metadata: dict[str, Any]) dict[str, Any][source]
Returns template variables for PDS4 label generation.
- Parameters:
image_file – The image file being processed.
nav_metadata – Navigation metadata dictionary.
backplane_metadata – Backplane metadata dictionary.
- Returns:
Dictionary mapping variable names to values for template substitution.
- static supported_grouping() list[str][source]
Returns the list of supported grouping types.
- Returns:
The list of supported grouping types. For Cassini ISS, only ‘botsim’ is supported.
- yield_image_files_index(**kwargs: Any) Iterator[ImageFiles][source]
Yield filenames given search criteria using index files.
- Parameters:
group – How to group the results. Only ‘botsim’ is supported.
**kwargs – Arbitrary keyword arguments, usually used to restrict the search.
- Yields:
ImageFiles objects containing information about groups of selected image files.
- class DataSetPDS3CassiniISSCruise(pds3_holdings_root: str | Path | FCPath | None = None, *, index_filecache: FileCache | None = None, pds3_holdings_filecache: FileCache | None = None, config: Config | None = None)[source]
Bases:
DataSetPDS3CassiniISSImplements dataset access for PDS3 Cassini ISS Cruise data (volumes 1001-1009).
- class DataSetPDS3CassiniISSSaturn(pds3_holdings_root: str | Path | FCPath | None = None, *, index_filecache: FileCache | None = None, pds3_holdings_filecache: FileCache | None = None, config: Config | None = None)[source]
Bases:
DataSetPDS3CassiniISSImplements dataset access for PDS3 Cassini ISS Saturn data (volumes 2001-2116).
- class DataSetPDS3VoyagerISS(pds3_holdings_root: str | Path | FCPath | None = None, *, index_filecache: FileCache | None = None, pds3_holdings_filecache: FileCache | None = None, config: Config | None = None)[source]
Bases:
DataSetPDS3Implements dataset access for PDS3 Voyager ISS (Imaging Science Subsystem) data.
This class provides specialized functionality for accessing and parsing Voyager ISS image data stored in PDS3 format.
- static pds4_bundle_path_for_image(image_name: str) str[source]
Maps image name to bundle directory path.
- pds4_bundle_template_dir() str[source]
Returns absolute path to template directory for PDS4 bundle generation.
- pds4_image_name_to_browse_lid(image_name: str) str[source]
Returns the browse LID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LID.
- Returns:
The browse LID.
- pds4_image_name_to_browse_lidvid(image_name: str) str[source]
Returns the browse LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LIDVID.
- Returns:
The browse LIDVID.
- pds4_image_name_to_data_lid(image_name: str) str[source]
Returns the data LID for the given image name.
- Parameters:
image_name – The image name to convert to a data LID.
- Returns:
The data LID.
- pds4_image_name_to_data_lidvid(image_name: str) str[source]
Returns the data LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a data LIDVID.
- Returns:
The data LIDVID.
- pds4_lid_part_to_image_name(lid_part: str) str[source]
Returns the image name for the given LID part.
- Parameters:
lid_part – The LID part (an on-disk product filename stem).
- Returns:
The image name that produced the given LID part.
- class DataSetPDS3GalileoSSI(pds3_holdings_root: str | Path | FCPath | None = None, *, index_filecache: FileCache | None = None, pds3_holdings_filecache: FileCache | None = None, config: Config | None = None)[source]
Bases:
DataSetPDS3Implements dataset access for PDS3 Galileo SSI (Solid State Imager) data.
This class provides specialized functionality for accessing and parsing Galileo SSI image data stored in PDS3 format.
- static pds4_bundle_path_for_image(image_name: str) str[source]
Maps image name to bundle directory path.
- pds4_bundle_template_dir() str[source]
Returns absolute path to template directory for PDS4 bundle generation.
- pds4_image_name_to_browse_lid(image_name: str) str[source]
Returns the browse LID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LID.
- Returns:
The browse LID.
- pds4_image_name_to_browse_lidvid(image_name: str) str[source]
Returns the browse LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LIDVID.
- Returns:
The browse LIDVID.
- pds4_image_name_to_data_lid(image_name: str) str[source]
Returns the data LID for the given image name.
- Parameters:
image_name – The image name to convert to a data LID.
- Returns:
The data LID.
- pds4_image_name_to_data_lidvid(image_name: str) str[source]
Returns the data LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a data LIDVID.
- Returns:
The data LIDVID.
- pds4_lid_part_to_image_name(lid_part: str) str[source]
Returns the image name for the given LID part.
- Parameters:
lid_part – The LID part (an on-disk product filename stem).
- Returns:
The image name that produced the given LID part.
- class DataSetPDS3NewHorizonsLORRI(pds3_holdings_root: str | Path | FCPath | None = None, *, index_filecache: FileCache | None = None, pds3_holdings_filecache: FileCache | None = None, config: Config | None = None)[source]
Bases:
DataSetPDS3Implements dataset access for PDS3 New Horizons LORRI (Low-Resolution Imaging Experiment) data.
This class provides specialized functionality for accessing and parsing New Horizons LORRI image data stored in PDS3 format.
- static pds4_bundle_path_for_image(image_name: str) str[source]
Maps image name to bundle directory path.
- pds4_bundle_template_dir() str[source]
Returns absolute path to template directory for PDS4 bundle generation.
- pds4_image_name_to_browse_lid(image_name: str) str[source]
Returns the browse LID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LID.
- Returns:
The browse LID.
- pds4_image_name_to_browse_lidvid(image_name: str) str[source]
Returns the browse LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a browse LIDVID.
- Returns:
The browse LIDVID.
- pds4_image_name_to_data_lid(image_name: str) str[source]
Returns the data LID for the given image name.
- Parameters:
image_name – The image name to convert to a data LID.
- Returns:
The data LID.
- pds4_image_name_to_data_lidvid(image_name: str) str[source]
Returns the data LIDVID for the given image name.
- Parameters:
image_name – The image name to convert to a data LIDVID.
- Returns:
The data LIDVID.
- pds4_lid_part_to_image_name(lid_part: str) str[source]
Returns the image name for the given LID part.
- Parameters:
lid_part – The LID part (an on-disk product filename stem).
- Returns:
The image name that produced the given LID part.
- class DataSetSim(*, config: Config | None = None)[source]
Bases:
DataSetDataset that yields a series of YAML-described simulated images.
Each dataset value is a direct file path to the YAML scene file; selection simply returns that one file.
- static add_selection_arguments(cmdparser: ArgumentParser, group: _ArgumentGroup | None = None) None[source]
Adds dataset-specific command-line arguments for image selection.
- Parameters:
cmdparser – The argument parser to add arguments to.
group – Optional argument group to add arguments to. If None, creates a new group.
- static pds4_bundle_path_for_image(image_name: str) str[source]
Maps image name to bundle directory path.
- pds4_bundle_template_dir() str[source]
Returns absolute path to template directory for PDS4 bundle generation.
- pds4_path_stub(image_file: ImageFile) str[source]
Returns PDS4 path stub for bundle directory structure.
- pds4_template_variables(*, image_file: ImageFile, nav_metadata: dict[str, Any], backplane_metadata: dict[str, Any]) dict[str, Any][source]
Returns template variables for PDS4 label generation.
- static supported_grouping() list[str][source]
Returns the list of supported grouping types.
- Returns:
The list of supported grouping types.