Data Access and Management

Data Access and Management#

The satrain.data module provides functionality for accessing, downloading, and loading SatRain datasets.

Key Functions#

get_files(base_sensor: str, split: str, input_data: str | List[str], geometry: str, domain: str = 'conus', subset: str = 'xl', data_path: str | Path | None = None, download: bool = True) → Dict[str, List[Path]][source]#

Get files in SatRain dataset.

Parameters:

  • base_sensor – The base sensor of the dataset.

  • split – Which split of the data to get (training, validation, testing).

  • input_data – List of the input data sources (‘gmi’, ‘atms’, ‘geo’, ‘geo_t’, ‘geo_ir’, ‘geo_ir_t’, ‘ancillary’)

  • geometry – For which retrieval geometry to download the data.

  • domain – Name of the domain for the testing data (‘austria’, ‘conus’, ‘korea’)

  • subset – The subset to download (xs, s, m, l, xl).

  • data_path – Optional path pointing to the path to store the data.

  • download – Download missing data.

Returns:

A dictionary listing locally available files for each input data source and the target data.

download_missing(dataset_name: str, base_sensor: str, geometry: str, split: str, source: str, subset: str = 'xl', domain: str = 'conus', destination: Path = None, progress_bar: bool = False) → None[source]#

Download missing file from dataset.

Parameters:

  • dataset_name – The name of the dataset, i.e., ‘satrain’ for the Satellite Rain Estimation and Detection (SatRain) dataset.

  • base_sensor – The base sensor (‘gmi’ or ‘atms’)

  • geometry – The viewing geometry (‘on_swath’, or ‘gridded’)

  • split – The name of the data split, i.e., ‘training’, ‘validation’, or ‘testing’.

  • subset – The subset, i.e, ‘xs’, ‘s’, ‘m’, ‘l’, or ‘xl’; only relevant for ‘training’, ‘validation’, or ‘testing’ splits.

  • domain – The name of the test domain. Only relevant if split=’testing’.

  • destination – Path pointing to the local directory containing the SatRain data.

  • progress_base – Whether or not display a progress bar displaying the download progress.

Returns:

A list containing the local paths of the downloaded files.

load_tabular_data(dataset_name: str, base_sensor: str, geometry: str, split: str, subset: str, retrieval_input: List[str | Dict[str, Any] | InputConfig], target_config: TargetConfig | None = None, data_path: Path | None = None)[source]#

Load data in tabular format.

Parameters:

  • dataset_name – The name of the dataset.

  • base_sensor – The base sensor.

  • geometry – The geometry, i.e., ‘on_swath’ or ‘gridded’.

  • split – Training or validation.

  • subset – The subset: ‘xs’, ‘s’, ‘m’, ‘l’, ‘xl’

  • retrieval_input – A list specifying the retrieval input.

  • target_config – A config dict or object defining the target data configuration.

  • data_path – Optional path pointing to the local data path.

Returns:

A tuple input_data, target with input_data being a dictionary containing the retrieval input as separate xarray.Datasets and target containing the target data.

list_local_files() → Dict[str, Any][source]#

List available SatRain files.

Dataset Subsets#

The SatRain dataset is available in multiple subset sizes to accommodate different use cases:

  • xs - Extra small (~1-5 GB): Quick testing and tutorials

  • s - Small (~10-20 GB): Development and prototyping

  • m - Medium (~50-100 GB): Algorithm validation

  • l - Large (~200-500 GB): Model development

  • xl - Extra large (~1-2 TB): Full-scale training

Usage Examples#

Basic file access:

from satrain.data import get_files

# Get files for small subset
files = get_files(
    base_sensor='gmi',
    split='training',
    subset='s',  # Small subset for development
    geometry='gridded'
)

Download missing data:

from satrain.data import download_missing

# Download small subset for quick start
download_missing(
    base_sensor='gmi',
    split='training',
    subset='xs'  # Start with extra small
)

Load tabular data efficiently:

from satrain.data import load_tabular_data

# Load progressively larger subsets

# Start small for development
data_small = load_tabular_data(
    base_sensor='gmi',
    subset='xs',
    inputs=['gmi']
)

# Scale up for training
data_large = load_tabular_data(
    base_sensor='gmi',
    subset='l',  # Large subset for serious training
    inputs=['gmi', 'geo', 'ancillary']
)

Subset Selection Guidelines:

# For learning and quick experiments
subset = 'xs'  # 1-5 GB, ~1K-5K scenes

# For algorithm development
subset = 's'   # 10-20 GB, ~10K-20K scenes

# For validation and comparison
subset = 'm'   # 50-100 GB, ~50K-100K scenes

# For model development
subset = 'l'   # 200-500 GB, ~200K-500K scenes

# For production training
subset = 'xl'  # 1-2 TB, ~1M+ scenes

All Functions#

satrain.data#

Provides functionality to access and download the SatRain data.

download_dataset(dataset_name: str, base_sensor: str, input_data: str | List[str], split: str, geometry: str, domain: str = 'conus', subset: str = 'xl', data_path: str | Path | None = None) → Dict[str, List[Path]][source]

Download SatRain dataset and return list of local files.

Parameters:

  • dataset_name – The SatRain dataset to download.

  • base_sensor – The base sensor of the dataset.

  • input_data – The input data sources for which to download the data.

  • split – Which split of the data to download.

  • geometry – For which retrieval geometry to download the data.

  • domain – Name of the test domain (optional).

  • subset – The subset to download (xs, s, m, l, xl).

  • data_path – Optional path pointing to the local data path.

Returns:

A dictionary listing locally available files for each input data source and the target data.

download_file(url: str, destination: Path) → None[source]

Download file from server.

Parameters:

  • url – A string containing the URL of the file to download.

  • destination – The destination to which to write the file.

download_files(base_url: str, files: List[str], destination: Path, progress_bar: bool = True, retries: int = 3) → List[str][source]

Download files using multiple threads.

Parameters:

  • base_url – The URL from which the remote data is available.

  • files – A list containing the relative paths of the files to download.

  • destination – A Path object pointing to the local path to which to download the files.

  • progress_bar – Whether or not to display a progress bar during download.

  • retries – The number of retries to perform for failed files.

Returns:

A list of the downloaded files.

download_missing(dataset_name: str, base_sensor: str, geometry: str, split: str, source: str, subset: str = 'xl', domain: str = 'conus', destination: Path = None, progress_bar: bool = False) → None[source]

Download missing file from dataset.

Parameters:

  • dataset_name – The name of the dataset, i.e., ‘satrain’ for the Satellite Rain Estimation and Detection (SatRain) dataset.

  • base_sensor – The base sensor (‘gmi’ or ‘atms’)

  • geometry – The viewing geometry (‘on_swath’, or ‘gridded’)

  • split – The name of the data split, i.e., ‘training’, ‘validation’, or ‘testing’.

  • subset – The subset, i.e, ‘xs’, ‘s’, ‘m’, ‘l’, or ‘xl’; only relevant for ‘training’, ‘validation’, or ‘testing’ splits.

  • domain – The name of the test domain. Only relevant if split=’testing’.

  • destination – Path pointing to the local directory containing the SatRain data.

  • progress_base – Whether or not display a progress bar displaying the download progress.

Returns:

A list containing the local paths of the downloaded files.

enable_testing() → None[source]

Enable test mode.

get_data_url(dataset_name: str) → str[source]

Returns the URL from which the SatRain data can be downloaded.

Parameters:

dataset_name – The name of the dataset (‘satrain’).

Returns:

A string containing the URL.

get_files(base_sensor: str, split: str, input_data: str | List[str], geometry: str, domain: str = 'conus', subset: str = 'xl', data_path: str | Path | None = None, download: bool = True) → Dict[str, List[Path]][source]

Get files in SatRain dataset.

Parameters:

  • base_sensor – The base sensor of the dataset.

  • split – Which split of the data to get (training, validation, testing).

  • input_data – List of the input data sources (‘gmi’, ‘atms’, ‘geo’, ‘geo_t’, ‘geo_ir’, ‘geo_ir_t’, ‘ancillary’)

  • geometry – For which retrieval geometry to download the data.

  • domain – Name of the domain for the testing data (‘austria’, ‘conus’, ‘korea’)

  • subset – The subset to download (xs, s, m, l, xl).

  • data_path – Optional path pointing to the path to store the data.

  • download – Download missing data.

Returns:

A dictionary listing locally available files for each input data source and the target data.

get_files_in_dataset(dataset_name: str) → Dict[str, Any][source]

Lists all available files for a given dataset.

Parameters:

dataset_name – The name of the dataset, i.e., ‘satrain’ for the Satellite Rain Estimation and Detection (SatRain) benchmar dataset.

Returns:

A nested dictionary containing all files in the dataset.

get_local_files(dataset_name: str, base_sensor: str, geometry: str, split: str, subset: str = 'xl', domain: str = 'conus', relative_to: Path | None = None, data_path: Path | None = None, check_consistency: bool = True) → Dict[str, Path][source]

Get all locally available files.

Parameters:

  • base_sensor – The name of the referene sensor.

  • geometry – The viewing geometry.

  • split – The split name.

  • subset – The subset name (only relevant for training and validation splits).

  • domain – The domain name (only relevant for testing split).

  • relative_to – If given, file paths will be relative to the given path rather than absolute.

  • data_path – The root directory containing IPWG data.

  • check_consitency – Whether or not to check consistency of the found files.

Returns:

A dictionary mapping data source names to the corresponding files.

list_local_files() → Dict[str, Any][source]

List available SatRain files.

list_local_files_rec(path: Path) → Dict[str, Any][source]

Recursive listing of SatRain data files.

Parameters:

path – A path pointing to a directory containing SatRain files.

Returns:

A dictionary containing all sub-directories

load_json_maybe_gzipped(path: Path)[source]

Loads a JSON file, handling both plain and gzipped (.gz) files.

Parameters:

path – A Path object pointing to the file to read.

Returns:

The deserialized Python object.

load_tabular_data(dataset_name: str, base_sensor: str, geometry: str, split: str, subset: str, retrieval_input: List[str | Dict[str, Any] | InputConfig], target_config: TargetConfig | None = None, data_path: Path | None = None)[source]

Load data in tabular format.

Parameters:

  • dataset_name – The name of the dataset.

  • base_sensor – The base sensor.

  • geometry – The geometry, i.e., ‘on_swath’ or ‘gridded’.

  • split – Training or validation.

  • subset – The subset: ‘xs’, ‘s’, ‘m’, ‘l’, ‘xl’

  • retrieval_input – A list specifying the retrieval input.

  • target_config – A config dict or object defining the target data configuration.

  • data_path – Optional path pointing to the local data path.

Returns:

A tuple input_data, target with input_data being a dictionary containing the retrieval input as separate xarray.Datasets and target containing the target data.

progress_bar_or_not(progress_bar: bool) → Progress | None[source]

Context manager for a optional progress bar.