satrain.evaluation

Evaluation functionality for the IPWGML SatRain dataset.

This module provides the Evaluator class that implements a generic retrieval evaluator based on the test data split of the IPWG SatRain dataset.

The evaluator takes care of downloading the data and loading it in the format required by the retrieval. The interface to the retrieval has to be provided in the form of a retrieval callback function. The evaluator calls this function with an xarray.Dataset containing the retrieval input data in the format requested by the user and expects the retrieval callback function to return the corresponding retrieval results. The evaluator than assess the results against the reference estimates using various metrics.

Members

class Evaluator(base_sensor: str, geometry: str, retrieval_input: List[str | Dict[str, Any | InputConfig]] | None = None, domain: str = 'conus', target_config=None, data_path: Path | None = None, download: bool = True)

The Evaluator class provides an interface to evaluate a generic retrieval implemented by a retrieval callback function using the IPWG SatRain dataset.

Parameters:

• base_sensor – The name of SatRain reference sensor

• geometry – The geometry of the retrieval. ‘gridded’ for retrievals operating on the regridded input observations; ‘on_swath’ for retrievals operating on the nativ swath-based observations.

• retrieval_input – The retrieval inputs to load. Should be a subset of [‘gmi’, ‘mhs’, ‘ancillary’, ‘geo’, ‘geo_ir’]

• domain – The domain over which to evaluate the retrieval.

• data_path – An optional path to the location of the ipgml data.

• download – A boolean flag indicating whether or not to download the evaluation files if they are not found in ‘data_path’.

evaluate(retrieval_fn: Callable[[Dataset], Dataset], tile_size: int | Tuple[int, int] | None = None, overlap: int | None = None, batch_size: int | None = None, input_data_format: str = 'spatial', n_processes: int | None = None, output_path: Path | None = None)

Run evaluation on complete test dataset.

Parameters:

• retrieval_fn – The retrieval callback function.

• tile_size – The tile size to use for the retrieval or ‘None’ to apply no tiling.

• overlap – The overlap to apply for the tiling.

• batch_size – Maximum batch size for tiled spatial and tabular retrievals.

• input_data_format – The retrieval kind: ‘spatial’ or ‘tabular’.

• output_path – If not ‘None’, retrieval results will be written to that path.

evaluate_scene(index: int, tile_size: int | Tuple[int, int] | None, overlap: int | None, batch_size: int | None, retrieval_fn: Callable[[Dataset], Dataset], input_data_format: str, track: bool = False, output_path: Path | None = None) → Dataset

Run tests on a single scene.

Parameters:

• index – An index identifying the scene.

• tile_size – The tile size to use for the retrieval or ‘None’ to apply no tiling.

• overlap – The overlap to apply for the tiling.

• batch_size – Maximum batch size for tiled spatial and tabular retrievals.

• retrieval_fn – The retrieval callback function.

• input_data_format – Whether the retrieval expects input data in ‘tabular’ or ‘spatial’ format.

• track – If ‘True’ will track the retrieval results using the evaluator’s metrics. If ‘False’, results will not be tracked.

• output_path – If not ‘None’, retrieval results will be written to that path.

Returns:

An xarray.Dataset containing the retrieval results.

evaluate_scene_no_results(index: int, tile_size: int | Tuple[int, int] | None, overlap: int | None, batch_size: int | None, retrieval_fn: Callable[[Dataset], Dataset], input_data_format: str, track: bool = False, output_path: Path | None = None) → Dataset

Wrapper around evaluate_scene that discards the return value.

get_heavy_precip_detection_results(name: str | None = None, include_baselines: bool = True, baselines: List[str] | None = None) → DataFrame

Get scalar results from heavy precipitation detection metrics as pandas.Dataframe.

Parameters:

• name – An optional name for the retrieval algorithm.

• include_baselines – If ‘True’, results from retrieval baselines will be included in the results.

• baselines – List of names of the baseline to include in the results.

Returns:

A pandas.DataFrame containing the combined scalar results from the ‘heavy_precip_detection_metrics’ of this Evaluator object.

get_input_data(scene_index: int, format: str = 'spatial', batch_size: int | None = None, tile_size: Tuple[int, int] | None = None, overlap: int | None = None) → Dataset

Get retrieval input data for a given scene.

Parameters:

• scene_index – An integer specifying the scene for which to load the input data.

• format – The format in which to load the data. ‘spatial’ will load the data in the original 2D format of the spatial scenes. ‘tabular’ will load the data as a flattened sequence of pixels.

• batch_size – If format is ‘tabular’ or the data is tiled, the batch size can be used to load data in batches.

• tile_size – An optional tile size to use to tile the data input fixed-size tiles.

• overlap – An optional overlap to apply between neighboring tiles to avoid artifacts in the results.

Returns:

An xarray.Dataset containing the retrieval input data.

get_input_data_spatial(scene_index: int, tile_size: int | Tuple[int, int] | None = None, overlap: int | None = None, batch_size: int | None = 1) → Dataset

Get retrieval input data for a given scene.

Parameters:

• scene_index – An integer specifying the scene for which to load the input data.

• tile_size – Optional int or tuple of ints specifying the height and width of the tiling to apply to the input data.

• overlap – The width of the overlap between neighboring tiles. Defaults to a fourth of the tile size if not given.

• batch_size – Set to number larger than one to return batches of multiple tiles.

Returns:

An xarray.Dataset containing the input data or an iterator over the input data tiles.

get_input_data_tabular(scene_index: int, batch_size: int | None = None)

Get retrieval input data in for a given scene in tabular format.

Parameters:

• scene_index – An integer specifying the scene for which to load the input data.

• batch_size – Optional batch size to use to batch the input data.

Returns:

An xarray.Dataset containing the retrieval input data or an iterator over the batched input data.

get_input_files(index: int) → InputFiles

Compile retrieval input and target files for a given collocation.

Parameters:

index – The collocation index.

Returns:

An InputFiles object containing all available input and target files for the given collocation.

get_precip_detection_results(name: str | None = None, include_baselines: bool = True, baselines: List[str] | None = None) → DataFrame

Get scalar results from precipitation detection metrics as pandas.Dataframe.

Parameters:

• name – An optional name for the retrieval algorithm.

• include_baselines – If ‘True’, results from retrieval baselines will be included in the results.

Returns:

A pandas.DataFrame containing the combined scalar results from the ‘precip_detection_metrics’ of this Evaluator object.

get_precip_quantification_results(name: str | None = None, include_baselines: bool = True, baselines: List[str] | None = None) → DataFrame

Get scalar results from precipitation estimation metrics as pandas.Dataframe.

Parameters:

• name – An optional name for the retrieval algorithm.

• include_baselines – If ‘True’, results from retrieval baselines will be included in the results.

Returns:

A pandas.DataFrame containing the combined scalar results from the ‘precip_quantification_metrics’ of this Evaluator object.

get_prob_heavy_precip_detection_results(name: str | None = None, include_baselines: bool = True, baselines: List[str] | None = None) → DataFrame

Get scalar results from probabilistic heavy precipitation detection metrics as pandas.Dataframe.

Parameters:

• name – An optional name for the retrieval algorithm.

• include_baselines – If ‘True’, results from retrieval baselines will be included in the results.

Returns:

A pandas.DataFrame containing the combined scalar results from the ‘prob_heavy_precip_detection_metrics’ of this Evaluator object.

get_prob_precip_detection_results(name: str | None = None, include_baselines: bool = True, baselines: List[str] | None = None) → DataFrame

Get scalar results from probabilistic precipitation detection metrics as pandas.Dataframe.

Parameters:

• name – An optional name for the retrieval algorithm.

• include_baselines – If ‘True’, results from retrieval baselines will be included in the results.

• baselines – List of names of the baseline to include in the results.

Returns:

A pandas.DataFrame containing the combined scalar results from the ‘prob_precip_detection_metrics’ of this Evaluator object.

get_results() → Dataset

Combind results from all tracked metrics into a single xarray.Dataset.

property heavy_precip_detection_metrics

List containing the metrics used to evaluate the detection of heavy precipitation.

plot_precip_quantification_results(name: str | None = None, include_baselines: bool = True, other_results=None, n_col: int = 4) → plt.Figure

Plot precipitation quantification results

Produces a plot showing the results from the precipitation quantification metrics.

Parameters:

• name – Name to use for the results of the current retrieval.

• include_baselines – Whether or not to include results from the baseline retrievals.

• n_col – The number of colums to use for the plot.

Returns:

The matplotlib.Figure containing the plotted results.

plot_retrieval_results(scene_index: int, retrieval_fn: Callable[[Dataset], Dataset], input_data_format: str = 'spatial', tile_size: int | Tuple[int, int] | None = None, overlap: int | None = None, batch_size: int | None = None, swath_boundaries: bool = False, ax_width: int = 5, contour_legend: bool = True, include_metrics: bool = False, n_rows: int = 1, bounds: Tuple[float, float, float, float] | None = None) → plt.Figure

Plot retrieval results for a given retrieval scene.

Parameters:

• scene_index – An integer identifying the scene for which to plot the retrieval results.

• retrieval_fn – The retrieval callback function.

• input_data_format – The retrieval kind: ‘spatial’ or ‘tabular’.

• tile_size – The tile size to use for the retrieval or ‘None’ to apply no tiling.

• overlap – The overlap to apply for the tiling.

• batch_size – Maximum batch size for tiled spatial and tabular retrievals.

• swath_boundaries – If ‘True’ will plot swath boundaries of the GPM base_sensor.

• ax_width – The width of each axes objects in inches.

• contour_legend – Whether or not to draw a legend for the radar boundary contours.

• include_metrics – Whether or not to print metrics onto retrieval results.

• n_rows – The number of rows across which to distribute the plots.

• bounds – An optional tuple (lon_min, lat_min, lon_max ,lat_max) defining the longitude and latitude coordinates to use a x-axis and y-axis limits, respectively.

Returns:

The matplotlib.Figure object containing the plot.

property precip_detection_metrics

List containing the metrics used to evaluate precipitation detection.

property precip_quantification_metrics

List containing the metrics used to evaluate quantiative precipitation estimates.

property prob_heavy_precip_detection_metrics

List containing the metrics used to evaluate the probabilistic detection of heavy precipitation.

property prob_precip_detection_metrics

List containing the metrics used to evaluate precipitation detection.

property set_detection_metric

List containing the metrics used to evaluate precipitation detection.

property set_heavy_precip_detection_metrics

List containing the metrics used to evaluate the detection of heavy precipitation.

property set_prob_heavy_precip_detection_metrics

List containing the metrics used to evaluate precipitation detection.

property set_prob_precip_detection_metrics

List containing the metrics used to evaluate precipitation detection.

property stats: Dataset

Load surface precipitation statistics for this evaluator’s base sensor and domain.

Returns:

An xarray.Dataset containing precipitation statistics for the evaluator’s sensor and domain.

class InputFiles(target_file_gridded: Path, target_file_on_swath: Path, gmi_file_gridded: Path, gmi_file_on_swath: Path, atms_file_gridded: Path, atms_file_on_swath: Path, ancillary_file_gridded: Path, ancillary_file_on_swath: Path, geo_file_gridded: Path | None, geo_file_on_swath: Path | None, geo_t_file_gridded: Path | None, geo_t_file_on_swath: Path | None, geo_ir_file_gridded: Path | None, geo_ir_file_on_swath: Path | None, geo_ir_t_file_gridded: Path | None, geo_ir_t_file_on_swath: Path | None)

Helper class that holds the input files required for evaluation.

get_path(name: str, geometry: str) → Path | None

Get path to input data file for given input and geometry.

Parameters:

• name – The name of the input.

• geometry – A string specifying the geometry: ‘on_swath’ or ‘gridded’.

Returns:

A Path object pointing to the input file to load or None.

evaluate_scene(input_files: InputFiles, retrieval_input: List[InputConfig], target_config: TargetConfig, geometry: str, tile_size: int | Tuple[int, int] | None, overlap: int | None, batch_size: int | None, retrieval_fn: Callable[[Dataset], Dataset], input_data_format: str, precip_quantification_metrics: List[Metric], precip_detection_metrics: List[Metric], prob_precip_detection_metrics: List[Metric], heavy_precip_detection_metrics: List[Metric], prob_heavy_precip_detection_metrics: List[Metric], output_path: Path | None = None) → Dataset

Evaluate retrieval on a single collocation file.

Parameters:

• input_files – An input files record containing the paths to all retrieval input files.

• retrieval_input – A list defining the retrieval inputs to load.

• target_config – An optional TargetConfig specifying quality requirements for the retrieval target data to load.

• geometry – A string defining the geometry of the retrieval: ‘on_swath’ or ‘gridded’.

• tile_size – The tile size to use for the retrieval or ‘None’ if no tiling should be applied.

• overlap – The overlap to apply for the tiling.

• batch_size – If not ‘None’, inputs to ‘retrieval_fn’ will be batched using the given batch size. This only has an effect for tabular and spatial retrievals with tiling. Batches may include less samples than the batch size.

• retrieval_fn – A callback function that runs the retrieval on the input data.

• input_data_format – A string specifying whether the retrieval expects input data in spatial or tabular format.

• precip_quantification_metrics – A list containing the metrics to use to evaluate quantitative precipitation estimates.

• precip_detection_metrics – A list containing the metrics to use to evaluate the precipitation detection.

• prob_precip_detection_metrics – A list containing the metrics to use to evaluate the probabilistic precipitation detection.

• heavy_precip_detection_metrics – A list containing the metrics to use to evaluate the heavy precipitation detection.

• prob_heavy_precip_detection_metrics – A list containing the metrics to use to evaluate the probabilistic heavy precipitation detection.

• output_path – If given the retrieval results from the scene will be written to this path.

get_expected_dims(input_data: Dataset) → Tuple[str]

Given an xarray.Dataset containing the retrieval input data, calculate the expected dimensions in the retrieval results.

load_retrieval_input_data(input_files: InputFiles, retrieval_input: List[InputConfig], geometry: str) → Dataset

Load retrieval input data.

Parameters:

• input_files – A InputFiles dataclass object specifying the input files for the given collocation.

• retrieval_input – List of the retrieval inputs.

• geometry – The type of data to load: “on_swath” or “gridded”.

Returns:

An xarray.Dataset containing the input data from the sources specified in ‘retrieval_input’ as well as the latitude and longitude coordinates and meaurements times of the reference precipitation estimates.

process(retrieval_fn: Callable[[Dataset], Dataset], input_data: Dataset, coords: Tuple[int, int], result_tiler: DatasetTiler) → List[str]

Performs the retrieval on a single tile adds the retrieval results to the corresponding result tile.

Parameters:

• retrieval_fn – The retrieval callback function.

• input_data – An xarray.Dataset containing the tiled retrieval input data.

• coords – A tuple containing the row- and column-index of the tile that is being processed.

• result_tiler – The tiler providing access to the result dataset.

Returns:

A list containing the names of the retrieval variables that were present in the output from the retrieval callback function.

process_batched(retrieval_fn: Callable[[Dataset], Dataset], input_data: List[Dataset], spatial_dims: List[str], coords: List[Tuple[int, int]], result_tiler: DatasetTiler) → List[str]

Performs the retrieval on a batch of input data tiles and adds the retrieval results to the corresponding result tiles.

Parameters:

• retrieval_fn – The retrieval callback function.

• input_data – An xarray.Dataset containing the a batch of input data tiles.

• coords – A tuple containing the row- and column-index of the tile that is being processed.

• result_tiler – The tiler providing access to the result dataset.

Returns:

A list containing the names of the retrieval variables that were present in the output from the retrieval callback function.

process_scene_spatial(input_data: Dataset, tile_size: int | Tuple[int, int] | None, overlap: int | None, batch_size: int | None, retrieval_fn: Callable[[Dataset], Dataset]) → Dataset

Process an overpass scene using a given retrieval callback function for an image-based retrieval.

This function takes care of tiling and potentially batching of the input scenes.

Parameters:

• input_data – An xarray.Dataset containing all required input data for the scene.

• tile_size – The tile size expected by the retrieval function. Set to ‘None’ provide full scene as input data.

• overlap – The overlap between neighboring tiles.

• batch_size – The batch size expected by the retrieval function.

• retrieval_fn – The retrieval callback function to use to evaluate the retrieval on the input data.

Returns:

An xarray.Dataset containing the assembled retrieval results for the given input scene.

process_scene_tabular(input_data: Dataset, batch_size: int | None, retrieval_fn: Callable[[Dataset], Dataset]) → Dataset

Process a collocation scene with input data in tabular format.

Parameters:

• input_data – An xarary.Dataset containing the retrieval input data.

• batch_size – The batch size to use for processing.

• retrieval_fn – The retrieval callback function.

Returns:

An xarray.Dataset containing the retrieval results reshaped into their original 2D structure.