Testing Functionality

Testing Functionality#

The satrain package provides built-in testing functionality for making the retrieval evaluation easy and reproducible. The testing data of the SatRain dataset has been kept in the original collocation scenes that have irregular sizes. While irregular-sized input data is uncommon in most traditional machine-learning applications, they are a practical reality for most satellite retrievals. The original observation structure of the testing data was retained to make it easier to evaluate existing retrieval algorithms on the data. Morever, evaluating all retrievals against the gridded precipitation reference also ensures that retrievals operating on the on-swath and gridded geometries can be compared directly. To simplify running most retrievals on the testing data, the satrain.evaluation module provides functionality to batch and tile the input data and reconstructing the retrieval results.

The `Evaluator` class#

All evaluation functionality provided by the satrain package is implemented by the satrain.evaluation.Evaluator class. To use the Evaluator class to evaluate a retrieval, the user only needs to provide is an interface function to their retrieval. The evaluator class will then automatically load the input data and track the accuracy of the resulting retrieval results.

This documentation illustrates how the evaluator loads the input data in different formats and demonstrate how the evaluator can be used to evaluate retrievals.

We start by instantiating an evaluator object that will load gridded GMI observations and the total_precipitation field from the ancillary data over the CONUS domain.

%load_ext autoreload
%autoreload 2
from satrain.evaluation import Evaluator
evaluator = Evaluator(
    base_sensor="gmi",
    geometry="gridded",
    retrieval_input=["gmi", {"name": "ancillary", "variables": ["total_precipitation"]}],
    domain="conus"
)

The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload

Evaluating Precipitation Retrievals#

Above we have seen how the evaluator can be used to load the testing data in different formats. However, it is generally not necessary to use these function directly to evaluate a retrieval. Instead, the evaluator will take care of the data loading and evaluating the retrieval results in one go. All we need to do for this is write an interface function that runs the retrieval on the input data loaded by the evaluator.

Below we define a very simple retrieval function, which simply returns the ERA5 total precipitation from the input data but renames it to the surface_precip variable expected by the evaluator.

import xarray as xr

def retrieval_fn(input_data: xr.Dataset) -> xr.Dataset:
    """
    Retrieval interface function returning the total precipitation from the ancillary data.

    Args:
        input_data: A xr.Dataset containing input data loaded by the evaluator.

    Return:
        A xr.Dataset containing the 'surface_precip'.
    """
    print(f"Input data shape: ({input_data["ancillary"].shape})")
    return xr.Dataset({
        "surface_precip": 1e3 * input_data.ancillary[{"features_ancillary": 0}] # Remove feature dimension and convert to mm/h.
    })

Once we have the retrieval_fn interface function defined, we can use it to visualize the retrieval results. Note that the retrieval_fn defined above works with any possible input data format since it simply returns the ER5 precipitation from the input data. Note that depending on whether the input is tiled, or batched, or both, the function may be called multiple times.

evaluator.plot_retrieval_results(
    1637,
    retrieval_fn=retrieval_fn,
    input_data_format="spatial",
    tile_size=(256, 256),
    batch_size=2,
    include_metrics=True
);

Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((2, 1, 256, 256))
Input data shape: ((1, 1, 256, 256))

../_images/2ee9518aa9b907f81d8b7f4801dc05d75c3c849aff7c4e31b828de9ed296fc19.png

Result format#

For the evaluator to be able to evaluate the retrieval, the results returned by the interface function must adhere to the expected data format. The evaluator expects an xarray.Dataset with the same dimensions as the provided input data (except for the feature dimensions). The table below lists the expected variables.

Note: that the retrieval can provide any subset of them as they will be evaluated independently.

Task	Variable name	Meaning	Data type	Value range
Precipitation Quantification	`surface_precip`	The surface precipitation rate in mm h$^{-1}$	floating point	$0 \leq$
Prob. Precipitation Detection	`probability_of_precip`	Estimated probability of observing a raining pixel	floating point	[0, 1]
Precipitation Detection	`precip_flag`	Flag indicating whether a pixel is predicted to be raining or not	bool	{`True`,`False`}
Prob. Heavy Precipitation Detection	`probability_of_heavy_precip`	Estimated probability of observing heavy precipitation (RR > 10 mm $h^{-1}$)	floating point	[0, 1]
Heavy Precipitation Detection	`heavy_precip_flag`	Flag indicating whether a pixel is predicted to contain heavy precipitation	bool	{`True`,`False`}

Metrics#

The evaluator keeps track of the accuracy of separate metrics for each of the five tasks that the retrieval is evaluated on. If the target variable is not present in the retrieval results that accuracy of these results is not tracked.

Precipitation quantification#

By default, the quantitative precipitation estimates returned by the retrieval callback function (surface_precip) are evaluated using the following metrics.

Bias
The mean absolute error (MAE)
the mean squared error (MSE)
The mean absolute percentage error (SMAPE) for precipitation rates exceedin 0.1 mm h$^{-1}$.
The linear correlation coefficient
The spectral coeherence from which the effective resolution is computed

Precipitation detection#

By default, the deterministic precipitation detection (precip_flag) and deterministic heavy precipitation (heavy_precip_flag) estimates returned by the retrieval callback function are evaluated using the following metrics.

POD
FAR
Heidke Skill Score (HSS)

Probabilistic precipitation detection#

The probabilistic precipitation detection (precip_flag) and probabilistic heavy precipitation (heavy_precip_flag) estimates returned by the retrieval callback function are evaluated using the following metrics.

Precision-recall curve (PR curve)

Running the Evaluation#

Evaluating the retrieval on the full testing dataset is then as easy as calling the evaluator.evaluate function. Below, we run it on the Austria domain as it is the smallest testing dataset. We also redefine the interface function to not print the input data shape on every invocation.

Note: The evaluator support running the evaluation in parallel using multiple processes via the n_processes keyword argument. However, this requires the retrieval function to be pickleable and safe for multiprocessing and thus may not work with any retrieval_fn.

def retrieval_fn(input_data: xr.Dataset) -> xr.Dataset:
    """
    Retrieval interface function returning the total precipitation from the ancillary data.

    Args:
        input_data: A xr.Dataset containing input data loaded by the evaluator.

    Return:
        A xr.Dataset containing the 'surface_precip'.
    """
    return xr.Dataset({
        "surface_precip": 1e3 * input_data.ancillary[{"features_ancillary": 0}] # Remove feature dimension and convert to mm/h.
    })
    
evaluator = Evaluator(
    base_sensor="gmi",
    geometry="gridded",
    retrieval_input=["gmi", {"name": "ancillary", "variables": ["total_precipitation"]}],
    domain="austria"
)
evaluator.evaluate(retrieval_fn=retrieval_fn);

Displaying the Results#

After the evaluation has been run, the results can be accessed using corresponding getter functions. The results are returned as pandas.DataFrame that displays nicely in the notebook environment.

evaluator.get_precip_quantification_results(name="Test retrieval")

/home/simon/src/satrain/src/satrain/baselines.py:61: FutureWarning: In a future version of xarray the default value for join will change from join='outer' to join='exact'. This change will result in the following ValueError: cannot be aligned with join='exact' because index/labels/sizes are not equal along these coordinates (dimensions): 'scales' ('scales',) The recommendation is to set join explicitly for this case.
  results = xr.concat(results, dim="algorithm")

	Bias [$\%$]	MAE [$mm h^{-1}$]	MSE [$(mm h^{-1})^2$]	SMAPE$_{0.1}$ [$\%$]	Correlation coeff. []	Effective resolution [$^\circ$]
Test retrieval	76.231029	0.112298	0.265545	137.783548	0.315425	inf
ERA5	27.824237	0.113220	0.173934	141.840027	0.458022	inf
GPROF V7 (GMI)	27.085501	0.112640	0.211739	96.359131	0.782981	inf

Testing Functionality

Contents

Testing Functionality#

The `Evaluator` class#

Loading Input Data#

Tiling the Input Data#

Loading Tabular Data#

Evaluating Precipitation Retrievals#

Result format#

Metrics#

Precipitation quantification#

Precipitation detection#

Probabilistic precipitation detection#

Running the Evaluation#

Displaying the Results#

Task	Variable name	Meaning	Data type	Value range
Precipitation Quantification	`surface_precip`	The surface precipitation rate in mm h\(^{-1}\)	floating point	\(0 \leq\)
Prob. Precipitation Detection	`probability_of_precip`	Estimated probability of observing a raining pixel	floating point	[0, 1]
Precipitation Detection	`precip_flag`	Flag indicating whether a pixel is predicted to be raining or not	bool	{`True`,`False`}
Prob. Heavy Precipitation Detection	`probability_of_heavy_precip`	Estimated probability of observing heavy precipitation (RR > 10 mm \(h^{-1}\))	floating point	[0, 1]
Heavy Precipitation Detection	`heavy_precip_flag`	Flag indicating whether a pixel is predicted to contain heavy precipitation	bool	{`True`,`False`}

Testing Functionality

Contents

Testing Functionality#

The Evaluator class#

Loading Input Data#

Tiling the Input Data#

Loading Tabular Data#

Evaluating Precipitation Retrievals#

Result format#

Metrics#

Precipitation quantification#

Precipitation detection#

Probabilistic precipitation detection#

Running the Evaluation#

Displaying the Results#

The `Evaluator` class#