satrain.metrics#
Defines the metrics used to evaluate precipitation retrievals.
The metrics objects are for iterative evaluation, i.e., results are passed to
the metric iteratively for each collocation scene. The metric object keeps
track of the necessary quantities required to compute the metrics. Finally, the
value of the metrics over all considered scene can be computed using each
metric’s compute function. The metrics use shared memory to track required
quantities so that evaluation can be performed in parallel using multiple
processes.
Usage#
While the metric classes defined here can, in principle, be used on their own,
their intended use is with the satrain.evaluation.Evaluator class, which
holds the metrics to track in its precip_quantification_metrics,
precip_detection_metrics, prob_precip_detection_metrics,
heavy_precip_detection_metrics, and prob_heavy_precip_detection_metrics.
evaluator.precip_quantification_metrics = [Bias()] # Track only bias
evaluator.precip_detection_metrics = [POD()] # Track only POD
evaluator.prob_precip_detection_metrics = [PRCurve()] # Track only PR curve
evaluator.heavy_precip_detection_metrics = [POD()] # Track only POD
evaluator.prob_heavy_precip_detection_metrics = [PRCurve()] # Track only PR curve
The metrics are used by the ipgml.evaluation.Evaluator to
- class Bias(relative: bool = True)[source]#
The bias, or mean error, calculated as the mean value of the difference between prediction and target values:
\[\begin{split}\\text{Bias} = \\mathbf{E}\{y_\\text{pred} - y_\\text{target}\}\end{split}\]where the mean is calculated over all results passed to the ‘compute’ method for which the target values are finite.
- Parameters:
relative – If True, the bias is calculated as percent of the mean reference precipitation. Else the bias is calculated as absolute value.
- class CRPS[source]#
The continuous ranked probability score supporting both deterministic and quantile predictions.
This metric calculates the CRPS using .. math:
\text{CRPS} = 2.0 \int_0^1 (\tau - I_{y_\text{target} < y_\text{pred}}) (y_\text{target} - y_\text{pred}) d\tau
If the prediction is deterministic, this metric simply calculates the mean-absolute error. If the prediction consists of several quantiles the integral above is approximated using the trapezoidal rule.
- compute() Dataset[source]#
Calculate the CRPS for all results passed to this metric object.
- Returns:
An xarray.Dataset containing a single, scalar variable ‘crps’.
- update(prediction: ndarray, target: ndarray, taus: ndarray | None = None) None[source]#
Update metric using a probabilistic prediction.
- Parameters:
prediction – An np.ndarray containing the predicted values. This can be deterministic values or a probabilistic prediction in the form of several quantiles of the distribution.
target – An np.ndarray containing the reference values.
tau – The quantile fraction corresponding to the probabili
- class CorrelationCoef[source]#
The linear correlation coefficient between predictions and target values.
\[\begin{split}\\text{Correlation coeff.} = \\mathbf{E}\\frac{ (y_\\text{pred} - \\mu_{y_\\text{pred}})(y_\\text{target} - \\mu{y_\\text{target})} }{ \\sigma_{y_\\text{pred}} \sigma_{y_\\text{target}} }\end{split}\]where the mean is calculated over all results passed to the ‘compute’ method for which the target values are finite and \(\\mu\) and \(\\sigma\) are used to denote the mean and standard deviations of the distributions of \(y_\text{pred}\) and \(y_\\text{target}\).
- class DetectionMetric(buffers: Dict[str, Tuple[Tuple[int], str]])[source]#
Helper class to identify metrics to assess precipitation detection.
- class Distribution(bins: ndarray | None = None)[source]#
Calculates a 2D histogram or retrieved and reference precipitation.
- compute() Dataset[source]#
Calculate the joint and marginal distribution as well as KL divergence.
- Returns:
An xarray.Dataset containing a the joint distribution (‘joint_distribution’), the marginal distributions (‘retrieved_distribution’ and ‘target_distribution’), and the KL divergence for the retrieved precipitation rates.
- class FAR[source]#
Metric to calculate the false alarm rate (FAR) for precipitation detection. The FAR is the fraction of false positive predictions and total number of positive predictions.
\[\text{FAR} = \frac{\#\text{False positive}}{\#\text{True positive} + \#\text{False positive}}\]
- class HSS[source]#
Metric to calculate the Heidke-Skill Score for precipitation detection. The HSS is using the formula given here.
- class MAE[source]#
The mean-absolute error calculated as the mean value of the absolute value of the difference between prediction and target values:
\[\text{MAE} = \mathbf{E}\{|y_\text{pred} - y_\text{target}|\}.\]where the mean is calculated over all results passed to the ‘compute’ method for which the target values are finite.
- class MSE[source]#
The mean-squared error calculated as the mean value of the squared difference between prediction and target values:
\[\begin{split}\\text{MSE} = (\\mathbf{E}\{y_\\text{pred} - y_\\text{target}\})^2\end{split}\]where mean is calculated over all results passed to the ‘compute’ method for which the target values are finite.
- class Metric(buffers: Dict[str, Tuple[Tuple[int], str]])[source]#
Base class for metrics that manages shared data arrays and can be used to manage the access to those arrays.
- class POD[source]#
Metric to calculate the probability of detection (POD) for precipitation detection. The POD is the ratio of true positive predictions and the total number of observed events.
\[\text{POD} = \frac{\#\text{true positive}}{\#\text{True positive} + \#\text{False negative}}\]
- class PRCurve(n_bins: int = 100, range: Tuple[float, float] = (0.0, 1.0), logarithmic: bool = False)[source]#
Calculates the precision recall curve for probabilistic detection results. The precision recall curve is a probabilistic detection metrics and thus expects predictions to be probabilities normalized to lie within \([0, 1]\). If the probabilities are not normalized the
rangeargument can be used to define a customized value range.The precision and recall are defined as:
\[\begin{split}\\text{Precision} = \\frac{\# \\text{True positive}}{\# \\text{True positive} + \# \\text{False positive}}\end{split}\]\[\begin{split}\\text{Recall} = \\frac{\# \\text{True positive}}{\# \\text{True positive} + \# \\text{False negative}}\end{split}\]Both precision and recall are calculated for a range of detection thresholds, i.e., values of the threshold probability above which an even is classified as positive. The values yield a curve representing the trade off between recall and precision as the detection threshold is increased.
- class ProbabilisticDetectionMetric(buffers: Dict[str, Tuple[Tuple[int], str]])[source]#
Helper class to identify metrics to assess probabilistic precipitation detection.
- class ProbabilisticQuantificationMetric(buffers: Dict[str, Tuple[Tuple[int], str]])[source]#
Helper class to identify metrics to assess precipitation quantification.
- class QuantificationMetric(buffers: Dict[str, Tuple[Tuple[int], str]])[source]#
Helper class to identify metrics to assess precipitation quantification.
- class SMAPE(threshold: float = 0.1)[source]#
The symmetric mean absolute percentage error (SMAPE) with threshold \(t\).
\[\begin{split}\\text{SMAPE}_t = \\mathbf{E}_{t \\leq y_\\text{target}}\\{\\frac{|y_\\text{pred} - y_\\text{target}|}{ 0.5 (|y_\\text{pred}| + |y_\\text{target}|)}\}\end{split}\]where the mean is calculated over all results passed to the ‘compute’ method for which the target values are finite and for which the absolute value of the exceeds the given threshold value.
- Parameters:
threshold – Minimum target value for samples to be considered in the calculation.
- class SpectralCoherence(window_size=32, scale=0.036)[source]#
Metric to calculate spectral coherence curves and effective resolution for retrieved precipitation fields. Spectral coherence and effective resolution are calculated as described in:
Pfreundschuh, S., Guilloteau, C., Brown, P. J., Kummerow, C. D., and Eriksson, P.: GPROF V7 and beyond: assessment of current and potential future versions of the GPROF passive microwave precipitation retrievals against ground radar measurements over the continental US and the Pacific Ocean, Atmos. Meas. Tech., 17, 515–538, https://doi.org/10.5194/amt-17-515-2024, 2024.
- Parameters:
window_size – The size of the window over which the spectral coherence is computed.
scale – Spatial extent of a single pixel. Defaults to 0.036 degree which is the resolution used for the gridded data of the SatRain dataset.
- class ValidFraction[source]#
This metric tracks the number of predictions that are left out because the retrieved value is NAN.
- iterate_windows(valid, window_size)[source]#
Iterate over non-overlapping windows in which all pixels are valid.
- Parameters:
valid – A 2D numpy array identifying valid pixels.
window_size – The size of the windows.
- Returns:
An iterator providing coordinates of randomly chosen windows that that cover the valid pixels in the given field.