Input and Target Data

This section describes the file structure of the data files containing the input data (satellite observations and ancillary data) and the target data. While this information is relevant for users who want to develop their own data loaders or evaluate external retrievals, the satrain dataset also provides ready-to-use PyTorch datasets that avoid manual data handling.

Input Data

The SatRain dataset distinguishes a total of seven input sources. The data are stored in separate files in the format <prefix>_<timestamp>.nc with a uniqe prefix identifying the data source and a shared timestamp corresponding to the median time of the PMW overpass. The different input sources are:

• PMW observations: atms and gmi, depending on the base sensor

• Geostationary Vis and IR: geo for the geo observations closest to the PMW overpass and geo_t for the time-resolved observations.

• Geostationary IR: geo_ir for the Geo-IR observations closest to the PMW overpass and geo_ir_t for the time-resolved observations.

• Ancillary data: ancillary

Below, we take look the file structure of these input data.

GMI and ATMS Observations

The GMI and ATMS observations contain the variables observations and earth_incidence_angle containing the PMW observations and the corresponding earth-incidence angles. Since the data loaded below is in gridded format, the spatial dimensions are latitude and longitude. For the on-swath data they are scan and pixel.

Note: The PMW files also contain the GPM L1C file and the start and end of the extracted scan range, which can be helpful to extract retrieval results from external retrieval algorithms.

import xarray as xr
from satrain.data import get_files
training_files = get_files("gmi", split="training", input_data=["gmi"], geometry="gridded", subset="xs")
# Load the first training file from the 'gmi' files in the training dataset.
xr.load_dataset(training_files["gmi"][0])

<xarray.Dataset> Size: 14MB
Dimensions:                (latitude: 256, longitude: 256, channel: 13)
Coordinates:
  * latitude               (latitude) float64 2kB 46.55 46.51 ... 37.31 37.27
  * longitude              (longitude) float64 2kB -77.76 -77.72 ... -68.48
Dimensions without coordinates: channel
Data variables:
    observations           (latitude, longitude, channel) float64 7MB 245.8 ....
    earth_incidence_angle  (latitude, longitude, channel) float64 7MB 52.79 ....
    scan_time              (latitude, longitude) datetime64[ns] 524kB 2018-01...
Attributes:
    satellite:       GPM
    sensor:          GMI
    preprocessor:    2407c
    scan_start:      1009
    scan_end:        1413
    gpm_input_file:  1C-R.GPM.GMI.XCAL2016-C.20180107-S191142-E204417.021938....
    lower_left_row:  147
    lower_left_col:  293
    pixel_start:     391
    pixel_end:       647

xarray.Dataset

• Dimensions:
  • latitude: 256
  • longitude: 256
  • channel: 13
• Coordinates: (2)
  • latitude
    (latitude)
    float64
    46.55 46.51 46.47 ... 37.31 37.27

    array([46.546079, 46.509701, 46.473323, ..., 37.342361, 37.305982, 37.269604])

  • longitude
    (longitude)
    float64
    -77.76 -77.72 ... -68.52 -68.48

    array([-77.75869 , -77.722312, -77.685934, ..., -68.554972, -68.518593,
           -68.482215])

• Data variables: (3)
  • observations
    (latitude, longitude, channel)
    float64
    245.8 242.1 246.3 ... nan nan nan

    array([[[245.83, 242.1 , 246.26, ..., 235.5 , 245.76, 247.33],
            [245.75, 240.05, 246.49, ..., 245.52, 246.85, 250.29],
            [245.75, 240.05, 246.49, ..., 245.52, 246.85, 250.29],
            ...,
            [251.01, 243.87, 249.62, ..., 232.31, 243.28, 242.  ],
            [250.86, 244.01, 249.95, ..., 233.67, 242.82, 243.27],
            [250.86, 244.01, 249.95, ..., 233.67, 242.82, 243.27]],
    
           [[245.83, 242.1 , 246.26, ..., 235.5 , 245.76, 247.33],
            [245.31, 241.07, 246.33, ..., 235.5 , 245.76, 247.33],
            [245.59, 241.75, 245.65, ..., 234.88, 245.9 , 245.75],
            ...,
            [250.86, 244.01, 249.95, ..., 233.67, 242.82, 243.27],
            [250.86, 244.01, 249.95, ..., 233.67, 242.82, 243.27],
            [250.86, 244.01, 249.95, ..., 233.67, 242.82, 243.27]],
    
           [[245.31, 241.07, 246.33, ..., 235.5 , 245.76, 247.33],
            [245.31, 241.07, 246.33, ..., 235.5 , 245.76, 247.33],
            [245.31, 241.07, 246.33, ..., 235.5 , 245.76, 247.33],
            ...,
    ...
            ...,
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan]],
    
           [[260.38, 256.52, 262.16, ..., 257.55, 249.8 , 257.51],
            [260.8 , 256.59, 262.55, ..., 255.88, 249.21, 257.35],
            [260.8 , 256.59, 262.55, ..., 255.88, 249.21, 257.35],
            ...,
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan]],
    
           [[260.8 , 256.59, 262.55, ..., 255.88, 249.21, 257.35],
            [260.8 , 256.59, 262.55, ..., 255.88, 249.21, 257.35],
            [260.8 , 256.59, 262.55, ..., 255.88, 249.21, 257.35],
            ...,
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan]]])

  • earth_incidence_angle
    (latitude, longitude, channel)
    float64
    52.79 52.79 52.79 ... nan nan nan

    array([[[52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            [52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            [52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            ...,
            [52.83, 52.83, 52.83, ..., 49.16, 49.16, 49.16],
            [52.83, 52.83, 52.83, ..., 49.16, 49.16, 49.16],
            [52.83, 52.83, 52.83, ..., 49.16, 49.16, 49.16]],
    
           [[52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            [52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            [52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            ...,
            [52.83, 52.83, 52.83, ..., 49.16, 49.16, 49.16],
            [52.83, 52.83, 52.83, ..., 49.16, 49.16, 49.16],
            [52.83, 52.83, 52.83, ..., 49.16, 49.16, 49.16]],
    
           [[52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            [52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            [52.79, 52.79, 52.79, ..., 49.11, 49.11, 49.11],
            ...,
    ...
            ...,
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan]],
    
           [[52.8 , 52.8 , 52.8 , ..., 49.13, 49.13, 49.13],
            [52.8 , 52.8 , 52.8 , ..., 49.13, 49.13, 49.13],
            [52.8 , 52.8 , 52.8 , ..., 49.13, 49.13, 49.13],
            ...,
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan]],
    
           [[52.8 , 52.8 , 52.8 , ..., 49.13, 49.13, 49.13],
            [52.8 , 52.8 , 52.8 , ..., 49.13, 49.13, 49.13],
            [52.8 , 52.8 , 52.8 , ..., 49.13, 49.13, 49.13],
            ...,
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan]]])

  • scan_time
    (latitude, longitude)
    datetime64[ns]
    2018-01-07T19:47:14 ... NaT

    array([['2018-01-07T19:47:14.000000000', '2018-01-07T19:47:14.000000000',
            '2018-01-07T19:47:14.000000000', ...,
            '2018-01-07T19:47:40.000000000', '2018-01-07T19:47:40.000000000',
            '2018-01-07T19:47:40.000000000'],
           ['2018-01-07T19:47:14.000000000', '2018-01-07T19:47:12.000000000',
            '2018-01-07T19:47:12.000000000', ...,
            '2018-01-07T19:47:40.000000000', '2018-01-07T19:47:40.000000000',
            '2018-01-07T19:47:40.000000000'],
           ['2018-01-07T19:47:12.000000000', '2018-01-07T19:47:12.000000000',
            '2018-01-07T19:47:12.000000000', ...,
            '2018-01-07T19:47:40.000000000', '2018-01-07T19:47:40.000000000',
            '2018-01-07T19:47:40.000000000'],
           ...,
           ['2018-01-07T19:44:37.000000000', '2018-01-07T19:44:37.000000000',
            '2018-01-07T19:44:37.000000000', ...,
                                      'NaT',                           'NaT',
                                      'NaT'],
           ['2018-01-07T19:44:37.000000000', '2018-01-07T19:44:37.000000000',
            '2018-01-07T19:44:37.000000000', ...,
                                      'NaT',                           'NaT',
                                      'NaT'],
           ['2018-01-07T19:44:37.000000000', '2018-01-07T19:44:37.000000000',
            '2018-01-07T19:44:37.000000000', ...,
                                      'NaT',                           'NaT',
                                      'NaT']], dtype='datetime64[ns]')

• Indexes: (2)
  • latitude
    PandasIndex

    PandasIndex(Index([46.546079223928864,  46.50970088924819,  46.47332255456751,
           46.436944219886826,  46.40056588520615,  46.36418755052547,
           46.327809215844795,  46.29143088116411, 46.255052546483434,
            46.21867421180276,
           ...
            37.59700889248182,  37.56063055780113, 37.524252223120456,
            37.48787388843978,   37.4514955537591, 37.415117219078425,
            37.37873888439774, 37.342360549717064,  37.30598221503638,
             37.2696038803557],
          dtype='float64', name='latitude', length=256))

  • longitude
    PandasIndex

    PandasIndex(Index([-77.75869037995149, -77.72231204527081, -77.68593371059013,
           -77.64955537590946, -77.61317704122877, -77.57679870654809,
           -77.54042037186741, -77.50404203718674, -77.46766370250606,
           -77.43128536782538,
           ...
           -68.80962004850444, -68.77324171382375, -68.73686337914307,
            -68.7004850444624, -68.66410670978172, -68.62772837510104,
           -68.59135004042037, -68.55497170573969, -68.51859337105901,
           -68.48221503637834],
          dtype='float64', name='longitude', length=256))

• Attributes: (10)

  satellite :

  GPM

  sensor :

  GMI

  preprocessor :

  2407c

  scan_start :

  1009

  scan_end :

  1413

  gpm_input_file :

  1C-R.GPM.GMI.XCAL2016-C.20180107-S191142-E204417.021938.V07A.HDF5

  lower_left_row :

  147

  lower_left_col :

  293

  pixel_start :

  391

  pixel_end :

  647

The ATMS files are only available when the base_sensor is ATMS. They contain a different number of channels.

import xarray as xr
from satrain.data import get_files
training_files = get_files("atms", split="training", input_data=["atms"], geometry="gridded", subset="xs")
# Load the first training file from the 'atms' files in the training dataset.
xr.load_dataset(training_files["atms"][0])

<xarray.Dataset> Size: 10MB
Dimensions:                (latitude: 256, longitude: 256, channel: 9)
Coordinates:
  * latitude               (latitude) float64 2kB 32.07 32.03 ... 22.83 22.79
  * longitude              (longitude) float64 2kB -82.92 -82.89 ... -73.65
Dimensions without coordinates: channel
Data variables:
    observations           (latitude, longitude, channel) float64 5MB nan ......
    earth_incidence_angle  (latitude, longitude, channel) float64 5MB nan ......
    scan_time              (latitude, longitude) datetime64[ns] 524kB NaT ......
Attributes:
    satellite:       NOAA20
    sensor:          ATMS
    preprocessor:    2412c
    scan_start:      2181
    scan_end:        2993
    gpm_input_file:  1C.NOAA20.ATMS.XCAL2019-V.20180107-S050836-E065004.00070...
    lower_left_row:  137
    lower_left_col:  110
    pixel_start:     36
    pixel_end:       292

xarray.Dataset

• Dimensions:
  • latitude: 256
  • longitude: 256
  • channel: 9
• Coordinates: (2)
  • latitude
    (latitude)
    float64
    32.07 32.03 31.99 ... 22.83 22.79

    array([32.067502, 32.031124, 31.994745, ..., 22.863783, 22.827405, 22.791027])

  • longitude
    (longitude)
    float64
    -82.92 -82.89 ... -73.68 -73.65

    array([-82.924414, -82.888036, -82.851657, ..., -73.720695, -73.684317,
           -73.647939])

• Data variables: (3)
  • observations
    (latitude, longitude, channel)
    float64
    nan nan nan ... 256.9 248.5 241.2

    array([[[   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            ...,
            [166.39, 168.17, 225.39, ..., 260.44, 252.61, 245.1 ],
            [166.39, 168.17, 225.39, ..., 260.44, 252.61, 245.1 ],
            [166.39, 168.17, 225.39, ..., 260.44, 252.61, 245.1 ]],
    
           [[   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            ...,
            [166.39, 168.17, 225.39, ..., 260.44, 252.61, 245.1 ],
            [166.39, 168.17, 225.39, ..., 260.44, 252.61, 245.1 ],
            [166.39, 168.17, 225.39, ..., 260.44, 252.61, 245.1 ]],
    
           [[   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            ...,
    ...
            ...,
            [224.98, 212.79, 272.81, ..., 256.95, 248.5 , 241.16],
            [224.98, 212.79, 272.81, ..., 256.95, 248.5 , 241.16],
            [224.98, 212.79, 272.81, ..., 256.95, 248.5 , 241.16]],
    
           [[   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            ...,
            [224.98, 212.79, 272.81, ..., 256.95, 248.5 , 241.16],
            [224.98, 212.79, 272.81, ..., 256.95, 248.5 , 241.16],
            [224.98, 212.79, 272.81, ..., 256.95, 248.5 , 241.16]],
    
           [[   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            [   nan,    nan,    nan, ...,    nan,    nan,    nan],
            ...,
            [222.95, 207.63, 269.3 , ..., 257.47, 249.38, 240.44],
            [224.98, 212.79, 272.81, ..., 256.95, 248.5 , 241.16],
            [224.98, 212.79, 272.81, ..., 256.95, 248.5 , 241.16]]])

  • earth_incidence_angle
    (latitude, longitude, channel)
    float64
    nan nan nan ... 29.85 29.85 29.85

    array([[[  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            ...,
            [40.42, 40.42, 40.38, ..., 40.4 , 40.4 , 40.4 ],
            [40.42, 40.42, 40.38, ..., 40.4 , 40.4 , 40.4 ],
            [40.42, 40.42, 40.38, ..., 40.4 , 40.4 , 40.4 ]],
    
           [[  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            ...,
            [40.42, 40.42, 40.38, ..., 40.4 , 40.4 , 40.4 ],
            [40.42, 40.42, 40.38, ..., 40.4 , 40.4 , 40.4 ],
            [40.42, 40.42, 40.38, ..., 40.4 , 40.4 , 40.4 ]],
    
           [[  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            ...,
    ...
            ...,
            [29.86, 29.86, 29.82, ..., 29.85, 29.85, 29.85],
            [29.86, 29.86, 29.82, ..., 29.85, 29.85, 29.85],
            [29.86, 29.86, 29.82, ..., 29.85, 29.85, 29.85]],
    
           [[  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            ...,
            [29.86, 29.86, 29.82, ..., 29.85, 29.85, 29.85],
            [29.86, 29.86, 29.82, ..., 29.85, 29.85, 29.85],
            [29.86, 29.86, 29.82, ..., 29.85, 29.85, 29.85]],
    
           [[  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            [  nan,   nan,   nan, ...,   nan,   nan,   nan],
            ...,
            [29.85, 29.85, 29.81, ..., 29.85, 29.85, 29.85],
            [29.86, 29.86, 29.82, ..., 29.85, 29.85, 29.85],
            [29.86, 29.86, 29.82, ..., 29.85, 29.85, 29.85]]])

  • scan_time
    (latitude, longitude)
    datetime64[ns]
    NaT NaT ... 2018-01-07T06:18:19

    array([[                          'NaT',                           'NaT',
                                      'NaT', ...,
            '2018-01-07T06:15:47.000000000', '2018-01-07T06:15:47.000000000',
            '2018-01-07T06:15:47.000000000'],
           [                          'NaT',                           'NaT',
                                      'NaT', ...,
            '2018-01-07T06:15:47.000000000', '2018-01-07T06:15:47.000000000',
            '2018-01-07T06:15:47.000000000'],
           [                          'NaT',                           'NaT',
                                      'NaT', ...,
            '2018-01-07T06:15:47.000000000', '2018-01-07T06:15:47.000000000',
            '2018-01-07T06:15:47.000000000'],
           ...,
           [                          'NaT',                           'NaT',
                                      'NaT', ...,
            '2018-01-07T06:18:19.000000000', '2018-01-07T06:18:19.000000000',
            '2018-01-07T06:18:19.000000000'],
           [                          'NaT',                           'NaT',
                                      'NaT', ...,
            '2018-01-07T06:18:19.000000000', '2018-01-07T06:18:19.000000000',
            '2018-01-07T06:18:19.000000000'],
           [                          'NaT',                           'NaT',
                                      'NaT', ...,
            '2018-01-07T06:18:22.000000000', '2018-01-07T06:18:19.000000000',
            '2018-01-07T06:18:19.000000000']], dtype='datetime64[ns]')

• Indexes: (2)
  • latitude
    PandasIndex

    PandasIndex(Index([  32.0675020210186,  32.03112368633792,  31.99474535165724,
           31.958367016976563, 31.921988682295883, 31.885610347615206,
           31.849232012934525, 31.812853678253845, 31.776475343573168,
           31.740097008892487,
           ...
            23.11843168957155, 23.082053354890874,  23.04567502021019,
           23.009296685529513, 22.972918350848836,  22.93654001616816,
           22.900161681487475, 22.863783346806798,  22.82740501212612,
           22.791026677445437],
          dtype='float64', name='latitude', length=256))

  • longitude
    PandasIndex

    PandasIndex(Index([-82.92441390460792, -82.88803556992723, -82.85165723524655,
           -82.81527890056587,  -82.7789005658852, -82.74252223120452,
           -82.70614389652384, -82.66976556184316, -82.63338722716247,
            -82.5970088924818,
           ...
           -73.97534357316087, -73.93896523848018,  -73.9025869037995,
           -73.86620856911883, -73.82983023443815, -73.79345189975747,
            -73.7570735650768,  -73.7206952303961, -73.68431689571543,
           -73.64793856103475],
          dtype='float64', name='longitude', length=256))

• Attributes: (10)

  satellite :

  NOAA20

  sensor :

  ATMS

  preprocessor :

  2412c

  scan_start :

  2181

  scan_end :

  2993

  gpm_input_file :

  1C.NOAA20.ATMS.XCAL2019-V.20180107-S050836-E065004.000708.V07A.HDF5

  lower_left_row :

  137

  lower_left_col :

  110

  pixel_start :

  36

  pixel_end :

  292

Geo Observations

The Geo Vis and IR observations contain all observations in a single variable observations. Visible channels are stored using reflectance in percent and thermal IR channels use K. The geo_t files contain an additional dimension containing all 10-minute observations within an hour of the overpass.

Note: Since the Vis/IR observations for different testing domains are derived from different instruments, the number of channels and their spectral coverage changes across the testing domains.

import xarray as xr
from satrain.data import get_files
training_files = get_files("gmi", split="training", input_data=["geo"], geometry="gridded", subset="xs")
# Load the first GEO training data file.
xr.load_dataset(training_files["geo"][0])

<xarray.Dataset> Size: 9MB
Dimensions:       (latitude: 256, longitude: 256, channel: 16)
Coordinates:
  * latitude      (latitude) float32 1kB 46.55 46.51 46.47 ... 37.34 37.31 37.27
  * longitude     (longitude) float32 1kB -77.76 -77.72 -77.69 ... -68.52 -68.48
    time          (latitude, longitude) datetime64[ns] 524kB 2018-01-07T19:43...
Dimensions without coordinates: channel
Data variables:
    observations  (latitude, longitude, channel) float64 8MB 21.9 ... 254.1

xarray.Dataset

• Dimensions:
  • latitude: 256
  • longitude: 256
  • channel: 16
• Coordinates: (3)
  • latitude
    (latitude)
    float32
    46.55 46.51 46.47 ... 37.31 37.27

    array([46.546078, 46.5097  , 46.473324, ..., 37.34236 , 37.30598 , 37.269604],
          dtype=float32)

  • longitude
    (longitude)
    float32
    -77.76 -77.72 ... -68.52 -68.48

    array([-77.75869 , -77.72231 , -77.685936, ..., -68.55497 , -68.51859 ,
           -68.482216], dtype=float32)

  • time
    (latitude, longitude)
    datetime64[ns]
    2018-01-07T19:43:39.500000 ... 2...

    array([['2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000', ...,
            '2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000'],
           ['2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000', ...,
            '2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000'],
           ['2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000', ...,
            '2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000'],
           ...,
           ['2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000', ...,
            '2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000'],
           ['2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000', ...,
            '2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000'],
           ['2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000', ...,
            '2018-01-07T19:43:39.500000000', '2018-01-07T19:43:39.500000000',
            '2018-01-07T19:43:39.500000000']], dtype='datetime64[ns]')

• Data variables: (1)
  • observations
    (latitude, longitude, channel)
    float64
    21.9 19.35 21.06 ... 261.0 254.1

    array([[[ 21.9 ,  19.35,  21.06, ..., 237.6 , 236.7 , 232.79],
            [ 21.78,  18.61,  20.82, ..., 236.92, 236.15, 232.79],
            [ 21.66,  18.55,  20.82, ..., 237.15, 236.42, 232.79],
            ...,
            [ 12.97,  10.01,  11.61, ..., 237.49, 237.02, 233.68],
            [ 12.97,  10.04,  11.61, ..., 237.32, 236.75, 233.15],
            [ 12.85,   9.54,  11.49, ..., 237.03, 236.59, 233.15]],
    
           [[ 21.9 ,  18.55,  20.82, ..., 238.23, 237.57, 233.85],
            [ 22.51,  18.99,  21.78, ..., 238.28, 237.73, 234.03],
            [ 21.04,  17.34,  19.63, ..., 238.95, 238.32, 234.56],
            ...,
            [ 12.48,   9.39,  11.01, ..., 238.06, 237.62, 233.68],
            [ 13.22,  10.43,  12.09, ..., 238.34, 237.89, 234.21],
            [ 13.09,  10.31,  12.09, ..., 237.94, 237.57, 233.85]],
    
           [[ 20.92,  17.22,  19.15, ..., 238.45, 237.84, 234.21],
            [ 19.94,  16.27,  17.59, ..., 237.43, 236.7 , 233.32],
            [ 19.58,  16.13,  17.24, ..., 239.67, 239.12, 235.08],
            ...,
    ...
            ...,
            [ 25.32,  22.36,  24.53, ..., 261.25, 261.04, 253.76],
            [ 23.98,  22.06,  22.86, ..., 261.04, 260.87, 253.76],
            [ 24.1 ,  23.24,  24.89, ..., 261.04, 260.87, 253.9 ]],
    
           [[ 10.53,   4.9 ,  10.77, ..., 267.39, 266.7 , 256.86],
            [  9.55,   4.17,   9.58, ..., 266.77, 266.03, 256.3 ],
            [ 13.46,   8.51,  14.84, ..., 267.71, 266.98, 257.  ],
            ...,
            [ 24.1 ,  21.92,  22.98, ..., 261.38, 261.12, 254.05],
            [ 24.96,  21.15,  23.22, ..., 261.13, 260.87, 253.9 ],
            [ 24.59,  19.2 ,  22.86, ..., 261.25, 261.04, 253.9 ]],
    
           [[  9.79,   6.03,   8.86, ..., 266.73, 266.03, 256.3 ],
            [  9.67,   5.49,   9.22, ..., 264.01, 263.22, 254.05],
            [ 10.04,   6.09,  11.73, ..., 263.81, 263.02, 254.05],
            ...,
            [ 24.71,  22.71,  23.93, ..., 261.25, 261.  , 254.05],
            [ 23.86,  20.73,  22.14, ..., 261.21, 261.  , 253.9 ],
            [ 23.25,  18.7 ,  22.38, ..., 261.25, 261.  , 254.05]]])

• Indexes: (2)
  • latitude
    PandasIndex

    PandasIndex(Index([46.546077728271484, 46.509700775146484, 46.473323822021484,
            46.43694305419922,  46.40056610107422,  46.36418914794922,
            46.32780838012695,  46.29143142700195,  46.25505065917969,
            46.21867370605469,
           ...
           37.597007751464844, 37.560630798339844, 37.524253845214844,
            37.48787307739258,  37.45149612426758,  37.41511535644531,
            37.37873840332031,  37.34236145019531,  37.30598068237305,
            37.26960372924805],
          dtype='float32', name='latitude', length=256))

  • longitude
    PandasIndex

    PandasIndex(Index([ -77.7586898803711,  -77.7223129272461,  -77.6859359741211,
            -77.6495590209961, -77.61317443847656, -77.57679748535156,
           -77.54042053222656, -77.50404357910156, -77.46766662597656,
           -77.43128204345703,
           ...
           -68.80962371826172, -68.77323913574219, -68.73686218261719,
           -68.70048522949219, -68.66410827636719, -68.62773132324219,
           -68.59134674072266, -68.55496978759766, -68.51859283447266,
           -68.48221588134766],
          dtype='float32', name='longitude', length=256))

• Attributes: (0)

Geo-IR Observations

The Geo IR contain globally merged observations from the IR window channels around \(11 \mu m\). The multi-timestep files containg 16 half-hourly timesteps centered on the overpass time.

import xarray as xr
from satrain.data import get_files
training_files = get_files("gmi", split="training", input_data=["geo_ir"], geometry="gridded", subset="xs")
# Load the first Geo-IR training data file.
xr.load_dataset(training_files["geo_ir"][0])

<xarray.Dataset> Size: 1MB
Dimensions:       (latitude: 256, longitude: 256)
Coordinates:
  * latitude      (latitude) float64 2kB 46.55 46.51 46.47 ... 37.34 37.31 37.27
  * longitude     (longitude) float64 2kB -77.76 -77.72 -77.69 ... -68.52 -68.48
    time          (latitude, longitude) datetime64[ns] 524kB 2018-01-07T20:00...
Data variables:
    observations  (latitude, longitude) float64 524kB 249.0 248.0 ... 263.0
Attributes:
    BeginDate:       2018-01-07
    BeginTime:       16:00:00.000Z
    EndDate:         2018-01-07
    EndTime:         16:59:59.999Z
    FileHeader:      StartGranuleDateTime=2018-01-07T16:00:00.000Z;\nStopGran...
    InputPointer:    merg_2018010716_4km-pixel
    title:           NCEP/CPC 4km Global (60N - 60S) IR Dataset
    ProductionTime:  2018-01-09T00:19:23.000Z

xarray.Dataset

• Dimensions:
  • latitude: 256
  • longitude: 256
• Coordinates: (3)
  • latitude
    (latitude)
    float64
    46.55 46.51 46.47 ... 37.31 37.27

    units :

    degrees_north

    standard_name :

    latitude

    array([46.546079, 46.509701, 46.473323, ..., 37.342361, 37.305982, 37.269604])

  • longitude
    (longitude)
    float64
    -77.76 -77.72 ... -68.52 -68.48

    units :

    degrees_east

    standard_name :

    longitude

    array([-77.75869 , -77.722312, -77.685934, ..., -68.554972, -68.518593,
           -68.482215])

  • time
    (latitude, longitude)
    datetime64[ns]
    2018-01-07T20:00:00 ... 2018-01-...

    standard_name :

    time

    array([['2018-01-07T20:00:00.000000000', '2018-01-07T20:00:00.000000000',
            '2018-01-07T20:00:00.000000000', ...,
            '2018-01-07T20:00:00.000000000', '2018-01-07T20:00:00.000000000',
            '2018-01-07T20:00:00.000000000'],
           ['2018-01-07T20:00:00.000000000', '2018-01-07T20:00:00.000000000',
            '2018-01-07T20:00:00.000000000', ...,
            '2018-01-07T20:00:00.000000000', '2018-01-07T20:00:00.000000000',
            '2018-01-07T20:00:00.000000000'],
           ['2018-01-07T20:00:00.000000000', '2018-01-07T20:00:00.000000000',
            '2018-01-07T20:00:00.000000000', ...,
            '2018-01-07T20:00:00.000000000', '2018-01-07T20:00:00.000000000',
            '2018-01-07T20:00:00.000000000'],
           ...,
           ['2018-01-07T19:30:00.000000000', '2018-01-07T19:30:00.000000000',
            '2018-01-07T19:30:00.000000000', ...,
            '2018-01-07T19:30:00.000000000', '2018-01-07T19:30:00.000000000',
            '2018-01-07T19:30:00.000000000'],
           ['2018-01-07T19:30:00.000000000', '2018-01-07T19:30:00.000000000',
            '2018-01-07T19:30:00.000000000', ...,
            '2018-01-07T19:30:00.000000000', '2018-01-07T19:30:00.000000000',
            '2018-01-07T19:30:00.000000000'],
           ['2018-01-07T19:30:00.000000000', '2018-01-07T19:30:00.000000000',
            '2018-01-07T19:30:00.000000000', ...,
            '2018-01-07T19:30:00.000000000', '2018-01-07T19:30:00.000000000',
            '2018-01-07T19:30:00.000000000']], dtype='datetime64[ns]')

• Data variables: (1)
  • observations
    (latitude, longitude)
    float64
    249.0 248.0 247.0 ... 263.0 263.0

    units :

    K

    standard_name :

    brightness_temperature

    array([[249., 248., 247., ..., 244., 244., 244.],
           [249., 248., 248., ..., 244., 244., 244.],
           [246., 246., 247., ..., 244., 244., 244.],
           ...,
           [270., 270., 266., ..., 263., 263., 263.],
           [270., 270., 270., ..., 263., 263., 263.],
           [270., 269., 269., ..., 263., 263., 263.]])

• Indexes: (2)
  • latitude
    PandasIndex

    PandasIndex(Index([46.546079223928864,  46.50970088924819,  46.47332255456751,
           46.436944219886826,  46.40056588520615,  46.36418755052547,
           46.327809215844795,  46.29143088116411, 46.255052546483434,
            46.21867421180276,
           ...
            37.59700889248182,  37.56063055780113, 37.524252223120456,
            37.48787388843978,   37.4514955537591, 37.415117219078425,
            37.37873888439774, 37.342360549717064,  37.30598221503638,
             37.2696038803557],
          dtype='float64', name='latitude', length=256))

  • longitude
    PandasIndex

    PandasIndex(Index([-77.75869037995149, -77.72231204527081, -77.68593371059013,
           -77.64955537590946, -77.61317704122877, -77.57679870654809,
           -77.54042037186741, -77.50404203718674, -77.46766370250606,
           -77.43128536782538,
           ...
           -68.80962004850444, -68.77324171382375, -68.73686337914307,
            -68.7004850444624, -68.66410670978172, -68.62772837510104,
           -68.59135004042037, -68.55497170573969, -68.51859337105901,
           -68.48221503637834],
          dtype='float64', name='longitude', length=256))

• Attributes: (8)

  BeginDate :

  2018-01-07

  BeginTime :

  16:00:00.000Z

  EndDate :

  2018-01-07

  EndTime :

  16:59:59.999Z

  FileHeader :

  StartGranuleDateTime=2018-01-07T16:00:00.000Z; StopGranuleDateTime=2018-01-07T16:59:59.999Z

  InputPointer :

  merg_2018010716_4km-pixel

  title :

  NCEP/CPC 4km Global (60N - 60S) IR Dataset

  ProductionTime :

  2018-01-09T00:19:23.000Z

Ancillary data

The ancillary data files contain the ERA5 data, GPROF surface type, and surface elevation all in separate variables.

import xarray as xr
from satrain.data import get_files
training_files = get_files("gmi", split="training", input_data=["ancillary"], geometry="gridded", subset="xs")
# Load the first ancillary training data file.
xr.load_dataset(training_files["ancillary"][0])

<xarray.Dataset> Size: 6MB
Dimensions:                          (latitude: 256, longitude: 256)
Coordinates:
  * latitude                         (latitude) float32 1kB 46.55 ... 37.27
  * longitude                        (longitude) float32 1kB -77.76 ... -68.48
    time                             (latitude, longitude) datetime64[ns] 524kB ...
Data variables: (12/19)
    ten_meter_wind_u                 (latitude, longitude) float32 262kB 1.49...
    ten_meter_wind_v                 (latitude, longitude) float32 262kB 5.10...
    two_meter_dew_point              (latitude, longitude) float32 262kB 255....
    two_meter_temperature            (latitude, longitude) float32 262kB 258....
    cape                             (latitude, longitude) float32 262kB 0.46...
    sea_ice_concentration            (latitude, longitude) float32 262kB nan ...
    ...                               ...
    total_column_water_vapor         (latitude, longitude) float32 262kB 7.33...
    total_precipitation              (latitude, longitude) float32 262kB 0.00...
    convective_precipitation         (latitude, longitude) float32 262kB 0.0 ...
    leaf_area_index                  (latitude, longitude) float32 262kB 1.54...
    surface_type                     (latitude, longitude) float32 262kB 11.0...
    elevation                        (latitude, longitude) float32 262kB 308....
Attributes:
    Conventions:  CF-1.6
    history:      2021-03-16 13:19:36 GMT by grib_to_netcdf-2.16.0: /opt/ecmw...
    scan_start:   1009
    scan_end:     1413

xarray.Dataset

• Dimensions:
  • latitude: 256
  • longitude: 256
• Coordinates: (3)
  • latitude
    (latitude)
    float32
    46.55 46.51 46.47 ... 37.31 37.27

    array([46.546078, 46.5097  , 46.473324, ..., 37.34236 , 37.30598 , 37.269604],
          dtype=float32)

  • longitude
    (longitude)
    float32
    -77.76 -77.72 ... -68.52 -68.48

    array([-77.75869 , -77.72231 , -77.685936, ..., -68.55497 , -68.51859 ,
           -68.482216], dtype=float32)

  • time
    (latitude, longitude)
    datetime64[ns]
    2018-01-07T19:48:00 ... 2018-01-...

    array([['2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000', ...,
            '2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000'],
           ['2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000', ...,
            '2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000'],
           ['2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000', ...,
            '2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000'],
           ...,
           ['2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000', ...,
            '2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000'],
           ['2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000', ...,
            '2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000'],
           ['2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000', ...,
            '2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000']], dtype='datetime64[ns]')

• Data variables: (19)
  • ten_meter_wind_u
    (latitude, longitude)
    float32
    1.49 1.474 1.457 ... 3.124 3.142

    units :

    m s**-1

    long_name :

    10 metre U wind component

    array([[1.4903749 , 1.4738501 , 1.4573252 , ..., 4.6529317 , 4.6129146 ,
            4.5728974 ],
           [1.4307564 , 1.410248  , 1.3897396 , ..., 4.640805  , 4.6032763 ,
            4.5657477 ],
           [1.3711379 , 1.346646  , 1.322154  , ..., 4.628678  , 4.593638  ,
            4.5585985 ],
           ...,
           [0.44747722, 0.4369765 , 0.42647573, ..., 3.1326468 , 3.1475084 ,
            3.16237   ],
           [0.44330314, 0.4324879 , 0.4216727 , ..., 3.1191344 , 3.1357212 ,
            3.1523082 ],
           [0.43912902, 0.42799935, 0.41686967, ..., 3.1056218 , 3.1239343 ,
            3.1422465 ]], dtype=float32)

  • ten_meter_wind_v
    (latitude, longitude)
    float32
    5.108 5.087 5.067 ... -7.923 -7.939

    units :

    m s**-1

    long_name :

    10 metre V wind component

    array([[ 5.1075287 ,  5.087323  ,  5.067118  , ...,  0.52380145,
             0.51866394,  0.5135265 ],
           [ 5.0133553 ,  4.9975767 ,  4.981798  , ...,  0.45077264,
             0.44381022,  0.43684784],
           [ 4.9191823 ,  4.90783   ,  4.8964787 , ...,  0.3777438 ,
             0.36895648,  0.36016917],
           ...,
           [ 1.9375613 ,  1.9179219 ,  1.8982826 , ..., -7.9780025 ,
            -7.997375  , -8.016747  ],
           [ 1.9097931 ,  1.8898349 ,  1.8698767 , ..., -7.9426637 ,
            -7.960365  , -7.9780664 ],
           [ 1.8820251 ,  1.861748  ,  1.8414708 , ..., -7.907325  ,
            -7.9233546 , -7.9393845 ]], dtype=float32)

  • two_meter_dew_point
    (latitude, longitude)
    float32
    255.6 255.5 255.5 ... 272.6 272.6

    units :

    K

    long_name :

    2 metre dewpoint temperature

    array([[255.56487, 255.52306, 255.48125, ..., 248.7918 , 248.81383,
            248.83585],
           [255.55246, 255.5154 , 255.47833, ..., 248.78279, 248.80441,
            248.82603],
           [255.54005, 255.50774, 255.4754 , ..., 248.7738 , 248.79501,
            248.81622],
           ...,
           [251.84477, 251.92336, 252.00195, ..., 272.5687 , 272.5698 ,
            272.57086],
           [251.85397, 251.93425, 252.01454, ..., 272.58585, 272.58936,
            272.5929 ],
           [251.86317, 251.94516, 252.02715, ..., 272.60303, 272.60895,
            272.6149 ]], dtype=float32)

  • two_meter_temperature
    (latitude, longitude)
    float32
    258.9 258.9 258.8 ... 277.8 277.9

    units :

    K

    long_name :

    2 metre temperature

    array([[258.9192 , 258.87057, 258.82193, ..., 255.81094, 255.85866,
            255.90636],
           [258.96683, 258.9203 , 258.87372, ..., 255.81871, 255.86774,
            255.91676],
           [259.01447, 258.96997, 258.9255 , ..., 255.82648, 255.87682,
            255.92715],
           ...,
           [269.2886 , 269.2649 , 269.2412 , ..., 277.64923, 277.6813 ,
            277.7134 ],
           [269.30872, 269.28546, 269.26218, ..., 277.72787, 277.75888,
            277.7899 ],
           [269.32886, 269.306  , 269.28317, ..., 277.8065 , 277.83643,
            277.86633]], dtype=float32)

  • cape
    (latitude, longitude)
    float32
    0.4607 0.4508 ... 12.16 12.18

    units :

    J kg**-1

    long_name :

    Convective available potential energy

    array([[ 0.46073768,  0.45081612,  0.44089457, ...,  0.        ,
             0.        ,  0.        ],
           [ 0.49205357,  0.47957984,  0.46710613, ...,  0.        ,
             0.        ,  0.        ],
           [ 0.52336943,  0.5083436 ,  0.49331766, ...,  0.        ,
             0.        ,  0.        ],
           ...,
           [ 0.        ,  0.        ,  0.        , ..., 12.7323885 ,
            12.711081  , 12.689773  ],
           [ 0.        ,  0.        ,  0.        , ..., 12.438542  ,
            12.435477  , 12.432413  ],
           [ 0.        ,  0.        ,  0.        , ..., 12.144696  ,
            12.159875  , 12.175053  ]], dtype=float32)

  • sea_ice_concentration
    (latitude, longitude)
    float32
    nan nan nan nan ... 0.0 0.0 0.0 0.0

    units :

    (0 - 1)

    long_name :

    Sea ice area fraction

    standard_name :

    sea_ice_area_fraction

    array([[nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           [nan, nan, nan, ..., nan, nan, nan],
           ...,
           [nan, nan, nan, ...,  0.,  0.,  0.],
           [nan, nan, nan, ...,  0.,  0.,  0.],
           [nan, nan, nan, ...,  0.,  0.,  0.]], dtype=float32)

  • sea_surface_temperature
    (latitude, longitude)
    float32
    nan nan nan ... 293.5 293.5 293.5

    units :

    K

    long_name :

    Sea surface temperature

    array([[      nan,       nan,       nan, ...,       nan,       nan,
                  nan],
           [      nan,       nan,       nan, ...,       nan,       nan,
                  nan],
           [      nan,       nan,       nan, ...,       nan,       nan,
                  nan],
           ...,
           [      nan,       nan,       nan, ..., 293.6309 , 293.6195 ,
            293.60812],
           [      nan,       nan,       nan, ..., 293.5862 , 293.58044,
            293.5747 ],
           [      nan,       nan,       nan, ..., 293.54153, 293.5414 ,
            293.5413 ]], dtype=float32)

  • skin_temperature
    (latitude, longitude)
    float32
    259.0 258.9 258.9 ... 292.9 292.9

    units :

    K

    long_name :

    Skin temperature

    array([[258.9801 , 258.92432, 258.86853, ..., 254.70152, 254.73746,
            254.77339],
           [259.02106, 258.96698, 258.91293, ..., 254.70712, 254.74692,
            254.78673],
           [259.06198, 259.00967, 258.95734, ..., 254.71272, 254.7564 ,
            254.80005],
           ...,
           [270.35883, 270.30212, 270.24545, ..., 293.01367, 293.00375,
            292.99384],
           [270.3741 , 270.31766, 270.2612 , ..., 292.9715 , 292.96698,
            292.96246],
           [270.38943, 270.3332 , 270.27692, ..., 292.92932, 292.9302 ,
            292.9311 ]], dtype=float32)

  • snow_depth
    (latitude, longitude)
    float32
    0.04219 0.04199 0.04179 ... 0.0 0.0

    units :

    m of water equivalent

    long_name :

    Snow depth

    standard_name :

    lwe_thickness_of_surface_snow_amount

    array([[0.04218891, 0.04198768, 0.04178645, ..., 0.09443753, 0.09542508,
            0.09641261],
           [0.04164097, 0.04144382, 0.04124667, ..., 0.09420027, 0.09521844,
            0.09623662],
           [0.04109303, 0.04089997, 0.0407069 , ..., 0.093963  , 0.09501181,
            0.09606062],
           ...,
           [0.00272006, 0.00284704, 0.00297401, ..., 0.        , 0.        ,
            0.        ],
           [0.00276712, 0.00289597, 0.00302482, ..., 0.        , 0.        ,
            0.        ],
           [0.00281418, 0.0029449 , 0.00307562, ..., 0.        , 0.        ,
            0.        ]], dtype=float32)

  • snowfall
    (latitude, longitude)
    float32
    0.0005889 0.0006067 ... 2.673e-07

    units :

    m of water equivalent

    long_name :

    Snowfall

    standard_name :

    lwe_thickness_of_snowfall_amount

    array([[5.8894278e-04, 6.0666859e-04, 6.2439439e-04, ..., 3.3597830e-06,
            3.1177374e-06, 2.8756917e-06],
           [5.7953177e-04, 5.9796119e-04, 6.1639067e-04, ..., 3.3233880e-06,
            3.0802341e-06, 2.8370805e-06],
           [5.7012070e-04, 5.8925379e-04, 6.0838688e-04, ..., 3.2869928e-06,
            3.0427309e-06, 2.7984690e-06],
           ...,
           [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 6.1250608e-07,
            5.0712180e-07, 4.0173759e-07],
           [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 5.0999461e-07,
            4.2224787e-07, 3.3450118e-07],
           [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 4.0748310e-07,
            3.3737393e-07, 2.6726477e-07]], dtype=float32)

  • surface_pressure
    (latitude, longitude)
    float32
    9.73e+04 9.73e+04 ... 1.031e+05

    units :

    Pa

    long_name :

    Surface pressure

    standard_name :

    surface_air_pressure

    array([[ 97296.77 ,  97301.64 ,  97306.516, ...,  98902.55 ,  98975.055,
             99047.56 ],
           [ 97368.984,  97376.18 ,  97383.375, ...,  98915.12 ,  98983.76 ,
             99052.4  ],
           [ 97441.195,  97450.71 ,  97460.234, ...,  98927.69 ,  98992.46 ,
             99057.24 ],
           ...,
           [102455.3  , 102501.77 , 102548.25 , ..., 103076.65 , 103073.42 ,
            103070.19 ],
           [102455.77 , 102502.52 , 102549.28 , ..., 103074.87 , 103071.695,
            103068.52 ],
           [102456.25 , 102503.27 , 102550.305, ..., 103073.09 , 103069.98 ,
            103066.86 ]], dtype=float32)

  • total_column_cloud_ice_water
    (latitude, longitude)
    float32
    0.2347 0.2396 ... 0.01566 0.01576

    units :

    kg m**-2

    long_name :

    Total column cloud ice water

    array([[0.2347124 , 0.2396043 , 0.24449618, ..., 0.03289587, 0.0312475 ,
            0.02959914],
           [0.23805223, 0.24319023, 0.24832822, ..., 0.03380498, 0.03214161,
            0.03047824],
           [0.24139206, 0.24677616, 0.25216025, ..., 0.03471408, 0.03303571,
            0.03135733],
           ...,
           [0.00360507, 0.00372531, 0.00384555, ..., 0.0152003 , 0.01527647,
            0.01535265],
           [0.00330729, 0.00342412, 0.00354095, ..., 0.01538466, 0.01546951,
            0.01555437],
           [0.00300952, 0.00312294, 0.00323635, ..., 0.01556902, 0.01566255,
            0.01575608]], dtype=float32)

  • total_column_cloud_liquid_water
    (latitude, longitude)
    float32
    0.2272 0.2214 ... 0.2938 0.2937

    units :

    kg m**-2

    long_name :

    Total column cloud liquid water

    array([[0.22718571, 0.22143298, 0.21568026, ..., 0.        , 0.        ,
            0.        ],
           [0.23732325, 0.22977464, 0.22222601, ..., 0.        , 0.        ,
            0.        ],
           [0.2474608 , 0.2381163 , 0.22877178, ..., 0.        , 0.        ,
            0.        ],
           ...,
           [0.        , 0.        , 0.        , ..., 0.28595683, 0.28552046,
            0.2850841 ],
           [0.        , 0.        , 0.        , ..., 0.28991482, 0.28965804,
            0.28940123],
           [0.        , 0.        , 0.        , ..., 0.2938728 , 0.2937956 ,
            0.2937184 ]], dtype=float32)

  • total_column_water_vapor
    (latitude, longitude)
    float32
    7.333 7.297 7.261 ... 5.92 5.922

    units :

    kg m**-2

    long_name :

    Total column water vapour

    standard_name :

    lwe_thickness_of_atmosphere_mass_content_of_water_vapor

    array([[7.333405 , 7.2973294, 7.261254 , ..., 2.967647 , 2.9610486,
            2.9544501],
           [7.353893 , 7.3174744, 7.2810564, ..., 2.9699013, 2.9631584,
            2.9564152],
           [7.37438  , 7.3376193, 7.3008585, ..., 2.9721558, 2.965268 ,
            2.95838  ],
           ...,
           [4.5616865, 4.5692744, 4.576863 , ..., 5.8369055, 5.8377986,
            5.8386917],
           [4.5635014, 4.5709004, 4.578299 , ..., 5.8774586, 5.878894 ,
            5.880329 ],
           [4.5653167, 4.572526 , 4.5797358, ..., 5.9180117, 5.919989 ,
            5.921966 ]], dtype=float32)

  • total_precipitation
    (latitude, longitude)
    float32
    0.0005891 0.0006068 ... 3.879e-05

    units :

    m

    long_name :

    Total precipitation

    array([[5.8905850e-04, 6.0679228e-04, 6.2452612e-04, ..., 3.2374021e-06,
            2.9729404e-06, 2.7084786e-06],
           [5.7962496e-04, 5.9804652e-04, 6.1646808e-04, ..., 3.2155401e-06,
            2.9396870e-06, 2.6638336e-06],
           [5.7019142e-04, 5.8930070e-04, 6.0841005e-04, ..., 3.1936784e-06,
            2.9064336e-06, 2.6191888e-06],
           ...,
           [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 2.9608609e-05,
            2.9645447e-05, 2.9682285e-05],
           [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 3.3324770e-05,
            3.3781675e-05, 3.4238576e-05],
           [0.0000000e+00, 0.0000000e+00, 0.0000000e+00, ..., 3.7040936e-05,
            3.7917900e-05, 3.8794864e-05]], dtype=float32)

  • convective_precipitation
    (latitude, longitude)
    float32
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    units :

    m

    long_name :

    Convective precipitation

    standard_name :

    lwe_thickness_of_convective_precipitation_amount

    array([[0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           ...,
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)

  • leaf_area_index
    (latitude, longitude)
    float32
    1.545 1.542 1.54 ... 0.0 0.0 0.0

    array([[1.5454112, 1.5424844, 1.5395576, ..., 2.8880725, 3.0771809,
            3.2662895],
           [1.5558375, 1.5521992, 1.548561 , ..., 2.7712657, 2.9641995,
            3.1571336],
           [1.5662639, 1.5619141, 1.5575644, ..., 2.654459 , 2.8512185,
            3.0479777],
           ...,
           [3.8703792, 4.1055512, 4.3407235, ..., 0.       , 0.       ,
            0.       ],
           [3.9266117, 4.157391 , 4.3881707, ..., 0.       , 0.       ,
            0.       ],
           [3.982844 , 4.209231 , 4.435618 , ..., 0.       , 0.       ,
            0.       ]], dtype=float32)

  • surface_type
    (latitude, longitude)
    float32
    11.0 11.0 11.0 11.0 ... 1.0 1.0 1.0

    array([[11., 11., 11., ..., 18., 11., 11.],
           [11., 11., 11., ..., 18., 11., 11.],
           [11., 11., 11., ..., 18., 11., 11.],
           ...,
           [10., 10., 10., ...,  1.,  1.,  1.],
           [10., 10., 10., ...,  1.,  1.,  1.],
           [10., 10., 10., ...,  1.,  1.,  1.]], dtype=float32)

  • elevation
    (latitude, longitude)
    float32
    308.4 365.7 357.8 ... -500.0 -500.0

    array([[ 308.37634 ,  365.65036 ,  357.82312 , ...,  246.79338 ,
             208.18872 ,  224.18974 ],
           [ 294.5039  ,  366.42844 ,  348.88458 , ...,  230.91454 ,
             220.35715 ,  228.0542  ],
           [ 270.35638 ,  317.7898  ,  360.50027 , ...,  259.60086 ,
             203.0638  ,  204.8569  ],
           ...,
           [  73.702805,   73.20642 ,   89.54027 , ..., -500.      ,
            -500.      , -500.      ],
           [  84.63796 ,   74.7578  ,   57.481377, ..., -500.      ,
            -500.      , -500.      ],
           [  54.902924,   75.34764 ,   66.292336, ..., -500.      ,
            -500.      , -500.      ]], dtype=float32)

• Indexes: (2)
  • latitude
    PandasIndex

    PandasIndex(Index([46.546077728271484, 46.509700775146484, 46.473323822021484,
            46.43694305419922,  46.40056610107422,  46.36418914794922,
            46.32780838012695,  46.29143142700195,  46.25505065917969,
            46.21867370605469,
           ...
           37.597007751464844, 37.560630798339844, 37.524253845214844,
            37.48787307739258,  37.45149612426758,  37.41511535644531,
            37.37873840332031,  37.34236145019531,  37.30598068237305,
            37.26960372924805],
          dtype='float32', name='latitude', length=256))

  • longitude
    PandasIndex

    PandasIndex(Index([ -77.7586898803711,  -77.7223129272461,  -77.6859359741211,
            -77.6495590209961, -77.61317443847656, -77.57679748535156,
           -77.54042053222656, -77.50404357910156, -77.46766662597656,
           -77.43128204345703,
           ...
           -68.80962371826172, -68.77323913574219, -68.73686218261719,
           -68.70048522949219, -68.66410827636719, -68.62773132324219,
           -68.59134674072266, -68.55496978759766, -68.51859283447266,
           -68.48221588134766],
          dtype='float32', name='longitude', length=256))

• Attributes: (4)

  Conventions :

  CF-1.6

  history :

  2021-03-16 13:19:36 GMT by grib_to_netcdf-2.16.0: /opt/ecmwf/eccodes/bin/grib_to_netcdf -S param -o /cache/data2/adaptor.mars.internal-1615900733.829179-28806-27-384385e8-9820-4dda-aaba-bfabf09f8b33.nc /cache/tmp/384385e8-9820-4dda-aaba-bfabf09f8b33-adaptor.mars.internal-1615900733.8296843-28806-11-tmp.grib

  scan_start :

  1009

  scan_end :

  1413

Target data

Finally, the target files contain the precipitation estimates as well as the radar-quality index (RQI), the gauge-correction factor (GCF), and the precipitation type fraction.

import xarray as xr
from satrain.data import get_files
training_files = get_files("gmi", split="training", input_data=[], geometry="gridded", subset="xs") # Target data is always included
# Load the first ancillary training data file.
xr.load_dataset(training_files["target"][0])

<xarray.Dataset> Size: 5MB
Dimensions:                  (latitude: 256, longitude: 256)
Coordinates:
  * latitude                 (latitude) float64 2kB 46.55 46.51 ... 37.31 37.27
  * longitude                (longitude) float64 2kB -77.76 -77.72 ... -68.48
Data variables:
    surface_precip           (latitude, longitude) float64 524kB 0.91 ... nan
    radar_quality_index      (latitude, longitude) float64 524kB 0.0 0.0 ... nan
    gauge_correction_factor  (latitude, longitude) float32 262kB 0.9156 ... nan
    valid_fraction           (latitude, longitude) float64 524kB 1.0 1.0 ... 0.0
    precip_fraction          (latitude, longitude) float64 524kB 1.0 1.0 ... nan
    snow_fraction            (latitude, longitude) float64 524kB 1.0 1.0 ... nan
    hail_fraction            (latitude, longitude) float64 524kB 0.0 0.0 ... nan
    convective_fraction      (latitude, longitude) float64 524kB 0.0 0.0 ... nan
    stratiform_fraction      (latitude, longitude) float64 524kB 0.0 0.0 ... nan
    time                     (latitude, longitude) datetime64[ns] 524kB 2018-...
Attributes:
    lower_left_col:  2419
    lower_left_row:  1046

xarray.Dataset

• Dimensions:
  • latitude: 256
  • longitude: 256
• Coordinates: (2)
  • latitude
    (latitude)
    float64
    46.55 46.51 46.47 ... 37.31 37.27

    array([46.546079, 46.509701, 46.473323, ..., 37.342361, 37.305982, 37.269604])

  • longitude
    (longitude)
    float64
    -77.76 -77.72 ... -68.52 -68.48

    array([-77.75869 , -77.722312, -77.685934, ..., -68.554972, -68.518593,
           -68.482215])

• Data variables: (10)
  • surface_precip
    (latitude, longitude)
    float64
    0.91 1.09 1.25 1.39 ... nan nan nan

    array([[0.91, 1.09, 1.25, ..., 0.  , 0.  , 0.  ],
           [1.16, 1.44, 1.75, ..., 0.  , 0.  , 0.  ],
           [1.23, 1.41, 1.54, ..., 0.  , 0.  , 0.  ],
           ...,
           [0.  , 0.  , 0.  , ...,  nan,  nan,  nan],
           [0.  , 0.  , 0.  , ...,  nan,  nan,  nan],
           [0.  , 0.  , 0.  , ...,  nan,  nan,  nan]])

  • radar_quality_index
    (latitude, longitude)
    float64
    0.0 0.0 0.0 0.0 ... nan nan nan nan

    array([[0.        , 0.        , 0.        , ..., 0.53543307, 0.55905512,
            0.57086614],
           [0.        , 0.        , 0.        , ..., 0.5       , 0.5       ,
            0.50393701],
           [0.        , 0.        , 0.        , ..., 0.47244094, 0.48425197,
            0.49212598],
           ...,
           [0.59448819, 0.5984252 , 0.60629921, ...,        nan,        nan,
                   nan],
           [0.5984252 , 0.6023622 , 0.64566929, ...,        nan,        nan,
                   nan],
           [0.6023622 , 0.62204724, 0.69291339, ...,        nan,        nan,
                   nan]])

  • gauge_correction_factor
    (latitude, longitude)
    float32
    0.9156 0.9341 0.9509 ... nan nan

    array([[0.9155638 , 0.9341001 , 0.9509074 , ..., 1.        , 1.        ,
            1.        ],
           [0.92000943, 0.93374383, 0.93910277, ..., 1.        , 1.        ,
            1.        ],
           [0.90631545, 0.92085993, 0.93365943, ..., 0.99999994, 0.99999994,
            0.99999994],
           ...,
           [1.        , 1.        , 1.        , ...,        nan,        nan,
                   nan],
           [1.        , 1.        , 1.        , ...,        nan,        nan,
                   nan],
           [1.        , 1.0000001 , 1.        , ...,        nan,        nan,
                   nan]], dtype=float32)

  • valid_fraction
    (latitude, longitude)
    float64
    1.0 1.0 1.0 1.0 ... 0.0 0.0 0.0 0.0

    array([[1., 1., 1., ..., 1., 1., 1.],
           [1., 1., 1., ..., 1., 1., 1.],
           [1., 1., 1., ..., 1., 1., 1.],
           ...,
           [1., 1., 1., ..., 0., 0., 0.],
           [1., 1., 1., ..., 0., 0., 0.],
           [1., 1., 1., ..., 0., 0., 0.]])

  • precip_fraction
    (latitude, longitude)
    float64
    1.0 1.0 1.0 1.0 ... nan nan nan nan

    array([[1.        , 1.        , 1.        , ..., 0.        , 0.        ,
            0.        ],
           [1.        , 1.        , 1.        , ..., 0.00393701, 0.        ,
            0.        ],
           [1.        , 1.        , 1.        , ..., 0.        , 0.        ,
            0.        ],
           ...,
           [0.        , 0.        , 0.        , ...,        nan,        nan,
                   nan],
           [0.        , 0.        , 0.        , ...,        nan,        nan,
                   nan],
           [0.        , 0.        , 0.        , ...,        nan,        nan,
                   nan]])

  • snow_fraction
    (latitude, longitude)
    float64
    1.0 1.0 1.0 1.0 ... nan nan nan nan

    array([[1.        , 1.        , 1.        , ..., 0.        , 0.        ,
            0.        ],
           [1.        , 1.        , 1.        , ..., 0.00393701, 0.        ,
            0.        ],
           [1.        , 1.        , 1.        , ..., 0.        , 0.        ,
            0.        ],
           ...,
           [0.        , 0.        , 0.        , ...,        nan,        nan,
                   nan],
           [0.        , 0.        , 0.        , ...,        nan,        nan,
                   nan],
           [0.        , 0.        , 0.        , ...,        nan,        nan,
                   nan]])

  • hail_fraction
    (latitude, longitude)
    float64
    0.0 0.0 0.0 0.0 ... nan nan nan nan

    array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ...,
           [ 0.,  0.,  0., ..., nan, nan, nan],
           [ 0.,  0.,  0., ..., nan, nan, nan],
           [ 0.,  0.,  0., ..., nan, nan, nan]])

  • convective_fraction
    (latitude, longitude)
    float64
    0.0 0.0 0.0 0.0 ... nan nan nan nan

    array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ...,
           [ 0.,  0.,  0., ..., nan, nan, nan],
           [ 0.,  0.,  0., ..., nan, nan, nan],
           [ 0.,  0.,  0., ..., nan, nan, nan]])

  • stratiform_fraction
    (latitude, longitude)
    float64
    0.0 0.0 0.0 0.0 ... nan nan nan nan

    array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           [ 0.,  0.,  0., ...,  0.,  0.,  0.],
           ...,
           [ 0.,  0.,  0., ..., nan, nan, nan],
           [ 0.,  0.,  0., ..., nan, nan, nan],
           [ 0.,  0.,  0., ..., nan, nan, nan]])

  • time
    (latitude, longitude)
    datetime64[ns]
    2018-01-07T19:48:00 ... 2018-01-...

    array([['2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000', ...,
            '2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000'],
           ['2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000', ...,
            '2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000'],
           ['2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000', ...,
            '2018-01-07T19:48:00.000000000', '2018-01-07T19:48:00.000000000',
            '2018-01-07T19:48:00.000000000'],
           ...,
           ['2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000', ...,
            '2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000'],
           ['2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000', ...,
            '2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000'],
           ['2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000', ...,
            '2018-01-07T19:44:00.000000000', '2018-01-07T19:44:00.000000000',
            '2018-01-07T19:44:00.000000000']], dtype='datetime64[ns]')

• Indexes: (2)
  • latitude
    PandasIndex

    PandasIndex(Index([46.546079223928864,  46.50970088924819,  46.47332255456751,
           46.436944219886826,  46.40056588520615,  46.36418755052547,
           46.327809215844795,  46.29143088116411, 46.255052546483434,
            46.21867421180276,
           ...
            37.59700889248182,  37.56063055780113, 37.524252223120456,
            37.48787388843978,   37.4514955537591, 37.415117219078425,
            37.37873888439774, 37.342360549717064,  37.30598221503638,
             37.2696038803557],
          dtype='float64', name='latitude', length=256))

  • longitude
    PandasIndex

    PandasIndex(Index([-77.75869037995149, -77.72231204527081, -77.68593371059013,
           -77.64955537590946, -77.61317704122877, -77.57679870654809,
           -77.54042037186741, -77.50404203718674, -77.46766370250606,
           -77.43128536782538,
           ...
           -68.80962004850444, -68.77324171382375, -68.73686337914307,
            -68.7004850444624, -68.66410670978172, -68.62772837510104,
           -68.59135004042037, -68.55497170573969, -68.51859337105901,
           -68.48221503637834],
          dtype='float64', name='longitude', length=256))

• Attributes: (2)

  lower_left_col :

  2419

  lower_left_row :

  1046