Common submodule

This module provides routines that are used throughout the library.

rainforest.common.io_data : read various MCH data formats

rainforest.common.retrieve_data : retrieve MCH files (COSMO, polar, Cartesian) from storage

rainforest.common.radarprocessing : radar processing routines

rainforest.common.constants : define constants used throughout the code

rainforest.common.graphics : creating plots for QPE realization and evaluation

rainforest.common.lookup : computing and reading lookup tables that are used for interpolation and geolocalization of data

rainforest.common.lookup : various small utilities used throughout the code

rainforest.common.io_data module

Functions to read MeteoSwiss products

Daniel Wolfensberger MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch December 2019

Modified by Rebecca Gugerli MeteoSwiss/EPFL March 2022

rainforest.common.io_data.read_NWP_HZT_cart(cart_file)

Reads the HZT fields that are produced for operational use and returns a grid object

Parameters:: fileIn – path with filename
Returns:: Masked array of data
Return type:: data

rainforest.common.io_data.read_cart(cart_file)

Generic function that reads a Cartesian radar file, either in gif or binary format (ELDES or as generated by the qpe module of this library) and converts its data to floating numbers

Parameters:: cart_file (str) – Full path of the file to read as a a string
Return type:: The cartesian data in a numpy array

rainforest.common.io_data.read_gif(gif_file)

Reads a Cartesian radar file in gif format

TODO: Replace with function rad4alp_gif_readers.py from pyart

Parameters:: gif_file (str) – Full path of the file to read as a a string
Return type:: The cartesian data in a numpy array

rainforest.common.io_data.read_polar(polar_files, physic_value=True)

Reads a polar radar metranet file as a pyart radar instance, if multiple files corresponding to multiple elevations (sweeps) are provided they are merged into a single pyart instance

Parameters:

polar_files (str or list of strings) –

Can be either: - Full path of the file to read as a a string - string with wildcard pointing to the files to read,

e.g. …../MLA192711055.*

a list of full filepaths

Returns:

sweepnumbers (list) – list of sweeps numbers corresponding to all files that were read
merged (a pyart Radar instance)

rainforest.common.io_data.read_station_data(gauge_file)

Reads gauge data as given by the Climap software

Parameters:: gauge_file (str) – Full path of the file to read as a a string
Return type:: The cartesian data in a numpy array

rainforest.common.io_data.read_status(status_file, add_wet_radome=False)

Reads a radar xml status file

Parameters:

fname (str) – Full path of the status xml file to be read
add_wet_Radome (boolean (optional)) – For older files, there is not information about the wet radome. If this is true, the script will estimate the wet radome precipitation as a 3 x 3 mean of the RZC product at the given time (as is done for more recent files)

Return type:

The status as a python dict

rainforest.common.io_data.read_vpr(xml_file, radar=None)

Reads a vpr xml file and returns an interpolator that can be used to obtain the vpr correction at any altitude

Parameters:

vpr_file (str) –

Full path of the file to read radar : char (optional)

the radar is required to know the vpr reference height, it needs to be either ‘A’,’D’,’L’,’P’ or ‘W’, if not specified, it will be inferred from the vpr filename

Returns:

An interpolator that returns the vpr correction as a function of
the height, the correction is a multiplicative factor

rainforest.common.io_data.read_xls(xls_file)

Reads an excel file such as those used for CPC vlaidation

Parameters:: fname (str) – Full path of the excel file to be read
Return type:: The excel file as a Pandas dataframe

rainforest.common.io_data.save_gif(gif_file, precip)

Saves a precipitation map in float to a gif file (AQC format)

Parameters:

gif_file (str) – Full path of the file to write as a a string
precip (2D array) – 2D array containing the precipitation intensities

rainforest.common.retrieve_data module

Functions to retrieve MeteoSwiss products from the archives

Daniel Wolfensberger, Rebecca Gugerli MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch, rebecca.gugerli@epfl.ch December 2019, July 2022

rainforest.common.retrieve_data.retrieve_CPCCV(time, stations)

Retrieves cross-validation CPC data for a set of stations from the xls files prepared by Yanni

Parameters:

time (datetime.datetime instance) – starting time of the time range
stations (list of str) – list of weather stations at which to retrieve the CPC.CV data

Returns:

Three arrays with the variables CPC.CV, CPC and AQC at the given stations
extracted from the cross-validation files.

rainforest.common.retrieve_data.retrieve_hzt_RT(tstep)

Retrieves the preprocessed HZT products: A version adapted to real time implementation Only used in for the function retrieve_hzt_prod

Parameters:: tstep (datetime) – directory where to store the unzipped files
Return type:: A list containing all the filepaths of the retrieved files

rainforest.common.retrieve_data.retrieve_hzt_prod(folder_out, start_time, end_time, pattern_type='shell')

Retrieves the preprocessed HZT products from the CSCS repository for a specified time range, unzips them and places them in a specified folder

Parameters:

folder_out (str) – directory where to store the unzipped files
start_time (datetime.datetime instance) – starting time of the time range
end_time (datetime.datetime instance) – end time of the time range
pattern_type (either 'shell' or 'regex' (optional)) – use ‘shell’ for standard shell patterns, which use * as wildcard use ‘regex’ for more advanced regex patterns

Return type:

A list containing all the filepaths of the retrieved files

rainforest.common.retrieve_data.retrieve_prod(start_time, end_time, product_name, folder_out=None, pattern=None, pattern_type='shell', sweeps=None, hdf5=False)

Retrieves radar data from the CSCS repository for a specified time range, unzips them and places them in a specified folder

Parameters:

start_time (datetime.datetime instance) – starting time of the time range
end_time (datetime.datetime instance) – end time of the time range
product_name (str) – name of the product, as stored on CSCS, e.g. RZC, CPCH, MZC, BZC…
folder_out (str) – directory where to store the unzipped files, if set to None will read the file to memory
pattern (str) – pattern constraint on file names, can be used for products which contain multiple filetypes, f.ex CPCH folders contain both rda and gif files, if only gifs are wanted : file_type = ‘*.gif’
pattern_type (either 'shell' or 'regex' (optional)) – use ‘shell’ for standard shell patterns, which use * as wildcard use ‘regex’ for more advanced regex patterns
sweeps (list of int (optional)) – For polar products, specifies which sweeps (elevations) must be retrieved, if not specified all available sweeps will be retrieved
hdf5 (bool) – If True will retrieve the hdf5 files for the given product (beware hdf5 is not available for all products)

Return type:

A list containing all the filepaths of the retrieved files

rainforest.common.retrieve_data.retrieve_prod_RT(time, product_name, pattern=None, pattern_type='shell', sweeps=None)

Adapted function from rainforest.common.retrieve_data: Here, it reads the data per timestep, and in the real-time operation, the radar data is not zipped

Parameters:

time (datetime object) – timestamp to extract
product_name (string) – Name of the product to be extracted
sweeps (list) – List of sweeps if not all want to be extracted. Defaults to None.

Raises:

ValueError – If no data is found

Returns:

dictionary containing with the the file list

Return type:

dict

rainforest.common.radarprocessing module

Created on Mon Dec 16 17:38:52 2019 @author: wolfensb, gugerlir

rainforest.common.radarprocessing.HZT_cartesian_to_polar(hzt, radar, sweeps=range(0, 20))

Transform a Cartesian HZT object (height of freezing level) to a polar Radar object with the freezing level at every gate on the Rad4Alp domain

Parameters:

hzt (pyart Grid object) – Grid object that contains the hzt data, as obtained with pyart.aux_io.read_cartesian_metranet
radar (char) – name of the radar, ‘A’,’D’,’L’,’P’ or ‘W’
sweeps (list) – list of Rad4Alp sweeps, from 0 to 20 to include in the polar radar object

rainforest.common.radarprocessing.HZT_hourly_to_5min(time, filelist, tsteps_min=5)

Function to interpolate the hourly isothermal fields to 5min resolution to make them consistant with the radar fields

Parameters:

time (datetime object) – timestep to calculate
filelist (dictionnary) – list with timesteps, path and filename of HZT files only typically derived in database.retrieve_radar_data.Updater.retrieve_radar_files()
tsteps_min (int) – resolution of new fields in minutes

Return type:

dictionnary with datetime objects as keys and numpy.ma.core.MaskedArray (Cartesian coordinates)

class rainforest.common.radarprocessing.Radar(radname, polfiles, statusfile=None, vprfile=None, temp_ref='TAIR', metranet_reader='C')

Bases: object

A class that contains polar radar data and performs some pre-processing before adding that data to the database or computing the QPE product The different elevations are stored in a dictionary rather as in a single pyart radar instance as this was found to be faster in practice

Creates an Radar class instance

Parameters:

radname (char) – Name of the radar, either ‘A’,’D’,’L’,’P’ or ‘W’
polfiles (list of str) – List of full filepaths of the radar files for a given timestemp, one file for every elevation, typically obtained with the retrieve_prod function in the common submodule
statusfile (str (optional)) – Full path of the status file that corresponds to this particular radar and timestep, used to compute noise estimates
vprfile (str(optional)) – Full path of the vpr xml file that corresponds to this particular radar and timestep, used to compute VPR correction
metranet_reader (str(optional)) – Which reader to use to read the polar radar data, can be either ‘C’ or ‘python’

add_cosmo_data(cosmo_data)

Adds COSMO data to the radar instance

Parameters:

cosmo_data (dict) –

dict of COSMO data at polar coordinates obtained from the get_COSMO_variables function in the common submodule this dict must have the following format

dic[variable][sweep]

add_height_over_iso0(): Function to derive the relative height over the 0° isothermal altitude in meters by subtracting the altitude of the 0° isothermal altitude from the altitude of the radar gate

add_hzt_data(hzt_cart)

Transform a Cartesian HZT object (height of freezing level) and adds the fields to this radar object.

Parameters:: hzt (Masked array as output from HZT_hourly_to_5min) – from Grid object that contains the hzt data, as obtained with pyart.aux_io.read_cartesian_metranet(filelist[tstamp_hzt0]).fields[‘iso0_height’][‘data’][0]

compute_hydro(): Computes the hydrometeor classification using Nikola Besic’ algorithm, all necessary fields ZH, ZDR, RHOHV, KDP, T (COSMO) must be available

compute_kdp(dscfg)

Computes KDP using the simple moving least-square algorithm

Parameters:: dscfg (dict) – dictionary containing the following fields RMIN: RMAX: RWIND: ZMIN: ZMAX:

compute_noise(): Computes a noise estimate from a status file

correct_attenuation(): Corrects for attenuation using the ZPHI algorithm (Testud et al.) using the COSMO temperature to identify liquid precipitation OR using the 0° isothermal altitude

correct_gate_altitude(): Following the Swiss standard, we change the calculation of the gate_altitude that is automatically done in pyart. Instead of the constant radar scale factor of (ke) of 4/3, we now use 1.25

get_field(sweep, field_name)

Gets a radar variable at given elevation (sweep)

Parameters:

sweep (int) – Sweep number from 1 to 20
field_name (str) – name of the variable, e.g. ZH, ZDR, RHOHV, SW, …

snr_mask(snr_threshold)

Masks the radar data at low SNR

Parameters:: snr_threshold (float) – Minimal SNR to consider in dB

visib_mask(min_visib, max_visib_corr)

Masks the radar data at low visibility and corrects the reflectivity for visibility

Parameters:

min_visib (int) – Minimal visibility below which the data is masked
max_visib_corr (float) – Maximum visibility correction factor, the visibility correction is 100/VISIB (with VISIB in %) and can be thresholded with this parameter. This is usually set to 2 at MeteoSwiss

rainforest.common.radarprocessing.hydroClass_single(radars, zh, zdr, kdp, rhohv, temp, weights=array([1., 1., 1., 0.75, 0.5]))

Computes the hydrometeor classes for columnar data, note that all input arrays except weights must have the same length

Parameters:

radars (ndarray of char) – Array of radar IDs, (‘A’,’D’,’L’,’P’,’W’)
zh (ndarray) – Array of radar reflectivity in dBZ
zdr (ndarray) – Array of diff. reflectivity in dB
kdp (ndarray) – Array of specific phase shift on propagation in deg / km
rhohv (ndarray) – Array of copolar correlation coefficient
temp (ndarray) – Array of radar temperature in Celcius
weights (ndarray (optional)) – The weight of every input feature, zh, zdr, kdp, rhohv, temp in the hydrometeor classification

Returns:

The hydrometeor classes as ndarray with values from 0 to 8, corresponding to
the classes – 0 : no data 1 : aggregates (AG) 2 : light rain (LR) 3 : ice crystals (CR) 4 : rimed particles (RP) 5 : rain (RN) 6 : vertically aligned ice (VI) 7 : wet snow (WS) 8 : melting hail (MH) 9: dry hail / high density graupel (IH/HDG)

rainforest.common.radarprocessing.hydroClass_single_over_iso(radars, zh, zdr, kdp, rhohv, H_ISO0, weights=array([1., 1., 1., 0.75, 0.5]))

Computes the hydrometeor classes for columnar data, note that all input arrays except weights must have the same length

Parameters:

radars (ndarray of char) – Array of radar IDs, (‘A’,’D’,’L’,’P’,’W’)
zh (ndarray) – Array of radar reflectivity in dBZ
zdr (ndarray) – Array of diff. reflectivity in dB
kdp (ndarray) – Array of specific phase shift on propagation in deg / km
rhohv (ndarray) – Array of copolar correlation coefficient
iso0_height (ndarray) – Array of 0° degree isotherm altitude
weights (ndarray (optional)) – The weight of every input feature, zh, zdr, kdp, rhohv, temp in the hydrometeor classification

Returns:

The hydrometeor classes as ndarray with values from 0 to 8, corresponding to
the classes – 0 : no data 1 : aggregates (AG) 2 : light rain (LR) 3 : ice crystals (CR) 4 : rimed particles (RP) 5 : rain (RN) 6 : vertically aligned ice (VI) 7 : wet snow (WS) 8 : melting hail (MH) 9: dry hail / high density graupel (IH/HDG)

rainforest.common.constants module

Set of constants regarding MeteoSwiss radars and QPE

rainforest.common.constants.MODE(x)

rainforest.common.graphics module

Set of functions to display QPE precipitation data and verification scores

class rainforest.common.graphics.MidpointNormalize(vmin=None, vmax=None, transition=None, clip=False)

Bases: Normalize

Normalizing that is linear up to a certain transition value, logarithmic afterwards

Parameters:

vmin (float or None) – Values within the range [vmin, vmax] from the input data will be linearly mapped to [0, 1]. If either vmin or vmax is not provided, they default to the minimum and maximum values of the input, respectively.
vmax (float or None) – Values within the range [vmin, vmax] from the input data will be linearly mapped to [0, 1]. If either vmin or vmax is not provided, they default to the minimum and maximum values of the input, respectively.
clip (bool, default: False) –
Determines the behavior for mapping values outside the range [vmin, vmax].

If clipping is off, values outside the range [vmin, vmax] are also transformed, resulting in values outside [0, 1]. This behavior is usually desirable, as colormaps can mark these under and over values with specific colors.

If clipping is on, values below vmin are mapped to 0 and values above vmax are mapped to 1. Such values become indistinguishable from regular boundary values, which may cause misinterpretation of the data.

Notes

If vmin == vmax, input data will be mapped to 0.

class rainforest.common.graphics.QPE_cmap

Bases: LinearSegmentedColormap

Colormap that uses purple tones for small values, and then a transition from blue to red for values above the transition

Create colormap from linear mapping segments

segmentdata argument is a dictionary with a red, green and blue entries. Each entry should be a list of x, y0, y1 tuples, forming rows in a table. Entries for alpha are optional.

Example: suppose you want red to increase from 0 to 1 over the bottom half, green to do the same over the middle half, and blue over the top half. Then you would use:

cdict = {'red':   [(0.0,  0.0, 0.0),
                   (0.5,  1.0, 1.0),
                   (1.0,  1.0, 1.0)],

         'green': [(0.0,  0.0, 0.0),
                   (0.25, 0.0, 0.0),
                   (0.75, 1.0, 1.0),
                   (1.0,  1.0, 1.0)],

         'blue':  [(0.0,  0.0, 0.0),
                   (0.5,  0.0, 0.0),
                   (1.0,  1.0, 1.0)]}

Each row in the table for a given color is a sequence of x, y0, y1 tuples. In each sequence, x must increase monotonically from 0 to 1. For any input value z falling between x[i] and x[i+1], the output value of a given color will be linearly interpolated between y1[i] and y0[i+1]:

row i:   x  y0  y1
               /
              /
row i+1: x  y0  y1

Hence y0 in the first row and y1 in the last row are never used.

rainforest.common.lookup module

Functions to read and compute lookup tables

The available lookup tables are

qpebias_station : dict with the bias correction of RZC at every station

cosmo1_to_rad : dict which maps COSMO grid to polar radar grid it has keys [sweep][coord_type], coord_type is ‘idx_0’ : first index of COSMO grids, ‘idx_1’: second, ‘idx_3’: third, mask’ is 1 for points that fall outside of COSMO domain. This lookup table is valid only for COSMO data stored in /store/s83/owm/COSMO-1/

cosmo2_to_rad : same thing but for COSMO 1E data

cosmo2_to_rad : same thing but for COSMO 2 data

cosmo1T_to_rad : same thing but for netCDF files of COSMO1 temperature extracted for MDR and stored in /store/s83/owm/COSMO-1/ORDERS/MDR/

cosmo1T_to_rad : same thing but for netCDF files of COSMO2 temperature extracted for MDR and stored in /store/msrad/cosmo/cosmo2/data/

station_to_rad : maps the SMN stations to radar coordinates, it is an extraction of the more generic but less convenient qpegrid_to_rad table It is list of 3 elements, first element is a dict with keys [station][sweep][ncode] and gives the polar gates (azimuth_idx, range_idx) that fall within a Cartesian pixel at a given radar elevation and for a given station neighbour (00 = station location, -1-1 = one to the south-west, 22 = two km to north and 2 km to east) second element is a dict giving distance from every station to the radar third element is a dict of keys [station][sweep] giving the height above ground of the radar observations above that station at a given elevation (sweep) MISSING KEYS IMPLY NO RADAR VISIBILITY

cartcoords_rad : gives the Cartesian (Swiss LV03) coordinates of all polar gates. It is a dict that gives for every sweep a 3D array of shape ncoords x nazimuth x nrange, ncoords is 3, first slice is Swiss Y coordinate (west to east), second is Swiss X-coordinate (south to north) and last is Swiss Z coordinate (altitude)

qpegrid_to_height_rad : gives for every pixel of the Swiss Cartesian QPE grid the average height of radar observations for a given radar, for every sweep. It is a dictionary where the key is the sweep number ( 1 - 20) and the value is a 640 x 710 array of heights above ground

station_to_qpegrid : maps every station to the corresponding QPE gridpoint it is a dict of keys [station][ncode] and gives the index of every neighbour of every station in the QPE 2D grid 640 x 710 pixels

visibility_rad : gives the (static) visibility of every polar gate for a given sweep number in the form of a 2D field of size nazimuth x nrange

rainforest.common.lookup.calc_lookup(lookup_type, radar=None)

Calculates a lookup table and stores it in the /data/lookup_data folder

Parameters:

lookup_type (str) –
the lookup table type, must be one of
- qpebias_station
- cosmo1_to_rad
- cosmo1e_to_rad
- cosmo2_to_rad
- cosmo1T_to_rad
- cosmo2T_to_rad
- station_to_rad
- cartcoords_rad
- qpegrid_to_rad
- station_to_qpegrid
- visibility_rad
radar (char or list of chars (optional)) – the radar for which to retrieve the lookup table, needed only if the lookup_type contains the term ‘rad’, must be either ‘A’, ‘D’, ‘L’, ‘W’ or ‘P’

rainforest.common.lookup.get_lookup(lookup_type, radar=None)

Read a lookup table from the /data/lookup_data folder

Parameters:

lookup_type (str) –
the lookup table type, must be one of
- qpebias_station
- cosmo1_to_rad
- cosmo1e_to_rad
- cosmo2_to_rad
- cosmo1T_to_rad
- cosmo2T_to_rad
- station_to_rad
- cartcoords_rad
- qpegrid_to_rad
- station_to_qpegrid
- visibility_rad
radar (char or list of chars (optional)) – the radar for which to retrieve the lookup table, needed only if the lookup_type contains the term ‘rad’, must be either ‘A’, ‘D’, ‘L’, ‘W’ or ‘P’

Returns:

lut – The lookup table in the form of a python dict

Return type:

dict

rainforest.common.utils module

Set of functions that can be useful

Daniel Wolfensberger MeteoSwiss/EPFL daniel.wolfensberger@epfl.ch December 2019

rainforest.common.utils.LV03toWGS84(east, north, heights)

Converts a set of WGS84, lat/lon/heights to Swiss CH1903 coordinates

Parameters:

east (ndarray) – Easterly Swiss coordinates (CHY)
north (ndarray) – northerly Swiss coordinates (CHX)
heights (ndarray) – heights a.s.l in WGS84 coordinates

Return type:

lat, lon and height coordinates in WGS84

rainforest.common.utils.aggregate_multi(array_3d, agg_operators)

Aggregates a 3D numpy array alongs its first axis, using different aggregation operators

Parameters:

array_3d (ndarray) – 3D numpy array, of shape (N x M x L)
agg_operators (ndarray of integers) – array of aggregation operators as defined in the constants.py file, must have the same length as the first dimension of array_3D (N)

Return type:

An aggregated array of size M x L

rainforest.common.utils.check_random_state(seed)

Turn seed into a np.random.RandomState instance.

Parameters:: seed (None, int or instance of RandomState) – If seed is None, return the RandomState singleton used by np.random. If seed is an int, return a new RandomState instance seeded with seed. If seed is already a RandomState instance, return it. Otherwise raise ValueError.
Returns:: The random state object based on seed parameter.
Return type:: numpy:numpy.random.RandomState

rainforest.common.utils.chunks(l, n): Divides a list l into n sublists of similar sizes

rainforest.common.utils.dict_flatten(mydict): Flattens a nested dictionary

rainforest.common.utils.envyaml(filename)

Reads a yaml configuration file while parsing environment variables. Environment variables must be written as ${NAME_OF_VAR} in the yaml file

Parameters:: filename (str) – path of the input yaml file
Returns:: the yaml content in the form of a python dict
Return type:: dict

rainforest.common.utils.get_qpe_files(input_folder, t0=None, t1=None, time_agg=None, list_models=None)

Gets the list of all qpe files in a folder (as saved by qpe_compute) and separates them by qpe type and timestep

Parameters:

input_folder (str) – main directory where the qpe files are saved, it contains one subfolder for every qpe model (type) that was used
t0 (datetime (optional)) – Starting time of the period to retrieve, will be used to filter files, if not provided starting time will be time of first file
t1 (datetime (optional)) – End time of the period to retrieve, will be used to filter files, if not provided end time will be time of last file
time_agg (minutes (optional)) – Will aggregate all files to a reference time in minutes (e.g. use 10 to put together all files that correspond to a single gauge measurement)
list_models ((optional)) – List of qpe types to retrieve , if not provided all folders in input_folder will be used

Returns:

A dictionary where every key is a QPE model and every value is a list
with all files in chronological order

rainforest.common.utils.get_qpe_files_multiple_dirs(input_folder, t0=None, t1=None, time_agg=None, list_models=None)

Gets the list of all qpe files from multiple folders sorted according to DOY (as saved by qpe_compute) and separates them by qpe type and timestep

Parameters:

input_folder (str) – main directory where the qpe files are saved, it contains one subfolder for every qpe model (type) that was used
t0 (datetime (optional)) – Starting time of the period to retrieve, will be used to filter files, if not provided starting time will be time of first file
t1 (datetime (optional)) – End time of the period to retrieve, will be used to filter files, if not provided end time will be time of last file
time_agg (minutes (optional)) – Will aggregate all files to a reference time in minutes (e.g. use 10 to put together all files that correspond to a single gauge measurement)
list_models ((optional)) – List of qpe types to retrieve , if not provided all folders in the first input_folder will be used

Returns:

A dictionary where every key is a QPE model and every value is a list
with all files in chronological order

rainforest.common.utils.get_version()

rainforest.common.utils.hex_to_rgb(value)

rainforest.common.utils.idx_cart(x, y): Returns the Cartesian index of a set of coordinates x and y

rainforest.common.utils.nanadd_at(a, indices, b): Replaces nans by zero in call to np.add.at

rainforest.common.utils.nearest_time(dt, reference)

Gets the nearest earlier reference timestep to a given datetime, for ex. if dt = 1 Jan 2020 10:12, and reference is 10 it will return

1 Jan 2020 10:10, or if dt = 1 Jan 2020 10:59 and reference is 60 it will return 1 Jan 2020 10:00

Parameters:

dt (datetime) – The datetime to check
reference (int) – The reference timestep in minutes

Return type:

The closest earlier timestep in datetime format

rainforest.common.utils.nested_dict_gen(d): The generator for the previous function

rainforest.common.utils.nested_dict_values(d): Extracts all values from a nested dictionary

rainforest.common.utils.perfscores(est_data, ref_data, bounds=None, array=False)

Computes a set of precipitation performance scores, on different data ranges. The scores are

scatter: 0.5 * (Qw84(x) - Qw16(x)), where Qw is a quantile weighted by ref_data / sum(ref_data) and x is est_data / ref_data in dB scale

RMSE: root mean square error (linear error)

bias: (ME/mean(ref_data) + 1) in dB

ED: the energy distance which is a measure of the distance between two distributions (https://en.wikipedia.org/wiki/Energy_distance)

Parameters:

est_data (ndarray) – array of estimates (ex. precip from QPE)
ref_data (ndarray) – array of reference (ex. precip from gauge)
bounds (list (optional)) – list of bounds on ref_data for which to compute the error metrics, by default all data will be used (unbounded), note that even if you prescribe bounds the scores for the overall data will always be added in the output
array (boolean (optional)) – Whether or not to convert the output dict to a numpy array

Returns:

all_metrics – a dictionary containing all the scores, organized in the following way all_metrics[bound][score]

Return type:

dict or ndarray

rainforest.common.utils.quantile(data, weights, quantile)

Weighted quantile of an array with respect to the last axis.

Parameters:

data (ndarray) – Input array.
weights (ndarray) – Array with the weights. It must have the same size of the last axis of data.
quantile (float) – Quantile to compute. It must have a value between 0 and 1.

Returns:

quantile – The output value.

Return type:

float

rainforest.common.utils.quantile_1D(data, weights, quantile)

Compute the weighted quantile of a 1D numpy array.

Parameters:

data (ndarray) – Input array (one dimension).
weights (ndarray) – Array with the weights of the same size of data.
quantile (float) – Quantile to compute. It must have a value between 0 and 1.

Returns:

quantile_1D – The output value.

Return type:

float

rainforest.common.utils.read_df(pattern, dbsystem='dask', sqlContext=None)

Reads a set of data contained in a folder as a spark or dask DataFrame

Parameters:

pattern (str) – Unix style wildcard pattern pointing to the files, for example /store/msrad/folder/*.csv will read all csv files in that folder
dbsystem (str) – Either “dask” if you want a Dask DataFrame or “spark” if you want a spark dataframe
sqlContext (sqlContext instance, new: SparkSession instant) – sqlContext to use, required only if dbystem = ‘spark’

Return type:

A spark or dask DataFrame instance

rainforest.common.utils.read_task_file(task_file): Reads a database processing task file

rainforest.common.utils.rename_fields(data): Rename pyart fields from pyrad names to simpler names, according to the dictionary PYART_NAMES_MAPPING in the constants.py module

rainforest.common.utils.round_to_hour(dt): Returns the sweep number of a polar file based on its name

rainforest.common.utils.split_by_time(files_rad): Separate a list of files by their timestamp

rainforest.common.utils.stack_uneven(arrays, fill_value=nan)

Fits mmltiple into a single numpy array, even if they are different sizes, assigning a fill_value to parts with no data

Parameters:

arrays (list of np arrays) – list of numpy array to stack, they can have different dimensions
fill_value ((float, optional)) – the fill value with which to fill the missing pixels

Returns:

a np.ndarray with size N x M, where N is the sum of the number of
rows of all arrays and M is the maximal number of columns in all arrays

rainforest.common.utils.sweepnumber_fromfile(fname): Returns the sweep number of a polar file based on its name

rainforest.common.utils.timefromfilename(fname): Returns the datetime of a file based on its name

rainforest.common.utils.timestamp_from_datestr(datestr)

rainforest.common.utils.timestamp_from_datetime(dt)

rainforest.common.utils.wgs84toCH1903(lat, lon, heights)

Converts a set of WGS84, lat/lon/heights to Swiss CH1903 coordinates, east, north and height

Parameters:

lat (ndarray) – latitudes in decimal format (degrees)
lon (ndarray) – longitudes in decimal format (degrees)
heights (ndarray) – heights a.s.l in WGS84 coordinates

Return type:

east, north and height coordinates in CHLV190

rainforest.common.wgs84_ch1903 module

class rainforest.common.wgs84_ch1903.GPSConverter

Bases: object

GPS Converter class which is able to perform convertions between the CH1903 and WGS84 system.

CHtoWGSheight(y, x, h): Convert CH y/x/h to WGS height

CHtoWGSlat(y, x): Convert CH y/x to WGS lat

CHtoWGSlng(y, x): Convert CH y/x to WGS long

LV03toWGS84(east, north, height): Convert LV03 to WGS84. Return a tuple of floating point numbers containing lat, long, and height

WGS84toLV03(latitude, longitude, ellHeight): Convert WGS84 to LV03. Return a tuple of floating point numbers containing east, north, and height

WGStoCHh(lat, lng, h): Convert WGS lat/long (° dec) and height to CH h

WGStoCHx(lat, lng): Convert WGS lat/long (° dec) to CH x

WGStoCHy(lat, lng): Convert WGS lat/long (° dec) to CH y