Main configuration file

The main configuration file is used to define the global settings, notably the paths to the different sources of data. The parameters of the main configuration file are described in Table 2.

Table 2: Configuration parameters of the main configuration file

Name	Type	Description
name	STRING	Name of the data processing. This name is used in the path of the saved products in the following manner: <saveimgbasepath>/<name>/<YYYY-MM-DD>/<datasetname>/<prodname>/<outputname>
datapath	STRING	Base directory of the rainbow raw data. This field must have a trailing ‘/’. The raw data files of a scan can be found using the following file path: <datapath>/<scanname>/<YYYY-MM-DD>/<YYYYMMDDHHMMSS00datatype>.<ext>
configpath	STRING	Absolute base directory of the configuration files. This directory contains clutter maps, filter coefficients, antenna pattern, and the data processing configuration files. If any path within a pyrad config file is provided as a relative path, it will be converted to an absolute path, by prepending it with the path specified with “configpath”. If not provided, the directory of the main config file will be used.
locationConfigFile	STRING	File nameof the location configuration file. If only the filename (and not directory) is provided it will be prepended with configpath. Described in Section 3.2.
productConfigFile	STRING	File name of the product configuration file. If only the filename (and not directory) is provided it will be prepended with configpath. Described in Section 4.
lastStateFile	STRING	File name (with full path) of the file containing the time of the last processed scan. Used in particular for real-time processing.
imgformat	STRING STRARR	File format(s) of the images. The following formats are supported: eps, png, and jpg. If saveimg is set to 0, this field is not used.
saveimgbasepath	STRING	Base directory for the images to save. The directory structure looks as follows: <saveimgbasepath>/<name>/<YYYY-MM-DD>/<datasetname>/<prodname>/<outputname> If saveimg is set to 0, this field is not used.
s3EndpointRead	STRING	OPTIONAL. Url to an S3 endpoint containing input radar data. The format must be https://endpoint.domain (e.g. https://eu-central-1.linodeobjects.com/), the https:// is not mandatory.
s3BucketRead	STRING	OPTIONAL. Name of an S3 bucket containing input radar data. It has to be used together with s3EndpointRead and s3PathRead to be able to read input radar data from a bucket. The procedure will only work by setting the environment variables S3_KEY_READ and S3_SECRET_READ.
s3PathRead	STRING	OPTIONAL. Path where to find input radar data in an S3 bucket. The data will be retrieved at url https://s3BucketRead.s3EndpointRead/s3PathRead/*
rm_s3_file	INT	OPTIONAL. If set, input radar data files downloaded from an S3 bucket will be removed after reading.
s3EndpointWrite	STRING	OPTIONAL. Url to an S3 endpoint where to store output data. The format must be https://endpoint.domain (e.g. https://eu-central-1.linodeobjects.com/), the https:// is not mandatory.
s3BucketWrite	STRING	OPTIONAL. Name of an S3 bucket for output radar data. It has to be used together with s3EndpointWrite and s3PathWrite to save output data to a bucket. Requires S3_KEY_WRITE and S3_SECRET_WRITE environment variables.
s3PathWrite	STRING	OPTIONAL. Path where to save output radar data in an S3 bucket. The data will be saved at url https://s3BucketRead.s3EndpointRead/s3PathRead/filename
s3SplitExtensionWrite	INT	OPTIONAL. If set to 1, files with multiple extensions are stored separately in the S3 bucket. For example, myfolder/out.csv becomes myfolder/csv/out.csv. Default is 0 (False).
s3Verify	INT	OPTIONAL. If set to 0, disables SSL verification for S3 operations. Default is 1.
s3Certificates	INT	OPTIONAL. Path to SSL certificates, only needed if custom certificates are used.
MySQLhost	STRING	REQUIRED. Hostname or IP of the MySQL server. Use localhost when the database runs on the same machine.
MySQLdatabase	STRING	OPTIONAL. Name of the MySQL database to connect to. The database must already exist.
MySQLuser	STRING	OPTIONAL. Username used to authenticate with the MySQL server. The user must have appropriate privileges on the selected database.
MySQLport	INTEGER	OPTIONAL. Port number of the MySQL server. Default is 3306. Use a different value (e.g. 3307) if the server is exposed on a custom port.
loadbasepath	STRING	OPTIONAL. Base path of saved data. Defaults to saveimgbasepath.
loadname	STRING	OPTIONAL. Name of the saved data processing. Used for saved volume loading. Defaults to name.
gecsxbasepath	STRING	OPTIONAL. Base path of saved visibility fields generated by the GECSX tool.
gecsxname	STRING	OPTIONAL. Name of the GECSX dataset containing the visibility fields. Used for GECSX volume loading.
dempath	STRING	OPTIONAL. Base directory of the Digital Elevation Model (DEM) files. Used to load radar visibility.
smnpath	STRING	OPTIONAL. Base directory of SwissMetNet stations data. Used for comparison with rain gauges.
disdropath	STRING	OPTIONAL. Base directory of disdrometer data. Used for comparison with disdrometers.
solarfluxpath	STRING	OPTIONAL. Base directory of solar flux data. Used to plot calibration bias from sun monitoring.
iconpath	STRING	OPTIONAL. Base directory of ICON (NWP model) data files.

Note on s3 and MySQL writing

Every pyrad product output (png or csv) can be written to a s3 bucket, provided that the keywords s3EndpointWrite, s3BucketWrite are provided in the main configuration file (plus optional keywords), and the variables S3_KEY_WRITE and S3_SECRET_WRITE (bucket key ID and secret) are available as environment variables during runtime. Note that the python library boto3 has to be installed.

Every csv pyrad product output can be written to a MySQL database, provided that the keywords MySQLhost, MySQLdatabase, MySQLuser are provided in the main configuration file (plus optional keywords), and the variable MYSQL_PASSWORD is available as environment variable during runtime. Note that the python libraries sqlalchemy cryptography and pymsql have to to be installed.

For every single pyrad product the two keywords s3write and MySQLwrite can be used to specify whether a given product should be written to s3/MySQL. By default s3write is True, meaning that all products will be written to s3 (if possible) and MySQLwrite is False, meaning that no product will be written to MySQL.