dvas_recipes package

Copyright (c) 2020-2022 MeteoSwiss, contributors listed in AUTHORS.

Distributed under the terms of the GNU General Public License v3.0 or later.

SPDX-License-Identifier: GPL-3.0-or-later

Subpackages

Submodules

dvas_recipes.dynamic module

Copyright (c) 2020-2023 MeteoSwiss, contributors listed in AUTHORS.

Distributed under the terms of the GNU General Public License v3.0 or later.

SPDX-License-Identifier: GPL-3.0-or-later

Module content: dynamic variables, that get modified on the fly by the Recipes.

dvas_recipes.dynamic.RECIPE = None

name of the dvas recipe

Type:

str

dvas_recipes.dynamic.CURRENT_FLIGHT = None

[12345, 1]

Type:

list

Type:

current event_id, rig_id of the flight to be processed next. E.g.

dvas_recipes.dynamic.CURRENT_VAR = None

‘temp’

Type:

dict

Type:

current variable name to be processed, and associated uncertainties, e.g.

dvas_recipes.dynamic.INDEXES = None

variable database name associated to the ‘tdt’ and ‘alt’ indexes

Type:

dict

dvas_recipes.dynamic.CURRENT_STEP_ID = None

current step id

Type:

int|str

dvas_recipes.dynamic.ALL_STEP_IDS = []

all the recipe steps id

Type:

list

dvas_recipes.dynamic.N_CPUS = 1

number of cpus to use to run the dvas recipe

Type:

int

dvas_recipes.errors module

Copyright (c) 2020-2022 MeteoSwiss, contributors listed in AUTHORS.

Distributed under the terms of the GNU General Public License v3.0 or later.

SPDX-License-Identifier: GPL-3.0-or-later

Module contents: Error management

exception dvas_recipes.errors.DvasRecipesError

Bases: Exception

General error class for dvas_recipes.

ERR_MSG = 'dvas_recipes Error'

dvas_recipes.hl_commands module

Copyright (c) 2020-2022 MeteoSwiss, contributors listed in AUTHORS.

Distributed under the terms of the GNU General Public License v3.0 or later.

SPDX-License-Identifier: GPL-3.0-or-later

This file contains the very-high-level dvas_recipe commands, to initialize and run them.

dvas_recipes.hl_commands.init_arena(arena_path=None)

Initializes a new dvas processing arena.

Parameters:

arena_path (pathlib.Path, optional) – relative Path to the processing arena to initialize. Defaults to None == ./dvas_proc_arena/.

dvas_recipes.hl_commands.optimize(n_cpus=None, prf_length=7001, chunk_min=50, chunk_max=300, n_chunk=6)

Measures which chunk size provides the fastest computation time for the combine() function.

This function is relatively dumb: it will try all requested chunk sizes, and measure which is the fastest.

Warning

For best results, it is recommended to not run anything else in parallel to this test, in particular if all cpus are requested.

Note

To reduce computing time, profiles get “sliced” when they are to be combined, in order to exploit, if available, multiple cpus. The size of the individual chunks is a parameter that can be adjusted manually by the user of dvas recipes. The best chunk size will be system dependant. A size too low will increase the computation time through additional loops. A size too large will drastically increase (and possible top-up) the RAM requirements, thus also increasing the computing time.

Parameters:

  • n_cpu (int, optional) – number of cpus to use for the test. Defaults to None = all available.

  • prf_length (int, optional) – length of the test profiles. Default to 7001 elements.

  • chunk_min (int, optional) – minimum chunk size to test. Defaults to 50.

  • chunk_max (int, optional) – maximum chunk size to test. Defaults to 300.

  • n_chunk (int, optional) – number of chunk samples to take. Defaults to 6.

dvas_recipes.hl_commands.run_recipe(rcp_fn: Path, fid_log_fn: Path, fid_to_treat: list = None, from_step_id: str = None, until_step_id: str = None, debug: bool = False)

Loads and execute a dvas recipe.

Parameters:

  • rcp_fn (pathlib.Path) – path to the specific dvas recipe to execute.

  • fid_log_fn (pathlib.Path) – path to the log linking flight ids, event ids, rig ids, and event datetimes.

  • fid_to_treat (list of str, optional) – list of flights ids to treat specifically.

  • from_step_id (str, optional) – if set, will skip all processing until this step_id value. Defaults to None.

  • until_step_id (str, optional) – if set, will skip all processing step beyond this step_id value. Defaults to None.

  • debug (bool, optional) – if True, will force-set the logging level to DEBUG. Defaults to False.

  • The “fid_log_fn” file should contain one tuple of flight if, event id, rig id, and

  • event datetime per line, e.g.:: – # This is a comment F01,e:12345,r:1,2022-08-19T19:00:00.000Z T04,e:12340,r:2,2022-08-19T15:45:00.000Z

  • Event ids are meant to be GRUAN flight ids, in order to link GDPs with specific flights.

  • Rig ids are also extracted from GRUAN parameters in GDP files to distinguish different rigs

  • in case of multiple simultaneous launches.

  • Event datetimes are not used by dvas directly, but allow for an easier identification of

  • flights in UPPS (UAII Plot Preview Software)

  • Flight ids are user-defined reference strings, used to identify specific flight in a given

  • campaign - these are typically defined before GRUAN event ids (that are generated at the

  • creation of GDPs).)

  • TODO – At present, it is not possible to link fids to eids, because the information is missing from the iMS100 GDP. This products would require the field ‘g.measurement.InternalKey’ to be present, like the RS41 GDP, e.g.:

    :g.Measurement.InternalKey = “UAII2022_F08”

dvas_recipes.recipe module

Copyright (c) 2020-2023 MeteoSwiss, contributors listed in AUTHORS.

Distributed under the terms of the GNU General Public License v3.0 or later.

SPDX-License-Identifier: GPL-3.0-or-later

This file contains specific recipe classes and utilities.

dvas_recipes.recipe.for_each_flight(func)

This function is meant to be used as a decorator around recipe step functions that need to be executed sequentially for all the specified flights.

Parameters:

func (function) – the recipe step function to decorate.

dvas_recipes.recipe.for_each_var(incl_wdir=True, incl_wspeed=True, incl_wvec=False, incl_latlon=False)

Parameteric decorator, to enable the looping over different variables, with the option to select different wind elements.

Parameters:

  • incl_wdir (bool) – whether to include the wdir variable, or not.

  • incl_wspeed (bool) – whether to include the wspeed variable, or not.

  • incl_wvec (bool) – whether to include the wvec variable, or not.

  • incl_latlon (bool) – whether to include the lat and lon variables, or not.

dvas_recipes.recipe.for_each_oscar_var(func)

This function is meant to be used as a decorator around recipe step functions that need to be executed sequentially for all the OSCAR variables.

Parameters:

func (function) – the recipe step function to decorate.

class dvas_recipes.recipe.RecipeStep(func, step_id, name, run, kwargs)

Bases: object

RecipeStep class, to handle individual recipe steps and their execution

__init__(func, step_id, name, run, kwargs)

RecipeStep initialization function.

Parameters:

  • func (function) – dvas_recipe function to be launched as part of the step.

  • step_id (str|int) – id of the recipe step, e.g. ‘01a’.

  • name (str) – name of the recipe step, e.g. ‘uaii2022.sync.sync_flight’

  • run (bool) – if True, the step is supposed to be executed as part of the recipe.

  • kwargs (dict) – dictionnary of keyword arguments to be fed to func.

property step_id

I.D. of this recipe step

property name

name of this recipe step

property run

Whether this step should be executed, or not, as part of the recipe

execute()

Launches the processing associated with this recipe step.

class dvas_recipes.recipe.Recipe(rcp_fn, eids_to_treat=None, debug=False)

Bases: object

Recipe class, designed to handle the loading/initialization/execution of dvas recipes from dedicated YAML files.

__init__(rcp_fn, eids_to_treat=None, debug=False)

Recipe initialization from a suitable YAML recipe file.

Parameters:

  • rcp_fn (pathlib.Path) – path of the recipe file to initialize.

  • eids_to_treat (list, optional) – list of (‘fid’, ‘e:eid’, ‘r:rid’) tuples. If None, will process all the flights found in the DB.

  • debug (bool, optional) – if True, will force-set the logging level to DEBUG. Defaults to False.

property name

Returns the name of the Recipe.

property n_steps

Returns the number of steps inside the Recipe.

static init_db(reset: bool = True, data_in_db: bool = True)

Initialize the dvas database, and fetch the original data required for the recipe.

Parameters:

reset (bool, optional) – if True, the DB will be filled from scratch. Else, only new original data will be ingested. Defaults to True.

static get_all_flights_from_db()

Identifies all the radiosonde flights present in the dvas database.

Returns:

2D ndarray of str –

the array of event_id and rig_id tags for each flight, with the

following structrure:

[['e:12345', 'r:1'], ['e:12346', 'r:1']]

execute(from_step_id=None, until_step_id=None)

Run the recipe step-by-step, possibly skipping some of the first ones.

Parameters:

  • from_step_id (str|int, optional) – if set, will start the processing from this specific step_id. Defaults to None = start at first step.

  • until_step_id (str|int, optional) – if set, will end the processing after this specific step_id. Defaults to None = go until the end of the recipe.

dvas_recipes.utils module

Copyright (c) 2020-2022 MeteoSwiss, contributors listed in AUTHORS.

Distributed under the terms of the GNU General Public License v3.0 or later.

SPDX-License-Identifier: GPL-3.0-or-later

This module contains general dvas_recipe utilities.

dvas_recipes.utils.demo_storage_path()

Returns the absolute path to the stored dvas demo locations, from where the dvas demo scripts can be copied locally.

dvas_recipes.utils.recipe_storage_path()

Returns the absolute path to the stored dvas recipes location, from where they can be copied locally.

dvas_recipes.utils.default_arena_path()

Returns the default relative location and name of the dvas processing arena for a specific recipe: ‘./dvas_recipe_arena/’

Args:

Returns:

pathlib.Path – the default relative Path.

dvas_recipes.utils.fn_suffix(fid=None, eid=None, rid=None, var=None, mids=None, pids=None, tags=None)

Returns the default suffix of filenames given a set of info provided by the user.

Parameters:

  • fid (str, optional) – the flight id

  • eid (int, optional) – the event id

  • rid (int, optional) – the rig id

  • var (str, optional) – the variable name

  • mids (list of str, optional) – the list of mids

  • pids (list of str, optional) – the list of pids

  • tags (list of str, optional) – the list of tags associated with the data

Returns:

str – the filename suffix, that can be fed to the dvas.plots.utils.fancy_savefig().

dvas_recipes.utils.format_tags(tags)

Formats a list of tags according to some basic rules. Any tag set by the user should be fed to this routine.

Parameters:

tags (str, list) – the tags to format

Returns:

list – the cleaned-up tags

dvas_recipes.utils.rsid_tags(pop=None)

Returns the list of rsid (recipe step ID) tags, possibly by removing some of them.

Parameters:

pop (list, optional) – if set, any tags in this list will be removed from the returned list.

Returns:

list – rsid tags, minus the popped ones.

dvas_recipes.utils.cws_vars(incl_latlon=False)

Return the list of variables that are present in CWS.