dvas.config package

Distributed under the terms of the GNU General Public License v3.0 or later.

SPDX-License-Identifier: GPL-3.0-or-later

Submodules

dvas.config.config module

Distributed under the terms of the GNU General Public License v3.0 or later.

SPDX-License-Identifier: GPL-3.0-or-later

Module contents: User configuration management.

dvas.config.config.instantiate_config_managers(*args, read=True)

Generate a dictionary with instances of all specified ConfigManagers

Parameters:

args (ConfigManager) – ConfigManager to instantiate
read (bool, optional) – Read config automatically after instantiation. Default to True.

Returns:

dict – instances of ConfigManager

class dvas.config.config.ConfigManager

Bases: object

Abstract class for managing YAML config

REQUIRED_ATTRIBUTES = {'DOC_TYPE': <class 'type'>}

DOC_TYPE = None

[dict, list].

Type:: type
Type:: Type of document. Choices

document

Config document. Must be redefined as well to avoid list/dict reference overlap

Type:: dict

init_document(): Initialise document attribute

append(value)

Append value to document attribute

Parameters:: value (dict or list) – Value to append

abstract read(doc_in): Abstract read method

class dvas.config.config.OneLayerConfigManager

Bases: ConfigManager

Abstract class for managing ‘one-layer’ YAML config.

‘one-layer’ means YAML file of such type:

label1: value_label1
label2: value_label2
...
labelN: value_labelN

REQUIRED_ATTRIBUTES = {'CLASS_KEY': <class 'str'>, 'DOC_TYPE': <class 'type'>, 'LABEL_VAL_DEF': <class 'dict'>, 'PARAMETER_PATTERN_PROP': <class 'dict'>}

DOC_TYPE: alias of dict

PARAMETER_PATTERN_PROP = None

Parameter JSON 7 schema. Constant value.

Must be a dict like:

{"type": "object",
 "patternProperties": ANY,
 "additionalProperties": False
}

Type:: dict

LABEL_VAL_DEF = None

Default values of labels

Type:: dict

CLASS_KEY = None

Class key denomination

Type:: str

property json_schema

JSON schema. Constant value

Type:: dict

read(doc_in=None)

Read config from config files

Parameters:: doc_in (str, optional) – Default None -> read from directory Else read from doc_in.
Raises:: ConfigReadError – Error in reading YAML config data.

read_yaml(yaml_doc)

Read YAML document.

Parameters:: yaml_doc (str | pathlib.Path) – YAML document
Returns:: dict

class dvas.config.config.CSVOrigMeta

Bases: OneLayerConfigManager

CSV original metadata config manager

PARAMETER_PATTERN_PROP = {'^\\w+$': {'oneOf': [{'type': 'null'}, {'type': 'string'}, {'type': 'number'}, {'type': 'boolean'}]}}

Parameter JSON 7 schema. Constant value.

Must be a dict like:

{"type": "object",
 "patternProperties": ANY,
 "additionalProperties": False
}

Type:: dict

LABEL_VAL_DEF = {}

Default values of labels

Type:: dict

CLASS_KEY = 'CSVOrigMeta'

Class key denomination

Type:: str

document

Config document

Type:: dict

class dvas.config.config.OneDimArrayConfigManager

Bases: OneLayerConfigManager

Abstract class for managing ‘one-dim-array’ YAML config.

‘one-dim-array’ means YAML file of such type:

- label11: value_label11
  label12: value_label12
  ...
  label1N: value_label1N

- label21: value_label21
  label22: value_label22
  ...
  label2N: value_label2N
...
- labelM1: value_labelM1
  labelM2: value_labelM2
  ...
  labelMN: value_labelMN

REQUIRED_ATTRIBUTES = {'CLASS_KEY': <class 'str'>, 'CONST_LABELS': <class 'list'>, 'DOC_TYPE': <class 'type'>, 'LABEL_VAL_DEF': <class 'dict'>, 'NODE_GEN': <class 'str'>, 'PARAMETER_PATTERN_PROP': <class 'dict'>}

DOC_TYPE: alias of list

CONST_LABELS = None

Constant labels

Type:: list

NODE_GEN = None

Node name able to be generated by regexp. Use empty str to assign none node.

Type:: str

property json_schema: JSON schema. Constant value

read(doc_in=None)

Read config

Parameters:: doc_in (str, optional) – Default None -> read from directory Else read from doc_in.
Raises:: ConfigReadError – Error in reading YAML config data.

class dvas.config.config.Model

Bases: OneDimArrayConfigManager

Instrument type config manager

PARAMETER_PATTERN_PROP = {'^mdl_desc$': {'type': 'string'}, '^mdl_name$': {'oneOf': [{'pattern': '[a-zA-Z0-9]+(([a-zA-Z0-9])|([\\-\\_\$\$]))*[a-zA-Z0-9]', 'type': 'string'}, {'const': ''}]}, '^mid$': {'type': 'string'}}

Parameter JSON 7 schema. Constant value.

Must be a dict like:

{"type": "object",
 "patternProperties": ANY,
 "additionalProperties": False
}

Type:: dict

LABEL_VAL_DEF = {'mid': ''}

Default values of labels

Type:: dict

CLASS_KEY = 'Model'

Class key denomination

Type:: str

CONST_LABELS = [{'mdl_desc': 'Null instrument type', 'mdl_name': ''}]

Constant labels

Type:: list

NODE_GEN = ''

Node name able to be generated by regexp. Use empty str to assign none node.

Type:: str

document

Config document

Type:: dict

class dvas.config.config.Prm

Bases: OneDimArrayConfigManager

Parameter config manager

PARAMETER_PATTERN_PROP = {'^prm_cmt$': {'type': 'string'}, '^prm_desc$': {'type': 'string'}, '^prm_name$': {'type': 'string'}, '^prm_plot$': {'type': 'string'}, '^prm_unit$': {'type': 'string'}}

Parameter JSON 7 schema. Constant value.

Must be a dict like:

{"type": "object",
 "patternProperties": ANY,
 "additionalProperties": False
}

Type:: dict

LABEL_VAL_DEF = {'prm_cmt': '', 'prm_desc': '', 'prm_plot': '', 'prm_unit': ''}

Default values of labels

Type:: dict

CLASS_KEY = 'Prm'

Class key denomination

Type:: str

CONST_LABELS = []

Constant labels

Type:: list

NODE_GEN = 'prm_name'

Node name able to be generated by regexp. Use empty str to assign none node.

Type:: str

document

Config document

Type:: dict

class dvas.config.config.Flg

Bases: OneDimArrayConfigManager

Flg config manager

PARAMETER_PATTERN_PROP = {'^bit_pos$': {'maximum': 62, 'minimum': 0, 'type': 'integer'}, '^flg_desc$': {'type': 'string'}, '^flg_name$': {'type': 'string'}}

Parameter JSON 7 schema. Constant value.

Must be a dict like:

{"type": "object",
 "patternProperties": ANY,
 "additionalProperties": False
}

Type:: dict

LABEL_VAL_DEF = {}

Default values of labels

Type:: dict

CLASS_KEY = 'Flg'

Class key denomination

Type:: str

CONST_LABELS = [{'bit_pos': 0, 'flg_desc': 'No original data existing for this point (i.e. out of bound)', 'flg_name': 'no_original_data'}, {'bit_pos': 1, 'flg_desc': 'Invalid data (e.g. NaN value for real GDP measurement)', 'flg_name': 'is_invalid'}, {'bit_pos': 2, 'flg_desc': 'Invalid data (e.g. NaN uct value for real GDP gph measurement) that was corrected following official GRUAN instructions', 'flg_name': 'was_invalid'}, {'bit_pos': 3, 'flg_desc': 'Interpolated values', 'flg_name': 'is_interpolated'}, {'bit_pos': 4, 'flg_desc': 'GDPs are incompatible with one another', 'flg_name': 'is_GDP_incompatible'}, {'bit_pos': 5, 'flg_desc': 'A valid CWS measure exists for this measurement point', 'flg_name': 'has_valid_cws'}, {'bit_pos': 6, 'flg_desc': 'Prelaunch data', 'flg_name': 'is_prelaunch_data'}, {'bit_pos': 7, 'flg_desc': 'Ascent data', 'flg_name': 'is_ascent_data'}, {'bit_pos': 8, 'flg_desc': 'Descent data', 'flg_name': 'is_descent_data'}, {'bit_pos': 9, 'flg_desc': 'Measurement is in the PBL (i.e. x < PBLH)', 'flg_name': 'is_in_PBL'}, {'bit_pos': 10, 'flg_desc': 'Measurement is in the FT (i.e. x < tropopause and x > PBLH)', 'flg_name': 'is_in_FT'}, {'bit_pos': 11, 'flg_desc': 'Measurement is in the UTLS', 'flg_name': 'is_in_UTLS'}, {'bit_pos': 12, 'flg_desc': 'Measurement is in the MUS', 'flg_name': 'is_in_MUS'}]

Constant labels

Type:: list

NODE_GEN = ''

Node name able to be generated by regexp. Use empty str to assign none node.

Type:: str

document

Config document

Type:: dict

class dvas.config.config.Tag

Bases: OneDimArrayConfigManager

Tag config manager

PARAMETER_PATTERN_PROP = {'^tag_desc$': {'type': 'string'}, '^tag_name$': {'type': 'string'}}

Parameter JSON 7 schema. Constant value.

Must be a dict like:

{"type": "object",
 "patternProperties": ANY,
 "additionalProperties": False
}

Type:: dict

LABEL_VAL_DEF = {}

Default values of labels

Type:: dict

CLASS_KEY = 'Tag'

Class key denomination

Type:: str

CONST_LABELS = [{'tag_desc': 'None', 'tag_name': ''}, {'tag_desc': 'Original profile unaltered by dvas', 'tag_name': 'is_original'}, {'tag_desc': 'Cleaned-up profile', 'tag_name': 'is_cleaned'}, {'tag_desc': 'Profile is resampled onto a regular time-step grid', 'tag_name': 'is_resampled'}, {'tag_desc': 'Synchronized profile', 'tag_name': 'is_synchronized'}, {'tag_desc': 'GRUAN Data Product', 'tag_name': 'is_gdp'}, {'tag_desc': 'Combined working measurement standard', 'tag_name': 'is_cws'}, {'tag_desc': 'Profile minus CWS', 'tag_name': 'is_delta'}, {'tag_desc': 'Empty data', 'tag_name': 'is_empty'}]

Constant labels

Type:: list

NODE_GEN = 'tag_name'

Node name able to be generated by regexp. Use empty str to assign none node.

Type:: str

document

Config document

Type:: dict

class dvas.config.config.MultiLayerConfigManager

Bases: OneLayerConfigManager

Abstract class for managing ‘multi-layer’ YAML config.

REQUIRED_ATTRIBUTES = {'CLASS_KEY': <class 'str'>, 'DOC_TYPE': <class 'type'>, 'LABEL_VAL_DEF': <class 'dict'>, 'NODE_PATTERN': <class 'list'>, 'PARAMETER_PATTERN_PROP': <class 'dict'>}

DOC_TYPE: alias of dict

NODE_PATTERN = None

Node pattern (order matter)

Type:: list

get_val(node_labels, final_label)

Return single node_labels value

Parameters:

node_labels (list of str) – Node keys. If the escape character is missing in the prefix of the node, it is added automatically.
final_label (str) – Key parameter

Returns: object

Raises:: ConfigGetError – Error in getting config label value

get_all(node_labels)

Return all values for a given node labels. Only values specified in defaults labels will be returned.

Parameters:: node_labels (list of str) – Node keys
Returns:: dict
Raises:: ConfigGetError – Error in getting config label value

get_default(): Return all default values

property json_schema: JSON schema. Constant value

class dvas.config.config.OrigData

Bases: MultiLayerConfigManager

Original data config manager

PARAMETER_PATTERN_PROP = {'^csv_comment$': {'enum': ['#'], 'type': 'string'}, '^csv_delim_whitespace$': {'type': 'boolean'}, '^csv_delimiter$': {'anyOf': [{'type': 'null'}, {'type': 'string'}]}, '^csv_encoding$': {'anyOf': [{'type': 'null'}, {'type': 'string'}]}, '^csv_header$': {'oneOf': [{'type': 'null'}, {'minimum': 0, 'type': 'integer'}, {'const': 'infer'}]}, '^csv_index_col$': {'oneOf': [{'type': 'null'}, {'const': False}]}, '^csv_na_values$': {'anyOf': [{'type': 'null'}, {'type': 'string'}, {'items': {'type': 'string'}, 'minItems': 1, 'type': 'array', 'uniqueItems': True}]}, '^csv_skip_blank_lines$': {'type': 'boolean'}, '^csv_skipfooter$': {'minimum': 0, 'type': 'integer'}, '^csv_skipinitialspace$': {'type': 'boolean'}, '^csv_skiprows$': {'oneOf': [{'minimum': 0, 'type': 'integer'}, {'type': 'string'}]}, '^csv_use_default$': {'type': 'boolean'}, '^edt$': {'type': 'string'}, '^metadata$': {'oneOf': [{'type': 'null'}, {'additionalProperties': False, 'patternProperties': {'^[\\w\\.]+$': {'oneOf': [{'type': 'null'}, {'type': 'string'}, {'type': 'number'}]}}, 'type': 'object'}]}, '^pid$': {'type': 'string'}, '^srn$': {'type': 'string'}, '^tags$': {'items': {'type': 'string'}, 'type': 'array', 'uniqueItems': True}, '^value$': {'type': 'string'}}

Parameter JSON 7 schema. Constant value.

Must be a dict like:

{"type": "object",
 "patternProperties": ANY,
 "additionalProperties": False
}

Type:: dict

LABEL_VAL_DEF = {'csv_comment': '#', 'csv_delim_whitespace': False, 'csv_delimiter': ';', 'csv_encoding': 'utf_8', 'csv_header': 'infer', 'csv_index_col': None, 'csv_na_values': None, 'csv_skip_blank_lines': True, 'csv_skipfooter': 0, 'csv_skipinitialspace': False, 'csv_skiprows': 0, 'csv_use_default': False, 'metadata': {}, 'tags': []}

Default values of labels

Type:: dict

CLASS_KEY = 'OrigData'

Class key denomination

Type:: str

NODE_PATTERN = ['[a-zA-Z0-9]+(([a-zA-Z0-9])|([\\-\\_\$\$]))*[a-zA-Z0-9]', '(?:(([a-z0-9]+(([a-z0-9])|([\\_]))*(?:(?<!_flag)))|([a-z0-9]+(([a-z0-9])|([\\_]))*(?:(?<=_flag)))))']

Node pattern (order matter)

Type:: list

document

Config document

Type:: dict

class dvas.config.config.ConfigExprInterpreter

Bases: object

Abstract config expression interpreter class

Notes

This class and subclasses construction are based on the interpreter design pattern.

classmethod set_callable(fct): Set strategy :Parameters: fct (callable) – Function/Methode called by ‘get’ expression

abstract interpret(): Interpreter method

static eval(expr, get_fct)

Interpret expression.

Parameters:

expr (str|ConfigExprInterpreter) – Expression to evaluate.
get_fct (callable) – Function use by ‘get’

Syntax:: cat(<str_1>, …, <str_n>): Concatenate str_1 to str_n

Raises:: ExprInterpreterError – Error while interpreting expression

Examples

>>> import re
>>> mymatch = re.match('^a(\d)', 'a1b')
>>> print(ConfigExprInterpreter.eval("cat('My test', ' ', get(1))", mymatch.group))
My test 1

class dvas.config.config.NonTerminalConfigExprInterpreter(*args)

Bases: ConfigExprInterpreter

Implement an interpreter operation for non terminal symbols in the grammar.

interpret(): Non terminal interpreter method

abstract fct(*args): Function between expression args

class dvas.config.config.CatExpr(*args)

Bases: NonTerminalConfigExprInterpreter

String concatenation

fct(a, b): Implement fct method

class dvas.config.config.ReplExpr(*args)

Bases: NonTerminalConfigExprInterpreter

Replace dict key or list index by its value. If key is missing, return key

fct(a, b): Implement fct method

class dvas.config.config.ReplStrictExpr(*args)

Bases: NonTerminalConfigExprInterpreter

Replace dict key or list index by its value. If key is missing, return ‘’

fct(a, b): Implement fct method

class dvas.config.config.LowerExpr(*args)

Bases: NonTerminalConfigExprInterpreter

Lower case

fct(a): Implement fct method

class dvas.config.config.UpperExpr(*args)

Bases: NonTerminalConfigExprInterpreter

Upper case

fct(a): Implement fct method

class dvas.config.config.SmallUpperExpr(*args)

Bases: NonTerminalConfigExprInterpreter

Upper case 1st character

fct(a): Implement fct method

class dvas.config.config.ToDatetime(*args)

Bases: NonTerminalConfigExprInterpreter

Get str and convert to datetime value

fct(a): Convert a string to datetime

class dvas.config.config.TerminalConfigExprInterpreter(arg)

Bases: ConfigExprInterpreter

Implement an interpreter operation for terminal symbols in the grammar.

class dvas.config.config.GetExpr(arg, totype=None)

Bases: TerminalConfigExprInterpreter

Get catch value

__init__(arg, totype=None)

Init function

Parameters:

arg (object) – argument to get.
totype (type, optional) – if set, will convert the value to that type. Defaults to None = use native type.

interpret(): Implement fct method

class dvas.config.config.SplitSelect(arg, spl='_', sel=1)

Bases: GetExpr

Split a string a select one item

interpret(): Split string ‘arg’ using ‘spl’ and return item ‘sel’

class dvas.config.config.NoneExpr(arg)

Bases: TerminalConfigExprInterpreter

Apply none interpreter

interpret(): Implement fct method

dvas.config package

Subpackages

Submodules

dvas.config.config module