Changelog

last-commit issues

The dvas changelog (reproduced below) is stored in the code repository on Github.

All notable changes to dvas will be documented in this file.

The format is inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).


## [vX.Y.Z]
### Added:
### Fixed:
### Changed:
### Deprecated:
### Removed:
### Security:

## [v1.1.0]
### Added:
 - [fpavogt, 17.08.2023] Enable the ingestion of manual flag files by the uaii2022 recipe
 - [fpavogt, 24.07.2023] Add DOI & pypi info to docs following first public release
### Fixed:
 - [fpavogt, 07.09.2023] Fix #295
 - [fpavogt, 11.08.2023] Fixes #288 [GDP SHC info is now extracted by dvas]
### Changed:
 - [fpavogt, 11.08.2023] Request python>=3.11 and pandas>=2.0
 - [fpavogt, 24.07.2023] Change Github Issues to use the new format
### Deprecated:
### Removed:
### Security:

## [v1.0.0]
### Added:
 - [fpavogt, 24.07.2023] Add new pypi CI action for automated release
 - [fpavogt, 19.05.2023] Ingest SHC info into the profile metadata
 - [fpavogt, 16.06.2023] Update docs for v1.0.0

## [v0.9.5.dev0]
### Added:
 - [fpavogt, 20.04.2023] Ingest and carry lat/lon data for GDPs and CWS (no uncertainties)
 - [fpavogt, 18.04.2023] Introduce and deploy "wvec", a new variable for the Wind vector calculations
### Fixed:
 - [fpavogt, 14.04.2023] Fix a bug associated to pandas 2.0.0 in fancy_bitwise_or()
### Changed:
 - [fpavogt, 14.04.2023] Parameter names are now proper abbreviations (fixes #282)
 - [fpavogt, 14.03.2023] Units of wdir are now typset properly in LaTeX as $^{\circ}$
 - [fpavogt, 13.03.2023] Units of RH are now %RH

## [v0.9.4.dev0]
### Added:
 - [fpavogt, 23.02.2023] Add first_timestep(.uncertainty) to the CWS netCDFs
 - [fpavogt, 22.02.2023] Add new "comment" field to dvas prms (fix #277 wdir info)
### Fixed:
 - [fpavogt, 22.02.2023] Fix #280 (optimize DB queries to avoid superfluous conditions)
 - [fpavogt, 22.02.2023] Fix #278 and #275
 - [fpavogt, 22.02.2023] Fix #276 (correct UTLS, add MUS, rm troposphere)
 - [fpavogt, 22.02.2023] Fix #277 (gph AMLS)
 - [fpavogt, 08.02.2023] Adjust pylint action for pylint 2.16
### Changed:
 - [fpavogt, 08.02.2023] Break-up dot products to keep memory use low

## [v0.9.3.dev0]
### Fixed:
 - [fpavogt, 08.02.2023] Adjust pylint action for pylint 2.16
### Changed:
 - [fpavogt, 08.02.2023] Break-up dot products to keep memory use low

## [v0.9.2.dev0]
### Fixed:
 - [fpavogt, 31.01.2023] Ignore NaNs when unwrapping angles (fix #273)

## [v0.9.1.dev0]
### Added:
 - [fpavogt, 26.01.2023] Add option to store profile data on disk instead of the db
 - [fpavogt, 23.01.2023] Add choice of mid to sync from val
### Fixed:
 - [fpavogt, 17.01.2023] Left align ylabels accross suplots (fix #272)

## [v0.9.0.dev0]
### Added:
 - [fpavogt, 2023-01-12] Add synop cloud code to GDP and CWS metadata
 - [fpavogt, 2023-01-06] Add clean TOD option to biglambda recipe step (fixes #249)
 - [fpavogt, 2023-01-05] Sync via global match can be restricted to a specific range of values (fix #269)
 - [fpavogt, 2022-11-18] Add/connect UTLS, PBL, and Free Tropopause region flags
 - [fpavogt, 2022-11-10] Add a WAS_INVALID flag to keep track of fixed GDP points, e.g. via #205
### Fixed:
 - [fpavogt, 2022-11-22] Fix #266 by keeping track of KS alpha level in metadata of CWS
 - [fpavogt, 2022-11-14] Include fid in plots and filenames (fix #261)
### Changed:
 - [fpavogt, 2023-01-11] Refactor the corr. coeffs. routine to improve performances
 - [fpavogt, 2022-11-10] Cleanup filename of KS plots
### Removed:
 - [fpavogt, 2021-01-05] Removed 'ucr' in favor of 'ucu' (fixes #268)

## [v0.8.1.dev0]
### Fixed:
 - [fpavogt, 2022-11-09] Fix #262 - handle angular wrapping in resample
 - [fpavogt, 2022-11-09] Fix Runtime warning caused by uc_tot=0 cases in gdp_vs_cws plots
 - [fpavogt, 2022-11-08] Fix #235 and #236 (wdir wrap-around)

## [v0.8.0.dev0]
### Added:
 - [fpavogt, 2022-10-28] Add netCDF export recipe step
### Fixed:
 - [fpavogt, 2022-10-31] Fix #253 (no more Int64 flags) and #259 (resampled points now inherit the anchor flags)
 - [fpavogt, 2022-10-31] Fix #252 and #256
 - [fpavogt, 2022-10-28] Fix #254
### Changed:
 - [fpavogt, 2022-10-28] Clean-up TAGS and FLAGS, incl. rename of "raw" to "original" (fix #255)

## [v0.7.1]
### Added:
 - [fpavogt, 2022-09-13] Add new INVALID flag for GDP points with NaN uncertainties (see #244)
 - [fpavogt, 2022-08-19] Add kn2ms conversion function
 - [fpavogt, 2022-08-05] Add support for the M20, iMS100, ATMS-3710, and WxR-301D sondes
 - [fpavogt, 2022-07-13] Set resampled points to NaN if they are >=1s away from an original point (fix #233)
 - [fpavogt, 2022-07-12] Add support for the PS01B3M sonde
 - [fpavogt, 2022-07-12] Modify GetreldtExpr() to optionally round time stamps, and add dedicated test
 - [fpavogt, 2022-07-12] Add support for the RS41-SG, the ATMS-3710, and the HT03G-1U sondes
 - [fpavogt, 2022-07-07] Add support for the iMet-54-AA sonde
 - [fpavogt, 2022-07-06] Add support for the DFM-17 sonde
 - [fpavogt, 2022-07-01] Add subplot with relative contributions of GDPs to CWS in gdp-vs-cws (fix #226)
### Fixed:
 - [fpavogt, 2022-10-07] Correct bug in RS92 GDP config file, cleanup demo script
 - [fpavogt, 2022-09-29] Fix #242
 - [fpavogt, 2022-08-23] Fix #241
 - [fpavogt, 2022-08-22] Fix #240
 - [fpavogt, 2022-08-05] Fix #238
 - [fpavogt, 2022-07-15] Fix #232 - wdir curves no longer jump from 0 to 360 in plots
 - [fpavogt, 2022-07-12] Deal with the floating point errors associated to pd.dt.total_seconds()
 - [fpavogt, 2022-07-12] Fix typo in model mid field
### Changed:
 - [fpavogt, 2022-08-19] Add new metadata fields for first, launch, and burst points (fix #231)
 - [fpavogt, 2022-07-15] Changed InfoManager sorting rule to include mid (fix #234)
 - [fpavogt, 2022-07-12] Check that time stamps are strictly increasing in the resampling strategy
 - [fpavogt, 2022-07-01] Boost version to 0.7.0.dev0, after move to MeteoSwiss organization

## [v0.6.0.dev1]
### Added:
 - [fpavogt, 2022-06-29] Cleanup the tropopause function, to support the WMO, MCH and GRUAN algorithm
 - [fpavogt, 2022-06-07] Basic recipe step can now issue eid-edt files for the UAII plot preview software
 - [fpavogt, 2022-06-03] Add new recipe step to flag zones of interest, compute the tropopause, etc ...
 - [fpavogt, 2022-06-03] If no burst point is specified, flag any points beyond the max altitude as "descent"
 - [fpavogt, 2022-05-25] Add participant_preview plot and associated recipe step
 - [fpavogt, 2022-05-19] Allow DB to store timezone info with datetime fields
 - [fpavogt, 2022-05-17] Add ability to skip recipe steps beyond a certain point
 - [fpavogt, 2022-05-11] Add colors to on-screen logging
 - [fpavogt, 2022-05-05] Fix #209 by returning full covariance matrices and plotting them
### Fixed:
 - [fpavogt, 2022-06-30] Fix #169 - remove pampy dependancy
 - [fpavogt, 2022-06-30] Fix #222 - prf.has_flg() now returns bool instead of int64
 - [fpavogt, 2022-06-30] Fix #223 - add --debug option to dvas_run_recipe entry point
 - [fpavogt, 2022-05-17] Fix wrong convention in get_shifts_from_alt
 - [fpavogt, 2022-04-26] Fix #205 with a dedicated fix in the uaii2022.basic.cleanup recipe step
 - [fpavogt, 2022-04-14] Fix #208 by improving dvas plots
### Changed:
 - [fpavogt, 2022-07-01] Make 3.10 the minimum required Python version
 - [fpavogt, 2022-07-01] Remove pytest-datafiles dependancy
 - [fpavogt, 2022-06-10] Change day_night tag to timeofday, and make it "optional" by relying on the RS41 GDP to apply if to all
 - [fpavogt, 2022-06-08] Refactor hardcoded variable names
 - [fpavogt, 2022-05-30] Add official support for Python 3.10
 - [fpavogt, 2022-05-25] Cleaned-up config files
 - [fpavogt, 2022-05-25] Changed names of Parameter config file and associated db table
 - [fpavogt, 2022-05-17] The flight overview plot is now using the index i for x (rather than time), to show the synced profiles
### Deprecated:
### Removed:
### Security:

## [v0.5.0] (2022-04-14)
### Added:
 - [fpavogt, 2022-04-13] New CI action to check the code version is incremented for new releases
 - [fpavogt, 2022-04-01] Connect `from_step_id` to recipe HL entry-point, incl. automatic disabling of DB reset
 - [fpavogt, 2022-03-31] Fix-up assembly of Delta Profiles, and implement dedicated dtas plots
 - [fpavogt, 2022-03-29] Add the `reset_db` recipe parameter to allow not systematically reseting it
 - [fpavogt, 2022-03-21] Add new uc_budget diagnostic plot to dvas
 - [fpavogt, 2022-03-16] Enable the cropping of descent data in the "cleanup" recipe step
 - [fpavogt, 2022-03-15] Propagate flags through process_chunk(), weighted_means(), and delta()
 - [fpavogt, 2022-03-08] Flag descent data in basic cleanup step. Add new 'descent' flag
 - [fpavogt, 2022-03-03] Enable the resampling of profiles with a new cleanup recipe step
 - [fpavogt, 2022-03-01] Add prf_summary recipe (fix #177)
 - [modolol, 2022-02-28] Global DB extract
 - [fpavogt, 2022-02-25] Add support for iMS-100 GDP beta 2
 - [fpavogt, 2021-11-30] Add new "force-all-valid" strategy to combine GDPs
 - [fpavogt, 2021-11-26] Add ability to feed lambda fcts to csv_skiprows
 - [modolol, 2021-11-03] Add `mid` in DB search options
 - [fpavogt, 2021-07-02] Add `has_tag()` convenience method to `Profile` and `MultiProfile` classes
### Fixed:
 - [fpavogt, 2022-04-13] Fix #84
 - [fpavogt, 2022-03-18] Fix #163
 - [fpavogt, 2022-02-25] Fix #179
 - [fpavogt, 2022-02-21] Fix #192
 - [fpavogt, 2021-12-02] Fix issue #181
 - [GonzagueRomanens, 2021-11-26] Fix issue #187
 - [fpavogt, 2021-11-19] Fix issue #178
 - [fpavogt, 2021-11-03] Fix issue #172
 - [fpavogt, 2021-09-16] Fix issue #166 and #165
 - [modolol, 2021-07-12] Fix issue #185
 - [modolol, 2021-07-12] Fix issue #160
### Changed:
 - [fpavogt, 2022-04-05] Adjust paths input structure of the dvas recipes, improve recipe tag handling
 - [fpavogt, 2022-03-18] Modified & improved gdp_vs_cws plots into a dedicated recipe step
 - [fpavogt, 2022-03-01] Simplified the logging by removing the specific names
 - [fpavogt, 2022-03-01] Add dev option to setup.py
 - [modolol, 2022-02-28] Remove mandatory tag creation in config
 - [fpavogt, 2022-02-25] Remove all correlations between distinct GDP models
 - [fpavogt, 2021-11-26] Update RS41 GDPs to v1
 - [modolol, 2021-11-24] Modify 'header' rules in origdata
 - [fpavogt, 2021-11-22] Various logging improvements
 - [fpavogt, 2021-10-18] Clean-up default csv na values in origdata

## [v0.4.1] (2021-07-16)
### Added:
 - [fpavogt, 2021-07-14] Add `csv_encoding` parameter to config files, and set it to 'altin_1' for RS-41
 - [fpavogt, 2021-07-02] Add `has_tag()` convenience method to `Profile` and `MultiProfile` classes
### Fixed:
 - [modolol, 2021-07-12] Fix issue #160
### Deprecated:
 - [fpavogt, 2021-07-14] Make Python 3.9.0 and pandas 1.3.0 minimum required versions for dvas

## [v0.4.0] (2021-06-30)
### Added:
 - [fpavogt, 2021-06-23] Add `get_validities()` routine to identify which levels to combine in CWS
 - [fpavogt, 2021-06-16] Add `dvas_optimize` entry point to find ideal CHUNK_SIZE value
 - [fpavogt, 2021-05-25] Add recipe_config.yml and associated initialization function for recipes
 - [fpavogt, 2021-05-18] Add pytest fixture to test LaTeX plots using the command line argument "--latex"
 - [modolol, 2021-05-11] Add interpreter to mix raw parameter in config file
 - [fpavogt, 2021-04-19] Add config info for variables other than T, and for the M10 raw file
 - [modolol, 2021-04-13] Add DB exploring tool
### Fixed:
 - [modolol, 2021-06-10] Fix missing correction + bugs in PR #147
 - [modolol, 2021-06-08] Cleanup towards v04 (config module)
### Changed:
 - [fpavogt, 2021-06-30] Ensure that the default db config files are only found in`dvas_demo`
 - [fpavogt, 2021-06-23] `has_flg()` now returns 0 if flags are <NA>
 - [fpavogt, 2021-06-18] Rename `corcoefs()` as `coeffs()`, and move it to new `dvas.tools.gdps.correlations` module
 - [fpavogt, 2021-05-18] Improve looks of plots, and split-up "orig_data_config.yml"
 - [fpavogt, 2021-05-18] Change parameter names in tests, for consistency with main code

## [v0.3.0] (2021-04-12)
### Added:
 - [fpavogt, 2021-03-15] Add extract() class method to MultiProfile
 - [fpavogt, 2021-03-15] Make MultiProfile iterable using __getitem__() (see #114)
 - [modolol, 2021-03-11] Import flags from CSV files
 - [fpavogt, 2021-03-10] Add resampling strategy for RS and GDP Profiles
 - [fpavogt, 2021-03-03] Add new gdps sub-module in dvas.tools
### Fixed:
 - [fpavogt, 2021-04-01] Fixed sync routines following #129, now with dedicated tests
 - [fpavogt, 2021-03-31] Bring the ks_test() routine back from the dead
 - [fpavogt, 2021-03-31] Profile.__getattr__() now always returns a pd.Series, even for an index (#130)
 - [fpavogt, 2021-03-05] Fix test_utils.py tests
 - [modolol, 2020-02-25] Fix issue #113
 - [modolol, 2020-02-17] Fix issue #96 (enhancement toward v0.3)
### Changed:
 - [fpavogt, 2021-03-31] MultiProfile.get_prms() now returns a DataFrame instead of a list (#129)
 - [fpavogt, 2021-03-29] Use masked arrays when dealing with Jacobians and correlation matrices to correctly handle NaNs
 - [fpavogt, 2021-03-03] Re-design GDP routine to use bigger Profile chunks to speed things up
 - [modolol, 2020-02-25] DB access methods
 - [modolol, 2020-02-23] Change evt_dt -> edt
 - [modolol, 2020-02-23] Change evt_id -> eid, rig_id -> rid
 - [modolol, 2020-02-22] Change InstrType -> Model. Change mdl_id in tag -> mid from Model table
 - [modolol, 2020-02-22] Add data source to InfoManager

## [v0.2.0] (2021-02-16)
### Added:
 - [fpavogt, 2021-02-01] Add dvas_recipes package, to host the demo and high-level UAII routines
 - [modolol, 2020-01-29] Add instrument, tag and parameter informations in MultiProfiles
 - [modolol, 2020-12-29] Add metadata management in config and DB.
 - [fpavogt, 2020-12-17] Add a rebase strategy and fix #83.
### Fixed:
 - [modolol, 2020-02-11] Disconnect global db from each tests
 - [fpavogt, 2021-02-02] Fix #102 & #87. Lift cap on pandas version.
 - [modolol, 2020-01-29] Fix #94
 - [fpavogt, 2020-12-21] Fix #60
 - [fpavogt, 2020-12-15] Signature-changing "deepcopy" decorator now leads to correct decorated signatures (see #70).
### Changed:
 - [fpavogt, 2021-02-05] Additional restructuring of dvas_recipes, to give users propers packages.
 - [fpavogt, 2021-02-05] Renamed dvasError to DvasError.
 - [modolol, 2020-01-29] Standardize table field names.
 - [modolol, 2021-01-14] Removes instrument from the mandatory configuration (and all dependencies) + add product id to instrument
 - [modolol, 2020-12-18] Modify logger
 - [modolol, 2020-12-18] Add dvas.dvas facade
 - [modolol, 2020-12-16] Move data.data.update_db to new module data.io
 - [modolol, 2020-12-16] Modify config metadata interpreter syntax
 - [modolol, 2020-12-15] Modify config generator interpreter syntax
### Security:
 - [fpavogt, 2020-12-21] Add suggestion for contributors to sign their commits with GPG keys inside CONTRIBUTING.md

## [v0.1.1] (2020-12-14)
### Fixed:
 - [fpavogt, 2020-12-14] Fix auto-publications of docs + minor code tweaks

## [v0.1.0] (2020-12-14)
### Added:
 - [GonzagueRomanens, 2020-12-19] add SampleDatasetMultipayload
 - [fpavogt & modolol, 2020-12-07] Cleanup towards v0.1
 - [fpavogt & modolol, 2020-12-03] Winter cleanup towards v0.1
 - [fpavogt, 2020-11-05] Refactor ProfileManager class into Profile, grouping 'alt', 'val' , and 'flag' in a pandas DataFrame
 - [modolol, 2020-11-02] Tropopause calculation function
 - [fpavogt, 2020-07-23] Add GRUAN submodule, with GDP correlation rules
 - [modolol, 2020-07-16] Add dummy examples
 - [modolol, 2020-07-03] .dev_utils directory in package
 - [fpavogt, 2020-06-30] Add docs CI & a basic netCDF4 test
 - [modolol, 2020-06-12] Add tags table in DB and config items generator
 - [fpavogt, 2020-06-10] Create a plot submodule, including dedicated style sheets
 - [modolol, 2020-05-25] Add logging possibilities
 - [modolol, 2020-05-18] Add pytest-env, pytest-datafiles in pip install CI_pytest.yml
 - [fpavogt, 2020-05-14] Github Action to check if this CHANGELOG gets modified :)
 - [fpavogt, 2020-05-14] Add README, CODE_OF_CONDUCT, CONTRIBUTING files
 - [fpavogt, 2020-05-07] Github Actions to validate syntax using pylint and run pytests
 - [fpavogt, 2020-05-07] CHANGELOG file based on https://keepachangelog.com
 - [fpavogt, 2020-05-07] Sphinx documentation with primary pages and todo list

### Changed:
 - [modolol, 2020-11-02] Data classes types
 - [modolol, 2020-07-16] load data search syntax
 - [modolol, 2020-07-14] Use design pattern chain of responsibility for update_db
 - [fpavogt, 2020-07-02] Set GPL v3.0 license
 - [modolol, 2020-06-23] Remove instr_id. Use SN instead (adaptation for GPD integration)
 - [modolol, 2020-06-22] Licence and copyright files headers
 - [modolol, 2020-06-18] Modify TypedProperty to use pampy PyPI package
 - [modolol, 2020-05-18] Adapted setup.py and CI_pylinter.yml

### Fixed:
 - [modolol, 2020-06-17] Fix bug in data loading (dvas.database.database)
 - [modolol, 2020-06-17] Remove automatic make dir in module import and global path assignment
 - [fpavogt, 2020-06-12] Automatically load the `base` style sheet for plots & new tests
 - [modolol, 2020-06-12] Set os.chmod(777) -> skip for Windows and check for R/W for Mac and Linux
 - [modolol, 2020-05-25] Adapt for passing multi OS_CI
 - [modolol, 2020-05-18] Code cleaning (bis)
 - [modolol, 2020-05-15] Code cleaning