Changelog
The dvas changelog (reproduced below) is stored in the code repository on Github.
All notable changes to dvas will be documented in this file.
The format is inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [vX.Y.Z]
### Added:
### Fixed:
### Changed:
### Deprecated:
### Removed:
### Security:
## [v1.1.0]
### Added:
- [fpavogt, 17.08.2023] Enable the ingestion of manual flag files by the uaii2022 recipe
- [fpavogt, 24.07.2023] Add DOI & pypi info to docs following first public release
### Fixed:
- [fpavogt, 07.09.2023] Fix #295
- [fpavogt, 11.08.2023] Fixes #288 [GDP SHC info is now extracted by dvas]
### Changed:
- [fpavogt, 11.08.2023] Request python>=3.11 and pandas>=2.0
- [fpavogt, 24.07.2023] Change Github Issues to use the new format
### Deprecated:
### Removed:
### Security:
## [v1.0.0]
### Added:
- [fpavogt, 24.07.2023] Add new pypi CI action for automated release
- [fpavogt, 19.05.2023] Ingest SHC info into the profile metadata
- [fpavogt, 16.06.2023] Update docs for v1.0.0
## [v0.9.5.dev0]
### Added:
- [fpavogt, 20.04.2023] Ingest and carry lat/lon data for GDPs and CWS (no uncertainties)
- [fpavogt, 18.04.2023] Introduce and deploy "wvec", a new variable for the Wind vector calculations
### Fixed:
- [fpavogt, 14.04.2023] Fix a bug associated to pandas 2.0.0 in fancy_bitwise_or()
### Changed:
- [fpavogt, 14.04.2023] Parameter names are now proper abbreviations (fixes #282)
- [fpavogt, 14.03.2023] Units of wdir are now typset properly in LaTeX as $^{\circ}$
- [fpavogt, 13.03.2023] Units of RH are now %RH
## [v0.9.4.dev0]
### Added:
- [fpavogt, 23.02.2023] Add first_timestep(.uncertainty) to the CWS netCDFs
- [fpavogt, 22.02.2023] Add new "comment" field to dvas prms (fix #277 wdir info)
### Fixed:
- [fpavogt, 22.02.2023] Fix #280 (optimize DB queries to avoid superfluous conditions)
- [fpavogt, 22.02.2023] Fix #278 and #275
- [fpavogt, 22.02.2023] Fix #276 (correct UTLS, add MUS, rm troposphere)
- [fpavogt, 22.02.2023] Fix #277 (gph AMLS)
- [fpavogt, 08.02.2023] Adjust pylint action for pylint 2.16
### Changed:
- [fpavogt, 08.02.2023] Break-up dot products to keep memory use low
## [v0.9.3.dev0]
### Fixed:
- [fpavogt, 08.02.2023] Adjust pylint action for pylint 2.16
### Changed:
- [fpavogt, 08.02.2023] Break-up dot products to keep memory use low
## [v0.9.2.dev0]
### Fixed:
- [fpavogt, 31.01.2023] Ignore NaNs when unwrapping angles (fix #273)
## [v0.9.1.dev0]
### Added:
- [fpavogt, 26.01.2023] Add option to store profile data on disk instead of the db
- [fpavogt, 23.01.2023] Add choice of mid to sync from val
### Fixed:
- [fpavogt, 17.01.2023] Left align ylabels accross suplots (fix #272)
## [v0.9.0.dev0]
### Added:
- [fpavogt, 2023-01-12] Add synop cloud code to GDP and CWS metadata
- [fpavogt, 2023-01-06] Add clean TOD option to biglambda recipe step (fixes #249)
- [fpavogt, 2023-01-05] Sync via global match can be restricted to a specific range of values (fix #269)
- [fpavogt, 2022-11-18] Add/connect UTLS, PBL, and Free Tropopause region flags
- [fpavogt, 2022-11-10] Add a WAS_INVALID flag to keep track of fixed GDP points, e.g. via #205
### Fixed:
- [fpavogt, 2022-11-22] Fix #266 by keeping track of KS alpha level in metadata of CWS
- [fpavogt, 2022-11-14] Include fid in plots and filenames (fix #261)
### Changed:
- [fpavogt, 2023-01-11] Refactor the corr. coeffs. routine to improve performances
- [fpavogt, 2022-11-10] Cleanup filename of KS plots
### Removed:
- [fpavogt, 2021-01-05] Removed 'ucr' in favor of 'ucu' (fixes #268)
## [v0.8.1.dev0]
### Fixed:
- [fpavogt, 2022-11-09] Fix #262 - handle angular wrapping in resample
- [fpavogt, 2022-11-09] Fix Runtime warning caused by uc_tot=0 cases in gdp_vs_cws plots
- [fpavogt, 2022-11-08] Fix #235 and #236 (wdir wrap-around)
## [v0.8.0.dev0]
### Added:
- [fpavogt, 2022-10-28] Add netCDF export recipe step
### Fixed:
- [fpavogt, 2022-10-31] Fix #253 (no more Int64 flags) and #259 (resampled points now inherit the anchor flags)
- [fpavogt, 2022-10-31] Fix #252 and #256
- [fpavogt, 2022-10-28] Fix #254
### Changed:
- [fpavogt, 2022-10-28] Clean-up TAGS and FLAGS, incl. rename of "raw" to "original" (fix #255)
## [v0.7.1]
### Added:
- [fpavogt, 2022-09-13] Add new INVALID flag for GDP points with NaN uncertainties (see #244)
- [fpavogt, 2022-08-19] Add kn2ms conversion function
- [fpavogt, 2022-08-05] Add support for the M20, iMS100, ATMS-3710, and WxR-301D sondes
- [fpavogt, 2022-07-13] Set resampled points to NaN if they are >=1s away from an original point (fix #233)
- [fpavogt, 2022-07-12] Add support for the PS01B3M sonde
- [fpavogt, 2022-07-12] Modify GetreldtExpr() to optionally round time stamps, and add dedicated test
- [fpavogt, 2022-07-12] Add support for the RS41-SG, the ATMS-3710, and the HT03G-1U sondes
- [fpavogt, 2022-07-07] Add support for the iMet-54-AA sonde
- [fpavogt, 2022-07-06] Add support for the DFM-17 sonde
- [fpavogt, 2022-07-01] Add subplot with relative contributions of GDPs to CWS in gdp-vs-cws (fix #226)
### Fixed:
- [fpavogt, 2022-10-07] Correct bug in RS92 GDP config file, cleanup demo script
- [fpavogt, 2022-09-29] Fix #242
- [fpavogt, 2022-08-23] Fix #241
- [fpavogt, 2022-08-22] Fix #240
- [fpavogt, 2022-08-05] Fix #238
- [fpavogt, 2022-07-15] Fix #232 - wdir curves no longer jump from 0 to 360 in plots
- [fpavogt, 2022-07-12] Deal with the floating point errors associated to pd.dt.total_seconds()
- [fpavogt, 2022-07-12] Fix typo in model mid field
### Changed:
- [fpavogt, 2022-08-19] Add new metadata fields for first, launch, and burst points (fix #231)
- [fpavogt, 2022-07-15] Changed InfoManager sorting rule to include mid (fix #234)
- [fpavogt, 2022-07-12] Check that time stamps are strictly increasing in the resampling strategy
- [fpavogt, 2022-07-01] Boost version to 0.7.0.dev0, after move to MeteoSwiss organization
## [v0.6.0.dev1]
### Added:
- [fpavogt, 2022-06-29] Cleanup the tropopause function, to support the WMO, MCH and GRUAN algorithm
- [fpavogt, 2022-06-07] Basic recipe step can now issue eid-edt files for the UAII plot preview software
- [fpavogt, 2022-06-03] Add new recipe step to flag zones of interest, compute the tropopause, etc ...
- [fpavogt, 2022-06-03] If no burst point is specified, flag any points beyond the max altitude as "descent"
- [fpavogt, 2022-05-25] Add participant_preview plot and associated recipe step
- [fpavogt, 2022-05-19] Allow DB to store timezone info with datetime fields
- [fpavogt, 2022-05-17] Add ability to skip recipe steps beyond a certain point
- [fpavogt, 2022-05-11] Add colors to on-screen logging
- [fpavogt, 2022-05-05] Fix #209 by returning full covariance matrices and plotting them
### Fixed:
- [fpavogt, 2022-06-30] Fix #169 - remove pampy dependancy
- [fpavogt, 2022-06-30] Fix #222 - prf.has_flg() now returns bool instead of int64
- [fpavogt, 2022-06-30] Fix #223 - add --debug option to dvas_run_recipe entry point
- [fpavogt, 2022-05-17] Fix wrong convention in get_shifts_from_alt
- [fpavogt, 2022-04-26] Fix #205 with a dedicated fix in the uaii2022.basic.cleanup recipe step
- [fpavogt, 2022-04-14] Fix #208 by improving dvas plots
### Changed:
- [fpavogt, 2022-07-01] Make 3.10 the minimum required Python version
- [fpavogt, 2022-07-01] Remove pytest-datafiles dependancy
- [fpavogt, 2022-06-10] Change day_night tag to timeofday, and make it "optional" by relying on the RS41 GDP to apply if to all
- [fpavogt, 2022-06-08] Refactor hardcoded variable names
- [fpavogt, 2022-05-30] Add official support for Python 3.10
- [fpavogt, 2022-05-25] Cleaned-up config files
- [fpavogt, 2022-05-25] Changed names of Parameter config file and associated db table
- [fpavogt, 2022-05-17] The flight overview plot is now using the index i for x (rather than time), to show the synced profiles
### Deprecated:
### Removed:
### Security:
## [v0.5.0] (2022-04-14)
### Added:
- [fpavogt, 2022-04-13] New CI action to check the code version is incremented for new releases
- [fpavogt, 2022-04-01] Connect `from_step_id` to recipe HL entry-point, incl. automatic disabling of DB reset
- [fpavogt, 2022-03-31] Fix-up assembly of Delta Profiles, and implement dedicated dtas plots
- [fpavogt, 2022-03-29] Add the `reset_db` recipe parameter to allow not systematically reseting it
- [fpavogt, 2022-03-21] Add new uc_budget diagnostic plot to dvas
- [fpavogt, 2022-03-16] Enable the cropping of descent data in the "cleanup" recipe step
- [fpavogt, 2022-03-15] Propagate flags through process_chunk(), weighted_means(), and delta()
- [fpavogt, 2022-03-08] Flag descent data in basic cleanup step. Add new 'descent' flag
- [fpavogt, 2022-03-03] Enable the resampling of profiles with a new cleanup recipe step
- [fpavogt, 2022-03-01] Add prf_summary recipe (fix #177)
- [modolol, 2022-02-28] Global DB extract
- [fpavogt, 2022-02-25] Add support for iMS-100 GDP beta 2
- [fpavogt, 2021-11-30] Add new "force-all-valid" strategy to combine GDPs
- [fpavogt, 2021-11-26] Add ability to feed lambda fcts to csv_skiprows
- [modolol, 2021-11-03] Add `mid` in DB search options
- [fpavogt, 2021-07-02] Add `has_tag()` convenience method to `Profile` and `MultiProfile` classes
### Fixed:
- [fpavogt, 2022-04-13] Fix #84
- [fpavogt, 2022-03-18] Fix #163
- [fpavogt, 2022-02-25] Fix #179
- [fpavogt, 2022-02-21] Fix #192
- [fpavogt, 2021-12-02] Fix issue #181
- [GonzagueRomanens, 2021-11-26] Fix issue #187
- [fpavogt, 2021-11-19] Fix issue #178
- [fpavogt, 2021-11-03] Fix issue #172
- [fpavogt, 2021-09-16] Fix issue #166 and #165
- [modolol, 2021-07-12] Fix issue #185
- [modolol, 2021-07-12] Fix issue #160
### Changed:
- [fpavogt, 2022-04-05] Adjust paths input structure of the dvas recipes, improve recipe tag handling
- [fpavogt, 2022-03-18] Modified & improved gdp_vs_cws plots into a dedicated recipe step
- [fpavogt, 2022-03-01] Simplified the logging by removing the specific names
- [fpavogt, 2022-03-01] Add dev option to setup.py
- [modolol, 2022-02-28] Remove mandatory tag creation in config
- [fpavogt, 2022-02-25] Remove all correlations between distinct GDP models
- [fpavogt, 2021-11-26] Update RS41 GDPs to v1
- [modolol, 2021-11-24] Modify 'header' rules in origdata
- [fpavogt, 2021-11-22] Various logging improvements
- [fpavogt, 2021-10-18] Clean-up default csv na values in origdata
## [v0.4.1] (2021-07-16)
### Added:
- [fpavogt, 2021-07-14] Add `csv_encoding` parameter to config files, and set it to 'altin_1' for RS-41
- [fpavogt, 2021-07-02] Add `has_tag()` convenience method to `Profile` and `MultiProfile` classes
### Fixed:
- [modolol, 2021-07-12] Fix issue #160
### Deprecated:
- [fpavogt, 2021-07-14] Make Python 3.9.0 and pandas 1.3.0 minimum required versions for dvas
## [v0.4.0] (2021-06-30)
### Added:
- [fpavogt, 2021-06-23] Add `get_validities()` routine to identify which levels to combine in CWS
- [fpavogt, 2021-06-16] Add `dvas_optimize` entry point to find ideal CHUNK_SIZE value
- [fpavogt, 2021-05-25] Add recipe_config.yml and associated initialization function for recipes
- [fpavogt, 2021-05-18] Add pytest fixture to test LaTeX plots using the command line argument "--latex"
- [modolol, 2021-05-11] Add interpreter to mix raw parameter in config file
- [fpavogt, 2021-04-19] Add config info for variables other than T, and for the M10 raw file
- [modolol, 2021-04-13] Add DB exploring tool
### Fixed:
- [modolol, 2021-06-10] Fix missing correction + bugs in PR #147
- [modolol, 2021-06-08] Cleanup towards v04 (config module)
### Changed:
- [fpavogt, 2021-06-30] Ensure that the default db config files are only found in`dvas_demo`
- [fpavogt, 2021-06-23] `has_flg()` now returns 0 if flags are <NA>
- [fpavogt, 2021-06-18] Rename `corcoefs()` as `coeffs()`, and move it to new `dvas.tools.gdps.correlations` module
- [fpavogt, 2021-05-18] Improve looks of plots, and split-up "orig_data_config.yml"
- [fpavogt, 2021-05-18] Change parameter names in tests, for consistency with main code
## [v0.3.0] (2021-04-12)
### Added:
- [fpavogt, 2021-03-15] Add extract() class method to MultiProfile
- [fpavogt, 2021-03-15] Make MultiProfile iterable using __getitem__() (see #114)
- [modolol, 2021-03-11] Import flags from CSV files
- [fpavogt, 2021-03-10] Add resampling strategy for RS and GDP Profiles
- [fpavogt, 2021-03-03] Add new gdps sub-module in dvas.tools
### Fixed:
- [fpavogt, 2021-04-01] Fixed sync routines following #129, now with dedicated tests
- [fpavogt, 2021-03-31] Bring the ks_test() routine back from the dead
- [fpavogt, 2021-03-31] Profile.__getattr__() now always returns a pd.Series, even for an index (#130)
- [fpavogt, 2021-03-05] Fix test_utils.py tests
- [modolol, 2020-02-25] Fix issue #113
- [modolol, 2020-02-17] Fix issue #96 (enhancement toward v0.3)
### Changed:
- [fpavogt, 2021-03-31] MultiProfile.get_prms() now returns a DataFrame instead of a list (#129)
- [fpavogt, 2021-03-29] Use masked arrays when dealing with Jacobians and correlation matrices to correctly handle NaNs
- [fpavogt, 2021-03-03] Re-design GDP routine to use bigger Profile chunks to speed things up
- [modolol, 2020-02-25] DB access methods
- [modolol, 2020-02-23] Change evt_dt -> edt
- [modolol, 2020-02-23] Change evt_id -> eid, rig_id -> rid
- [modolol, 2020-02-22] Change InstrType -> Model. Change mdl_id in tag -> mid from Model table
- [modolol, 2020-02-22] Add data source to InfoManager
## [v0.2.0] (2021-02-16)
### Added:
- [fpavogt, 2021-02-01] Add dvas_recipes package, to host the demo and high-level UAII routines
- [modolol, 2020-01-29] Add instrument, tag and parameter informations in MultiProfiles
- [modolol, 2020-12-29] Add metadata management in config and DB.
- [fpavogt, 2020-12-17] Add a rebase strategy and fix #83.
### Fixed:
- [modolol, 2020-02-11] Disconnect global db from each tests
- [fpavogt, 2021-02-02] Fix #102 & #87. Lift cap on pandas version.
- [modolol, 2020-01-29] Fix #94
- [fpavogt, 2020-12-21] Fix #60
- [fpavogt, 2020-12-15] Signature-changing "deepcopy" decorator now leads to correct decorated signatures (see #70).
### Changed:
- [fpavogt, 2021-02-05] Additional restructuring of dvas_recipes, to give users propers packages.
- [fpavogt, 2021-02-05] Renamed dvasError to DvasError.
- [modolol, 2020-01-29] Standardize table field names.
- [modolol, 2021-01-14] Removes instrument from the mandatory configuration (and all dependencies) + add product id to instrument
- [modolol, 2020-12-18] Modify logger
- [modolol, 2020-12-18] Add dvas.dvas facade
- [modolol, 2020-12-16] Move data.data.update_db to new module data.io
- [modolol, 2020-12-16] Modify config metadata interpreter syntax
- [modolol, 2020-12-15] Modify config generator interpreter syntax
### Security:
- [fpavogt, 2020-12-21] Add suggestion for contributors to sign their commits with GPG keys inside CONTRIBUTING.md
## [v0.1.1] (2020-12-14)
### Fixed:
- [fpavogt, 2020-12-14] Fix auto-publications of docs + minor code tweaks
## [v0.1.0] (2020-12-14)
### Added:
- [GonzagueRomanens, 2020-12-19] add SampleDatasetMultipayload
- [fpavogt & modolol, 2020-12-07] Cleanup towards v0.1
- [fpavogt & modolol, 2020-12-03] Winter cleanup towards v0.1
- [fpavogt, 2020-11-05] Refactor ProfileManager class into Profile, grouping 'alt', 'val' , and 'flag' in a pandas DataFrame
- [modolol, 2020-11-02] Tropopause calculation function
- [fpavogt, 2020-07-23] Add GRUAN submodule, with GDP correlation rules
- [modolol, 2020-07-16] Add dummy examples
- [modolol, 2020-07-03] .dev_utils directory in package
- [fpavogt, 2020-06-30] Add docs CI & a basic netCDF4 test
- [modolol, 2020-06-12] Add tags table in DB and config items generator
- [fpavogt, 2020-06-10] Create a plot submodule, including dedicated style sheets
- [modolol, 2020-05-25] Add logging possibilities
- [modolol, 2020-05-18] Add pytest-env, pytest-datafiles in pip install CI_pytest.yml
- [fpavogt, 2020-05-14] Github Action to check if this CHANGELOG gets modified :)
- [fpavogt, 2020-05-14] Add README, CODE_OF_CONDUCT, CONTRIBUTING files
- [fpavogt, 2020-05-07] Github Actions to validate syntax using pylint and run pytests
- [fpavogt, 2020-05-07] CHANGELOG file based on https://keepachangelog.com
- [fpavogt, 2020-05-07] Sphinx documentation with primary pages and todo list
### Changed:
- [modolol, 2020-11-02] Data classes types
- [modolol, 2020-07-16] load data search syntax
- [modolol, 2020-07-14] Use design pattern chain of responsibility for update_db
- [fpavogt, 2020-07-02] Set GPL v3.0 license
- [modolol, 2020-06-23] Remove instr_id. Use SN instead (adaptation for GPD integration)
- [modolol, 2020-06-22] Licence and copyright files headers
- [modolol, 2020-06-18] Modify TypedProperty to use pampy PyPI package
- [modolol, 2020-05-18] Adapted setup.py and CI_pylinter.yml
### Fixed:
- [modolol, 2020-06-17] Fix bug in data loading (dvas.database.database)
- [modolol, 2020-06-17] Remove automatic make dir in module import and global path assignment
- [fpavogt, 2020-06-12] Automatically load the `base` style sheet for plots & new tests
- [modolol, 2020-06-12] Set os.chmod(777) -> skip for Windows and check for R/W for Mac and Linux
- [modolol, 2020-05-25] Adapt for passing multi OS_CI
- [modolol, 2020-05-18] Code cleaning (bis)
- [modolol, 2020-05-15] Code cleaning