{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retrieving MeteoSwiss products from CSCS with pyrad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most MeteoSwiss products can be retrieved from the CSCS or MeteoSwiss servers using the functions documented [here](https://meteoswiss.github.io/pyrad/API/generated/pyrad.util.html#data-retrieval-utilities).\n",
    "\n",
    "The reading/writing functions for these files are available in the [MeteoSwiss Py-ART fork](https://github.com/MeteoSwiss/pyart). Please also check the following [notebook](https://meteoswiss.github.io/pyart/notebooks/read_mch_metranet_data.html) for an example of reading/writing such files.\n",
    "\n",
    "Please also check the [internal MeteoSwiss confluence page]()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Most products can be fetched with the *retrieve_mch_prod* function. You need to specify an output directory, make sure there is sufficient space in the output directory before you run the command!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: reading polar data from Albis at low-resolution (500m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "60\n",
      "/scratch/wolfensb//MLA2413110100U.001\n",
      "dict_keys(['reflectivity', 'signal_to_noise_ratio', 'reflectivity_vv', 'differential_reflectivity', 'uncorrected_cross_correlation_ratio', 'uncorrected_differential_phase', 'velocity', 'spectrum_width', 'reflectivity_hh_clut'])\n"
     ]
    }
   ],
   "source": [
    "from pyrad.util import retrieve_mch_prod\n",
    "import pyart\n",
    "import datetime\n",
    "\n",
    "OUTPUT_DIRECTORY = '/scratch/wolfensb/temp/' # Adjust to your needs\n",
    "T0 = datetime.datetime(2024,5,10,10,10)\n",
    "T1 = datetime.datetime(2024,5,10,10,20)\n",
    "\n",
    "files = retrieve_mch_prod(OUTPUT_DIRECTORY ,T0, T1,  product_name='MLA')\n",
    "\n",
    "print(len(files))\n",
    "print(files[0])\n",
    "\n",
    "radar = pyart.aux_io.read_metranet(files[0])\n",
    "print(radar.fields.keys())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: same as example 1 but getting only certain sweeps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12\n",
      "[np.str_('/scratch/wolfensb//MLA2413110100U.001'), np.str_('/scratch/wolfensb//MLA2413110100U.002'), np.str_('/scratch/wolfensb//MLA2413110100U.003'), np.str_('/scratch/wolfensb//MLA2413110100U.004')]\n"
     ]
    }
   ],
   "source": [
    "files = retrieve_mch_prod(OUTPUT_DIRECTORY ,T0, T1, product_name='MLA', sweeps = [1,2,3,4])\n",
    "print(len(files))\n",
    "print(files[0:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: getting a Cartesian product, RZC = radar QPE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "[np.str_('/scratch/wolfensb//RZC241311010VL.001'), np.str_('/scratch/wolfensb//RZC241311012VL.001'), np.str_('/scratch/wolfensb//RZC241311015VL.001'), np.str_('/scratch/wolfensb//RZC241311017VL.001'), np.str_('/scratch/wolfensb//RZC241311020VL.001')]\n"
     ]
    }
   ],
   "source": [
    "files = retrieve_mch_prod(OUTPUT_DIRECTORY ,T0, T1,  product_name='RZC')\n",
    "print(len(files))\n",
    "print(files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 4: getting a Cartesian product, CPC = radar-gauge QPE, but only at hourly accumulation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "[np.str_('/scratch/wolfensb//CPC/CPC2413110100_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110200_00060.801.gif')]\n"
     ]
    }
   ],
   "source": [
    "files = retrieve_mch_prod(OUTPUT_DIRECTORY ,T0, T1,  product_name='CPC', pattern='*00060*')\n",
    "print(len(files))\n",
    "print(files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 5: more complex filtering of retrieved files with regex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Extracting all CPC files except those at 5 minute resolution (CPC*_00005.801.gif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[np.str_('/scratch/wolfensb//CPC/CPC2413108100_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108100_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108100_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108100_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108100_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108200_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108200_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108200_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108200_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108200_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108300_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108300_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108300_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108300_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108300_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108400_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108400_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108400_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108400_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108400_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108500_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108500_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108500_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108500_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413108500_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109000_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109000_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109000_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109000_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109000_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109001_02880.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109003_04320.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109100_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109100_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109100_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109100_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109100_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109200_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109200_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109200_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109200_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109200_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109300_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109300_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109300_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109300_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109300_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109400_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109400_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109400_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109400_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109400_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109500_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109500_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109500_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109500_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413109500_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110000_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110000_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110000_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110000_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110000_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110001_02880.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110003_04320.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110100_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110100_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110100_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110100_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110100_01440.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110200_00060.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110200_00180.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110200_00360.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110200_00720.801.gif'), np.str_('/scratch/wolfensb//CPC/CPC2413110200_01440.801.gif')]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<>:3: SyntaxWarning: invalid escape sequence '\\.'\n",
      "<>:3: SyntaxWarning: invalid escape sequence '\\.'\n",
      "/tmp/ipykernel_136299/2933435368.py:3: SyntaxWarning: invalid escape sequence '\\.'\n",
      "  files = retrieve_mch_prod('/scratch/wolfensb/',T0, T1,  product_name='CPC', pattern='^(?!.*00005\\.801\\.gif$).*\\.gif$', pattern_type='regex')\n"
     ]
    }
   ],
   "source": [
    "T0 = datetime.datetime(2024,5,10,8,10)\n",
    "T1 = datetime.datetime(2024,5,10,10,20)\n",
    "files = retrieve_mch_prod(OUTPUT_DIRECTORY ,T0, T1,  product_name='CPC', pattern='^(?!.*00005\\.801\\.gif$).*\\.gif$', pattern_type='regex')\n",
    "\n",
    "print(files)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pyart_new",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}