Computing difference fields (experimental)
==========================================

Brief summary
-------------

* Met.3D is able to perform simple difference computations to derive the difference of two data fields on-the-fly.
* Example uses include (but are not limited to) computation of tendency fields 
  (current time step minus time step X hours earlier) and or forecast error fields 
  (forecast data field minus analysis field).
* This functionality is still experimental (as of Met.3D 1.6/20.10.6). 
  It is not available through the GUI, only via user-defined pipeline.cfg files specified on the command line. 
  You will need some inside knowledge about internal Met.3D pipeline requests 
  (see Sect. 4.2 in `<https://doi.org/10.5194/gmd-8-2329-2015>`_).
* *Please let us know if this function is useful for you, and you would require further development.*


Description
-----------

Difference computations implemented as a "filter" in Met.3D's data pipeline 
(see Sect. 4.2 in `<https://doi.org/10.5194/gmd-8-2329-2015>`_). 
The filter can be configured in the "pipeline configuration file" passed to Met.3D on the command line 
(i.e. you do not load your data through the GUI but define your data source in advance to starting Met.3D).

* If you don't yet have such a file, copy the default pipeline file at 
  `<https://gitlab.com/wxmetvis/met.3d/-/blob/master/config/default_pipeline.cfg.template>`_.
* Start Met.3D from the command line with the argument ``–pipeline=yourpipelinefile.cfg``
  Datasets need to be specified using the ``--datasets`` command line argument, to be available
  for the difference computation, see :ref:`loading data from command line`.

In the pipeline configuration file, the difference module can be configured in the 
[ConfigurablePipeline] section that needs to be added to the file. 
The difference filter maps an incoming request to two "base requests" that are passed on to the 
difference filter's input sources.

As an example, your "pipeline.cfg" could include the following:

.. code-block:: 
    :linenos:
    
    [ConfigurablePipeline]
    size=2
    
    1\type=DIFFERENCE
    1\name=Difference to z(an)
    1\input1=ECMWF ENSEMBLE ENSFilter
    1\input2=ECMWF ANALYSIS ENSFilter
    1\baseRequest1="LEVELTYPE=REQUESTED_LEVELTYPE;VARIABLE=REQUESTED_VARIABLE;INIT_TIME=REQUESTED_INIT_TIME;VALID_TIME=REQUESTED_VALID_TIME;MEMBER=REQUESTED_MEMBER"
    1\baseRequest2="LEVELTYPE=1;VARIABLE=z;INIT_TIME=REQUESTED_VALID_TIME;VALID_TIME=REQUESTED_VALID_TIME;MEMBER=0"
    1\schedulerID=MultiThread
    1\memoryManagerID=NWP
    1\enableRegridding=false
    
    2\type=DIFFERENCE
    2\name=ECMWF NAWDEX PL ENSEMBLE 6h tendency
    2\input1=ECMWF NAWDEX ANALYSIS ENSFilter
    2\input2=ECMWF NAWDEX ANALYSIS ENSFilter
    2\baseRequest1="LEVELTYPE=REQUESTED_LEVELTYPE;VARIABLE=REQUESTED_VARIABLE;INIT_TIME=REQUESTED_INIT_TIME;VALID_TIME=REQUESTED_VALID_TIME;MEMBER=REQUESTED_MEMBER"
    2\baseRequest2="LEVELTYPE=REQUESTED_LEVELTYPE;VARIABLE=REQUESTED_VARIABLE;INIT_TIME=TIMEDIFF_-6_HOURS_REQUESTED_INIT_TIME;VALID_TIME=TIMEDIFF_-6_HOURS_REQUESTED_VALID_TIME;MEMBER=REQUESTED_MEMBER"
    2\schedulerID=MultiThread
    2\memoryManagerID=NWP
    2\enableRegridding=false


In the [ConfigurablePipeline] section, two filter instances are defined: 
(1) to compute the difference between the ensemble forecast and the analysis (to compute forecast error), 
and (2) to compute the difference between the requested valid and init times 
and the corresponding times six hours earlier (six-hour tendency).
Both specify two input fields, which specify the names of the input datasets.
These datasets need to be provided by the ``--datasets`` command line argument, that way they are loaded before
the configurable pipeline.

* Base requests need to define the pipeline keywords ``LEVELTYPE``, ``VARIABLE``, ``INIT_TIME``, 
  ``VALID_TIME``, ``MEMBER``.
* Values mapped to the keywords can

    * define explicit values, e.g. ``VARIABLE=z`` or ``MEMBER=0``
    * pass on a value in the incoming request, e.g. ``LEVELTYPE=REQUESTED_LEVELTYPE``
    * pass the value of the incoming request plus/minus a time difference applied, 
      e.g. ``INIT_TIME=TIMEDIFF_-6_HOURS_REQUESTED_INIT_TIME``

        * here, the ``-6`` can be any integer
        * units can be ``SECONDS``, ``MINUTES``, ``HOURS``, ``DAYS``, ``YEARS``


Code
----

Difference computation is implemented in ``MDifferenceDataSource``, located at

* `<https://gitlab.com/wxmetvis/met.3d/-/blob/master/src/data/differencedatasource.h>`_
* `<https://gitlab.com/wxmetvis/met.3d/-/blob/master/src/data/differencedatasource.cpp>`_

See the source code for further information about how the filter works.