Gridded data (horizontally regular) =================================== Met.3D supports gridded, structured data with a **regular longitude-latitude grid** in the horizontal, and either **levels of constant pressure** (in the following "pressure levels") or **hybrid sigma-pressure levels** (as used, e.g., by the ECMWF integrated forecast system) in the vertical. Internally, a single 2D or 3D scalar data field (i.e., one forecast variable, one time step, one ensemble member) is treated as the smallest data entity. A single data field is stored in a class derived from :code:`MStructuredGrid`, these objects store all contextual information required to visualise the field (including latitude and longitude coordinates, pressure or hybrid level, time step and additional metadata). Example of derived classes are :code:`MRegularLonLatGrid` for 2D fields, :code:`MRegularLonLatStructuredPressureGrid` for 3D fields on pressure levels, :code:`MLonLatHybridSigmaPressureGrid` for 3D fields on hybrid sigma-pressure levels. Time series and ensemble sets are composed of these basic data field entities. .. Should internal data handling really be described with c++ classes in the user documentation? That should rather move to the developer documentation and user documentation should be simplified. Gridded data in NetCDF format ----------------------------- Gridded, structured data can be read from NetCDF files that follow the `Climate and Forecast (CF) Metadata Conventions`_. For more information on how to load these, see :doc:`../data_handling/loading_data`. .. note:: Forecast variables, time steps and ensemble members can be arbitrarily distributed over the files that match the file filter. You can store all ensemble members in one file but have different files for each time step, or vice versa. Or everything can be stored in a single file. Examples of how NetCDF files need to be structured -------------------------------------------------- The following example contains a NetCDF header of a file containing an **ensemble forecast** on **pressure levels**. .. code-block:: :linenos: netcdf somegriddeddata.pl { dimensions: lon = 101 ; lat = 41 ; isobaric = 12 ; time = 1 ; ens0 = 51 ; variables: float lat(lat) ; lat:units = "degrees_north" ; float lon(lon) ; lon:units = "degrees_east" ; float isobaric(isobaric) ; isobaric:units = "hPa" ; isobaric:long_name = "Isobaric surface" ; isobaric:positive = "down" ; int time(time) ; time:units = "Hour since 2012-10-15T00:00:00.000Z" ; time:standard_name = "time" ; int ens0(ens0) ; ens0:standard_name = "ensemble_member_id" ; float Geopotential_isobaric(time, ens0, isobaric, lat, lon) ; Geopotential_isobaric:long_name = "Geopotential @ Isobaric surface" ; Geopotential_isobaric:units = "m2.s-2" ; float Temperature_isobaric(time, ens0, isobaric, lat, lon) ; Temperature_isobaric:long_name = "Temperature @ Isobaric surface" ; Temperature_isobaric:units = "K" ; ... } Note the following: * The **latitude and longitude dimensions** are recognised according to their units keyword; cf. the `longitude/latitude section in CF-conventions`_. * The **vertical pressure level dimension** requires units of pressure, as well the positive attribute being defined; cf. the `vertical coordinate section in CF-conventions`_. * The **time dimension** is identified by its units attribute; cf. the `time coordinate section in CF-conventions`_. .. important:: The time encoded in the units attribute of the time dimension is used as the **forecast base/initialisation time**. If you are not using forecast data (e.g. if you are using reanalysis data), Met.3D requires you to either use the same reference time in all files of your dataset, or to have the reference time equal the valid time (i.e. lead time = 0) - this essentially means that you are using one file per time step. * The **ensemble dimension** is **optional**. If you don't have an ensemble dimension, the variables should have the dimensions (time, isobaric, lat, lon). .. important:: The **ensemble dimension** is currently not specified in the CF conventions. Met.3D simply searches for a variable with a ``standard_name`` attribute set to ``ensemble_member_id``. For now, the ``_CoordinateAxisType`` used by the netcdf-java library (``_CoordinateAxisType = "Ensemble"``) is also acceptable. (See the Met.3D source code method ``NcCFVar::getEnsembleVar()`` in ``nccfvar.cpp``). The following example contains a NetCDF header of a file containing an ensemble forecast on hybrid sigma-pressure levels; the required keywords of the vertical variable are different; cf. `Appendix D of the CF-conventions`_. .. code-block:: :linenos: netcdf somegriddeddata.ml { dimensions: lon = 101 ; lat = 41 ; hybrid = 62 ; time = 1 ; ens0 = 51 ; variables: float lat(lat) ; lat:units = "degrees_north" ; float lon(lon) ; lon:units = "degrees_east" ; float hybrid(hybrid) ; hybrid:units = "sigma" ; hybrid:long_name = "Hybrid level" ; hybrid:positive = "down" ; hybrid:standard_name = "atmosphere_hybrid_sigma_pressure_coordinate" ; hybrid:formula = "p(time,level,lat,lon) = ap(level) + b(level)*ps(time,lat,lon)" ; hybrid:formula_terms = "ap: hyam b: hybm ps: Surface_pressure_surface" ; int time(time) ; time:units = "Hour since 2012-10-15T12:00:00.000Z" ; time:standard_name = "time" ; int ens0(ens0) ; ens0:_CoordinateAxisType = "Ensemble" ; double hyam(hybrid) ; hyam:long_name = "hybrid A coefficient at layer midpoints" ; hyam:units = "Pa" ; double hybm(hybrid) ; hybm:long_name = "hybrid B coefficient at layer midpoints" ; hybm:units = "1" ; float Temperature_hybrid(time, ens0, hybrid, lat, lon) ; Temperature_hybrid:long_name = "Temperature @ Hybrid level" ; Temperature_hybrid:units = "K" ; float Specific_humidity_hybrid(time, ens0, hybrid, lat, lon) ; Specific_humidity_hybrid:long_name = "Specific humidity @ Hybrid level" ; Specific_humidity_hybrid:units = "kg/kg" ; ... } .. important:: Met.3D requires the **surface pressure field** as well as the **ak/bk coefficients** to reconstruct the 3D pressure field. The names of these variables are specified in the "hybrid:formula_terms" attribute. Make sure that this attribute lists the correct variable names. A mismatch is a common source for errors - Met.3D will not be able to find the required fields! .. _`Climate and Forecast (CF) Metadata Conventions`: http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html .. _`longitude/latitude section in CF-conventions`: http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#latitude-coordinate .. _`vertical coordinate section in CF-conventions`: http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#vertical-coordinate .. _`time coordinate section in CF-conventions`: http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#time-coordinate .. _`Appendix D of the CF-conventions`: http://cfconventions.org/cf-conventions/v1.6.0/cf-conventions.html#dimensionless-v-coord Gridded data in GRIB format --------------------------- Met.3D provides support for GRIB files written by ECMWF’s `ecCodes`_ library (as output, e.g. by the ECMWF MARS archive or written by Metview). In the GUI, where you create or load dataset configurations, simply change the file format to ``GRIB``. Alternatively you can also edit the dataset configuration flile directly and change the ``fileFormat`` entry to ``GRIB``. Note the following: * GRIB messages may be arbitrarily distributed over different files that match the specified ``fileFilter``. * Support is currently only provided for a horizontally regular longitude/latitude grid (GRIB ``gridType`` = ``regular_ll``) and either pressure (GRIB ``typeOfLevel`` = ``isobaricInhPa``) or hybrid sigma-pressure levels (GRIB ``typeOfLevel`` = ``hybrid``) in the vertical. * Hybrid sigma-pressure model levels additionally require the surface pressure field (GRIB ``shortName`` = ``sp``) or its logarithm (GRIB ``shortName`` = ``lnsp``) to be present in the dataset. * Analysis (GRIB ``dataType`` = ``an``), forecast (GRIB ``dataType`` = ``fc``) and perturbed/control (i.e., ensemble forecast; GRIB ``dataType`` = ``pf/cf``) messages are interpreted. If your dataset contains more than one of these types, it is recommended to load these as separate datasets. * For 3D fields, a consistent consecutive list of vertical levels must be present **for all time steps and ensemble members** of a dataset. Levels need not be complete (i.e., there can be missing levels at the top or bottom), but the same levels need to be provided for all data fields in a dataset. * For a given dataset, all GRIB messages must have the same horizontal extent. .. important:: Unlike NetCDF files, GRIB files do not contain global headers that summarize the information contained in the file. Hence, to know which data is stored in a file the entire file needs to be read. This, of course, is inefficient when the file is accessed multiple times from different Met.3D sessions. Met.3D hence creates index files that store such a "global header" when first accessing GRIB files. You will find these files along with the GRIB files in the directory in which the GRIB files are stored. Note that your user ID hence requires write access to the directory in which the GRIB files are stored. If you don't have write access, we recommend placing symlinks to the files in a directory to which you have access and in which the index files can be stored. When you are first loading GRIB files, creation of index files can take some time! .. _`ecCodes`: https://software.ecmwf.int/wiki/display/ECC/ecCodes+Home