Adding a new data reader

A data reader makes a file format on disk available to Met.3D. In Met.3D, existing data readers are not just file parsers, they are both a reader and a data source, so they can be placed directly at the head of a data pipeline. Everything that the pipeline provides (task graphs, producing the data item, caching in the memory manager) is inherited from the data source side.

Currently implemented readers: NetCDF/CF (MClimateForecastReader), GRIB (MGribReader), DWD radar HDF5 (MDWDHDF5RadarReader), and precomputed trajectories (MTrajectoryReader). Their individual behaviour is described in Data readers.

Adding a new data reader requires two steps:

Implement the reader class.
Integrate it into the application, so that the format can be selected by a user, and a pipeline is built from it.

Choosing a base class

Kind of data	Base class
Gridded NWP data (produces `MStructuredGrid`)	`MWeatherPredictionReader` (`src/data/weatherpredictionreader.h`)
Radar volume scans (produces `MRadarGrid`)	`MRadarReader` (`src/radar/radarreader.h`)
Anything else (e.g. trajectories)	`MAbstractDataReader` plus a matching data source base class

MAbstractDataReader (src/data/abstractdatareader.h) is the common root of all three. It contributes only the file-system side: setDataRoot(path, fileFilter), the file-filter helpers getAvailableFilesFromFilters() and getEnsembleMemberIDFromFileName(), and the pure virtual scanDataRoot(). It deliberately knows nothing about data sources. If you want to read data without using the Met.3D pipeline architecture, you can just use the data reader interface. If you want to make use of the Met.3D pipeline, thus, producing a data item for Met.3D to use and cache, also inherit from a data source.

The remainder of this page describes the gridded NWP case in detail; the other cases follow the same structure with a different readGrid() signature.

Note

Gridded weather prediction data is called NWP throughout the code base (MWeatherPredictionReader, NWP_PIPELINE, MNWPPipelineConfigurationInfo, initializeNWPPipeline()). This page uses “gridded” when talking about the kind of data and keeps “NWP” whenever it names an actual type in the code. Both refer to the same thing.

Implementing a gridded reader

MWeatherPredictionReader inherits from MWeatherPredictionDataSource and MAbstractDataReader. It already implements produceData(), createTaskGraph() and locallyRequiredKeys(), so a subclass never touches the request API. What it must provide are the availability and metadata queries, and readGrid().

What the base class already does

produceData() decodes the request keys, validates the requested base and valid time against the availability information, and calls the pure virtual readGrid().

For HYBRID_SIGMA_PRESSURE_3D it additionally reads the surface pressure field named by variableSurfacePressureName() and attaches it to the returned grid.
For AUXILIARY_PRESSURE_3D it links the 3-D pressure field named by variableAuxiliaryPressureName().
If the requested field does not exist, an empty EMPTY_LEVELS_3D grid filled with M_MISSING_VALUE is returned.

Because these two level types make the reader load a second field, the base class also adds the corresponding parent task in createTaskGraph().

Scanning the data root

scanDataRoot() is called once from setDataRoot(), i.e. at pipeline creation time. Its job is to build the complete availability catalogue of the dataset: which variables exist, on which level types, for which base times, valid times and ensemble members, and in which file each field is stored. Try to make the heavy work in scanDataRoot(), and choose a performant and thread-safe data structure for the catalogue, so later on readGrid() (called for every data request once) is as fast as possible.

scanDataRoot() should also call setDataRange(westLon, eastLon, southLat, northLat) with the horizontal extent covered by the dataset.

Use getAvailableFilesFromFilters() to obtain the file list to scan, and take care of the %m placeholder in the file filter. If it is present, the ensemble member is encoded in the path and must be extracted with getEnsembleMemberIDFromFileName().

Availability queries

Met.3D asks the reader for the dataset content through the following methods. They must be implemented in a thread-safe manner. An availability query might be called from the UI thread while a background thread is currently reading some of the data.

Method	Returns
`availableLevelTypes()`	The `MVerticalLevelType` values present in the dataset.
`availableVariables(levelType)`	Variable names for one level type.
`availableEnsembleMembers(levelType, var)`	Member IDs; a single `0` for non-ensemble data.
`availableInitTimes(levelType, var)`	Forecast base times.
`availableValidTimes(levelType, var, initTime)`	Valid times of one forecast.
`variableLongName()` / `variableStandardName()` / `variableUnits()`	CF long name, CF standard name, and units string.
`variableSurfacePressureName(levelType, var)`	Name of the associated surface pressure variable (hybrid levels only).
`variableAuxiliaryPressureName(levelType, var)`	Name of the associated 3-D pressure variable (auxiliary pressure levels only).
`variableHorizontalGridType(levelType, var)`	`MHorizontalGridType` of the variable.

MWeatherPredictionReader provides the MVariableInfo struct and the MLevelTypeMap hierarchy (a mapping of level type → variable name → base time → valid time → member → file) as a template for the catalogue, but it does not create one. Each reader declares its own map and implements the accessors on top of it. MGribReader uses an extended variant (MGribVariableInfo) that additionally stores per-message offsets.

Met.3D identifies and displays a variable by its name and its vertical level type, so classifying every variable into one of the values of MVerticalLevelType is a core functionality of the gridded readers. The available values and what they mean are listed under Vertical level types. variableHorizontalGridType() is the second classification the reader has to make; regular, rotated and projected lon-lat grids are discussed in Experimental projection support. Both determine which grid class readGrid() has to allocate.

Reading a field

readGrid(levelType, variableName, initTime, validTime, ensembleMember) returns a newly allocated MStructuredGrid and hands ownership to the caller. Do not store it in the memory manager yourself. MScheduledDataSource stores it, and it also sets the generating request on the item. The typical sequence is:

Look up in the catalogue which file holds the field, and open it (see the thread-safety notes below). Return a nullptr and print an error message if the field cannot be located.
Cache per-file and per-variable metadata if you access that file for the first time so subsequent reads skip the metadata work.
Allocate the grid class matching the level type, for example MRegularLonLatGrid (SINGLE_LEVEL), MRegularLonLatStructuredPressureGrid (PRESSURE_LEVELS_3D), MLonLatHybridSigmaPressureGrid (HYBRID_SIGMA_PRESSURE_3D, including its ak/bk coefficients) or MLonLatAuxiliaryPressureGrid (AUXILIARY_PRESSURE_3D). See Grid types.
Fill the coordinate axes.
Set the metadata: setMetaData(initTime, validTime, variableName, ensembleMember), one setAvailableMember() call per member of the dataset, and setHorizontalGridType().
Read the data values for the requested time and member into the grid.

If your data type provides missing values, fill the grid with M_MISSING_VALUE.

Finally, mind the unit conventions of the grid classes: ak/bk coefficients and auxiliary 3-D pressure fields are expected in hPa, surface pressure fields in Pa. You can use scaleFactorToHPa(units) to convert the vaules.

Thread safety

readGrid() is called from worker threads and can run concurrently for different variables, times and members. Two rules follow:

ALL NetCDF access in Met.3D must be guarded with MAbstractDataReader::staticNetCDFAccessMutex, because the NetCDF C/C++ library is not thread-safe. This applies to every NetCDF call, in scanDataRoot() as well as in readGrid(), and also to readers of other data types that happen to use NetCDF (MTrajectoryReader). Hold the lock only around the library calls.
Guard per-file states (open file handles, cached metadata) with their own mutex, so that two threads reading different files do not block each other.

Radar and other readers

A radar reader derives from MRadarReader, which pairs MRadarDataSource with MAbstractDataReader in the same way, and declares readGrid(variable, elevation, interpolateAzimuth, useUndetectedValue) returning an MRadarGrid. MDWDHDF5RadarReader is the reference implementation. All the concepts from above also apply to adding a new radar data reader.

For any other data type, derive from MAbstractDataReader for the file-system part and from the data source base class of that data type for the pipeline part, as MTrajectoryReader does. If no such data source interface exists yet, it has to be written first; the reader itself then only implements scanDataRoot() plus the availability and read methods of that interface.

Both cases also need their own pipeline setup, see Pipelines for other data types.

Integrating the reader into the application

The steps below have the goal of building a pipeline for the newly added data reader, so the data source producing data items from this data reader is registered in the system.

They describe adding a new format to an existing pipeline type, using gridded data as the example. A reader for a data type that has no pipeline type yet needs a few more pieces; these are collected in Pipelines for other data types below.

1. Extend the type enum. PipelineType (src/system/mpipelineconfigurationinfo.h) enumerates the pipeline types. For a reader of gridded data you reuse NWP_PIPELINE, and MNWPReaderFileFormat in the same header enumerates the file formats within that type. Add a value for the new format there.

2. Extend serialisation. MAbstractPipelineConfigurationInfo is the generic super class for the dataset configuration and reads/writes it as an INI file. For gridded data, extend MNWPPipelineConfigurationInfo: add the new format string to both getNWPFormatString() and getNWPFormatFromString(). If the reader needs configuration parameters of its own, add a member to the class and read and write it in loadConfiguration() and saveConfiguration().

3. Add the format to the dataset dialog. MAddDatasetDialog (src/gxfw/adddatasetdialog.{h,cpp,ui}) has one tab per pipeline type, and getSelectedPipelineType() derives the type from the active tab. Add an entry to the format combo box of that tab (nwpFileFormatCombo for gridded data) plus any additional widgets your reader needs, and read them in the corresponding get…PipelineConfigurationInfo() method.

Warning

The dialog converts the combo box selection with (MNWPReaderFileFormat)(ui->nwpFileFormatCombo->currentIndex() + 1). The order of the combo box entries must therefore match the order of the enum values exactly. Append the new entry at the same position in both places.

4. Construct the reader in the pipeline. In MPipelineConfiguration (src/system/pipelineconfiguration.cpp), the readers and the corresponding pipelines are instantiated, one method per pipeline type. For a new gridded data reader, add a branch for the new format in initializeNWPPipeline():

...
else if (pipelineConfigInfo.dataFormat == MY_FORMAT)
{
    nwpReaderENS = new MMyFormatReader(dataSourceId, pipelineConfigInfo);
}
else
{
    return;
}
nwpReaderENS->setMemoryManager(memoryManager);
nwpReaderENS->setDataRoot(pipelineConfigInfo.fileDir,
                          pipelineConfigInfo.fileFilter);
...

The two calls at the end are important: setMemoryManager() must happen before any request reaches the source. setDataRoot() triggers scanDataRoot(), so it should be the last configuration step. Everything the scan depends on must already be set.

The rest of initializeNWPPipeline() is format-independent: the reader is followed by the filter chain, and the last source of each chain is given the dataset name with setObjectName() and registered with MSystemManagerAndControl::registerDataSource(). That registration is what exposes a pipeline endpoint to actors under the dataset name. For gridded data the reader itself is not registered — only the endpoints are (the raw chain under the dataset name, the derived chain under "<name> derived", and the probability-region variants if enabled). See Variable pipeline.

Finally, every initialisation method ends with the same optional step: if addTimesAndMembersToSyncControl is set, the new data source is added to the session’s default time and ensemble controls via addDatasourceToDefaultTimeControls().

Pipelines for other data types

Radar and trajectory data follow the same pattern with their own set of classes, and a genuinely new data type needs:

A PipelineType value, and a subclass of MAbstractPipelineConfigurationInfo instead of a new format in an existing one — see MTrajectoriesPipelineConfigurationInfo as an example. It implements saveConfiguration(), loadConfiguration(), isValidConfiguration() and getFormat(), stores its settings in an INI group of its own, and detects that group with a static is…Config(settings) helper. The group name is what identifies the configuration file later.
A branch in MPipelineConfiguration::loadDatasetFromFile(). It determines the type of a dataset configuration file by trying each configuration class in turn: the first one whose loadConfigurationFromFile() succeeds (i.e. whose INI group is present) wins, and its initialisation method is called.
An initialize…Pipeline() method in MPipelineConfiguration, following the same order as above: get the memory manager from MSystemManagerAndControl::getMemoryManager(), construct the reader, setMemoryManager(), setDataRoot(), then build any downstream sources.
Its own tab in MAddDatasetDialog with a matching get…PipelineConfigurationInfo() method, and a case in MAddDatasetDialog::openDialog() that calls the initialisation method.

Note that the endpoint convention differs between pipeline types. The radar and trajectory pipelines register the reader itself as a data source (the radar pipeline registers both the reader and the regridder that follows it), because there is no long filter chain in front of it. Register whichever sources actors are supposed to be able to select.