Data items

A data item is the unit of data produced by a data source and stored in the memory manager. Every object that can be stored and retrieved through the memory management system inherits from MAbstractDataItem.

Items are created in produceData() and then handed off to the memory manager, which takes ownership and caches them for future reuse.

The MAbstractDataItem interface

MAbstractDataItem is the base class every data item must inherit from. It provides the bookkeeping that the memory manager needs, plus one pure virtual method that every subclass must implement:

virtual size_t getMemorySize_KiB() const = 0;

This method must return the total memory footprint of the item in kibibytes, including any dynamically allocated sub-structures (arrays, nested objects, etc.). The memory manager uses this value to enforce the memory budget and decide which items to evict.

Two further methods track how and where the item was stored. Both are set automatically by the memory manager and do not need to be called manually:

  • getGeneratingRequest() / setGeneratingRequest(MDataRequest) records the request that produced this item.

  • getStoringObject() / setStoringObject(MAbstractDataSource*) records the data source that owns the item.

MAbstractDataItem itself inherits from MMemoryManagementUsingObject, which is the common base for both data items and data sources. It assigns each instance a unique integer ID via getID(), which is used internally by the memory manager.

Items cannot be shared across sources

The memory manager keys every cached item by the pair (source, request). A data item is owned by exactly one data source; it cannot be registered in the memory manager under a different source, and it must not be returned from produceData() of a source that did not originally create it.

If two sources need access to the same underlying data, the downstream source should request it from the upstream source, and receive a data lease (see below), rather than sharing the raw item pointer.

MWeatherPredictionMetaData

For NWP data, items typically also inherit from MWeatherPredictionMetaData. This mixin adds the standard NWP metadata fields:

  • initTime and validTime (QDateTime)

  • variableName (QString)

  • ensembleMember (int)

They are set via setMetaData(initTime, validTime, variableName, ensembleMember) and read back through the corresponding getters.

It also carries a processing information record, a list of key/value pairs that document what operations were applied to produce the item. These need to be set by the producing source. This information is written as attributes to an exported NetCDF file.

Data items referenced inside other items

Some data items hold a reference to another data item as a member. A typical example is MLonLatHybridSigmaPressureGrid, which contains a MDataLease<MRegularLonLatGrid> surfacePressure member for the surface pressure field needed during pressure-level computation. Similarly, MLonLatAuxiliaryPressureGrid holds a MDataLease<MLonLatAuxiliaryPressureGrid> auxPressureField_hPa member.

MDataLease<T> is a RAII handle that pins a cached item so the memory manager cannot evict it. As long as the lease exists, the referenced item stays in memory. When the outer item (e.g., the hybrid-sigma grid) is itself evicted and destroyed, its MDataLease member is destroyed too, releasing the pin on the inner item. See Memory management for more details.

This has two practical consequences when implementing such a data item:

  1. The inner item must be acquired as a lease (not a raw pointer) before it is stored in the outer item.

  2. getMemorySize_KiB() of the outer item must not count the memory of the inner item a second time; the inner item is already tracked independently by the memory manager.