Data pipeline
Almost all gridded data in Met.3D flows through one shared, demand-driven pipeline. Nothing is computed until something asks for it, results are cached and reused, and memory is reclaimed automatically based on an least-recently-used (LRU) heuristic. This page gives a broad overview; the sub-pages document each part in detail.
Contents:
Core concepts
A data source is a module that, given a request, produces exactly one data item. Data sources are chained. A data source pulls its required inputs from parent data sources, forming a pipeline.
A data request is a string of key/value parameters that uniquely identifies which item to produce (variable, level type, times, member, …). It forms the cache key.
A task graph turns a request into a directed graph of tasks which the scheduler executes in parallel, based on the graph dependencies. See Scheduling in Met.3D.
A data item is the result a source produces (for example a grid, see MStructuredGrid), stored in the memory manager under a key composed of the producing source and the request.
A lease (
MDataLease) is a handle to a cached item that pins it in memory for as long as the lease lives. See Memory management for the lease and eviction rules.
Life of a data request
The diagram below traces one request for a data field from an actor variable’s request to the render-ready state of the field.
Fig. 49 Life of a data request, from the request of an actor variable to the GPU texture used for rendering.
1. The actor variable builds a request. MNWPActorVariable::asynchronousDataRequest()
assembles a request from its current state (variable name, level type, base and valid
times, ensemble member, plus the parameters of all attached request properties) and
passes it to the data source it is attached to via requestData(). This data source
is the end of its variable pipeline.
2. The source turns the request into a task graph. requestData() calls
getTaskGraph(), which strips all keys that are not in requiredKeys() (see
data requests) and looks the resulting key up in the memory
manager. If the item is cached, an invalid task is returned, no work is scheduled,
and dataRequestCompleted() is emitted immediately: the whole upstream graph is
skipped. Otherwise, createTaskGraph() recursively asks every input source for its
own task graph, so one task per source and request is created.
3. The scheduler executes the graph. Tasks without unfinished parents are run first, so upstream sources produce their inputs before their consumers, and independent branches run in parallel. See Scheduling in Met.3D.
4. Each task produces and caches one item. Every task calls produceData()
on its source, which returns one data item. The item is stored in
the memory manager under the (source, request) key. So, every intermediate
result of the pipeline (not only the final field) is cached and can be reused by
other requests.
5. Completion is signalled, not returned. requestData() is asynchronous: it
returns a QFuture immediately, and the source emits dataRequestCompleted(request)
once the item exists. The actor variable’s slot asynchronousDataAvailable() is
invoked on completion of the request. It calls getData(request), which normalizes
the request the same way as in step 2 and acquires the item from the memory manager,
handing out an MDataLease to it. The source itself never stores the item.
6. The grid is uploaded to the GPU. The variable calls getTexture() on the
grid, which creates the texture and registers it with MGLResourcesManager, or
returns the already created one. The texture handles are kept in the variable
(textureDataField, textureLonLatLevAxes etc.) and bound by the actor when it
renders. When the variable receives a new field, releaseDataItems() releases both
the textures and the lease before the new ones are acquired.
When one request fans out
The sequence above produces one field from one request. Many user actions are not like that: an ensemble statistic, a derived variable, or an actor showing several variables all expand into many items. This is where the task graph earns its keep, and it is worth seeing once what that expansion looks like.
Take a volume raycaster displaying the probability that horizontal wind speed exceeds 45 m/s, over all M ensemble members, with vertical regridding enabled. The pipeline it is attached to is a straight chain, with no branches:
Reader -> Smooth -> Partial derivatives -> Ensemble filter
-> Vertical regridder -> Ensemble filter -> (actor)
The stages and their order are described in Variable pipeline. Most of them do nothing here: a stage whose keys are absent from the request passes it straight upstream, so only the trailing ensemble filter and the regridder act.
The actor variable issues a single request; the ensemble operation and the member selection are ordinary request keys:
ENS_OPERATION=P>45;SELECTED_MEMBERS=0/1/.../50;VARIABLE=wind_speed;...
The fan-out happens in step 2 above.
MStructuredGridEnsembleFilter::createTaskGraph() strips its own keys and then
asks its input source for a task graph once per member, each time with a
different MEMBER key:
for (unsigned int m : selectedMembers)
{
rh.insert("MEMBER", m);
task->addParent(inputSource->getTaskGraph(rh.request()));
}
Each of those calls recurses further up the chain, so the graph ends up holding M read tasks, M regrid tasks and one probability task depending on all of them — part (c) of the figure in Design rationale.
Two consequences are worth knowing:
When the probability task finally runs,
produceData()loops over the members and callsgetData()for each. Those are cheap lookups: every member grid has already been produced by a parent task and stays pinned in the memory manager until the child task completes.The per-member regridded fields are cached like any other item. Switching from the probability field to a single member, or to the ensemble mean of the same variable and time step, reuses them and recomputes only the final operation. So if an ensemble operation feels slow, the question is almost always whether those upstream items were cache hits, not whether the operation itself is efficient.