Design rationale

Why does Met.3D have a task-graph-based, asynchronous pipeline with a global memory manager, instead of a straightforward chain of modules that each process one request at a time? This page collects the reasoning, largely following Chapter 5.3.1 of Rautenhaus (2015).

The smallest data entity

The single most important assumption in the whole architecture is this:

Assumption

The data field of one variable, one ensemble member and one time step is small enough to fit into both system memory and GPU memory.

Everything follows from that. Such a field can be held in one 3-D texture, which means a visualization algorithm running on the GPU has full random access to the entire field and never has to deal with paging or bricking. It also means that the field is a sensible unit of caching, which is why the memory manager is keyed by (data source, request) with exactly this granularity, and why Grid types describes a grid as a flat array rather than a chunked structure.

The assumption still holds for the domain sizes Met.3D is typically used with, but it is an assumption, not a guarantee. A rough estimate: a field of nlon x nlat x nlev grid points in single precision needs nlon * nlat * nlev * 4 bytes, and the raycaster needs the pressure field alongside the data field. The user-facing version of this rule of thumb is in System requirements.

Datasets that violate the assumption are not supported today. The way out was sketched in the original design but explicitly left out of scope: progressive access handled at the level of the scheduler and the memory manager, and streaming or a level-of-detail representation for fields that do not fit on their own. None of it has been implemented since. See Known limitations.

Why a classical pipeline is not sufficient

In the textbook visualization pipeline, each module processes one request at a time: requests propagate upstream from the sink, and every module asks each of its inputs for at most one data item. That works well for techniques that do not care about the time or ensemble behaviour of a dataset — contouring, slicing, volume rendering.

Weather forecast data breaks the assumption almost immediately. Most interesting operations need several fields at once:

• An ensemble statistic (mean, standard deviation, threshold probability) needs every member.

• A trajectory computation needs several time steps, sometimes two at a time to interpolate in time.

• A derived variable needs several input variables (wind speed needs u and v).

• An actor is a sink that routinely requests several variables simultaneously — a horizontal section showing geopotential height, wind speed and wind barbs requests four fields for every time step change.

So a pipeline module must be able to request many items from one input.

Three groups of tasks

Following Biddiscombe et al. (2007), who analysed the same problem for time-dependent processing in VTK, the operations a Met.3D pipeline must support fall into three groups. “Time” here means both the valid time and the initialisation time dimension; the same classification applies to ensemble members and to variables.

Group	Description
Single item	The module needs exactly one data item from each input at a time. Examples: regridding a field, smoothing a field, rendering an isosurface of one scalar field.
Progressive access	The module needs many items, but only one (or a few) at a time. Examples: computing an ensemble mean by accumulation, computing trajectories time step by time step.
Concurrent random access	The module needs arbitrary items simultaneously. Examples: displaying several variables in one actor, comparative visualization of different members or time steps, statistics over a selected subset of members.

Met.3D implements groups 1 and 3 directly: a data source may request as many items from its inputs as it needs, and they are all produced before its own task runs. Group 2 is not implemented as a distinct mode — a source that needs many items gets them all at once, which is fast but requires them all to be resident. This can become a bottleneck, for example, when computing trajectories in Met.3D: integrating trajectories forward for a long time frame requires all time steps of all three wind components to be resident in memory at the same time for trajectories to be computed.

Requirements and their current status

The requirements below are those listed in Table 5.1 of Rautenhaus (2015) for the efficient execution of the three task groups, keeping the original lettering. The two marked with an asterisk were outside the scope of that work. The status column reflects the current code.

#	Requirement	Status
a	Global memory management for CPU and GPU memory, caching as many intermediate results as possible, so that dynamic user behaviour (stepping back and forth in time or member) is cheap.	Implemented, but split: three independent LRU managers (NWP, Analysis, Trajectories) handle system memory, while MGLResourcesManager handles GPU resources.
b	Thread-safe algorithms and memory management, so that pipeline and task parallelism are possible.	Implemented. produceData() runs in a worker thread and must be thread-safe.
c	The possibility to use GPU shaders inside pipeline algorithms.	Implemented. Tasks can be flagged with MTask::setGPUTask(); the scheduler then runs them in the main thread, which owns the OpenGL context.
d	Requirement c in turn requires all data items to be GPU-aware, that is, some relation between the CPU and the GPU representation of an item.	Implemented per data item type, not as a general contract. MAbstractDataItem has no GPU interface; each type that needs one provides its own (MStructuredGrid::getTexture(), MTrajectories::getVertexBuffer(), MRadarGrid::getHSecDataTexture()), and items that never reach the GPU have none. The item also does not hold a second, GPU-side reference as originally envisaged: it creates the resource on demand and registers it with MGLResourcesManager, which owns it.
e	Variable, time and ensemble metadata must be available everywhere in the pipeline.	Only by convention: MStructuredGrid and trajectory items carry MWeatherPredictionMetaData, set by the readers and propagated by createAndInitializeResultGrid(), but nothing enforces it. MAbstractDataItem and MRadarGrid have no such field.
f	No overhead for algorithms that make no use of a particular dimension.	Implemented through request pass-through: a stage that is disabled does not insert its keys, so the request — and therefore the cache key — is identical to one that never passed through that stage.
g*	Support for pre-loading and pre-execution of likely upcoming requests, and storage of the results in the cache if memory is available.	Not implemented, and still open. See Known limitations.
h*	Support for streaming via domain decomposition.	Not implemented, and still open.

Where the parallelism comes from

It is tempting to look for parallelism in the shape of the pipeline: a section that fans out into two branches can be executed on two threads. In practice pipelines in real working environments are rarely complex enough for this to pay off; a Met.3D variable pipeline is essentially a straight chain.

The parallelism in Met.3D comes from somewhere else. By sending many requests into the same pipeline (one per ensemble member, per time step, per variable), a single user-level operation produces many independent tasks even though the pipeline has no parallel sections at all. Computing an ensemble mean over 51 members means reading 51 fields and, with regridding enabled, regridding 51 fields; those operations are independent of one another although the pipeline that produced them is a straight line. The task graph makes that fan-out explicit, and the scheduler executes it.

Fig. 42 Pipeline concept of Met.3D. (a) Data sources are connected to form a pipeline into which an actor sends data requests. (b) Sample pipeline that visualizes the probability of horizontal wind speed exceeding 45 m/s; a request for the probability triggers further requests up the pipeline. (c) The task graph generated by the pipeline in (b). Note that while the pipeline topology in (b) has no parallel sections, the task graph does. Figure reproduced from Rautenhaus et al. (2015, doi:10.5194/gmd-8-2329-2015), Fig. 10, © Author(s) 2015. CC Attribution 3.0 License.

See When one request fans out for what this looks like in the code.

Comparison to VTK

The task-graph approach with global memory management and a central scheduler is the main structural difference between Met.3D and VTK, and it is worth knowing if you are coming from VTK or ParaView:

• In VTK, considerably more responsibility for memory management and execution control stays with the individual modules. In Met.3D, a data source only produces an item; storing it, caching it, evicting it and deciding when it runs are all handled centrally.

• VTK’s modules are not thread-safe, which prevents task-parallel execution of simultaneous requests. Met.3D requires produceData() to be thread-safe and relies on it.

• Approaches that add task parallelism to VTK-style pipelines generally parallelise pipelines that fan out topologically. Met.3D’s dynamic, request-dependent branching covers a broader set of cases, because the branching is a property of the request rather than of the pipeline.

The price is that a data source has to declare its locallyRequiredKeys() correctly and has to be thread-safe. Both are easy to get wrong, and both are described in Data sources.

References

• Rautenhaus, M. (2015): Interactive 3D visualization of ensemble weather forecasts, PhD thesis, Technical University of Munich, Chapter 5.3.

• Biddiscombe, J., Geveci, B., Martin, K., Moreland, K., Thompson, D. (2007): Time dependent processing in a parallel pipeline architecture, IEEE Trans. Vis. Comput. Graph., 13(6), 1376-1383.

• Moreland, K. (2013): A survey of visualization pipelines, IEEE Trans. Vis. Comput. Graph., 19(3), 367-378.