Directions for solving current limitations

This page is intended to outline our ideas for the limitations listed in Known limitations. These are not meant to be strict specifications, but merely guidelines for developers that want to solve them.

Replacing glfx

As outlined in Shader effects still depend on glfx, we want to move away from glfx. One option would be to move to raw GLSL, which would require rewriting all shaders. However, this is not ideal, as it would result in the loss of the modularity provided by glfx.

Instead, we could rewrite our shaders in Slang. It is a relatively new shading language, based on HLSL that compiles to multiple different outputs. The Khronos Group is starting to use this language in their Vulkan tutorials and recommends it over GLSL.

Slang can compile shaders to the SPIR-V binary format. SPIR-V is also supported by OpenGL 4.5. However, SPIR-V generated for Vulkan is not necessarily compatible with OpenGL, as the two APIs have different requirements and execution models.

For this reason, it is recommended that the replacement of glfx be carried out together with the migration from OpenGL to Vulkan. This allows the shader rewrite to target Vulkan directly rather than introducing an intermediate shader architecture that would subsequently have to be adapted again.

Migrating from OpenGL to Vulkan

As outlined in A move from OpenGL to Vulkan is under discussion, we plan to migrate Met.3D to Vulkan at some point. Our motivation is, that OpenGL is effectively a legacy API, that no longer receives major feature revisions. Newer GPU features are primarily exposed through extensions, if at all.

Ray-tracing is one example of a feature unavailable in OpenGL. Modern hardware ray-tracing functionality is not available through OpenGL in the same way it is exposed through Vulkan’s ray-tracing extensions.

Another motivation is portability. OpenGL is no longer supported by Apple on macOS. Vulkan applications can, however, run on macOS through MoltenVK. It translates Vulkan calls to Apple’s Metal API.

The initial Vulkan implementation should prioritize a straightforward migration of the existing rendering architecture. More advanced functionality, such as multi-GPU rendering, asynchronous compute, and sophisticated resource scheduling, should only be introduced where it is required by the existing architecture or provides a clear benefit. The abstractions should not unnecessarily prevent these features from being added later.

This section outlines some challenges involved with a Vulkan migration and provides guidelines on how it could be achieved.

The basics

Vulkan is a considerably more explicit API than OpenGL. It requires additional abstractions for commonly used code. Otherwise, the graphics code will grow unmaintainable.

However, we advise against implementing a fully fledged Render Hardware Interface (RHI). An abstraction that hides all Vulkan-specific functionality would be a substantial subsystem, that requires specialized knowledge to maintain. It would also make it more difficult to expose new or bleeding-edge Vulkan functionality and GPU features. Whenever a student or researcher wanted to use such functionality not supported by the RHI, they would first need to integrate it into the API.

Instead, only a lightweight abstraction layer around selected Vulkan objects should be introduced.

Additionally, we require per-device memory management, similar to our memory management implemented for OpenGL. Resource types, such as buffers, images (Textures) and shaders should be wrapped into memory-managed RAII containers. These abstractions can be implemented similarly to the existing OpenGL abstractions.

Qt provides support for Vulkan, which can be used where appropriate. However, it is preferable to use the Vulkan C++ API, vulkan.hpp, for most Vulkan functionality. It provides type-safe enumerations, STL container support and other conveniences over the C API or Qt’s Vulkan API.

Qt’s Vulkan API does abstract away some platform-specific functionality. For this reason, the Vulkan instance should, for example, use QVulkanInstance. It requests platform specific extensions that would otherwise need to be queried at runtime.

The following sections describe possible approaches for implementing these abstractions and discuss some challenges involved.

Render subsystem

A render subsystem could be implemented to manage the global Vulkan instance. It should be a singleton that is created when Met.3D starts.

Some parts of Met.3D may require specific instance-level extensions. The subsystem should therefore provide a way to request these, before it is initialized. Otherwise, we would have to hard-code all requirements directly in the subsystem.

During initialization, the render subsystem creates the Vulkan instance with the required features and extensions. It then selects a default physical device that is most suitable for the workloads performed by Met.3D.

Selecting a suitable device requires checking whether it supports the required presentation capabilities. This may require creating an invisible window or surface during initialization.

Devices

Vulkan supports multiple devices by design. This is an advantage over OpenGL and could eventually allow Met.3D to make use of multiple GPUs for compute workloads.

For example, a system with two GPUs could use one GPU for rendering, while using the other to process GPU tasks submitted to the scheduler.

For the initial migration, however, Met.3D should use a single rendering device. Nevertheless, support for multiple devices should be kept in mind when designing the Vulkan abstractions.

Each Vulkan device requires an abstraction within Met.3D. They require their own memory allocator (see Resource allocation) and resource manager for caching resources created on the device.

Additionally, each device should expose up to three different command queues:

1. A graphics queue

2. A compute queue, preferably a dedicated queue for asynchronous compute, when available.

3. A transfer queue, preferably a dedicated queue for asynchronous transfers, when available.

In practice, the graphics queue will often support transfer and compute operations. Having separate queues available for different types of work can provide opportunities for further optimization.

Resource allocation

The Vulkan Memory Allocator (VMA) library provides a useful abstraction for managing memory allocations for images and buffers. It should probably be used for resource allocation rather than implementing a custom memory allocator.

Resource caching

Graphics resources, such as Buffers and Images, are created for a specific Vulkan device. Each device should therefore manage its own resources.

Similar to the MGLResourcesManager, each device should have a resource manager responsible for resources created for that device. It should utilize smart-pointers for reference counting. The individual resource types should be implemented as RAII wrappers around native Vulkan resources.

A challenge is determining when resources can safely be destroyed. Reaching a reference count of zero on the CPU does not guarantee that the GPU has finished using the resource.

Since Met.3D does not continuously render frames, one possible solution is to perform resource cleanup whenever the GPU is known to be idle. At that point, all previously submitted work has completed, so resources whose CPU-side references have been released can safely be destroyed.

To track used and available memory, we could utilize VK_EXT_memory_budget. It allows us to query the current memory budget and usage. These reported budget and usage can be used to determine when cached but unused resources should be evicted.

Additionally, VK_EXT_memory_priority could be used to assign different priorities to resources, helping ensure that important resources remain in device-local memory when memory pressure occurs.

Scene views and resources

Vulkan generally requires applications to account for resources that are modified while previous frames are still being processed by the GPU (frame in flight). For example, while the GPU is rendering one frame using a buffer, the CPU may need to update the data for the next frame. The buffer therefore cannot simply be overwritten while it is still in use.

A particularly difficult challenge arises from Met.3D’s use of scene views. Since each scene view can render independently, we might require one copy of certain resources per scene view and per frame in flight.

One solution would be to force Met.3D to process only a single frame at a time. However, this would unnecessarily limit GPU/CPU parallelism and could significantly reduce performance.

Another approach would be to transfer all modified buffers, textures, and other resources before starting a frame and wait for those transfers to complete before rendering begins. All scene views participating in that frame could then access the same resource versions. We would therefore only require resource duplication per frame in flight rather than per scene view and frame in flight.

The feasibility and performance implications of this approach need to be investigated during the migration.

Actors and shader pipelines

Vulkan is considerably more explicit about graphics pipeline creation than OpenGL. Most pipeline state has to be specified when the pipeline is created, including the shaders, descriptor layouts, vertex input state, and relevant attachment formats.

This creates a challenge for Met.3D because actors can participate in multiple scene views. A naïve implementation could require an actor to create a separate graphics pipeline for every combination of view and rendering configuration. Changes to the rendering configuration could then require additional pipelines to be created.

Vulkan 1.3 simplifies this problem significantly through dynamic rendering. Dynamic rendering removes the requirement to define traditional render-pass and framebuffer objects when creating a graphics pipeline. Instead, attachment formats are specified through dynamic-rendering pipeline state.

For this reason, a Vulkan rewrite should probably target Vulkan 1.3 or later and make use of dynamic rendering.

Render passes

Met.3D currently uses multiple render passes, as outlined in Rendering. OpenGL allowed these passes to be implemented easily, because resource usage and synchronization was largely implicit.

Vulkan, in contrast, requires resource usage and synchronization to be described explicitly. For example, a shadow-mapping pass must complete before the resulting shadow map can be read by the main rendering pass.

Before dynamic rendering, the render passes did specify pass dependencies and solved this problem.

If we use dynamic rendering, we need to explicitly synchronize resource usage between the passes.

This is an appropriate use case for a frame graph. A frame graph would allow Met.3D to describe which passes are executed for a given frame and the dependencies between those passes. From this information, the renderer could determine the required execution order, resource transitions, and synchronization.

The frame graph should also provide a way for actors to declare custom passes and their dependencies. For example, a raycaster may require the complete scene depth buffer to be available before it can perform raycasting, while its lightmap may be produced earlier in the frame and therefore does not need to wait for all other rendering work.