Automated Testing & Continuous Integration

Met.3D uses GitLab CI together with CMake’s CTest framework to automatically build and test Met.3D, and to build and smoke test the packages on demand. Tests are currently based on visual regression: Met.3D renders scenes from synthetic meteorological data in batch mode, and the resulting screenshots are compared against a set of reference images stored in a dedicated repository. The similarity is measured using an image similarity metric (currently SSIM). A configurable threshold per test case (typically 0.95-0.98) determines whether a test passes or fails.

CTest is CMake’s built-in test runner. It collects the tests registered in CMakeLists.txt, executes them, and collects the results. GitLab CI automates the full build–test–package pipeline on self-hosted runner machines whenever it is triggered.

The relevant files in the Met.3D repository are:

.gitlab-ci.yml

Definition of the GitLab CI pipeline (jobs, stages, trigger rules, artifacts).

tests/ci/CMakeLists.txt

Registers one CTest per test case. Only included if CMake is configured with -DMET3D_ENABLE_CI_TESTS=ON.

tests/ci/build_test_pack_windows.cmd

Build, test, report and packaging script of the Windows job.

tests/ci/cases/*.json

One JSON file per test case.

tests/ci/sessions/, tests/ci/actors/

Session and actor configuration files used by the test cases.

tests/ci/*.py

The scripts executed by each test case (see How a Test Case Works).

tests/ci/templates/

Templates for the generated dataset and batch configuration files, and for the HTML report.

CI Pipeline Overview

The pipeline consists of the stages build, test and packaging. Two independent job chains run the same test suite on Windows and Linux.

start

:run-full-ci (gate);

fork
  partition "Windows (met3d-windows)" {
    :build-test-pack-windows-vcpkg
    (vcpkg + MSVC + Intel Fortran:
    build, CTest, report, CPack ZIP);
    :smoke-test-windows-vcpkg
    (Met3D.exe --help from the ZIP);
  }
fork again
  partition "Linux (met3d-linux)" {
    :build-linux-conda
    (micromamba env, glfx, build);
    :ctest-linux
    (CTest, report);
    :build-linux-conda-package
    (conda build);
    :smoke-test-linux-conda-package
    (install package, Met3D --help);
  }
end fork

stop

Fig. 59 Jobs of the Met.3D CI pipeline (.gitlab-ci.yml).

run-full-ci

An empty gate job. All other jobs depend on it, so triggering it starts the whole pipeline (see the trigger rules below).

build-test-pack-windows-vcpkg

Bootstraps (or updates) vcpkg, configures Met.3D with the CMake preset win-x64-release and the vcpkg toolchain, builds it, runs the CTests, generates the report and packs a ZIP via CPack. Everything is done in one job by tests/ci/build_test_pack_windows.cmd, so that the build directory does not have to be passed between jobs as an artifact. The directory is too large to be uploaded to GitLab and to be used as artifact. If any CTest fails, CPack is skipped. The job timeout is set to 300 minutes, since vcpkg may need to build all dependencies from source if its binary cache is empty or outdated.

smoke-test-windows-vcpkg

Extracts the ZIP package and runs Met3D.exe --help.

build-linux-conda

Creates a conda environment with micromamba from conda_environment.yaml, builds glfx (branch qt6) into local/, and builds Met.3D in release mode. The build directory is passed to the following job as an artifact.

ctest-linux

Runs the CTests on the build from build-linux-conda and generates the report.

build-linux-conda-package

Builds the conda package from the recipe in util/conda/. Only runs if ctest-linux succeeded.

smoke-test-linux-conda-package

Installs the package into a fresh environment and runs Met3D --help.

On both platforms, Met.3D is configured with -DCMAKE_COMPILE_WARNING_AS_ERROR=ON, so compiler warnings make the pipeline fail.

Both platforms produce an HTML report with side-by-side image comparisons, as well as a JUnit XML file (tests/ci/report/junit.xml) that GitLab displays in the Tests tab of the pipeline and in the merge request widget. Reports, screenshots, difference images and logs are kept as job artifacts for 7 days.

Pipeline trigger rules

Pipelines are only created in the following cases. Normal pushes to a branch do not create a pipeline.

  • Merge requests: A pipeline is created for each merge request (and each push to it), but run-full-ci is a manual job, so nothing runs until it is started by clicking Run on that job in the merge request’s pipeline. The other jobs then follow automatically.

  • Master: To test the current state of master, go to Build → Pipelines in the Met.3D GitLab project, click New pipeline, keep master as the selected branch and confirm with New pipeline. All jobs then run without further manual steps. Since master is a protected branch, this requires permission to merge into master.

All jobs are interruptible. If a new commit is pushed while a pipeline is running, the running pipeline is cancelled.

How a Test Case Works

Each test case is described by a JSON file in tests/ci/cases/. tests/ci/CMakeLists.txt registers one CTest ci_<casename> per JSON file. In addition, the setup test ci_python_deps (ensure_python_deps.py) runs before all test cases. It installs the Python packages needed by the test scripts (numpy, netCDF4, pillow) via pip if they are missing.

Each test case is executed by run_ci_case.py, which runs the following steps and aborts at the first one that fails:

  1. Data generation (generate_data.py) — Creates the synthetic NetCDF datasets listed in dataset_generators in tests/ci/data/<id>/, and other files listed in file_generators (e.g., images for the image actor) in tests/ci/data/.

  2. Config generation (generate_configs.py) — Creates one dataset configuration per generated dataset and a batch configuration file in tests/ci/cases/<casename>/, based on the templates in tests/ci/templates/ and the information from the test case JSON.

  3. Running Met.3D (run_met3d.py) — Launches Met3D in batch mode (--datasets, --batch and optionally --actors). Met.3D loads the session, renders the time or camera animation and writes screenshots to tests/ci/cases/<casename>/screenshots/. The output of Met.3D is written to tests/ci/cases/<casename>/<casename>.met3d.log.

  4. Image comparison (compare_images.py) — Clones the reference image repository (environment variable MET3D_CI_REF_REPO_URL) into tests/ci/ref_repo/ if not already present, or updates it otherwise. Each screenshot is then compared against the image with the same file name in ci/reference_images/<casename>/ of that repository. The test fails if a screenshot is below the threshold, if images are missing or additional ones were produced, or if the image sizes differ. Difference images are written to tests/ci/cases/<casename>/diff/, the results to compare_results.json.

The duration and exit code of each case are written to run_results.json. All generated files are ignored by git.

Reporting (report.py) is not part of the CTests. The CI jobs run it after all CTests have finished. It aggregates the results of all cases into an HTML report and a junit.xml file in tests/ci/report/. You can run it locally as well (python3 tests/ci/report.py from the repository root) to inspect the results in a browser.

Note

The similarity metric is a global SSIM computed over the whole image, not the usual sliding-window SSIM. It is robust against tiny rasterization differences between GPUs and drivers, but also less sensitive to small local changes. Keep that in mind when choosing the threshold.

Test case description

An example test case JSON file:

{
  "description": "Hybrid data HSecs with image actor.",

  "dataset_generators": [
    { "id": "hyb", "generator": "hybrid_global_wave" }
  ],

  "file_generators": [
    { "generator": "synthetic_image_png", "output": "image_actor_test.png" }
  ],

  "session": {
    "base_session": "tests/ci/sessions/hyb_hsecs_image.session.config",
    "actors": []
  },

  "batch": {
    "mode": "timeAnimation",
    "time_range_secs": 36000,
    "time_step_secs": 3600
  },

  "compare": {
    "metric": "ssim",
    "threshold": 0.96
  }
}
dataset_generators

Datasets to generate. id is the dataset name in Met.3D and the data directory name, generator one of the generators implemented in generate_data.py: pl_global_wave (pressure levels), hybrid_global_wave (hybrid sigma-pressure levels) and auxp_global_wave (auxiliary pressure field).

file_generators (optional)

Additional files to generate into tests/ci/data/. Currently only synthetic_image_png.

session.base_session

The session file to load, relative to the repository root.

session.actors (optional)

Actor configuration files to load in addition, relative to the repository root.

batch

mode is the batch mode animation type, time_range_secs and time_step_secs the animated time range and step. Currently, only timeAnimation is supported, since generate_configs.py does not set a camera sequence file. Further batch mode settings are taken from tests/ci/templates/template.batch.conf, e.g., the screenshot format (.webp). See Batch mode.

compare

metric (only ssim), threshold (default: 0.985), and optionally ref_subdir, the directory of the reference images in the reference repository if it differs from the case name.

Running CTests Locally

Enable the CTests during CMake configuration with -DMET3D_ENABLE_CI_TESTS=ON (see /04_contributing/met3d_development/setup_devel_env). When CMake processes the project, it prints the registered test cases.

The image comparison needs to know where to find the reference images. Set the following environment variables before running the CTests (in the CI jobs, they are already set):

export MET3D_CI_REF_REPO_URL=https://gitlab.com/Christoph.Fischer/met.3d.tests.git
export MET3D_CI_REF_REPO_BRANCH=master   # default if not set

After a successful build, run all tests from the build directory:

ctest --output-on-failure

or a single test case, e.g., ctest -R ci_pl_topo_vsec --output-on-failure. The tests run Met.3D, so they need a display and OpenGL; the Met.3D window opens while a test case runs. If you use an IDE, consult its documentation for how CTests are integrated. For example, in CLion you can create a run configuration of type CTest Application.

The tests use the Python interpreter found by CMake. If required Python packages are missing, they are installed into that interpreter by the ci_python_deps test. On Linux, run the tests from within the activated met3d conda environment, which already contains them.

On Windows, the CTests additionally need to locate the vcpkg DLLs (e.g., netcdf.dll) and the Qt plugins at runtime. tests/ci/CMakeLists.txt sets VCPKG_DLL_DIR and QT_PLUGIN_PATH for the tests automatically from the vcpkg installation of the build directory.

Warning

The reference repository is checked out to tests/ci/ref_repo/ and reset with git reset --hard origin/<branch> before each comparison. Do not make changes there that you want to keep; use a separate clone of the reference repository instead.

Adding or Updating Test Cases

To add a new test case:

  1. Create a session in Met.3D using the synthetic datasets (run an existing test case once to generate them in tests/ci/data/), and save it to tests/ci/sessions/. Actor configurations that are loaded separately go into tests/ci/actors/.

  2. Add a test case JSON file tests/ci/cases/<casename>.json (see above). If needed, add a new data generator to generate_data.py and register it in available_generators (or available_file_generators).

  3. Re-run CMake so that the new case is registered, then run it with ctest -R ci_<casename>. The comparison fails, since no reference images exist yet.

  4. Check the screenshots in tests/ci/cases/<casename>/screenshots/, and copy them to ci/reference_images/<casename>/ in a separate clone of the reference repository. Commit and push them there.

If a change in Met.3D intentionally changes the rendered images, update the reference images in the same way. Windows and Linux share the same reference images, so check that the new images pass on both platforms, e.g., by triggering the pipeline of the merge request.

GitLab Runners

A GitLab Runner is an agent installed on a machine that picks up CI jobs from GitLab and executes them. Runners are associated with a project via tags. Met.3D uses the tags met3d-windows and met3d-linux to send jobs to the correct platform. Both use the shell executor, i.e., jobs run directly on the host and not in a container. See the official GitLab Runner documentation for background.

Important

The test jobs start Met.3D, which requires an OpenGL context on a GPU. The runner therefore has to run inside a logged-in desktop session of the runner machine, not as a system service. This is why the runners are started at login below.

Setting Up a Windows Runner

Prerequisites

Install git and CMake via winget.

winget install -e --id Git.Git --source winget
winget install -e --id Kitware.CMake --source winget

Microsoft Visual Studio

Download a recent version of Microsoft Visual Studio. During installation select the Desktop development with C++ workload.

Note

We recommend disabling the vcpkg integration bundled with Visual Studio. The CI script bootstraps its own vcpkg instance to have control over the version and cache location.

Intel Fortran Compiler

Download the Intel oneAPI Toolkit and follow the installation instructions. Note the installation root directory (e.g. C:\Program Files (x86)\Intel\oneAPI) for configuring the runner.

PowerShell Core (pwsh)

The GitLab Runner shell executor invokes pwsh.exe (PowerShell Core) rather than the classic powershell.exe. Until this upstream issue is resolved, install PowerShell Core.

winget install --id Microsoft.PowerShell --source winget

Python does not need to be installed separately: vcpkg installs Python as a dependency of Met.3D, and the report generation uses that interpreter from the build directory.

Installing and registering the runner

Follow the official instructions at https://docs.gitlab.com/runner/install/windows/ to download the runner binary.

Register a new runner for your project:

  1. Go to Project → Settings → CI/CD → Runners on GitLab.

  2. Click New project runner.

  3. Set the tag to met3d-windows and add a descriptive name.

  4. Select Windows as the platform and follow the displayed instructions.

  5. Paste the generated gitlab-runner register command into a local PowerShell window. When prompted, choose shell as the executor and accept the default https://gitlab.com host.

Starting the runner automatically at login

Create a PowerShell script that launches the runner. For example, if your runner executable is at D:\gitlab-runner.exe, create a file called start-runner.ps1:

Start-Transcript -Path "D:\runner.log" -Append
Set-Location "D:\"
.\gitlab-runner.exe run
Stop-Transcript

Register this script as a scheduled task so it runs whenever you log in:

  1. Open Task Scheduler (taskschd.msc).

  2. Click Create Task and give it a name.

  3. Under General, select Run only when user is logged on and Run with highest privileges.

  4. Under Triggers, create a new trigger: At log on.

  5. Under Actions, create a new action (replace path):

    • Program/script: powershell.exe

    • Arguments: -NoProfile -ExecutionPolicy Bypass -File "D:\start-runner.ps1"

  6. Under Settings, enable If the task fails, restart every … and disable Stop the task if it runs longer than.

  7. Confirm and reboot to verify the runner starts automatically.

Configuring the runner (config.toml)

Open the runner configuration file (config.toml, typically located next to the runner binary) and add the required environment variables to the [[runners]] section:

[[runners]]
  environment = [
    "VCPKG_DIR=D:\\vcpkg",
    "INTEL_ONEAPI_ROOT=C:\\Program Files (x86)\\Intel\\oneAPI"
  ]
VCPKG_DIR

The directory where vcpkg is (or will be) bootstrapped. The CI script clones and bootstraps vcpkg here on the first run, and pulls and re-bootstraps it on every following run. The vcpkg binary cache is placed next to it, in vcpkg_binary_cache (e.g., D:\vcpkg_binary_cache), so that built dependencies are reused across jobs.

Warning

Keep this path short (e.g. D:\vcpkg). Long paths can cause build failures because Windows has a default maximum path length of 260 characters, and deeply nested vcpkg trees can exceed this limit. See for example https://github.com/microsoft/vcpkg/issues/47075

INTEL_ONEAPI_ROOT

The root directory of the Intel oneAPI installation. The CI script sources setvars.bat from this path to make the Fortran compiler (ifx) and MSVC tools available.

The script checks both variables at the beginning of the job and fails with a corresponding message if they are missing.

Known issues on Windows

Smart App Control

Windows Smart App Control may block Met3D.exe because it is an unsigned binary. Disable Smart App Control in the Windows Security settings.

Antivirus / file-system races during CPack

windeployqt.exe can crash non-deterministically (exit code 0xc0000409, STATUS_STACK_BUFFER_OVERRUN), probably due to race conditions on files copied by the Qt deployment. The CI script automatically retries the CPack step up to three times.

vcpkg rebuilds all dependencies on every job

If vcpkg selects its default binary cache location while the runner runs as a different user, the cache may be written but not read (see https://github.com/microsoft/vcpkg/issues/35638). The CI script therefore sets VCPKG_DEFAULT_BINARY_CACHE explicitly (see above).

Setting Up a Linux Runner

On Linux, CI jobs use micromamba to create a conda environment from conda_environment.yaml in the Met.3D repository, i.e., the same environment developers use. It contains the compilers, CMake and all other dependencies. micromamba itself is downloaded on the first run into $CI_BUILDS_DIR/.cache/micromamba, which also holds the package cache. The environment is recreated on every job, so changes to conda_environment.yaml are picked up automatically.

Besides the runner itself, the host only needs git, wget, tar and bzip2, as well as a GPU with working OpenGL drivers and a desktop session (see above).

Installing and registering the runner

Install the GitLab Runner binary via your system package manager. On some older distributions the packaged version may be too old to be compatible with the current GitLab instance; in that case either build the runner from source or install it inside a conda environment.

Register a runner the same way as on Windows (see above), but:

  • Set the tag to met3d-linux.

  • Select Linux as the platform.

  • Use shell as the executor.

Starting the runner automatically

Add the runner binary (gitlab-runner run) to your desktop environment’s Startup Applications, so that it runs inside the desktop session and has access to the display. Make sure the runner is always started from the same working directory so that it can find config.toml and its associated paths.