Using Met.3D on the Levante HPC System
======================================

.. note:: 
    The following description is current as of April 2025.

This guide provides instruction for using Met.3D on GPU Nodes on the HPC System Levante of the DKRZ.
It works with both Linux and Windows.
It might also work with MacOS if you can install the required programs.

Prerequisites
-------------

- SSH-Client
- For Windows: `PuTTY <https://www.putty.org/>`_
- `TurboVNC <https://github.com/TurboVNC/turbovnc/releases>`_
- `VirtualGL <https://github.com/VirtualGL/virtualgl/releases>`_
- Access to a GPU-Node on Levante (with that your ``<dkrz-user-id>`` and ``<dkrz-project-id>``)


Step 1: Installing Met.3D
-------------------------

Login on Levante with your DKRZ provided user-id:: 
    
    ssh -X <dkrz-user-id>@levante.dkrz.de

Loading and initializing Conda
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Conda is already installed but must be loaded explicitly with the following command::

    module load python3/unstable

Please read the `instructions from the DKRZ <https://docs.dkrz.de/doc/levante/code-development/python.html#location-of-conda-environments-and-packages>`_, particularly the sections "Locations of conda environments and packages" and "Initialization", to correctly initialize Conda on Levante.
If multiple people in your project use Met.3D, there may already be an existing conda environment in the work directory.
In that case use ``conda config --add envs_dirs <path>`` and ``conda config --add pkgs_dirs <path>`` to access the exsisting environments.
You may need to restart your shell to fully complete initialization.

Installing Met.3D
~~~~~~~~~~~~~~~~~

.. note:: 
    For your own sanity you should replace ``conda`` with ``mamba`` when following either installation method, because ``conda`` is very slow.
    Also, when this was last tested on Levante, ``conda`` would crash when installing the :doc:`Linux binary conda package <conda>`.
    The reason for this is currently unknown, however ``mamba`` works.

You can now install Met.3D either using the :doc:`Linux binary conda package <conda>` or by :doc:`Compiling Met.3D from source code: Linux (conda) <src_conda>`.
The installation may take a while.

Step 2: TurboVNC Session
------------------------

This step is nearly identical to `The VNC Solution <https://docs.dkrz.de/doc/visualization/sw/paraview/paraview-on-levante.html#the-vnc-solution>`_ for Paraview in the DKRZ documentation.
In the following two Terminals are needed.

Terminal 1
~~~~~~~~~~

Login on Levante::
    
    ssh -X <dkrz-user-id>@levante.dkrz.de

Before the first time
_____________________

Before starting the first TurboVNC session two things must be done:
A password for the TurboVNC sessions must be set with::

    /opt/TurboVNC/bin/vncpasswd

Additionally, creating the following script makes the process of starting the VNC Server easier.
Adapt the resolution ``3840x2160`` to your screen::

    #! /usr/bin/bash 

    export VGL_DISPLAY=egl0
    /opt/TurboVNC/bin/vncserver -geometry 3840x2160 -wm /sw/bin/vncsession &  
    exit 

Starting VNC Server
___________________

Allocate a GPU-Node::

    salloc --mem=100G --gpus=1 -p gpu -A <dkrz-project-id>

If you have access to the Visualization-Nodes lg0, lg1, lg2 and lg3, you may instead either of the following for a full or half node respectively (half a node should be enough in most cases)::

    salloc --exclusive --gpus=2 -p visualize -w lg0 -A <dkrz-project-id> --mem=960GB --time=8:00:00 
    salloc --gpus=1 -p visualize -w lg0 -A <dkrz-project-id> --mem=480GB --time=8:00:00

Now start the VNC Server by executing the script.
It should print some output including the text `started on display <url>:<number>`.
The parts ``<url>`` and ``<number>`` are required in Terminal 2.


Terminal 2
~~~~~~~~~~

Create a ssh tunnel with (Windows user must use ``plink`` instead of ``ssh``)::

    ssh -L 5901:<url>:590<number> <dkrz-user-id>@levante.dkrz.de


TurboVNC Client
~~~~~~~~~~~~~~~

Now start the TurboVNC Client and connect to ``localhost:1``.
When using the client, all graphics programs must be started with ``vglrun``.

Step 3: Starting Met.3D
-----------------------

Within the TurboVNC Client first activate the Met.3D Conda environment with ``conda activate``.
If using the :doc:`Linux binary conda package <conda>` find the path to the Met.3D executable with ``which Met3D``.
Then start Met3D using::

    vglrun <path to Met3D>

If you followed :doc:`Compiling Met.3D from source code: Linux (conda) <src_conda>` use::

    cd ~/met.3d-base/build/
    MET3D_HOME=~/met.3d-base/met.3d MET3D_BASE=~/met.3d-base vglrun ./Met3D