Amalia LLM: inference on Deucalion

This document describes a simple process for setting up AMALIA-9B inference on Deucalion using the vLLM engine and a singularity container.

Environment Setup

vLLM Container

We are going to use a container with vLLM installed, available in the following directory:

/share/data/models/llms/vllm_0.21.0.sif

Model

AMALIA-9B is available on Deucalion in the following directory:

/share/data/models/llms/AMALIA-9B-0626-DPO

Serving the model

AMALIA-9B can run on a single A100 40GB GPU. Follow the steps below to serve the model.

SSH to Deucalion

ssh <user>@login.deucalion.macc.fccn.pt

Allocate a GPU on Deucalion
```
salloc -A <project_name> -n 1 -c 32 -t 01:00:00 -p normal-a100-40 --gres=gpu:1
```
Info

This command allocates an interative session on a single node (-n 1) with 32 threads per process (-c 32). The session is allocated on a GPU partition (normal-a100-40), using 1 gpus (--gres=gpu:1) and is configured to last 1 hour (-t 01:00:00).

You can find your <project_name> by running the command billing on Deucalion.

Run the vLLM container

singularity shell --nv \
    --bind /share/data/models/llms/AMALIA-9B-0626-DPO:/AMALIA-9B-0626-DPO \
    /share/data/models/llms/vllm_0.21.0.sif

Serve the model

vllm serve /AMALIA-9B-0626-DPO \
    --host 0.0.0.0 \
    --port 8000

If you followed the steps correctly, you should see a message Application startup complete indicating the model is being served succesfully.

Serve via sbatch

Alternatively, you can serve the model by submitting the following bash script to the Slurm queue.

submit_vllm_serve.sh

#!/bin/bash
#SBATCH --job-name=vllm_amalia
#SBATCH --partition=normal-a100-40
#SBATCH --account=your_project_account
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus=1
#SBATCH --cpus-per-task=32
#SBATCH --error="slurm-vllm-serve.err"
#SBATCH --output="slurm-vllm-serve.out"


MODEL="AMALIA-9B-0626-DPO"
MODEL_PATH="/share/data/models/llms/${MODEL}"
CONTAINER_PATH="/share/data/models/llms/vllm_0.21.0.sif"

CMD_VLLM="vllm serve /${MODEL} \
    --host 0.0.0.0 \
    --port 8000 \
    --served-model-name ${MODEL}"


singularity exec --nv --bind ${MODEL_PATH}:/${MODEL} ${CONTAINER_PATH} ${CMD_VLLM}

Submit the job using sbatch:

sbatch submit_vllm_serve.sh

Monitor the status of your job on the queue:

squeue --me

Check the logs to ensure the model is being served successfully:

tail slurm-vllm-serve.out

The Application startup complete message will indicate when the model is being served succesfully.

Interact with Amalia

After the successful startup, the model will be served in the following endpoint:

http://<NODE_HOSTNAME>:8000/v1/

Where <NODE_HOSTNAME> is the gpu node that is being used to serve the model (e.g. gnx516).

You can make requests to the model as in:

curl http://<NODE_HOSTNAME>:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "messages": [
            {"role": "system", "content": "Você é um assistente prestável."},
            {"role": "user", "content": "Qual é a capital de Portugal?"}
        ]
    }'