Set up model masters

After deploying CosmicAC, add a model master for each supported model so its serving parameters prefill new Managed Inference Jobs.

After you deploy CosmicAC, add a model master for each supported model. A model master holds a model's default serving parameters. CosmicAC prefills the Serving configuration from these parameters when you create a Managed Inference Job for the model. For more detail, see Model masters.

You add each model master by sending requests to the app-node API from any machine that can reach your CosmicAC base URL. This guide uses curl to send the requests from a terminal. To manage model masters, see Manage model masters.

The base URL is the address where your CosmicAC UI is reachable, such as localhost, a server IP, or a domain name. It depends on how your team deployed CosmicAC and where you connect from. If you need it, ask your administrator. CosmicAC serves the app-node API under /api, so the requests below use <base-url>/api.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
Your CosmicAC base URL.
A terminal with curl.

Steps

Create a model master for each supported model

For each model below, send a POST request to <base-url>/api/v1/model-masters. Each request uses the model's recommended configuration. For the source values, see Recommended model parameters.

Runtime image format

Set runtime_image to a Docker image reference, such as vllm/vllm-openai:v0.15.1. CosmicAC serves the model on the image you name here.

The earlier label format, such as vLLM 0.15.0 + CUDA 12.9, no longer works. If a model master still uses that format, CosmicAC serves the model on a fallback image chosen by your deployment rather than the version you set.

Qwen3-VL-235B-A22B-Thinking-FP8

curl -X POST <base-url>/api/v1/model-masters \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "INFERENCE_VLLM",
    "base_os_image": "Ubuntu 22.04 + CUDA 12.9",
    "disk_gb": 500,
    "cuda_driver_version": "CUDA 12.9",
    "model_name": "Qwen/Qwen3-VL-235B-A22B-Thinking-FP8",
    "runtime_image": "vllm/vllm-openai:v0.15.1",
    "data_type": "Auto",
    "quantisation": "Fp8",
    "tensor_parallel": 8,
    "gpu_memory_utilisation": 0.9,
    "max_model_length": 131072,
    "max_concurrent_sequences": 256,
    "reasoning_parser": "deepseek_r1",
    "multimodal": true,
    "replica": 1,
    "require_auth_header": true,
    "inference_param_overrides": {
      "root_disk_size_gb": 500,
      "env": [
        { "name": "TRUST_REMOTE_CODE", "value": "true" },
        { "name": "ENABLE_EXPERT_PARALLEL", "value": "true" },
        { "name": "ENFORCE_EAGER", "value": "true" }
      ]
    }
  }'

MiniMax M2.5

curl -X POST <base-url>/api/v1/model-masters \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "INFERENCE_VLLM",
    "base_os_image": "Ubuntu 22.04 + CUDA 12.9",
    "disk_gb": 500,
    "cuda_driver_version": "CUDA 12.9",
    "model_name": "MiniMaxAI/MiniMax-M2.5",
    "runtime_image": "vllm/vllm-openai:v0.15.1",
    "data_type": "Auto",
    "quantisation": "None",
    "tensor_parallel": 4,
    "gpu_memory_utilisation": 0.85,
    "max_model_length": 131072,
    "max_concurrent_sequences": 64,
    "reasoning_parser": "deepseek_r1",
    "multimodal": true,
    "replica": 1,
    "require_auth_header": true,
    "inference_param_overrides": {
      "root_disk_size_gb": 500,
      "env": [
        { "name": "TRUST_REMOTE_CODE", "value": "true" },
        { "name": "ENABLE_EXPERT_PARALLEL", "value": "true" },
        { "name": "ENFORCE_EAGER", "value": "true" },
        { "name": "SWAP_SPACE", "value": "0" },
        { "name": "SAFETENSORS_FAST_GPU", "value": "true" }
      ]
    }
  }'

Qwen2-VL-2B-Instruct

curl -X POST <base-url>/api/v1/model-masters \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "INFERENCE_VLLM",
    "base_os_image": "Ubuntu 22.04 + CUDA 12.9",
    "disk_gb": 100,
    "cuda_driver_version": "CUDA 12.9",
    "model_name": "Qwen/Qwen2-VL-2B-Instruct",
    "runtime_image": "vllm/vllm-openai:v0.15.1",
    "data_type": "Auto",
    "quantisation": "None",
    "tensor_parallel": 1,
    "gpu_memory_utilisation": 0.9,
    "max_model_length": 27000,
    "max_concurrent_sequences": 128,
    "reasoning_parser": "None",
    "multimodal": true,
    "replica": 1,
    "require_auth_header": true,
    "inference_param_overrides": {
      "root_disk_size_gb": 100,
      "env": [
        { "name": "TRUST_REMOTE_CODE", "value": "true" },
        { "name": "ENFORCE_EAGER", "value": "true" },
        { "name": "SWAP_SPACE", "value": "0" }
      ]
    }
  }'

Confirm the model masters

List the active model masters to confirm CosmicAC created them and to read back their current parameters:

curl --request GET \
  --url '<base-url>/api/v1/model-masters?status=active&limit=20&offset=0' \
  --header 'content-type: application/json'

The list shows each model master you created along with its ID and stored parameters. When you next create a Managed Inference Job for one of these models, CosmicAC prefills the Serving configuration with the model master's values.

You can edit most prefilled values on the job form, but not the runtime image. The web interface offers the model master's runtime_image as the only runtime image for that model, and interactive cosmicac jobs create applies it without prompting. To change it, update the model master. See Manage model masters.

Set up model masters

Prerequisites

Steps

Create a model master for each supported model

Confirm the model masters

Next steps

On this page