Hardware

What it runs on

From a CPU-only box to a DGX Spark. Add machines over Weirdnet as you get them.

Ymir detects what a machine has (ymirctl hw) and works out a model budget. These are the design's tiers. They are planning targets: so far the setup has run on one NVIDIA PC (an RTX 3060), and vLLM has run on that card only.

TierHardwareWhat you get
Minimum (CPU)x86_64 with AVX2, or ARM64; 8 GB RAM; 20 GB free diskllama.cpp on CPU, a small model at Q4, a few tokens per second. Fine as a pool member for small jobs or embeddings.
Entry GPUNVIDIA with 8 GB VRAM (Turing or newer), AMD RDNA2+ with 8 GB, or Apple Silicon with 16 GBllama.cpp GPU offload with a small chat model plus a 7 to 9B Q4 model.
Comfortable16 to 24 GB VRAM, or a 32 to 64 GB MacA 27 to 35B Q4 model plus a small stand-in and an embedding model.
vLLMLinux, NVIDIA Ampere or newer (24 GB+), or AMD Instinct / RDNA3 on the ROCm listvLLM as an extra backend: higher concurrency and FP8 or NVFP4 checkpoints. Linux only.
DGX SparkGB10, 128 GB unified memoryThe target for the m1/m2 setup on one box. vLLM on the Spark's GPU is untested so far.

Backends

  • llama.cpp, through llama-server, on Linux, macOS and CPU-only boxes. Ymir starts it on a random loopback port with a per-process key, or uses a path you give it.
  • vLLM, managed or wrapped, on Linux with NVIDIA. Windows and macOS builds are not done yet.
  • A stock server (vLLM, llama-server or anything OpenAI-compatible) already running on loopback, wrapped by Ymir. Unloading only detaches; Ymir never stops a server it didn't start.

Disk and downloads

Models are big. Setup shows the download size and asks before fetching, then checks size and SHA-256.

Platforms

Linux x86_64 and ARM64 build today. macOS, Windows and the BSDs are planned (Windows service and BSD packages in M9). Nothing is published yet.