Ymir detects what a machine has (ymirctl hw) and works out a model budget. These are the design's
tiers. They are planning targets: so far the setup has run on one NVIDIA PC (an RTX 3060), and vLLM has run on that
card only.
| Tier | Hardware | What you get |
|---|---|---|
| Minimum (CPU) | x86_64 with AVX2, or ARM64; 8 GB RAM; 20 GB free disk | llama.cpp on CPU, a small model at Q4, a few tokens per second. Fine as a pool member for small jobs or embeddings. |
| Entry GPU | NVIDIA with 8 GB VRAM (Turing or newer), AMD RDNA2+ with 8 GB, or Apple Silicon with 16 GB | llama.cpp GPU offload with a small chat model plus a 7 to 9B Q4 model. |
| Comfortable | 16 to 24 GB VRAM, or a 32 to 64 GB Mac | A 27 to 35B Q4 model plus a small stand-in and an embedding model. |
| vLLM | Linux, NVIDIA Ampere or newer (24 GB+), or AMD Instinct / RDNA3 on the ROCm list | vLLM as an extra backend: higher concurrency and FP8 or NVFP4 checkpoints. Linux only. |
| DGX Spark | GB10, 128 GB unified memory | The target for the m1/m2 setup on one box. vLLM on the Spark's GPU is untested so far. |
Backends
- llama.cpp, through
llama-server, on Linux, macOS and CPU-only boxes. Ymir starts it on a random loopback port with a per-process key, or uses a path you give it. - vLLM, managed or wrapped, on Linux with NVIDIA. Windows and macOS builds are not done yet.
- A stock server (vLLM, llama-server or anything OpenAI-compatible) already running on loopback, wrapped by Ymir. Unloading only detaches; Ymir never stops a server it didn't start.
Disk and downloads
Models are big. Setup shows the download size and asks before fetching, then checks size and SHA-256.
Platforms
Linux x86_64 and ARM64 build today. macOS, Windows and the BSDs are planned (Windows service and BSD packages in M9). Nothing is published yet.