A GPU Scheduler for Local LLMs: What Broke on Real Hardware

Several things want my GPU at the same time: the chat I'm typing into, a coding agent, an indexer working through incoming documents, and a model that reads my screenshots. There is one card — an RTX 4060 Ti with 16 GB. Give each app its own llama-server and they know nothing about each other: while memory lasts, they split the GPU evenly, and when it runs out, something crashes. Ollama loads models while they fit and unloads idle ones to make room, but it serves waiting requests in the order they arrived, with no idea that a chat matters more than indexing.

So in June I started writing GridCore, a scheduler that sits between applications and the GPU. The video shows what it does. This note is about how it's built — and what only broke once it met real hardware.

A control plane, not another inference engine

GridCore doesn't run inference itself. llama.cpp does that, one llama-server process per model. GridCore starts and stops those processes, accepts requests over the OpenAI API, and decides who gets the GPU right now.

For applications, only the address changes. The endpoints are the same, and aliases in the config let an existing client keep sending "model": "gpt-4o" without knowing that a local Gemma is behind it. The one extra thing an app can tell GridCore is its class — interactive, background or batch — through a header, a field in the request body (any OpenAI SDK can send it via extra_body), or a suffix on the model name:

curl localhost:8080/v1/embeddings \
  -H 'X-GridCore-Class: background' \
  -d '{"model": "nomic-embed", "input": ["...", "..."]}'

A request without a class is treated as interactive.

The first question I get: why bother, when llama-server has had router mode since December 2025? Router mode also runs one process per model and also loads and unloads them, but it evicts the least recently used model once their number exceeds --models-max. It doesn't account for VRAM, it has no priorities, and its child processes are hidden inside it — so there is no way to measure how much memory each model actually takes. And those numbers are what every scheduling decision is built on. llama-swap does the same job from the outside and works with almost any server, but which models may run together is declared by hand, and it has no priorities or memory accounting either.

GridCore is written in Go: a single static binary with no cgo, two dependencies (yaml.v3 and the Prometheus client), running as a systemd user service on Linux or a launchd agent on a Mac. The daemon is about 9,700 lines of code, with 181 tests.

A few hard rules instead of one clever algorithm

I wanted the scheduler's behavior to be predictable from the config alone. So at its core are a handful of simple rules:

• Strict class order. Interactive always goes first; within a class it's FIFO. No aging, where a job that has waited long enough gradually climbs in priority: sooner or later that would let a batch job overtake a chat.
• Preemption only between steps. A chat response is one indivisible step: GridCore never interrupts a generation halfway through. Background work, by contrast, is cut into small steps: a large /v1/embeddings request is split into chunks of 32 inputs, each scheduled separately, and the client still gets a single assembled response. Between steps, the GPU can go to the chat.
• Interactive mode. While an interactive request is running, and for two seconds after it, no new background steps start. The two seconds keep the card from going to background work in the pause between two messages of the same conversation.
• Residency tiers. A model used interactively in the last five minutes is hot: background work can't evict it. The rest are cold, and the ones cheapest to lose go first: reload time weighed by how often each class asked for the model lately, so a model used every few seconds outlives one used once a moment ago. Pinned models — for me, a 0.4 GB embedder — are never evicted.
• Invariant number one. A process with requests in flight is never stopped. The scheduler first stops giving it work, waits for the running requests to finish, and only then unloads it.

All state belongs to a single goroutine: HTTP handlers only enqueue a job and wait for permission to run. A model that starts loading reserves its memory immediately, so two concurrent decisions can't claim the same space twice.

Here is what that looks like in the event log when a user starts chatting 300 ms after background indexing began:

11:20:27.194 enqueue   background embedding nomic-embed steps=32
11:20:27.194 dispatch  background step 1/32 -> nomic-embed
11:20:27.500 enqueue   interactive chat gemma4-12b steps=1
11:20:27.500 dispatch  interactive step 1/1 -> gemma4-12b
11:20:27.500 mode      gpu              interactive
11:20:28.731 complete  interactive gemma4-12b
11:20:30.833 mode      gpu              idle
11:20:31.032 dispatch  background step 32/32 -> nomic-embed

The chat got the GPU in the same millisecond, with no time in the queue. The background job waited for the answer to finish and the two-second window to pass, then picked up where it had left off.

Count first, load second

To decide whether a model fits, you need to know its VRAM footprint before loading it. The first version estimated it from the file size and was off by anywhere from −8% to +94%. The worst case was Gemma 4 E2B: an estimate of 5678 MB against an actual 2924. That model has a huge per-layer embedding table, which llama.cpp keeps in system RAM rather than on the card, so the file size tells you almost nothing.

Now GridCore reads the GGUF header and counts tensor by tensor: the weights, minus the embedding tables that stay in RAM; the KV cache, layer by layer, accounting for sliding-window layers, Gemma's shared-KV layers and hybrid SSM architectures; plus the vision projector and runtime buffers. On the models I tested, the error is between −1% and +10%. After the first load, the estimate is replaced by the memory the process actually took on the card.

Two mistakes worth knowing about if you do this math yourself:

• --ctx-size in llama-server is the total context pool shared by all slots, not a per-slot size. As long as I multiplied it by the number of slots, the Qwen3-14B estimate came out at 12.7 GB against a real 9.7.
• Placement flags change everything. Ornith-1.5-35B-A3B with --cpu-moe keeps 18.6 GB of experts in system RAM and takes 2.2 GB on the card — and still generates 46 tokens per second, as fast as a 9B model sitting entirely on the GPU. Without accounting for the flag, the estimate was 21 GB and the scheduler refused to load the model at all. Now it understands -ngl, --cpu-moe, --n-cpu-moe and -ot: 2229 MB estimated against 2244 measured.

This also explains a decision that looks odd at first glance: GridCore doesn't use the automatic placement fitting that llama.cpp has added. Auto-fit adapts to whatever memory is free at the moment of loading, while the scheduler needs a predictable footprint before it. So placement is set explicitly in each model's config.

For the same reason, GridCore has no KV cache saturation to fight and never frees llama.cpp's slots by hand. llama-server allocates the entire KV cache once, when the model loads: --ctx-size is the total pool, --parallel splits it evenly into slots, and that amount doesn't grow at runtime. GridCore doesn't touch the slots either: it tracks how many are busy itself and never sends a model more requests than it has slots. Everything else waits in GridCore's priority queue rather than in llama-server's internal queue — which is exactly what lets a chat overtake background work. If a request is longer than one slot's context, llama-server returns exceed_context_size_error, and GridCore passes that error back to the client as is, without truncating anything. The price of this approach is that KV cache memory is reserved for the worst case, even while slots sit idle. In exchange, you know in advance exactly how much memory a model will take.

What only broke on real hardware

On the simulated test GPU and simulated runtime, every scenario test passed. Then I ran a five-minute load test on the real card — 6 chat clients, 8 background, 4 batch and 2 embedding clients — and it found what the tests couldn't see.

• Complete background starvation. Six chat clients never once let the GPU leave interactive mode, and in five minutes the background jobs completed zero requests. Under constant load, "background waits while chat is quiet" means "background never runs". Now, if the head of the background queue hasn't made progress for 3 seconds, one background step may run alongside the chat. The first version of that rule didn't work either: the oldest starved job was waiting for a model that couldn't fit next to two hot models, and it blocked embedding jobs whose model was already loaded. Now the scheduler walks the starved jobs from oldest to newest and runs the first one that actually can run.
• Excess evictions. To free 200 MB, the scheduler unloaded an 8.5 GB model, then another 6.8 GB one. The first victim was still finishing its requests and its memory still counted as used — so the scheduler picked a second victim. Now the memory of models already marked for eviction counts as being freed.
• Phantom memory. nvidia-smi reports total used memory and the per-process list in two separate calls. If a llama-server exits between them, for one poll its memory looks like ~11 GB taken by some outside process. The scheduler concluded there was no room and, in the same millisecond, unloaded a second, perfectly healthy model. Now the estimate of outside memory is frozen while a process is shutting down and for two polls after.
• 502s out of nowhere. Three errors in five minutes, and nothing in the llama-server logs. The HTTP server inside llama.cpp closes a connection after five requests, and Go kept picking already-closed connections from its pool. One connection per request over loopback turned out to be cheaper than retries.

Each of these bugs became a regression test, and the simulated GPU learned things it couldn't do before — such as holding on to the memory of a process that is already gone from the list.

The numbers

All on the same RTX 4060 Ti 16 GB. During the load test, four models lived on the card at once — Gemma 4 12B for chat, Gemma 4 E4B and E2B for background and batch work, and the embedder — using 15.5 of 16 GB, with not a single unnecessary load and not a single error in five minutes:

The honest cost: under mixed load, chat p50 went from 1.9 to 3.4 seconds. Part of the difference is six clients sharing two model slots, part is that one background step running next to the chat. If chat needs full exclusivity, background_max_starvation: 0s restores it — but then, under constant chat, background work doesn't move at all.

v0.2 adds a knob in between: background_share caps how much of the time background steps may run next to a constant chat. Re-running the same load test back to back, chat p50 was 4.0 s without the cap and 2.7 s with the default of 0.5 (p95 4.8 → 3.5 s), while background work moved at half speed for as long as the chat lasted. Between messages it runs as before.

What v0.2 added

Two of the three things that were next in line after v0.1 are done.

• Cost-based eviction instead of LRU. Among the models the rules allow, GridCore evicts the set that is cheapest to lose: reload time times recent demand, a request count per class that halves every five minutes. Two models that don't fit together now take turns (min_residency) instead of pushing each other out on every request: a config where E4B and E2B were 200 MB over budget reloaded them 60 times in five minutes under v0.1, and five times under v0.2.
• Model families. A request can name gemma4, and the scheduler picks the 12B, E4B or E2B variant — the Gemma 4 family fits this almost perfectly, with one architecture and one prompt format. A person waiting gets the best variant it can have; background work gets the one that disturbs nothing. On a card already holding the 12B for chat and Ornith-35B for a coding agent, a screenshot request lands on the E4B next to them instead of waiting for the 12B, and nothing is evicted. Requests with images or audio only go to variants that can take them.

What v0.3 added: Apple Silicon

v0.3 runs on a Mac, with llama.cpp's Metal backend; I tested it on an M4 Pro with 24 GB. On paper it's the same problem: Metal's working-set limit is two thirds of RAM, 16,384 MB on this machine, practically the 4060 Ti's 16 GB. The scheduler core didn't change at all. Almost every memory number, though, lied in a new way:

• A model is all the RAM it takes. On unified memory the parts llama.cpp keeps on the CPU, like Gemma's per-layer embeddings and the input copy of the token embeddings, come out of the same pool. The 12B takes 9,212 MB here against 8,486 on the 4060 Ti, the E2B 4,815 against 2,924. GridCore now measures the whole process footprint, and the estimate counts the host side too: −2% to +4% on the Gemma 4 models.
• Overcommit is silent. On CUDA a model that doesn't fit fails to load. On Metal a second 12B "loaded" and answered /health, but its warmup had already failed with kIOGPUCommandBufferCallbackErrorOutOfMemory. From then on every request to both models returned 500, including the one that had been working fine. GridCore now fails such a load, counts OOM errors from running instances and unloads the model loaded last.
• Processes grow after they were measured. llama-server's RAM prompt cache (8 GB by default) took the 12B from 8,816 to 18,237 MB over a dozen distinct prompts, and context checkpoints added about 1 GB more. On a discrete GPU that's host RAM nobody counts; on a Mac it's the same memory the models live in. With mmap, Metal also wrapped the whole model file in one GPU buffer, CPU-side tensors included. On unified memory GridCore starts llama.cpp with --load-mode none, --cache-ram 0 and --ctx-checkpoints 0, unless a model's args ask otherwise.
• The OS has a say. macOS reports memory pressure, and with a browser and an IDE open a single loaded 12B was enough for "warn". At warn, background work runs only on models already loaded. At critical, cold models are unloaded one at a time, followed by a minute without background loads: before that, unloading a model lowered the pressure and background work loaded it straight back, three times a minute.

The load test found one more, and this one could hit any device. Once loads went one at a time, a starved background job kept slipping its load in while a chat request waited for an eviction. The chat then evicted that model the moment it was ready, and around it went: 64 loads in three minutes, not one chat answered. Now lower classes start no loads while an interactive load is pending.

With the fixes, the same mixed load test on the Mac, apps open: every model loaded once, no failed requests, chat p50 6.6 s and p95 7.9 s. That's slower than the 4060 Ti's 2.7 s mostly because the M4 Pro processes prompts several times slower. Re-run on the 4060 Ti, all three load tests matched v0.2.

v0.3 also has a dashboard at /admin/ui: one page compiled into the binary that shows the memory budget split by model, ten minutes of memory and queue history, the resident models, the queue with the reason each job waits, and every scheduler event.

What it still doesn't do

Deliberately left out: interrupting a chat mid-generation, multiple GPUs, AMD cards, vLLM and MLX runtimes, cloud fallback and authentication. Radeon support is next; AMD's APUs share memory the way a Mac does, so most of the unified-memory work carries over.

The bottom line

The scheduling algorithm itself turned out to be the easy part. The hard part was accounting. The scheduler makes its decisions based on memory numbers, and nearly every serious bug came down to those numbers lying: an estimate based on file size, two inconsistent nvidia-smi calls, the memory of a process that hadn't finished unloading. The Mac proved it again: the scheduler didn't change, and everything that broke was a number that stopped meaning what it meant on a discrete GPU. A scheduler is only as good as its bookkeeping.

The code is on GitHub under the MIT license: w512/GridCore. If you're working on a similar problem — on a single GPU or a whole rack — I'd like to compare approaches: get in touch.