Team 2
Split-Precision KV-Cache LLM Serving Engine
Routing coding sessions between BF16 and FP8 KV-cache workers.
A serving gateway that treats KV-cache precision as a scheduling choice, sending each session to the BF16 or FP8 worker with the shortest predicted completion time.
3.7%
lower p95 end-to-end latency
Predictive routing against least outstanding requests, at concurrency 1.
2.0%
higher output throughput
7.280 against 7.141 tokens per second on the held-out trace.
24.2%
lower time to first token
FP8 KV cache at 512 input tokens, in profiling.
Figures are the team's own measurements from the hackathon weekend, as reported in their submission.
The problem
Coding assistants alternate between large repository prompts and short follow-ups. Which KV-cache precision is faster depends on prompt length and concurrency. The team asked whether predicting completion time while keeping a session on one worker could reduce waiting and improve GPU utilization.
What they built
A CPU gateway exposes one OpenAI-compatible API through Open WebUI. Two vLLM workers load the same Qwen3.8-27B checkpoint on the two ThinkStation PGX machines. One uses a BF16 KV cache and the other FP8. Model weights are unchanged.
For a new session, the routing policy picks the eligible worker with the smallest predicted wait plus processing time, using prompt length, reusable context, output budget and remaining work. Reservations record each assignment right away, and session affinity keeps follow-ups on the worker that may already hold the prefix cache.
How they used the cluster
- Fuzzball manages workflows, resources, placement, endpoints and diagnostic logs.
- VAST Data supplies a shared checkpoint directory and stores traces and configurations.
- Each worker loads its own weights and keeps its own cache on an NVIDIA GB10.
What they measured
In profiling, average wall-clock time favored BF16 above about 5,050 input tokens and FP8 below that point and at lower concurrency. The team notes that this crossover is specific to the workload.
On a held-out trace at concurrency 1, with 72 requests per policy and three repetitions, predictive routing cut p95 end-to-end latency from 37.145 s to 35.759 s and raised output throughput by 2.0%, with no failed requests. At concurrency 4 predictive routing regressed. Coding correctness was not measured.
What comes next
- Collect loaded gateway traces with queue and KV-cache telemetry to calibrate the predictor under concurrent load.
- Build a larger prompt set grouped by input length.
- Study how quantizing model weights alongside the KV cache changes latency and throughput.