Team 1 · Inference at Scale
CoopMoE
Cooperative expert caches for a model too large for both machines.
Serves a 235B mixture-of-experts model that does not fit in the memory of both machines combined, by letting each node borrow experts from the other before it reads its own disk.
Chalmers Phua, Edoardo D’Alessio, Animesh Saxena
227 GB
of expert weights
Against about 70 GiB of expert cache on each machine.
7,960
distinct experts held by the pair
Up from 5,697 with the link off, with no duplicates.
80%
fewer NVMe reads
About 7.2 GB/s down to 1.3 GB/s across both machines.
Figures are the team's own measurements from the hackathon weekend, as reported in their submission.
The problem
Qwen3-235B-A22B in FP8 has about 227 GB of expert weights. Each machine has 128 GB of unified memory, and about 70 GiB of that is left for experts once the dense weights, KV cache and runtime are loaded. Neither machine can hold the model, and the two together cannot either.
What they built
Each machine keeps the full model on its own NVMe and a 70 GiB least-recently-used cache of experts in unified memory. Both machines serve requests at the same time over a ConnectX-7 link. When one needs an expert it does not hold, it asks the other machine first and reads its own disk only as a last resort.
An eviction rule the team calls duplicate-first removes experts the other machine also holds before unique ones, so the two caches drift apart and together cover more of the model. The engine is the Hugging Face transformers Qwen3-MoE implementation with its continuous batching loop. Only the expert module in each layer is replaced.
How they used the cluster
- Two ThinkStation PGX machines, each serving four requests at a time, with unified memory holding 15 GiB of dense weights, the KV cache and the expert cache.
- Node NVMe: each machine copies the 243 GB expert store from shared storage at job start, in about 2 minutes, and reads misses from it at about 3.6 GB/s.
- ConnectX-7 carries cache-residency bitmaps and the data for misses the other machine can satisfy.
- Fuzzball runs a staging job per model that downloads the checkpoint and repacks it into an expert store of 12,032 experts, and a driver script launches each run and collects a report.
- A two-node job could not mount node-local volumes on both machines, so the team ran two single-node jobs that find each other through a rendezvous file on shared storage.
What they measured
Both configurations served the same 32 requests, 128 generated tokens each, with four requests in flight per machine. The baseline is the same engine with the link switched off.
With the link on, the two caches held no duplicates and NVMe reads fell by about 80%. Time to first token was 20.4 s against 28.1 s. Throughput in that first design was 1.52 tokens per second against 1.71 for the baseline, because the other machine ran the missed expert and both GPUs were already saturated. The team changed the design to copy the expert over the link and run it locally. Results for that version were still pending at submission.
What comes next
- Measure the raw link and move from plain TCP to RDMA over the ConnectX-7.
- Run FP8 experts natively, since they are converted to BF16 on every use today.
- Test uneven load, slower storage and more than two machines.
- Repeat the runs. Each number here is from a single run of 40 to 45 minutes.