Team 3 · Multi-Node Training
Alchemy
Turning private data into fast, fine-tuned local models.
An agent that runs recipe, training and inference experiments on your own hardware and hands back one Fuzzball workflow that fine-tunes a model and serves it.
Tushar Sonthalia, Kunlin Han, Sebastian Lueders, Akshat Singhania
8.0×
training throughput
356 to 2,858 tokens per second on two nodes, with the loss unchanged.
17.8%
lower held-out loss
0.730 to 0.600 on all 374 held-out LAMMPS files after 2M tokens.
1.98×
from the second node
1,011 to 1,998 tokens per second on the same configuration.
1.79×
serving throughput
BF16 to FP8 in vLLM at 16 concurrent requests, loss up 0.2%.
Figures are the team's own measurements from the hackathon weekend, as reported in their submission.
The problem
National labs, hospitals, banks and defense contractors hold data that cannot go to a cloud API. Local models solve that, but these institutions often run smaller models that fall short of frontier quality. Fine-tuning on the use case closes the gap, and doing it well takes three scarce skills: ML research, training performance engineering and inference performance engineering.
What they built
An agent that does all three as experiments on the hardware in the room. Every experiment is a Fuzzball workflow the agent writes and launches itself.
- Recipe search: many short LoRA trials run in parallel, one per node. A successive-halving optimizer stops weak trials early and the winner is chosen by held-out loss.
- Training performance search: batch sizes, activation checkpointing, single-node against multi-node training, and custom kernels for the GB10. A change is kept only if the loss curve is unchanged.
- Inference performance search: precision and serving settings, rejecting any configuration that falls below a quality floor.
- Emit: one commented Fuzzball workflow that runs the tuned fine-tuning job and stands up the inference server.
The demo
LAMMPS, a large public C++ simulation code, stands in for a private codebase. The base model is Qwen3-Coder-30B-A3B, with whole LAMMPS files held out for evaluation.
The team wrote Triton kernels for the hardware: a fused linear and cross-entropy kernel over the 152k vocabulary that ran 2.1 to 2.35 times faster and cut 6.2 GiB of peak memory in training, and a grouped expert kernel that ran 1.9 times faster than the per-expert loop. The fine-tuned model served 441 tokens per second at 64 concurrent requests.
What the second node added
Data-parallel training across both GB10 nodes ran as one Fuzzball multi-node job at 99% scaling efficiency, with gradient sync taking 0.05 to 0.5 s of a 33 s step. Parallel search trials shared one copy of the weights and data on VAST Data storage, with no copying.