Team 3 · Multi-Node Training

Alchemy

Turning private data into fast, fine-tuned local models.

An agent that runs recipe, training and inference experiments on your own hardware and hands back one Fuzzball workflow that fine-tunes a model and serves it.

Tushar Sonthalia, Kunlin Han, Sebastian Lueders, Akshat Singhania

8.0×

training throughput

356 to 2,858 tokens per second on two nodes, with the loss unchanged.

17.8%

lower held-out loss

0.730 to 0.600 on all 374 held-out LAMMPS files after 2M tokens.

1.98×

from the second node

1,011 to 1,998 tokens per second on the same configuration.

1.79×

serving throughput

BF16 to FP8 in vLLM at 16 concurrent requests, loss up 0.2%.

Figures are the team's own measurements from the hackathon weekend, as reported in their submission.

The problem

National labs, hospitals, banks and defense contractors hold data that cannot go to a cloud API. Local models solve that, but these institutions often run smaller models that fall short of frontier quality. Fine-tuning on the use case closes the gap, and doing it well takes three scarce skills: ML research, training performance engineering and inference performance engineering.

What they built

An agent that does all three as experiments on the hardware in the room. Every experiment is a Fuzzball workflow the agent writes and launches itself.

  • Recipe search: many short LoRA trials run in parallel, one per node. A successive-halving optimizer stops weak trials early and the winner is chosen by held-out loss.
  • Training performance search: batch sizes, activation checkpointing, single-node against multi-node training, and custom kernels for the GB10. A change is kept only if the loss curve is unchanged.
  • Inference performance search: precision and serving settings, rejecting any configuration that falls below a quality floor.
  • Emit: one commented Fuzzball workflow that runs the tuned fine-tuning job and stands up the inference server.

The demo

LAMMPS, a large public C++ simulation code, stands in for a private codebase. The base model is Qwen3-Coder-30B-A3B, with whole LAMMPS files held out for evaluation.

The team wrote Triton kernels for the hardware: a fused linear and cross-entropy kernel over the 152k vocabulary that ran 2.1 to 2.35 times faster and cut 6.2 GiB of peak memory in training, and a grouped expert kernel that ran 1.9 times faster than the per-expert loop. The fine-tuned model served 441 tokens per second at 64 concurrent requests.

What the second node added

Data-parallel training across both GB10 nodes ran as one Fuzzball multi-node job at 99% scaling efficiency, with gradient sync taking 0.05 to 0.5 s of a 33 s step. Parallel search trials shared one copy of the weights and data on VAST Data storage, with no copying.