Team 6

Honest Harvest

Give AI research agents the idle GPU time of their own brain.

Runs training experiments on the idle time of the GPU that hosts an agent’s LLM, and pauses them at the next training step whenever an agent needs the model.

Peter Kwaterski, Austin Koske, Miles Trompeter

1.7×

research throughput

30 minutes against 52 for a dedicated split, two runs each.

23 ms

per token from the LLM

Naive sharing slowed it to 45 ms.

0 of 12

wrong keep or discard calls

In every arm, checked against 66 isolated runs.

78 to 92%

of the LLM’s GPU sat idle

Measured on the event cluster with 4 to 6 agents.

Figures are the team's own measurements from the hackathon weekend, as reported in their submission.

The problem

Self-hosted AI research agents loop between an LLM that proposes a change and a short training run that tests it. The GPU that hosts the LLM sits idle most of the time. Using that time must not slow the LLM or change the answer an experiment gives.

What they built

  • Harvest: when the clean node is busy and the LLM is idle, experiments run on the LLM node.
  • Step aside: an LLM request pauses them at the next training step, in 0.2 to 0.5 s once running.
  • Fixed work: 900 steps on checksummed data, so a pause changes when a result arrives and never what it says.
  • Calibrated: the keep or discard threshold is twice the seed noise measured across 66 isolated runs.
  • Safe: one clean rerun for inconclusive calls, plus memory admission, RSS limits and a pressure guard.

How they used the cluster

  • Two NVIDIA GB10 nodes: node A runs the LLM (gpt-oss-20b on vLLM) plus harvested work, and node B runs clean experiments.
  • Fuzzball holds one long allocation per node and the team schedules inside it. Workflows also ran the 66 reference runs, code sync and the dashboard.
  • VAST Data is the shared volume for code, data and every result.
  • Apptainer: Fuzzball pulls the container images and checks architecture, and the team ships a definition for the whole stack.

What they measured

All arms replayed the same 16-decision trace. Honest Harvest finished in 30 minutes against 52 for a dedicated split, at the same 23 ms per token. Naive sharing was as fast overall but made the LLM about twice as slow per token.

Harvested runs gave the same answers as clean ones: the score difference was 0.0005 across 98 runs, with identical data checksums. The team lists the limits plainly. There were one or two runs per arm, and 0 of 12 has a 95% interval of 0 to 24%.

What comes next

  • Launch only in guaranteed-idle windows, which removes the 2 to 6 s first pause.
  • Pipelined launch and checkpointing harvested runs to shared storage.
  • Fractional GPU allocations and a yield signal in the platform, so the scheduling can move out of the application.