Team 14

wxrouter

A weather service on two desk-side machines that keeps answering while it learns.

A global weather model trained from scratch, served behind one address across two GB10 machines so forecasts keep flowing while a machine fine-tunes or a server dies.

Cory Gardner, Niravkumar Patel, Chandan Kumar, Ruthwik Reddy Adapala, Arin Mahapatra, Rayan Ahmed Khan

0 of 5,910

requests failed

Under five routing rules, a real fine-tune and a killed server.

10×

shorter wait while a machine trains

0.32 s with the least-loaded rule against 3.2 s without, at p95.

5 h 14 min

to train the weather model

19M parameters, from scratch, on a separate GB10 system.

11 to 18%

lower 1-day forecast error

After a 12-minute fine-tune on a machine that was serving.

Figures are the team's own measurements from the hackathon weekend, as reported in their submission.

The problem

AI weather models make a global forecast cheap. The team’s model computes a 48-hour forecast in about 80 ms. A lab, a utility or a county office could run that on two desk-side machines with the data on site, but only if the service stays up while the model keeps learning from newer data.

What they built

The team trained a global weather model from scratch during the hackathon, then built the service a small lab needs around it: one address in front of both machines, so forecasts keep flowing while a machine fine-tunes, a forecast server dies or a new model goes live.

  • One address, five routing rules and failover. Both forecast servers were killed in turn under traffic: 298 requests sent, 298 answered, and each server came back by itself in 12 to 14 s.
  • One GPU, three workloads. A supervisor on each machine keeps the weather model and an 18.7 GB LLM serving and runs a fine-tune beside them.
  • An assistant takes its forecasts from the weather model, and diagnosis agents read telemetry and propose the likely cause of a fault.
  • One forecast on two machines: a 5-day forecast with 20 ensemble members took 2.0 to 2.2 s split across both GPUs, against 4.2 s on one.

How they used the cluster

  • Fuzzball 4.3.1 starts everything from one workflow file: 2 jobs and 5 services, a GPU per machine, a shared volume and an endpoint, all up 36 s after a start.
  • NVIDIA GB10 in every machine. The weather model is the AFNO architecture from PhysicsNeMo with the team’s own weights, and the assistant runs on Nemotron 3.5 Lightning.
  • The two Lenovo ThinkStation PGX machines are where every event number was measured.
  • One shared volume on VAST Data storage: one LLM file serves both machines, and a new model goes live by pointing one link at it.

Limits the team states

The model is small and was trained once, in one night, on a separate NVIDIA DGX Spark with the same GB10 chip. Leading operational and AI weather models have less than half its 3-day error, and it has no rain output. Every other number comes from one run each on the event machines. The team’s own adaptive routing rule lost to a simpler one.

What comes next

  • A nightly job that tunes on the day’s NOAA analyses and rolls the result out only if it scores better.
  • A spare router and tests that take down a whole machine.
  • A finer grid, with every measurement repeated.