A long job on a failing node can stall or die with little warning, and a model you serve on your own GPUs gets a new URL every time its workflow is resubmitted, so every agent connected to it has to be reconfigured. Fuzzball 4.4 addresses both. The alert rules in your Prometheus and Alertmanager stack now decide when a failing node comes out of service, and a persistent endpoint keeps the URL your agents use when you replace the model it serves. The same release lets a workflow build the container image it runs in.
Node health runs on your Prometheus stack
Each compute node runs the standard Prometheus Node Exporter, and a custom Fuzzball exporter adds metrics that Node Exporter can't see. When you deploy Fuzzball with its Kubernetes operator, it deploys Prometheus, Alertmanager, and Grafana by default, preconfigured with alert rules, routes, and dashboards. To use your existing Prometheus instead, turn the bundled stack off and load the same rules, routes, and dashboards into it.
You set thresholds and wait windows in your alert rules, and Fuzzball acts on the alerts Alertmanager sends it. A firing alert can place a hold on a node: a cordon hold stops new work from being placed there, and an evict hold also takes running work off. A resolved alert releases the hold, unless the rule requires an administrator to release it. By default, monitoring can hold at most 20 percent of a pool's nodes at once, so a misfiring rule can't empty your cluster.
Fuzzball reports each node's health, your alert rules decide what matters, and every hold records who took a node out of service and why.
The workflow author decides what happens to work on an evicted node. A step marked retry.on: node starts again from the beginning on a healthy node, and every other step fails, with the reason recorded in the workflow events.
Evaluating Fuzzball for your cluster? The Fuzzball solution brief summarizes what Fuzzball runs and where it runs.
Replace a model without reconfiguring your agents
In Fuzzball 4.3, a service endpoint keeps one URL for the life of the workflow that declared it. Stop that workflow and start another, and the URL changes.
Fuzzball 4.4 adds persistent endpoints. You create one by name, and a service attaches to it. To replace a model, submit a new workflow against the same endpoint. By default, the running workflow keeps serving until the new service is ready, and then Fuzzball cancels the previous workflow. You manage persistent endpoints from the CLI or the Endpoints page of the web interface, and agents that start workflows over the Fuzzball MCP server confirm each takeover.
A persistent endpoint belongs to no single workflow, so replacing the model it serves leaves every agent's configuration unchanged.
Build a container image inside the workflow
A workflow can now describe the container image it runs in: a base image, files to add, and commands to run, such as package installs. Fuzzball builds the image before the jobs that use it start and saves it to its object cache, where you or your group can reuse it. Your users can go from a public base image to a running job without building on a workstation or pushing to a registry.
Image builds are on by default. A build runs as root inside a confined container on a compute node, so an administrator who doesn't trust every workflow author with that access can turn builds off.
Before you upgrade
Fuzzball 4.4 changes some behavior on an existing cluster, including the monitoring stack, node commands, workflow retry settings, and endpoint URLs. Read the Fuzzball 4.4 release notes before you upgrade, and start with the Fuzzball documentation to plan an upgrade or a new deployment.
