Diagnose failed jobs and re-prioritize queued work in place with Fuzzball

We previously announced that Fuzzball 4.3 makes it easy to run models and agents in a sovereign, sandboxed environment. But 4.3 added several other improvements, fixes, and quality-of-life changes as well.

Re-prioritize a queued workflow without resubmitting it

One command changes the priority of a workflow that's already queued:

fuzzball workflow update WORKFLOW --priority N

Fuzzball re-stamps the value onto every unfinished stage, and each stage records a priority_updated entry in the activity log. The scheduler applies the new priority on its next pass, so nothing is canceled or resubmitted.

The ceiling matches workflow start --priority: unless you're an organization owner, the value has to be zero or lower. Fuzzball 4.3 also fixes workflow start --priority rejecting a positive value from an organization owner, so it now matches workflow update.

Find the failed rank, its node, and its exit status in one error

Every rank in a multi-node job now reports its own exit code. When one rank is killed, runs out of memory, or crashes, Fuzzball fails the whole job, and the error names the failed rank, its node, and the exit status.

AMD GPU faults now feed node reliability, alongside NVIDIA

ECC errors and overheating on AMD accelerators now feed the node reliability score, cordon, drain, and fault eviction, alongside NVIDIA faults. Fuzzball already cordoned, drained, and evicted on NVIDIA faults; 4.3 adds the AMD signals.

Fuzzball also stops degrading substrate nodes for IO errors on disks that back no mounted filesystem or swap.

We've got big things coming for cluster monitoring, so make sure you check back in when v4.4 is released!

Collect cluster diagnostics in one command

fuzzball debug is a new CLI command for cluster troubleshooting. Its dump subcommand includes collectors for everything your Fuzzball cluster does, and makes it easy to pass information to CIQ support when troubleshooting an issue:

fuzzball debug dump

Improved workflow insights

fuzzball workflow why explains more, too. This release fixes three causes of jobs left queued indefinitely: whole-node-exclusive jobs placed on nodes that already had other work, the scheduler picking a provisioner definition with fewer nodes than the job needed, and workflows staying Pending after the scheduler had finished their allocations. When a job can't fit, workflow why now reports INSUFFICIENT_RESOURCES and names the definition and both node counts.

Web interface improvements

The web UI gets three changes for reading a workflow. Each workflow's detail page has a Fuzzfile tab showing the submitted definition as read-only YAML, next to the Events tab. The workflow editor graph draws a connection between each volume and the jobs and services that mount it. And workflow search now matches email, ID, cluster ID, parent workflow ID, and error text as well as name.

Node and volume provisioner controls

Fuzzball 4.2.0 shipped organization compute policy grants as an API with enforcement, and calling that API was the only way to manage them. Fuzzball 4.3 adds a CLI and a web UI for managing access to node provisioners, at organization and group level.

Volume access is tighter by default. New volumes are created with mode 0750 instead of 0755, so they're no longer world-readable or world-traversable unless you set --permissions explicitly. The volume creation form in the web UI now has a Permissions control for the POSIX mode.

Remove finished workflows by hand or on a schedule

fuzzball workflow delete WORKFLOW permanently removes one finished, failed, or canceled workflow. fuzzball workflow prune removes them in bulk:

fuzzball workflow prune --older-than DURATION --dry-run

To age workflows out automatically, set a workflow.retention block.

apiVersion: deployment.ciq.com/v1alpha1
kind: FuzzballOrchestrate
metadata:
  name: fuzzball
spec:
  fuzzball:
    workflow:
      retention:
        enabled: true
        maxAge: 2160h     # 90 days, counted from when the workflow ended
        interval: 6h      # how often the sweep runs; defaults to 1h
        batchSize: 200    # max deletions per sweep; 0 = server default (100)

Workflow deletion ships disabled in Fuzzball 4.3, so no one can delete or prune a workflow until a cluster administrator enables the feature for specified roles.

workflow:
  delete:
    clusterAdmin: true
    organizationOwner: true
    groupOwner: false
    workflowOwner: false

Resolve domain-based endpoint URLs on Docker Compose deployments

Workflow service endpoint URLs now resolve on a Docker Compose deployment, with a default base domain of <host-ip>.nip.io. The stack can also optionally configure a DNS service and resolver configuration to use a custom domain.

Check four changes before you upgrade

We encourage everyone to move to the latest release of v4.3! A few things to be aware of when you do:

1. Catalog repository URIs are now restricted to HTTPS and publicly routable hosts by default, and repositories that don't qualify stop syncing until you re-add them over HTTPS. On an operator-managed deployment, spec.fuzzball.workflowCatalog.restrictions.allowPrivateAddresses relaxes the restriction, and allowedHosts, maxCloneBytes, and cloneTimeout tighten it.
2. Catalog entry IDs will change on the first sync after upgrade. Update any saved links or scripts that refer to entries by UUID.
3. Ingress routes on the kong and nginx classes now require TLS and redirect plaintext callers with a 308.
4. A leftover pending-restore marker now fails the deployment, where it used to skip the required pre-upgrade database dump without warning. Delete the pending-pg-restore ConfigMap manually if necessary.

Full release notes: the Fuzzball v4.3.0 release notes list every change in this release, including the upgrade notes and the security fixes.

To plan a Fuzzball deployment or upgrade, talk to CIQ.