Fuzzball v3.2: Quick jobs, better visibility, and self-service passwords

Fuzzball v3.2 is out! This release is mostly about usability — specifically, things that were technically possible but involved more friction than they should have. For example: running a quick one-off job from the command-line now takes one command. Workflows now tell you what they're doing while they're doing it, both from the command-line and the web interface. And now users can manage their own passwords within Fuzzball, without needing help administrating their identities in the identity provider.

Here's what's new.

fuzzball run: One command gets you a shell or a GPU job

A common thing our users have wanted to do with Fuzzball — that previously required writing a Fuzzfile — is run a quick command or get an interactive shell to poke around in an environment. That's what fuzzball run does.

The simplest invocation drops you into a shell:

$ fuzzball run
Workflow "7fa8bd3b-5ca6-4671-ab22-ad8e8ab96f08" started.
Waiting for job to start...
(fuzzball:/)$

When you exit, the workflow stops. It uses Alpine by default, but any container image works:

$ fuzzball run --image docker://rockylinux:9

To run a specific command instead of an interactive shell, put it after --:

$ fuzzball run --device nvidia.com/gpu=1 --image docker://nvidia/cuda:12.0-base -- nvidia-smi

You can even run a local script: --script copies it into the job and executes it:

$ fuzzball run --cores 8 --device nvidia.com/gpu=1 --script ./build.sh

Workflow events show what's happening while it happens

Fuzzball workflows have always reported the status of each stage, job, and service as it executes; but some internal workflow processes - particularly cloud provisioning and image management - take long enough that it helps to have a bit more information about their activity.

v3.2 adds an events system that tracks what's happening throughout a workflow's lifecycle. The event log is visible from the command-line:

$ fuzzball workflow events <workflow-id> --follow

You'll see something like this as it runs:

3:15:00 PM [WORKFLOW]  Workflow started
3:15:00 PM [IMAGE]     pull-cuda: Stage started
3:15:00 PM [IMAGE]     pull-cuda: Pulling image: docker://nvidia/cuda:12.0-base
3:15:04 PM [IMAGE]     pull-cuda: Image cache hit
3:15:04 PM [IMAGE]     pull-cuda: Stage finished
3:15:04 PM [JOB]       train: Waiting for my-data to finish
3:15:05 PM [JOB]       train: All dependencies satisfied, stage ready for scheduling
3:15:05 PM [JOB]       train: Provisioning 1 nodes (definition: gpu-a100)
3:15:38 PM [JOB]       train: Provisioning completed: 1 nodes
3:15:38 PM [JOB]       train: Resources allocated (node: gpu-node-1, cpu: 4, memory: 16 GB, devices: 1)
3:15:39 PM [JOB]       train: Container started on node: 10.0.5.23:9876

For cloud workflows, the additional visibility around instance provisioning gives added confidence that the workflow is progressing correctly. For image-heavy workflows, you get pull progress and cache hit confirmation. The dependency events (Waiting for my-data to finish, All dependencies satisfied) show exactly which stages are blocking which.

Events are available through the CLI, the OpenAPI, and the web UI. In the web interface, the Events tab appears on any workflow, stage, job, or service detail view and refreshes automatically while the stage is running.

You can filter to specific stages by name, filter by kind (--kind image, --kind job, etc.), or get JSON output for programmatic use.

Learn more about Fuzzball from CIQ

Users manage their own passwords without touching Keycloak

Fuzzball uses Keycloak as its identity store. Until v3.2, managing passwords meant interacting with Keycloak directly — going through its admin console or API. That was workable if you knew Keycloak, but most users don't, and "ask your admin to log into Keycloak" is not a reasonable password change workflow.

There are three things you can do now that you couldn't do before through Fuzzball itself:

Users can change their own passwords. In the web interface, there's a "change password" option in the user menu. From the CLI:

$ fuzzball profile set-password

Admins can reset member passwords. From the organization member list in the web UI, or from the CLI:

# Auto-generate a temporary password
$ fuzzball organization update-member user@example.com --generate-password

# Or specify one
$ fuzzball organization update-member user@example.com --password='TempPassword1!'

It's also easy to require new users to change their password the first time they log in.

$ fuzzball organization add-member user@example.com --require-update-password
GENERATED PASSWORD (save this - it won't be shown again):
  Kx9@mN4#vP2^rQ8*wJ7...

The user will be required to change this password on first login.

These features only apply to local Keycloak passwords. Organizations using LDAP federation or external SSO manage credentials through those systems, and that won't change.

Other notable changes

GPU annotation matching. The scheduler can now match jobs against GPU metadata — model name, memory, and other annotations exposed by the NVIDIA and AMD device plugins. If you want to target a specific GPU tier rather than just "a GPU," you can now do that in your workflow definition.

CoreWeave provisioner (preview). Fuzzball can now provision compute and storage in CoreWeave. Configuration follows the same pattern as the AWS provisioner and is available through both the CLI and web interface. This is currently in preview status.

Multiple device nodes per device. device-plugins.yaml now supports devices that expose multiple device nodes — useful for hardware like the NextSilicon Maverick that creates multiple /dev entries per physical device.

files: section in the web interface. Workflow definitions using the files: section — which lets you include files inline in the workflow — are now fully supported in the web UI workflow editor.

Fuzzball v3.2 is available now.

See the documentation to get started with the new features.

Download the Fuzzball solution brief