Fuzzball: The future of performance computing is here

The next generation of performance computing has arrived. We are incredibly excited to announce the launch of Fuzzball, a groundbreaking container-first workflow orchestration platform designed to streamline and accelerate high-performance computing (HPC). Purpose-built for rapid R&D work, Fuzzball simplifies complex computational workflows with its intuitive, browser-based interface, offering granular configuration of workflows, secrets, storage, data, and more.

Fuzzball redefines the standard for HPC by focusing on workflow orchestration, with a containerized architecture and a comprehensive set of APIs. Most importantly, it empowers HPC users to focus on high-value research and development by automating the orchestration of systems, data, and resources.

Build performance-intensive workflows in a web UI

One of Fuzzball’s most significant innovations is its user-friendly, web-based workflow builder. With Fuzzball, anyone can design, configure, and execute performance-intensive workflows without needing to interact with a command line interface. Researchers are no longer required to navigate intricate Linux commands or manually manage the complexities of traditional HPC environments, which often involve a tangled web of shell scripts, Python scripts, storage drivers, and advanced Linux administration. (Don’t worry, you can still do all your HPC in your CLI if that’s your style.)

The user interface does more than provide usability — it also ensures that workflows are standardized and reproducible across all environments. No matter how many users create workflows within a single organization, you can be sure that they will be portable and compatible with any other Fuzzball instance or cluster.

Building a workflow can be done in three steps:

Step 1: Create a new workflow. Import a workflow from a local YAML file, copy from a previous workflow run, or start from scratch. You can also use a built-in workflow template to get a quick start for on-premise deployments.

Create a Fuzzball workflow from a local YAML file, copy a previous workflow run, or start from scratch.

Step 2: Define your job. Workflows are fully parameterized so you can easily edit every step and detail. Get as technical as you need for your HPC job.

Fuzzball workflows are fully parameterized so you can easily edit every step and detail.

Step 3: Customize your environment. Quickly define the infrastructure and resources for your workflow, right down to instance sizing and storage volumes. Fuzzball integrates with many modern key stores, like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager.

Fuzzball allows you to define the infrastructure and resources for your workflow, right down to instance sizing and storage volumes.

Fuzzball allows you to configure node/instance sizing for any workflow.

Once you’ve built your workflow, you can run, share, and reuse it. Fuzzball workflows are modular and portable. You can run them on any Fuzzball clusters. Create templates, save your workflows, and share them with your team. The data and results from one workflow can be the input to another workflow.

Fuzzball allows you to run, share, and reuse workflows because they are modular and portable.

Fuzzball unifies the compute backend enabling any developer, researcher, data engineer, data scientist, or builder to scale HPC and AI applications by removing infrastructure complexities. Data experts and researchers can spend their energy on their areas of expertise without worrying about data access, resource allocation, or technical administration tasks.

“Before Fuzzball, customers were required to master decades old, site-specific technology to run their performance intensive computing tasks. A customer would typically use the Secure SHell (ssh) program to log into an HPC machine via a text-only interface, then create bash or python scripts to execute workflows and submission scripts to run them using a batch scheduling system. Orchestrating multiple jobs to run in parallel or in sequence was complicated and error-prone. Fuzzball modernizes this archaic pattern and abstracts bespoke configuration details making workflows easy, manageable and portable.” – Maxine Slaveck, Principal Product Manager for Fuzzball

How Fuzzball works

Fuzzball reimagines container orchestration using a job-based model instead of a service-based model. Its container-first architecture is built on Kubernetes and designed to avoid the overhead costs that typically come with container-based HPC designs. The following diagram shows the primary components of how Fuzzball works:

• Fuzzball Orchestrate (Control Plane): Orchestrate handles the secure management and coordination of workflows, data, volumes, compute pipelines, and resources. Hosted in Kubernetes, it manages a set of Fuzzball Substrate resources, either statically or dynamically provisioned, across environments.
• Substrate (Compute Resources): Substrate is a low-level container runtime environment installed on every compute resource in a Fuzzball cluster. It runs workloads as close to the hardware as possible, maximizing performance through advanced resource allocation and device awareness.
• Interfaces: Users can access Fuzzball via CLI, web UI, or the SDK without worrying about getting less functionality from any of them. There’s nothing the CLI can do that the SDK or UI can’t.

The functional components of Fuzzball and their interactions.

Rather than depend on manual data volume and storage configurations, Fuzzball automates data management and maintains security policies. Traditional HPC environments have very few ways of monitoring access and usage of data. Fuzzball, however, provides auditable logs of how and what data is accessed by a job. Because users are not SSH’d into a remote machine, data can be managed for compliance and regulatory concerns. The entire Fuzzball architecture can be deployed on-premise, even in air-gapped environments. Workflows can run in multiple on-prem or cloud environments following prioritization set by users.

Key features of Fuzzball

• User-friendly interface: Easy-to-use web-based graphical interface for workflow creation and management.
• Automated resource provisioning: Dynamic allocation of compute resources based on workload needs.
• Container-first architecture: Ensures portability, security, and enhanced performance using technology most enterprises already support.
• Integrated data orchestration: Seamless management of data workflows with full visibility and traceability.
• IDP and tool integrations: Use the tools you already know and love for your HPC workloads without worrying about data access issues or compliance risk.
• On-premise and cloud capable: Dynamically provision and manage your resources wherever they are — on-premise, public cloud, or private cloud
• Fully automatable: Plug Fuzzball workflows into your CI/CD pipelines using the API to automate deployments of jobs and resources

Get a demo of Fuzzball today

Fuzzball can be used across a wide variety of use cases, from AI/ML to genomic sequencing to automobile crash simulations. Users can scale their workflows however they see fit while Fuzzball automatically determines which jobs can run in parallel and which are sequential. Unlike traditional HPC use cases, they’ll never be leaving performance on the table thanks to Fuzzball’s intelligent dynamic resource management.

The CIQ team has been hard at work developing Fuzzball for years, with some customers using it in an invite-only limited preview. Greg Kurtzer, CEO of CIQ, shares:

“Researchers and scientists have grown tired of having to be Linux experts just to run their applications and meet the demands of innovation for their companies. Knowledge gaps, skill gaps, unknown resources or waiting for resources for their business groups has slowed them down building and innovating on-premises. While customers have spent billions of dollars in on-premises resources, those resources are either too slow to access, too difficult for the builder or unknown and sit idle. And the architecture of HPC hasn’t changed in more than 30 years. Fuzzball solves that problem, starting today.”

Schedule a demo to see first hand the power of Fuzzball here!