What's New in Fuzzball 4.2

But previous releases that led us up into here, 4.0 established a new volume management system. So, there's quite a bit of flexibility now in terms of what storage volumes can be attached to Fuzball. Before fuzzball kind of always owned uh the provenence of a volume. Now it's easier to bring in existing storage that you might have at your site, especially at an existing HPC center, let's say. But really any local storage that you might already have, it's a lot easier to bring that into ball and attach it to arbitrary workflows and arbitrary jobs and services and use it however you need. 4.1 added a lot of features to ouruler and introduced backfill scheduling where uh it it's not always running things in uh in kind of first order of submission.

It's it's can look at work that might be later in the queue uh that there are resources to run now when it can't run the top job. So uh that was added in 4.1 as well as preeemption. So we have a lot of people that want to be able to have let's say in an AI context they want to run both inferencing and training in the same cluster but if there isn't inferencing load they want their training to expand to use it but then when inferencing a kind of uh interactive inferencing load comes up they want to carve off more of that dynamically and use it.

we'll use preeemption to uh to stop certain portions of the training workload to free up capacity for inferencing and vice versa. And it's not just an AI feature uh but um that is that's one place that that people have been interested to use it. And that brings us to 4.2 today which is adding aentic uh connector capability um integrated API access from the workflow and then node health monitoring. So let's talk about driving fuzzball with an AI agent. So uh an an MCP server, this is a model context protocol server, uh tells an agent which then passes that on to a language model actions that it's able to take.

They're they're called tools, I think. Uh and some of those actions in the fuzzball MCP server are submitting a job or submitting a workflow. Um it's looking at the work that's currently in the cluster. Um but it's also looking up something in the fuzball documentation. And this is something that makes the Fuzbball MCP really interesting is that the the Fuzball server always has a current copy of the documentation that reflects the capabilities of the server that you're talking to. And the MCP agent is able to look up exactly that. So you don't have this problem of it looks up a different version of documentation kind of out on the internet and maybe it's looking at an out-of-date version or a version that's newer than what you're running.

It can see exactly what your server is capable of and decide what it needs to do. based on your prompt uh based on the server's own documentation. Similarly, the workflow catalog in fuzball uh is a source of documentation for the MCP agent as well. So if you tell it you want to build a certain kind of workflow, it's able to look at other workflows in the catalog to see what other workflows how they were crafted, how they work and compare that against documentation and iterate to solution very quickly. So we we have a demo today of this happening uh where um we started off with like a quick hello world workflow and then iterated to uh a more complex chromax workflow.

So I b brought this into cloud desktop myself. Normally I do this with cloud code but cla uh demos a little better. I just told it uh after connecting the MCP server, I want to run a hello world fuzzball workflow and fuzball or sorry uh the the MCP the cloud desktop agent here went and looked at what tools it had available and made sure it was logged in. That's what this context show is. Looked to see what the MCP server was capable of and then started looking at the documentation by itself to understand what it means to submit a hello world workflow to fuzzball.

what resources does it have available and did this all autonomously uh without me having to do anything other than please run a hello world workflow and that hello world workflow that we ran you know you can go find it in the fuzzball interface yourself you log in you can see it just as though you or any other person had built that workflow you can find it in the execution side and check its logs check how it ran how it executed you could even open it from here in the editor and continue iterating maybe you want to use the MCP server to start a new workflow. But then once you have kind of a basis you want to iterate yourself, you can bounce back and forth any way you want.

You can reference a workflow that ran in the past and ask Fuzball to pick up from there uh or to ask your agent to pick up from there and make further modifications. So you're not uh you're not partitioned away from fuzball. You can uh in the normal fuzball interface, you can kind of bounce back and forth between an agentic process and a classical process at any time. And then by the end, uh, it has this workflow. It ran. It actually waited for it to finish so it could show me the output. You know, it went a little bit above and beyond and and did some things I didn't ask for.

It printed the host name and the date that it ran and the kernel that it ran on. Totally fine. I could iterate from here on exactly what I want this workflow to do. But let's put that aside and do something a little bit more complicated. Um, I wanted to make a Gromax workflow. Gromax is a molecular dynamics code that's pretty common in bio research and high performance computing. I won't pretend to be a bio research scientist myself, but I do recognize the name and even I armed with cloud code armed with uh with the MCP server am able to have it develop a workflow to do the scientific prompt.

The scientist doesn't have to do anything uh know how fuzzball works, how it's going to make this workflow, know where to get the Gromax container. The scientist is just able to specify what research type thing they want to do. and the uh the agent, the cloud desktop in this case, enhanced with the fuzball MCP server and with access to the fuzball docs and frankly anything else on the internet that it can do to learn about Gromax and how to do things with it, it's able to put together a working workflow uh that that runs exactly as the scientist would want it to. And then if it doesn't, they can see the output.

they can iterate this way without having to know anything about how the system actually did the execution, how the workflow syntax works, um, or come up with it from nothing themselves. And then the way Claw Desktop works is once it's built the workflow, we have it in I I forget what it calls it, but there's this this side pane here that records the uh the workflow that it generated and the output from it once it's executed. that Gromax job uh is running in the same fuzzball web interface as any other fuzzball job. You can monitor it while it's running. The the way the the agent tends to respond as it just waits for the whole workflow to end and then report on the end of that.

Uh but you can check in here through the web interface and see each step as it executes, see the logs that it's generating and inspect it or edit it just like any other fuzzball job. One thing that is interesting uh that that I've experienced while dealing with uh the fuzball MCP server is seeing how it chooses to build workflows and what I can even learn about the best way to make a fuzball workflow uh from what is effectively a new outside perspective. So without being prompted to it discovered that we have this object cache in fuzball and made sure that the output from the run was saved in the object cache and I could go view it directly in the object cache.

Um it understood how to inline input data in the workflow. And it that's that's kind of a hard syntax to get right if you're doing it by hand. But the LLM didn't care. It's just a YAML document. So it just put the input file right there in the workflow. And now that workflow is portable even for its input data. I can take that and run it on a different cluster with the same input and it'll run the same way that you would want. Something I've been saying about uh the MCP server. I was a little cynical about it. I'm one of these kmagins who is a little bit suspicious of all the new AI hype and I thought does Fuzball really need an MCP server?

And then you know one of our engineers was working on it and had it up ready for at least a certain level of evaluation. How is this working? Is this useful? And it was immediately useful. It was immediately apparent how this would change the way even I would interact with fuzzball. And there's no better example of that than just the fact that I used claw desktop to build the demos for this. So I was able to start just with a conversation and say, "Hey, I'm putting together a demo for this a uh agentic workflow capability of fuzzball. Let's talk about some good demos that would be useful.

how do we show them off and iterate there? And then I was even able to say, what's a good prompt to make sure that another iteration of the MCP server does what we want it to do in a predictable way. And this whole thing was just it was trivial compared to how much uh work would have to go into putting I'm not a a biioiniraticist or a a genome researcher but I was able to put together this workflow and it's it's not quite the same as complete uh you know complete AI generated um a document or something like that because it's it's going and using a container that exists right it went and got it I think off the Nvidia container registry and uses that without having to build it.

It's accessing the fuzzball docs and basing uh the way that it's constructing this workflow based on an example Gromax workflow we already had in the catalog. So, it's not completely from nothing. It's not uh it's not as likely to be just maybe a little bit of nonsense that uh an AI has hallucinated. It's just accelerating your time to solution because you don't have to go do all that Google searching yourself. You don't have to go figure it all out. You can start with the actual scientific result you're trying to do and it can help bring in all of these resources and do that automatically and go get compute resources for you.

Let's move on a little bit and talk about workflows that launch other workflows. So uh this comes up mostly for us with something like next flow where you have a pipeline system that itself wants to make decisions and then launch computationally intensive work. When we've integrated Nexflow with fuzzball in the past uh Nexflow has kind of had to run outside of fuzball and it talks to the fuzball API or with the command line and it submits work.

um in a in other environments it would you know submit slurm jobs or submit PBS jobs uh for us it was submitting fuzzball workflows but we didn't have a way bring next flow itself into fuzball and now we do now we don't have our our next flow integration updated quite to that point yet uh but what we did uh to demonstrate this capability was integrate another popular piece of software the Jupyter notebook suite and Jupyter lab so as I was describing Earl earlier with building these with uh um with cloud desktop itself. You know, there's a lot to get right here. I had never called out to a REST API from a Jupyter notebook before, but I was able to build this workflow using the MCP server and it it it just worked.

It just worked and it was great. So, let let's look at this workflow here. Here I have, you know, it's a pretty simple workflow. It looks like any other Jupyter workflow I might otherwise have run. It's running this Jupyter service. It's running in a in a um container image that it got from a public registry. I can connect to it using Fuzball's service endpoints feature. All of this is stuff that's been in Fuzball for a while. But uh one of the cells in here is a prompt for a stable diffusion workflow. So it's going to it it has code at the top of this notebook for connecting to the Fuzbball API.

And it's able to do that because it has a an access key, an access token for the fuzzball API that's generated for this workflow. And so while this workflow is running, it has an environment variable that allows anything that can talk to the fuzball API to speak as me because this is my workflow and do things in fuzball uh that um that I would have otherwise been able to do. So here it's submitted a separate stable diffusion workflow separate from the Jupiter workflow that's running in parallel. Um I don't think I show here in the in the actual demo video, but there's a parent child relationship between them.

Fuzball understands which workflows spawned which other workflows. But ultimately the end result is once it's done this notebook here notices and the image that was generated appears in this notebook which is crazy. you can just be in this uh notebook interface that people especially like in data science have come to appreciate and and use for lots of different things. But now you can spawn arbitrary fuzzball work on arbitrary external computing resources and the results will just appear right here in this uh in this notebook interface. And lastly, let's talk about node health and and monitoring and self-healing scheduling. So this is an ongoing process. Uh we're we're kind of iterating and continuing to work on how we want this to work.

Right now all of the node monitoring in fuzzball is very internal to fuzzball. It has a set of parameters about each compute node that it's monitoring and it scores those to give a health metric for uh you know between 0 and 100. How healthy is this node? And then you can set policies for actions that the cluster might take when it falls below a certain threshold. So maybe once a score drops to 90, uh, it stops scheduling work there. And maybe once it drops to 70, it actively starts trying to move data off of it. Maybe there's preemptable work on it. Um, and it tries to preempt that work and move it to healthier nodes.

Uh, or if it gets so bad, it just doesn't even care. It's like, this this work isn't even worthwhile anymore. The nodes are not trustworthy. Let's just cancel all the work that's on it, and hopefully that work can be scheduled elsewhere. All those capabilities are are part of fuzzball today. Um but the the only metric we have for making those decisions is this kind of 0 to 100 node health metric that is a distillation of a bunch of different health criteria um like the temperature of the node and uh the error rate of memory on the node different things like that. Where this is most visible today uh is in the list of nodes.

So any user on the cluster can access the list of nodes that are currently provisioned or connected to the cluster. And if you are in fuzball 4.2 you will see that it's added I think just the health column. I think the status column was always there for for whether the fuzzball substrate um Damon the the the process for that was running and in an okay state a schedulable state. But we've added this health uh um column as well as a score column that's reporting fuzball's view of that node and its health and and the the health column itself here is whether it's taking any actions uh for for that node.

But then you can also see the status for a given node uh in the nodes detail. So if you click on one of the nodes uh in this list, you get this detail pane on the side and that'll report the score as well. I think there's somewhere where we can see um see like changes in the health. But this this takes me to kind of our our intent for continuing to iterate on this. So like I said, right now the the node health monitoring in fuzzball 4.2 is very um it's very internal to fuzzball. It can monitor the things it can monitor. It can take the actions it can take.

But that's it's kind of only what we've developed. A true version of fuzball. the the current plan is to integrate that more with an existing ecosystem of external node exporters. So you can pull node health kind of metric exporters off the shelf from existing projects like Prometheus and that whole ecosystem of node monitoring. You can add your own node exporters to monitor things that are specific to your environment. And then there will be one for fuzzball for monitoring things specific to the fuzball cluster in the fuzzball environment.

And then from that, once that data is exported from the compute node through orchestrate, you'll be able to have whatever monitoring system that you want to plug into it take actions in cluster like draining nodes, like preempting work off of problematic nodes and not be confined only to the actions and the single integer metric of health that that we have in 4.2 today. So, this is a work in progress. We're continuing to work on it. If you uh if you have a 4.1 cluster today and you upgrade to 4.2, um we'd love to hear about your experiences and your preferences and like what would make the most sense in your environment, what you need to monitor, how we can support that, how we can make this the the best and most resilient environment that we can for you.

So there are of course other things in 4.2 that aren't, you know, one of these three headline uh um features. We've added additional functionality for um uh separating different groups and different organizations within the fuzzball cluster. It's always been a multi-tenant, multi-org and multi-group cluster, but there's better support now for uh configuring what volumes a given group is able to access and who's able to mount it versus who's able to use it versus who's able to create volumes, that kind of thing.

there's more uh access control lists for controlling who can do what with especially in storage and then we have added uh kind of similar to the the node health monitoring an iterative process towards uh more and more resource accounting so that when people run in the cluster you can see who did what work when how much resource it consumed how long it ran all of that data is there but we're breaking it out into more accounting reports and we'll talk more about that as a headline feature as it solidifies and there's more and more of it um and as that comes to completion. Second thing for this uh we have had support for AMD uh GPU computing for a while.

We have a plugin just like the Nvidia plugin that uh adds the ability to detect and use and schedule AMD GPUs and Nvidia GPUs in two separate plugins per compute node. Um but something we did add in the 4.2 two time frame is now uh when the dynamic provisioners in AWS or Azure or Oracle or Google when they provision new compute nodes those compute nodes now have the AMDGPU plug-in enabled and available. So if you're running in a dynamically provisioned environment that has AMDGPUs you can now see them and schedule against them and run them. Most of the people that we've worked with that were doing AMD GPU compute were doing it in kind of a local on-prem statically provisioned environment, which is why we hadn't noticed that this was missing before.

But now, uh if you're doing dynamic uh compute provisioning in an AMDGPU environment, that will work out of the box where it would have needed some additional configuration before. And then finally, uh we have a a one command debug bundle uh process now where it's a lot easier to just run something akin to an SOS report. Uh run a command and it will get all manner of debug and logs out of uh the fuzball cluster through the fuzzball command line and that makes it easier to kind of bundle that up and send it off to our support team and engineers who can then help with whatever trouble or questions that you might be having with your cluster.

A few things that I'm really excited for uh in uh upcoming versions of Fuzball. The what we're working on right now for fuzzball 4.3 is what we're calling turnkey models and agents. So we have demos that we've shown here and in other places for running uh LL large language models on fuzball and then connecting to them with an agent. Um but that's a really kind of hands-on process crafting that workflow making it work. Um, the catalog in Fuzball version 4.3 will include a whole collection of models that you might want to run out of the box ready to go along with agents to connect them to things like open code and autodiscocovery between them.

So if you start up a model server and you start up an agent, that agent will just know that you have that model server available and they'll work together and you can just start doing AI code generation right there in fuzzball without any threat to uh you or concerns uh with what that agent might do on your local workstation. It's all in a sandboxed environment inside of a fuzzball job that only can see what you've given it access to. We're also excited now that we have this MC MCP interface to make that even more accessible where you don't need an external agent to make use of it.

So the demos today were run using Cloud Desktop, but we want the web interface for Fuzball to just have a chat interface on the side that you can talk to about workflows that you want to generate or questions that you have about what's happening in the Fuzzball cluster with the full access to the Fuzzball documentation and the uh actions and tools that the MCP server provides just right there in the web interface. So that probably won't be in 4.3, but we're excited to bring that out in a future version of Fuzball. And then another thing that's coming soon is the ability to build your um your workflow container images in the workflow itself.

So right now you have to have built those images using some kind of external process or maybe you are um pulling public images from DockerHub or the GitHub container registry or the Nvidia container registry. But sometimes you need to modify those containers or maybe you want to build a whole new container based off of a a kind of minimal base image. Right now that's that's not something Fuzzwell can do for you. In the future, the image step in a fuzell workflow will instead of only supporting pulling an image, we'll be able to build one from a definition that you provide. So you can install new packages, an existing image or start a whole new image and then run your workflow directly in it without needing to use any other external processes.

And with that, thank you all for your time and for being here. And I'll pass it off to Brian to close us out. >> Thanks, Jonathan. That is some really great stuff. I can't wait to get my own hands on uh that MCP agent and start doing things that I don't even know how it does it in Fuzball um right away in Cloud Desktop. So, what's next step for you in Fuzball 4.2 for everybody who's watching? Well, it depends upon what kind of camp you're in. Uh if you're already running Fuzzball, uh you probably want to get your hands on this, start evaluating it, starting kicking the tires on some of these new features.

Uh the place for you to go is docs.ciq.com. Um that's where we have the upgrade steps for you to run through. It also has uh instructions on kind of anything that you have to do uh to uh validate context. Um and if you're doing anything that has to do with AWS, if you have to do an upgrade path before you uh even touch the cluster, we've got instructions on how to do that as well. If you're just seeing Fuzzball for the first time or you're in the process of evaluating for Fuzzball, here's what we would like love for you to do. Book an evaluation with us, uh, our sales engineers can bring up a fuzzball environment, kick the tires with you, run through any questions you have or any specific um specific scenarios that you want to touch on to make sure that we cover.

Um, they can also help set up a demo fuzzball environment for you where you can kind of kick the tires, including that MCP agent. Um, and so you can see that agent working in the loop. Um, it's pretty awesome and it gets you started extremely fast. With that said, I want to thank everybody for being here and taking their time out of their day for viewing this. Uh, we hope to see you next time uh when the next version of Fuzball comes out. Talk to you later.