How to modernize your on-prem HPC without rebuilding it

Are they still sharing? >> Oh, there we go. >> There we go. Excellent. Yeah. Thanks, Hope. So fuzzball um we're we're primarily talking about fuzball today and fuzzball is our HPC orchestration system. It's hybrid across onrem and cloud infrastructures and we're we're we try to present it as a write once run anywhere workflow system uh for performance inensive computing and we we genericize that beyond like a strict HPC context because we also want to support uh AI and machine learning workflows and and these things. um you know in some ways they look a bit like HPC but in other ways uh they do not.

Uh so yeah we're we're we're servicing that whole gamut of uh any any performance sensitive computing that you want to do um whether you want to do it in a cloud or in an on-prem data center that you have in a colo or locally in your data center. And like Hope said, we've um we've talked about fuzzball here before many times. Uh but for people that are new to the platform, Fuzzball uh just a broad architectural overview real quick here. I'm still an engineer at heart. Um we have two parts of Fuzball Orchestrate, which is our control plane that does everything for you. It's what starts jobs.

It's what provisions resources and you can talk to it as you see over here on the left. um with a traditional command line interface. Although I I say traditional, there are some some things that make it kind of a better experience than than other similar systems. Um but we have a a customer end userfacing web interface as well as a full SDK and and API system uh to talk to that control plane. So you can do kind of anything you want with this control plane from any place. And then the compute, your actual performance sensitive workloads run very close to the metal with our runtime that we call substrate.

And this runs on your cloud instances or it runs on your local data center compute nodes. And that's what starts jobs and does data management and and uh is orchestrated by the orchestrate system. And then substrate can talk to volumes and an object store. And this diagram uh is I I would say a little bit out of date. It's on purpose. This is where we are coming from. But today we're talking a lot about um improvements to the fuzzball system in the recent version 4 release. So this is a major release for fuzball and it it makes some changes to the way volume management works, but also some improvements um to uh like bringing the object storage and and the container registry into the platform.

So we'll be talking about that and demonstrating it a bit. But first let's talk about multicloud. So um in the past fuzball has primarily been developed uh in AWS. That's where we developed it and then we build it and test it in environments that simulate and then in our customers environments are on-prem data centers. So we had this hybrid story between a cloud and your local data center. Um but over the last few point releases and culminating in version 4, we've added among some other deployment targets the the kind of four major primary clouds at least in the the US space um for uh AWS which we already had but we've added Google Cloud, Microsoft Azure and Oracle Cloud.

So you can start up a fuzball cluster in any of these clouds. Um you don't have to move from wherever you're running. you can move to where the infrastructure that you want to run is or where it's available like if if you're trying to get a certain kind of GPU and it's only available today in this cloud over there you can run over there. Um and we've also if you've done a fuzzball deployment before it's it's been a bit of a an experience in the past we've simplified that uh pretty well. So now it's all a very simple command.

you run this fuzzball cluster target deploy and it's a guided process that walks you through the process of standing up a fuzzball cluster in that environment and then uh something I I didn't mention with our diagram earlier is that fuzball supports federation between these environments so you can have fuzzball in AWS and Oracle and Google and on-prem and then have a federation layer on top of that so you have one interface that your uh end users are in one interface that they submit jobs or work or workflows to and it's delegated out to the the right cluster where the uh where the infrastructure that it needs is.

One other thing that we've added, this is technically uh this came in version three but kind of at the tail end of version 3 and so we're talking about it a bit more in version 4 is um a simplified local deployment experience. So for on-prem deployments, we've historically done that with uh you kind of bring your own Kubernetes and or we help you deploy a Kubernetes and then you deploy the fuzzball orchestrate control plane into that Kubernetes. Um but today uh we now have a docker compose deployment that is greatly simplified and this is useful for small production deployments but also especially for just getting up and running with fuzball on a local machine to test it out and evaluate and maybe even just develop fuzball workflows locally.

And I'd like to um demonstrate that real quick. Uh it's it's so simple. I would like to do a fuzzball deployment for us right now. So let me switch over to this command line. >> Nice. >> So uh this is I'm running on my MacBook right here and uh I have Docker and Docker Compose for Docker Desktop uh installed already. So that is a prerequisite. It's a Docker based uh deployment. Um and I've listed to see if I have any Docker Compose deployments right now. There are none. Uh but if I run this Oh, not this command. That's for later. If I run this command which is even this is more complex than it would be for you.

I'm running a release here that's a little bit beyond the 4.0.0 that we released just to get some additional kind of work that's been done since that release. So I'm I'm having to specify that I'm getting it from a development server and what version I want to deploy. But in general, you would just install this command line and say cluster docker compose deploy and then optionally up if you want it to come up immediately. And it asks you some questions about the domain that you want it to be served on and if you need authentication credentials for that and usernames that you want or if you want to specify a password, it'll generate passwords for you if not.

And you know, it just goes through the process of pulling some containers out of our uh portal system. CIQ has a a portal and depot system that we use to publish packages and uh and container images. It's where all this gets published and where it lives. And so this command line can just go and fetch it from that container registry and stand it up uh automatically. And it'll take maybe upwards of 30-ish seconds to get this keycloak uh container started. Keycloak we use it for the backend authentication system. We actually can run without Keycloak now. also um but we we left it in the kind of the traditional deployment for for now by default.

So when we're done we get this output of like where to find our uh our werewolf web interface for example or sorry werewolf talking werewolf now our fuzball web interface and what the authentication credentials are. So, if we switch back over to uh to that window I had before and it's this one. All right. I can open just this new window. Go here. It's a new deployment. So, I don't have that. And I can enter the credentials from uh that I just got from this system. And that is as long as it takes to do a brand new fuzzball deployment in a local docker compose environment with a web interface with an authentication system that supports multi-user if you want if you're doing it as a a kind of proof of concept deployment in a shared environment.

We even set up uh I think you can see here. Yeah, great. We set up um an example compute node in the docker compose environment. And of course, if you were running this for a real production use case or even just to test actual hardware, you would decommission this test node and and add additional compute nodes into it. This was just so that you can run a really quick workflow and see it running very uh like immediately. So, so that's a lot of improvements that we made on uh on the deployment process. And this is like one half of what we're trying to do with fuzzball 4.

One is making it easy to deploy fuzzball and get it into your environment wherever that environment is. If it's uh a local Kubernetes cluster that you're deploying your orchestrate into, you can do that. If you have a smaller like single server or even just a local workstation that you want to test it on, you can deploy it there. If you are on any of these clouds, you can deploy there. Again, a single command to do so. But the second half of the the kind of overarching goal for um for Fuzball 4 is that once you have it deployed, we want it to be as useful out of the box without requiring as many external dependencies and integrations as as necessary.

So Fuzball is coming from kind of two worlds. One is a a uh kind of in an industry cloud perspective where there's elastic resources and you provision things when you need them and you're always interconnecting with other external systems. And another is an HPC mindset where everything's very tightly integrated and provisioned locally and in one environment. And so we we keep learning back and forth between these two perspectives. And so one of the the things we realized, Fuzball was expecting to integrate with a container registry and Fuzball was expecting to integrate with an object store. So it wants to have your volumes be um it wants to have them be ephemeral and kind of always ephemeral and uh you're staging data in from an object store so that your data is always available regardless of where you run.

And then when you're done, you stage it back out and your jobs are coming from a container registry and that's out on the network somewhere. So whichever cluster you run on, uh, it can access the applications and the operating environments that you need to run in. All of that is well and good and you can still do that. Um, that's the primary way we would expect Fuzzball to be used, but we also want uh you to be able to run Fuzzball without having to have all of this external infrastructure. So we we've done three things to enable that. One is uh better support for persistent volumes uh that are managed outside of fuzzball itself.

And so we'll talk about that in detail in a minute. I've got details of that here on the slide. But another is an integrated object store. So you no longer need to work with an external object store. We have a uh we try to call it an object cache. Actually I'm violating our own naming convention here. Uh it's not meant to replace like an enterprisegrade uh scale out object store, but it's a place where you can put data into fuzzball and then your jobs can use it immediately and then it's a place that your jobs can eress data into and you can view it immediately through the command line or the web interface.

And then once we have that, you can also use that object store as a container registry. you can put a a singularity image format uh container um that we use with app both appainer and fuzzball. You can put that into the object store and use it directly. So we'll see how that works and and talk through it in detail. But let's let's start with volume management. So, if you're a local HPC center uh or even in some cloud environments, you probably have existing campaign storage um or home directories. Uh and maybe they're on like an advanced parallel file system. Uh examples that that I have seen most often in the field are luster, GPFS or whatever IBM's calling it today and BGFS and of course there are others.

Uh so Fuzball version 4 adds additional capabilities for adding these volumes into fuzball and or adding the places where these volumes are stored. So we see this as a directory of directories. And so that's a a provisioning location, a place we can get volumes from and uh and then the ability to discover those and attach them to a running job. So for example, here here's the the Fuzball version 4 web interface. And uh I'm I'm in our integration cluster here. So this is kind of the latest and greatest. Everything that we've developed uh gets built into here. In fact, we've we've had to ask the developers, hold off for a minute so we can show people things and and uh and not have it be changing while we're doing the demonstration.

But we have these volume provisioners here. And this is a little bit of a legacy deployment because this was a version 3 cluster that's been migrated. So, um, these may not be laid out exactly how we would do it if this were a fresh version 4 cluster, but we can see here I've got like 24 volumes in this persistent um, provisioner. And this uh, is attached, this is deployed in an Amazon cluster. So, it's using Amazon's EFS file system. If we make a new provisioner, uh, we can attach it to a generic NFS system. Uh, we have Amazon EFS, Azure files, OCI, FSS. We noticed a couple days ago that the um the Google provisioner isn't or provision driver isn't showing up here in the list.

So we need to figure out why that's missing. It is implemented but we need to add it to this interface. Um uh but then this last one is host path. And this is a thing that we've been missing historically. Um that we uh every driver or every bit of volume storage that we wanted to interact with had to come in kind of network transparency uh transparently um through a CSI driver. And the end user, the not the end user, but the the cluster administrator, our customer was responsible for figuring out how to configure that and make it work. And that really just meant everyone was just doing it with NFS, which isn't ideal in an HPC environment where you have a real parallel file system and and you want to access it natively.

So now um while we we may continue to add uh kind of specific bespoke support for different file systems um like those parallel file systems, especially in cloud environments where they need special support to provision them in the cloud and whatnot. um especially for local uh deployments, HPC deployments, those those data center deployments, we now have this host path driver where you can mount your file system just like you normally would on your compute nodes. You have like a a slashcratch or a sluster or a slash DPFS, something like that. And then you can just tell Fuzball where that is and it can discover the volumes there and make use of it.

And in fact, I I hadn't done that with BGFS myself just because we we tend to have cloud environments available and instead of local HPC systems because we're software developers, not HPC admins right now. Uh but uh Wolf Gang got a BGFS deployment working for us just before the um just before the webinar here. Oh, wait. Is this the right place? This is Oh, this is the one I just provisioned. That won't have BGFS. this one. Um, we have a BGFS provisioner here that's using the host path driver and I was able to run a workflow in it and it's just a simple workflow. It's this hello world workflow, right?

It's it's just printing a message up here. But another thing that it's doing is showing us what's mounted. And so we can see here that I have at /mount/bgfs this file system that is a native BGFS file system that'll get all of the same performance benefits of being a real parallel file system wherever it's being served from of course. Um, but now we can expose that as well as really any of these Luster, GPFS. Um, I I would say Panassis, but or or Vadura, but that that's an NFS capable file system already, so that's not that's not really an issue. But whatever file system you're using, if you can mount it locally, you can make it available to Fuzball.

that also ends up making kind of the the um the persistent storage and the shared storage story um within Fuzball really simple. So here I have let's let's go back here. I have a couple of workflows running uh I have a few actually but two that I have running are a um a virtual desktop environment. This is this uh VNC virtual GL environment. And so this was a workflow that stood up two volumes. I have a a home directory, let's say, and a temporary one. So this one's persistent and this one's ephemeral. And then I pulled an image and and now I'm running this desktop. And so here I have that desktop running.

And I have access to my my home directory that's mounted from wherever home directories are stored. Mine happens to be in that persistent provisioner, but a a relatively normal thing that I would expect is that you would have a provisioner that's pointed at just the /home that your environment is serving and it's serving out your your home directories as volumes and I can say give me mine and attach it to my work and then I have this separate temporary scratch space um that I can also make use of. Uh, and again, this this desktop here running in the remote environment and it's proxied here to me through the web interface without me having to do any extra work.

Something that I did want to demonstrate that I didn't have working before um just as an aside um that this is even a a 3D accelerated desktop. So, not only uh like it's easy to render something there, but we can see that it's actually speaking to the GPU on this GPU node to do this render, which I knew we could do, but I was glad to finally get working. And once I got it working, Wolf Gang told me he'd already done that a few weeks or months ago, and I I should have just talked with him, but >> that was very cool. >> Yeah.

Anyway, it's all so like the the goal of this, we've talked about this before on webinars, but the goal of this is to be able to generate some uh some simulation on one side and then render it uh with parave or some 3D rendering system on the other side and do this in real time. So the other job that I have working in this environment here is a Jupyter notebook which could be a parallel uh like an MPI job just for point of visual reference here it's this and uh this job here is running in this workflow here. Yeah. So it's just another workflow that has that same data volume attached uh and then got a Jupiter uh image that we didn't have to build.

We didn't have to do anything for it. It just pulled it directly off of the where where the Jupiter project puts its uh its images in quate.io and started this Jupyter service and I'm able to access it directly through a proxy from the foss web interface and run a Jupyter notebook and it's able to generate you know images and then I can see those at the same time over here in uh this desktop. So that is Oh yeah, it's just right here. So again, our our work or our um our use cases here relatively simple. Uh but the uh the point we're making is you can have these volumes.

They can come from anywhere. They can be stored on any file system and then they can be attached to multiple jobs and they can use them to communicate. One job can be generating output into one place and another can be accessing it in real time or after the fact and uh and you don't have to move that data around to make it accessible u from fuzzball. But the the last part of that uh kind of data storage process that we wanted to show. So again in the past let me let me show how this would work. So we have this workflow here. This is a running workflow.

Actually, you know what? I'll go from the the one that I was actually going to demonstrate this with. We'll go to this is a Blender workflow that I ran a little bit ago. And there there's a little bit of a rendering bug here where because this was a a task array job uh that did many many tasks, I got many many dots here in the graph for each task that ran. But when it was done, it was done rendering it and it assembled a video. And historically what you've done is when it's uh when that work is done you have some data in your ephemeral scratch volume and then you egress that to an object store because that volume's going to go away when this workflow is done.

It's temporary storage. So um our kind of original plan for how you would interact with fuzzball in that way is that when it was done it would egress the data to an object store but that needed to be some other object store that you were responsible for setting up that you were responsible for configuring and then you told fuzball how to contact it. There was just a lot of steps for the first time you're using fuzball. Maybe you don't need to or want to deal with all of that. So what we've done now is introduced uh a local object store inside of Fuzbball Orchestrate um that that you can egress data into um uh directly uh and so then you can see those objects.

So that's that's what this workflow did when it was done running. It had generated this output.mpp4 mp4 and it saved it into my local user space um uh in our local object store and then you can browse that directly in the fuzball web interface here in this object list here in my user private space and I have a few different files in here. Here is the the movie output. I've got a few other images here that I just put in here for sake of of demonstration. This is uh a an event that some colleagues of mine and I were at earlier. This is our announcement of the uh participation at ISC that we'll be at next week, the international supercomputing conference in Hamburg.

Uh this is an inside joke for someone else on the CIQ staff, but um we don't have preview for everything right now. We don't have video previews working, but it's still really simple to just download this directly from here. And then let's see if I can play it in the browser. I had forgotten that I it's somewhat uh Yeah, great. and then just render it here. So the very soon we'll have uh a preview capability for the videos as well in that pane. But it still gives you this immediate access without having to have gone out to an external object store. Um you can also use this for inputs.

And so I have another workflow running. This one is an open web UI and Olama workflow. I have it right here. Um, it's just kind of a standard chat GPT style interface where you have a model running. You can select what model you want to be talking to and you can give it prompts. Uh, this was a a really simple like question or response. Are you working? Show me that you can generate uh an output. But then I also had uploaded some data into it. Um, what I like to do is just grab data off of Project Gutenberg, these free ebooks that are available on uh on Gutenberg.org, I think and throw those into an LLM and then ask it questions about it.

And historically I've done that because it just is plain text files that it can eress directly and I could just sidestep the issue of not having an object store and it was very easy. But now I'm also able to tell it with this running workflow here that I want to ingress data that happens. Oh yeah, it happens in the in the volume. So here I have a volume just for documents that I'm going to ingest into my language model. I'm going to give it the files that I put into this volume to process so that I can ask it questions about it. And I can do that ingress from the object store.

And so here I have, you know, I've given it a copy of Romeo and Juliet in plain text format because it's easy for it to ingest that um and stored it on this volume temporarily. And then this workflow was responsible for processing that with the model before it made that model available to me for for prompting and inferencing. Uh and so this makes it then easier to get data into whatever workflow it is. Here we're using an AI model. Uh but you could also use it for input data into a simulation, input data into some data analysis, whatever you need. You no longer need to have put that in an external place outside of Fuzball or done it kind of on the side where you have this volume that you're loading data into without Fuzball knowing.

You can just upload it directly into Fuzball uh with a little interface here or you can put it with um uh with command line if you prefer uh and use it there. And then lastly before I pass on to Wolf Gang um for a little bit about the multicloud stuff once we have this object store you don't have to use it for just application data you can use it for um for the containers themselves so every stage in a fuzball workflow if I go back to our open web UI one this is a kind of large multi-stage workflow does lots of things sets up volumes but every job or service in fuzball is running in a container And those containers often are just available.

They're on DockerHub or they're on some projects registry or they're in the GitHub container registry. And a lot of our workflows just pull them and use them directly. Uh and that's great because it means that you don't need to do any translation yourself. It's all handled by Fuzball. You don't need to store those yourself anywhere. Um you can just use these services that are or you know this data these applications that are directly available online anywhere. But if you do need a custom image like uh like I did for the um this virtual GL system, I created the image that uh that we're or I started it.

I think actually the one that we're running was something that Wolf Gang improved. Um but you know, we here at CIQ, we created this image that we're running a virtual desktop in and we put custom packages in it. We made sure all of the the OpenGL and VirtualG stuff was configured properly. And historically, we've pushed that into CIQ uh portal or we've pushed it into the GitHub container registry, but now you can put it directly in the fuzzball object store uh and use it from there. And so I I have just for sake of uh of demonstration rather than than rerun it and switch back and forth between the terminal again uh this is what that looked like um I just logged into our fuzball context for the cluster I was in and ran uh a put to put this uh VNC.

It's a a singularity image format container uh that we put into the object store just as a regular file. And once it's there, my uh my service here, my virtual desktop service can just access it directly and use it without having to go out to an external container registry without having to redo the conversion because every time you're using an OCI image, it does cache it anyway. Um, but it uh it's already in the the right format in the object store and it can just use it directly from there. because I said it caches. I have realized one thing. I know I said I'd hand off to to Wolf Gang here uh after that, but one more thing I neglected to say.

Every time we are doing an ingress or every time we're using an external container, we're using this object store to cache that. And that data is visible here in this cache name space inside of our object store. So here we can see images that came from the the Docker container registry. This is, you know, we often use Alpine for small things. We can see uh, you know, various stages of of Rocky Linux that we've pulled in here. This is, uh, I I did a previous hello world with a Rocky Linux image. So that's where this got converted and cached. But then also just any data that we pull from outside gets cached here automatically.

And those um, that cached data has an automatic, you know, time to live value. We default it to 7 days. you can change that or you can set it explicitly. Um, and so data that you're using frequently stays in the cache and doesn't have to get reingested into fuzzball every time. But then data that you aren't using frequently uh gets flushed and it'll have to reingest it the next time that workflow runs just transparently without you having to do anything. you can configure it to bypass the cache if you really want a newer updated version and then it will update the version that's in the cache and you'll have that for the future.

So I imagine there will be questions about that but uh in the meantime I'll pass it over to Wolf Gang and see what he has to show us about uh running Fuzball and and the same workflow in multiple clouds which is an exciting thing. >> Thank you. Okay. >> Yeah. So, um, thanks thanks for that, Jonathan. Um, so I just wanted to give you a basic a basic flavor of what it looks like if you're running on different clouds and and the goal is to basically for you to not see the difference. Um, so I'm not showing deployments for all the clouds that we support because they will get repetitive, but um, I basically have here AWS, Oracle Cloud, um, and Google Cloud.

And I'm just going to show you what it looks like to run the same workflow on the different um clouds and and also um on on a cluster that we just deployed on prem. Um okay, so starting out here. So this is our workflow catalog. Jon has already used it. Basically a set of templated applications. Um the particular one I'm going to use is is is called bolts 2 which is in the bio space. also for those from the bio space that's a that's a um tool that's used for structural prediction and affinity prediction. So you can use this to screen virtual compound library. Um so in this particular case we're running um we're running a protein against a a fixed set of a few hundred compounds and and within these applications you set a few parameters.

Um so for example there might be slight differences in how storage is set up between the clouds but um so you know we have to tell it which volume to use but then because it gets mounted in the same place all the paths uh within the volume stay the same. So any volume that you specify here gets mounted as slash data so your workflows referring to um pass within within the mounted um mounted volumes don't have to change at all. Um, so in this case all I all I did is specify a persistent data volume which is going to use to to cache its data and then store its outputs and it uses a firml as scratch space to set up the actual run.

So that's that's what that looks like in AWS. Um, it sets up a bunch of volumes and then it runs three different jobs set up predict and then it does some cleanup. Um, this one runs on GPU. So, this may take a second to actually start up. Um, so this was AWS. Um, but now this is Oracle. So, this is running on on OCI. Um, looks exactly the same, right? So, we can we we specify volumes. In this case, the volume setup is slightly different, but I don't have to change anything else because again, the persistent volume gets mounted in the same same place.

the job from the jobs perspective it run it uses the exact same paths um I can start that here but unfortunately I've been unable to get GPUs this morning on on Oracle cloud which is how you know it's a live demo um so this may not actually start um it may fail to start if it can't require if it can't acquire a GPU um uh yeah and there it is yeah couldn't get a GPU um same and this is Again, this is Google Cloud. Um, same exact thing. Um, it's I have to tell it the the setup of the volumes. Um, where to put the output data and you start it and it looks the same.

Um, I just realized it doesn't look the same because it didn't actually tell it where to save the output, but you get the idea. So, uh, deployment to clouds is very to different clouds is very easy. And once you are working um and and once you have it deployed it attempts to look identical as much as as possible um within adjusting for you know different setups of storage and things like that. And again if you deploy this uh on prem so this is an on-rem deployment I just did this morning to to set up the BGFS. Um again looks the same. Um I can't run this particular workflow here because I don't have a GPU.

only have CPU nodes, but if you started any of the other workflows, it would look exactly the same. And and not only not only that, um there's also a system on top of this called federate that should present you essentially the same interface, but allow you to actually submit two different post instances running in different clouds or onrem. So that making hybrid computing much easier. >> Yeah.

And that's really the point like it's all well and good to say oh look I can be in this cloud or that cloud or onrem and it looks the same and there's value to that your your your customer your user your researcher they have the same experience wherever they are but once it's the same experience they don't need to go to those different places they can be in the one federate place and do whatever they would do set up their workflow submit it and it doesn't matter where it ends up as long as the resources are available and fuzzball can make that determination it doesn't need to change anything depending on what cloud it's in because it's all abstracted away.

It just does the right thing depending on where it runs which is >> and if and if someone is let's say they have a multicloud setup and one of the clouds has an outage now they have an ability to uh not just be stuck due to that outage, right? that they can uh shift to the one that is still operating >> or even just an availability issue like if we had federate set up across these you know three clouds and onrem that that uh that Wolf Gang has set up here then uh if it sees oh there aren't GPUs available on the OCI instance right now we'll run it somewhere else instead um and you don't have to make that determination it just moves the workflow to where the resources are available.

Yeah, I didn't set up Federery for this, but this is this workflow is actually a good example. It it would have it failed in Oracle cloud because currently none of the GPUs are available that that would have worked for this job. So if we had federate setup, we could have rerouted it to a different cluster. >> And and I should also mention that so not only sort of does it behave the same across the different clouds, there's also different ways to use this onrem that essentially look the same. You can have it run your entire cluster with dedicating nodes to file as compute nodes. Uh you can also use an existing if you have an existing cluster running slurm or pds as your scheduleuler.

You can have fusball run on top of that and basically allocate itself temporarily workers from the pool of slur or PBS nodes. So you don't have to give up on on a traditional. You can run this in parallel and on top of slur and PBS as well. And again essentially looks the same from a user perspective. >> So um one of the questions I have how does fuzzwall handle a site with multiple parallel file systems that could be mounted differently across nodes? >> Yeah. So um the provisioner configuration is per cluster. So uh if you had different file systems in different locations uh in different environments, you would just configure one time in each cluster the provisioner that maps that file system into fuzball and then all of the volumes in it become available.

And once that's done, uh your fuzball users don't have to care where that data is mounted on the compute nodes or even what file system it's in. uh they just say I want this volume with this name or they if it's ephemeral they just say give me a volume with these attributes and the the kind of native attribute that we worry about is uh well the provisioner if you want to specify the provisioner typically we tell people to to let that be discovered but then the size of volume if they're creating a new ephemeral volume but then the site can also uh add key value

pair annotations to any provisioner so they can say this is a flash provisioner or this is a GPFS provisioner or this is a fast and this is a slow provisioner and so people can say I want a volume uh with these characteristics and it will match it with a provisioner that has those characteristics regardless of where it's mounted or what file system it is on your actual local site >> go ahead I can show an example by running >> well I was gonna I was just gonna show what this looks like so this Um so this ah it's type um so these are the provisioners that are available on on this particular system.

So if we look at type So that that illustrates what what John was saying. So this this is the BGFS um provision provisioner. It uses basically BGFS mounted on the compute nodes. Um and it it has an annotation that that specifies it as a fast tier. And then there's a there's an NFS. So data here is NFS and specified as slow. and then um to use that. So for example, if you just needed some large slow scratch space um you could tell it this is how much I need and anything that's annotated with tier slow basically and that then will within your job this will be SLcratch no matter where it's being served from.

>> Okay, very good. Thank you. Um one more. Uh so if the team is adopting fuzzball and they already are submitting through slurm what what changes for them if anything? Yeah. So that's where Wolf Gang mentioned we have, you know, we have provisioners that like we call the static provisioner where you just have fuzzball substrate running on compute nodes. That's kind of the most traditional on-prem environment. And then we have provisioners for clouds. So AWS, GCP, Oracle, and uh and Azure. And so those are are elastic dynamic provisioners. You you put work into fuzzball and it says, "Oh, I don't have resource for this." and it goes out to the the cloud that it has access to with the resource shapes that it has access to and provisions a node for you or an instance for you to run on.

Uh we have dynamic provisioners for slurm and PBS that let you uh configure fuzball to be able to get resources from an existing slurm or PBS system and then run fuzzball workflows on them. And it's it's doing that as though your slurm or PBS environment is an elastic cloud. So fuzball sees it as though it's provisioning an instance. Uh but the end result is that you have a a fuzz or a slurm or PBS job that's running fuzzball substrate and then the fuzball work runs through that. So in that environment, especially in a transitional state, um your existing slurm or PBS users could continue to use slurm or PBS as they are today and um and submit work that traditional way.

Fuzzball users could sim uh simultaneously use fuzzball to craft workflows that are portable across uh fuzball instances that'll go out to the cloud or run locally. uh and it would dynamically allocate resources out of the slurm cluster let's say um just like it would if you were submitting them natively and another aspect there that uh can help in transition between the two is you can you know appptainer is is pretty popular in a traditional HPC environment you can use the same appainer containers in outside of fuzball to run with slur or PBS and then continue using those same containers in the fuzball environment environment. So you can move back and forth.

You can use them to to get something portable in the traditional environment and then bring that into Fuzball and then it can run anywhere, not just on that one cluster. >> Very cool. Um, one of the last comments I will make is uh everything just seemed to move very very quickly, right? It seems that getting started is and I know there's prep, you know, for a webinar, >> but I think um this this is not a system that would would take necessarily hours, right, or or an application that would necessarily take hours just to get up and running. the the deployments into the cloud can take a little bit, but those as demonstrated by the fact that we can deploy locally really fast.

The issue is not so much the fuzzball deployment as it is the provisioning of all the resources in the cloud that we then install it onto. So that that remains a little bit of an issue. It can take like 20 to 40-ish minutes to deploy into a cloud, but it's not a huge deal and it's it's simpler all the time with the additional interfacing that we're putting on that. there are fewer gotchas, fewer ways that it can go wrong, which when you're talking about a process that takes 20 to 40 minutes, it going wrong in the middle is always frustrating. So, we're we're trying to take more control over that process and kind of make it a more guided experience.

>> Very nice. Well, we are at time. Fantastic demos. Um great uh great overview. Thank you so much uh to both of you, Jonathan and Wolf Gang. And as a reminder, this will be available on YouTube. If you have questions for the team, you can always reach out at info do uh info@ciq.com and you can always go to our website. So, thank you very much for joining us today and we will see you in our next webinar. >> Thanks, Hope. >> Thank you.