Fuzzball (HPC 2.0): Demos & Updates

research i've worked at the nih and a few other places in that capacity and i also have some background in high performance computing and a little bit of background also with apptaner excellent thank you dave and then we're going to go to greg who's uh the mystery man this week hiding in the dark why am i the mystery man oh it's just the background oh yeah you just did it so if i don't move for a little while it readjusts uh the light settings so um i i think the trend here at this point is is biologists and uh so i'm a biochemist by degree

but turned into high performance computing and an open source um guy and i'm i've been lucky enough in my career to be part of you know a variety of different open source projects and and community endeavors and some cool companies oh oh i don't know if everybody can see this rocky linux v9 so be ready for it was up in the background that one no no no it was just a random one that came up when i when i first installed and you can see i'm actually using it but i put it on a back uh on a screen that just has a nice pretty

background showing that it is nine very cool very cool all right so brian i know you've spent quite a bit of time getting something together to show us uh go ahead tell us what what you're gonna show us and tell us a little bit about it uh show us and tell us why why you think it's important cool uh so what i've prepared today um is a open foam uh demo for running through fuzzball uh so with cfd simulations a lot of the time um depending on the size of your model you may require a significant amount of compute resources and this can be on

the order of one in the order of hundreds of cores so you know when you're on your on-prem hpc system you submit your job sometimes you'll be stuck in the queue waiting and if through the use of fuzzball we can connect your on-prem resources with uh cloud resources and that way you can bypass the queue get your simulation running on the cloud and get your results a lot faster so yeah uh let's jump into it let me start off by sharing my screen while brian's doing that we do have a few a few ideas that have come in so if you have anything else shoot

over to us okay so uh can everybody see my screen we can cool cool so i am in this open phone demo directory right now uh in this directory i have a open phone.yaml so uh how i'm gonna be running this demo today is i'm gonna kick off the workflow and um the the workflow will provision some cloud resources once those cloud resources are up the uh workflow will begin running on those cloud resources so let me just start off by kicking off the workflow first and while we wait for the resources to provision we can jump into the open foam yaml and we can

talk about what's going on there so first to kick off the workflow i'm going to run fuzzball workflow start users instant accident.co i'm going to name this job webinar open foam demo and give it a unique id of one and then i will call open phone cool and as you can see my workflow has started and for me to check the status of this workflow i can call fuzzball workflow status uh okay and i can we can watch this and just copy this id here boom so uh as you can see our workflow has started uh the volume has been created the image has been

pulled and our job is in a pending state uh and what's happening in the background as we speak is cloud resources are being provisioned and uh and yeah and once they come up our job should be in a started state uh so let's jump into the open foam demo and let's see what's going on in here okay so within this uh workflow yaml file um we have um a volume uh so this volume is a ephemeral volume and this is where um our workflow is going to execute out of uh in this workflow we have um a single job uh this job is called run

motorbike simple foam uh within this job we are going to be using a image and this image is being pulled from docker hub it's open cfd's uh latest uh default open phone container and uh the command we're running here is um some bash and basically what is happening is we're going to be taking the uh motorbike in example from within the installation copying it to our working directory uh setting up our environment by sourcing this bashrc file and then executing the all run script so within the all run script what is hap what is going to be happening is we're going to take our motorbike

model uh decompose that model into six parts because we're going to be running on six cores and uh this model will be meshed and then the simulation will be run uh using uh the solver simple foam for 500 iterations once the iterations have completed uh the results will be uh reconstructed so next in our workflow yaml we have um some environment variables that we're setting uh wm projector we'll just point to our open phone installation uh the next two um allow us to run some mpi commands as root uh these are running within the container so that should be all good um next we have

our current working directory so uh this job will be executing out of slash data so jumping into our resources like i mentioned earlier we're going to be running this simulation on six cores um and we are we will be requesting um one gig of memory so with our uh ephemeral volume we're going to be mounting it um to slash data and uh which is also our working directory for our job so uh let's jump back and see where our our jobs at so it looks like our job is still pending and we are waiting for um the resources to provision so uh once the job

is in a started state we can begin um live tailing the uh logs of this job and uh to do that we can execute fuzzball workflow and we can add a dash f here to uh till it and we can put in the name of our workflow so let's just copy and put that there and we could also give it the name of our job as well so as you can see above um our job has started and um and we can begin live tailing so let's do that okay so um as you can see uh our model has been decomposed with uh decomposed par

and currently we are running um snappy hex mesh which will match our motorbike model so i think it'll take a couple minutes for that to complete um cool so so i got i got i'm sorry i got bounced out of the studio for just a minute um due to some technical reasons but i'm back in i don't know if you already covered this but i wonder um you know for for somebody who's who doesn't really know what open foam is might not have heard of open film um did you kind of you know could you talk a little bit more high level about what it's

for and what the motorbike simulation is and so on uh yes yes so open foam is a open source uh cfd software that um where uh engineers can run various types of physics simulations um is typically used for uh just some uh yeah so um happening in this model uh we are basically uh we're basically going to visualize the velocity of the wind against uh the motorbike model um and afterwards uh after running these 500 iterations we should be able to visualize these results in a visualization software such as a pair of view so you might use open film i guess if you were trying

to design a more aerodynamic motorcycle or maybe uh trying to design an airplane wing yes um that would provide optimal lift or something along those lines yes exactly cool cool so um as you can see um our workflow has progressed um so uh our model's been meshed the initial conditions have been set and right now we are currently at the state of running simple foam and for 500 iterations which will should take another couple minutes [Music] and yeah we can jump into the comment section just to see if we have any questions i think we've got a question coming in from youtube yeah there we

go oh i was saying that a lot on mute sorry guys control picking the same way as you would without fuzzball if read modifiable is equal true uh yes you should be able to modify the control dict so uh within your workflow before executing um so once your resources are set within the workflow uh you can set you would modify control dict and then from there when you execute uh decompose par uh your model should be decomposed accordingly thank you brian yeah by the way these comments coming in are hilarious i've been i've been uh contributing and talking with that in the youtube uh channel

yeah there's a number of good ideas and comments coming through and i was just about to say dave is well known in this industry for dad jokes we should we should have some dad jokes as a um yeah interlude you gotta you gotta say which dave though there's so many days true i guess we'll start calling you god love god do you have any bad dad jokes there's no such thing um but i i i do have uh one cute this is one my son has been telling for you i have a six-year-old son and he's been telling this probably for three years he loves

this joke and the joke is um how much does it take to pierce a pirate's ear no idea the answer of course is a buccaneer oh that's great that's great and on that note so as we can simple on that note jumping back into our demo uh ripple foam has completed reconstruct par mesh has run so our results the mesh has been reconstructed and then reconstruct part will take all of the partitions that the model was decomposed from and combine it all back together cool and and in a more practical case um it if you have a bigger model you would want to run this

on um more on multi-node on hundreds of cores so um if you guys are interested in that please leave it in the comments and uh we will be definitely we'll be happy to address that in a future webinar cool so that will conclude uh my demo for today and i will pass the mic back to zayn we do have one more question brian yes for sure yeah is there some resource monitoring feature i'd like to find out if all the cores are 100 for the simple phone case uh yes uh i believe you should be able to so how i would check if the resources

were fully being used is x you can within fuzzball you can uh get a shell within into your job and from there you could uh check if all the cpus are being utilized from there so yes the answer is absolutely you can do that so you could do it that's a little bit of a brute force method but that is a really good idea um i'm gonna bring that back and i think you know uh forrest brian and dave will probably do doing the same bring this back over to the engineering crew and you know see if if real time monitoring is something that we

could put in for a later release it's a great idea who's taking that note got it zayn i guess that's you i got it i wrote it down we're good awesome all right thank you very much brian that that was great we appreciate it so i don't know if you haven't been reading the comments this is this is getting funny um i'm gonna turn it over to forest forest we were playing with this earlier today this has been a lot of fun uh you talk about a way to kill a lot of time really quick and get some really interesting uh laughable results so first

what are we doing today hey everyone so the other demo that we're looking at today is an ai model that's kind of making rounds around you know the metaverse at the moment uh there is a company called openai that produced they've produced a few different very interesting ai models over time text-based ones i think they call them like jeep gpt or something like that um and then also some image based ones they have one at the moment called doll e2 uh dolly 2 is massively proprietary it's something they've trained you have to like wait on a waitlist to get access to it that type of

stuff um so it's a little bit difficult to you know really really mess around with um but someone has built kind of a miniature version of it that they call doll e mini um doll e mini much like the the larger versions like dolly two essentially takes in some text input um it's been trained on essentially hundreds of millions billions of different images of all different types of you know annotated training data type stuff um and you give it a prompt and it generates images based on that prompt so if you give it you know a cat sitting on a cliff by a lighthouse you'll

get a hypothetically images of cats sitting on cliffs near lighthouses um it extends out to being able to like do things in different art styles and i think i might pull these up here in a second because i do have some results for this but we've got you know for example you can do things like a picasso style painting of gregory kurtzer or a person something like that um or of just you know basically anything you know the cat and it'll give you picasso style painting of a cap so dolly mini typically you'd get access to it there's a version of it that runs online

out there there's like a website you can go to put in your request i got sick of having to wait for other people's requests to get done um so i kind of looked into it and realized that oh well this model is freely available out there you can go and download and mess with it yourself so i figured i would turn it into a fuzzball workflow and we would see how well that works um so i'm going to go ahead uh first off and just kind of run through the uh like workflow and stuff like that and then here in a bit we'll take a

look at some of the results that we've generated we've had some of these comments coming in so far i think i've generated a rainbow colored cow fortune teller a farmhouse with a cow on the sun and a squirrel wearing a sombrero in the jungle so i've generated those i have some results here i'm going to look through or i'm going to explain this workflow real quick so we know exactly where these came from and how this works and then i'll go ahead and take you through the results we've generated so far um see i'll go ahead and share my screen we'll take a look at

this right here we got some questions about the schedule we'll get to after we talk about this cool okay so yeah you can see here in front of me i've got basically just a vs code window here um yes the font is a little small now that i look at it um so my apologies there but just to kind of take you guys through exactly what this workflow is doing this is essentially uh broadly an example of running ai inference on fuzzball so this is not like training an ai model this is taking a pre-trained model and then you know actually running as i said

inference with it so just to explain what we're looking at here up here at the top we have a file ingress section so we're basically just going to we have a volume sorry a uh a data volume setup section so we're going to set up a data volume called v1 this is an ephemeral volume so it only exists for the lifetime of this workflow we're not ingressing any data into this because the input to this workflow is basically just um the model itself and then some um piece of text basically but we are egressing out these images into an s3 bucket so that's how we're

getting access to these in the end so this workflow runs it gets to the end it generates the images it tars them back up and then pushes them out to an s3 bucket um this is the and i've named this job a little bit wrong it should have been you know infer model or something like that but i think i copied this template from elsewhere this is not training this is inference my apologies um the image that we're pulling down here is basically a um it's an obtainer image that is based on this recipe right here so i've taken the script i've basically just taken

the code that the dolly mini people provide for their inference pipeline as a part of a notebook that they have like a jupiter notebook type thing i basically taken that code put it into a script and then kind of taken the rest of the setup that they give you in order to get the docker file for this working and put it in there as well i built this pushed it out to a container registry and i've got it now deployed on fuzzball you can see i'm providing credentials to that registry there the command we're actually doing is the script with the inputs so you can

see that right now we're processing a saddle on a cow in a car race a screaming gopher caught in a spider web and richard nixon in a tutu throwing confetti so we'll see how those come back we have cwd slash data so this is just um this command is being run inside slash data um we have a few uh environmental variables the most important one is this right here uh the pre-trained model i'm pulling down from website called weights and balances or weights and balances weights and biases and i basically just stored my api key as a secret within this fuzzball cluster so it can

be templated out and i don't have to display it openly or manage it outside of just inserting it into the fuzzball cluster we're using an aws g4dn.8x large i believe for this so we've got 16 cores 120 gb bytes of ram we're requesting and then one gpu um we're mounting that data volume uh and then once this is actually done generating the images we go down here to tar up the results so we basically just take the nine images that we generated put them into this drop that into this directory here and then up here at the top this egress will trigger and we get

the images out so that's that as mentioned the script file that i'm using is basically this it's basically just their code that they provide for the inference pipeline here but i've taken it out of their notebook and just basically put it into a script um so we can run it that way um yeah so that's that i've been running it let's see what this is doing down here if this is finished or what this is up to the vs code terminal sometimes doesn't like the scroll back here so let's see what this latest one i'm running is up to oh it's finished oh good okay

cool um awesome okay that means that we have more results now sitting in s3 to look at so just to kind of explain exact um you know what i kind of did there on the command line uh you know you can see i'm i'm basically able to control this from the integrated terminals that the s code provides so i'm doing you know fuzzball workflow status i'm doing fuzzball i can do like a fuzzball workflow log and this kind of produces a lot of log barf um so we're probably gonna see a bunch of random stuff yeah so we get a bunch of different um you

know kind of log output there you can see the prompts being printed um you can see it saving off the images um so i can you know control the execution of this workflow from within my ide and uh yeah without further ado let's actually take a look at the results that we've got so far so this is in essence um the dolly mini ai model running in friends orchestrated with the fuzzball system um and we're you know taking live requests of what we want that to do so let's take a look at what those have yielded shall we and the exciting part i know right

let's see here go ahead and download these latest pack of images i've got it set to generate for time's sake um i think if you go to their website and mess around with it it generates you nine images per prompt but just for time sake i have it generating uh three per prompt i just download that 24 i did let's stop my screen sharing oh and i do want to point out um that container that we're using is running on rocky linux so this model and what we're looking at here is yeah running from an app tainer that's based on a rocky linux space image

so just want to point that out um here is our file of results from the first this is already smiling that's hilarious um so yeah once again this was um a rainbow colored cow fortune teller a farmhouse with a cow on the sun and a squirrel wearing a sombrero in the jungle we're just going to go ahead and we will get all the questions that are coming up after after we see so there's our rainbow colored cow fortune teller i don't know if this counts as an lol cow demo but you know we're getting there i think um so yeah we've got that i think

these are uh i think on the sun may have gotten lost it looks like we're in the sun here but um here's some cows in a field here's a scoop it works remarkably well apparently for generating squirrels and sombreros so yeah so we've got that here's another cow oh there you go here's more cows in the sun headless cow here's another squirrel with a sombrero here's i think our final of the three cows and uh there's us back in the sun again and then here's our last scroll with this umbrella so so yeah um you can see that this works quite well um i'll go

ahead and pull up the other file so we can see some more of our very creative audience's results um this one right here is gonna be let's see a saddle on a cow in a car race a screaming gopher caught in a spider web and richard nixon in a 2-2 uh throwing confetti so let's see how this turns out oh yeah there we go okay so there's a saddle on a cow i think um let's see here there's a gopher maybe nebulously caught in the spider web perhaps yeah maybe there's richard nixon throwing confetti maybe not the tutu apparently not 2-2 might be a little

bit too specific for this model but we'll see there's another half created cow part of a cow there's another gopher kind of you know doing something here's richard nixon partying down again and then we have our final few which in general did not come out as close as we got for the last ones but you know he has hooves it looks like he's given us his signature wave there actually wow so fantastic yeah so that's up um that's the dolly mini model i'll go ahead and uh start running some of these additional requests and i think our plan is to follow up with a post

or something like that that'll kind of show off some of the results that we got absolutely um but yeah i'll fire off some of these if we want to maybe hop over to some questions or something like that that was all right uh thank you all for tuning in that was uh you know dolly many running on fuzzball yeah thanks for putting that together force it's also nice to actually show using somebody else's model to to run thank you very much all right let's start having some questions i think there have been quite a few come through here i know greg has been answering some

of them as they go trying to go back as far as we can here uh so we're talking about schedulers so does fuzzball offer a scheduler service similar to something like slurm it doesn't matter newbie dave it's all good so kind of so the way it does scheduling is very different than slurm because it's more like an orchestrator than just a straight scheduler in the sense that it can do services it can do other types of processes but it's very much like slurm as opposed to more like an orchestrator and the fact that it can also do rich policies and allocation of very specific hardware

configurations so it can do its pneuma aware you can do multi-node processing and so on and so forth it can do a management of consumable resources and it knows when those consumable resources have been consumed again more like a slurm scheduler in that case but again it's running as a micro service uh entity within our microservice stack so the general architecture of fuzzball kind of you can envision it as um that there's there's two clusters running in one cluster there's the management side which is running kubernetes and on top of the management side we have a bunch of orchestrating services that run so we have

an image service a data mover service a volume service scheduler service and get all these different microservices that are all working together and able to scale up and whatnot independently in traditional microservice style and then we have the compute cluster now the compute cluster is running a very lightweight container hypervisor or runtime vm hypervisor and it is api driven so all of the interaction with it is over api and it is uh kind of focused on leases so you can ask fuzzball and this lower level by the way is called fuzzball substrate you can ask substrate for example i'm going to need 10 gpus i'm

going to need 30 cores and i'm gonna need this much memory and if fuzzball substrate has that available and can actually allocate that for the amount of time requested it will respond and say here's here's a lease id and whenever you want to use that that resource just send me this lease id like a token and and then we can go ahead and schedule that now when you have thousands of substrate instances you have to manage them now with orchestrate and orchestrate is the next level up in the fuzzball stack and orchestrate is that microservice platform so you can think of it again as running

your your orchestrate instance on top of anything from kubernetes to vmware tanzu to any one of the kubernetes stack we have a version of kubernetes that we put together that's super easy to install just to kind of facilitate that deployment so you can very easily kind of deploy that management cluster and then you can run the compute cluster using werewolf for example or whatever you want to use to provision your resource and and then that cluster can scale independently uh now when you're running in the cloud it's a little bit different in the fact that that orchestrate cluster is going to run on whatever cloud

service provider is offering as a kubernetes stack you can run your own kubernetes stack in the cloud but it's easier just to use whatever they're providing and you can run fuzzball orchestrate on that stack so when it's not doing anything that's the only thing running and it's a very lightweight small stack sitting on that kubernetes resource as soon as jobs start coming into that kubernetes uh instance uh assuming that that fuzzball orchestrate instance uh it automatically provisions the compute resources that you need and so it'll automatically scale up and scale down so what you just saw in today's demos is i believe we ran everything

up in in one of the clouds i don't even know which cloud was probably aws or gcp but but one of the clouds and um uh when a workflow was submitted into fuzzball it automatically provisioned those those cloud instances to then go and run the job and it tears those cloud instances down when the job is done so it gives you that ability to kind of elastically scale up and scale down as needed as opposed to when you're running on prem and you could theoretically run out of resources and then you have a job queue that develops and that q is now managed according to

priorities just like you'd see in slurm not something you'd see typically more like out of a kubernetes scheduler i i kind of used this question to actually touch on a number of questions that actually came up in the chat and to talk through all of those so sorry for going on so long but i was actually answering a few different questions there nope that's great i was scrolling through i think that answers todd's question as well can i jump in and underscore one of the things that greg said certainly so he covered so many things but i think you know i'm fairly new to fuzzball

and i'm still looking at it with fresh eyes you know as i think brian also too and so one of the things that kind of blew me away about fuzzball when i first started looking at it and understanding it and greg greg mentioned this but i just want to kind of underscore it is um so you know slurm a traditional cluster you've got all these nodes sitting there spinning doing nothing you know or working on other jobs or whatever but they're all sitting there there's all these resources and then slurm says okay you you have a job you want to run on these resources let

me go look at them and see what they're doing and then run your job but that big pool of resources is always there the cool thing about fuzzball when it runs in the cloud especially is that there's no resources there at first fuzzball says oh let me build your cluster for you and based on what you need and then it basically builds you a custom cluster out you know and says here's your new cluster just for your job runs everything and then tears it all back down and that's super cool there's another there's another facet of this which we we haven't officially announced so this

is i guess a little bit of a soft leak that we're putting out there but there is a a lot of interest uh recently in composable hardware specif specifically around cxl you know pci switching and and whatnot um imagine if there was a way that a workflow can come in and say i don't need 10 gpus i need 24 gpus and you obviously don't have that in one system but using some sort of something like a composable switch we can now actually build that resource and set that resource up and just like in the cloud spin that up with that specific hardware configuration then run

the workflow on it and then again tear that back down and then put all of the resources back into the pool so it's quite um capable in terms of how you would do things like this and a lot of that is because uh we we did a couple things when we built an architect at fuzzball one of them was uh we took what was working in the hpc community and we took what was working in the enterprise cloud and hyperscale communities and we basically picked and choose what is the best for everything that we need to do across this entire you know across the entire

ecosystem let's pull those together and then let's further innovate on top of that to give us the best platform that we can possibly imagine for doing these sorts of computing tasks and and that's fuzzball is the result of that so that's our version of of what the best hpc kind of platform could look like uh and one of the motivating factors for this was you know we've been building hpc clusters the pretty much the same way for the last 30 years the beowulf design has been incredibly uh fantastic for us to create hpc systems and drive science drive research innovation and build these sorts of

capabilities and scale them up but you know fairly recently we started to see a lot of uh new types of innovations again coming out of enterprise cloud and hyperscale in such a way that they weren't really compatible with what how we've been doing hpc systems and we've also seen a much greater increase of diversity of workflows to the point where this long tail of science that we used to call it is now actually starting to become the the lion share of the workloads the the majority of these workloads and we're starting to see that traditional hpc application those tightly coupled highly parallelized mpi focused applications

um are now no longer the dominant and majority applications running on this system so all this together really kind of puts right now is the perfect time to start thinking about how do we modernize our hpc environment and how do we take a better look a new look a fresh look at cloud how do we how do we merge cloud and on-prem in a way that absolutely makes sense so you're running the right workloads where they need to be where they need to be run so all of that is kind of what went into fuzzball and what we were what we were thinking about when

we did this yeah the next question a great question too does fuzzball replace some of those capabilities for provisioning like werewolf or is it complementary so great question uh werewolf is if you're not familiar just for um if anybody's out there not familiar werewolf is a cluster management and provisioning toolkit it was created back in 2001 when i was at the department of energy and it is still actively maintained today it is still highly utilized today as this open source kind of cluster toolkit it does a few things somewhat differently than how you may be familiar with kind of managing systems at scale and it

works on the concept of imaging so instead of managing operating systems on n number of nodes you manage a image that goes out to all of those nodes and those nodes boot on in a dynamic sense so every time you turn on a node it automatically does a net boot and it's going being provisioned this image that you have defined for a single node or thousands of nodes and you can use the same image for thousands of nodes and that makes it very advantageous for doing clustering and managing clusters with something like werewolf now fuzzball sits on top of werewolf generally speaking just like slurm

um and or kubernetes if somebody wanted to provision a kubernetes cluster statelessly uh you can do that also on top of werewolf so werewolf is the tool responsible for managing the operating system on all of these these resources but what you run on top of that operating system is completely up to you so in this particular case fuzzball wouldn't replace anything uh in werewolf as a matter of fact it is 100 complimentary in the sense that werewolf would be kind of booting and running the operating system that's now running the fuzzball services um and it works very very good together from that perspective and you

can use werewolf both for the compute portion of the fuzzball cluster as well as the orchestration in the microservice portion of the of the fuzzball cluster uh however running kubernetes again on stateless is a little bit different because most people think of running kubernetes on stateful resources few things you have to do there and just in order to make kubernetes work properly and feel at home on a stateless system but it absolutely can be done and and that is one way that we do deploy fuzzball for customers today stuck on me again thanks craig there was a question mr ignite welcome back it's always good

to see you uh he's asking if there's gonna be an ide plug-in that has a menu bar for workflow management in real time i'm gonna open that one up to i think forrest you've been kind of thinking about this and if if it maybe maybe as well others but um forrest tell us how you were kind of using fuzzball from vs code and uh and maybe what are some of the plans and ideas around that yeah so just to kind of elaborate on vs code a little bit um i use that a lot for kind of working on you know generating my workflows that type

of stuff um it's got the integrated terminal that i can use to you know run fuzzball commands right from the ide all that so that basically kind of makes it the all-in-one easy text editing and working on um workflows in that way solution um some other things that we're kind of exploring around that um i think we're still kind of uh situating exactly what our plans are going to be to integrate like a fuzzball plug-in for that type of thing um on the workflow side itself something that we're exploring is the ability to like take the s code and run it as a workflow so

you can get a vs code terminal into a compute node somewhere so that's something that we're looking at and then we also have of course our gui that we're working on that's complementary to the cli that you've seen brian and i kind of demo that most of the functionality that you see there on the cli will be kind of available through so be able to do some of that workflow design execution management that type of stuff through our gui system as well so that's one kind of big solution there um but i know we're still situating what exactly like ide plug-ins themselves will look like

for fuzzball and i have heard talk from the engineers um about doing a plug-in as well and i think there is already for apptainer uh for others i think this is a great idea too so for us we ever put that demo into a tutorial document yeah we can definitely put that out there um i want to point out that it is an open source model so if you just look up dolly mini github you'll get the github for it um like i said my modifications there have been to move their um docker-based container over to apptaner and then to take their code out of

the notebook and kind of reconfigure it a little bit so it can save off the images instead of just displaying them into a notebook um so all credit uh for you know the model and stuff like that goes to i believe his name is boris daimon or something like that if you look at his github there um so i'll credit for you know that itself goes there but yeah we can definitely make available some of the tutorial materials for people to mess around with like i said i modified it a little bit um pretty simply to make it take in uh input on the command

line and then to make it just save off those images um so yeah that's something that we can i'm sure we can make available how long did it take you to get it to workforce when you started to play with it it took a little bit um most of the stuff i ended up doing was just kind of code level debugging as far as the actual fuzzball part of it itself once i had the image going and the script itself kind of all sorted out so there weren't any errors with that it was essentially just as simple as dropping it into a fuzzball workflow and

then figuring out what resources it took to run optimally so um yeah the workflow creation process was fairly simple like the script debugging that took a bit but um dropping that into a workflow and getting that workflow running was pretty simple of course the bigger the bigger lift would be getting a fuzzball cluster uh running and this is something that we are planning to put up uh demo instances up on the net so people can actually use this um timing for that still a little bit further out because that technically is kind of like a cloud resource a cloud platform um so that's that's a

little bit further out but uh yeah that will be something that um uh you're gonna see here you know pretty soon very cool thank you thomas united has another question so i think we talked about this a little bit last week in the round table but giving that university labs have a lot of high-end workstations can you federate over a campus multiple campuses to those high-end workstations can you put them into a single cluster during their idle times yes so um in terms of workstations is a little tricky uh most people are not building hpcs out of workstation hpc systems excuse me out of work

stations that is something that was actually kind of a big deal a little while ago as a matter of fact it was one of the reasons why a lot of people like the name werewolf because they can convert their their labs of workstations to a cluster at night under the full moon right so anyway um that was one of the reasons why you know a lot of people liked uh werewolf and it actually kind of went out um and it was got very popular because of that because werewolf is completely stateless so you can actually have locally installed windows systems um have them boot to

pixie first and then you know when you reboot those systems when you turn on your werewolf control service uh they will automatically all turn into cluster notes so we actually did there was a lot of people in computer labs doing that now since then that was again kind of early-ish 2000s mid-2000s since then most people have been focused on building kind of just uh production you know uh clusters that are dedicated for that sort of uh those sorts of work jobs and whatnot but in either case if you want to build up multiple hpc systems i've talked a little bit about you know fuzzball substrate

and fuzzball orchestrate those two is what makes up a single kind of cluster resource there's another level of fuzzball which sits above both of those called fuzzball federate which does exactly what you're kind of i think asking uh which gives us the ability to basically link together and join and unite any number of separate fuzzball clusters and these clusters don't always have to be just on-prem they can be in the cloud so you can have let's say fuzzball resource in different colleges within a campus you can have different campuses you can have them a fuzz ball you know multiple let's say a fuzzball cluster in

aws a fuzzball cluster in in azure but maybe in aws you actually want a fuzzball cluster in multiple availability zones to ensure that you can always hit gpus when you need gpus they're actually getting kind of hard to get uh in various clouds so this gives you the ability to automatically kind of choose where do you want to run and maybe that's maybe that's an azure maybe that's in gcp maybe that's in uh another campuses you know camp you know a server room or data center so there's a lot of optionality and a lot of functionality in terms of this architecture and what you're seeing

and what i think everything we demo today uh was just going straight to a single cluster at some point in the future we'll start really demonstrating and showing off federate but right now we're still focused on on orchestrate and just kind of demonstrating through orchestrate that's great thank you i think that was all the questions we had if you guys have any more questions post them now give that a minute um i want to thank brian thank you for putting that together that was great thank you for spending the time for us as always we really appreciate it dave how to say yeah i really

really love demos like this demos that you can kind of visualize and think about you know um like uh like what brian put together as far as modeling and you know you can really kind of understand what's going on on an intuitive level and also just like really fun demos as everybody you know who knows my history knows that i really like to make sure that there's some fun and some humor and demos to keep keep you engaged and also to you know they're just they're just fun to work on and so i just i love stuff like this absolutely thanks for the dad jokes

too there's more where that came from really quickly um i have the last two batches of results from i think the other requests that we had if we want to close out by looking through those absolutely let's do it once again this is a fuzzball workflow that i engineered to use this model to use fuzzball to orchestrate out actually running things with this model so i i just want to make it very clear that everything you're seeing here was generated from that workflow and that uh you know was all run on fuzzball at um an aws data center on resources that it orchestrated there so

very very cool let me just go ahead and share my screen and i'll show you our last few fuzzball results here did we also post all of these images somewhere go uh maybe from the youtube uh link so people can actually go and and download them and hang them on the walls and maybe make some amazing nfps from them exactly yeah okay so um this is a diver drinking coffee in a mountaintop cafe a screaming spider caught in a gopher web and three dogs playing with an elephant on mars so let's see how these turned out let's see here there it is okay so there's

a diver not exactly on a mountaintop but a diver at least a spider in a web so maybe it's a little too novel to have a uh whatever i just said it a screaming what is it it's a screaming spider caught in a gopher web so maybe that spider is screaming and we just can't hear it because this uh this ai doesn't generate sound um here's a bunch of elephants and dogs it looks like maybe playing some soccer or something like that on the martian surface um here's some divers looks like having a cuppa at the bottom of the ocean coffee there you go um

that's cool looks like maybe even have like a little coffee cake or something like that right next to it there's another spider in a web but noticeably distorted here's a little bit more of the the great i guess animal conference going on on mars a little bit of a mishap and our mishappen diver there another random of the spider and then more uh going on on mars and then i've got um the last one i i hope i got them all i think i got all the requests that came in i don't think i missed any um but this last one right here is a

bison on a keyboard a space shuttle backpack riding on an elephant and i didn't have a third one that i could see so i just put in a ciq branded super computer just to see what we got let's see didn't do the van gogh style painting of gregory kurtzer i didn't in this round um we can prob i've got a couple of those i think uh so i think that we could oh yeah that looks really good so i think that we could probably not do me i'm just saying let's see here so we've got a bison on a keyboard uh let's see here what

was this a space shuttle backpack riding on an elephant that one that's actually pretty appetizing that's looking good yeah it looks like we've got like a strap there and stuff um here's our ciq branded super computer this looks like it's the iq colors yeah it's got a little bit of a green hue to it this looks like the um the rack of servers that jonathan is in that photo that's everywhere in uh you know what i thought you're right we got it we got to bring that up with jonathan um here's it looks like here's a uh a bison sitting on a space bar it

looks like here's another elephant with a uh with a space shuttle backpack maybe backwards maybe it might be here's a green super computer yeah that that's definitely yeah i got a color getting a little bit more you know like your brand there's a lot of presentations nowadays i think i'm gonna use that one perfect here's a bison on a kind of free form maybe touch pad keyboard i think i might have to use that one too i'm not sure what i'm going to put it into a presentation somewhere it looks like we've got not only the uh the spaceship but the whole crew here [Music]

and uh one more super computer so not the super computers i guess they're maybe not um you know they're pretty uniform i guess when you look at millions of them at once or millions of pictures of them at once i guess um cool yup so that's all those um yeah fun times thanks everyone for tuning in and uh for having some fun with us on those so i think what what i just learned is every time i need an image for a presentation i think i'm gonna start bugging forest with a bunch of keywords of course i'm going to be in your best interest to

um yeah post that that uh workflow as a template somewhere live so i'm not bugging you every time i'm making a new presentation perfect um as a service in the marketplace so you can just go download your own really quickly um i shared my vs code screen i just want to point out just to link it back to what we said about there real quick um once again this is the workflow this ran from these are like the logs being printed out of it and all of this execution for example um like starting this workflow logging this workflow checking the status of it that type

of thing as i mentioned i did that all from within vs code and their terminal and stuff in there um so very useful but just kind of random note to show just once again how that you know how i've been running these commands here typically i pull up a terminal but just to show you guys where those are coming from so while you're in the middle of a webinar you did all that i appreciate the creativity and the responses that we got that made it very very fun so i also like dave's last comment dave ingram's last comment about uh rocky linux 9 codename bison

keyboard blue onyx is the code name for rocky linux9 but bison keyboard could be a good one for the next maybe earlier rock or color but you know exactly most ergonomic keyboard ever looking through the comments exactly cool i love it well thank you guys very much we appreciate you joining us this week uh look for these images to be posted somewhere we'll link to it from youtube so we'll get those out there and forest will work on getting a tutorial out on how to get this done for yourself so you can waste a lot of time spitting out random images so i really appreciate

it thanks for your time thanks guys for joining us you