Fuzzball (HPC 2.0): Workflow Demonstrations

let me remind everyone for those of you guys who are watching feel free to ask questions let's keep it live and interactive whatever you have just throw in the chat we'll we'll address that but i also want to point out that forest is doing this live so it's always fun and exciting to do live demos yes it is oh let's indeed here all right can everyone see my screen all right okay there we go i'm not sure if i can make that text bigger but i will roll with that so as you can see i'm here on my just vs code terminal um that's where

we're gonna take a look at the yaml definition that's gonna power this qmc pack one um i'm actually now that i think about it just one moment because this takes a couple of minutes and just to make sure we don't run out of time in the meantime we are actually going to go ahead and start this off to begin um so you can see i'm here in the command line we're just going to do a quick fuzzball workflow start and then we'll just provide a name for it so we're going to call it users effer ciq.co pack demo and then we just provide the path

to where that's at so that's in qd or where that workflow um definition that i'm about to go over is that so let's do this animal there we go okay perfect so we've just gone ahead and started that workflow i'm going to put its status up here so we can monitor it you know kind of switch back and forth you'll see we have a few different things going on here um i will explain everything so let's switch back over really quickly to that um vs code window that i was on and we'll kind of run through what everything is doing there so this qmc pack

workflow has a few different jobs in it and also has a few images a few volumes it's a little bit it kind of uses uh you know more than just one of those things like maybe we've shown in some of our past demos so just beginning up here at the top of this qmc pack workflow the first thing that we're doing is creating two different volumes we're creating one for some data to be ingressed into and then we're creating some data to be egressed out of this is more done just for demonstration purposes there's no specific need to create an entire directory just to do

an ingress or do an egress you could do it with one just as easily but just to kind of illustrate moving data around between volumes and that type of thing i chose to do this with two you can see that i'm basically just reaching out to an s3 bucket on aws in this case i'm going to pull down this file for qmc pack to use from that bucket and then we're going to place it into this first volume qmc pack data at this location this is an ephemeral volume meaning both or both of these are femoral volumes meaning that they only exist for the lifetime

of the workflow so this data will be ingressed the jobs will be able to operate on it in a persistent manner relative to other jobs and once the egress here is done the ephemeral volume will essentially be dropped on the floor and the data and it lost so anything not egressed will be uh yeah lost there so like i said we have two volumes we'll move to the jobs now we have one two three four jobs in this case the first one is just an untar this is basically just untarring this tar gz that we ingressed in this first step so we're just mounting that

data volume that we ingressed into with this mounts field here to the or to this location inside of the container as we've noted in fuzzball everything is a container all these workflow jobs that this computational workflow is going to be doing or run out of a specific container in this case because we have not really any other need for like specific tooling we're just using a small alpine container being pulled from the docker hub to provide tar in this case um for this initial entire job the next job up here is and you can see that we have you know eight cores requested 30gb bytes

of memory um and the command is basically just a tar command so this is basically just spinning up a standard cpu compute node to just do an ontar on we now have the run qmc pack step which is a little bit more complex you can see the image we're using is something that we're pulling from nvidia's ngc um so this is something that they've put together to run this on um this is something they've put together to run especially on gpus so you can see that uh in this case as far as gpus go we're requesting one of them we're also requesting four cores and

60 gb bytes of memory um this is going to match more or less the definition that i've set up that'll pull us down a node off this public cloud provider that'll have one nvidia b100 gpu in it um and because down here with this mpi field we're requesting four nodes with an open mpi implementation of our mpi wrapper uh you can see that we'll actually end up getting four of those compute nodes so we'll have four v100s in total with our mpi wrapper managing uh parallelizing that over all those nodes um so this will run this step takes 15 to 20 minutes or so to

run through all of that we have an environmental variable here that we're setting just to make sure that we're using the right process management library relative to openmpi um and so yeah and then you can see the uh the command we're actually running here is basically just qmc pack based on some input data you'll notice that we're mounting the qmc pack directory um to the ho to this host pwd directory inside of the container um and we're also saying that to be our current working directory um we've done that because we untarred in this step all that data into that host pwd directory um and

so in this case yeah we just need to specify our working directory and then what file in there we're going to be feeding into qmc pack the next couple jobs are pretty simple one is just an ls so we can have some visibility into what data was produced from this run once it's done there's nothing too special there it's once again just alpine just doing an ls latrh there and then it requires you can see what these requires things that we're building up a directly cyclic graph of execution here and we do that because obviously the data has to be untarred the simulation has to

be run and so we have a path of execution that we're trying to follow here and so this ls command or this ls job here um yeah it's basically just going to show us what's in that host pwd directory and then our last job is tar outputs the volume egress directory and that um oh i think i must have copied that over and forgot to switch that there shouldn't be a problem there it should be um just fine but that is not the container i meant to use there um it should be fine otherwise um yeah it should be fine uh it's just doing a

tar command so we could do something just as minimal as help in there and it would be fine um but as we'll get to notice we've got now three different containers being orchestrated and pulled from somewhere and managed for this workflow so that's kind of a neat aside there uh so you can see this uh tar outputs job like i said it's just gonna use just something very minimal to tar up what's in host pwd and place that into egress and then once this job is done the end or once this job is done the workflow itself will complete we'll go back down here to

the bottom at this egress and this file which we tarred up in that step um just to point out oops i haven't accidentally pasted a bunch of stuff um you can see that this is the directory that we've created that tarball there and then you can see that we're going to egress that same directory up to this s3 bucket right there i'm just tab into the other screen and see how this is working looks like it is in fact let me switch over real quick so i'll show you guys this running and then we might get into a couple of the other ones because like

i said this will take about 20 minutes or so so just moving back over to this really quickly you can see that we've been watching this status output here and just a map what we've just seen in that workflow to what we're seeing here in our status output we have all these lines here along the left each one of these represents a different part of this workflow you see that we have workflow up here at the top that's just kind of delineating the workflow as a whole um it's kind of representing the time that overall that it was started and stopped um and then or

started and finished and then uh its name basically we have the two volumes that we set up right here those finish basically instantly um those are just setting up that qmc pack data and um egress directory that we're going to be using to store some data for these jobs um so those are those two volumes being set up we then have the two file transfers one of them says finished one of them says pending you'll notice this one shows something moving in from s3 you'll notice this one shows something moving out of the cluster and on to s3 so this one is finished because this

represents our data ingress that we showed there this one is pending because it represents the data egress that won't happen until all these jobs are finished we have these images here that have been downloaded as well each one of these image lines is a container that has been downloaded for this workflow to use as i mentioned we're using alpen to provide just kind of basic guitar stuff um we're using qmc pack for qmc pack and then apparently we're also using quantum espresso for tar as well but as mentioned ignore that just a typo um so you can see that we then have the four jobs

here untar run qm cpac ls tar outputs um you can see untar is finished and it got done about three minutes or so ago so right now tar outputs are not tar outputs sorry right now run qmc pack will be spooling um and i'm just gonna of course i may be jumping ahead a little bit but the egress on that can be to anything it doesn't matter it doesn't have to be s3 it can be to whatever story you want it to go to right um i mostly use it in the context of s3 if we had set up as a persistent volume we'd be

able to perhaps move it out onto a physical architecture but at the moment i mostly use that to push an s3 you'll notice that in that workflow i have there i have some very specific parameters that i've set let's see or where's it at you'll notice here that i've set a few different parameters s3 axis key id s3 axis key and s3 region of course sorry oh my heavens sorry bro let's see here is that better there we go okay cool so you'll notice that down here in the secret section of this um this egress and for that matter the ingress we have s3 axis

key id s3 axis let's see s3 axis key id s3 axis key and s3 region these allow you to specify uh basically just standard s3 api compliant credentials that allow you to reach out to any s3 compliant api um object storage out there um so some places um the s3 api isn't just provided by amazon s3 it's also provided by some other storage platforms um so you can utilize the same you know commands and such that you would use uh if you you can basically just treat it as if it was an aws endpoint um we also have it's not shown here because i am

hitting aws s3 here um but we do also have the ability to set an s3 endpoint that allows you to specify that custom endpoint if some other cloud storage provider happens to have a different endpoint that they need you to use thank you cool um let's see here i want to go to this oh yeah so um what i want to point out here is that as this is running oh it started okay cool so we should be able to see this different now okay so um this is kind of cool you can see that right here this job was pending and if i go

to fuzzball provision resource list which gives us the list of all of the compute resources that are currently running on the cluster you'll see that we have four of these creating now i won't make the more verbose output come out of this just um because i'm not sure if i want to have that shown here but if we were to look at the more verbose output from this we'd be able to see that these are running um p3 dot uh 2x larges for each one of these and you'll notice that when it was pending these were creating when it was started these are running and

so that shows that this job is now running these are the resources that it's running on um it i don't think it uses the gpu for every bit of the computation so if this doesn't um show an incredible amount of usage that'll be why but if we were to do something like fuzzball workflow exec um real quick if you don't if you don't mind me jumping in real quick um i i have some a slide i can share real quick that'll kind of graphically describe what it is that we're seeing with regards to the data ingress and ecress and whatnot so here i'm going to

share this slide real quick yeah so what this is demonstrating is on the left side you'll see the user's workflow coming into fuzzball orchestrate that workflow is what forest was describing it has within that it has the context of ingress it has the context of the job pipeline and then the context of the egress that comes into fuzzball orchestrate now forest is running this up in the cloud right now so when that workflow comes in it's going to start doing a bunch of things kind of simultaneously you can see some of these things depicted on the right side so for example you can see the

data lake the data lake in this case could be you know any form of an s3 data store could be http endpoint it could be anything we're going to ingress all of the bits of data we need for this job you're going to see the ingress that was under the volumes in that you're going to see the containers coming in uh from that as well and then we're going to set up the volumes underneath that now in parallel to that because we're running this up in the cloud fuzzball is going to be provisioning resources cloud resources instances to actually run that job pipeline and it

will satisfy all of the hardware uh requirements of that workflow in order to uh to basically run that job so if you're running an mpi job if you're running task arrays uh if you need gpus it will actually uh provision that resource out and compose that resource as well if you need and if you have composable hardware once that is done it'll start running that job pipeline on top of this volume and then at the very end we're going to do that egress so what you saw forest sharing was a lot of these pieces being depicted by the uh the monitor basically by what is

actually uh running it and you can follow these different steps so i'm gonna actually go back let's go back to forest screen now and he can actually show exactly what that what that is and what stage it's in oh you're still muted thank you there you go um so as i as i kind of pointed out here we have all of these gpu nodes running if i go ahead and do a fuzzball workflow exec and then i provide the name of the workflow the job that the workflow is running and then uh arbitrary command if i were to specify tty here i could do slash

bin bash and get a shell into this i'm just running nvidia smi though in this case um you'll see we actually have the output from nvidia smi coming from one of these compute nodes uh you'll see that at the moment we don't have any utilization like i said it's not going to use the gpu for every part of it so we don't expect that but we do have processes running there um and then we do have some amount of memory that is being used and if we go down here we can see we have you know four percent so not much but starting to use

the gpu and that memory changing um and so you can kind of see there that we have some gpus that we're running on here i'll move back to the status of the workflow and we'll see where that's at uh you can see it's still on this run qmc pack step let me just follow the logs for a second and see what this is outputting there we go um we can see that we get this pending message when there's nothing being printed to the log at the moment but we can see so far what's been run as a part of this simulation so basically just a

lot of info here um i'm trying to find the line that references gpus yeah here we go um total number of mpi groups one 16 ranks ranks per node four accelerators per node one so we see that we have one gpu per node there um there's another i think a bit that says something about kyoto but i'm not seeing it right off the top of my head but anyway you can see that we're finding accelerators and stuff there um we just get a memory usage report from it as it runs um then yeah we'll like i said get this pending message might wanna stifle that

a little bit make a note um but yeah so we can see that uh this is is running um this will take another few minutes to run so if we wanted to we could actually start something else up right now and kind of go through that yeah go for it sounds great already i'm going to hop out of this and i'm going to hop over to my desk example really quickly so so like i said this is a little bit of kind of a basic example um this just basically shows das running in general within this platform and kind of how more the purpose here

is to show how a task array works and how it can allow you to process many many input files at once in an embarrassingly parallel manner so we have this workflow here um the sorry i'm looking at how big it is on the actual screen should be good um so we have this workflow here we're going to once again reach out to my s3 bucket and pull down moredates.targz moredates.targz has 100 input file or has 100 input folders in it each one of them kind of looking like it's got a random date basically attached to it and inside of each one of those is a file prices.txt that has 20 000 random numbers in it between 50 and 100.

um so we'll bring that down and that'll get pulled into this data volume that we have here v1 um this is an ephemeral volume so once again this will last for the lifetime of the workflow but then be destroyed once the workflow is done so we'll pull down this bit of data from that s3 bucket we'll basically just do a similar untar job to what we did before um i'm using just kind of notably here a private docker hub registry um you know by one that i got uh i'm using a private docker registry here to pull this down um so just showing you how

uh you know you can pull down from directly a private repo on the docker hub itself um you can see i have an access token there configured within our secret system and stuff like that to be able to get in so i'm pulling down this desk compute image that i made this basically just includes das and some other components once that's pulled down and is untarred we're going to pre-process the data so we'll use this little script right here called mungfolders.py and there's a few different ways you can do this this is kind of a pythonic way when we have a task array the uh

interface that we get into being able to assign different parts of that task array to or assign the resources that task array is bringing up to different subtasks of the overall computational job we're trying to run is through this fb task id variable here um so we essentially get depending upon what the task number here is in this task right and i'll explain this a little bit more i'm kind of getting a little ahead of myself explaining because i have to explain this script right here uh but this fb task id is set with the number that this um the task number that is relative

to what's set here in this task array um so essentially what munch folders does is it goes through the 100 input files that we have and appends to the front of each one just a running count from 1 to 99 so that that file right so that the task array can then have a way to pull down and link a specific input file to a specific computational resource that's brought up um so this is basically just going to take that dates file lunge a little bit add those uh numbers to the front of it and then once that's done and you can see we're using

very minimal resources for that that's a really really quick task um we then get to the last one which is run dask and like i mentioned this is a task array i don't think that we've shown one of these before so i'll elaborate a little uh a task array enables what's called the embarrassingly parallel model of high performance computing um we've gone over mpi quite a bit which is actual quarter core and node node communication happening uh in a high performance computing simulator parallelized simulation embarrassingly parallel is essentially the case where in contrast to mpi where incredible effort is required to write an mpi

application with the embarrassingly parallel model little to no effort is necessary in order to separate the input into discrete computational tasks that can be run independent of any other task so in this case we have 100 input files we're gonna start it might i might be having off by one error here but i'm gonna ignore that and say it's a hundred um so i have a hundred input files here um i will start numbering them at task one i'll stop and they are numbering them at task 99 and i'll run 10 of them at once so what we're going to see this do is

bring up 10 nodes at once to run this and then this is going to basically do this processing right here in this das compute file you can see that we are essentially opening that prices.txt file i'm reading everything in from it and then feeding everything that we had in it is this one moment there we go sorry i i've confused myself once or twice with that um this is the one that yeah sorry this is the one that we're looking at the other one there is uh um the other one there is kind of the working copy this is the one that we're actually running

in that um which will make like this make a little bit more sense here because that's uh covers a corner case in it anyway um so like i said this is uh essentially going to take those 20 000 input numbers from those 100 input files um do essentially raising each one of them in those lists to the power of two um it's gonna sum every one of those twenty thousand now raises the power of two numbers together um and then divide that average by twenty thousand um or divide that number by twenty thousand so we essentially get um you know just average plus num 20 000.

yeah so we get the average more or less of what those 20 000 numbers is in there um sorry can't math on the webinar for some reason um but yeah so that's what this will do uh moving back to this task array over here with those one hard input files yeah it'll map to that we can see that python3 runs and then in this file we take in the first member of the arguments array and so that's why we pass it to it right there so you can see how we link the task array into some actual code in this case i'm going to go

ahead and run this and in the meantime we'll return also to qmc pack because it looks like it is doing something so head back over here real quick so you can see qmp qmc pack is still working uh the run qmc pack step is finished the ls step is finished and the outputs are actively being tarred um so once that's done this egress will continue and in the meantime let's just open this up real quick so yeah we can see here qmc pack execution completed successfully from the run qmc pack job um and you can see here tar outputs we're gradually getting an output of

what's being tarred in that directory um and that we'll have on s3 in just a little bit uh let's go ahead and start desk before we run out of time uh let's see here that's desk demo number task wait where's that oh sorry i was completing the path and stuff there there we go number desk slash numbers slash yeah number task and you can see some other stuff in here that's essentially just um you know testing other just scripts stuff like that just kind of the mess from getting this working that i forgot to clean up we'll go ahead and watch this status for a

moment because this will be kind of fun to watch unroll so um this has this workflow started we've got this s3 ingress happening right now so we're bringing in that more dates.tar.tar.gz file from s3 we've already got the image cached basically because i've been running this and then here in just a moment we'll start to see these jobs kick off we'll watch this here for just a moment um i might keep an eye on that let's see when the um when the task array kicks up it'll essentially unroll 10 tasks out of it and this output will become much more kind of complex and i

want to make sure that we capture that here so once we see that uh we'll kind of let this run for a second and i think we are going to pivot really quickly to um quantum espresso because that takes about 15 minutes to run and i see that it's 11 30. i'm i'm simple-minded i'd love to see the output actually changing as it's going through the different tasks and watching it you know something started something finishes something you know is happening over here now and you can actually see all of the dependencies of the job graph being resolved in real time uh and implemented in

real time as you watch this output it's super cool yeah exactly so as you've seen here this um this initial or this job up here run das has started and it's unrolled 10 tasks out of it this has started this kind of top level job for this task array because at the moment it's spooling 10 nodes to run these on um so these are all pending we'll kind of start to see those come up here uh within the next couple of minutes or so i'll go to a fuzzball provision resource list because we can maybe is it a whole bunch of them going oh that

is super cool is it possible to kind of is it possible to pre-spool or pre-provision some of these um resources if you know that you're going to have some resources uh that you're going to want to run so the the cluster is ready for it yeah we just do um fuzzball provision resource create and then um i think we would just do something like uh user slash fbert ciq.com default cpu definition is that what it's called cpu node yeah there we go okay yeah so that uh spooled up a resource um so that if we were to kind of look closer at this i mean

this is a bit of a mess at the moment i think that there's a few different workflows conflating here um so that's why we see some are running uh some are creating i suspect most of the ones creating are the 10 going for this um which actually let's return to it before i forget about that yeah okay so those are still spooling um so yeah we're just kind of waiting a couple minutes for those to come up sorry for throwing you off on that oh no it's fine that's totally fine it's um it's a good point uh that yeah you can pre-provision those i'm

not sure uh how sophisticated at the moment the linking of those created resources is um uh like whether or not it'll still want to provision some i'd have to kind of investigate a little more but oh there we go okay cool we see this starting and if this was if this was a physical cluster that was on prem and all the resources and nodes are already there there's no spooling up all this happens instantly right yep exactly because the resources are already there it's not having to do any negotiation with the public cloud provider you know anything like that to bring these up with whatever

um you know playbooks whatever like that it's yeah so we yeah so we see these starting here um these are uh this will start to like finish these tasks yeah there we go and we'll start to see more starting um that concurrency in case i didn't make it clear that concurrency one means we'll have 10 nodes running at once so we'll have 10 desk tasks maximum running at one time and so we can see we have at the moment one two three four five six seven as we're waiting for a couple more nodes to spool i think and it looks like it's reusing those nodes

so so nodes that have already spooled up and been provisioned it's not shutting those down and then restarting them so we're actually seeing them come back up much faster now in this case yeah i want to point out that um there are nuances to that depending upon where you're looking but in the case of a task array yes it will just continually land those tasks on the resources it's already done which you can see the usefulness of if you know the instance you're trying to use is you know thirty dollars an instance or whatever just throwing a number out there you wouldn't wanna have to

spin up thirty or a hundred thirty dollar instances you just wanna do ten of them and then have it run so we can see that this is going to go through um and process those files pretty handily we can go ahead and take a look at the log output as this goes like i mentioned this is just going to print the average of those numbers and because they were all generated from about the same random space we should see that the numbers being printed out will all be within some small amount of each other so let's go ahead and do this and yeah so what

we see here is all of these tasks gradually printing out their outputs as they go we can see the task name right there we sometimes get these heartbeat ones failed but if you uh you know look we've got one on task 22 but then task 22 completes successfully right there so we're not worried about those um you can see that we're essentially just printing out what the name of the file that we derived this data here from was um so in this case this is meant to represent january 30th 1999 but in this case we assigned this to task four so we munged this little

four dash here at the front um just to make sure that we can map that to a task so if we come down to the bottom we see that this is still running and still moving i'm gonna give just one moment i'm gonna check the resource counts to see or a little closer and see if everything is cleaned up uh let's see here oops oops um one two three four five six seven eight nine ten eleven twelve okay um yeah i think we're good i will go ahead and again while forest is bringing that up if you have any questions please start entering those into

the chat so we can answer those as we go absolutely um see i'm gonna uh while that finishes up i think it's fairly apparent what's going on there i'm going to move over really quickly and show the quantum espresso workflow because that's kind of cool and we still need to show some of the aws stuff and we're starting to run up on time so this one right here uh q espresso data volume kmc pack egress directory did i accidentally paste it on this one as well qe outputs au surf oh no that's an egress sorry i was thinking over here of egress directory so i

was reading egress as its own volume and i was like why do i have two volumes set up on this one um no no no no this is one volume okay so in this q espresso workflow you can see that we have one volume we're creating here q espresso data volume we're going to bring down once again some data from that s3 bucket just these qe inputs um the inputs for qmc pack and quantum espresso in this case are basically just available out there i believe that there are the examples that are on the nvidia ngc if you visit the container pages for these pieces

of software there so you can see we're going to do an ingress so we'll bring that bit of data in here and then once that data is done um we'll egress that out to this s3 bucket once again you can see we have one two three jobs in this case um the first one is an untar not like anything we haven't seen before um so that's pretty simple um just gonna untar that data the second job here is going to do the same thing resource wise that qmc pack just did it's going to bring up four nodes with the open mpi implementation of the mpi

wrapper looking for four cores 60 gb bytes in memory and one gpu on each one of those nodes so in total we'll have four v 100s coming up you can see that we are making our cwd the inputs file that we untard in the prior step and then we're just running just the qmc pack executable here over that input data um there's two files uh that get unpacked from that um this tarball right here whoops let's do that again uh and so yeah and then you can see the image we're pulling down from the ngc um there's not much else to see on that one

i don't think and then up here at the top we once again tarp the outputs and then we place those into that data directory that we have so that we can egress those out to s3 afterward um we'll also notice we have requires here run q espresso antar so we're creating a basic directed acyclic graph there for this to follow and we'll move over really quickly to the command line once again and then we'll go ahead and run that one so we'll go ahead and do sorry like i said i um look to some things i don't quite want on camera uh and so we'll

go ahead and find where i put there we go so you're typing that we do have one question for you forest or michael what's up what does the workflow do if a node fails during the run oh so if a node fails during the run it depends upon what the behavior you defined in your workflow is and we have an ability to set policy on a workflow job and so you can set two types of policies a timeout policy and a retry policy a timeout sets how long the i need to make sure i start this there we go um a timeout policy defines how

long the workflow job should be allowed to run for before it gets terminated due to timeout and then retry will tell you how many times a job should retry if it fails for any reason and so that's just as simple as adding um that policy retry to your workflow so pretty simple just add a line and it'll it'll try i'm um i'm looking for a workflow that has it on it but i can't cut you out on my top of my sorry um cool really quickly just to link what we're doing here to what that definition we just looked at is um you can see

that uh once again we have the workflow up here start and finish time of the workflow the one data volume that we created in that workflow definition we have the file ingress that we did so that's already finished we have the file egress that we're going to do which is pending because these jobs haven't finished we have the untar we have one image that we're using just quantum espresso in this case and then we have um antar here uh like i said finished and we're just waiting on quantum espresso now so this will take similar to qmc pack a few minutes to spool up those

nodes um but speaking of which in the meantime let's hop back over to qmc pack and see what it's done in the meantime if i can find where that status is there we go ah it's finished including our egress and everything um and so if i just hop to a little aws s3 ls and do let's see our s3 convert storage we can see 1128 428 so just about 10 minutes ago or so qmc pack results full tar.gz and aws s3cp um this year put that there this to the current directory right yeah cool there we go all right we'll give this a moment

to download and then i'm just gonna um tar xf that and just to show the results that we have in there i don't have any um you know fun graphs or anything that i can bring up and show with it but we should be able to see fairly easily what the output files here are and how we've just recovered those you're just showing off what a what a reasonable internet connection looks like amazing yeah i definitely like my uh i like my one gigabit down they tell me that the local fiber place can get me one gigabit up which is very useful for container pushes so i'm exploring that as well i'm not going to tell you what mine is where i'm currently at sometimes i get 20.

screaming i know oh let's see here another question so how do numa architectures interact with a workflow and are there some examples yeah um give me just one second to uh what's what screen do i have oh i've got this one okay good um give me just one second to get into these what's it called uh just one second what is that on up host pwd okay there we go okay um because that tart up the host pwd directory that's what it unfortunately gets decompressed as but if we go in here you can see that we have all the files that we just wanted to

recover from it our initial uh data files that we brought in still targey z'd um so we may want to avoid doing that if those are significant because we wouldn't want to just be copying those over but anyway we have all of the kind of information that we provided to this we have i believe this is the actual input file itself and then we have all these scalar.dat files that represent the outputs i don't have a little qmca demo spooled up to actually pull like i said some graphs or something out of this um but you can see those are the results and if we

go do something like sorry i'm gonna find there we go and if we go to this and we print out all the logs um once again as i showed earlier we can scroll back through this um we've got you know device memory allocated via the q2 allocator so talking about you know allocating gpu memory um we've got uh the ls here that we ran so that's just showing us kind of what's in that data directory that we ended up tarring up and bringing down that i'm trying to see it might be quantum espresso's output that has a very obvious point where it says using um

q2 accelerated mpi or something like that oh here we go running on an nvidia gpu via qd acceleration that's useful okay um so yeah so that's qmc pack um you can see once again execution completed successfully we have our timing and information like that here at the output um really quickly while quantum espresso is probably still working we'll um hop back over to desk just to finish that out and show what that looks like um so we can see in this output we have the single run desk and then 100 or well 99 tasks coming off of that um all of them starting and finishing

you know 128 well that's the whole job itself i guess we expect that to take a few minutes all of these finishing fairly quickly you know 11 31 15 11 31 48 and you can see how as we progressed through all of these input files over the course of seven minutes or so we processed all 100 of them and so if i go do a log dash f we can extract all that data out of it and once again like i noted uh we get this odd error sometimes but we ultimately still get results and as i mentioned you'll notice these are all somewhere in the vicinity of 5800 and about 40.

um like i mentioned all the random numbers here were generated from the same between 50 and 100 so we expect to see about the same thing on each one of those which is good that we do so that's desk and qmc pack let's move back to quantum espresso and see how it's doing we can see it's running right now why don't we just take a look at that while it goes there we go once again we still get that pending message we'll have to note that um that should be stifled when there's uh so that's not just constantly printing um but we can see that

we're starting to get some output here i imagine we'll get some more here in a second but if we look through this there like i mentioned gpu acceleration is active gpu aware mpi enabled um and then we can go and we can see 16 mpi processes four threads per mpi process mpi processes distributed on four nodes um then we can just get information about the simulation and we can see that this is printing except for when it's pending um i think this does 20 iterations and it's on iteration one uh so that will take a little bit to run so we'll come back over to

the status and what else do i want to do take a look at this so yeah you can see that at the moment we only have five nodes running compared to the very large amount that we had earlier so all of those nodes that we spilled up for das all the nodes that we spooled up for qmc pack um all of those have been spun down and they're obviously no longer visible here in our resource list we have five we expect four for this run quantum espresso one but it's probably just uh something taking a little bit to spin down so i'm sure it'll disappear

here in a moment um but yeah so that's quantum espresso like i said this will take probably about 12 minutes or so to run and so once that's done um really quickly even though we're at time i think we should hop over and take a look at that s3 bucket and show how those results have been uploaded there as well and real quick maybe i missed it did we address the pneuma question well no we didn't thank you for the reminder you got it um the numa question uh let's hop back over really quickly uh i'm gonna bring up my um text editor because that's

the easiest spot so uh just to pick a random one of these we'll use the run qmc pack job because it's the one that we're most likely to actually have you know some use for numa node on you'll notice that when we set the cpu cores that we want here and whether or not we want hyper threads to be present or enabled you'll notice that we can set a pneuma affinity here this affinity can do there's two other possible affinities i can never remember the name of the one that's for the whole node but i believe they are numa socket and core at the moment

and so core is the most course let me back up the affinity that we're looking at here is where we're going to choose the cpu cores on this node from so obviously if we have 32 cores on a node um we depending upon what's going on with the server we may want to land all of the cores that we're using within a certain subset of the total amount of cores on the node available this is typically done when we have what's called a non-uniform memory architecture underlying in the server like the compute node and numa is essentially where you know say you have a

32 core machine you have you know 200 gigabytes of memory on that node um if you have four pneuma nodes you would probably have something like eight cores to a pneuma node each one of them with 50 gigabytes of that memory that they can access fundamentally faster than any other cores in that system can and so a pneuma architecture being pneumo where in your simulations is a really easy way to instantly add efficiency because you know say we only need eight cores of that 32 it becomes highly advantageous to land all of those cores on the same pneuma node so they can all have a

space in memory that they can access faster than any other core um so that's the pneuma affinity in this case with four cores um i don't think uh it's gonna be too important we don't we're not um under subscribing this compute node we're using all the cores available on it so by default um uh kind of our affinity isn't really too important there but you can see how if we were doing um like a subset of a node or something along that type of thing numa would be immediately or that affinity there would be immediately a way to add efficiency to your simulation with just

a couple types on the keyboard as i noted there are two other possible affinities socket and core socket basically just minimizes the cores to the number of sockets so you know if you've got a compute node that has two 16 core cpus in it and you specified that you wanted 16 cores and with the socket affinity it would land all of those cores on one socket um so that once again if you're looking for socket-based affinity that's how you do that um and then core basically is the like i said most course and it just um basically just picks from anywhere on the node so

that's how we do numa awareness um just another kind of interesting similar option is by core and by core allows us to specify whether or not we want the amount of memory that we tell it to go and look for to be for each core so in this case we're saying false because we don't want to need 240 gb bytes in total 60 times 4.

so we say by core false and so we're only looking for 60 for all four cores in total i'm going to switch this and we're going to take a look back on the command line and see what's what still running like i mentioned this will take a few minutes but let's kind of see what this is doing right now but actually looks like it's finishing cool um yeah so these are these are just like random ending warnings that i presume happen because this is literally written in fortran 90. um so i'm sure we're going to get some interesting input out of anything written in fortran 90.

um as i noted earlier when we look through these results that we have here we can see that it is very obvious that we're running on four nodes it's very obvious that we're using some type of gpu acceleration we can see we have 56 atoms here of gold that we are simulating in this case uh if i read all of this correctly you'll see we get kind of some little bit of information here this is the amount of ram that we'll need so one of the reasons why we're using um well this is a little scaled down from the example but it doesn't matter we

can see that we did 20 iterations so one two three four five six blah blah blah um we did 13 actually but we did we did 13 actually sorry my test runs did 20 maybe i changed something um but you can see we get yeah all the numbers that we expect all the output we have a note here that says we have a output data file being written that we're going to push to s3 in just a second um you'll see we have job done uh this run was terminated at this time and we get cpu and wall clock there and then we also get

gpu time so we can see uh i guess we do only get cp on that as we see we get wall clock and gpu time printed out there and then we're just getting the kind of tarring up of the outputs here at the moment before those will be uploaded so once tar output's finished we'll see that data egress start and then just like we did before i'll hop back over to um kind of just aws s3 ls and uh i will show that there and also while i'm at it let me bring up my aws console so that i can just show the bucket view

there as well it's three okay cool so while we're waiting for that here's my um oops i've lost the stream yard in my tops there we go so uh let's see there we go there's nothing i don't want to reveal here yeah we should be good okay there we go can everyone see that okay this is just the aws bucket interface you can see that i'm here on my burt storage bucket that i've been referring to this whole time you can see we have more dates.tar.gz which is the quantitative finance data that we pulled in you can see we have s32 example.tar gz which is

what we pulled in for qmc pack and you see we have qeinputs.tar.gz which is what we pulled in for quantum espresso to use as an input and then we have qmc pack results uh fulltar.gz here which is the um uh which is the qmc pack results that we just uploaded it looks like the quantum espresso workflow is just finished in the background so actually if i refresh this boom there we go qe outputs ausurf.targz which is exactly what we expected there and then if i go back over to the command line you'll see that this workflow is finished the data egress finished there while we

were looking at that bucket view and then much like i did before um i will do uh let's see here i am just going to make a quick little temporary directory to move this into just to make sure i don't conflict with anything it's just showing off your internet speed again yeah here we go let's see grab this we don't have enough time on a webinar for me to download stuff like that i'm gonna pull two gigabytes here in just a moment so in just a moment we'll have that coming on out that might take a little bit to untar but um we'll be

able to look at those results here in just a moment look at that let's do this and then uh we'll just wait a second for that to run so that's about what i had as far as demos um yeah that's about what i am excellent well thank you very much for having me putting a lot of time getting this together for us it's very cool to get to show off so you can see um here now we can go into this qe inputs file like i said that thing is a little rough here i'm not really uh concerned about not you know preserving what the

volume looked like when it was attached to it but you can see if we go in here we continue to go down these directories we have our output files there so so that is uh everything i had um yeah fantastic that's everything on so well thank you very much forest for that uh michael i don't know if you were i was lost michael there he's back michael greg i don't know if you guys have anything you want to add before we close i just wanted to i just started interrupting i just wanted to say that was super awesome for us you put a lot of

work into that we can tell and uh it really demonstrates the use cases of what it is that that fuzzball and and more generally an hpc 2.0 cloud native you know federated computing platform how it can actually work how it can actually help users help researchers and and whatnot i think you did a great job uh demonstrating that so thank you so much thank you and if i can just one more time share my screen i am i've decided to gonna show this really quickly um i can't show this live uh just because there will more than likely be issues on the public cloud providers

side with having enough of these available um a problem with their capacity not anything to do with our system but if i run this right here this is something that i ran yesterday we've featured lamps examples on here we had kind of our most advanced one yet yesterday and so we can see and i'll pull the workflow off of this in just a moment as well but we can see that this run i did yesterday um like i mentioned they don't always have capacity in these instances so it was a bit of crap shoot getting it running so i only have results here um but

we can see that we have this four by two by four mpi processor grid and we are using up to four gpus per node this is running on uh this is also running an mpi as we can kind of see due to the timing loss and communication here um and then also through the workflow and so just wanted to show we've done a lot of single gpu node examples thus far but just wanted to make sure i point out we have no problem running across nodes that have multiple gpus as well on them and so these results you're looking at here were generated from a

run of lamps that was done over two nodes at once with four gpus on each node and so in total the results that we see right here are generated from a run of lamps it's running over eight v 100 gpus at once across two nodes on this public cloud provider so like i said i can't easily demo that live but i did want to share those results because it did work at least once they were able to issue me one of those um so it didn't work so this is multi gpu we've seen this now several times where we're trying to provision resources up in

the cloud and i know that the idea of the cloud is you have infinite number of resources whenever you need them available yeah we're actually not seeing that um you know depending on what availability region we're on what cloud we're on because we're testing on all the major clouds at this point uh we're getting kind of mixed feedback regarding where can we actually spin up big resources big jobs and uh yeah we've hit the top of capacity now a number of times up in you know the cloud and whatnot and this is one of the reasons why it's so incredibly important to have a multi-availability

zone multi-cloud solution such that you can find where those resources exist and bring your job to those resources because if you're just running in one you're probably going to hit hit scalability issues because the cloud is not actually infinite especially when we're doing really kind of hardcore compute focus jobs so being able to distribute that as wide as you can possibly uh distribute that find the best prices the best resources that you need match that up and automatically have that job run in the right location it's kind of a big deal absolutely awesome well yes thank you everyone for joining and for uh tuning in

to the webinar it like i said or like uh you know everyone kind of touched on like i said it was fun to put all this together and to get to show it off um so i hope it kind of pushed the envelope for what we've shown fuzzball can do so very cool that's great thank you very much and guys we always ask that you like and subscribe so you can stay in touch with us and we can keep putting out things for you so watch and ask us questions on we appreciate it and we will see you again next week thanks guys bye everyone