Fuzzball HPC-2.0

ecosystem were doing and said how do we make use of that there's value there we could be leveraging that and how do we use that so that was really just the first step in terms of modernization and looking at hpc from another lens so what is it what is fuzzball um it is a cloud native a cloud hybrid uh federated meta orchestration platform for uh compute focused workflows and data that's what it is so that's a lot to take in so i guess can we break that down a little bit in terms of like why is that important and what does that really mean again

everybody steps back i know i know oh it's just grinning eagerly why is it important well there's a number of reasons first off um as i kind of alluded to traditional hpc has been really looking into what are hyperscalers doing what are cloud clouds doing what do people need to do in terms of computing in the cloud and what is enterprise doing and how do we make good use of these capabilities in high performance computing but it definitely doesn't end there um going back now maybe about three or four years ago i started getting you know right when i was really you know kind of

right in the middle of singularity and building up containers i started getting a lot of calls from enterprises who all of a sudden started to see that they are going to be hpc like consumers uh they even said we need hpc and how to actually bring hpc into the enterprise is a little tricky and i heard some really funny quotes about it one of which was we're trying to get our system administrators off of ssh and here you're telling us that all of our users and and access into this system requires ssh um no it's completely the opposite direction from where we're trying to go

with our infrastructure so no we need a cloud native style system an api based system to do this we need a system that's going to fit in with our existing infrastructure and whatnot so it was really it's the hpc legacy architecture that we've been leveraging within within the the computing industry pervasively again has been fantastic but due to its kind of legacy um way of operating and and whatnot it's it just doesn't fit with more modern ways of looking at how to do compute and that was really what started us going in this direction uh i've been talking now so much and just kind of

riffing i don't even remember what the question was zayn so it's really back to i mean you said a lot of words in the beginning of what fuzzball is from a high level i mean it obviously integrates a lot of different things it has a lot of different parts and pieces and it can do the orchestration amongst hpc and cloud on-prem kind of that hybrid cloud model so it's a lot of things to a lot of people right but why is it important it's a great one robert ian anybody else forced you guys want to jump in so i'll share a little bit of something

um one of the big things that makes fuzzball important is that it's an automated uh not sorry an automation native solution um we see in hpc that as technology progresses we have both a need to bring these modern development practices into it um and we also see kind of the practice of hpc itself complexifying in terms of you know what workloads we want to be able to run what pipelines we want to be able to represent and so fuzzball on both sides of that gives the opportunity to codify and you know kind of formalize these different pipelines and stuff as complex as they are in

a way that can both be you know easily modified to account for other uses for that data pipeline as well as easily managed through modern ci cd practices so you know from the user side of fuzzball it improves things immensely as i said in both the management of high performance computing work as well as the actual perform you know doing and performance of high performance computing work many hpc centers have been asked as well you know going back i think 15-ish years how do we make use of cloud and you'll see that you know many hpc centers have been had a very tough time really

making good use of cloud resources part of that again stems right back into that base found fundamental architecture being very monolithic and flat how do you extend that how do you create some sort of infrastructure that can span between regions between clusters between you know infrastructures how do we build all of that in such a way that we can pull it all together and then make very informed and intelligent decisions on where should jobs run right when we think about where should jobs run we get to think about some really cool policies things like not only architecture and resource availability but cost of computing include

the data into the model as well i mean we've been talking about you know in high performance computing that we need to be thinking about the data as a first-class citizen in terms of um you know scheduling and orchestration but you know we we haven't done a very good job up until now on how to do that right when we think about data we have to think about data locality where is that data data mobility can you move that data and how easy is it to move at some point the data just gets too big and you end up with data gravity which is just

too big so the jobs kind of get sucked into the data and then you can also have data security policies that pervasively need to be applied across the entire realm of both you know movement of data and where is everything going as well as um sorry got a call i just had to go mute probably spam nowadays um but but how do we put all of that into a policy and think about that from a meta orchestration federated perspective and how do we pull all of that and unite all those resources together in a way that makes sense so we haven't had the ability to

make those sorts of decisions before so that's super cool and again i just want to double on what forest said regarding the diversity of of jobs and workflows has been massively expanding uh hpc used to be something very specific with regards to tightly coupled parallel applications and anything that was not tightly coupled parallel applications some people wouldn't even call it hpc now hpc is pretty much anything that uh needs to run fast and whether it's a single process single thread or you're you're scaling out or you're doing some sort of parallel training or you're even you have a service that's running that's doing inferencing or

some sort of in uh data stream or pipelining of data and ingestion of that and being able to commute compute in real time on on what's all of this is now high performance computing so the scope and realm of what is high performance computing has increased massively and our traditional hpc architectures are not very good at dealing with this now that's interesting because whenever i used to work for in the airline industry one of the things that they did a lot of every month they had to come up with a schedule for planes and it had to make sense for where planes were where the

crews were how much the fuel was how efficient the plane was and it was something they typically just ran on it used to just be on a bunch of hp ux boxes and then it became kind of a small linux cluster but those jobs would run for 30 40 days until they were optimized enough but it sounds like that would be a really good use case for enterprise i.t actually using an hpc type environment and is that something fuzzball would help with yeah yeah absolutely and again because you know when we started thinking about how do we modernize high performance computing uh the first stabs

that we took the first first when we were first thinking about this it was what do we need to put on top of traditional hpc what capabilities what apis um and it was it was very difficult because we're basically you know um i know i'm gonna get backlash for this but we're trying to put a saddle on a cow um it's not exactly the same thing because it's more like putting a saddle on a high-performance computing race car uh maybe that's a better analogy in this particular case but it's just it it became overly complicated it became full of duct tape and shoelace to pull

everything together and really what we needed to do is think about this from the ground up so we went the other direction we started with kubernetes and we said what do we need to do to make kubernetes good at this sort of thing and again maybe that's a little bit more like putting the saddle on the cow though it it just it's the wrong tool for the job especially when we're talking about high performance computing um and trying to run that through a architecture that was designed for microservices it's just it was not a good fit we had to come to the rationalization and the

understanding that we we have to rebuild this we have to think about this from the ground up and almost reinvent high performance computing and the or the architecture for high performance computing into something that is again i love the string of buzzwords cloud native cloud hybrid federated meta orchestration for performance critical and performance intensive workflows jobs and data and that's what we that's what we basically set out to solve and um it's a big problem to solve to be honest and that's why we were founded that's why we were funded to create that and um we've been working on it for about two years now

it's been fairly hush-hush in terms of what it is we've been building and so we're just coming out to market um now we're just starting to do some announcements we're letting this out in terms of what we've been working on we have a number of organizations that are currently in proof of concepts right now early access beta users and we are going to be releasing this around late q2 maybe early q3 and and bringing this ga in the meantime we are working with early use cases and early individuals and organizations who want to be part of this want to see what it is and have

a strategic interest in terms of bringing ideas back into the platform and um and really helping us to make sure that we are solving all of these problems in the in the the use case that you just described zane is massive um and and that's just a perfect example of how these sorts of architectures can now can now support organizations who were never doing high performance computing before but all of a sudden are looking at at hpc and thinking of themselves as being hpc consumers sure let's see we got a question that popped up but i wanted to pick on robert before we answer that

question just real quick i know that you talk to a lot of different companies a lot of different people out there what are the dominant use cases that you're running across right now that are people that are interested in fuzzball what are they trying to solve what are those cases they're trying to solve with fuzzball today [Music] um so it's very uh very varied like you were saying like greg was saying so um it's all the way from traditional high performance computing uh which we do extremely well and can do on any hardware and you know any cloud now and tie that together in into

one unified resource um but it's also uh machine learning ai uh starting to come into play in a lot of different places and that's where the flexibility of the solution uh really comes into play more so that that's what we're starting to see more and more even down the running you know jupiter notebooks from the hpc resources vdi um we're starting to run into a lot more persistent jobs as well so yeah i mean it's it's flexible enough to solve all those varied problems at bare metal speed on the when it's deployed on hardware itself um but have that flexibility to be dynamic in the

clouds as well so um yeah i mean that's what we're seeing over and over again excellent all right so tron your question is what are the components or technologies in fuzzball so i think that's an interesting question and it could be a big one yeah so there you go this might take a second um but we can kind of start at the base level so we have um several kind of layers which is a typical way of describing software stacks to uh what we would call like a full fuzz ball cluster so the first level is fuzzball substrate which is really just a single node

resource manager that is able to spawn containers with um a leasing model that allocates resources to containers based off of an api so that allows us to have a set of nodes manage all the resources on the nodes with a higher level orchestration tool and run containers on all of those nodes in order to run basically any application stack that is suitable for the hardware resources available for that node just to give a little bit of a parallel um for traditional hpc with like a batch scheduler if you're familiar with container technologies like um apptaner or others that are available um what you would typically

do in those environments is have a batch scheduler that spawns a process running as your user on that node and then you have that process call out to like a cli tool that spawns your container fuzzball substrate actually just takes the direct job request essentially for running a container and immediately spawns that user container for us so we don't have to have another tool in the middle so this is part of what greg was talking about about reducing the complexity in the stack and building it from the ground up so we don't have as many integration points and we can manage everything um basically from

a single control plane so on top of fuzzball substrate is what we call a fuzzball orchestrate and that's actually a set of microservices that run in kubernetes so we use a kubernetes cluster to manage our microservice stack because kubernetes is a very good tool for that job we could use another uh orchestration framework for running microservices if we wanted to but that's what we chose because it's best in class at the moment at least in our opinion and that will um basically allow us to manage the inventory of compute nodes uh that are available in the cluster so we have a set of control plane

nodes that run our kubernetes cluster with all of our micro services on top of it that manage our compute nodes then we have our compute nodes that just run fuzzball substrate so we basically can start up a container on football associated with a certain set of resources and then get out of the way and let it run as it would normally run for any other type of kind of batch work that you might be familiar with um so with that physical orchestrate stack we can do things like manage container images run what forest will show us with our workflow definitions where we execute a directed

acyclic graph of steps in order to complete um a computing pipeline and do other things like manage data or manage policies around users being able to use certain resources or research quotas things like that so that's kind of the second level of um the stack and then out on top of that we have what would i think we called it a fuzzball federate um so that layer is essentially taking all of our fuzzball orchestrate clusters and connecting them together so we can make high-level decisions about how we want to schedule workflows so at the fuzzball orchestrate level we're doing things like making decisions about how

we schedule directly to nodes whereas at the first ball federate level we're making decisions about how we schedule two data centers essentially so we might have several different clusters in several different locations maybe some on-prem maybe some in the cloud and we want to make decisions about which cluster or data center is the best to run a particular workflow based off of different types of suitability whether that's resource like amounts or if it's actually like resource classifications or overall policy about what um what what resources that user is allowed to access um so i think uh greg you just pulled up a slide or someone

did um would you uh like to talk through that slide a little bit you you do this a little bit more than i do from the slides perspective at least you you were talking already to this so i just wanted to put up the visual um this is this is kind of what a federated meta orchestration uh platform using fuzzball would look like where you basically have uh workflows coming into this federation layer and this is where you can make policy decisions based on things like um you know resource and architecture availability as i was saying before cost as well as uh you know where's

the data data gravity data mobility data security so we can actually say so for example if a workflow comes in and this workflow is requiring data that exists in let's say s3 we can make the decision do we want to run this workflow up in aws so we don't in a particular availability region or zone so we don't have to incur any sort of egress cost of moving that data but now we're actually incurring some sort of cost to actually run the run the compute if we move the data to an on-premise cluster maybe in that particular case we'd actually get better performance maybe it'd

be less expensive but we get this ability now that we can start thinking about it at this level and to kind of go back a little bit in terms of what ian was describing in terms of um components and and what the architecture looks like you'll see if your traditional if you're if you're familiar with traditional hpc this is kind of like you know a traditional hpc compute cluster you've got some sort of base provisioning you've got a small operating system and you got in this case fuzzball substrate running across all of your compute nodes this could be tens this could be thousands of compute

nodes um on this side you're going to end up with you know some sort of scratch file system parallel file system luster gpfs or just a high performance nfs type system to share out data and volumes across this compute cluster now uh as as ian was saying you know fuzzball orchestrate is a microservice platform in itself so we're going to want to run it on top some on top of some sort of microservice platform like architecture or service provider like kubernetes and in this particular case this is how we would ingest a workflow via an api here in fuzzball orchestrate this workflow will come in

we'll be able to basically ingress data as the workflow defines ingress data create the volumes run the job pipeline and then egress any data and what that means is this data lake kind of becomes the home directory storage of what we're what we're using today as home directory storage and high performance computing we still need as i said that scratch file system that's running here because we're running a bunch of volumes and we're shuffling data across a bunch of nodes so we need something highly performant here but in terms of that persistent storage we don't need to use some sort of very expensive you know

enterprise grade nfs home directory storage which is what we have been using now we can use a data lake and the nice thing about the data lake going back to this slide is data lakes are actually and object stores are much better at doing distributed and redundancy and and spreading out across geographies so this gives us the ability to leverage that data store as a distributed you know data lake in a matter of speaking and and now we can run jobs anywhere and we are working with partners as well in terms of organizations and providers of a distributed data lake and to figure out what's

the best way of doing this and maybe it makes sense you know while i have some slides up if nobody minds me standing on my soapbox a little bit longer to talk a little bit about the workflows and i think forest you have some some demonstrations that we're going to be be running but just to give a quick introduction in terms of what is in a workflow and how a workflow looks we basically need to support three different aspects in a workflow so workflow basically is going to be consisting of you need to define your data and volumes and you need to describe your job

pipeline in this case the volume we're going to set up a volume named v1 it'll be an ephemeral volume we'll have some ingress data we will pre-populate this volume with and fuzzball will automatically manage the movement of this data it'll manage the caching of this data so subsequent jobs that require the same data will land on the same resources and be able to leverage that at the end of the job pipeline we'll be able to egress things that we want to persist and retain after running this job or the workflow excuse me now jobs is basically it is a full acyclic graph of individual jobs

each job has a number of attributes associated with it so ncl just to jump at the end requires ufs ufs requires untar and untar has no requirements so that one will run first you'll see it's going to run every job is going to run completely within a container so the first thing we're going to do is we're going to pull the docker alpine or the alpine container out of docker we're going to basically spin up that container and we're going to run this command inside of that container when we spin up this container we're also going to mount up v1 this volume inside that container

and slash data now once this is done which is basically a data prep we're going to untar what we downloaded up here then we're going to run the ufs weather model now this is an mpi job so we're going to run it across 128 nodes we're going to do the complete mpi wire up using openmpi so historically with running containers and mpi on high performance computing systems you're typically running in like a hybrid environment where you have part of the mpi outside the container part of the mpi inside the container in this particular case we're actually running 100 within the container so uh these this

this this container would basically run across 128 nodes if you're running infiniband that will automatically be be managed and and leveraged and we're going to run this command within it we're going to start that command in this directory notice in slash render and that's because that's where we're mounting volume one which we again from up here at slash render and then ncl is a a quick visualization program that we're going to basically run uh to basically take the output of this now turn that into a couple images and at the very end of this job we're going to take those two images and here you

can see we're just putting them to a local file system but we can very easily upload that into something like s3 where it can be visualized globally or elsewhere we have a graphical environment that we're working on as well that will be released shortly um or when we release fuzzball and this graphical environment can be used to both create and make workflows but it can also be used to templatize workflows such that each workflow now becomes in a matter of speaking a very simple compute focused application that you can now create almost like a web form to go create an application to go run workflows

where the inputs the outputs or any other templates that you put into that workflow will be automatically replaced with the contents of what's in that that web form so there's a lot of kind of options um and uh and things you can do with a system like this and keep in mind even when we go up to this level that workflow will come in and we will figure out where it makes the most sense to run it based on the policies that the organization requires and has set forth one other quick note there is no ssh anywhere in the system so all of this is

going over apis that are governed by im policies across the entire platform so every aspect of this can be secured tightly and system administrators and again organizational policies can define what is allowed to run where is it allowed to run how is it allowed to run and what are users or groups of users allowed to do with that so i've been talking for a while anything else anybody wants to add to that yep so that from from the workflow perspective that file that you showed that's that's what a user would create at some point there will be the gui to help them create that so

that's user kind of the experience and i know forest is going to show us that in a little bit but from an administrative standpoint and setting up those policies for where things land how hard is that and how does that from an administrative perspective what does that look like ian you want to take that yeah so um basically we have uh sets of curated policies where like for initial deployments and proof of concepts we're working with organizations to understand what their needs are um and then building the policies with them in order to properly i guess set user roles and manage resource access and do

things like that the policy system is very expressive which means you can do a lot with it but it also means that there's a lot there to document and to understand and so it's basically a learning process where we want to set up um specific roles of policies so that it's easy for like a standard deployment to work how administrators would expect based off of what they need and then if they want to dig in and do very specific things and allow very specific things to happen they can go ahead and do that as well so it's really about making sure this the common use

case or for the common deployments that administrators want to make that is straightforward and you get kind of the roles and abilities that you would expect and if they want to go on and tweak things then they have a very expressive system to do that so they can get it exactly as they'd like excellent thank you so now we are right at half an hour forest i want to leave you enough time to make sure that you get to show us what you brought so if you want to go ahead show us your demo sounds good everyone uh i will go ahead sorry i'll go

ahead and share my demo or my uh terminal screen here really quickly give me just one moment yeah why is firing that up we got a couple of other questions and there's so the first one what about a migration path from existing end users from a traditional hpc environment what does that look like that's probably another question for me and greg yeah so the migration path would really be about using either a proof of concept environment that you might set up so you would like to set up a side environment from your production and give users access to that to start to test it or

you would give work with us and give users access to our cloud deployments off as well so that they can get familiar with the platform and get a feel for how it works from a user aspect from a um like administrator managing infrastructure it would be a hard switch in terms of having your nodes be managed by fuzz ball versus whatever your current infrastructure is from a user aspect there's definitely paths to migration where users can get familiar with the platform understand how the workflows work and even use the same containers that they're currently using if they're using uh apptainer containers on the fuzzball platform

as a part of their workflows that uh farce will demonstrate um and get familiar with the system great and the other one real quick so yes fuzzball is cloud vendor agnostic it doesn't really matter but whenever you say i am greg policy that sounds like an aws term i know we're not talking about aws yeah so sorry about that um we are using the same acronym acronym uh for identity and access management but we it is a policy that we have created specifically for fuzzball it is not related to aws i am at all however in name however i would add that one of the

design principles is that the end user and the administrator have the exact same experience whether it's in aws gcp azure on-premise connected as a unified platform it does not matter that i am policies do transfer no matter where they're running them and that end user and admin experience will be unified that's extremely important to what we're trying to accomplish thanks robert thanks for the questions all right forrest back to you all right we're good to go all right if everyone can see that all right i'll go ahead and start into this so i'm here on my command line um as greg mentioned i'm not ssh

to anything i've just gone ahead and done the fuzzball login process which is a web-based flow essentially um so i'm going to go ahead and run a couple of workflows here the first is going to do um it's going to run jupyter notebook and i'm going to show you how you can interact with that while it's running on a compute node and then the second we're going to do is more traditional batch processing with a molecular dynamic suite called lamps so i'll go ahead and start this workflow off here and here on the cli we do fuzzball workflow start and then i provide the name

of the workflow and my account information so we'll do jupiter ai training we'll just say this is the first one of those we're going to run and we'll specify the path to the workflow yaml file that we're going to be using um that contains the workflow that fuzzball is going to run we'll go ahead and start that off can you increase the font a little bit sorry about that yes does that look better uh yeah if you can do one more i think that'd be great there we go okay so um you'll see that this workflow is now started um we can go ahead and

do fuzzball workflow status and we'll watch this to watch it as it runs you can see we have actually one second i'm going to decrease that because that's going to want to print [Music] yeah we do need a certain amount of character with in order to display all the information and like the the table describing the components of the workflow as they're running um so i'm not sure if you can widen your terminal force that you're sharing or if we just need to lower the uh the size of the font uh this is as wide as it goes i'll lower the font a little bit

uh you can see that a little bit better um does that still look good on everyone else's end cool we're waiting for this to start anyway so um really quickly um i'll get well let me just clear my terminal really quickly so just to explain what exactly we're looking at here um you'll see this wraps around a little bit but this last column is uh basically just any errors that happen with any of these workflow components as they run we have three different things represented here workflow volume image job workflow over here is just kind of giving the actual status of the entire workflow itself

volume and image are giving the status of the setup of the data volume that's going to be attached to this workflow that it's going to be able to use to store data in or bring data out of when the workflow is finished and also persist data in between workflow jobs so they can each access that same data that's been you know pulled in to this workflow or generated from somewhere else in it you'll see that we have also an image here that's being pulled this is coming down from a git lab registry so i've provided some secrets inside of this workflow that are essentially my

credentials to access the skit lab registry which is what gives me the ability to um yeah be able to pull that container down because otherwise you know wouldn't have access you can also see one job here and that's the job that here in a moment i'm going to um do some stuff with so that i can get access to the jupyter notebook that's running in it at the moment as we run status and as mentioned we can watch this but with the width of the terminal it prints a little weird so for clarity i'm just going to keep it like this um you can see

that at the moment this job is basically just sitting here pending what's going on is that we're essentially provisioning this node right now um so that we can launch this job here on it and be able to have access to it within this notebook that i'm going to show you in a moment um so we're essentially just waiting for this job to start while we're waiting do we want to go through the workflow definition that backs this really quickly um i'll just jump in and say that the reason why this is taking some time to start is because on the back end we're actually provisioning

a vm within a cloud provider in order to run this workflow based off of the resource constraints specified as a part of um the workflow definition so when you show the workflow definition we'll see the resource constraints um so i think we should do that next but i just want to say that like if we're working with like a static node pool of like on-premise infrastructure it'll start running immediately unless there is like a queue for those nodes cool thank you ann very good to point out there um yeah so really quickly i'll take you through what this uh workflow definition here looks like so

you can see this is basically just a flat yaml file and as greg kind of showed with that example earlier we have a few different sections we have basically just a version header that tells what version of our dsl this is based on and then we have a list of jobs that are going to be executed when this workflow runs in this case we basically just have one job and that's the name jupiter but you can see we have a bunch of different things inside of this job that are being done in order to properly set it up first off i'm going to kind of

go out of order here first off we have the container image that this job is being pulled from and that is landing on that compute node and that the actual computational job here is being run inside of you can see as i mentioned this is pulling from git labs this is just a custom container that i've put together that has everything we need in order to do the notebook that we're going to do in a second in it you can see i've also provided some secrets and a um basically some credentials there and i've also stored some of those secrets inside of our fuzzball cluster

for easy templating when that workflow runs um so i don't have to manage those credentials in an unsafe manner myself um and you know for ease of use uh you'll also see that uh after the image we have this command here um this command is what's going to actually run inside this container when it all starts up on the node um so in this case we're basically just launching a jupiter notebook instance on port 8888 um with a couple of other options there we also do a few things to kind of set up uh the other parts of the environment here we have some policies

that are set in this case we basically just have a timeout policy that means that this workflow will stop automatically after 30 minutes and then we also have some environmental variables here that will be injected into this container so that anything running inside the container can access them as well you'll see that we are mounting some type of data to this i'll move down here you'll notice that in addition to the job section we have a volume section this is actually allowing you to um bring up a storage place as i mentioned for workflows to use um so this is an ephemeral type volume this

will disappear and basically only be active for the lifetime of this workflow so this is sort of a scratch space to bring things into and use as a space for jobs to operate on the same data within so you can see we set that volume up there and then we attach it to this job right here at this location which i'll show you in just a moment once we actually hop into this notebook and then here we also have the resource counts that this is going to be looking for so as we've discussed this allows you to kind of set on the fly exactly what

resources you want available to a workflow um and then the scheduler will do all the logic in order to figure out where the best place for that workflow to run is so you can see we have cpu memory and devices here um cpu allows you to specify the number of cores that you want and then how you want those cores to be kind of selected from the compute node in this case we're doing enuma affinity so this will minimize the number of cores onto as few or will not minimize the number of cores it will minimize the number of pneuma nodes that are being used

on a given compute node um if it supports a pneuma architecture um so that's efficient that's trying to um you know land things all in a space that you know all those cores are going to be able to access the same memory um location with pneuma um but we have other affinities that you can choose that are a little bit more broad and um you know less granular there we also specify the amount of memory that we want in gb bytes so that's going to give us 12 gb bytes worth of ram that this workflow will be able to use and then we also specify

that we want an nvidia gpu through this device's specification so this basically is just going to reach out and look for a node that says it has a gpu attached to it and then serve that back to us so we can use that our workflow i'm going to go ahead and as this is provisioning live sometimes it can take a little bit of a variable amount of time for the node to come up so i'll status this again and hopefully we'll see that this job is started and that means that the node is brought up but it hasn't yet so we're waiting on this um

this typically takes i would say about five to ten minutes or so um so we'll probably be waiting on this for a second longer but it shouldn't be too awful much longer i don't hope or i hope i can jump in on this question while we're waiting for that so um basically if we if you go back and i don't have the the slides up right now if you have put a cluster in aws for example we could be leveraging that cluster if you have a cluster that's on prem you can run a federation or fuzzball federate on top of both of those and then

fuzzball federate will decide which cluster it goes to there is no infrastructure within this demo that is like a cloud sas type solution although we have been asked to do this so this is going to come online where we are going to be hosting a fuzzball cloud for people if they wish to to run on top of that and then we would basically make determinations in terms of where things would run again based on policy but for the for the moment it is going to wherever fuzzball is currently installed thank you greg and it looks like we are still waiting on this so it's always

for a live demo huh of course you know my tests it was you know five minutes or so reliably but we'll uh we'll see how long this ends up going for here murphy's law there is an option as well um that we've used in the past and we're not doing it now apparently where you can actually provision resources to be spinning and be up and running kind of on standby most organizations when we're talking about the cloud compute side of this really like to keep all of their costs down to a minimum and that's one of the nice things about cloud so a lot of

organizations that we're talking to especially if they have multiple resources they've got an on-prem resource they've got a cloud resource the on-prem resource is kind of statically defined it already exists so typically you know you're going to have all of that running and available and ready [Music] on the other side of this if you have a a cloud side a cloud cluster you don't want to keep any compute nodes running just for the sake of keeping them running so typically that will automatically shrink down to just the bare minimal requirements for fuzzball to operate which is really just that kubernetes cluster and we'd sit on

top of like eks or or whatever you happen to be using for your kubernetes infrastructure up in the cloud we would just sit on top of that and run there and then as soon as a job comes in we would automatically provision a new uh instance a new cloud instance for that job meeting the needs and specification of the workflow coming in um and whatnot so hopefully that answers the question um and i see that it is started so i'm going to stop blabbling and handing hand the mic back over to forest okay so we're good to go um that note is provisioned as you

can see by the status over here listing has started that means that this job is actively running on the provision node um that we should be able to access it and do some different things with that so the first thing that i need to do in order to get the link and token that i need in order to access this jupiter notebook is just to open up the log of this workflow really quickly and you can see we have some output there we're going to be looking at this namely and because this is running you know at a separate data center i'm being a public

cloud provider i actually have to port forward this workflow back to my laptop here in order to be able to get onto this notebook that's being run here so i'll just go ahead and do fuzzball workflow forward and then we'll provide the name of the job and then the name of our sorry the name of the workflow and i'll provide the name of the job in that workflow that we want to connect to and then a local and remote port to forward from we'll run this and you'll see we're listening i'm going to go ahead and switch over my sharing really quickly so you can

more easily see what i'm doing here it's important to bring up that's not an ssh port forward either you get asked that question quite often whenever forrest is going through this it's actually i'll probably say it wrong greg api tls so we're using uh grpc as the foundation of our api and so we can use a streaming uh rpc setup to pass the data from um like your container process all the way across a couple proxies when we uh do like these federated setups to your terminal so that's how the port forward works when uh force shows something like exactly into the container as well

um that'll also be done in the same way so we're not opening up ssh ports um to the node itself we're doing everything over our api thank you i think it's awesome oh sorry i think it's also cool to note as well is that port forward right now we're using that for for jupiter but you could use that for pretty much anything i mean we have uh customers that are that are using that for um uh remote desktop you know accelerated desktop visualization and and vdi um you can use it for i mean all sorts of stuff anything that does network i o access you

can actually do that port forward and that that the grpc api the streaming api gives us the ability to not only do ports but we can also do pseudo ttys so you can actually get a shell in one of the containers that from a user perspective you may not even know if that's running up in aws or what availability zone it's running in aws or is that running on prem all of that is completely transparent but it all just works bringing it right back to the user's workstation exactly as they would expect and all of this can also hide behind a graphical environment which we're

again we're working on right now to simplify and one thing i'll add is because it's through our api this directly interacts with our im system so if an administrator wants they can completely turn off forwarding or access to running containers and kind of uh put a curtain up essentially so if you have like requirements around uh what machines can view like the data being operated on by a container you could completely turn off these types of features and only allow workflow submission and nothing else um of course you're probably ready but real quick i think we should answer stefano's question um so in order to

get started we need a k-8 cluster compute nodes um which could uh if if you want if this isn't optimal for um like performance and context wishing uh we could uh have those within the case cluster but outside is is better um a uh a storage system that allows all the compute nodes to uh have the same view of data and that that's all you need to get started so if if you're interested in looking at more i would have you uh talk to robert about a potential poc or or anything like that um yeah go ahead for us cool so just to make sure

that we're all clear on exactly what we're doing here we've had this workflow that's ran here i've gone ahead and got the log from it and then i've gone ahead and port forwarded that workflow back to my laptop here when i control click this um it's probably going to bring up twice close the old one when i control click that you see that it brings up here in my web browser the homepage of this jupyter notebook this is a full interface into this compute node that i can you know basically use for whatever i want to do in jupyter i'll go ahead in this case

and upload a notebook that i have from my local machine here and then we'll go ahead and run that so we have a pie chart demo notebook here oops oops whoops sorry actually touched something we'll go ahead and push upload there and you'll see that that has gone ahead and uploaded so this is now sitting here on this compute node i'll go ahead and click this to open it well first off i'll grab a terminal into this so we can show you something else i'm about to do here in a moment but i'll go ahead and open this up we'll see that this notebook opens

as you would expect this is a terminal that's sitting on that compute node that we were just on so i'll do nvidia smi so you can see that we have a gpu available as we're going to be doing some gpu-based ai training here i'll also swap back over to this really quickly and show you our empty data directory this is where that volume that we attached to the workflow was put on or was landed inside of the container so this can be written to and we can persist data between workflow jobs and such like that inside this directory so i'll go over here and we'll

take a look at this notebook that we have this is basically just a pytorch tutorial notebook that does some ai training i'll go ahead and run through the code in it we'll do some quick imports this is going to do as we discussed fuzzball has a very robust ingress system that allows you to bring in data from different places and object storages it also supports custom data movements of data movement with inside jobs themselves if that's something you need to do we'll go ahead and run this and we'll see that this is going to hopefully download a little faster than that um because it was

downloading faster earlier um but if not i believe i'm well give that a second um yeah it might be going faster than we think all right uh so yeah like i said um this dan download will progress um i was expecting this to go faster than it is because it did in all my testing um demoing in the cloud is always fun yeah i uh like i said this should be like a quarter of the way done by now or so um so we'll give this a moment uh maybe we have another question or something we can discuss in the meantime anything that this will

take a few moments yes this is kind of a statement and then a question so it's talking about provisioning sounds like setting up fuzzball could be tricky but the next question that follows that is can a tool like werewolf help with that there you are or are there existing cloud formation templates to assist with with that provisioning so this is not a talk on on werewolf but thank you for bringing this up because it's super important one of the things about werewolf that allows um that really facilitates how a um uh how you would basically use werewolf to stand up a custom resource is b

being able to pull in i think by the way i think i'm i'm hidden behind the icon there um i don't know i thought that's just what you there we go thanks ian um uh one of the i completely lost my journey to thought now um because werewolf basically takes oci based containers and we provision them out to bare metal it's very easy for us to basically set up things like node images things like even kubernetes images and be able to splat that out to bare metal hardware spin that up in such a way that we can very much we can facilitate how it um

how it basically sets up you know these resources so yes there is there is a significant amount to actually set up a fuzzball cluster but aside from the kubernetes piece there's nothing that you would expect that is more than a traditional hpc system you still have shared file system you should still have to deal with the network infiniband provisioning out those resources and again werewolf will help with that we will be offering images for fuzzball that you can just basically download import those into werewolf and then blast those out to compute nodes so if you've got a thousand plus compute notes werewolf will be able

to provision those out very very efficiently and and get you going with um with with fuzzball at least on the compute section of that cluster very easily for the resource management uh of that cluster you will have to have a kubernetes setup as ian mentioned earlier it does not have to be in direct proximity proximity as a matter of fact we do have one uh use case um at an hpc center right now going on where they basically stood up uh the fuzzball microservices fuzzball orchestrate basically within a vmware resource so basically they're within it's vmware setup they basically installed you know they had kubernetes

running there they installed all of the fuzzball requirements for orchestrate there and then they basically brought in substrate on a system that's even on a route a different routed network from that vmware system and they're basically able to bring up um the compute nodes over there and then the compute nodes are talking to the orchestrate cluster that's back over in vmware so it's kind of a cool way of of being able to build this but if you do already have a kubernetes cluster we can definitely leverage that i'm looking up now at the screen and it looks like did we get past we did yeah

it started moving a lot faster than it was there so we're good to go yeah mom's in the cloud yeah we'll continue on with this we'll show a couple of those training images that we just downloaded here so we can have a little bit of an idea of what our data set looks like we'll go ahead and actually set up our neural net that we're going to be using for this run some of the other code that we need and as i mentioned this is sitting on a gpu node you may have noticed in the workflow that i showed there um i was able to

or i was specifying that i wanted an nvidia gpu and as i indicated over here i can also use nvidia smi and we have one of these tesla t4s on this node i'll go ahead and start this code right here which is actually going to train this neural net on that gpu so we can see uh one moment i'm going to reshare just my browser there we go okay so this should um begin giving us some output here in just a second we should there we go um so you can see this is starting to print the status of training um we have a loss

value over here that basically represents how well trained our model is and we're going to do two epics so we'll see this um stop here in about a minute or so if i move over here i can do nvidia smi again you'll notice that because of you know quirks of how containers work we don't see exactly what process is running here but we do have an indication that there is a process running here because we no longer see it there's none running and we also have 11 hiring 42 maybe bytes of gpu memory taken up which is essentially our model sitting on that gpu so

we move back over here we'll run this one more time you can see our utilization over here is at 24 indicating that this gpu is actually running at the moment um and you can see that this is still training on that so we'll give this a few moments um while this finishes we'll go over here yeah so you can see that from this kind of top-level view of everything if i go into the data directory um within this code that's essentially where i told this to download to so this is now in that volume that we set up at the start of this um you

can see that data that we spent a little bit ingressing there from within our code in just a moment this will there we go we'll finish training so that's done we'll go ahead and save off the trained model that we just worked with and if we go back over here you'll see that that model right there is available and i can also do things like download i'm not sure if you can actually see this but i'm able to download that directly to my laptop from this interface as well so if i want to move things around there i can do that if we go over

here we'll show a few images from the test set so we've got a cat a ship a ship and a plane there we'll do some code really quickly to get these test set images to the model and then we'll go ahead and actually run essentially some live inference over our test data set so you can see our model is predicted that this is a cat a car ship and a plane which is not too bad for something pretty basic like this and then we'll also go ahead and run the neural net that we just trained over the entire model so we'll get kind of some

information on how exactly it is how accurate the model is over the entire data set in general you can see it's about 55 percent accurate and we'll go ahead and run this as well which will give us kind of the output per category that's in the scifir10 dataset we're using to train this so you can see that for each one of these different categories we have a value for exactly how well the model was able to recognize that type of object and yeah once again this is all running via fuzzball workflow this is sitting on a node at a major cloud provider's data center and

you can see that we're able to just with the container that i've created instantly bring it up um and then upload you know material and stuff from my own laptop i could ingress things in from the internet to it but you can see essentially here i'm able to um open this workflow open the jupyter notebook inside of it and then you know upload something from my own laptop and be able to run that effectively you know using a gpu as well so so that's jupiter notebook through fuzzball excellent we're kind of getting close to time here guys but there is one question i get quite

often greg and i'm gonna i'm gonna direct this one to you because it is something that you ask pretty much every time i talk about fuzzball and is why is it named fuzzball where did the name come from what inspired robert yeah that should be the robert robert robert named fuzzball um it was essentially uh a play on uh as greg said earlier you know the pandora's box to get us to fuzzball was uh containers um and singularity uh that greg founded is is a piece of that um obviously fuzzball is uh a singularity as as well but in string theory right so um and

then also it ties together uh everything into one and that's really what we're doing here right tying together multiple different resources into one and giving end users and admins an extremely efficient and similar experience across everything as well as uh providing applications the ability to have freedom to run wherever it is best for the applications to run this is what's real important about fuzzball it gives the power back to the researcher power back to the administrators of of those researchers and gives them the ability to really do the science that they want to do and that's really what we want to do is really enable

those folks to do what they do great um at the end of the day that that's really where fuzzball came from and it's it's really our ethos of what we're trying to do as a company i'd only add i think robert nailed it i i'd only add as well we've got the cutest little mascot too it's a little fuzzy um that you'll see here come out here shortly so so yeah tron's asking how how can you get his hands on fuzzball that that's going to be a probably a me question so i will reach out to you tron and and i will work with you

and we will figure out how to give you access to fuzzball so if we don't have your contact info tron please someone reach out to try and get that from if we can without making it too public so i appreciate you guys i know we are actually about a minute over thanks for the time today we appreciate you joining us again remember like and subscribe we will see you next time thank you very much real quick real quick hold on oh sorry we are growing fast we have a whole slew of open positions right now uh if this looks like something that somebody is interested

in being part of please go to our website and check that out absolutely thanks again bye