Fuzzball On-Prem 101

can dive into the things that have changed a little bit or that you guys have really been focused on um including kind of what IQ was what an operator has turned or what that's turned into from an operator perspective but if you don't mind just kind of high level Brian what is fuzzball so fuzzball is a workflow orchestration platform uh which is I guess part of what ciq calls HPC 2.0 uh with this um workflow orchestration platform it's got its own workflow definition language and where you could build workflows consisting of various jobs and you could also do the same in um a web UI

as well and this whole platform basically enables performance Computing and at the end of the day uh Al also um reproducible uh scientific Computing uh yeah uh Jonathan is there anything you'd like to add to that uh maybe the only other thing that I'd add to that is uh everything is developed kind of API first and so the parts of fuzzball talk to each other over a defined API and then an API is also exposed to the end users so um some of our customers are using that to develop or you know they use a python SDK that we generate from that API and uh

allows the whole system to be orchestrated and automated even from outside of fball very nice thank you so one of the things I remember when we started this was hearing a lot about icbe so if we could kind of go into the icbe history what it is why it existed and kind of what has it become today or what are we doing differently now sure so to my understanding ibe was a provisioning tool developed by ciq which encapsulates um I guess a ciq flavor of kubernetes and also the fuzzball orchestrate platform and the goal of IQ was to be able to uh provision the fuzzball

orchestrate platform with the kubernetes engine that it runs on onto an umrem system and through this icbe Tool uh it was also designed to be able to also deploy in air gapped type type of systems as well but yeah Jonathan do you have like nod your head make have some of that yeah I that that seems right to me that um a lot of it just comes down to where your or what what we consider the scope of of our domain what what are we responsible for and uh the the biggest thing to know as a point of history is that I Cube considered uh

the kubernetes environment part of our domain and we're that's part of a shift in our on-prem deployment strategy that I expect we'll talk about in a moment absolutely so I think that that actually leads right into the shift and now we're talking about creating an operator so maybe not leveraging IQ as much or at all and and moving more towards a real kubernetes operator so what have we done there and what is that mean so uh you using the um operator model which is uh I guess a common uh type of pattern used uh when administrating kubernetes systems uh We've basically designed um the operator

to be able to deploy uh the fuzzball orchestrate platform onto a existing kubernetes cluster um the specific platform that we've tested on is Rancher kubernetes Engine 2 uh which is what we will be kind of demoing today um in our Des so yeah yeah Jonathan is there anything else You' like to I'll just add on to that like that's that's what we and I say we very specifically like Brian and I have been using internally because rke has been really great to kind of spin up really quickly it's been easy to get up and running with and then we we were doing some of this

work it was what one of our clients was specifically interested in using so it was our first Target but we have had other people in the company Deploy on other kubernetes platforms without any reported issues so we're working on getting that kind of proven out and and uh you know just developing a list of what we consider a tested platform but uh so far we haven't had any reason to think that it wouldn't work on any reasonably standard compliant kubernetes right so I think one of the things I want to make sure we delineate here is we keep talking about kubernetes and orchestration and then

high performance Computing can we differentiate what kubernetes is doing for us in this environment vers what goes on inside of an HPC cluster so not necessarily running kubernetes on HPC cluster if you work but using kubernetes to do what uh so basically we're using kubernetes to run uh the various microservices to manage um the uh I guess the workflows uh and data uh software images that are being orchestrated for uh your particular workflow but at the end of the day uh your workflow is actually running on bare metal uh which the orchestrate platform uh I guess runs the yeah which orchestrate basically will run the

job on bare metal through uh orchestrating the workflow using these micro Services yeah I I don't want to uh like maybe you were planning on this at another time but this might be a good opportunity to put the diagram that you've put together for this Brian that might make it and then we can like gesture to that and say what parts of it are in kubernetes and what part are not for sure for sure let's get this shared out perfect uh just gota give it a second sounds like changing the flow of the conversation guys sorry yeah no cool so uh so when jobs are

submitted to um so fuzzball orchestrate is running on raner kubernetes engine which as you can see here on this controller node uh jobs are submitted into uh fuzzball orchestrate and fuzz on your compute nodes we have something called fuzzball substrate which is you can can kind of think of this as uh a task Runner that runs uh various jobs uh that are submitted to orchestrate so orchestrate will uh stage the various uh data and uh software containers uh onto the substrate node and eventually uh end up running your job on this particular compute node yeah so if you're coming from like a traditional HBC environment

if you've run like slurm Fuzzball orchestrate is kind of like the slurm control d slurm control d and slurm DVD rolled together things like that and then subrate substrate is like the slurm D that runs on your compute node and if you're coming from a PVS environment then fuzzball orchestrate is like the PBS server and fuzzball substrate is like the PBS mom that kind of thing but it it adds extra functionality you have um like substrate is also a container runtime because every job in fuzzball is is containerized so it does all of that transparen transparently as part of the agent I don't want to

go too far into it but from a when we when you say containers and a container runtime what container formats will it run uh so yeah the the um the runtime on fuzzell substrate can ult technically speaking it's always running like a cth a singularity or Apptainer style image it can transparently convert oci container images as part of the workflow so um um you can bring containers to it in either uh CIF or oci format and it will do the right thing so any kind of oci compliant container yeah okay thank you I think now we kind of started down the whole uh diving into

an overview of orchestrate high level again I took you a little bit off track sorry so if we want to go into that now and talk about workflow orchestration Resource Management scheduling I mean you kind of touched on it Jonathan but is there more that we should go into um I don't know like maybe just just reiterate like a fball fuzzball executes workflows those workflows are collection of jobs that can be uh related to each other through dependencies and then the the workflows are processed in orchestrate and dispatch jobs uh which are either the the compute jobs that you want to run or kind of

ancillary jobs like pulling an image or doing Ingress egress that kind of thing uh those all execute outside of kubernetes on your compute resources that are running substr thank you and I've heard talks of fuzz files what are fuzz files go for it Brian so uh fuzz files are basically uh the fuzz are your workflow definitions which are uh submitted to fuzzball orchestrate so uh with within this workflow definition you can have a collection of jobs like Jonathan mentioned uh which can be related uh using either the CLI or the web UI you can take this fuzz file submit it to orchestrate and have your

workflow uh run on the on your various compute nodes running fuzzball substrate these are typically yaml formatted files um I think they can be Jason as well but uh but we pretty much always do them with the AML very nice thank you I think now you guys have something to show us right you g to see this thing in action uh yes yes so uh so for today's demo uh what we will be demoing is uh the installation of the fuzzball operator uh from ciq Mountain uh and then once we have the operator we are going to use it to uh deploy fuzzball orchestrate into

uh the uh RK cluster that we have running on this controller node and then finally we will uh go and install substrate on our compute node uh and just to get a compute node up and running and then from there we will switch over to a deployment I have up and running as this takes uh some time to get up in running so uh we'll switch over to that where I'll demo what a completed deployment will look like and from there we'll uh submit a couple jobs into uh fuzzball Orchestra straight and I can show you how uh jobs running within fuzzball kind of looks

like from both a CLI perspective and also a web uat perspective and Brian I see that we got a question from LinkedIn at some point just asking for clarification on what an operator is so either now or as you're doing the demo can you make sure that that we give that because I think it's easy for us to be like just say it and we know what that means but we could give a little bit more detail uh yeah oh would you be able to take that one sure I'll do that the most well in kubernetes uh so that one please the operator pattern is

just the idea that you deploy a container uh into the kubernetes cluster that itself manages aspects of the cluster so for a relatively complex deployment like fuzzball is Um this can be a good strategy for having um you know a lot of flexibility in how uh your Fuzzball cluster is deployed and managed and it also makes upgrades a lot easier so it's like we took a chunk of what I Cube was and instead of running it outside of the cluster and and having it be kind of proprietary and bespoke we we deploy it as a container inside the cluster and so it has full

access to the control plane of the kubernetes cluster that we're deploying fuzzball orchestrate into and then coupled with that is a custom type so if if you haven't used kubernetes before it's kind of like a a big collection of objects that Define uh typically in yaml format how you want things to be deployed um so we have a fuzzball type custom type that we've uh defined that says how you want your your uh your cluster your fuzzball cluster to be deployed and the operator controller sees that definition and makes your cluster be configured the way that you want it to be and so if you

want to make configuration changes you can just edit that object in your cluster and the operator controller applies those changes throughout the cluster as necessary if you upgrade uh you just upgrade the operator or the well you upgrade the operator which upgrades the controller which puts the new version out into the cluster and it's all been quite seamless and uh a good experience uh so and then it integrates into um into Ci's Mountain platform as well because all of the containers that are needed to do this deployment are just pulled right out of Mountain um and the uh the installation of the operator into your

cluster uh in the first place is done directly from Mountain so it's it's kind of the more at least a more native or Community uh typical way to do this kind of a a kubernetes deployment and yeah we think it'll be much more streamlined in addition to uh supporting deploying into uh customers existing uh kubernetes environment cool cool thank you Jonathan uh all right so uh that being said I guess we can jump into the demo so give me a second I'm just going to switch my screen share real quick okay so cool um is the font size okay here go another couple just to

make it a little bit bigger there we go there we go maybe thank is that is that better yes now that I think now that it's wider make it bigger again it did reset oh okay how's that that seems good that works okay cool cool cool cool cool all right so um I'm on this uh controller node uh that that was in the diagram that we previously shown uh I guess to start off uh I am going to install first start off by installing the uh fuzzball operator so to do that uh I am going to first use Helm uh to log into uh Mountain

uh so I have a mountain access key which I have set in my environment and this will basically use Helm to log CIQ's Mountain so that I could pull the uh fuzzball operator into uh the kubernetes cluster so start off by doing that one once logged in uh now I'm going to set a couple of environment variables uh to I guess pass into my Helm command uh to pull the actual fuzzball operator so as you can see here things like uh the version that I'm pulling uh the URI to uh the helm chart and the uh URL to the image uh that I want

to pull are being set so once those are set um I'm going to just paste this Command right here I'm going to uh I guess call Helm and uh pull the Fuzzball operator from Mountain uh and in doing so uh I'm basically uh passing in things that I set before like the version uh pointing it to uh Mountain uh right here and then also providing it some credentials to actually access uh these artifacts which are provided through Mountain um I should also mention uh everything that I'm showing today is uh available um in our documentation at beta. fuzz.

iocs so uh if you ever want to reference uh anything that I'm running today you can definitely take a look there uh so from now that I got this command pasted I'm just going to run that and it'll start installing the fuzzball operator so uh from there um we can see that I have C control uh so I have Coupe control on this node which is running an rk2 cluster uh and from using coup control uh I can take a look at uh the uh the fuzzball operator being installed uh Jonathan could you provide a little bit more uh color on what this PV command

does oh sure so there's a number of just smoke test things that are in there are docs for seeing that the operator is installed properly the the PVC is a it's it stands for persistent volume claim this is a kubernetes thing this is the the operator needs a place to store its information and seeing that the fuzzball operator controller manager uh persistent volume claim has been created and is bound to a location that's bound to this volume that was generated on its behalf this just one step along the way showing that the operator installation has been successful it has a location to storeage State gotcha

gotcha thank you Jonathan so the next command I'm going to run is uh C control get crd I think I believe crd stands for custom resource definition uh and basically the fuzzball orchestrate platform is this custom resource definition uh that we're going to be deploying so to just get that uh that's just another smoke test there and then lastly we can see um the pods that are running the fuzzball um operator and as you can see we have three instantiations of this that are all currently running yeah and I was actually corrected on that that the three out of three is not um this isn't

like three replicas of it there are three containers that make up the operator controller manager that are all involved in this process so um it's not quite three instances but we do see that the three containers that are expected are running gotcha gotcha thank you Jonathan uh all right so now that we have uh the operator installed uh we can begin installing um the uh fuzzball orchestrate platform so to do that I'm first going to set a another few environment variables uh for to configure fuzzball orchestrate so uh ip1 here I believe is the IP of the load balancer that is used by kubernetes in

this case uh since we're deploying on Prem uh the cluster uses metal lb uh the NFS server uh an NF NFS server is defined and this NFS server is basically used to store uh the substrate configuration uh so that the substrate node uh can basically pull the basic the sub Dr mounts this NFS server as well and can basically pull the configuration from there so it knows uh what orchestrate cluster to actually talk to and uh the realm ID is basically a uu ID of um a keycloak realm I believe which is used for authentication and uh used to kind of organize sets of users

uh within this um fuzzball Orchestra deployment um Jan did did I get most of that right yeah that's all fine uh the the realm ID here is a a universal unique ID a uu ID that um this deployment as I I I know how we're going to do it but uh is going to generate a realm in a keycloak that will be stood up on our behalf using this U ID um alternatively if you have an existing keycloak um it can be configured differently to access an existing keycloak realm and so then you would plug that value here and a cat is also necessary oh

you don't see this I'm sorry I I'm looking at myself on camera but a cat just jumped up into the the frame but um yeah so all great um this is going to do a totally normal deployment oh maybe one thing I'll say is that this IP address is the one that we want to use to export the fuzzball external services this is the IP 1 so as a prerequisite to this we expect the local kubernetes environment to have a metal lb uh load balancer deployed with a pool of addresses and to make it repeatable we're saying which address from that pool we want to

use and and we happen to know because we defined the pool that 1099 is one of one such available address cool cool thank you Jonathan um so uh now that I have these variables set I'm basically going to create a configuration for um the fuzzball orchestrate um platform which is uh in this yl definition that I'm going to paste so and this is basically going to take some of the variables I've set up here and kind of throw them into this configuration file uh called fuzzball Doo so as you can see uh just pasted that into there so uh and this basically just takes some

of the various uh variables I've set in the environment and uh sets them in this uh fuzzball config so now that I have a configuration uh I am ready to start deploying um the fuzzball orchestrate platform and to do that I'm going to run Coupe CTL apply and and pass this fuzzball yo file in and uh as you can see um I have a deployment started and to kind of watch um the various um to watch it deploy you can basically run this command here which is outlined in our documentation to kind of just watch the logs of this entire platform kind of getting stood

up so yeah uh so um while we wait for that uh I can switch over to our compute node and we can kind and get that installed uh so uh to install fuzzball substrate let just make this quick cool uh oh before we do that Jonathan is there anything else you'd like to add uh the one thing that I think might be cool is to demonstrate like that object that we just applied into the cluster it's resident in the cluster so you could like get it you can see its definition and the coolest thing to me is that you can just edit it in place

and we shouldn't change anything about it during the deployment but you can run like Cube C edit fuzzball I think fuzzball uh um orchestrate is what we called it um and just edit it in place and then the next time the operator wakes up it sees that that has happened and makes the change in the cluster which is a good experience in my in my experience gotcha gotcha so uh and just to show some of that uh we can just do something like get pods so it's not pods it's literally a fuzzball not pods oh yeah so you're getting the fuzzball called fuzzball orchestrate fball

fuzzball orchestr so you need an additional fuzzball between get and because you're getting an object of type fuzzball uh named fuzzball orchestrate capital fuz I think it doesn't matter what you have there should be fine yeah so there we have one and instead of Ed instead of get you can say edit um which just puts the yaml definition up in an editor and you can just change it here so like if you scroll down a bit um you can see the like you've got some metadata up here that makes it harder to see but there's the the metadata that we had specified in that fuzz. yaml so you can either keep that Fuzzball.

yaml and you know put it in get or something like that and then if you want to make changes reapply it to your cluster the same way that's one thing you can do or if you just want to man manage it stateful you can just make changes right here and they update the configuration of the cluster and that that's the real value of the the operator is that it's it's really Dynamic and it's responsive to the configuration that you apply here gotcha thank you Jonathan all right so uh so uh for so now that this uh is deploying uh we can just go over to

our compute node real quick and uh get that stood up and go through the install process of that so uh to do that first I'm just going to configure some firewall settings just to uh and this will basically Ure that um so this will basically put the the inter the internal interface in the right firewall Zone and uh make sure that our nodes are communicating correctly uh with each other uh next I will start by installing ciq mountain and enrolling it uh and enrolling this node into mountain and then installing uh fuzzball substrate onto it so to do that just go paste these commands real

quick so just installing the mountain tool right here and then enrolling my server uh using the access key and then from there I'm going to enable the uh fuzzball product fuzzball orchestrate and then afterwards it's a simple dnf install to install fuzzball substrate and while this is going maybe one uh one little Nuance here because that fuzzball substrate orchestrate might seem odd to people so there is a separate fuzzball substrate package that just installs fuzzball substrate it's you can see it here in the dependency list fuzzball substrate orchestrate is the plugin that allows fuzzball substrate to connect to and receive jobs from fuzzball orchestrate so

installing that installs the whole stack gotcha um and this is basically just one way of getting um substrate running on a node um you know if you're running like let's say a Warewulf cluster you could probably create like something like a substrate node image which you could deploy to uh your compute nodes so you wouldn't have to do it by hand like you say we could you could we have of course and we we intend to provide that as a thing that if you're using both Warewulf and fuzzball uh you should be able to get from us and or at least a base image a

starting point that we could help you configure them you beat me to it I was going to say I feel like somebody on this webinar has done that before maybe a few times cool so uh after uh substrate is installed uh I'm just going to set some uh environment variables for that service right here let me just uh let me just paste this just going do this and then and this will basically tell uh fuzzball substrate uh where to get its um config from and and this is basically um the NFS uh server which um orchestrate is publishing the configuration to for substrate to

pick up so uh once I have that Set uh I can just go ahead and CTL startall substrate like so and then just hit it with a quick status and we can see that this is running so uh uh it takes a little bit for um the orchestrate platform to uh fully get fully get deployed and up and running so we're going to change gears a little bit I'm going to switch my screen share over to a deployed environment and we can show you what that kind of looks like once it's done deploying so let me switch over to this guy here cool so I

am on a controller node uh that has uh fuzzball orchestrate um fully deployed on on this uh on this node um and to show you uh kind of what that looks like on the tail end of the logs uh you should see something like this uh at the end of your logs when it's fully um deployed and to show you that um we have all of our we don't have any uh PLS that are in an error State uh or in some weird State and they're all running or succeeded uh we can just run that command as just a smoke test and from there uh

we can run this command here which calls the fuzzball admin pod and calls fuzzball admin and it basically checks uh the compute resources uh which are connected to this fuzzball orchestrate deployment and as you can see here I have this compute node uh that is talking to um my orchestrate um uh deployment so uh this node has um a the fuzzball CLI installed on it um the fuzzball CLI is also installed through mountain and uh once you have your orchestrate deployment up and running you can basically uh create a fuzzball context and I can show you here that I have one already created uh so

uh once your fuzzball context is created you can log into it so I can you can do that by running fuzzball context login direct and I'm just going to pass in so just throw in my credentials there and as you can see uh I'm logged in as this admin account and uh and I can start uh submitting jobs so uh as you can see here I have uh two fuzz files uh basically uh created and let's first take a look at this hello world fuzz file uh as you can see with this Hello World um fuzz file it's a workflow of a single job uh

named hello this job uses um an Alpine container and uh as it's uh I guess software environment and within this uh environment we are basically calling a shell and echoing uh hello world in the shell and uh the resource requirements for this particular job is a single core and uh and one gig of memory so uh now that we've seen this um workflow let's go ahead and submit it so so to do that I'm just going to call fuzzball workflow start hello world.

fuzz and I should get a uyu ID which is returned and from there using this uu ID I could um check the status uh of this particular workflow using fuzzball workflow describe and passing in this uu ID and as you can see um this workflow has finished since it it since it's just printing hello world uh you can see the status of the various stages like the image pull and the job and the job actually running when it started when it finished uh additionally uh using the CLI you could do you could also gather logs from uh this particular workflow uh by pointing it to

a specific job which you want to check the logs of to do that we can just do Fuzzball workflow log pass in um The UU ID and specify the job name uh which you'd like to uh look at and from there we can see the log this particular job has printed hello world which we saw in the fuzz file before um to do something slightly more interesting we can take a look at this printer um fuzz file that I have also on here uh this uh fuzz this workflow is um a single job as well uses uh name this jobs name printer the image that

it's using is also an Alpine uh container and basically what this job does is it prints uh integers 1 to 300 with a 1C sleep in between so and resource requirements for this also one core one gig and um one additional thing uh that we have in this particular workflow is a policy which you can see here and this is basically setting a uh wall time for this particular job of 10 minutes so I'm just going to go ahead and submit that so fuzzball workflow start and I'm just going to pass in printer boom okay so as we can see um we have a uu

ID which uh fuzzball return to us from here uh we can do something like workflow log pass in the uuid and pass in the job and as you can see it's basically printing these integers uh the last thing I'd like to show uh on this demo is give me one second let me just go ahead and refresh this cool so uh let me switch my screen share and switch to we do have one question while you're doing that Jonathan you may want to take a stab at it yeah so the question on LinkedIn was was can the config be sourced from anywhere else other than

NFS uh with an example of like an S3 endpoint or other type of share so the use of NFS here is optional um you could get the configuration like it's just generating config file actually a couple config files that could be placed anywhere but there's not currently any support for publishing the config from orchestrate directly to an S3 endpoint uh or pulling it directly from an S3 endpoint 2 substrate you'd have to manage that separately you could take the config that orchestrate is publishing that it's generating and your yourself publish it to S3 and then as part of your compute node bring up pull that

down to a local file and read that in from uh with substrate when it comes up so that could be done um and I I think that would be relatively straightforward if it were uh like if a requirement like that came to us um the biggest issue would just be handling the credentialing for it I would think but um it' probably be possible to do something like set a well there are secrets in it so you would want a credential for reading it as well but yeah plausible um possible to do right now with with some integration work um but also doesn't seem as I

sit here like it would be a huge ask if it were necessary to do be more integrated into the product cool so uh I guess as you can see here I have pulled up uh the uh fuzzball web UI which is uh I guess connected to the fuzzball orchestrate uh uh deployment which I was submitting jobs to as you can see here uh my printer.

fuzz file which I submitted previously is currently running and if I wanted to uh look into this particular workflow I could click into that and if I wanted to view JW uh logs uh from the printer job I can just simply click on the printer job click on logs and as you can see here we can kind of see that it's printing uh an integer um and increasing after every second you should click on that terminal tab too oh yes and uh I can also get a shell on this particular um on this particular job inside the container so I can do something like M

you see I'm inside a cluz ball job I could run commands in this uh hello uh you can uh touch test maybe just do like a PS in there too so we can see that the job is running in it yeah yep so yeah so running a PS we can see that uh my job command from the fuzz file is currently running uh in this in this uh container yeah cool cool well that is uh all I had uh prepared for today's demo uh if there are any if there's anything the audience would like to see yeah feel free to drop that in the comments

uh your use cases uh all that type of stuff we are definitely interested in what our audience would like to run using FL ball try to see if we have any more questions thank you for that Brian that's fantastic yeah Jonathan left and he's back and he's back no I appreciate the the demo it's been good to see it and I'm glad to see all the changes that are going on it's been interesting having people come in and ask like how does this actually what is this part of uh kubernetes like how do they play together what does it mean so thank you for explaining

that Jonathan you and Brian that's very helpful yeah one thing that might be we've said this on the webinar before but just to be clear for people who might not be familiar with fuzzball in the different container platform that we're dealing with here um the the big thing to take away from that from an HPC standpoint from a performance Computing uh standpoint or performance sensitive standpoint is that we we use kubernetes and the kind of traditional oci ecosystem as our application platform but it's not where the performance sensitive part of your workflow runs the performance sensitive part is still containerized but it's containerized with technology

that is more akin to Singularity or Apptainer that's designed for execution in an HPC environment and doesn't have some of the same uh performance characteristics of traditional oci where oci is really built for resiliency and reliability for services of web services especially uh Apptainer and now fuzzball substrate are designed around um providing a container experience but for an HPC use case um not having as many layers between uh various parts of the network stack for example is a big part of it thank you Jonathan well if we don't have any more questions I really appreciate you guys jumping on and doing this and look forward

to see more as these workloads workflows come out and I know there are other things that you guys are working on with other software vendors and being able to run those I don't if you want to hit on those real quick we've got some exciting stuff coming up with uh with like running larger things that have been difficult to containerize in the past I don't want to like make too many promises for how quickly we'll be able to show them we've done them in the past but we've made some changes or we we've done them internally uh but we've made some changes to how user

IDs get mapped into containers that need a little bit more baking time for how that affects storage and accessing storage from within the container so watch the space hopefully in the next few weeks we'll be able to show something more publicly on uh running some larger isv type applications that uh that do a little bit more than say hello world to your screen not that you can't do more things now but they're just not any more flashy than that a lot of the time sure sure great yes yes flashy you have any closing thoughts to come yes yes you've been doing all the demo work

just been sitting here being quiet enjoying what I'm watching so anything to close I guess other things that we're we're kind of working on right now uh things like running services within uh fuzzball so in particular like one use one use case that comes to mind is you're training some AI model you want to deploy this model as a service and then run inference against this service uh to uh I guess validate your particular model uh so that is coming down the pipe and then also uh possibly running uh fuzzball on the cloud as well with you on various uh Cloud providers so uh look

out for that yeah very good well thank you guys thank you for joining today thank you everybody for watching we appreciate it and again stay tuned subscribe like love to have you guys come back and see what else these guys have been up to and what they're working on so really appreciate you we'll see you again next week awesome thanks see you later bye