Fuzzball: Latest Updates and Workflows

which does actually affect our life so that's why we talk about it because there it is all around us um we are going to be talking about fuzzball um a little bit today and I believe Forest that you have a little something to show us we want to jump right into that part I like to see it so I don't know I'm always like jump right into seeing it but you want to give us a little um a little update well yes absolutely um so today uh we're here to discuss just a little bit of kind of an update about fuzzball kind of you know

month or so Post Release uh you know as everyone knows ciq released both mountain and fuzzball two of our big Flagship products last month um Mountain kind of our big solution around Rocky Linux for bringing patching and uh you know packages package management all that type of thing out to that um kind of in an Enterprise format and then fuzzball of course our big Flagship platform around high performance Computing that uh you know is kind of our new way of combining the best of Enterprise Computing with the best of high performance Computing um as I said it's been about a month since fuzzball's been out

uh and we're here today to kind of give a little bit of kind of a review of what's going on with buzz ball since then um which we'll get to uh I don't have anything too specific to show on um you know the GUI or Etc like that today unfortunately uh I was tinkering with something but I didn't quite get it working in time uh so maybe in a future release we'll be able to show up but we'll certainly talk about it a little bit um it's kind of yes was it AI based a little bit yeah it's kind of something that um we'll just

go ahead and discuss them a little bit I guess a capability this is something that kind of came out of one of our pocs so part of the reason why I don't specifically happen to show is because it uh it's still a little bit wrapped up in one of those so it's you know I would have to kind of take it apart be constructed a little bit to you know you know you'd all have to sign an NDA otherwise um but essentially uh one big thing that we've kind of been working on lately uh kind of in my spare things as far as workflows go

uh is bringing as I said capabilities around AI to fuzzball we've already done um quite a bit on this channel uh with fuzzball and AI you know last summer I think about a year ago we were showing Dolly mini and that type of thing um since then we've run stable diffusion uh we've done a lot with uh just in general kind of the pie torch tensorflow sphere that type of thing um but one big thing that we've kind of worked on recently uh is hyper parameter tuning so in case you don't know hyper parameters are something that kind of go into AI training it's different

things like uh but or could really reasonably explain here um but you know batch sizes um different aspects of how many um result uh kind of bends basically a model has different aspects of kind of How It's trained how data is kind of divided up and delineated out to the different job compute host that you're using as a part of your AI training hyper parameter tuning is commonly done before you're doing like a big run of AI training so you can know that you're actually doing that training as optimal as possible it's kind of like a meta-analysis of your training setup before you actually you

know commit tens of thousands of dollars or whatever to go run for a week or two weeks or a month on training your AI um so one big platform that a lot of places use for this is called weights and biases um weights and biases is a big um like kind of ml Ops platform that allows you to streamline a lot of different operations around AI uh as I said hyper parameter tuning is one of them um so you can create sweeps there where you essentially give it your code uh give it you know the parameters that you want to optimize for that type of

thing uh and then given a certain bunch of compute sources you can actually have it run and test different hyper parameter combinations to determine what's going to work best um as I said that's kind of something that's a little bit locked in a POC at the moment um but we've been using the fuzzball architecture to do things like uh take with just an API key stored in our fuzzball cluster um as a secret we've discussed previously um fuzzball's ability to for example in the cloud have credentials and other bits of information stored inside of it so that it can be templated out when a workflow

reaches it for example so users don't have to store their own credentials in it uh so you don't have to store their credentials and workflows they can just store it within the like vaults on the server basically um so you can store an API key and essentially given just your link that you would normally use to run this weights and biases agent command um you can go ahead and essentially like a link to your your sweep when you when you create a sweep on the site it gives you a link that you can then use and run sweep agents against uh and you can send

the sweep agent's scale embarrassingly parallel uh in an embarrassingly parallel manner so as you run more and more sweep agents um you essentially can just run more parameter sweeps at once um and so we've been running testing with fuzzball's task array interface where we can essentially you know run one type of task across a whole bunch of different instances at once you know provide uh kind of embarrassingly parallels a lot as a um it's kind of generalized HPC architecture where you know you've got 20 000 input files you need to do the same little bit of data analysis or data munging or whatever on them

on all those files uh so you you know take your 100 compute nodes and then just run sequentially um those 20 000 files throughout so in this case we use the ability of fuzzball to have concurrent instances running in a task array um to actually run multiple agents at once and be able to control those and be able to see actually out of the weights and biases site these instances being run through fuzzball out on uh in the example we're working with the AWS Cloud actually on the weights and biases site as compute instances um this is kind of similar to uh we never got

to show it much because it got obsoleted but this is similar to how like crypto mining for example worked when we were messing with this on fuzzball um we're a third party server could uh delineate work out to a workflow um in an embarrassingly parallel manner um in this case weights and biases serves as kind of a third party server that sends out work to all these embarrassingly parallel instances uh and you're able to do your you're able to essentially scale out your hyper parameter sweeps as arbitrarily as you'd like um so that's interesting like I said I tried to get a kind of um

uh public version of that together but I wasn't quite able to uh but that's something very interesting that we've worked on lately I know that's kind of a big capacity um that uh you know I know a lot of places are doing a lot of hyper parameter sweeps and are expending compute resources on that um so fuzzball definitely makes that easy to uh kind of delineate um this um there was a blog post recently that we put out uh or the I put together that was about how to do these hyper parameter sweeps using obtainer and the natural extension of that once I had a

container was to take it to fuzzball and see if it would run there and as I said with fuzzball's task array interface it was really simple to just take the same command I've been running out on these instances when I was testing it myself and be able to scale that out through fuzzball to multiple instances so so that's uh kind of the biggest um most interesting kind of workflow type update that I have at the moment uh and that's uh kind of some of the capabilities that we've been looking at with buzzball we've got um of course Lots on our roadmap um we're always uh

working and iterating on all of our objectives in light and uh you know relative to each other um but fuzzball remains a uh a topic of great interest at ciq and one that uh me and uh my colleagues here on the fa team especially are uh are always eager to be exploring new use cases new workflows Etc and so um yeah so that's yeah that's kind of one of the most interesting things we've been looking at recently um like I said from especially in the AI space that's kind of a big thing um you guys have been working quite a bit on containerizing things and

like you said you've containerize things and try them out in fuzzball see what happens I know I hear you guys talking a lot about things like open phone pair of view uh open radio could you talk a little bit about where those are in terms of containerization and then kind of what those are used for like what disciplines of science would use those yeah so we have a number of different applications that we've run on fuzzball um you know informally we've tested these things we've kind of seen them run uh we're working on you know the concepts and more formally uh you know testing those

and being able to verify that things work on fuzzball um but for now we have quite a few applications that we've tested built out use cases for built-in workflows around um like I said the open foam open radio so what was the other one that you said Zane um yeah so those applications uh we all have running on fuzzball uh my colleague Brian Phan has done quite a bit of work in the cfd space on open foam and bringing that to us um so we've got uh the capability to do batch Computing around uh you know simulations in that space we've got our motorbike example

for example uh that I'm sure many people in that space have seen running on fuzzball um and then we can take that and within you know one workflow or even a single workflow be able to run the entire um you know the two containers that it takes to do the open foam simulation and then a separate paraview container that generates a visualization off of that um so end to end with open phone for example we can uh you know ingest what's necessary um so I'll admit I'm not 100 sure right off the top of my head if there is data that gets ingested into that

right off the start I think there is um you can adjust whatever data would be necessary for that simulation it runs the whole open foam uh kind of aerodynamics cfd simulation on it um and then like I said we can take another container and uh organize Power view to generate a visualization of that all within one workflow so yeah that's really cool I know that's quite an uh an application of industrial interest in quite a different or a few different places um I'll admit right off the top of my head I'm not as familiar perhaps as I should be with how the open radios workflow

goes um but my colleague yoshi-san and yoshiakisanda over in Japan has done quite a lot of work on that um I believe we've got a decent amount of blog posts and stuff like that that kind of goes into Rocky Linux and open radios I think we just launched out on um I think the Oracle cloud or at least one of the clouds out there in image that you can go spin up right now that's got rocky Linux an open radio spilled into it so you can start iterating on that immediately um is the Oracle Cloud very cool yeah so that's out there um you can

go like I said work with open radius on a rocky leg space today by just spinning that up there um but on fuzzball uh like I said I I'm perhaps not as um terribly familiar with exactly how that workflow goes as they should be um but as far as I've seen that runs on puzzball um we're able to take the same similar types of examples that we've run um kind of I believe in his blog posts and things like that I'm going to put them into yaml's uh run those tested them around um as I said that's another application of large industrial interest um and

so with that kind of entering the open source sphere after Altair released that um yeah as we've seen being able to run that on fuzzball and do all those same types of things that you've you know seen them do in his blog posts Etc um is very significant within uh the husband system so and I believe that David godlove re or took Yoshi's work and actually did it on a webinar so we actually got to see a car crash I'm not mistaken we can actually link to I think we can yeah I believe we did that I think I remember that not too long ago

wait we've done we've done so many of these by now that sometimes Dave godla was able to recreate it so we can put that link in there we can go watch that if you're interested in seeing it in action again thank you Yoshi and Dave for doing that yeah um so yeah we've like I said uh between Brian Yoshi we've got quite a lot in uh you know CFM where we've covered some interesting applications in cfd and um kind of finite element analysis here there um so yeah like I said those are two big industrial applications that are used in a lot of different places

um there of uh yeah a lot of interest to people so yeah no I just feel like I gotta pop in and say sorry forest for putting you on the spot for a demo I thought I thought it was gonna uh live surprise demos those are the best come on a surprise like can't you just do it live just like that's awesome I wish I had a you know my magic cop to pull something out of but like I said I I did try to get that one um one privatized but ultimately the uh the underlying code base uh is difficult to not reveal uh

and I haven't had a chance to kind of get together a little bit higher parameter tuning code of my own but at some point you know we will uh we'll definitely show that off there are um people in the private sphere that we're very interested to see that come out and be in action so yeah I feel like we should really make this interesting and give him a deadline oh add it to his jira board we have access to that can we go in there yeah I love that perfect oh man yeah well we've got um you know not only that we've uh you know

we kind of touched on a little bit there's quite a lot to be done with AI on fuzzball as well um just to kind of you know maybe go into a little bit of uh of a review of the applications and type of things that we've done uh on fuzzball so far um as I said AI has been something that we've been working on quite a bit kind of in general not only within fuzzball but in the Rocky link sphere um in the past year or so we've been working really hard to kind of create a basis of practice around AI on Rocky Linux um

by bringing uh you know big Frameworks and stuff like that to it um as I've noted uh in a number of different times we've run models like stable diffusion and things like that on Rocky Linux and then taking those directly to fuzzball and been able to run those there at this point stable diffusion is like one of my standard demos uh that I use it quite a few different things because it uh you know being just a pytor uh sorry being ultimately just a pi torch based AI model um pytorch runs great on Rocky Linux it installs just fine and so there's not really any

issue getting that up and running on gpus up and running on um that type of architecture and then especially Translating that to fuzzball um so as far as pytorch goes we've done quite a bit of sophistication on buzzball um just by virtue of the fact that a lot of the major um kind of interesting you know fun to demo use cases are uh are built on that you know I arrived at SC and I saw a couple other folks um you're also running stable diffusion on their architectures um so that's been interesting tensorflow uh We've also run on buzz ball we've got you know the

classic feminist example that type of thing um we haven't uh Dove too deep into um running uh some of the massively tensorflow based models out there uh just because um we haven't run into those use cases yet uh but yeah we've run tensorflow we've run examples like that as said we've uh um kind of in the sphere of more of Rocky Linux uh we're working at the moment on supporting the rocky Linux special interest group around AI um and so just here in the last couple of months or so that was founded um and myself and a couple of others are kind of um interested

in seeing where that goes kind of helping to support you know as I said I kind of bringing some of the practice of AI into Rocky Linux and giving people that you know here's the the actual guide that shows there are people that are you know working on this out there so um tensorflow and that type of thing um we've tinkered around with uh and as I said we've got definitely an example there that we've worked on with uh uh fuzzball other things in AI as I noted um let's see here I got a question for you where you're looking for us what's that that

special interest group that Sig is that something we should post a link for so yeah absolutely find it very quickly if we don't mind doing that yeah we'd love to have more people um coming on in there and contributing to as I said helping them make Rocky Linux the best environment for AI that it can be I think that could be a really cool place to spend some time we have meetings uh every Thursday I think our next one is next week uh yes next one is next week at 9 30.

uh Pacific am [Music] so yeah feel free to come and join us they've uh they've been a little quiet for a second uh see how we'd love to get um more people on and have more people involved there excellent but yeah so tensorflow Tinker breath um we've got things like GPT that we've worked on a little bit um as far as uh you know Jax things like that um I couldn't point to uh a complete workflow but mostly because a lot of my tinkering has been kind of just uh you know working with GPT in General on Rocky Linux um and just kind of as

I said the Jax framework that underlies some of that um kind of in the context of tpus that type of thing uh yeah but we're working on um kind of figuring out what's sticking around and kind of the open source standard of AI space uh you know of course we're real big on you know open source AI with fuzzball um that is kind of a very fast-moving space especially around large language models at the moment um so we're kind of you know seeing what sticks around seeing what um you know ends up being uh kind of the major use cases there um but you know

as uh we've already built out large scale models with pytorch and that type of thing so um as far as gbt inference all that that's what we're heading for um so yeah AI uh is proven in a lot of different ways out on fuzzball um and just kind of in General on Rocky Linux at this point uh so yeah AI Frameworks models those are always fun we have a other applications that you know we've run on fuzzball um I think that we've um tinkered with quite a bit on this webinar uh for example in bioscience we have quite a few uh different experts at ciq

kind of in that sphere um as HPC and bioscience tend to you know go hand in hand in many cases uh so for example um we've worked on grow Max which is the big molecular science our molecular Dynamics application that a lot of places have or a lot of places are probably familiar with um that gets used in Pharmacology that gets used in private academic research that gets used in biological research of all different types uh there's an interesting publication out there uh from AWS that kind of goes over price performance ratio with grow Max on AWS when you use their elastic fabric adapter um

and so something interesting that I was working on kind of earlier in this year you know my colleagues and I uh proving out um and gradually we've been working on this for quite some time but just always you know benchmarking and things like that I'm really sure I was using uh doing some benchmarking with the elastic fabric adapter support the fuzzball has when you're running an MPI on buzzball it with the AWS provider that we have at the moment um you're able to use EFA placement groups to make sure that uh all of your instances are getting a place in like cluster placement groups that

type of thing so they can communicate most effectively uh and looking at you know for example numbers that AWS has published for their performance that they're getting on instances um it's nearly identical to what we got at a fuzzball when we run basically that same test uh informally um but we're essentially you know we're getting very very close to the same nanoseconds per day performance that AWS gets out of that um to the point where uh with a little bit more tuning on the gromac side of things to kind of uh you know optimize how that's compiled on a container level um it would be

yeah it's already very nearly identical so there's not you know like I said most of the distance is just you know in kind of taking the time to compile things versus that specific architecture but um when we run this like I said versus aws's numbers we get um almost exactly the same performance that they report um and so fuzzball you know as far as being able to replicate that type of thing and the performance that people are used to with the EFA and stuff like that um for these large-scale Cloud batch Computing jobs um you know there's always testing to come but uh yeah for

from what I've seen at the moment you know just to give drum access an example um of an EFA enabled application that we've spent some time you know compiling with all that support uh and that we you know like to potentially be able to make available to people eventually um the performance that we've seen is almost exactly what uh we expect based on those type of Publications that are out there I'm further in bioscience um what about star yeah there's um yeah there's a whole host outside of grow Max of other things that we do in uh in bioscience um like genetic sequencing that type

of stuff um so being able to take you know the genes that are present in a sample match them versus a genome and figure out what uh those genes are this is I mean this is an incredibly common task in any you know genomics lab uh anywhere in the world and it's mostly high performance computing power task and so uh you know we've built out workflows around really common bio apps like Sam tools HTC uh star um those type of platforms uh that I'm sure many folks will be familiar with uh I can you know as with many things in HPC I you know I

casually know of these things but can't count myself as you know one of the very valued domain science experts that we have in them um but I know that there's a lot of use around star and those type of things out there and um those were some of the first I actually think the star workflow come to think of it the Sam tools HTC star workflow was the first workflow I built at ciq um that we converted from a GitHub repo that had a lot of python code and things like that but did like S3 transfers and data movements and a lot of things that

I was able to strip out and just kind of leave to fuzzball in the end uh and it seems like yesterday and it was almost two years ago yeah I know right absolutely crazy oh you're still going you're still going I I was gonna jump in here what's up Bros well so I get a lot of um calls and emails and requests from universities and there's several universities that we're working with now and lots of them that have been like coming in and asking questions I literally got one this morning uh from a university and they said fuzzball looks cool cool yeah absolutely let's talk

about it let's you know set up a meeting we'll do actual demo we'll figure out what it is that you need so talk to me about the academic world and where fuzzball kind of fits in there what we've been doing with these universities yeah so the academic world is very near and dear to my heart because that's how I got into HBC and that's kind of how I got specifically into kind of the concepts that we have uh around fuzzball and HPC um as we kind of noted I think at one of our past webinars I uh came from the academic HPC environment um was

there for about two years or two and a half years or so um and while I was there heard quite a bit from researchers that they wanted Solutions around automation like buzzball they were exploring you know other workflow engines they were exploring other platforms um but being you know kind of one of the overworked and overtaxed admins on this cluster already um you know those Solutions take a lot of time to put together those Solutions take a lot of time to investigate um that was you know kind of a whole different sphere of things that we were able to dip into a little bit but

we were never able to get to kind of the level of complete automation that we wanted to reach for our researchers so I was always looking for a better solution when I saw fuzzball I was like oh great this is very similar to what I you know I've been thinking would be um you know kind of the next phase here I think Rose I we alluded in a past webinar to my you know diagrammed out you know kind of simplistic concept of having you know all your containers on this side you know def files over here um and kind of this uh SSH based login

thing that would like drop a user into a container that they then couldn't like knock out the rest of the system from you know with like running on head nodes that type of thing because we were we didn't have a login node at that point so um you know anyway yeah so there was a lot of you know kind of interest and excitement that me and my colleagues in high performance Computing you know around me had for um kind of that whole sphere of Automation and uh so I think fuzzball a lot of folks in Academia will be very excited for that type of thing

um just in general the solutions and just the Simplicity that you can reach with the under with just the underlying technology of fuzzball like for example containers when I was working with containers in academic HPC instead of having to go set up a whole build environment you know in the Clusters area CH you know own the whole file system to my or the you know the new files to myself so I didn't you know accidentally knock out the rest of the applications folder build things you know scrub the area or uh you know scrub the file system there when the if a builds failed um

being able to you know take just a delineated instruction set of how to build an application in a container build that make it available in a shared space on the cluster uh for users to be able to access was such a game changer for kind of how as an admin I could do my work that I almost never wanted to go back to just like building applications manually I'm like oh my gosh I can use my workstation or wherever a spare compute node or whatever um or sometimes that was a little bit difficult because of how those were set up but I could use my

workstation or whatever to go build one of these containers upload it you know I can build it exactly like how I want I have my whole you know build script basically right there that uh will take you know this container from nothing to my application you know essentially automatically as long as I keep it updated um and once I have it like I said I was using Singularity which became obtainer so it's a single flat file on the system that you know I can just make available in a scratch space or something like that um yeah the technology that underlies it is fantastic uh and

I think when you combine that with the ability to do away with some of the um practices that really hold academic HPC back as far as management goes of clusters and kind of some of these technologies that um were not built with some of the technologies that we want to integrate in mind from the start if you can you know kind of you know ultimately with our vision for fuzzball if you can take that and replace that whole architecture with something that's much easier to manage um it's like as I said it's a complete game changer for academic HPC and kind of how that whole

sphere of things works um so y'all very excited yeah it's very exciting you know the implications as I said proposal for academic HPC um we've built out a number of applications kind of in that space some of the first open source applications I started looking at on fuzzball were things that I was familiar with from academics uh so like for example gromax was one that I started into lamps I knew it was a big use case um Quantum espresso qmc pack I don't think those have seen the light of day on the webinar for some time but I believe at one point in the past

we did show them off um at the very least those are things we've tinkered with um some of these applications are things like I said we're uh we're working in certain ways to you know bring uh to our customers and things like that uh but yeah academic research we also while it doesn't generally uh get used out in Industry um some people are interested in kind of working with uh what are called pgas networks um which are kind of a different way of doing multi-node parallelism that's separate from MPI where instead of having cores passing information over a network between each other you have one

basically big Global shared memory space that everything puts memory in and out of um and it kind of you know is more like the shared memory model of uh you know kind of communication than a network-based one uh and so if you're working with the chapel language we have specifically a multi-node runner that can um take chapel and gas net and wire that up so that you can uh run that on your system so that's very cool um so cool for us honestly you guys can see why one of my favorites you know it's like almost not an expert on that but let me just

you know bust out my Encyclopedia of knowledge about this question that you asked for so we could probably talk for the next like three hours just about what's going on and where things have been and where they are right now do you want to maybe just like as we're wrapping up here and then we're going to take some questions guys so if you have questions please pop them into wherever you're at we've got people out there looking and they'll pop them in here and we will answer all of your questions because we are live and we love to do this live um so wrapping up

here can you just are you at Liberty to tell us anything of like what we can expect what oh he's still there don't worry what's what's in the future for fuzzball because it's not a complete product I imagine that it will never be complete because we're always iterating on on what our customers are asking for what the community is asking for and you know we're constantly evolving but can you hint a couple things that are coming I believe I can a little bit uh within reason um really quickly uh yeah just really quickly one other interesting thing we've been working on recently are like benchmarking

tweets within uh fuzzball um hpl for Onyx test Suite uh moving on for Onyx head but veronic benchmarking Suite sorry um and then in the past the far past we did hplai and we're able to get once again matching performance numbers uh based on that Benchmark on some uh systems that we were some large-scale GPU uh and AI systems we were doing those on um so while those aren't immediately useful for people uh you know I'm sure everyone interested in you know still running that you know standard hpl lintac burn-in uh you can still do that same thing with fuzzball and um and get those

flops numbers out of it so uh yeah what's coming up next um two big things kind of uh in my space of things that I know we are kind of working on um in the workflow sphere of things uh we'd like to be able to bring um some popular Ides to fuzzball um so we're kind of working a little bit on uh tinkering around with bringing things like vs code Insight systems some workflows like that into something that fuzzball can do we'd like to be able to you know perhaps uh should be remote develop remote GPU profile etc those type of things it would be

very useful if somebody could take their AI code run a simple profiling workflow against it and then be able to get you know the profiling reports and everything like that uploaded out to S3 automatically once that's done so we're working especially kind of on um you know my team the essay side of things around stuff like that I know we've looked at uh for example um vnc's and I don't quite remember where our our exact uh where we've left that off at we're looking at bringing capabilities around R into fuzzball we'd like to once again another thing that I'm not I don't quite remember the

capabilities are off the top of my head of um but that we've worked on with our studio server and things like that um being able to um you know run like our development things like that remotely from uh fuzzwell workflow uh these all kind of in the same vein generally as our Jupiter notebook workflow that we've seen on this which I should know also works on fuzzball and the interactive sphere much of what I've discussed today has been batch Computing based workflows but we also have uh you know interactive capabilities and fuzzball around port forwarding uh and so um a lot of kind of the

interests that we have in workflows at the moment is around seeing how those capabilities can build out and like I said bringing more uh kind of benchmarking capabilities to uh to fuzzball around especially Ai and um kind of being able to burn in those systems and that type of thing I think we do have one question oh wait question time oh wait there we go questions oh is this backwards um all right all right that was good it was good so hyper parameter space exploration can get expensive and time and resources for us how do you forecast and condition it so that you get a

good balance of enough versus not enough well I think that comes down to knowing uh especially like for example from the fuzzball perspective what uh compute resources and that type of thing you're running on and ultimately the underlying cost of use for those um is this something where you can run these uh on-prem architecture or you can run your on-prem infrastructure um and it's not going to you know amass Cloud costs maybe as fast as using Cloud gpus are um in general uh being able to you know tunably scale kind of with how fuzzball makes that possible how many resources you're dedicating to it uh

being able to keep track of how long a workflow is run for that type of thing uh is very useful while it's you know kind of more so on our fuzzball roadmap metrics and tracking around that type of thing is something that uh we're working on um kind of putting together planning and you know code bases and stuff like that around so you can have visibility into costing and things like that easier uh wired into that um so while I you know once again I am only an amateur AI research scientist uh so I would have to say that I couldn't um specifically say how

you would forecast and condition it you know on a technical level um but like I said with regards to kind of how fuzzball and you know what we're building out with containers and that type of thing um it definitely gives you the ability to easily scale up scale down and kind of tune uh relative to what capabilities you want your users to have uh how quickly and how much that is going to end up costing you in the end amazing Forrest thank you so much I don't think that we have any more questions particularly for today for Forest about fuzzball but if you guys have

anything you're watching later and you want you want to know something just go to our website submit a little form you can type it in here in the comments below there's a million different ways to get a hold of us we want to talk to you we want to meet with you we want to chat with you about what's going on in your environment and how we can help and dive into details about how this particular product called fuzzball can help and support your amazing work that you're doing wherever you are so make sure that you give a like you give a subscribe we'll be

back here same time same place next week we love you very much have a great day thank you everyone thank you [Music]