Fuzzball-Ansible deployment with Ascender

and just fill out any form there and it will come to us and we'll get back to you so very happy to be here Zane say a little hello to the people hello people it's been a while I'm really excited about this demo because I didn't get to go and I haven't seen it and it's uh it uses a couple of my favorite tools so I'm really excited to see this cool yes I'm Zane I I'm on the sales team here at CQ been around for a long time almost three years three years next month wow congratulations almost congratulations super close that's awesome and

Mr Jonathan thank you so much for being here thanks Rose thanks for invite yeah I enjoyed putting this together and happy to get it in front of more people yeah good to see you Jonathan I don't get to see you anymore good to see you Tuesday sorry I'm I'm looking at one person and name the other all good so um okay so we have a little request to turn up your microphone Jonathan you can do that kind of get closer to it something how's this sound is that better or actually yes the same okay great yeah there's an auto adjust setting in it and it

uh it it never with this microphone it's always apparently too quiet um but anyway and now it's moving on its own hopefully we're good awesome so um there was a couple of shifts as we put you know more focus on fuzzball and internally you went to work on fuzzball so maybe you want to just give like a little overview to our audience what fuzzball is yeah so fuzzball is a um kind of cloud focused or not not just Cloud but like um inspired by some cloud computing paradigms uh HPC and performance Computing uh provisioning resource manager scheduler system so the the goal is to take

the experience that we have in a traditional HPC context and kind of pair it with experience that we have in a more Cloud native context and take the Best of Both Worlds and make something that uh can run bare metal high performance Computing workloads uh but run it anywhere uh with the flexibility and elasticity that you might expect in a cloud environment and and get some additional benefits along the way like maybe a little bit more reproducibility than you might get in a traditional HPC environment and yeah it's it's been coming along it's a cool project it has been coming along and anyone in this

this world uh can relate to having an idea going towards that idea and then having a couple of like I don't know stumbling blocks is really the right word but you know opportunities do things differently and to continue to iterate and do them better and so that's what I imagine kind of incorporating anible and our product Ascender into the mix so why don't you just like give us a little overview of what Ascender is as well sure so Ian it people may or may not already be familiar with anible but it's one of kind of a family of or or a set of peers um

kind of competitors in the space are like puppet and salt and Chef and things like that uh anible is a kind of configuration driven um configuration management system where you can put the how you want all of the nodes or all the systems in your entire infrastructure uh down in configuration files and then mix it together and deploy it and make it be true uh kind of the goal here is defined State configuration where instead of having just a person go and make changes on all of your computers you can say I want the computers to behave like this make it be that way please

and so anable is is one of and one of the most approachable uh um Tools in that space and then Ascender um is a web front end for that that that gives some additional functionality that we'll show off a little bit here um for bringing in your configuration data from different sources uh managing Secrets a little bit more um intuitively and then parallelizing that configuration process uh in in kind of a useful way and we'll get to see that a bit here um but to speak of Legacy and the the way that things kind of evolve and change over time like the the inspiration

from this starts with uh doing on-prem deployments of fuzzball so we we have like two ways of thinking of it either we're running the control plane for fuzzball out in a cloud infrastructure uh and we call that a cloud deployment or we're running the control plane uh what we call fuzzball orchestrate uh on on Prem in a local data center we happen to be still deploying it in a cloud environment here for testing um but it's deployed in the on-prem way the way you would do it uh locally at a site um oh is there is there some feedback no are we good yeah I'm

sorry I'm seeing notes down here that it seem a little bit kind of like a I don't know I think it's just Rose I don't hear it okay I'm gonna ignore it for now anyway so Brandon Brandon might hear he made comments historically we've used a tool that we developed in house called icbe to do these on-prem fuzzball deployments and the the thinking here was that for an HBC audience kubernetes was still uh kind of a a scary blackbox kind of thing um and and fuzzball because we're taking expertise from a cloud native and an HPC environment kind of mixing them together one of the

parts of that was that fuzzball orchestrate our control plane is developed as a kubernetes application so we needed a kubernetes to deploy it into but we thought that our customers might not have one and might not want to have to worry about that so we developed icbe which was kind of a minimal kubernetes distribution in a sense and then some automation around setting that up and deploying fuzzball and potentially other ciq applications into it over time as we worked with our customers we found that as as history has progressed our customers are more and more familiar with kubernetes and not afraid of it and in

fact have an existing kubernetes infrastructure that they want to use uh either they they just have experience with an existing kubernetes distribution that they already want to use or they have a fully fledged infrastructure that they want to install fuzzball into so we pivoted uh for that deployment process and are now using what's referred to as a a kubernetes operator to deploy fuzzball orchestrate into an existing kubernetes cluster and we provide some help around um around doing that deployment and and around uh setting up an uh kubernetes cluster if you don't already have one uh for internal testing we're targeting rk2 or the Rancher kubernetes

Engine 2 version two also sometimes called I think Rancher gov um was its maybe former name or maybe one of the product names around it uh that's been a good experience so far but some of the people at ciq have also used the exact same deployment mechanism to just deploy it into other kubernetes distributions and I won't list them here for fear of one implying that we're testing it as well yet or leaving one out but so far I I think it's just worked on every distribution we've tried so far uh even ones that I hadn't even known existed so that's been exciting um but

one of the benefits to that um because so the the a kubernetes operator is basically just a little application that you deploy as a kubernetes pod you deploy it into the cluster and then it runs as a an installer for your larger application so we have a little custom resource that is called fuzzle orchestrate that defines how you want your cluster to behave and by putting that configuration into the cluster and having this operator there watching for that configuration to exist it does some configuration management for your cluster it wakes up and says oh look I see that there is supposed to be a fuz

cluster and I will make this kubernetes environment have that but because kubernetes is a yaml based configuration engine cluster thing and anible and therefore Ascender is a yamel based configuration management thing uh it became very intuitive to put the two together um so we've been using anible internally to uh deploy fuzzball even with icbe um but it wasn't as straightforward as I would have wanted in order to put it in front of customers but as soon as we moved over the operator and started deploying it that way it just became something I was like oh yeah this customer and that customer they would actually

really appreciate this we should get that out there so uh I'll go ahead and share this just to walk through it a little bit um but let me find here um I think this is the right one that looks right um try and get my windows moved somewhere where I can see everything that's going on um so we we open sourced this so it's it's not there's not too much here but this is what's called a an anable collection so uh not going to do a whole ansible primer here but um configuration and anible is divided up into a number of different things that can

Define that configuration and some of that is reusable and so we've tried to push as as much as we can into reusable code here as uh as would be useful so there's a set of roles and these are are little bits of reusable configuration that can be applied to any host um so we have one here for setting up Depot which is a a Content distribution system that we're using here at ciq um we can deploy the CLI um the operator itself orchestrate anything that we need in order to set up a fuzzball cluster then I I had this already in a tab but I'll

skip over here to Playbooks once you have those roles you can assemble them together into what anible calls playbooks we can look at one as an example right here this is a really simple one uh we'll deploy the fuzzball command line interface and there's a little bit of kind of magic here for making it easy to Target which hosts you want to install that on and this one's just using this uh fuzzell CLI role and saying this node that we're going to run this Playbook against should have that role and then uh finally uh talking about kind of things being yaml uh yaml is a

a markup language that it looks like this right here it's just a a simple way to have configuration data in a text format and one of the benefits is if you saw a demo that we did earlier where we did a fuzzball orchestrate deployment kind of by hand the way if you were following a guide and just doing the instructions one part of that instruction is make a document like this and we give you an example and say here are some values that you might need to change a way we can do that with anible is just to encode the default settings and there's some

variables in here that you'll fill in uh but provide default settings and then uh in a bit we'll show how you can override any part of this in what anible calls the inventory um which lists the nodes you want to deploy on what parts of what things get deployed on what nodes and then configuration if you need to uh to override any part of that so I I I have demos of this but I'll I'll stop for a minute real quick in case there's any questions on the web or Zan or Rose if you have any questions for this so far I don't yet and

I don't see any so I think we're good to keep going Jonathan okay so let me switch over then to uh uh to my terminal I just had a personal question for you real quick actually Jan thinks as my mind is going um how familiar were you with anible in like your previous jobs did you ever use it there or personally like what what's your experience with it I had used anible a little bit at a former role so I I mostly had used puppet before um and I'm kind of a finicky person so none of these are my favorite uh they all have something

that I I wish was different I I wish all of them had a little bit of each other in them but the the really great thing about anible is it's so easy to get up and running with because it uses SSH as or it can use SSH as its deployment mechanism so you have a server you can write some code and say make that server be this way um and so it it ends up being really approachable easy to get up and running with um but I I had mostly used puppet and I in my kind of you said personal in my home lab environment

I had used salt a bit um which is has some technical stuff I like about it but is probably one of the least popular ones uh unless you're in some very Niche use cases um but these days I'm I'm using anible just because I'm I don't want to fight anymore I just want to use the one that people like and uh and get over myself so that's what I've been using even in my home lab stuff these days um just because it's easy you mean you stopped using the the one that everybody else stopped using a long time ago it has been resurrected though it

has been acquired and it is definitely seen some some new life yeah I could I could talk longer than I should right now about the differences between anible and salt you're an agent guy I can tell you're an agent guy it's not the agent ah you're gonna get me into it it's not the agent stuff I actually spent a long time trying to get the salt SSH uh client to work so that I could run salt the way that anible does because that's the part I like about anible and I had like bugs open with the project and their literal answer was we don't care

enough about uh salt SSH to fix this and they closed it so I stopped using it and here we are well that's totally separate topic sorry sorry get way off in the Weeds on this one all right so here we are I happen to be in uh so I I showed earlier like fuzzball anible is a gate repository it's public um we'll put a link to it in the show notes uh you can use it today you won't be able to use it to deploy fuzzball without credentials to Depot to get the containers and the packages that you need but if you're a fuzzball customer

you should have that and you could use this today um to deploy fuzzball so uh the first thing that you you can just use use it out of this git checkout um but another thing you can do is install it with anible Galaxy so anible has what's effectively a package manager for anible collections and so um there this this collection here actually has kind of two two faces two ways you can interact with it and one is as an anable collection um we can install it I'm forcing here to because it's already installed so otherwise it would just say oh you already have it but

to see the process um there is like a an anible Galaxy server that you can just install collections out of uh but you can also just point it at a get repo which is what we're doing here and then uh by default it'll check out this main branch uh and you'll get like the latest and greatest fuzzball um but we're also maintaining branches for each uh fuzzball release and so if you're on a specific point release of fuzzball you can put like 2.0.12 here and stay on that get get a check out of this that has been tested to work with that release is the

goal uh but oh not mail uh but I have this here and it's installed and so with that um one of the things we can do is run uh The Playbook so I'll I'll walk through I'll simplify this a little bit uh so we're running the an Playbook command we're going to run on the remote system as route which is pretty typical you either are run as root over there or you go over there as a regular Ro user and become root because we're install packages and things like that you need to be an elevated privilege um and we're going to read our inventory which is the configuration that we're deploying from hosts.

yl and I'll show that inventory in a moment um when I did this demo internally I realized I was exposing my Depot credentials to the demo when I opened up my uh uh my inventory so since then I have split those ential into an encrypted Vault so we're doing that and then asking I'll be prompted for a password in a moment and then I mentioned that little bit of um of kind of magic comment there in the Playbook this limit here lets us Target which host we want that to run on without having to do that designation inside of our uh inside of our inventory

and that becomes useful uh when we look at Ascender here in a moment so I can run this type my password correctly hopefully great um and we get a little bit of output this we're just going to deploy the CLI it's already there uh but it'll go through like I said that the point here isn't just to do the steps but to here is a state that I want my my system to be in make sure it is in that state so it's going to go through and make sure the depot is set up and then make sure that the CLI is uh is

installed presumably from Depot it won't make sure that that's where it had come from but it will make sure it is installed and at the correct version so there we have it uh from our Command we can see here I'm running this with kind of a name space you can also run playbooks from the command line at the the file these playbooks happen to be here in this playbooks directory so I could just point there um but if you install it this way you don't have to worry about get it all you don't have to do the checkout or clone or anything like that you

can just install the Playbook and then use it without having to wor about where all those files live and then I'll show uh my inventory here so I have uh a few things I I'm setting my fuzzball version even though I'm checking out the main version of the uh of the anible code I can still say what version of fuzzball I want to be deploying and then I have some overrides in here for how I want my environment to be configured so I don't have to replicate all of those default settings but I'm setting what IP address I want certain things to be on and

there's an NFS server that's used for shared State and I tell uh fuzzball where that server lives and how to connect to it um and then fuzzball uses keycloak for identity management and this distribution is setting up a keycloak for me uh so I have initial information here about how I want that keycloak to be configured all of this uh just gets overlaid on top of that default configuration and then becomes a combined configuration within the operator that uh that configures my eventual instance and then finally here's a list of hosts uh that it will be deployed on I'm doing everything on one host today

um but you could have an arbitrary number of hosts that have the command line interface on it of course and then we could set up kubernetes on multiple hosts and deploy the operator into that that kind of thing so the next step is to see how this looks in Ascender but any questions here before we move on no I think it's important to call out there are like 58 different ways to do variables in this so yes don't get confused it's easy to overwrite them at a whole bunch of different levels I think there's like 36 levels of inheritance so be very careful yeah

one one of the things that I was really pleased to experience so far I think there's not an example of it here but I didn't know before I tried it whether I could like set a variable somewhere else and then refer to it here in the inventory and it just worked the way I expected it to so I haven't like try to find the edges of it and where it might break but so far I've been able to Define variables in one place and use them in another and it's just done the right thing which has been nice very nice see there's a question about

trying out fuzzball Brandon thanks for being here man so he says is there a limited feature fuzzball or a trial fuzzball people can use so there's not an on-prem deployment trial um there is active work right now like our next deliverable I don't want to like too specify this but we're working on making a version of it available in the cloud so that people could more easily try it out we have it running in Cloud environments already but it's not like an easy onboarding process right now so we're trying to improve that specifically so more people can just try it out without having to do

a whole deployment thing yeah awesome that was a great question yeah all right we're going to switch over to uh switch back to my Chrome tab where I have Ascender running excellent so we've demoed Ascender here before um but not with fuzzball uh and uh like I said it it's a web interface for anible at in a reductive sense but you can already see there's like more data here and we plug it in with products like Ledger um that GA gather all the log data that comes out of um that come out of anible otherwise and Aggregates that and helps you get more

insight about what's going on in your cluster or your kind of your total it environment but today we're mostly going to use it to do a deployment so I mentioned that there's two uh kind of two faces to this fuzzball anible thing developed one is as a collection that you can install with uh anible Galaxy and you can use that directly from the command line like we demonstrated but I I was very interested to try and we we ended up succeeding making it also work as an Ascender project so uh the way Ascender typically works is you have um you make another git repository you

make a a place where you put your playbooks and playbooks are considered kind of local data that call potentially shared data from Collections and roles and things like that um and you're expected to to develop those playbooks and maintain them locally you can totally do that with fuzzball anable here but I wanted it to not be required so and you can also use fball anible as an Ascender Playbook so you can see that here we have this project created and it's just pointed at um fuzzball anible right on GitHub and uh by default it will also in install any collections that are required by that

project so that's all defined you don't have to do anything you just create a project in Ascender and add it with this uh with the same git repository that you you know that we'll be advertising in the description and you gain the ability to do fuzzball deployments in your environment there's one thing that isn't automatic uh but we're developing some documentation about and hopefully we'll be putting a an article out about this soon um in order to do Depot you can just put variables in your environment but that's a little bit unlovely uh We've also defined this custom credential type for ciq Depot that

lets you specify your Depot user and token and server in a secure way uh like how I was talking about wanting to separate the credentials from your inventory so we provide a template for doing that um and I don't know if there's a distribution mechanism for uh custom credential types or if we'll just need to document that and then people can do this kind of thing oh I'm in the wrong template um so here I have this Depot credential that says I'm going to use uh Depot to get packages as myself and then the token is encrypted and not shown as you would want um

but once we have this project and it pulls in the playbooks those playbooks are available uh uh to be run as what Ascender calls templates so we have a just to have a complete example here we have a template for each of those playbooks so we can see an example here's the uh the CLI template um it it is really just a reference to that same Playbook we can see the the path to it here uh but once we have all of those and I could just run that directly uh but you can tie them together as this workflow template and and this ends up

looking cool rather than go into it directly I'll I'll run it and then we can see what that looks like and every stage along the way whether it's in the inventory or in each of the the job templates or here in the workflow template we can specify additional configurations so just like we had in the inventory we can put that I want to be running fuzzball version uh 2011 here um just because that's what I was running when I did this deployment so we can go back here to our deploy fuzzball click this Little Rock ET here it may fail the first time there's a

bug in the version of Ascender that I'm using yeah it it failed but it will uh work the second time uh I've have been told that this bug has been fixed and I just didn't want to risk breaking the whole thing uh by upgrading my Ascender uh before the demo so on the second try it should be fine and we get this cool little visualization of these three tasks are running in parallel if you do this on the command line typically the way it runs by default each step would just run one at a time it' run on all the hosts each play would

run on all the hosts that it's applied to simultaneously but it could kind of do one play at a time but here uh in Ascender we have the flexibility to say these three are independent you can run them in any order or together but then orchestrate can't run until we have an rk2 environment so we're actually deploying a kubernetes cluster here and the NFS server is ready because orchestrate depends on both of those and then substrate which is the the compute agent this is the thing that runs on the compute resource and receives work from orchestrate we can deploy that as soon as this

NFS system is ready because even if orchestrate isn't ready to send it work we could still be setting up the compute nodes and so it gets this little dependency graph we get status information about what is done what's started um and what is left to be done and we can click into any of these and get that same log output that we were seeing on the command line before for each step as it happens the other thing I I know that this job doesn't require them but you can also have failures and roll backs so if any one of these tasks fail you could go

correct anything else on the other tasks that have already run yeah I haven't done any kind of responsiveness in this uh they just fail and I think maybe I've used retries but yeah as Rose was kind of alluding to before I'm still relatively new to the Ascender space it's been cool to have an excuse to use it and as soon as like I said as soon as we started moving to the operator it's like this is finally a thing that I would love to use with h with Ascender so it's the best way to learn a new tool a real world use case see

back here oh there we go come back here and see NOP trying to get back to my activity here we go so yeah it's it's still going we can come in here and see that this is running but this is the bulk of it uh I you know we have time for questions and stuff I'm happy to keep talking about this but that is the demo for what it is worth it's awesome it is very cool it's always cool when multiple tools come together and come together as a solution or at least making one easier to deploy like you've done it's really cool I I'll

be interested to see as customers get their hands on fuzzball with the ability for Ascender to touch all infrastructure how people start pulling all this back in So deploying massive HPC environments um working in the storage space network security and then pulling Ledger data back in it's goingon to be really cool to see how all this ties back together into a big nice picture yeah the the coolest single thing for me with this so far has been you know I I I showed let's go back to it so in this template here so this is the the workflow template for the entire deployment um

and you know a customer's deployment doesn't have to be exactly the one I have maybe they have an existing kubernetes cluster and they don't need to deploy one you know that that part they would a customer would do themselves but uh I like setting the fuzzball version right here at the top level and I've been able to upgrade by just editing changing this number and rerunning and I just have an updated fuzzball and that is uh quite an improvement in the experience of upgrading fuzzball over previous ways that we've we've had to do it so yeah glad to put this into people's hands and uh

at at various levels again you can use the the anible stuff today um if you're just using anible on the command line if you're interested in Ascender I I'm sure there are many people here at ciq that would love to show that off to and get that in your hands um and then fuzzball as well the other part that I would love to see you do is since all of Ascender is API driven do this all via API without ever logging in the actual goey yes that would be cool I have not done that yet in fact I don't I don't think I've

touched the Ascender API at all myself but yes that seems like a good really tying it into an actual workflow to upgrade your your entire cluster like that just from an API call it's pretty cool yep so cool I do not see any questions y well this was awesome Jonathan this is such a wonderful way to highlight all of the work that you have been doing and that ciq and all the engineers here have been doing behind the scenes with fuzzball and with Ascender and like seeing it kind of come to life a little bit more each day it's really awesome appreciate it it is

really cool uh I'm working on write up for this um so you know for people who would be interested in it and want to guide rather than having to watch a video so once that's published I'll try to make sure that gets linked also in the description for this um but yeah love to hear any questions or feedback that people have using it or looking at it regardless yeah awesome that's fantastic thank you Jonathan absolutely yes thank you thank you so remember you guys if questions do come up you can leave them in the little chat right there um you can also go to C

q.com we've got all kinds of blog posts and other demos and information and places where you can you know put in your specific questions and then that gets directed right to us so we can make sure that we can get you some like one-on-one time with us talk a little bit more about your infrastructure your environment what it is the problems and challenges that you're running into and how we can maybe help simplify things with Ascender and or fuzzball we'd love to chat with you more about that so thank you again Jonathan for your time and all your good work and everyone behind the

scenes that made all of this possible very very very grateful thank you Rose thank you Zay it's glad to be here absolutely good to see you yeah bye guys bye see you and