Hands on Warewulf: Solving Cluster Provisioning & Management

this kind of differently and i really wanted to focus on the idea of stateless and so what werewolf is is a way of uh administrating the operating system of lots of compute resources in an extraordinarily scalable and configurable way and that's what we've been able to create with werewolf as of the latest version it leverages the container ecosystem such that you can basically take a container you can build a container however you normally build your containers and then you can import that container into werewolf and then basically say i want to boot a whole bunch of nodes with this container and it will actually put

that container on the physical bare metal that you know that you want to provision out to and so it makes it very easy to maintain it makes it very easy to to provision large clusters and also to kind of provision different roles and purposes of your clusters via different container images so at a high level that's kind of what werewolf is excellent so thank you for that overview i think it's really appropriate that we have michael here now because the next question i have is what makes werewolf simple to deploy and use and administer and since you're new to it or you were new to

it you're not now what made that simple um assuming i think uh what made it simple was that um you know just being able to install an rpm and then um i actually went through the getting started guide on the werewolf's website and um it was from a system administrator viewpoint pretty quick to grasp all the different services that it's setting up for you so i didn't have to go configure dhcp i didn't have to set up tftp i didn't have to set up all those services um separately werewolf actually did it for me so it's just a simple config file set my ip address

and my control node i mean my head node and set that up run a quick wwctl configure all and boom you know werewolf was up and running uh then it was just learning how to configure um to set up my nodes so that i can get my nodes to configure and again it's just simple it's command line arguments that you're just passing into werewolf and it's doing a lot in the background just to make it easy so you can then um have your cluster ready and through ipx it just boots right up as soon as you know your your node can um can get its

configuration and get provisioned your node is running stateless so um that made it fairly simple very nice hey greg i know that was kind of the intent was to make it so that it was simple to deploy and use it in ministers i mean having someone new come in like michael and go through that process and now pretty much being an expert it's impressive for a piece of software that does something as complicated as what it's doing so i always like to ask you everything to add to what michael said i mean i know there was intent there when you went down that path but

making it simple so let you fill in that blank so you you can't take what michael says at face value um because he's he's a brilliant engineer so him coming into something and being able to pick it up very easily um is it's because he's awesome now with that being said we were making this we were trying to make this as simple as possible um but uh yeah yeah michael's just awesome so i'm glad to have him on the team absolutely and i'm having decades of experience this has been around for how long now over 20 like 2001 um i founded it uh and there's

some funny stories about the foundation of it as well and and how much i completely blundered the first the first kind of implementation of it um but yeah it stabilized pretty quickly and then once i once i kind of fixed the major i the major thing that was blundered and just so just everybody's probably curious now that i've said that um uh this was before so 2001 was before pxe was standard on network interface cards so we actually started off um not on pixi as a matter of fact we started off on bootable iso images so you'd actually use your werewolf control server to basically

create iso images for each one of your nodes and you boodle you burn a little cd-rom cdr and you put that into your node cd-rom and you and it knows how to boot from there uh that was the very first version of werewolf uh i went and presented it at linux world um and uh everybody the feedback that i got was uh everybody liked the administration model they liked how it operated they liked the tools but what the heck am i thinking regarding these cd-roms and isn't there a better way of doing that uh and the one one person raised that raised their hands and

said you know there's a booth in the back over in the expo hall for ether boot go talk to them i said oh great feedback awesome right after the talk answered a few questions went over to the back of the expo hall and met with the ether boot team and ether boot if you're not familiar was the predecessor at least in the open source world to pixie and they created uh images uh that you can flash on the roms of your nics or boot like on a floppy drive because again this is 2001 on a floppy drive or cd if you want but a tiny

little image that basically loads what's called an option rom into the bios which tells the bios that this device is actually bootable and it will now show up in your bios configuration you can select your network card as a bootable device like a scuzzy card select that as a bootable device and that option rom will then give the necessary capability to that network card to basically do a boot and it worked very similar to how pixie ended up kind of taking off uh and and growing from there but that was the predecessor predecessor to to pixie and uh so we then moved to ether boot

and for a while um we always had you know people had to flash their roms their nicks or get nicks that had special roms on them for ether boot or just use a floppy drive for so for a while actually even early werewolf you almost always had a little floppy disk that you put in that matched your nick put the floppy disk in and yes actually that was how you booted before pixie and then as soon as pixie started taking off that was a very easy transition uh and motivation so we just basically moved everything now from ether boot over to pixi and it's been

pixie ever since um well kind of no now it's ipixi but it's still pixy nice thank you for that so i think one of the other things i want to make sure that we talk about is we have support for x86 and arm right it's not just an x86 platform it supports arm as well fantastic yep and then let's talk about use cases so what are some of the use cases for doing a werewolf install i mean what different types so werewolf was originally designed for high performance computing clusters and the the predominant reason for that is in an hbc cluster you can have tens

hundreds or even thousands of compute nodes that are all virtually identical every one of them is running the same os same versions or you hope you want them to be otherwise you get version creep and you run into other problems so you want everything to be running never happens not with werewolf actually it never happens but you you want everything to be uh um as similar and homogeneous as absolutely possible now you could have groups of nodes that are different so you could have 100 node over 100 nodes over here it's running one version of the os 100 nodes over here that's running a different

version of the os 100 nodes over there that maybe has different configurations or different permissions on certain things on access controls and whatnot werewolf was designed for that werewolf is designed to be able to bring large amounts of resources kind of together now in terms of use cases hpc was the obvious one it was it was very easy it was what it was initially developed for but one of the interesting things that happened um i think was about 2000 before 2010 or maybe around 2010 is i started to hear of people using werewolf clusters for different purposes uh the first one that i heard was

uh the guitar center and musician's friend was using it to power their entire web infrastructure any musicians out there yeah for a while i doubt they're still using it but this was a while ago i think it was musician's friend in guitar center that reached out to me uh and i said well if you ever need any help i could use a les paul um and uh didn't happen but uh but yeah so there were people using werewolf just as an example for for different kinds of use cases and but but the majority of it is is high performance computing we have actually and michael

you may be able to talk a little bit more definitively on this uh we now have actually even considered and i think we've even now uh proof of concept the notion of being able to spin up kubernetes clusters and other types of clusters using werewolf so it's it's you know the fact that we're based on a container image means anything you can put into a container that you want to run on bare metal you can pretty much do excellent yeah michael i toss it over to you now have you been seeing that as a use case that you're running into yeah i mean one of

our customers we used werewolf to provision control plane nodes um and we're deploying kubernetes on those uh control plane nodes and then bringing up our compute nodes um the part of that has to do with um another product that we have here at ciq uh fuzzball substrate so that's where we've been using that so we have a a nice tool called icube which allows us to deploy from the werewolf control node to our three control play nodes and then bring up our compute nodes and they're all running fuzzball substrate immediately start talking to our cluster and you know within i'm gonna say depending on how

fast your machine is and say anything from 15 minutes to maybe half an hour you have a fully functional um cluster ready to go and run some workflows so that's one of the the use cases we've been using i've been using it a lot uh as well for testing so testing our products and it makes it easy because especially because it's um stateless uh one nice thing is i'm able to bring something up if it doesn't work i just reboot the machine and try it again um so you know and then with a little bit of scripts that you have to you know clean things

up but it just it's made it much easier for testing and development purposes so michael sorry sam jumping in michael brings up a really good point and it's something you know we didn't really talk about is the concept of stateless when we provision nodes in a stateless way we're basically provisioning them so when you turn off the computer the whole concept of what was installed the operating system everything kind of goes away it's like it was never installed and there's several different ways that you can achieve a stateless compute you know fabric and and the way that werewolf does by default is to actually run

it out of system memory so if you have let's say for example a container image and you want this container image to run on these hundred compute nodes it makes it very easy to manage if you just turn on those compute nodes and they all get that exact same container image and if you have one node that's doing something a little bonkers a little weird it's very easy to basically say well it's not software because the nodes above and below are are running just fine so you got some sort of weird hardware issue going on and you know you can start looking at the hardware

to see what's going on there i can tell you from some of the initial goals of why we created werewolf and and what we're trying to do with it is really to simplify the management of these sorts of systems and that helps a lot if in your cluster you are managing a large number of servers or workstations um the kind of equivalent right put air quotes around it again if that's what you're kind of that that's what your your administration model is then if you have a thousand nodes in your clusters and each one of them is is installed directly you have to manage each

one directly there are tools to facilitate the management configuration management tools and what not provisioning tools kickstart things like that but if each one is physically installed you cannot start over time you can end up with what we kind of alluded to before which is version drift where you can have slightly different versions of different aspects of your cluster and sometimes that's very difficult to find if you have a very big cluster um you know any sort of differences and whatnot what werewolf does again because every time you reboot that computer you can guarantee the image and integrity of that image as it's being provisioned

uh it makes it again very easy to install so whether you're doing something like a traditional hpc stack let's say you have you know infiniband drivers gpu drivers you have maybe luster file system drivers in there you can distribute all of that into one container when you provision that container out it's going to boot that whole stack if you provision it out to another cluster it'll boot that cluster with that exact same stack if you put that up in docker hub or or any of the public registries you can actually share that with other people which means for things like again a traditional hpc stack

an open hpc stack it makes it real easy now to distribute kind of that that system profile to any clusters out there and then werewolf will handle the configuration management that you need to to be running in order to manage differences between nodes ip addressing host names service configurations werewolf comes with a very small um simple configuration management system which is all templatized so you can have one file for all nodes that boot and when it boots it will get the template that you've configured for that node that could be things like ip addressing you know if you have infiniband on some nodes you can

define that as well so there's a lot you can do with with werewolf in this regard and and again kind of you know going back to the original question um regarding um you know use cases and and whatnot you can do things like again traditional hpc you can do an open hpc stack very easily and what michael kind of brought up which is fuzzball which is our hpc 2.0 stack and you can use werewolf to provision out that entire stack both the compute resource as well as the management resource and the control plane for that entire cluster so a lot you can do with it

oh and it's also i should mention it's also operating system neutral right so um you know we're we're very closely associated with with rocky linux but you don't have to run rocky linux with it you can run any of the enterprise linux derivatives but you can also run any of the debian derivatives you can run souza on it and through it as a matter of fact soos has been an amazing partner um with with werewolf and they're actually contributing and helping us with the development of werewolf and susa runs fantastic with werewolf so another aspect to consider is just how portable simple lightweight it is

um and it's just yeah very easy to deal with solution yeah the partnership part of that to me is exciting uh you did talk about when you have multiple clusters or you had several different clusters can you control all of that from one werewolf controller or do i need a separate werewolf controller for each type of cluster that i'm going to run you can control it from all of one werewolf control surface as long as you're managing your broadcast domain so michael mentioned that you know tftp dhcp or kind of you know they're required services whenever we're talking about pixi werewolf will configure that for

you but at some point you're scaling up your cluster so big that you actually have to start managing your broadcast domain uh because the last thing you want is storms or loops or something to all of a sudden take out take out your network uh when you're dealing with large clusters and the traditional beowulf model is very flat right so a lot of people say well we're just going to put that on just one gigantic network segment uh and not manage the broadcast domain you can do that for a thousand nodes for 2000 nodes you can even go 2500 nodes ish at some point you

really want to start dealing with your broadcast domains now there's multiple ways of dealing with your broadcast domains one is you just create relays between your routed networks and then route between them and then do dhcp uh relaying but the other way is you basically have multiple werewolf control surfaces um so maybe one on this this logical network and one on this you know logical subnet and you now have two werewolf servers now this also kind of mitigates some of the scalability factors um werewolf will scale many thousands of nodes uh with one werewolf server but at some point just for resiliency you probably don't

want to put many thousands of nodes on just one werewolf server so you may want to spread that spread that load out a little bit now this is a very long way in a roundabout way of answering your question regarding can it do multiple clusters with however you're managing your broadcast domains however you're managing your werewolf infrastructure werewolf is designed to be able to have multiple node groups via profiles and different node configurations so you can absolutely you can separate this into into network segments or subnetting and managing your broadcast domain between systems and have you know werewolf clusters kind of running all of that

but another way of doing this is to basically just have one big cluster that has different groups of nodes for different purposes or different research initiatives or different projects or pis um within your organization and you can then kind of break apart your nodes into different groups again based on these other factors you can break them apart could also be in terms of vintage of hardware many organizations building hpc clusters they go into it and say okay well this year we've got three million dollars we're going to spend for capital equipment purchases we're going to buy our base cluster but you know next year they

they have another million bucks and now they're going to add to it but the node profiles have now changed a little bit and you might be able to kind of still bulk them together but at some point if you keep doing this every year at some point you're going to get older nodes and you're going to get newer nodes they might still be compatible but maybe you have to put them on different infiniband fabrics because infinibrand has now changed so so much and you have to start managing your network fabric and whatnot and at some point your end of lifeing and you don't want to

have like an a big parallel mpi job running between the newest nodes and the oldest nodes because then you're going to get mixed performance and you're always going to go as fast especially on a tightly coupled application is going as fast as your slowest node so now you've got your newest nodes sitting there that are not completely utilized in any way long story short too late um you could basically um uh create pseudo partitions within werewolf based on configuration node groups and profiles different kernel images different os images different roles you could have file system i o based nodes you could have your luster storage

compute excuse me your luster storage nodes and then your compute nodes over here all right so there's a lot of ways you can kind of mix and dice how are you setting up your your your cluster infrastructure and werewolf is completely um flexible to all of that outstanding one question i do have is whenever you whenever you start working with a tool like werewolf one of the first things that people are going to want to know or that i would want to know is what kind of pre-configured cluster images are there am i gonna have to go build this thing from scratch or is there

something that i can just install and start using out of the box uh there are some pre-configured images out there uh we post some we have got some up in docker hub uh for note images you can get there in docker hub it's just under the uh the the organization name werewolf and you'll find rocky images there centos images there open hpc images there but for the most part you can use almost any image that's up in docker hub which is actually kind of kind of crazy to think about now there's one kind of gotcha there's actually maybe two gotchas the first gotcha is when

people create the default container images that they put up into docker hub they are technically they don't have a full boot capability associated with them system d has masked out a bunch of services they have core utils that are designed just for single user mode and not multi-user mode so you have to make a few changes to them but once you've made those changes and those changes by the way are public and werewolves so you can see how to how to change those once you've made those changes you can pretty much boot almost any image that's in docker hub today and actually provision that out

to bare metal hardware and then run that in a stateless way so in terms of turnkey solutions again we provide a few so if they're you know not to make this a a sales pitch but if you are interested in turnkey solutions uh around werewolf we can help um help you we've got ones that we've already made but we can also help tune and help create those uh for customers um open hpc as i mentioned before and uh hpc 2.0 our our cloud native cloud hybrid federated meta orchestration platform for performance critical workloads and data we have that too you need an acronym for that

just one long drawn it's kind of fun to say once you get going on it nice so one of the other things that that i find interesting is integration with other tools and when you look at configuration management i know we talk about where we'll be able to do configuration management of its own but i know it will also integrate with other configuration management tools correct i'll take this one and then i'm going to be quiet for a little while um i feel like i feel like i'm taking stealing uh michael's mic time uh sorry about that um the uh the configuration management system that's

inside inside of werewolf is really designed around um how do you provision configuration changes to a system before s been in knit is called and s been init in case you're not familiar is the parent of all processes you do a ps on your system it is pid one which means how do you make changes how do you get configurations onto a node before s bin in it is called is pid one and that's what werewolf does werewolf will make changes so if you have system configuration changes network configuration um you know ip addressing all of that sort of stuff it has to be there

before systemd starts and werewolf will be in charge of that you can use it for other things in dynamic files like um your your credentialed files your your etsy password etsy groups files and whatnot you can use it for that so you can ensure that you users have accounts on the system and whatnot and werewolf is good at that but if you need anything that's really more complicated than that base level of configuration management you really should be looking at just layering ansible or puppet or or cf engine or or chef or whatever you're you're comfortable and whatever you prefer using just put that into

the node image and configure it in the node image then when all the nodes boot they'll not only boot with the werewolf configuration that you've set through werewolf to ensure that everything's there when the system you know when system d is is running and is called but then you can actually then do further configuration management using whatever tools that you prefer so um yes to both excellent nicely we've got a question from jonathan that popped in he's asking about werewolf four so what's the status of open hpc support packaging documentation tutorials for werewolf four jonathan i was gonna be quiet here um but i'll answer

this question i'm going to try to be quiet again uh emphasis on the try uh i don't know yoda quotes are going through my head now um okay so uh open hpc support uh supports currently werewolf version three a really good call on that werewolf three uh is a little bit long in the tooth in terms of um age and and whatnot and it has not had a huge amount of active development in in quite some time uh everything has been now moved over to werewolf four but to your point open hpc has not adopted werewolf four yet uh we are currently working with the

openhpc team uh the openhpc team is actually even contributing fixes um patches and whatnot into werewolf 4 as it is being prepared to be brought into open hpc so it shouldn't be much longer but it is in progress but it's not quite there yet i would anticipate that it should get there in the next major release of open hpc and i'm sorry i don't recall what release that's going to be but um i have i have strong confidence that it is going to be an open hpc in the nearest future perfect thank you thanks for the question jonathan michael i want to give you some

time i know that you had prepared something to show us something we can look at so i will turn it over to you for that all right i'm going to give this a shot you know live demos are always fun so um you can make your screen a little bit bigger you don't want sure the font the file right there is good one more one more perfect all right so um i just want to show briefly how to um initially configure um a profile set up some nodes and i am actually using three nodes up on a vps provider so and these are not bare

metal machines they actually this provider um supports ipxi so booting so if i remove this screen you actually see i have my three nodes here and they're trying to communicate but i don't have these nodes set up yet so i it's actually just set up it's it's retrying every minute we're gonna do hopefully if my demo works we're going to watch these three nodes get their configuration as soon as it's configured in werewolf and boot right up so getting back to here the first thing i want to do is i want to set some uh kind of like defaults in a profile so now with

profiles you can have multiple profiles i believe uh greg mentioned that already um so this i'm just going to use the default profile to set some um some things for instance um i know that on my nodes um my three nodes that i showed you i actually have two interfaces on them one of the interfaces i have set up to be on a private network and that's where i'm going to manage over that private network from werewolf these three nodes so one of the first things i'm going to do is i'm going to i'm copying and pasting this to speed up the time um i'm

going to uh set my eth one device network interface because that is on the private network and i'm gonna set a default netmask now the reason why i want to do this so if i list this the reason why i want to do this is that this is going to be applied to all my nodes um i can add a new node that will be there i do not have to set that now for that node it's automatically going to be applied to all nodes that i add so that's one of the benefits of having profiles you can set a profile um based uh that

can be applied to multiple nodes at once without having to sit there and and do that for every single node that i'm bringing up um the other thing that i'm going to do is uh i probably should have done this first is i'm going to import a container okay because i'm also going to set if we notice over here i have the ability to set a default container so this again helps simplify my need to every time i add a node having to set a container per node so there's going to be some use cases where you're going to want to override that and you

can do that you can override these settings per node but if you know that the majority of the nodes are going to have the same container you can just set that up here so um i'm going to go ahead and i'm going to import the this image is uh what um greg mentioned earlier that we have some images ready to go on docker hub under uh werewolf namespace so i'm gonna go ahead i'm gonna bring in the rocky image now i had already done this before so you notice it went pretty quick um it was cached but right now i'm bringing in that rocky container

rocky linux okay and now it's building my container and then uh the next thing i want to do is as soon as it's done building is i'm gonna set this uh as my default container and i believe i probably could have done that in one command line um i probably should have done that to show you but i believe there's a flag that you can add to make that your default so instead of having two separate steps yes so it must be i saw i was getting the confirmation there so i'm gonna go ahead and set this container as my default container on my profile

so now if i list the values i've just set i now have my default container i have my default netmask for for that interface um the other thing i'm gonna do um is um i've already done this uh i'm just checking here the next thing i'm going to do now this is something that i know i need to do uh current with the current uh rocky image that was brought in um i'm going to enable the network so that when i boot up my nodes um it will automatically bring up the network interfaces um this is something actually i think is being worked on to

so we don't have to do this step but in my testing for this demo i saw that i needed to do this so i can actually now that i have the container set up and built uh here in werewolf i can actually go inside that container so it's just as simple as doing a wctl container exact um specify my container name and then i'm just gonna do bin bash and now i'm inside my my container here um and from within here i can just do a simple system ctl enable network it's going to create the sim link necessary and i'm going to exit um and

it's going to rebuild my container now the other thing i don't think i have time to show that today but the other thing that we do is you can go into your container you can bind in a directory and you can install uh packages so if you wanted to make sure uh for instance in this case rocky linux was up to date you can update the container by just executing into it and running a simple yum or dnf update um or you can install packages like i use it for installing our fuzzball substrate packages into containers um and then once the container's been rebuilt um

all you have to do is reboot your node and it will pick up the latest container um so i have that network set um the next thing i'm going to do is i'm going to also set our container our kernel um as well so what we're going to do is we're going to import if i show you right now let me clear my screen see if this makes it a little easier um we do a kernel list okay we don't have any kernels on the setup so i'm going to just go ahead and do a kernel import now this is going to import this command

i'm running now is going to actually um import uh the kernel that's been installed on the werewolf node so that's why so i'm going to just run this and it's going to bring in that kernel you can specify other kernels but i'm just going to go with the one that's currently running here on my werewolf node and you noticed i was able to set that as my default so now when i look at my my profile i now have not only my container but i now have the kernel that's going to boot up on those nodes so that should be it as far as this

setup is concerned now we want to set up our nodes we want to see some nodes boot so to speed things up i'm going to just uh right here i have one of my nodes um let me do the first note okay so this would be the command that we want to run i already know that my network card that that's its mac address there it's hardware address so with this command wwctl node add again i need to refer to which node net device network device we're going to use i can set its hardware address i can set its ip address and i'm going to name i can name my control however my node however i want to do it so in this case i named it control 1.

now um what's going to happen possibly is i'm going to go ahead and add that node and i didn't want to miss us seeing it actually um pick up that note so i'm going to start adding my other notes here right here so this is my second control node and then i'm gonna do uh my other one here i've added my three nodes one thing after you um let me just i can check um and take a look here and here you can see uh all three nodes that i just added and you can also see what is being set for them for instance you

notice they're going to pick up rocky8 because i set that in my profile my default profile they're also going to boot up this kernel uh here is my hardware address here is the ip address and here is um the netmask that came from that default profile and that's happening see if anything happens greg may have noticed i see if i missed anything um it didn't pick it up let me try a ctl overlay build we didn't get into um overlays or or anything here and if that doesn't do it i will try to re restart um you know of course when you're live right of

course absolutely doesn't matter how many times you tested it before it's just gonna fail yep not fail hiccup funny thing was uh i had him sitting here booting and i did it and immediately i saw oh this is gonna be awesome to show show everyone let me go ahead i'm gonna try to reload werewolf just to make sure while michael's doing this guys if you guys have questions go ahead and start dropping them in chat we'd love to hear from you also tell the log um yep yeah restarting the service um this was fixed in a later version to where you didn't have to reset

the restart the um the werewolf service or reload the werewolf service so hopefully that will do it see how it says it's an unknown unconfigured uh node right i'm just gonna speed this up instead of waiting the full minute there we go now i can see it's grabbing its image so that must be ways i'm i'm using werewolf 4.2 uh for the demo right now uh it's been a while since i've actually played with a production release i'm always playing with like the development um versions yeah so that that's actually been optimized so it automatically will reload um the configurations right yeah i i kind

of recall seeing that get fixed and uh i was gonna try to be have a dangerous demo here and run the latest development branch and then i said uh maybe not there's a couple subtle differences in the new version obviously this is being one of them but another version another difference is the kernel is no longer a required attribute the kernel is going to be taken from the container itself so the container that you imported actually has the kernel within it and this makes it super nice for us to be able to say well we're going to include in this container all of the infiniband

or gpu support or anything else that you need inside of that entire container so it gives us the ability to package up into an oci container the entire stack so um that was pretty much uh the extent i want to show there's a lot more obviously you can do here um but i just wanted to demo how easily once everything is configured and you saw it didn't take too long to configure the profile configure the nodes once you have that defined it's even faster you just bring up nodes and you're just adding their hardware address in a sense or you know that way you can

uh provision it and and then you're in uh i can you can actually see that the network is up and running here um so uh that is it i mean there's more to this but obviously we're we're short on time so um don't know if there's any other question anything else you want me to show uh greg while we're here if not i'm i'm done thank you that was an awesome demo one other facet as well is uh werewolf actually auto discover nodes as well so if you and you set up a node configuration and it does not have a mac address configured and you

set the discoverable flag in the node config werewolf will automatically as an as an unknown node will join we'll we'll check in it'll automatically just put that in the first available available node slot that is that is there and it will inject the the mac address into that node uh werewolf will also do ipmi uh so there's a whole group of power status commands power cycle power you know everything even um serial over lan sol support inside of werewolf so you can integrate directly into ipmi um too and werewolf can even configure your ipmi or actually actually i should say try to configure your ipmi

due to many vendors having slightly different versions of ipmi and how ipmi has is supposed to be a standard but everybody's adding different kind of capabilities and and things and it's not an extraordinary extraordinarily stable standard it does the best it can to configure ipmi for you and most systems it just works out of the box but um it will it will do the best it can and ipmy works pretty good on one of our customers they're using it quite a bit um and i was even able to set up a virtual ipmi on a kvm machine uh and do some testing that way uh

bringing up and bringing down kvm machines or get you know guess very cool so real quick greg i think we talked about this last time i just wanna make sure we talk about it again i think as a part of version four werewolf there's there's things that are happening or things that you're working on or the community's working on around moving things to apis would you like to brief it on that real quick yeah so um right now werewolf uh is a non-client server architecture with regards to the configuration and management of it um we are changing that so the werewolf server is going to

end up being kind of the uh the entire control plane for the for the system the cli then is going to interact with that werewolf control server over an api this gives us the ability to add additional features around that so for example we talked a little bit about the hpc 2.0 stack fuzzball is actually going to be able to send provisioning commands directly to werewolf without having to shell out everything will go over apis gives us the ability to also create nice graphical interfaces around a werewolf and those capabilities that process and that work is already underway so we are expect to see that

probably next this quarter q3 and we should be bringing that out at least for early early beta by about then and probably stable release probably endish of q3 excellent i think we have a question question just went away you said you had a good answer to this one uh can you remove the kernel parameter from all profiles uh the i think the questions in regards to they probably saw when i did a node list there were some default kernel arguments there yes you can actually override that you can set your kernel arguments that you want for that for that node in fact we have been

using the kernel arguments there for um like deploying kubernetes to use c groups uh version two so on rocky or or uh on rocky we wanna set that so that it enables c groups uh v2 and so we just we can override the kernel arguments so yes that's definitely configurable there i'd actually also answer this in a different way as well um i don't know if if nick was referring to the comment that i made regarding the kernel being inside of the container once you move to 4.3 and if you're using a kernel inside of the container you would be able to remove the kernel

parameter and i believe how it's currently stated it will override the kernel that's in the container if you don't remove it so the kernel argument is actually becoming a kernel override argument so if you specify your kernel or kernel override parameter it will override what is inside of the container and um but if you remove it and if there's a kernel inside of your container you can then remove it safely and werewolf will boot the kernel that exists inside of that container i think that answered next question yeah he was talking about colonel carter marks cool thanks for the question nick so last thing real

quick guys before we have to drop off uh ciq value ads so what are we doing to help with werewolf or what are we providing on top of werewolf to make it better i'll jump in here and then michael if i miss anything that you think of uh please fix or or correct or add to what i say um uh one of the things that we're working on um is fips compliance so you're gonna see a fips compliant version of werewolf coming out um probably q3-ish which is kind of a cool feature set of features to have especially as we're moving towards an api based

and we can then put guise and other sorts of interfaces on top of that that compliance is going to be able to guarantee and and validate the crypto uh that we're leveraging so you know you have a nice secure system um we're also going to be creating uh pre-configured and we kind of already talked about this a little bit but pre-configured node uh containers profiles um turnkey solutions around werewolf again traditional hpc as well as hpc 2.0 and fuzzball uh so we're definitely going to be bringing those uh making those available and then support and then services around that so that could be anything from

you know helping with uh solutions architecture helping with integration or just just support just if you have any sort of issues you can give us a call one of the things that we're doing and i know support is is totally not a cool subject to talk about because it's been done for the last i don't even know how many decades um but um one of the cool things that we're doing about with with support is actually focusing on the people focusing on supporting people not supporting cores sockets nodes or entitlements and that support model is actually really cool because again we become the escalation point

for people and teams within organizations and we don't care how many nodes or how many systems or cloud instances or containers or vms that somebody has to maintain that's their job that's the responsibility our responsibility is to make sure that they are successful and they have the means to get questions answered whenever they need those questions answered so that's our support model that is offered for not only werewolf but also singularity apptaner and rocky linux and if so if you have any questions if this is something you're interested in please do reach out to us and and contact us michael anything i left out that

you wanted to add i mean we're pretty much uh active on you know it's open source um so we're involved in that community so a lot of times there's a slack channel so a lot of times um us you know we're here at ciq we are keeping an eye and we help answer questions that way as well so uh or if anybody has an issue sometimes we're we're immediately looking into it ourselves um and trying to add um you know value in that way that we you know are helping to uh helping everyone to be able to do what they need to get done excellent

thank you and i see nick had one last question so he's asking about are the api's going to be version so it doesn't break everything the apis are built by um good engineers not me so yes it will definitely be versioned thank you for the question guys i think we're actually at the end of the time i want to thank greg as always for being here michael thank you for joining thank you for the demo glad it actually worked out it was fantastic and i really appreciate the time you put into it so thank you very much thank you for joining guys again don't forget

like and subscribe and we will see you again next week oh oh oh oh wait me me in the front quick quick point um we are hiring we are scaling the company up if if anybody is interested please do check out our careers page and reach out to us we'd love to talk with you there now i'm done excellent again appreciate it thank you very much thanks again michael thanks zane appreciate it thanks michael bye