Research Computing Roundtable - Turnkey HPC: Warewulf & OpenHPC

view it also as a software distribution i mean it's not low level like linux but what what it does is on a basic linux system it's going to sit on top of it it's going to provide a full collection of of hpc software including the provisioning system as well as the resource management and most other components that are needed and if i have uh i can't remember if we've got the timeline i don't have the timeline handy um but just a quick background uh it was originally launched in 2015 um with 1.0 uh the most recent update or major update was 2020 uh when we

launched 2.0 uh supporting uh at the time centos eight and celeste 15 or uh opensuse 15 uh it has now been uh updated to support uh rocky8 but that's a quick background that's great thank you so i know one of the pieces that's involved in openhpc is werewolf so greg created werewolf a long time ago i think we've talked about that before but how how does werewolf fit in the open hpc world greg i one thing that's a question for you well one thing i wanted to mention even before that is uh early on like when open hpc was just starting i was on one

of the technical i think was the technical steering committee jeremy you may remember better than i do um where in the organization but i was brought in as a representative from uh department of energy and berkeley lab and just interestingly gary was my boss at that point and approved my time to be spent on openhpc and uh and to help with the project so i mean i go way back with the project as well and i think i was invited in at that point uh because of the uh the interest that openhpc had in leveraging werewolf for that provisioning and uh so it's it's been

it's been fantastic to kind of been part of the project early on and then again just kind of watch it just take off and continue on so um very very cool now zane um completely forgot at this point what your question was asking how does werewolf fit in with open hpc like what role does it play in in that stack ah oh i accidentally just kind of touched on it or yeah so so uh werewolf is um an operating system provisioning and management platform uh specifically for clusters uh it was built around the idea of high performance clusters making it real efficient for system administrators

and engineers to build and manage their systems there are people using it outside of hpc but hpc is definitely its dominant use case openhpc has leveraged werewolf as i'd say the primary or one of the primary provisioning solutions that it uses within uh openhpc and um and it's been really focused on werewolf version three for pretty much the whole life of of openhpc and uh more recently we've started building towards werewolf version four and uh so i think in this conversation we're going to touch a little bit on werewolf version three versus versus four the benefits there and and maybe even kind of where the

open hpc direction there but uh but that's really where it's focusing within openhpc openhpc has a plethora of different libraries and applications and um and middleware and solutions really to make turnkey hpc as easy and reproducible as possible and they've done an absolutely fantastic job at that and again werewolf is really just sitting on the how do we help provision and manage those images on compute notes so i think one of the things we hear quite often when we talk about werewolf is images and just real quick jonathan how does werewolf handle portability and flexibility of node images so this is something that's changed and

in my opinion drastically improved uh even over uh other previous versions of werewolves so historically you've created kind of a special directory a true directory that you've installed packages into and kind of handled it manually up front and then you could shell into that image and and make changes to it similarly to how you would if you were sshinging or otherwise logging into a system and making changes to it and then distributing that image to other nodes uh werewolf four um refactors this uh partly through vocabulary but also through a little bit of integration on top of the semantics that have come up through the

open container initiative through the docker project and and similar things and so now rather than having to you still can but rather than having to create that uh node image from scratch by hand you can initialize it from an ups an upstream oci container through a registry or through an image that you bring with you and to my mind that makes the whole thing make a lot more sense you can start from that and then continue to shell into the image and modify it the way you know if you used previous versions of werewolf or other similar systems you can keep doing that or you

can do what i prefer and never change that image just uh keep modifying your container your container definition file and then you can keep that in git or some other uh revision control system and in that way uh track the the version history and the changes to that image over time without having to keep track of this big binary blob or special directory that contains your image that's great thank you misha i'm going to pick on you next because i know that you're actually going through this right now you're currently a werewolf 3 user in your environment and you're in the process of going through

that change over to four so i guess the question is how different are they in your mind so um i think uh werewolf 4 has been a great work compared to level 3 in terms of how to manage the provisioning um werewolf three had some issues for us uh one of the big issue was the database that you need to keep everything in the big database and you have to keep those blobs like gigabytes of gloves in the database and even extracting data out of them and shoot those images to the nodes was kind of like a pain for us sometimes not always but i

think the new design is is a game changer at this point because we're not going to have that big database we're going to have a brand new way of seeing the containers which used to be a vnfs things in a werewolf three and moving towards container i guess it's a whole better idea as uh jonathan mentioned that you can have a track of things that you have done inside your container now the big problem we currently have is we have too many images for different systems for provisioning and after a year and a half we exactly don't know what's going on in those images so

we didn't have a good documentation sometimes people just keep adding stuff into the image fixing things inside the image there is no track of them now i think having a good docker recipe and then keep it somewhere for your organization or your institute and and try to get that recipe for you every time that's the best way actually you can track because if you're gone the next person can pick up and work on it now we haven't done it yet we haven't moved to verbal four we're still working uh jonathan is helping us at this point so hopefully within uh we're targeting the end of

this year to fully move to verbal four um and all the notes in our cluster provisioning from variable four that's great thank you misha gary i've seen you nodding your head do a lot of this i know that you've had experience with with werewolf 3 as well so what do you see is the differences and how is that impacting you well you know um what i'm going to say is we're we we haven't used werewolf 4 yet um but we're i would say we are at werewolf three and a half and thanks to greg because since greg was at at the laboratory we saw a

lot of early thinking for werewolf iv in in in our version of werewolf and so someone was mentioning for example uh that uh you know the way you would you know cd into this directory and build your vnfs you know early on we uh realized that um that only the person who did all that work understood what was in that build and so we really had no way of reproducing it in a in in a easy way so when singularity came along that greg invented too then uh we started using singularity to manage our builds so it was kind of like a early form of

using containers to manage our vnfs builds so um so that that we we still do it that way so it's almost like a precursor to i believe what what we will be seeing is werewolf four we're looking forward to that great thank you jeremy from the openhpc perspective what do you see the difference or how they impacted you oh i can't hear you anymore sorry jeremy there we go off mute sorry about that um so the first the first major change i i think came in when we were doing some internal migration to uh open souci 1504 there were some changes in the structure of

where there's some changes in the way uh opensuse 154 compresses uh the modules and it's not compatible with werewolf three world three in its underlying structure requires this intermediate boot layer and that boot layer has to by design be able to support the operating system that it's that it's booting uh the beauty of wearable four is it jumps directly into booting the operating system so we can bypass all of those requirements and everything works great and so uh internally there is a move and a push even before it's available in open hpc to move into uh using world four because it has a lot more

capabilities so in the recent uh tsc um two weeks ago we had a discussion on on werewolf four um it is going to be moved into um it'll move it'll move into factory first uh in fact i just put in a new pr for a an updated build yesterday to include the uh the 4 3 source plus a few patches from commits and so i expect that to be go up to be available relatively soon but the answer the question is there's two options that we have if we continue supporting uh susie either we have to update werewolf three or we have to move to

werewolf four and i think the the clear the clear channels move to world war four great thank you glenn you've been playing with this lately in some interesting ways and kind of getting back integrated with it and playing with it what are your thoughts yeah it's a it's a big step up from uh from three to four um as jeremy and others have mentioned that the uh being able to have a dockerfile for your container as opposed to just creating a route and there's no record of what people have put into the truth you know for example to to know what's in there um i

think that's a it's a huge huge step forward and um as jeremy mentioned too there's no shim right so you could it just you know boots up your you know your kernel and your in your and your file system right away in a stateless manner so uh it's a little bit more agile a little bit quicker booting and um yeah and so a little a little bit more streamlined i think it's uh a really good really good step in that direction and i've been able to use it with uh uh with ubuntu and debian and you know a bunch of distros so it's it's been

really good that's great thank you forest i'll come to you last on this question they've been playing with this for a while as well yeah i've looked into it a little bit um i would say that for the most part i'm excited about what glenn is similarly excited about um i really like the reproducibility of the container being a central focus of it now beforehand as you mentioned with truths and stuff it could be a little bit difficult to keep track of exactly what went into that and that's you know something common that we talk about with those sandbox type containers that you could have

some original definition but then as you make modifications to it those could be potentially lost so i think that you know with one of the main goals of containers and that type of thing being reproducibility um one thing that i'm most excited about there is yeah the ability to integrate just containers from definition files directly um so that that can all be tracked a little bit better great thank you and i think we've touched on quite a bit of what to be excited about but jonathan i know there's one thing that's really interesting and you've touched on a little bit or someone did but talking

about overlays can you kind of explain that a little bit a little bit deeper yeah absolutely this is one of those things so the fact is i had heard of werewolf before you know before i came to ciq i was familiar with werewolf three it was something that a past colleague of mine had mentioned but i've never really used it and when i so my first kind of hands-on with it was with werewolf four and i i don't even know how to imagine running a system without the overlay system from werewolf four and then even 4.3 i came in right before 4.3 was released where

you could take so i'll back up a moment so overlays allow you to bundle up configuration files uh into they end up getting compressed into an image that gets shipped across to the node to customize your your node's state at boot time and then separately um on on some kind of interval by default it's once a minute and you can have different overlays for different nodes different groups of nodes and in 4.3 you can have multiple overlays that kind of get layered and concatenated together rather than having to have a single overlay image that goes to a given node and this just makes configuration management

such as it is in an image-based system just really and glenn said earlier really agile really straightforward it just kind of does what you want um and it's it's been really great to have access to um i i know there was you know a way to do this before uh but the the one that's in werewolf four and especially 4.3 is just so flexible and really easy to express the kinds of things that you're likely to want to do in a clustered environment like this that's great so greg the kernel question comes up quite often about kernels different versions of os's on things so can

you talk to me a little bit about how how easy or difficult is it to deal with kernels in werewolf four so let me start on this in a little bit of i'm gonna start actually with werewolf three so where three in every version of werewolf previous uh handled the node provisioning in two there was the the the operating system like the runtime user space portion of the operating system and then there was the bootstrap now the bootstrap is basically the kernel and the initial ram disk and initial kernel drivers that you need to kind of get the stack up and running but that was

all kind of tied to um tied to everything but but that was the kernel and and that bootstrap now you kind of had to play this mix and match game between what kernel are you running what drivers are in there and what is your user space going to sync up to or match again so this gets really complicated when you start thinking about things like gpu stacks you start thinking about lustre uh and and other things you know whenever you have a kernel component and a user component that has to match you actually had to do this kind of by hand in in previous versions

of both now where will 4.04.1 and i think even 4.2 all did it exactly the same way as as everything did in the past right we had a bootstrap uh and or kernel and then we had uh our user space component of the the operating system this made it very again difficult to go and just install like gpu drivers uh because you had to install the user space component into your into your your user space your portion stack the container or the vnfs and do it on the in the kernel as well so again very kind of disjoint where will 4.3 change that to the

point where the entire kernel stack exists within that container so this is really important as we start thinking about especially in my mind things like openhpc openhpc includes gpu drivers it includes luster and includes other facets that you want to make sure that you're managing kind of a close relationship between the kernel and your user space now with werewolf four we've talked about this a little bit there's some key differences if you're already used to playing with werewolf three and you've been using werewolf three or previous version uh there's some key differences and one of those key differences is uh we've kind of swapped out

the notion of the virtual node file system the vnfs for a container so this means and we've touched on this a little bit here so this means that we can basically take you know a dockerfile a singularity or an aptainer recipe or whatever build a container then import that into werewolf so that container build process could be manual could be a true root if you want or it could be coming out of a ci cd process where you're constantly managing and and controlling who has access to that what's going in there and you can share it between many systems so openhpc and one of the

visions that that i originally had with this and jeremy and i were talk talked about this now for i think years at this point jeremy is how do we take uh whether that be open hpc whether that be a custom image whether that be this image that somebody can expand but how do we take that image and make it as easy as possible to get it out to compute nodes so with werewolf 4 you can literally just import the open hpc container it includes the kernel it includes the kernel drivers it includes your gpu stack it includes luster if you wish everything you need you

just import it directly into werewolf and then you literally just start telling okay i want this node to run open hpc this open htc container i want this node to run that this node to run it or you can say i want this thousand nodes to run it and change it in just one configuration real quick as well i mean the the the configuration for werewolf has also changed it used to be in a database uh this has now come up i think at least once or twice that we've kind of eluded um now it's in a yaml configuration file uh and we've we've been

able to scale out that yaml configuration file um to north of 10 000 nodes just to make sure that it was a scalable uh interface for that now if anyone's running 10 000 nodes with a single werewolf controller on a flat broadcast domain i think your network team would probably like to have speaks with you a lot why that sounds like a great idea until it's not yeah exactly so misha i know you're going through this right now and kind of like greg said you're moving from three to four and things are are different in trying to make it easier what parts of that are

you looking forward to being able to do other than just getting away from the database i mean are you going to have those multiple images that's something that you're going to utilize oh yes so a couple of things which is kind of very important at this point migrating from level 3 to 4 is a very big process and you have to consider that your your your system is actually under production and you cannot have level three and four on the same system and even on the same network that's what i learned recently that we have to kind of like there is a conflict between the

dhcp on on two different nodes if you want to run werewolf three and four and you should be able to manage that now the things that we're looking forward at this point is to come up with the right design and implementation of the new werewolf 4 on our system and then we have to start building new images for different systems and the reason we need different images because we have different networks and sometimes uh we have different architectures and because of that you have to build some of the drivers install some of the drivers for omnipath is different from infiniband for instance and you need

different images for that and that that requires us to have a plan ahead of time and and migration from level three to four um is a long is it kind of like tedious job to do so we have to make sure that we're not mess we're not messing up with any services on the system at this point that's right thank you misha so jeremy on openhpc i mean i know there's a lot of things that are coming now but what else can we expect to see come out of the open hpc or what things are you really focused on right now in that environment or

in that that project uh so i'm focused on a couple things and uh the main one actually is is werewolf four um and i'm also looking at updates to uh overall to support uh the new uh latest open soucy builds uh so in terms of what uh is available that would actually be a good uh good discussion i wish i had some had some slide decks talking about some of the things that are coming out but um in the next builds will typically be updates to uh support the latest uh operating systems um like i said one of the pushes i'm certainly making is to

support opensuse 154 because it has been out for a while uh we also had an agreement to move the tool chain from um what is currently gcc nine four or nine five uh that be that will be moving to twelve point one um so there's definitely some there's a couple updates coming out um one of the things before uh on a on a side note one of the things i actually want to talk about when we're still talking about the the migration from world three to four and uh we haven't touched on it and that is security um one of i think one of the

key elements that we added with werewolf 4 is the ability to secure your provisioning system because this has always been kind of a sore point with with werewolf three um being fairly open for other systems to uh to get in um but there's a couple of specific features that were added to including the ability to basically wait for a uh you could actually store uh correctly explain this better you can explore a uh the ability for uh uh werewolf for to look for specific information coming from the node before it will actually send a send an image down to it um and so one of

the things that we're looking forward to is using world four because of the the additional security features no it's great greg i mean if you want to dive in a little bit on and talk about security in world before i guess that is something that i just kind of glossed right over and didn't even didn't even bring up so i think that's important this has been a general problem of provisioning systems since i think forever uh which has basically been if you can pixie boot a node and that node can start pulling all of the necessary bits layers and information and configurations from a control

system then what is stopping a user from doing that and if the user can do that is there any material that we are sharing with the compute nodes from the control node that could be controlled um anything or should be controlled i should say anything that might give them the ability to gain insights or or data on how to exploit and gain privilege within that system so werewolf 4 actually does a couple things you know fairly different the way that people typically manage this by the way is they have an out of band provisioning network and they use that out-of-band network just for provisioning and

management and when the the node operating system comes completely up that that uh network is not available it's either vlan off or it's not configured and you actually have to get to root to actually you know set up that network but if the users are not root they can't ever do that so you kind of end up with this um this this level of security that requires root to get root uh so usually usually that's good enough but not all systems and not all architectures are capable of building out-of-band provisioning network or fabric so we wanted to do some things that further enabled the security

and made it so the only way you can actually get this data get the provisioning data configurations and whatnot for your node is if there is some sort of you know secret key uh that we can leverage on the hardware and we ended up using uh asset keys for this and so most platforms support asset keys and you can put in a free form string in your firmware of your system for an asset key or leverage a random asset key for that can add that to the werewolf configuration and werewolf will only hand out those bits if those asset keys match and now a user

who's sshinging and coming into the system without physical access cannot actually get to the bios of those systems and don't have root won't be able to get those um asset keys so this gives it this gives us a level of protection which honestly in provisioning we have not really had before at least not in werewolf oh that's great so i wanted gary and saw you shaking your head again on some of that and i think security is an important aspect i want to give you an opportunity to talk about from your perspective the security around provisioning um you know that it's um we're glad to

see that addressed uh we actually we weren't all that concerned of the fact that somebody if they knew that obscure path could get that data off the masternode until we recently put into production a secure computing and research data platform for the for the university which will handle pii and hipaa data and and then when we had an outside firm come in to do the um uh evaluation to do the uh certification to 800 171 which is a uh nist uh standard for sensitive data um then it became an issue and so we had to figure out a work around for that so the fact

that we would be able to not worry about that with werewolf four is going to be great that's great thank you so jonathan i'm going to pick on you again because you've been playing with this quite often later quite a bit lately and just kind of from uh how you've been dealing with it and playing with it can you just give me what you see coming and kind of uh i don't know open it up and let you guys talk about whatever you want to what do you what has this been fun for you to deal with i don't know that i i don't know

that it's kind of broad um it is broad i think werewolf iv is great and i've been trying to get more people to use it so uh i i think that you know broad applicability in the the hpc space and especially the academic hbc space where i come up i i can't imagine wanting to do uh hpc provisioning with a different tool right now it really does seem like the best uh the best of breed um but i see opportunity for it outside of that as well i've been thinking a lot about um provisioning storage clusters in particular uh with werewolf um and uh i

forget if it was mentioned here but we've been talking a little bit about what it would look like to provision luster systems with it and i have experience with bgfs in the back as well i'd be interested in trying to to spin up that kind of a cluster um with werewolf but uh also just putting up libraries of of starter containers and uh so you know the work to create that container file is pretty minimal and especially working off of the examples that are already available in the werewolf um uh in the werewolf docker hub kind of space or account or whatever you can see

how we did that with rocky and and kind of replicate those experiences with your own container but there's no reason why we couldn't start putting up different versions of containers i have a small uh pile of containers to support like the melanox offed and infiniband using the built-in support with rocky and supporting omnipath that kind of thing um the ability to replicate those experiences go beyond just your own ability to replicate what you did locally once you can replicate it someone else can too and uh i i think there's as the community grows and i think that will only uh only happen uh more quickly

when where will force shows up in an open hpc more people will be using it and there will be more opportunity to share containers that people have had success with and i'm hopeful that we'll find a way to do that with uh with overlays as well um there's there's nothing in werewolf for that right now but i think as we see that system become packages of things that can kind of be composed together um we might see a use case for being able to to grab those out and export them and import them too something i want to jump in with real quick oh i

didn't mean to cut off the question um but i i i am kind of you know excited about what this looks like now to provision systems to provision open hpc systems or anything is the one process that you could use uh is you you go install whatever os you want as your controller kind of main operating system you install werewolf into that at some point hopefully we'll get into apple and and other things as well so it's super easy to do and then you just start importing containers so imagine if you want to go build an open hpc system you just go and import the

open hpc container from their documentation like here's our string here's the container it's pre-made if you're running this architecture here's one if you want to you know hear all the different variants of open hpc that you can choose from whether you're doing a a i o node whether you're doing a controller node whether you're doing a a compute node you can now go and just select the images that you want or you can use other images that other sites uh may have been basing on and you can just leverage that you can import that into werewolf you configure your nodes at the werewolf command line

at the wwctl command configure things like your your ip addressing your nodes um any other information you want with those nodes etc configure your overlay if you need the default overlay will probably work fine for pretty much everything that you need to run and you just go turn those nodes on and within literally minutes uh you know with a quick how-to you can actually be setting up and running your your cluster now we've done this at berkeley lab uh with a number of clusters and you know there's one story everybody already i think who knows me knows i like to tell stories there's there's this

story um of you know we built in this this uh geophysics clusters a while ago we built up this geophysics cluster and um i don't remember how many nodes it was but it for for the day and age it was a good size system i think it was like six or seven racks or something of of of compute it was a good size um and um we were integrating this and we had we had dell there we had the ncsa there and all these people there and whatnot and we first validated using the standard image that that dell and csa wanted to leverage it took

us i think two days to get the software stack up to get everything running and we validated it so we can get the system approved and um then we said okay well let's go let's go now re-spin this using our supported os and everyone's like well we can't stay for too long so we got a rush it's like oh this won't take long so but our plane leaves and another like you know we got to leave here in like another hour it's like yeah this won't take that long and i i think within the hour we not only had the whole system reinstalled reprovisioned using

our standard stack that we're using everywhere else in a completely supportable way but we actually ran the scientific code as well and validated it and the funny thing was is we actually got better performance and you put this whole thing together and it was just like it was so and from that point on actually we had a great relationship with dell and the ncsa and everybody else that was there to watch this um but that's the kind of experience we really want everyone to be able to see and be able to leverage and i mean imagine to be able to say i want an open

hpc cluster or i want this cluster or i already have this cluster running but you know what i'm going to go check out openhpc imagine that you can do that within minutes and reprovision your fabric reprovision your stuff and and test out different node images test out different operating systems for upgrades you know we're you know enterprise linux you know 8.6 over here but nine just came out we're gonna go try rocky nine um and you can import that image you can add in all your overlays add in everything you want on top of that image and literally it's just okay well i'm gonna go

reboot you know these five nodes with it and go start testing some jobs with those five nodes and um and you can actually start testing and validating this uh in a very sane way so that's kind of some of the benefits that werewolf offers and i think that the synergy between werewolf and openhpc is going to make just the idea of turnkey hpc which is kind of the whole thread of what we've been talking about in these uh you know research computing roundtables i think it's really just going to make that whole turnkey approach and turnkey aspect just so easy for people to deal with

and to get solid reproducible systems that have everything you need on it that you can just go and run with it anyway there's my there's my big spiel jeremy was i totally off on like what we could do with open hbc i mean i know we've talked about this it's close yeah no we can we can do quite a bit we've got we have open hpc running in fact that was when you were talking about the speed at which werewolf iv can operate um and i something i never thought of doing uh but we've tested we tested it internally and we actually have systems set

up where we template we templated the image so we have a base node image that operates as a template and when a job is submitted it takes that template copies it updates it and sends that out to the compute nodes and when that job's done the whole not only is not only are the nodes cleaned off but the entire uh node image itself is wiped out um but you can replicate and update that node and it happens in less than a minute i i wanted to jump in with something absolutely gary yeah about greg's story you know i was there the whole thing happened exactly

like he said um the uh the interesting thing that i just wanted to emphasize is the reason why we did it twice once with the dell stack and once with uh werewolf is because uh you know a lot of people when they buy or do procure clusters um if you say well i want this hardware i want it this way but we want to run our software stack we want this level of performance if they don't know what your software stack is then a lot of times the vendors are going to be reluctant to say give you or do it agree to an acceptance test

using your stack uh because they're not not familiar with it so in this particular case we said fine one you qualify the hardware with your stack and we can make sure that that works out fine and then we're going to roll our stack on it and then we can make sure that it works you know with ours and so that that was the reason for the back-to-back test but it uh the fact that you can provision both uh right next to each other uh makes it so that you could do your acceptance and then you can go on from there that's great and before we

throw up i think so we had a question before we throw that jeremy is there going to be we've kind of alluded to a little bit about different uh container definitions is open hbc going to host some that you guys have as a defined this is kind of the base that does this and then you host those so that people can just go download them uh i don't have an answer for that that's something we should we should look into there is an open hp container project um but that's a it's a good question and i don't have an answer for that that's great thank

you now we get to sobe's question so the question is what operating systems can't be provisioned with werewolf i know we touched on several of them but i will hear me i will throw it back to you well i have uh i i've personally provisioned uh centos uh rocky uh red hat proper red rel uh slazz open susie i know you can provision ubuntu with it um question back i don't know if anyone has experience with provisioning uh older version of centos sent to west seven i assume that it would work um i think you can go i think you have the ability to go

quite a ways back it's it's because of the nature of the way werewolf four is written it should be fairly it should have fairly broad support for most linux distros that's great i know glenn you've been you've been touching some not typically what you would run across in a normal environment type operating systems lately yeah i mean if there if there's one thing that i'd like to see the community do more of its posts uh containers for you know ubuntu or debian or you know whatever distro they're working on a place where we could gather all those because the like the werewolf uh the werewolf containers um are limited to like cinder west 87 and rocky 8.

so we want to we want to get some more up there but um because the the adding the kernels to different distros whether it's you know ubuntu or or rocky uh slightly different you know and getting the modules in there slightly different so you know just those things to smooth out those speed bumps for people would be i think would be uh really helpful actually i'd like to try things like you know uh tiny linux or pi pi core or whatever you know all sorts of others really really different things to see if uh if they would come up as well just just for fun

you know we learned something really interesting as part of this whole thing of trying to boot random containers up in like docker hub uh is that most of the base containers have actually been rendered unable to boot out of single user mode so there's a core utils single users a bunch of system d stuff that has been turned off and and optimized for container use if you go and look at inside the werewolf source code we got a directory right off the root called containers uh and then go into the docker directory and you'll see like look at the rocky 8 one you'll see how

many things we had to change in terms of the default or the stream base container to to get it to properly boot uh so there's a couple interesting things there so but being able to i mean the theory of being able to pull any container out of docker hub and run that container on bare metal is actually pretty neat and there's again there's just so much you can do with that with that piece right there um yeah anyway that's what that was the point i wanted to bring up there another great thank you so this is a good question so other than already running version

three are there any reasons to hold off on a werewolf four and i know misha you kind of touched on this a little bit earlier and i think one of the things that you're working through and looking at right now is the state full in version three versus state lesson version four so something like that i'll give you i'll let you go ahead and answer that and then i'll throw it up and everybody else well actually i got a good answer from jonathan a few days ago that we don't need stateful anymore so i think i'm pretty much convinced that stateless is going to work

for us um like i said uh there are a couple of things that you need to consider when you move from level three to four the first thing is if you currently have a management note for your werewolf three you cannot install variable four at the same time on the same note so you need different note and you have to make sure that you're not going to mess with the network because the way that wearable 4 is handling the dhcp is different from level 3.

that's what i learned so far now because of that we have this on and off option of dhcp during the time that you want to provision a few nodes for the test and once you make sure that everything is fun and you have to fully move to the verbal four to take care of the things for you meanwhile you have to make sure that you can handle the dhcp uh on both server nodes for for your werewolf which they're hosting the wearable for you there are a couple of other things that are going to happen when you move to the werewolf four one of the

thing is uh uh the uh public and private key that level three generates for all the nodes is going to be changed it used to be i think is it dsa or rsa 24 it's going to move up to rsa at 2049 i guess which is a kind of big change and more secure so your previous keys are not going to work you have to make sure that you have a brand new key for the werewolf four and uh pretty much that's it now but there is one thing that's i guess still you can do uh greg you can correct me on that if you

car if you if currently you have any image that you shred and then you make changes and then you create vnfs out of that you can still import them into the wearable four it is possible it's not it it doesn't have to be exactly the container so you can bring in your old images into the variable four but again at some point you probably want to move to the containers anyway yeah so with werewolf four you can just point it at it to root directory or vnfs directory and um and import that into werewolf four so it doesn't have to come from a container it

could come from an actual to root but you have to make sure that you have the kernel installed on the sheet so have to is a strong word you should you don't have to because werewolf four does support the ability to do something called kernel overrides and this basically means that it can operate the same way that werewolf 3 and previous versions did by overriding whatever kernel you may or may not have inside of your your container or vnfs it'll just override that and just say ah you just imported a kernel into werewolf four i'm just and you've configured that for a node i'm just

gonna use that it's very helpful for doing testing of different kernels especially because you can import a kernel out of a different container so let's say i've got container a and container b imported into werewolf container a has a newer kernel that i want to test i can actually take the kernel out of container a import that into werewolf and then apply it to container b so it's very very very flexible and not to make it sound confusing there's just a lot you can do with it so so what is sorry jonathan go ahead i i wanted to to give one more point to the

the question of is there any reason to hold off of of werewolf four and from a a functionality perspective i i don't think so i've actually had to go back for uh another reason and install werewolf three and going back has just been it's difficult to go back uh but the one place where i do think werewolf four has some deficiencies right now is in its documentation there's been a lot of new features and a lot of new functionality added even just between the different point releases of four and it's been easy for the website documentation especially to get out of date a bit this

is something that we're actively working on and i even i individually am actively working on even today and yesterday but with that i think the werewolf community in the development community is a really good one to get involved with i have never touched golang before werewolf iv is also a rewrite of the project in golang but i found the the code base as it exists to be pretty approachable if you're any good at googling what different programming semantics do to find what you're looking for in the code and you know maybe offer some pull requests or help write some documentation if that's a way that

you might be able to contribute that's great thank you sorry i got a little sidetrack looking for questions um what i want to go back to jonathan i know that you are since you've been playing with it quite a bit i feel like i picked on you a lot today but i know when we look at the community and people giving back to the community uh yeah there's the question right there how is that important how some people how should people go about it so sorry sorry jonathan there's that there's thewerewolf.org you can get involved with contributing to the open source project which is hosted

on github under the hpc ng uh account there uh there's also a slack team uh for the hpc ng project uh that most of the community werewolf commute uh conversation takes place in um but yeah and anything else greg i was basically going to say the same sort of thing the community is extremely friendly it's very helpful we talk about a lot of things even more than just oh if we talk about open hb obviously we talk about you know basically just how to build clusters how to deal with clusters and um so the easiest way is just to join in the slack just take

part of the conversations join in introduce yourself and um and go from there the one i did want to mention i wanted to get props out as well to susa souza has just done such a remarkable job in terms of not only joining the community but just being such an active uh group within the community um and it's been super helpful so when we were talking about operating systems and whatnot and support uh sousa opensuse as well as celes very very well supported uh inside of werewolf so it is not specific to a standard kind of enterprise linux red hat enterprise linux type ecosystem it

is you know souza and celeste is absolutely supported and again they've done great so um you know christian is is one of the developers over there and is um uh now one of the werewolf um uh committers uh like jeremy and myself and others one of the other things i want to go back to that sorry go ahead i just also want to uh uh for anyone interested in uh joining openhpc project um openhpc.community uh also check out our github page the wiki is uh fairly extensive and uh again you can join the project you can uh if you have any open hpc open source

uh components you'd like to see added to it there's a formal process for making that request and we are consistently adding new new components to the the to the system uh or to the distribution um and and for example and in every summer we have uh other projects we do like our summer internship project so um if you are interested in joining becoming a developer for openhpc we would certainly appreciate it i'll be great jeremy we'll add those links to bottom okay thank you there they go one of the things i want to go back to before we run out of time here we talked

about stateless and stateful and i think that comes up a lot on the conversations that i have i think it's important to clarify stateful stateless and then it doesn't mean disclose right i like jonathan or greg because they're the two that we've been talking with the most on these two topics so yeah so first of all definitions what we mean um a a stateful provisioning system uh the one i'm most familiar with is a system called foreman will usually do something like automate a red hat kickstart or i forget what debian calls that but something that automates the usual operating system installation process to lay

an os down on local disks just like you would if you were installing it by hand just automated werewolf is uh an image based uh remote uh stateless provisioning system so you have that process having created an image and then ship it over the network at boot time and boot it um there are sometimes though desires to have state with the node and this is where i like to differentiate the idea of that provisioning process being stateless in that the thing that was provisioned is stateless and happens every time you boot the node from the node itself being stateless so you could imagine for example

one of the things we hear is that um people want node logs to rot be retained on the node if they're not centralizing them through some other means maybe they want them to be retained on the node um so that if the a node crashes they can bring that node back up and see the logs still on the node they didn't get thrown away when the node rebooted there's nothing preventing you from having a partition on a local disk on the node and part of what your stateless image does is mount that into the operating system so that logs get written to a local location

uh an even more straightforward use case in an hpc context might be having local storage resources that are available for maybe high iops workloads that don't need to be shared between nodes you can mount some kind of local storage um uh onto a scratch path that users can write into that retains state on the node even though the provisioning process itself is stateless but then you can even still consider a stateless node that still has disks so one of the concerns with stateless provisioning is that the operating system that comes across the wire is entirely held in memory and there's a concern that that's eating

into the compute kit capacity of your node because you don't have that memory available for compute workloads well if you have disks in your node or an ssd or something you can create a swap partition and mount that swap partition and that allows as that memory is required it allows the operating system to be flushed to swap which is an inverted process from what you would normally do but ends up in the same state that bits of your operating system are on storage bits of it are in memory depending on when it was last used and all of the memory is returned for active use

to the uh to the running workload so there's a lot of flexibility there and it doesn't have to look like putting an operating system down on the disk thank you for that jonathan so we are actually up on time i'm going to give everybody an option to give your closing thoughts or anything you want to add to this gary i'll start with you again oh we're really looking forward to werewolf four so we're already making plans to do that great thank you for joining misha i think uh we're gonna have a good result by the end of this year by showing some progress moving from

level three to level four uh we're excited to look forward for that thank you mission glenn uh just semantics stateful does not equal uh or stateless did not equal diskless and uh images did not equal containers very true thank you jeremy uh yeah well uh with the the work we put into world four i'm really looking uh forward to its inclusion in openhpc which should happen fairly soon and uh just a reminder if you're attending super computing um open hpc buff it will be will be there as well as the openhpc booth and uh i don't know if werewolves also having its own location but

uh anyway come out uh uh come look for us great thank you jeremy jonathan yeah so i i talked a lot about werewolf today but i'm also really excited about open hpc and being more involved in that community which will only be bolstered by werewolf four being part of it um i've gained a lot of value from using the hard work that's come out of that project and not having to replicate it on my own so thanks to everyone in that community and if that's one that interests you uh definitely take a look at it they're doing good work thank you greg i don't have

much to say that hasn't been already said i think this was a fantastic conversation i just wanted to thank everybody for uh for being part of this thank you it's great enforced sorry we kind of made you be quiet in the corner i'm sorry don't worry zane um i'm just excited to see uh where will four uh kind of come to wider adoption out in the market um i'm excited to see how it'll end up improving the capabilities and capacity of open hpc in the end and more broadly how it ends up improving the capabilities of the next generation of high performance computing that we're

all working on building so very excited to see how it works it's great a great way to end we really appreciate you guys joining thanks to the panel misha thank you glad you were able to join the first time looking forward to having you back again thank you for the interaction today thank you for the questions in the chat uh feel free to hit us up if you have any more questions or if anything you would like help with go ahead and like and subscribe and we will see you again next week thank you thank you you