Warewulf 101

happy to reply to you and uh get you guys going so Warewulf um it is not spelled typically and I always laugh when internally right so one of us miss spells Warewulf or get auto corrected for you I had to go turn that off seriously I think we actually misspelled it in in like a release somewhere as well and it probably was an autoc correct that's right yeah give people a hard time so yes Warewulf what is it uh Jonathan in the context of uh how we see it at ciq yeah so thanks Rose Warewulf is a cluster provisioning system it sometimes gets looped in

with the cluster management conversation but it's pretty lightweight and that regard so Warewulf is built primarily for HPC though it's useful in other contexts as well for taking a set of Hardware that you mostly want to all kind of behave the same and not have to worry about the state of the software and operating system on that Hardware you want one server that says how all of the nodes should behave and then you just want to turn nodes on and they just become a a node in your cluster so if we can real quick uh get the share that I've got running right now just

to have in the background I can demonstrate so this is a virtual machine that is not running and I'm going to start it up and this virtual machine is connected to my Warewulf server that is also this is just you know in a test environment but this virtual machine doesn't have any discs associated with it it doesn't have any capacity for storing local storage you can do that with Warewulf uh but you don't have to um and this node will just start up ask the server what it should be doing and get a rocky Linux 9 image completely transparently over the network uh and then

you know if this were a compute cluster or something make itself available for for use and we don't have to just yeah and and all in memory again you can do stuff with local discs but the whole point here is uh if something goes wrong with your node you should be able to turn it off turn it back on and it will be back in the state that it was originally without you having to worry about what happened to it while it was running so Igor did ask why it's called Warewulf and I don't remember the story yeah the the this comes from first it's

a wear because it's like software uh so that's why it is spelled strangely and then uh or that side of it is and then uh the wolf is a reference to the Beowolf cluster and I'm too young to know why Beowolf clusters are called Beowolf clusters but the the bolf cluster was the um the pattern of building for typic research especially academic research at a university um uh building a compute cluster out of a bunch of desktop class Hardware so you would see these photos I've had one in a recent presentation as well of these baker racks of just completely normal like Dell Optiplex Towers

all plugged into a network and then used to do some kind of Ensemble or uh Cooperative uh um computation or simulation so yeah that that's it for for this part of the demo we can take this down for now but we'll do more parts of it in a bit that was very quick it came up very quick yeah uh it's it should be about as and this is just to put that in context running on both the server and the compute node there are running on I think a 15-year-old Dell server that is in my basement so uh with actual networking and actual Hardware it's

even faster fantastic oh that's awesome so I you said something about you don't have to worry about the state yes can you explain a little more so yeah so typically like if you just have your your normal computer the one that we likely each have sitting in front of us there's the hardware and it has like stateless memory things that it it loaded loads into memory and then it loses power it forgets all of that but then there's the SSD or the hard drive or whatever the thing that stores the information that's persistent from boot to boot uh for most use cases you know we

want to save things on our computer and then turn our computer off and turn it back on and have that stuff still be there in a clustered environment usually the state you care about is somewhere else it's in a shared file system somewhere it's on a network share somewhere uh that all of the compute nodes have access to um but typically you still deploy those servers or historically you still deploy those servers like you would normally they have a hard drive or an SSD and then you install the operating system onto it and over time as you make configuration changes or uh you upgrade things

that state is have you have to manage that state you have to upgrade the software that's on it or you have to change the configuration that's on it and that's a stateful operation is what we say and and that's fine for one computer but when you have 10 100 a thousand computers you have to worry about that state discreetly on each one and make sure they all stay in sync make sure they all stay the same uh and that can become quite onor so uh Warewulf is a system it's not the only system but it's a a pretty good and streamlined system for uh storing

that desired state in one place on the Warewulf server and then having the nodes get that from the server every time they turn on rather than depending on their own local record of what it should be doing what operating system it should be running what software and what versions it has what its configuration is it sounds super helpful yes I find it so yeah one of the things Jonathan that I've heard from talking to different people if they have older Hardware they're always concerned about if you do all this in memory am I really starting to impact my cluster in the amount of memory I

have available to actually do computation does that really matter it doesn't seem like it takes as memor as what you think that it would ultimately the the amount of memory you're sacrificing to to it is the size of the image that you're running but that is something that we're working on the one of the things that I intend to give a demo of today uh is what we're calling or what is called a two-stage boot process for a different reason for supporting larger image sizes than we've been able to support in the past but as we were implementing that it became very clear oh look

if you just changed this thing right here then you could provision directly to local storage instead of to memory and then you would still have stess deployments where it's getting the image every time over the network but instead of loading it into memory it's just caching it onto some local disc so uh we intend to alleviate that uh you know that concern in the future but the reality is like we're talking about a couple of gigabytes at most for most uh operating system images and then thing other things can come over the network anyway um and you know servers have this server that I'm running

downstairs right now has 6 4 gigabyt of memory and it's tiny um 128 uh 512 even terabyte or more of memory is not uncommon so it is something that you have to be aware of but it's less of a concern than it ever has been in the past and we expect it to go away anyway thank you awesome so what version of Warewulf are we on right now so just last week uh we released version 455 so we're we're on the 45x kind of development line and trying to uh you know bring out what new features we can into that Branch uh and definitely bug

fixes in that Branch without uh disrupting the feature set too much without you know breaking anyone's production um so there have been a number of new features there uh 4554 uh was what we kind of crafted this conversation around and the the features that were available in that release and that's that's where some bigger features were released uh 455 is mostly just bug fix and and regular maintenance type stuff we had a number of other issues that that didn't quite make it into uh into 4555 but what the community side of Warewulf development has been trying to do recently is just stick to a monthly

release Cadence where we give out a point release you know around the end of or beginning of a given month so we expect four five6 you know at the end of July beginning of August that kind of thing so on the on the point release you're saying every month what about the minor releases and the majors is there a schedule there so yeah the the the versioning of Warewulf doesn't quite match up with that traditional major minor bug fix thing um because four uh is so different it's like a different product from Warewulf 3 that we don't really have a plan to change change that

so within the version four kind of product nomenclature the5 and6 45 and 4.6 are probably what you would want to think of as a major release and for those we expect those to be that the current plan although this is relatively new we haven't done this yet uh is to have those twice a year uh and try to line them up with the schedule for the ISC the International Super competing conference and SC the ACM superc Computing conference so we're doing that in partnership with one of our larger Distributors the open HPC project um they are also trying to Target releases around those time frames

to have the next version of open HPC available for people at that conference and so we want them to have the next version of Warewulf in open HPC for that release for that conference so um we we did that for 4.5 and we are planning to do that uh in time for SC for 4.6 but it remains to be seen you do have a question Jonathan yeah oh having local dis with a swap space that is at least two times of the node image the node would move inactive Ram pages to swap to free up system memory yeah so this is something that we have

at least surmised in the past but I'm frankly though I expected this to work in our testing it has not the the the uh the memory of the OS image itself at least the way Warewulf is provisioning it seems to be pinned to memory in a way that it doesn't get swapped out to free it up so we're we're not directing people to this expectation anymore oh I may have lost you guys for a minute did I page out for a bit oh we okay great your your video froze for me so maybe like this sacrificing the video for for the transmission yeah so uh

we want that to work and I individually and we in the community do not yet understand why that is not working um it may be like a the the the temps that uh that Linux uses for the initial Ram dis behaves slightly differently than most temp FSS so it might be that if we tweak how it's getting provisioned or even if the the way that we're doing it with drait now that I'll show off in a bit it might be that it work in that circumstance but we haven't tested it there yet it should work um but it at least didn't in one of the

tests that we ran okay say that again jacket that's that's how pronounced yes or that's how I pronounce it anyway uh so uh most Linux environments have what's called a two-stage boot um where there's a really small OS image that gets built dynamically um and that thing is what gets started on your computer at first there's like the bootloader and then it runs this little Linux image and that Linux image is responsible for starting up all of your hardware and then switching to the real Linux that's going to run and so that that two stages of a Linux boot is is called a two-stage boot

uh Warewulf has historically tried to avoid that uh in order to just for Simplicity it it we thought we didn't need it we just treated the whole OS as one big image that we were going to send over the wire and let it boot and if I can uh if I can stop sharing here um and share this other one so this is my Warewulf server and uh you know I've got one compute node connected to it uh and it booted this uh Rocky n image and that all worked just fine it did exactly what I just described where we just sent it one big

image and it started but I have this uh you know unnecessarily large Rocky n image um that I can switch to and if I switch back to my compute node again not this uh this one here and start this up again oh well this is just still running from before uh and tell it to reboot and I'll let that go while I describe the issue so this is one of those places where I'm I'm liable to get into trouble because I I have an understanding or I think I have an understanding of what's going on but we've struggled to really understand it because it seems

to be very local system and local hardware dependent um depending on the hardware you're running on and the environment you're running in uh once your image gets to a certain size you get crazy unpredictable errors that we you know sometimes it would fail out before this point that you see here sometimes it would fail out later I actually experienced a worst case scenario that I am baffled was even possible of the boot succeeding but then when I logged into it there were files missing from the image which I suspect is what's happening in a lot of the cases but sometimes there are things that are

missing that disrupt the boot so much that it fails uh but other times the boot succeeded but it didn't know that things were wrong so the the current thought process the current understanding uh um modern uh x86 processors um where I say modern like within the last quarter Century uh have different modes of operation and and I kind hand waving refer to it as real mode and protected mode and so the the bootloader right now the bootloader I'm using is a piece of software called ipixie it's what Warewulf uses by default um um the bootloader runs in real mode it has direct access and it

runs the software that it's running in direct access to all of the hardware there's no abstraction between the memory that your application is using and the memory that uh is on the hardware um my understanding our current theory is that these bootloaders were simply not built to manage this much memory in real mode they're built to do a tiny little bit of work and then pass it off to something like Linux which switches to protected mode and then does things in a much more safe and and uh robust way and we when we started pumping gigabytes of image through these boot loaders it sometimes is

fine but sometimes it doesn't know that your system is reserving some of the memory for some purpose and it can't actually write there and so it tries to write the image over some protected memory and uh and then fails uh when it tries to boot so here we have one of our failure modes the kernel has started and it appears to start booting but then when it tries to access the root device it just fails um and this is a slightly less common failure mode in my experience it usually fails before this but this has been reliable for this image size so uh what we've

done is and I'll stop sharing here and switch back to to my server again real quick so it is somewhat predictable like you had a pretty good idea it was going to happen it's it's predictable for a given platform I'm on this this server I know that it does this in this oh I shared the wrong thing again it does this on this server but if your server firmware behaves differently we've had people that can do tens of gigabytes of an image on their server with like a fully upto-date firmware fully upto-date everything on that platform and then it it gives the bootloader enough memory

in a big chunk that it doesn't cause a problem but it's something to do with like bugs in your Hardware's firmware which you know we can't build our software to say well it's everybody else's Hardware's fault you need to go fix that like that sometimes that's the uh the uh that is an opinion that sometimes we encounter but we we can't do that right so we're looking for a solution and so we so can I just interrupt you for one second joh yeah so go for it um I wasn't paying attention sorry I mean I was paying attention but I didn't notice this part did

it give you a an error message like when you oh my God you froz in the best like little face for a second there but that was fun yeah so does it give you like the error message and did you know that those packages like how did you know those packages were missing that that first time oh that that that one time where I logged in and there were files just missing it was because I had been working in that environment and I knew what was supposed to be there and I was able to see that there were fewer files on the system then there

were so there wasn't a notification if you weren't familiar with it that's why I called that a worst case scenario the worst case scenario is it fails and doesn't tell you uh errors are bad but it fails and doesn't tell you is is real bad so we we really needed to find a solution to that uh I am totally lost on shares and things so I I can bring this back up and uh show you this one so in this environment yes there's an error here uh I I can't I think I can't oh yeah my cursor is here so we have um uh no

file system Comm Mount and Colonel panic and you know there's some errors here it's not necessarily clear why it is happening but this is I'm not going to boot anymore something is wrong uh and we have to sort it out so the the solution for this that we have been working on and this is I I refer to this as kind of a technical preview you um it's working pretty well we have a member of the community who's using this that um has submitted some enhancements for it that didn't make it into 455 but I'm hopeful for them to make it into 456 so uh

this is we're going to stick using this rock E9 large image so this one is too big for us to boot normally um and then because we're using the ipix bootloader we configure this one and we Chang this to drait and this takes a little bit of time uh but I'll let this go and then I'll describe what's happening here so uh we'll get back to here I think and then I'll tell this to reboot so uh it it sounds a little bit complex but it's it's not that different from what your computer normally does it's just complex compared to what we were trying to

accomplish with Warewulf so uh dra it or with Warewulf we now install a package into the compute node image that tells drackett this is a piece of software that's widely available it's it's available for pretty much any modern distribution definitely for Rocky Linux um it tells drackett how to build an image how to build a a little what's called an init Rd an initial Ram dis that uh normally what it does is start up and look for a dis on the local system and boot that we added a module to to it that lets it boot up and say oh I'm not supposed to boot

from the local machine I'm supposed to boot from a Warewulf server somewhere and it gets some information from the Warewulf server during the initial boot that tells it where the server is and what image it should be starting and that's what's Happening Here I do wish that drait was a little bit more verbose about what's happening um this is happening in like the starting dret pre-mount hook and there's a log file where it actually logs what's going on but right here it is is uh running dret and then pulling the image and so again we were talking about this being fast like this was like

a 3 gigabyte image uh where the previous one was around a gigabyte and dret boot ends up actually being faster than a standard boot um even though we've got more layers of indirection because Linux is so much better at moving that image across the network and doing the decompression than ipixie or even grub are so uh I mentioned we use ipixie for the boot loader here that's the default uh we also have GRUB boot available as a technical preview similar to drait and you can use drait with either of those so here we are in our artificially large it's really just got like a a

2 gigabyte file that's been placed in the root directory um uh we have this large image that just got pushed across the network uh didn't work before and with the the dret boot that we have now we can do that and so we've pushed this up to uh frankly the amount of memory that I have available in my test environment um without any trouble so this you know this will continue to mature and we'd love for people to uh get involved either on the community side or if you're a ciq customer and you uh are using or interested in Warewulf we'd love to get this

into your hands and get your thoughts on it and get your experience with it and continue to improve and and mature this this technique this additional method of booting with Warewulf so that uh because it's just better in a lot of ways it's a little bit more complex but it makes the whole system so much more flexible also seem like it made it a little faster yes and and then this is the thing so in the past we were just dependent on Warewulf's or sorry we were dependent on linux's ability to just take an image off the network and handle it uh now since we're

doing that intentionally and purposefully we're we're pulling that image ourselves and putting it somewhere and then booting into it we have more control over where that image goes and so that's where right now we create a temporary file system in memory and load the image in there but once we did that we were like wait a minute we could just have it nominate some random storage locally on the system and say let's put it there instead and so in the future this should let us completely obviate the need if you if you care about this kind of thing uh to to load the image into

memory you could just load it into local storage so and then it would persist past reboot so right now the plan is to have it not be stateful to not have it persist between reboots would we you could imagine that it could do that that it could say oh look I have an image there and I can tell that it's the same image that I'm being told to boot and there's not a newer one some kind of cash invalidation would have to happen there so it knows if it needs to pull a new one or not right now the plan is just to always load

the the image state LLY across the network like we already do but instead of writing it out to memory write it to local storage so I can still imagine you could still have like a logs directory that persisted so that you could actually go back and look happen we can do that today okay yeah we can do that today so uh one of the one of the bits of stateless configuration you can get from the server is you have some local discs Mount them for for things like local scratch for logs if you're a database server maybe you store your database there we have one

of our customers is running a a they can just plug another one in and tell the server to make it a file system node and it just starts up and works correctly um so yeah you can definitely have state but that's different than the operating system itself the the system being stateful awesome yeah let's put up uh Kevin's question here Hey Kevin what's up man thanks for watching we know Kevin all right he says I wonder which is more safe default between ipixie and drait yeah yeah so yeah and and I would Nuance this a little bit it's like whether we would use drait or

not we're using ipixie in either situation either ipixie or grub um the the thing with the at least the way we have drait implemented today in order for that to work there has to be dret support in your image and so we're publishing a package a new Warewulf drait package that you have to install inside of your image in order to allow that image to be booted via wolf with drait um but until we find a way around that where you can just boot an arbitrary not an arbitrary image um we will probably stick with not having drait on by default um and then you

know just have really good documentation or maybe some pointers in the UI saying hey if this image is beyond this size you might want to consider using drait for for boot that kind of thing yeah awesome thank you thanks Kevin good to see him we got one more another question y right there okay Gary what's up any guidance for scaling the performance of the pixie server to the number of nodes does it reach a point where it is best to stagger boots or to get multiple pixie servers yeah so my recommendation here just kind of drawing from like what I would do if I were

using this in a production environment uh I would scale my Warewulf server to be capable of booting a rack of comput so there's individual servers and then maybe you have like a multi-server chassis of some kind and then there's some set of nodes in a rack I would try and make it so that I can boot a rack of servers at a time just turn them all on uh and then because this is the way I would turn on a large system when I was a sedman uh I turning the entire system on all at once has its own problems and that is the unit

of of ring up that I I liked to use if you need to scale because that's the thing Warewulf's performance sensitivity is really only an issue at that initial turn on once everything's running it's a pretty low uh you know it's not a performance sensitive operation or or requirement um so that's what I would do but if you really needed to scale it we've seen people do a Warewulf server per rack like maybe you have a topof rack Warewulf server that does that um there's ways that you can cache what Warewulf is providing out ultimately the the image and the um the image and the

overlays just come over an HTTP server so you could put a cache in front of that um but what we've mostly talked about in the past is just having multiple Warewulf servers letting using static DHCP so it doesn't matter which server ends up responding to a given node they'll just get the same answer regardless and then scaling it that way so there's a few ways around it um and we would like to formalize that a bit um but for the most part people have been able to take what's there and make it work even if it's not like a first class part of the product

yet those different choices or different different types of architectures documented is that a part of the documentation you said formalized I don't know if it's part of the community documentation we've written some articles about it at ciq and I I think we've published some of that but at the very least we've shared it with customers and for sure could in the future thank you um on the community side this kind of a side question uh how common and frequent do you get requests for a goey So within the community not very often um but the people that tend to join a u like a cluster

management slack Channel tend to be the kind of people that are you know they very command line comfortable um I personally would be interested in a gooey for Warewulf but there's a couple of kind of first this then that steps to get there um and it's it's a lot of work and most of that is about having a fully featured API that a gooey can call into Warewulf with and I've got you know Grand visions and we talk within the the steering committee about what it would look like to get there but on the community side most of the time um you know we just

been focusing on the core product um we've also talked about that maybe being something that ciq uh might be able to commit time to but it would depend on having the right partner uh to drive that and you know that has that need that request and that'd be great yeah I mean it's it's interesting being a company that you know we provide support and training for Warewulf um in a variety of different ways uh and but then Warewulf has its own you know open source Community Driven it's got its own thing going on and of course we want to uh put all of those links

in in the chat for you guys you can also go to ciq.com and there'll be like a little Warewulf section where it gives all that the information for the community um so I think of that too and one of the reasons why I bring it up because if there's someone out there that has that skill set that would also like to see it that might be a wonderful place where they can kind of get involved is in the community yeah that that's absolutely true and uh I'd like to think that we're actually there are different kinds of Open Source communities sometimes the community aspect

of it is mostly around you can submit a bug if you like um but we try to be really welcoming to new contributors at pole requests too so uh if you have functionality that you want to see in Warewulf and you know you we would prefer to engage with Warewulf through the community rather than let's say a support contract with ciq that kind of thing um like awesome like come hang out uh there will be someone myself or a number of other people uh an increasingly large number of people um in the Warewulf slack that would love to talk with you about that and point

you in the right direction give feedback on any code changes that you're thinking about um and new functionality I see uh Zach hi Zach asking about a text user interface instead of a gooey that's not something that I individually have really thought about I think by the time you're you have a text user interface you're already logged into the server and so the maybe something that's kind of like that if I can put up another uh I did this earlier um but I didn't really remark on it so one of the things that something like a gooey or even a text user so for people

that don't know these terms so uh there's like the command line interface that's uh like node list and you know you execute a command and you get a response back and you're at a Shell like this and then a guy what we've imagined for Warewulf would be something like a web interface that you can log into and then see uh See the nodes that are in your cluster and the the um containers the the images that are associated with them status that kind of thing um but one of the big benefits of a guy aside from not having to deal with the command line is

discoverability um you have to know what commands exist or know how to get help about the commands uh um in order to use a command like and sometimes it can be int or sorry intimidating um but aui alleviates that by showing you hey these are things you might care about here's some status here's some uh here's you know buttons to click on instead of you have to know the commands um but a text user interface is kind of in between those where U I'm trying to think if I have an examp well do I have uh no I don't have either of them uh it's

it's a uh an application that would run on the command line but instead of just being command and response would be a user interface with maybe options and you can uh let's see here we go the network manager uh text user interface this is an example so I can run NM CLI and make changes to my network on this system or I can use this thing and you know not quite click around in it but I can see what's available and make changes here so the question is what about something like this for Warewulf and the closest that we have to it is that instead

of like node set where you have to know like that the container option sets the image that you're going to boot and you know know the Syntax for this we've added this node edit and then we also have an overlay edit and a profile edit things like this um and this opens your node um that you want to change in a text buffer and lets you change it interactively so up here in the comments we have examples of things you might set for this node and then down here we have the node itself and you can make these changes directly and you could change this

in the configuration file directly um but doing this lets Warewulf know that you're making the change and lets it um uh lets it update the server and make other changes that are that need to happen to to reflect the the modification you're making here so this is kind of moving in that direction it's not a full Tue in that sense but trying to make it a little bit more discoverable a little bit more approachable um you know in so far as we have capacity to right now thank you for the question Zachary that was good yeah awesome I also like links I haven't seen that

in a long time too bad I didn't have it installed some know awesome okay so where okay what no this no slides we have do we have a slides presentation oh Phillip hello Phillip enabling bash completion if it isn't by default is a huge help double tapping tab usually shows the available options with wwct yeah this is something that uh Christian go over at sus he's a member of the Warewulf community and a member of the technical steering community for or technical steering committee for Warewulf he's been just banging this out it's great um not only does it tab complete like the sub commands for

Warewulf but it'll also tab complete the the the things like uh the I think container names but definitely overlay files and so before you had to just kind of know what was in your overlay and just type it all and just it was not a great great experience but now um if we can switch back over to this again um I can say like wctl overlay uh edit and then see here's my list of overlays and then tab complete oh it's not working maybe I don't have the latest version installed on this environment because it has definitely worked in the past uh let's see if

it works in uh it show there here it's working uh uh debug and then it sees the files that are in it and that is just so much better of an experience and then I can say like render this for node one and we can see what this file looks like when it runs on that node um and in the past you just had to know these file names and or go look at them on the file system which was not great we going to have to uh going to have to file a bug for the fact that this is not working for the edit

subcommand still very cool back to slides question though oh yeah do we have some slides that you want to share with us yeah so I um I gave a presentation at the rocky M Rocky Mountain Advanced Computing Consortium um a maybe a month ago now um that was more generally speaking about uh getting involved in open source and open source communities and uh I you know would like to invite um anyone watching uh who's interested in Warewulf or is using Warewulf um to you know come join our community and part a big part of my oh the story that I was telling was that you

know these things aren't just software projects they are communities and communities are made of people um and so I I wanted to kind of present each of these well in the presentation I talked about Warewulf and I talked about Apptainer and I talked about Rocky Linux and I talked about open HPC um just talking about Warewulf today um but yeah I just figured we could put some of this up like if you're interested in getting involved in Warewulf we have a really well populated uh slack you can see see Zach uh talking there in this screenshot that I took before actually um the Warewulf slack

is great um and I I couldn't be happier with not only the members of the steer hearing committee and their involvement there but also with the the user community that has just joined and and supported the project really really well one of my favorite things is seeing a new person show up and ask a question and have a user that I don't know answering the question or at least trying to provide help and that's great and uh if you you know we talked about how Warewulf for is very different from Warewulf 3 uh and things that came before it but even a fair bit of

the Warewulf 3 Community has uh joined here you can see there in the left bar um that we've got a Warewulf 3 channel so if you are still running Warewulf 3 and are looking for people to to get help with there um you can join the same Community talk with some of the original maintainers at Warewulf 3 over there they're still you know providing some help for people that are still running it and then um yeah you're you're all plugged in with the people that are doing Warewulf 4 as well if and when you are ready to to switch to that um uh there's uh

twice monthly right now uh Warewulf steering committee meetings um this this made more sense before when it was in a a conference room but uh those meetings are open to the public uh if you're interested um I recommend joining the werewell slack and then asking for an invite and i' be happy or someone would be happy to send you the invite for that they're on Wednesday I think right now I forget I go when it's on my calendar um we we we tried to um we were trying to split it between like just technical steering stuff and Community Focus focused stuff but we haven't uh

we haven't really done a good job of starting up that bespoke community meeting we're hoping to do that maybe after August um but for now you know come to this meeting we've got a whole part in our agenda uh where you know we we say hi to anyone that's new and ask what's going on if there's any help you need um and the the notes for that are open to the public as well actually I think it's the notes that are behind that QR code um uh if you want to check out what we've been talking about and see how the project gets run and

how we make decisions that kind of thing and then because projects are made of people uh I just wanted to to let people know some of the faces and names behind the project so um you know people will know Greg Kurtzer are there at the top um he's the I think you're not Shing I don't think we're looking at the same slide oh oh my yeah we're still looking at the one with the slack Channel with the Warewulf slack Channel weird what's going on with Greg and uh okay just a moment Christian I believe his oh now yeah thank you here we there go see

yeah there the Q code I was talking about than we did see the slack channel before man Mac OS has changed this Behavior where it doesn't update Windows that aren't visible to the screen and I've noticed this with YouTube uh where if there's a video playing in the background if I tab back over to it it like has to catch up but that's what's going on here I had the actual display window behind other windows so I was like oh I don't need to change that okay so uh top uh kind of top left there uh people will know Greg Kurtzer our fearless CEO and

creator of Warewulf in the first place and Warewulf 4 in specific um but Christian who I mentioned is is pictured next to him you can see me there in the bottom left and then uh John gnog Hanks uh is works at CZ biohub he kind of represents the user community in a lot of ways he um uses and provides feedback on Warewulf and and provides a lot of really valuable production experience to the the steering committee and uh then Jeremy sidel on the bottom right uh works with uh Intel and helps keep us plugged into open HBC and other kind of industri um industry interests

that are stakeholders for Warewulf and then um Jason Yang on the right has been he's another ciq developer and has just been doing a lot of great work in Warewulf where uh you know if there's a bug I can just kind of assign it I say hey Jason could you take a look at this and what he he he lives in Japan so I do that at the end of my day and then when I wake up in the morning the the bug is fixed and it's it's a a great experience um and then like some ideas for how to you know if you're coming

from the community uh or you're interested in the community we're always looking for people to help do documentation work it's definitely getting better um documentation has been a little bit of a we've been behind on it for a while but we're iterating with each release and now we have it split out properly so uh we have a 4. 5.x branch for our docs as well as a main branch so that we're kind of documenting the release Behavior separately from the development behavior um and keeping those up to date that's been going really well um and then uh we're as I mentioned hoping to have a

more dedicated community meeting where maybe we highlight our user community and and what people are actually using Warewulf for um I had a or I got to see a really great um uh presentation exactly like what I'm hoping to see from Arizona State um at the Rocky Mountain Advanced Computing Consortium it's wonderful to see someone uh in the user Community giving a presentation on Warewulf and how they're using it so we're hoping to do more of that and if that sounds like something that you'd be interested in um get in touch uh we're looking for people to help organize that and then oh I I

didn't know I left this slide in here I passed out a signup sheet as part of the presentation that I gave um for people looking to get involved um but then you know there's a bunch of bugs and if you're as there always are and opportunities for enhancement and if you are a developer or you want to get into go programming Warewulf is a really great uh approachable code base for a systems person who generally knows or or wants to learn more about how this kind of systems management works I did not know go before getting involved with Warewulf and this has been a really

great code base to get to know go through um and then these were kind of my topof line generic getting involved in an open source Community pieces of advice um I part of my narrative was I had struggled in high school and college to start getting involved in open source stuff and I kept trying to look at it as like a software problem um but and I could just pick a piece of software off the shelf and start contributing to it is though that's how it works um but my my advice is to contribute to projects that you already use if you're using Warewulf you

will get value out of contributing to it um it's it's a really cool experience to put code into a produ project and then immediately get to benefit from it and then see other people benefit from it um join the Warewulf slack it's great and you'll you'll get a lot out of it um you're welcome to show up at meetings people that have Jo done that recently um reported that they they got a lot out of seeing how the project is run and how decisions get made how you know what we're thinking about what our priorities are and then you know reach out to the actors

contributors people that you see committing but also the steering community members um uh you know we we we love to talk about the project so reach out on slack reach out in any way uh and then if you really want to do stuff like help run a a community meeting uh when you hear opportunities like that say yes I will do it even if it sounds a little bit scary and there will be support the whole point is it's a community and no one has to do anything alone so uh let us know whatever interests you and and we'll do our best to get you

plugged in and enabled awesome now well said Jonathan well said I I I really appreciate you going over the value and the benefits and how to get involved in the community and giving some people guidance around that because those are questions that we get all the time like how do I get involved and I mean of course once you do it it seems simple and obvious but if you have not ever done it yet you're not even really sure where to start so that was very helpful um I can't believe we're already like 10 minutes to the hour so um I'm I feel like we

still have like a whole another hour that we could go talking about all the things that we want to talk about with Warewulf today um but I definitely want to give you just a few minutes to kind of wrap up and and and make sure that you get out the you know the updates and things like that that you really want to share I yeah I mostly that the thing I was most excited about and the thing I wanted to demo was the dret stuff I there are other features what I was showing um the ability to edit uh I showed editing the node you

can edit overlays we fixed some really cool well the bugs were not cool but it is cool that we have fixed them um um we also have made some improvements to sync user if you're using that functionality um there was some some inefficiencies and how that was uh happening that for large user counts ended up being a problem that's been resolved but you know we're adding new features and fixing bugs all the time uh in the future we're working on uh better support for IPv6 especially in IPv6 only environments um we want to have similar efficiency improvements for the container list command which for technical

reasons is kind of slow right now for for certain large containers and uh maybe the biggest one is we think there's a lot of power in the overlay system but it is awkward to use because all the overlays come from a single place that the system overlays are in so we're going to break those out and make it easier to mix and match the overlays that you want to use and then mix those with overlays that you're developing locally and manage those separately maybe some overlays will come from a package that you install from a provider like ciq that's giving you overlays to configure a

piece of software that uh that you want support for but maybe you also want to manage some overlays in a g repository somewhere and track that separately we're we're hoping to improve support for that so there's always new stuff always cool and exciting things happening and if you'd like to know more uh just come join slack join a community meeting uh look at the notes look in the GitHub issues there's the Milestones are pretty well organized in there too these days and if you want to see what we're looking at for 4.5x or 4.6.0 uh it's all right there Jonathan do you want to mention

the centos7 oh thank you yes so uh at some point in the recent past I think it may have been 453 but one one of the recent 4 five releases um we went ahead and dropped support support for Centos 7 this is something that we've been you know holding on to for a while but has become more and more difficult and now that sentos 7 is itself end of life um we just decided to stop worrying about it so we kept encountering go libraries that didn't work on Centos 7 that we couldn't upgrade to and uh GitHub has a function that it runs called dependabot

that tells you about security problems in your dependencies and uh we we had a number of of those that we wanted to update and fix and doing so broke compatibility with sentos 7 so with the E of that distribution uh we decided to just go for it you can still manage Centos so this this is worth clarifying you can still have sentos 7 compute nodes uh and that's fine um Warewulf can still manage a Centos 7 compute environment it just can't run the server on Centos 7 anymore so yeah to to run in a supported environment with the latest release you need a rocky 8

or rocky 9 or Susa uh leap environment or or SL environment uh those are all supported platforms within the Warewulf project um I think leap 15 something is the the version there um but yeah no longer Santa 7 thank you want to make sure we got that out there yes thank you it was in the notes but we we were just we were just talking so it's been yelling at me the whole time such a Hot Topic yeah yeah it sure is okay so is there any uh final thoughts before we wrap up here Jonathan oh Kyle hey Kyle that's a fun picture is Raspberry

Pi suitable as a Warewulf server there is a whole s a single board computer single board computer channel on the Warewulf slack full of very passionate individuals running Warewulf both server and compute on Raspberry Pi um there is I can't think of any reason why the server couldn't run on a pi that's just fine um we do there are some complexities in mixed architecture environments so the Raspberry Pi is an arm environment if you're then trying to provision x86 servers with it that can be it's possible you can do it but there are complexities around it and it's not necessarily how it's meant to work

if you happen to be trying to boot other arm machines it should be fine um booting compute nodes that are running on Raspberry Pi can sometimes be a struggle and that's where a lot of that community interest is coming from um my understanding and I am far from an expert uh is that the Raspberry Pi bootloader or boot environment is notoriously unreliable and getting it to network boot reliably has been a hobby project I'll I'll put it that way it is definitely not a a project targeted supported platform but if you are wanting to do that there is a community of people that will be

very happy to have you yes Dr Sil is very excited about doing that he oh man he's got a specific project that's just so cool it's a um it's like a single motherboard kind of environment that has slots for compute modules and so his whole you know platform his whole like product the thing he's working on building and has working is a Warewulf server in a box where both the the Warewulf server itself and all the compute nodes in the cluster are just in this one machine that you know you could imagine a a computer science department could use for teaching and instruction in a

class and make that really cheap to buy really cheap to build and easy to manage and that whole thing runs Warewulf and it's great awesome it's very cool okay Martin Herman coming in strong here at the top of the hour almost with a question that might take more than 30 seconds to answer so I am going to encourage you to go to the community but he would like your suggestion on the best 101 step byep to install Warewulf in a break brand new hardware I have a lab configuration of two ethernet networks with three computers for Warewulf slurm and the login node and nine compute

nodes I yeah that sounds fine that uh in terms of like what's missing there that that seems like a reasonable starting point um you really only need one ethernet Network for the the provisioning so one network that all those nodes are on you can boot the login nodes with Warewulf there are a lot of people that do that and and we we even recommend it um you just need to have some kind of shared NFS that can come off the Warewulf server as well um but there's there's a quick start guide on the Warewulf documentation website there's one for Rocky Linux and or I think

it's written as Enterprise Linux but uh one for Rocky Linux and one for Susa um and I think there's a new one for Debian though it might be incomplete at this point um all of those have a little bit of they need a little bit of handholding we're working on getting those improved so if you use one of those quick start guides and have questions that's a great reason to join the slack and we get questions like that all the time and then take those those issues those questions that people have enrolled them into documentation improvements whenever we can yeah awesome we're looking forward to

seeing you in the slack Channel Martin uh if you want to link to that pretty easy I mean you can go to warewulf.org and uh find some links there to the community and then of course if you would like our help right at ciq we do Provide support um training implementation there's all sorts of things that we can do to help you get your your your Warewulf uh environment up and running and uh functioning properly so Jonathan thank you so much for all the work that you're doing both at ciq and in the community and everyone in the community working on the project it's good

to see thank you so much for the conversation it's uh it's great we'll have to do another one because we still have there's a lot of bullet points we didn't get through Jonathan yeah there's a lot but especially with these monthly releases right like okay so we'll see you back here same time next month all right thank you guys have a wonderful day make sure that you like and subscribe and go to ciq.com and um you can ask any questions there and we'll get back to you thank you have a great day see you