What's new in Warewulf v4.6.0

Okay. So, maybe you want to do like a little overview of of Warewulf. And sometimes it's good to just kind of highlight the difference between what it means to work at a company that supports um an open source project and what it means to kind of be involved in the open source project. I'd love to hear a little bit more about that too. Yeah, for sure. So, um Warewulf is a a cluster management and provisioning system.

So it lets you kind of deploy one server with one source of truth that knows everything about your cluster and then you put other compute resources or it doesn't have to be compute but commonly compute resources on your cluster or on a network, turn them on, they check in with the server, get the operating system and the configuration that they should have and and turn on statelessly um and go from kind of a single server to a many hundred or thousand node cluster in minutes uh and not have to worry a big deal about configuration drift, things changing on one node and not being propagated to other nodes.

Everything is just kind of a reboot away from just being clean and pristine the way you want it to be. Ends up being a pretty good situation for most people. Um in terms of the company versus the community side of things, so Warewulf is an open source project. Um, we sell support and services for it here at CIQ and we do a little bit around the edges of providing like custom node images and things like that for our customers uh and developing custom overlays for them.

uh but a lot of the development is done in the open with the community and so we're big players in that uh CIQ and so my participation in that community is a sponsored effort uh of CIQ but we have collaborators from Intel and Open HPC and SUSA um and a few other from like the user community that are either users of the system or users of Warewulf and have a vested interest in its success um or are selling support services for it themselves which is you know great all in the community it is great and so thank you to everyone who is involved in the

community I mean any any open source project that is widely used um needs people behind the scenes you know kind of managing and taking care of things um did we want to talk a little bit about the um the channel the YouTube channel that you so I uh I wanted to put this out there so we just recently just this LA so we have twice monthly um steering committee meetings for the people that are kind of formally in the organizing body for the Warewulf project and we uh we've been doing those meetings for quite a while. Uh but we just recently started putting those online so I was going to uh show this here one of the many things that we'll talk about here.

So it's kind of unique though. I personally I have not seen any other open source project and maybe I'm just ignorant to you know the variety of them out there but that so we're you're recording the steering committee meeting and then posting it here on the YouTube. Yeah, that's right. So we have this one here. We'll have one playlist here that's for the meetings themselves and then mostly just to initialize it. U I kind of did a troll all through um through YouTube looking for people that had done things about Warewulf. And a lot of that is of course us. Oh, all right. Playing a video.

Uh, a lot of that is of course us. Um, but a fair bit of it is I I found other people that are doing uh that have kind of done their own deployments and recorded it. Um, put that in there. Um, and then there's some videos that Susa has put up uh that I included in there. Um, yeah. So any any video content from the community or with ourselves included um I'd like to highlight with this channel and kind of popularize out there. But yeah, the the meetings end up being pretty good and you know it's sometimes hard to find an hour in the middle of your day or maybe that's when you're eating lunch uh to to join it live.

It's open in general to the public if you just ask for an invite on the Slack channel. But otherwise um check the video out. There's there's good discussion especially around a release like what we're doing right now. There's a lot of a lot happening all at once. There is a lot happening. It's actually really exciting. There has been some some help from other provisioning tools that are taking different directions. Uh so people are looking around for you know a a provisioner an open source provisioner that also has support from um you know commercial companies and so Warewulf is stepping into that position. So it's actually really exciting.

So again thank you for all your work there. Cool. Okay. So that's a little bit of the background of what Warewulf is. Um did you want to go into a little bit of some of the features and the new stuff happening? Absolutely. So the the new release that's coming out uh hopefully within like a week or so. Uh we're about to put out another release candidate, but it's version 460. Uh which is is roughly analogous to what I would consider a major release for Warewulf. That kind of version 4 is more like a generational leap for the product. So, uh, we're we're in the version 4 generation of Warewulf.

Um, but 4.6 will be a relatively large release. And there's a number of things. Actually, I was going to switch over to uh to the terminal, but let's let's look at the release notes again real quick. The so here we go. So, the uh the the releases are available on the GitHub page already. Um, so you will have found like 458 which is the last official release. Um, and if you just download the latest release, this is what you'll get still. Uh, but if you click through the releases page, you will find this 460 RC1. So this is our first release candidate. Really big change log here.

We're not going to go into all of these things. Um, but and you know, we'll we'll kind of edit this and and organize it for the final release. Uh but I I encourage people to look through this and see what has changed. And you know, we're trying to highlight people that have contributed to this release. There's quite a number of people that have code in in both release candidate one and uh release candidate 2 and upcoming release candidate 3. So check that out. There are packages here. So if you want to install this, you even have to build it. You can just pull the packages down and install it on Rocky 8, Rocky 9 or SUSA.

And there's people that are doing this on Abuntu these days as well. Um, we don't have packages for that yet, but if you're willing to install it from source, there's documentation uh in the the Warewulf docs for doing that. Uh, and we're improving our Abuntu and Debbian support all the time. And I'm really grateful for uh Ian and and some other people that have been uh doing that work in the community. again that there's no commercial entity around doing that, but uh I'm always glad to bring in contributions or or to encourage contributions from our user community. Um but I do have a a short list of things that I wanted to highlight uh as new features that I think people will be pleased to see.

So now we'll switch. Awesome. So I know that in my mind as you're going I have a variety of questions that are coming to my mind. So if you also as participants and watchers of of what Jonathan is sharing have a question um just put it in the comments and and make sure I think because you know when when it's webinar you have to make it go to us and registrants I think is the is the term you kind of have a couple different options when you submit a question. So we want we want to hear your questions. Okay. So I'm I'm going to hold off and assume that you're going to answer my questions for a minute before I start asking them.

Okay. Well, we'll see. So yeah, this is going to be largely terminal driven. So we'll see that Warewulf is predominantly and we'll talk about that a little bit more in a bit but predominantly a command line interfaced driven system. So what I have here uh I have two servers running and uh it might be a little confusing but I'll try to point out Warewulf control 2 is the one that's running something very I think it's what we're about to release as release candidate 3. Uh but then I have this other server uh Warewulf control one. This is running the the current official release. So, this is running 458.

And I'll jump back and forth between these to demonstrate some of the differences sometimes. So, one of the things that seems like a really small detail but catches people out all the time. So, you know, Warewulf is a system for provisioning uh what we call node images, what Warewulf has Warewulf has historically called node containers uh out to compute nodes. So you can uh this is in previous version of Warewulf. You can say like wct container list and we can see that in this environment I have a rocky 8 9 container one that I was doing some experiments with sess linux uh a 9 and a 9.4.

Uh these are all different images that I can provision my compute nodes with. Uh but our use of the term container here has been a confusion to many people. uh it was meant to kind of indicate and uh and support the idea of using container ecosystem tools for building these node images. So you can use um frankly you can use docker or podman or Apptainer any of these to define a node image which ends up being a really good um a really good experience compared to the old way of kind of creating a directory and kind of hacking an image together manually. But this name led people to think that there was some kind of nested virtualization or containerization happening inside of the Warewulf node image that might impact performance.

Uh and it just ended up being a surprising um a surprising bit of nomenclature. So over in Warewulf 460 land, this has been renamed to image. Uh it behaves exactly the same. Uh we can see Oh. Uh long. Oh, I can't type and talk at the same time. Uh, there we go. Um, we can see information about the images in our cluster. Uh, but just for people whose muscle memory has already switched them to container, uh, that that's still present as an alias. But all of the documentation, all of the kind of line help, whether it's on the website or in the um in the command line tool here, um will expect to use the term uh image or node image now instead of container.

Again, not really a big functional change, but it was kind of a big deal to go through all of the code and change all of the places that it says container to be image and not break things. Um but that work is done and has been in place for quite some time now. Uh, and I at least am glad to to have implemented this kind of community responsiveness there. Um, we heard a lot of people get caught up by this. Um, happy to take questions on that, but I don't know if there's a lot of questions there. Oh, yeah. When I saw in the notes changing containers to images, I was like, why?

So, that was a very detailed explanation of why. It does make sense. Um I there there is some confusion about you know a container and what that means and yeah so thank you for for simplifying that. I mean I guess why image then but I guess it does doesn't image is at least a more historically that that name has been used in situations like this for a long time. Um so in the cloud you'll do the same thing you'll have a a node image or a cloud instance image that gets put up. Um so that that terminology is a bit more common for this kind of use case.

It might seem strange from the outside, but uh yeah, there's at least precedent for it. Great. Appreciate that. Yeah, one thing that might be useful to go into here uh is that Warewulf 460 comes with some more native support for managing upgrades uh from previous versions than in the past. you kind of had to deploy the image or sorry deploy the new version is what I mean to say uh and read the release notes because maybe there would be changes to the configuration files that you needed to go make manually. So an example of that if I go to uh my old version and I just look at /etc/warewulf node comp.

So this is the file that stores all of the information about your nodes and how they should be configured. Uh one example of that might be that this uh configuration field here says container and so if you're upgrading to 460 in the past you would have had to up upgrade Warewulf and then come in here and make those changes manually. Warewulf 460 includes a Warewulf upgrade or Warewulf control upgrade command that can uh update either where's my cursor? Here it is. uh config updates the warewulf.conf that's the server configuration file uh or nodes which is that nodes.conf file that I showed earlier. So using this command we can uh automatically load a config from another version of Warewulf.

And the target here is to support any previous version of Warewulf's config file that you might bring to it and convert it to a version that is compatible with Warewulf 460 and going forward with whatever the current version of Warewulf that you have is. So to demonstrate this in this directory this is the thing that's going to help a lot of people. Yes, it is because I' I've been hearing that as well. So, even 3 whatever or just Oh, sorry. I I did misspeak. I mean any previous version of Warewulf 4. So, uh we we went back through the the whole code history of of the Warewulf 4 generation of the system and we're we're uh migrating from that.

So, thank you for calling that out. No, this is not a three to four migration tool. Um, but uh in this directory I have copies of the warewulf.conf and the nodes.conf from that other environment from the one that's running 458. And so I can say uh we'll start with config. So normally you could just run this and it would upgrade the configuration file in the location that it is on this like the the normal location on the system. Here I'm specifying an input path because I want to upgrade this specific one that's in this directory and then an output path because I don't want to overwrite that file.

I want to write it out to a new location just so we can compare what it's going to do. So we get some warnings about things that uh perhaps we want to care about during an upgrade or that the system might not have known what to do. So there are some values that are no longer used that maybe never really were used but uh were there for future proofing but we never ended up using. So this this versioning scheme inside of the config file that's been dropped. We no longer log to SIS log. Um so that's been dropped. And then uh NFS mounts are configured differently now and we'll go into that a bit later.

But this is noticing that we have some NFS exports, some network file system exports. And so it's telling us how to make sure that those get imported properly into the new nodes.conf. So we'll do that also. So here we're running I've lost my cursor again. Uh uh we're going to upgrade our nodes.conf. We're going to reference a warewulf.conf that we had previously. So the old version and we're doing that as recommended by this warning up here in order to import the NFS exports from it. So we had um we had uh network file system exports in the old config file. Uh that used to implicitly mean that those should be mounted on compute nodes.

So now we're going to load that into our nodes.conf during this upgrade. We're going to read this previous config file and write it to this. Again, normally you wouldn't have to do this. We're only doing that to manage these specific files for the demonstration. And then if you run this command without these two options specified, I'll just comment them out for now. You'll be prompted that you must specify them. And you can either do it like I have here, which indicates true, uh, or you could say false to skip that. The reason being right now in 460, we're in a point where these two changes are relatively large.

You might not necessarily want to apply these in the future or every time you run an upgrade. Uh but we don't know when we're doing this upgrade whether you want this kind of change made to your config. So this is now defaults used to be compiled into Warewulf. They just were there whether you said them or not. Now we expect all of those default settings to be on a default profile. So this is whether you want to do that uh during this upgrade. We're going to leave that true. Um, and then we've also made some rather large changes to how overlays are organized. And this is whether we want to replace the previous generic and dubdubanit overlays with their kind of brokenout counterparts.

And we'll go into that in more detail in a moment. But for now, we will have those both happen because we want them in this example. We get some warnings similar to before. Um, we're not using a Warewulf internal variable to manage uh the version for this file anymore. And we didn't specify um whether our network devices are Ethernet or not. This might be something that we should add to the upgrade process. Maybe it should assume that it's uh an Ethernet device. But for now, it's just saying, "Hey, by the way, nothing is telling me what kind of network device this is. You might want to go set it explicitly." So we can see now uh here are changes that have been made to our warewulf.conf.

So it automatically dropped this syslog value which doesn't mean anything. This is just a move. This doesn't mean anything. This is similarly meaningless. But then here it has changed how NFS exports configured. So it's replaced all of these kind of combined export and mount configurations with just export configurations. So the server side is just for exports now and the mounts have moved to the client side and we'll see that when we move over to um uh when we move over to the nodes.conf and then here we changed container to image here as well. So that's been updated. Didn't have to do anything except for run the command.

Same over here for nodes. So one of the larger changes like I mentioned is that instead of just having two main overlays that we package, one for runtime being generic and the other for system being dubdub and now uh those have been broken out into specific overlays for their purpose. Um, so the main goal here is that it makes it easier to omit parts of the built-in overlays that you don't want to use. So a good example here is that we have historically packed multiple network configuration system overlays together into the system overlay. But if you're only using network manager, you don't want if config or Debian interfaces or the the SUSA wicked system.

So, you can just remove those now from your configuration rather than having to go modify the built-in overlays. Um, other smaller things that have been changed here, kernel args are now specified as a list instead of a string. Uh, just like a one one value. Uh, that makes it easier to move uh or to sorry mix and match um kernel arguments from profiles and nodes and a bunch of other things. Uh, here is where those NFS Oh, here's where those NFS client mounts are now configured. So, lots of detail here that people running 460 should read about in the release notes when they're published with the final release.

But the main goal here is to demonstrate. You don't have to know read up on all of that to move to 460 upfront anymore. uh you can just run a couple of Warewulf upgrade commands and it will update your configuration to a version or to a to a format that's compatible with the the new version that you're deploying and and we intend to keep that up to date as we release new versions over time. Awesome. Um I think maybe now's the time for we got a couple of questions. Is that okay? Good. Okay. Yes, Jess, definitely put them bring up the question. Okay. Hey, hello Tobias.

Will that upgrade tool also handle the changes in distribution overlay layout? For example, the separation of the default WW innit overlay into smaller overlays for individual components. Yeah. So that that's precisely what this um replace overlays flag is doing. This replace overlays flag is is seeing that okay, you are including that dubdubdubit because that flag is active. We're going to assume that means the previous dubdubanit one that was all the overlays packed together. So we're going to explode it out into its component parts. Um and same for the generic one. Um it doesn't know anything about the distributions like in the image itself but it is meant to make it easier than to uh to specify your configuration for a given distribution by including only the parts of the overlays that you want.

Great question. Oh hi submit. Okay, the question here is, does this mean that to upgrade from 456 to 461, I first need to install version 46 and then run the upgrade command with the configuration file? So that's correct. If you have a configuration from 456, uh, then you run this command to update the configuration. So that's that's a I imagine the question is because people might expect that the upgrade command would upgrade Warewulf itself and not just the configuration. That might be Yeah, this this uh it might be worth considering before we put out the final release renaming this command to like update comp or upgrade comp or something like that to avoid that confusion because I could see that um this does not upgrade Warewulf the program.

Uh in order to get access to this uh you need to install 460 or some future version um and then run this command to update your configuration if necessary. In fact, I'm going to take a note about that because I think that's a good idea. And then so 46 is going to be the first one that you can use this upgrade command with. Yeah, in in theory people could do it now. They can install the release candidate and it's there. Um and and working as you can see right now. Uh but the first actual like official release that has that command in it will be 460.

So, as time goes by, are you is it going to be that it always goes to the newest, latest, or are you going to be able to specify which version you want to upgrade to? Like, say they're on 460 for a while, a year, and they only want to go to 461. Yeah. So, at that point, they just install the 461 package, and the version of the upgrade command that comes with that package would update to the version that they're running. So, it'll always try to munge into the version of Warewulf that is installed, but it won't be able to downgrade. If you were running 470 in some future and wanted to backport to 460, the 47 or the 460 upgrade command won't know how to import necessarily um the config uh from 400.

So, it it's one direction, it goes up only. Okay, that makes sense. But if you did want to go backwards, you would just then install the lower one and do an update to there. Well, you you'd need to you'd probably need to do the updates manually at that point because the 460 version of the upgrade command wouldn't know how to read a 470 config file, right? Okay. Yeah, thank you for all that clarification. No problem. Questions. More questions or on to the next thing? What do we Oh, hey Ian. H yeah so I I was yes the question here whether um running the upgrade command should just be done automatically when you install the package.

Uh it's possible but especially like right now uh because we have these flags like add defaults and replace overlays that we explicitly are wanting the user to specify what they want in so doing. Um I wouldn't want to do it yet. I think um I I was thinking of that as I was saying it and as I was responding to the person previously asking if you'd have to run the commands manually. Um but I would find it hard to advocate for at this point just because there's just enough like manual like decision making required in running that command and people might not want their config file to just be upgraded.

So maybe what we could do is run an upgrade but then write the the file to a new location uh rather than actually replacing the one and then just telling people hey if you want to use the new config file just move these in place. I don't know but that's not that much more than running the command in the first place. No well this really highlights your responsiveness to the community and wanting to make sure that the product is something that people are needing and wanting and are as you know simplified as possible for them. So, if any of you guys have any interest in getting involved with the community or or have um some suggestions on to kind of the questions that Jonathan is is, you know, kind of speaking out loud as he's thinking more about um how to, you know, continue to improve Warewulf.

Um the community is a great place to be. So maybe Jess, if you wouldn't mind just putting the the community uh the Warewulf community Slack um uh link in in the the chat uh below because that's a great place to, you know, get involved and ask questions and and let your, you know, concerns be heard. Yeah, the the Warewulf community is kind of a perfect sweet spot of an open source community to get involved in if you're anything like a Warewulf user. Um the the code is reasonably approachable. Uh I hadn't done any Golang before I got involved in Warewulf or barely any. Uh and it's been an easy project to get involved with and we do everything we can to prioritize responsiveness to uh our user community.

So come join the Slack, talk about things, uh submit some poll requests. We'd love to have you. Yeah. Speaking of which, uh, this next feature is something that I personally was terrified of, but a member of our community was really interested in it and we started throwing some proof of concept code back and forth for it. And here it is in 460 and that is nested profiles. So, I'm going to look at the actual nodes.conf real quick. This is a bit of a contrived example that I set up just to demonstrate what's going on here. So um Warewulf has nodes. These are like these represent physical servers in your cluster.

But then it also has profiles. And profiles have all the same attributes or almost all the same attributes as nodes. But uh you can load a profiles values into a node. And this gives you some some grouping and inheritance and reusability in the code. So pretty normally you'll come with a default profile and I think yeah we can see that default profile right here. That's what's giving us these previously compiled in default values and you know a comment just to be there and what the overlay should be for our cluster. But you can include more than one profile and here I have one just called cluster because I didn't have a different name.

And that cluster overlay is defined here. But the new behavior that is new in 460 is you can see here this cluster profile itself has profiles which didn't used to be possible. So now I have an NFS profile which has my NFS my my file system mount configuration in it and a network profile that is just kind of setting a dummy value for a default network device. But you can imagine other network values being here. And so now you can break down your uh configuration not only across multiple profiles but with one profile referencing others and then build up more complex configurations through this kind of nested hierarchy of profiles.

So we can see what that looks like right here. So in this output, oh this isn't I'll get a pager here. Uh in this output uh we have our node here and then the field that we're configuring and this is the profile that it's coming from. And so we can see some of these values are coming from default. Some of them one of them anyway the image name itself is specified directly in that cluster profile. But some of our values like this NFS configuration for uh FS tab uh is coming from that NFS profile uh even though the node is ne not directly including this profile.

Same for this default net mask is coming from our network profile even though our node is not including that profile directly. So uh again this this uh frightened me a little bit to get in the code but we've got a lot of tests around it. It seems to be working well and eager to get this in people's hands and see uh what they make use of it. Uh the the one person in particular in the community uh he's he's excited to kind of rebuild his environment around this functionality once it's released. Um and I'm eager to see what people do with it. So what I mean I imagine there's lots of different things that you can do with it, but what what is he wanting to do with it?

What's the value to him? Does this allow you to have like manage multiple clusters doing different things from the same place? So you could imagine like if you have know it's more just like clarity in your configuration. If you have like like what I've done here, I imagine he'll want to do something like this where he'll have one profile that specifies I want to have these file systems mounted and then another one that says that maybe multiple profiles that are for different networks. So I have one that's on the network in this building or one that's on the network in this building or this high performance network or that high performance network.

Make one profile for each of them. And in the past if you then wanted to configure your nodes each node needed to say okay I have this network and I have this uh I have these file systems mounted and it had to reference all of those profiles. You could do that before but then that node would have like this everinccreasingly long list of profiles for the different bits of configuration that it had. Now you can group those together and say, "Okay, all of these subprofiles are part of cluster A." And so I can make a profile that is cluster A and cluster A has all these other profiles that go with it.

This other cluster, cluster B, has a different set of profiles and cluster C has another set of profiles. And then you configure those profiles once and now all you have to do to add a node to that cluster is assign it to that one profile instead of have to manage that list of profiles for each node that needs to access those things. That's the idea, but it it kind of presupposes a relatively heterogeneous and complex environment that you're doing with one Warewulf cluster. Um maybe more typical would be to have a Warewulf cluster for each environment and then this wouldn't be as useful. Uh, but some people are very far down the Warewulf rabbit hole and have one big master Warewulf environment that does all of their infrastructure.

Um, and we wanted to support that. So, here it is. Awesome. Thank you for that. Oh my. Um, any questions there or moving on to the next thing? Um, I don't see any so we will carry on. If you do have questions, put them in the chat and we'll get to them. So, the next one solves a pretty big group of of trouble that people have been having, similar to what's been going on with uh node image versus um versus container, but this one is a bit more of a functional change.

So when you boot a node, not just with Warewulf, but any any node in any environment like this, um you need an image, you need something that all your software is in that your kind of basic commands, not necessarily the end user software, that might come from containers or that might come from uh a file system that you mount in, but the the the programs of the operating system itself, you need an image for that. And then you also need a kernel that usually goes with that. When you just install Linux on a server, you'll get an operating system image and a kernel all together. Uh but when you're doing this over the network, you have to care about where your kernel is because it's not just on your hard drive.

It's not just on your server. It has to come from somewhere. And so one of the big innovations in Warewulf 4 is that instead of just having to manage those kernels separately, you can get that kernel directly out of your image. Um, and so when the Warewulf server boots, it says, "Okay, you have this image, but you need a kernel. Oh, I'll look in that image, and I'll see if there's a kernel there. And if there is one, then it serves that out." Uh, because that's a relatively recent innovation. And I think it's even beyond like the first release of Warewulf 4. I think that first came about in Warewulf 42 or 43.

Uh um Warewulf 4 has also had support for the first way of doing it where instead of loading the kernel from your image, um you would give Warewulf an explicit kernel and then hand that kernel out to nodes when it booted. it had to be compatible with the image and there was a lot of effort in making sure that it was a kernel that worked with the image and that it was m that it was upgraded at the same time. But over time that that automatic detection got better and better and better. Uh and the fact that you could import kernels directly into Warewulf just ended up being a confusion for people.

Every time someone would just go down the quick start and they'd reach the point where it said and now you can import a kernel. That is where people would break and we would just have to tell them skip that part and it will just work and then they would have a great time. Uh so the first part of this is that whole kernel import process completely gone. Remove from the docs, remove from the code. Um there are ways to do the same thing but it's all part of the auto detection uh path now. But what we couldn't do before with auto detection is if your image had multiple kernels in it, you couldn't explicitly say which kernel to boot and now you can.

So the way that works and finally I can put some stuff on the screen for this. Uh we now see kernels as a subfunction of images. So you can list the kernels that are available in an image with this kernels command under image. So you say Warewulf control image kernels. And now you can see not only the images that have kernels in them, but you can see when an image has multiple kernels that so like this Rocky Linux 9 image has a 427.37.1 kernel and then it was later upgraded and now also has a 427.40.1 kernel. In the past, we would have told people, well, go and remove the old one from your image to make sure that Warewulf boots the new one.

But now we can see in this list which one Warewulf is detecting. It sees that this one here is the default kernel for this image. We see that one of our nodes is booting it. But if we specify to a node um that we want to boot either this kernel we can give it by a path or this kernel we can give it by a version then it will boot that older kernel from this image explicitly. Uh so uh I could say for example ctl node set dubdub node one which is the name of my node and then kernel version is this parameter. It knows that this node has the Rocky Linux 9 image uh set.

So when I tab complete here, it's only going to show me kernel versions for the image that's already loaded. If I didn't have an image set, then it would I I forget if it doesn't show you any or if it shows you for all of them. But in any case, if you have an image set, it does the right thing. And so we can either specify it by a version number uh or we can specify it by a path if we want to get really specific. Maybe you have a a debug kernel of the same version. So you need to specify the explicit path to it.

So this uh is what allowed us to remove the functionality of importing explicit kernels into Warewulf which just ended up being a bunch of confusion for people because now you have all of the same flexibility of loading a specific kernel during boot uh for a specific node. Um, but it's all baked into the kernel autodetection features inside of Warewulf and ends up being again a really good situation and and just works for the common case and lets you be precise when you need to to make an exception. This is awesome. Is this something that you would then manage inside the profile? So yeah, you can absolutely do this in the profile or you can set it in the node either way.

So, normally you will just leave it and it will detect the newest one. And so, one of the things we did here was brought in some some third-party code that is allowing us to detect which of these version numbers is the the highest version number. Um, but if you are setting it explicitly, most of the time you would set it in the profile like you said, the same one that says uh what image to use maybe. Um, but then if you want to try a newer kernel uh on a single node, let's say, you can set that value specifically on a node, reboot that node, and it will use a different kernel from the image before you roll it out to the rest of your nodes.

Wow, that's got to be very helpful. Yes. For a lot of people. Oh, we were very excited to get this in. Yes. Yeah. Um, so I see here that we there's like a lot more things that we can talk about. Do we want to keep going or do we want to like leave a little suspense and maybe do another one um when 46 comes out? Uh I I will leave that to you. May maybe I I'll show I'll talk about overlays. Okay, perfect. Um, so one of the other big things in in Warewulf 46 is um, so in the past, like I said, there were just kind of two big overlays.

There was the generic one that was the runtime overlay and Warewulf init uh, that was the system overlay. And and system and runtime overlays have different uh, kind of behaviors. They they're the same uh the same thing, but they behave differently when they're applied to nodes in different ways. Uh, but all of those overlays were in one place. They would be in uh var lib Warewulf overlays. Um, but as you can see here now, I only have one and that's because we've split overlays between the ones that come with Warewulf which are now in user share Warewulf overlays. And this is where the what used to be generic and used to be Warewulf in it.

All of those have been split out here. and uh sit defined local overlays are in uh the previous location. Uh so the the big win here is people would have issues where they would modify an overlay uh or have an overlay of the same name as a distrib one that we were providing and then when they would upgrade Warewulf they would lose their changes. They'd have to back it up and then restore it later. Separating this lets us prevent that. Um, and it also uh lets us in the future, the goal is to create more overlays and kind of maybe not pack them in with Warewulf, but provide them as part of the project for more things that you might want to configure with Warewulf without having to write the overlays yourself.

And so in the future, the intent is to have a repository where you can get additional overlays as different packages. And when you install them, they'll go here. And then if you do modify them, as you can see here with issue, issue started life as a packed overlay right here. Uh but if you edit an overlay with Warewulf, the first thing it does is copy it to the site location and then makes the change there. And so any modified overlay that you have takes precedence. So if I say Warewulf control overlay list, we can see here all the overlays that Warewulf knows about what is in the overlay and then whether it's getting it from the site uh version of it or the distribution version of it.

So here the only one that I have in my site overlay section is issue which is coming true here. And then we've made other improvements to overlays as well. The big one is that uh so there's a set of um functions that are meant to be used with templates. So overlays the one of the biggest things about them is that uh you can define uh programmatic templates for how to generate configuration files on your system at runtime. Uh there are a bunch of functions that make that easier from a project called sprig sp i g and we're now including that in the template environment when you build an overlay.

So, uh, it makes, um, it makes our overlays better because now we have all access to all these functions that make the overlays clearer and and more featurerich, but it also makes it easier for sites to define more complex, more featurerich overlays in their environment, too, which is going to support this kind of additional growing library of overlays that we're going to start providing alongside Warewulf. Cool. Well, is there going to be a way to like template and share an overlay from one site to the next? Yes. So, Warewulf templates or sorry, Warewulf overlays already have templates. So, uh what we have been increasingly encouraging people to do uh is so go to our nodes thing here.

Uh so Warewulf nodes and profiles and nodes inheriting from profiles as well as network devices and a few other things um can have tags on them. And so the overlays can see these tags and then behave differently depending on how you've tagged the data in your uh node registry here. So that's already what we're recommending people do and and more and more functionality that we bake in even in the distribution overlays uh are being activated by tags. So we added the ability to configure DNS um with our network overlays, but rather than having kind of special cased support for that in Warewulf, you just define tags on your nodes that tell it what DNS servers to use.

But for even more complex things, we recently introduced what we call resources. And so all of the NFS client stuff has been moved to resources. These are uh a name to arbitrary data. It can be a list. It can be a single value. It can be what's called a dictionary or a hash where we have this value equals something. So this you know if you if you know how to read YAML syntax this is FS tab here is a list of two items and one of the items is a thing that has a file and amount hops and a spec and a VSS type and then a second item that has the same values but then we just uh let's see overlay show.

So, in our FS tab overlay, we're just reading that resource. Um, right. It's hard to read this uh kind of on the spot here, but we're Yeah, here we go. We're looking over all of the entries in that FS tab list that we got from our resources. And then we generate mounts for that. So there's nothing special in this like anyone could have written this template now that resources exist and share this out with the community and they don't have to make any changes to Warewulf. They can just make this overlay, share it out and all you have to do to activate it is configure some resources on your node or configure some tags to do DNS or whatever people want to do.

So, we're really excited about the addition of resources and the growing acceptance of tags as configuration parameters for overlays that that will encourage both us and other members of the community to publish and share overlays to configure whatever they want their cluster to do. It's going to be great. Oh man, so great. I I can just think in my mind right now how many of our customers and people I know using Warewulf, I'll be like, "Okay, it's time to upgrade. Let's go. Let's do it." Yeah, we've got things to share. That's really that's great. I I feel like we could go on and on. I know that we have a question from the field.

So, let's put the question up real quick before we close out. Hello, SET again. Does version 46 still only support stateless discless nodes or does it now also support stateful nodes? Good question. going right into the overlay share qu we have been increasingly supporting disk full nodes but that's different than stateful provisioning on them so uh I think at version 456 or something like that certainly in 458 uh there's support with ignition which is a a kind of disk partitioning automation system uh for creating partitions and file systems and mounts for those on nodes as they boot up. Uh that's still a a tech preview. Um but it's there and and there are definitely people using it.

Um what we don't currently support is writing the OS image onto a disk that has been provisioned in that way. So the OS image is still in memory and you can provision disk and mount it in many ways. Um but the operating system itself is still stateless in memory not written to disk. That is something that we are pursuing. In fact, I think there's a pull request currently tagged for 461. Um though I don't I don't know whether it will make it in 461. That's just kind of where things that aren't 460 have gone. um that uh that is targeting doing that as part of the if you've been following some developments in the two-phase boot process for Warewulf, the the current plan, the current kind of tact for that is to do that write of the image during the Dracket stage.

So when when we have our kind of first stage intermediate image running, then that thing would pull the image from the Warewulf server and write it to disk rather than to memory. But we're still working that out um and and finding the best way to do that that has the fewest caveats and exceptions for people. So it we do expect that to happen eventually, but it's not there in 460. Awesome. Those are some really great questions, Jonathan. My goodness. Thank you so much for your time. I think that we definitely are going to need to do this again in a couple of weeks. Um, I think before we go though, we did promise a sneak peek and and I probably shouldn't uh shouldn't tease that and then not do anything.

So, this won't be in 460, but something that we CIQ intend to contribute to the project uh hopefully for uh for merging um shortly after 460 um is a a REST API for Warewulf that we need for other work that we're doing. Um, but I just wanted to put that up here. This, uh, is a little bit out of date. This has not been rebased to the current work that's in 460 today. So, you'll notice like for example, it still says container here instead of image. Uh, so this is running in a slightly older build before some of the the changes that have come in more recently.

Uh, but this here is just a REST API with kind of an automatically generated web interface for it. um that you could well like the example that I did before is you could run a curl against this to get the nodes that are in your cluster. Oh, I need a password. There we go. So here it shows that you know it simulated running this curl against our API endpoint and we got some JSON back out of the list of nodes and uh well not the profiles but the list of nodes and their attributes and we you know we're filling this out. We've got nodes, we've got uh containers, what will become images up there.

Um, and then overlays and profiles are kind of the four main entities that we're focusing on in this API. So far, it's relatively complete within that scope. Um, you can do things like build overlays. You can't edit overlays today, but you can add add and remove things from overlays and update nodes. Uh, we're excited to get this out into the community and then build additional functionality on top of it. But uh since we kind of teased this in the uh in the description of the webinar, wanted to make sure we didn't leave without uh at least showing that it is a thing that exists. That is awesome.

I was going to mention it, but I didn't realize that we were able to show it. So, it's just this far. There there are other things that I would love to be able to show, but we're they're not ready yet. Yeah. Okay. Well, more on the horizon, which is really exciting. I mean that's what you want from any good open source project is more you know so and to be responsive. So again make sure that you join the community on Slack. There is an older one where Warewulf used to be um not that long ago. So sometimes people are a little bit confused. So we'll make sure to put the most current uh Warewulf slack channel link there in um the comments for you.

We also have the YouTube channel. We would love for you guys to be a part of it. If you have any questions, feel free to reach out to CIQ ciq.com. We have we offer training for Warewulf, which is really wonderful. You can kind of like sit down oneonone and go through, you know, everything about it. We also offer support um and um as well as kind of building out your cluster for you, helping you really envision and think about what it is that you want to do and how you want to set it up and help you set it up. So, lots of options for you.

And then of course on the horizon is that beautiful, we're calling it REST API. Okay, so gueies out. No more guey talk, huh? No, no, that so what you saw was not the guey. The rest is is um it's a technical term, but it's a type of API. So, yeah, the the the goal is uh to provide a web interface at top that, but the the REST API is a requisite for building a web interface on top of it. Okay. Okay. Well, thank you for that clarification. Now, now I get it. All right. Well, lots on the horizon, Jonathan. Again, thank you so much and the Warewulf community and everyone involved and all of the users.

Thank you so much for being here and I hope you have a wonderful day and we'll talk to you soon. Bye. Thanks all.