Seamless Upgrade Path to Warewulf Pro: Migrating Your HPC Cluster in Minutes

Um, but people just don't know it exists. And, uh, a lot of people having gotten to 45 have upgraded within 45, but have avoided going to 46 because of the pain of the 44 to 45 transition. So the the 46 command, the upgrade command is explicitly there to ease that transition around updating configuration files and how that's changed. But we also uh received a question from both a customer and a member of the community recently about how to plan that upgrade because the command doesn't exist until you have werewolf 46. So uh we came up with a procedure for that and we're going to demonstrate that today.

So if we can pull up the um the shared terminal here. So what I have uh is a it's it's a virtual machine but in a kind of a home lab environment. Um I have a demo werewolf server that's running 458 and we have one node in this environment and it happens to actually be running right now. I'm going to spare like switching back and forth between windows to demonstrate. I'll pull that up when we're booting it into the new environment. Um but we have this running right now and I'm I'm realizing I've got too many windows going. I have like a a whole script and everything and it's just all stacking on top of each other already.

Um, but this this procedure that we're recommending for how to trial out your upgrade process uh because I I don't have it right now, right? Uh if I run werewolf control, I don't have this upgrade command right now because I'm running for five. Uh is to use Appainer for it. So, we have this um >> this Appainer definition file. It's really simple. It's just a Rocky Linux uh base, like a really minimal Rocky Linux base. And then we installed a newer version of Werewolf into it. And so with that uh and I've already built it into the CIF, you can see right here. If you're not familiar with Appainer, it's a a container platform built for scientific computing and and HPC environments um that generates single files instead of kind of the the Podman or Docker style that that uses a backend registry for this.

So, we have this container image right here and we can run it. Uh, I'm going to run it with sudo because uh my Etsy uh werewolf location is protected. Uh, so we can run it. And here what we're going to do is inside the container we're going to run upgrade config and it's got two subcomands. One is for the config the the the werewolf.commp your main system configuration and the other is is nodes for your nodes.com. you need to update both of those files. Um or there there potentially could be changes to both of those files. So the the command supports them both. So we can check one and we get the newer version of uh werewolf running in that container and uh wait yeah yeah okay uh and we get a new werewolf.commp file here that is updated for the target version here.

We're going to 468 or sorry 464 458 to 464 and then uh we can do >> yeah we can do the same kind of command and you know when we do this for real um we're getting a recommendation here that we run uh um oh yeah we we run our nodes uh with werewolf.com and and I'll describe that in a minute. So when we're doing our our node update so that's what we're doing here. We're running the same container. We're bringing in our configuration from the local at C and putting that inside the container at /host at C. So this is giving our container access to our werewolf configuration so that it can read it in and then generate a new output for us.

And not only are we telling it here's our original nodes.com. This is the file that we're updating, but we're also telling it where our werewolf.com is. And this is because we got this this notice here when we were updating our configuration that there are NFS mount options specified in our configuration file. This is a change from 45 to 46. NFS client mount operations are now part of your nodes.com file rather than your werewolf.com serverside file. So this is telling us, hey, if you want those mounts uh ported forward into the new node side configuration, tell the node upgrade command uh where that file is and we'll do that.

So here we've done that. We've said um we want to start from our old nodes.com. Here's our old werewolf.commp. And then two settings that if we don't specify add defaults or replace overlays then the command will will die and will tell us hey you need to specify these values. Um add defaults is due to a change in werewolf 46 where before there was default behavior that was just kind of baked into werewolf and there was no way to change it or when there was it was pretty obscure. Um, now we're representing all of those default values in the nodes.com file as part of the default profile.

So where those were just assumed before, now they need to be set explicitly in nodes.com. If you have a new werewolf deployment, uh, the default configuration file will just have those settings in it. But if you're upgrading an old one, you need to tell it whether or not you want to apply those default values now into the file. So we we do want to do that. And then the second major change in the configuration file is the what what I handwavingly refer to as the great overlay split. And there there's two parts of that split and we'll talk about one of them right now. The first is that in the past we had uh small or a small number of large monolithic overlays.

There was like one runtime overlay that we called generic and one system overlay that we called where uh called dubdub andit or werewolf andit. Um we have werewolf supports multiple overlays though and they can all get combined at runtime and having them be large and monolithic made it impossible to deactivate portions of those big overlays that you didn't want. So, uh, we had like five different network configuration systems, for example, in the system overlay, in the dubdubanit overlay, and you couldn't turn off the ones you weren't using. Splitting them out into different overlays makes it so that you can. So, we'll go ahead and and uh run this, and we'll get another output.

And, you know, we get some information about things that are happening. uh but you know if if uh review them for your local environment if there's a problem then maybe you need to make a manual change but it will call that out and we'll look at the changes that were made for each of these. So first we'll look at the changes for werewolf.com. This is the the main configuration file for the server itself. And uh so at the top we we aren't tracking like an explicit version number anymore. we just look at what's in the file uh to decide how to parse it. Um logging in 46 no longer goes to SIS log.

We just write write to standard error I think but a standard output stream and expect it to be collected by systemd and data store used to be stored up here or configured up here. We've moved that down to there's there's a section here where all the paths are configured and we've moved that down here as datad down here. Uh this is a a no op. It's just a formatting change just because of the way it got output. Um but here is the change that we talked about NFS uh mounts changing. So Werewolf among other things supports configuring the server as a network file share or network file system server.

Uh and in the past you also configured how you wanted your cluster nodes to mount those shares in this file. Um, we can see here we still configure the exports, what file systems we want shared from the server here, but the mount options have been moved to the nodes.com file, and we'll see those in our new nodes.com. This here is also a noop. It's mostly just getting confused with the fact that container mounts which is a set of files that you want mounted from your host into the image when you're modifying it or when when you're working with it.

uh this year is bringing in uh DNS resolver configuration that has been renamed to image mounts because another one of the larger more visible changes in uh in werewolf 46 is that we've renamed the concept of containers to images and this was mostly just like it's it's completely a nomenclature change. It's just a name change. uh but the fact that we referred to it as container uh in the past caused a bunch of confusion for people thinking that uh there there might be container overhead or performance overhead or that there would be container complexity. We still support bringing in container images as your node image but we've renamed them from container to image and so this value has also uh been renamed to image instead of container.

Uh I think that's it. This is largely a noop. This is just something that was uh that doesn't need to be there if you're not specifying a value. So, it got removed from the output. Uh we'll we'll move on to the nodes file, but first I'll look at my my both audience and my audience captive here. And Ian, any questions about this so far? This looking okay so far? Okay, cool. >> Good job, man. Keep going. Yeah, the the image container to image nomenclature change is um funny how much people either liked that or didn't like it. >> I I don't think anybody's complained about the change and for I think for what uh like the the purpose was to reduce confusion and since we've done it no one has questioned what images are whereas we got questions all the time about the nature of containers.

So maybe we'll get more people uh running into this issue as they upgrade from 45 to 46. Um but it hasn't been so bad so far. And the commands themselves still support the the subcommand container. Like you can still type that command. Um it's just changes in the configuration file here. So we'll do a similar diff between our original nodes.com and the generated nodes.com and see what has changed. Uh a big thing here is that uh before we had uh smaller set of overlays. Where's the original? Oh, we didn't have them at all because they were just the default like I said. Exact. Uh of course.

So before you wouldn't have even seen what overlays were configured for this default profile because they were just assumed. It was just magical behavior. Now, not only are they explicit, but we can see that they are here as the list of the overlays that were previously included or the the functionality that was previously included in those default overlays. So, um this makes it easy if you don't want, for example, one one thing that people often turn off, there should be Oh, yeah, it's in the runtime overlay. This sync user overlay is used to synchronize users between the werewolf server and the cluster nodes. This is convenient in some use cases, but uh interferes with others.

And in the past, if you wanted to turn this off, you'd have to go modify uh one of the built-in overlays. Now, you can just remove it from your default profile and you're good to go. Same with all of these network configuration overlays. We have one for network manager, one for Debian, one for the legacy if config system. Now you can just remove the ones that you don't need in your environment or move them to other profiles if you uh need them for some nodes but not others. Um and mix and match a lot more easily. We're also getting default values for our kernel arguments. The formatting of kernel arguments also changed.

It used to just be a spaceepparated string. Now we have a list of arguments. This makes it easier to mix and match kernel arguments from different profiles and to add arguments on the node without having to respppecify everything that came from the profile. There was one other larger thing that I wanted to go over. Um, so this environment is configured for a two-stage boot. So it uses uh uh it uses Drait um which which creates a small node or a small image that will boot first and then that image is responsible for pulling down the full image and the overlays from the werewolf server. In the past in the first deployment of uh Dracket and two-stage boot support we implemented that as an alternative ipixie template.

So IPixie is our network bootloader or at least the default network bootloadader. Um you used to change the whole configuration for ipixie. Now we have an ipixie configuration that can do both the default single stage boot and also the dracket boot. And so instead of having to swap out that template, you just specify a tag that tells it which menu entry to boot. Now benefit to this is that if you want to test that out interactively when the node is booting you'll see a menu with a a short timeout and you can pick a dracket boot or a a single stage boot at runtime as well.

Um but you can just select which menu entry to boot by default. We can also see that here the home file system is configured to mount by default and this was ported in by our uh or from our werewolf.com. So, this wasn't in our file before. And because we told werewolf control upgrade nodes to refer to our older system configuration, it has that preserved here as well. Um, we're getting a a comment from uh a friendly customer and person in the community about a a previous colleague who would be glad about one of the changes, hopefully all of them, but Yeah, thanks for being here. Um, yeah.

Any What What do you think, Ian? How how are we doing? Any questions? Any any anything not clear here so far? No, it all looks It all looks correct to me. Um, and and I think I think we discussed Buntu and Debian will be moving to Drake by default. So, anyone who is a glutton for punishment like I am, pay attention to the drakit stuff because you will have to learn to live with it. >> Yeah, it it's still a little tech previewy in Werewolf. Um, it it all works. It's great and it ends up solving a lot of problems that people run into with um image size limits in a single stage boot.

Um, but it's a little bit more manual and we'll we'll be adding commands to to make that easier in the future, but for now it's it's a little manual. So, we're not just going to upgrade Werewolf from the community. Uh, this this was installed from the community version. It doesn't really make a difference, but to demonstrate this, we are CIQ. I'm going to demonstrate upgrading it to the the package that is built and available from CIQ. So the the next step is to get our our depot system. Depot is our our kind of content delivery RPM management thing. Um actually you would if you were a CIQ customer you would largely interact with this through portal which is our our web interface to it.

Uh but the depot command line utility is how you can add yum repositories dnf repositories to your local system. So we just have a a package that is publicly available that provides the the depot command. And then from there I'm going to log in and I happen to have already specified my username and password in environment variables so that I will not expose them here on the stream. Uh but these environment variables are useful to know as well because these are the the variables that allow werewolf to pull from protected repositories. So later on I'm going to demonstrate um importing a an image from CIQ portal or depot uh using the web interface, but if you were doing it from the command line, setting these environment variables is also how you would tell Werewolf what credentials to use to do that.

And then one last command here. Uh we will tell depot to enable the werewolf pro repository. And that's where we're going to get our updated werewolf package. So once the command is run, there's nothing really that secret to anything. It's just creating a yum repository configuration file here. And if we looked at that, we would see which repositories were enabled by default. And then there's a just a a key that's been generated to allow it to authenticate to the repository. But from that we can just do our normal DNF upgrade and we we could specify a version number here. Um I'm happy going to the latest that is available which will be uh 464.

Uh one thing that is a little bit uh it's worth noting anyway this is also pulling in IPMI tool which is what werewolf will use if you're doing power management commands turning nodes on and off. Um this was an oversight in previous versions of Werewolf. this dependency was not expressed in the package. So now it is and it's getting pulled in. >> And then uh so our we have kind of our updated configuration files here just for our inspection, but I'm going to run these commands again and it will do the upgrade in place. So we're upgrading our configuration first and by default it just outputs it to the original location.

If we look at Etsy Werewolf, well, I'll just look there. If we look at Etsy Werewolf, uh, I have not only like updated files from the package that I could use if I wanted, but I wanted to start from the ones I actually had, uh, but werewolf.com also has a werewolf.com- old. And this is where, uh, the previous configuration has been saved. And this is the updated one. And then we'll run one more upgrade command here. Oh. Oh, I misspelled sudo. So, we're upgrading our nodes file starting from ets werewolfnos.com. Actually, if I didn't specify that, it would just it would find it in the default location anyway.

Uh, and we're referring to the that old werewolf.commp file to get primarily those uh those NFS client configuration options. And as before, we're telling it we do in fact want to bring in the the default settings into this file and we want to expand any uh locally used overlays into their new equivalents. get some information here about the fact that it's updating to the the new Draet configuration and that it's setting some default values uh for the uh network device type setting it to Ethernet and same for the network mask setting that to I think before it was defaulting to 255 255 2550 and and it's doing that in the background.

So from that my my next recommendation would be to run werewolf control configure minus a. So this this sets up all of the local services on the on the local werewolf server. This minus a says all of them. You can specify them individually if you would like and then restart the werewolf server. This is something that we've seen at least a couple people miss where they've gone through all of the effort to upgrade werewolf and then nodes won't boot because their their werewolf damon is still running the older version. So there's uh at this point we're running 464 werewolf. >> So I have a question Jonathan.

>> Yeah. Yeah please. >> You called it an in place upgrade and then you you talked about a restart. So like what what is there any difference there? Maybe I'm thinking about the Linux world and how there's a big difference in an in place upgrade versus having to restart. Is it the same with werewolf or no? >> So So what I mean by in place I just meant that uh it's updating the configuration file where it is rather than just generating a new one that I would then have to move into place. So, uh, it by default these upgrade commands will and I so we'll go from here.

All I said was upgrade config. It looked for etsy werewolfwarewolf.com and then wrote the new version there. And so I'm just saying it's updating that configuration file in place. Um, you do still need to uh restart the the Damon because uh well, it just needs to be the new process. uh it needs to be running the new version. Um this would still I would consider it an in place upgrade. Uh the alternative to an in place upgrade of werewolf would be to set up a new werewolf server and then switch over to it like set up a whole new instance with a whole new server and and deploy it separately and then migrate my nodes to that.

>> Thank you for that. We do have Yeah, we have a question. Go ahead. Go ahead. Put it up. Hey Mac, I can see werewolf cop pick tech preview in depot, but would it only work on re and rocky9? That's a good question. So Jonathan is going to tell you where the mismatch is there. Yeah. >> So So it's first it's odd that you're still seeing werewolf cockpit tech preview. If that's still showing up somewhere, we need to fix that because um so it's final now and we're trying to prevent people from deploying the tech preview version of it. Um, it does only the the the werewolf cockpit, the web interface only runs on EL9.

So, you know, Rocky 9 or or anything like that. Um, and that's primarily because uh it it builds against a specific version of Cockpit, the the web interface that we're that we're building it into. um with an interface in EL9. We are planning out the deployment of the next version also supporting EL10 so that you could run it on Rocky 10, but we don't are not likely to make a version of the web interface that runs on Rocky 8, which is what most people are asking about when they ask this question. What is worth knowing though is that the web interface talks to werewolf over a network transparent REST interface.

So if you really wanted to use the web interface but you didn't but your your wearable server was running on Rocky 8 and you didn't want to upgrade um you could always just run a small virtual machine uh that's Rocky 9 with the web interface in it and then configure it to talk to your your Rocky 8 based werewolf server and that should be just fine. Um all right back here to what's the next thing in my script? All right. So, I mentioned that we have uh we're we have a two-stage boot. Um, and the two-stage boot depends on a Drait image that is built inside of uh and comes from the image you're booting.

Uh that's that's based on a package. Uh let's let's actually show that. Um so uh nope. Oh image shell. So we have this is the image that my node is running right now. Rocky Linux 8. And in here is a package called werewolf Draett. This tells Drakit how to boot from a werewolf server but it's still running the version from 458. So we want to fix that. And uh this is not the the greatest procedure right now. Um I'll I'll start it and then I'll talk through what's going on and how we're going to improve this.

Uh so this here is running a command in our uh in our image that is installing depot logging into depot and enabling werewolf pro just like we did on the server and then it's upgrading the werewolf bracket package community version of course you could just upgrade the package from GitHub or whatever as well but we want to make uh accessing depot simpler from within here it should be possible but As I was setting it up today, or not today, um yesterday or the day before, um I discovered a bug in Werewolf that was breaking our ability to bring the depot configuration into the image environment and just install the package that way.

And um something I didn't mention is that I am in fact running uh or I I'm running a werewolf 8. Sorry, there are too many versions and too many names of software. I am running a Rocky 8 image here to demonstrate that we can run a Rocky 8 compute cluster from a Rocky 9 server. Uh but that means that if I were to download that package locally and then try and install it, it would probably work, but technically that's a Rocky 9 build of the package and uh it it may or may not be compatible with Rocky 8 um when you install it. So, this is the the most complete way to do that right now.

I think we're planning on releasing these this process as a blog post shortly after this as a follow-up. So, um you don't necessarily need to like screenshot and dutifully take down all of these notes. We'll we'll make these commands available. Um but this is a way to do this um to get the package from the CIQ version of the package from the CIQ system and update it inside of your image. This takes a little bit because once it's done uh it's installed all the packages, it had to update or update the um the Dracket image and then it's updating our our image as well after the fact.

I actually didn't see it. I guess this is where it said it ran Draett and uh I just I expected more log output from it, but it's been working when I've used these commands in prep, so it should be fine. Uh should just take another moment here, but any more questions? Ian, what are you thinking? How are we doing? We're doing great. And actually, I was I was going back to the thinking, positing, and maybe something to try. But I wonder if a reload of werewolf as opposed to a restart would suffice. So >> that would be sufficient if it was only updating the config, but because we actually upgraded werewolf, then we actually do need to restart the Damon because otherwise we're still running the old code.

>> Yeah. Yeah, because everything is in memory and so we have to flush and re refill. >> Okay. Tell you what, while that is going, I can in fact do Can you Can you see this? Can you see the other tab? Oh, you can't see the other tab. That bad. Um, well, if I had this in T-Max, I even installed T-Max, so I wouldn't have this problem. We're we're mostly done. We've created our image and then it's it's compressing now and then it will be done. >> Okay. So, do we need to fill time? Because I always am curious about stuff. I you can ask questions, but our next step will be to build the overlays, which is kind of the last uh the last thing.

And then uh here we go. And now I can type. >> There you go. Uh whenever you're ready, we have another question, too. >> Great. So, the the last thing I'm going to do here before we switch over to kind of the werewolf pro part, I'm going to just boot this system or reboot it. Um and then we can take the question while that's going. So uh here is um I'm going to stop this and switch over to another window. This is the window for Oh, I think that's the wrong server anyway. Uh one moment the node, not the server. And here, thank you. And we will I know the text is tiny.

We're running uh Werewolf node one on Rocky Linux 8. Um we aren't reporting the version of Werewolf we booted from. So it's not going to be quite as good as I would like. Um but I'm going to hard reset this server and just demonstrate that it's still able to boot off of the new or I'm hard resetting this cluster node. There we go. and we'll demonstrate that it's still able to boot off of our updated server uh having updated the software and the image and the overlays and it will just come up and it will be beautiful. But yes, Rose, you said we had another question.

>> Um yes, I think Christopher, hi. I noticed that the EL10 package is available in the GitHub site. The last I attempted that with Rocky 10, I had problems with ISC DHCPD no longer being available and being replaced by KIA DHCP. Yes. >> Do you know if the host overlays have been updated to support KIA DHCP? >> So we have not updated the overlays to support KIA DHCP. Instead, what we are doing is moving our support and what we intend to do in the future is use um uh now I'm blanking on mask. >> DNS mask. There we go. Uh to use DNS mask by default.

Werewolf has had DNS mask support nominally as an optional feature for a while. Um but it has not been the default. And uh that is what OpenHPC is going to be using for their werewolf support in Enterprise Linux 10. Um and so the the packages that we have on GitHub now for EL10 put uh are configured to use DNS mask by default. There's they're early for sure and there's at least one gotcha with it where DNS mask isn't creating the TFTPD directory properly. um or it's it's not doing it and so we need to do it because in the past the the TFTPD um package was doing that for us.

Um so it's it's early days for EL10 support. Uh but it's it's coming and I see there's another question our I will point out that our server has in fact or our our cluster node has booted appropriately from our server and all is well. So >> celebrate that that tiny win and good job werewolf. Go back. Yay, werewolf. >> Uh, yeah, we're ready for the other question, Gwen. Great. Oh, hey, Mac. Would it be the same process to upgrade from 4631 to 4641? >> Uh, so yes, that is wonderful question and I love you for asking it. Thank you so much. So, one of the things, yeah, we've we've been very specific about the versions that we're using here.

That is not because it is a requirement. When we built these um the the werewolf control upgrade commands, they uh we we literally went back through the code uh and looked at every time the data structure that parses the config file has changed and built tests for that version of the config file. So, whatever version of Werewolf 4 that you are on all the way back to like pre-release versions, uh it will do its best to interpret the files that you have and generate a config that is reasonable today uh for whatever version you're on starting from version 460. So version 460 attempts to be able to upgrade from anything in the past and then any version after that is intended to our our goal is to make it able to upgrade from any prior version to itself.

Now just to be very clear that does not mean migration from werewolf 3 or before. That's a very different system and sometimes people have asked that because people have asked for that ability. They're just so different from each other. Automatically migrating from werewolf 3 to uh um to werewolf 4 is it's it's not a trivial operation. It's a better done by assisted men. So a little bit of job security on that one. >> Yeah, I'll be men can do it. >> I will say I'll be doing that soon. >> But it is something that we at CIQ can offer consulting services for. >> True. >> Uh but it's not something that we've been able to automate.

uh at least not certainly not completely. They're just too different. >> And we got another question from >> Oh, yeah. Yeah. Great. And I'll go ahead and get this uh this window shared back up. >> Okay, great. Oh, yeah. This is a good one. So, the other question is related to IPv6. In order to implement it on our nodes in a dual stack configuration, it's fine for us to boot over IPv4. I needed to add net tags for the prefix and gateway as well as modify the network manager overlay to support this. Do you know if IPv6 support is inbound? Yes. So our our uh one of the members of the steering committee uh is both active in open HPC but also works for internet 2 and I know he has been very interested in uh IPv6 support from you know his network networking background.

Uh I personally am also interested in it. We don't hear a lot of demand signals for it from a professional perspective within CIQ, but I'm continuing to raise and and carry the banner of we need to improve IPv6 support. It will probably a lot of it will still be in tags. Um, but not just core being able to address it. My understanding is that in 464 it kind of just behaves appropriately if you just put your IPv6 address in your IP adder field. Um, and I see uh Ian, are you doing that in production? Have you or have you just seen it? >> I'm not, but the discussions we've had in this steering committee and on Slack.

Um, >> yeah, >> it has been mentioned that and this is the favorite phrase that we've been using since Perk. It just works. Um, it is one of those things. It just works if you put it in. I think the only problem or hiccup might be it can't or dual stack might cause >> problem. That that was the question, wasn't it? Dual stack. >> Yeah. So, you have to either stick with V4 or go wholehog into V6 as of now. Uh there will be some testing to see if dual stack can be achieved. But yes, we you you have to do everything in V6 if you're going to go to IPv6.

>> So I'm looking at the command right now. We do have an IP adder 6 uh command that is obvious or or configuration setting. Yeah, thanks. Um so I I am not personally familiar with any problems or successes doing dual stack with just these, but maybe what I will say is we consider IPv6 a thing we want to be supporting. um we just lack have lacked experience in the past with it. So um for the most part, Werewolf doesn't need to know a lot about it. It's just generating config files after all. So if that's something you need and you have seen deficiencies, please open issues and we will uh we will intend to address them.

Um one thing I did neglect to mention, so I talked about the great overlay split and that's in two dimensions. So one is that monolithic overlays were split into multiple smaller overlays so that you could configure and remove them individually. The other thing that happened is that we separated what we are now calling distribution overlays. The overlays that come with werewolf or come from CIQ from uh um from your overlays, what we call site overlays. In the past, those were all just mixed together in var lib uh werewolf overlays. But now, if you look there in this environment, I didn't have any custom overlays. You will see that it is empty.

Uh when werewolf installs an overlay, now they are in user share werewolf overlays. And these are what we call the distribution overlays. If you use a werewolf command to edit one of these, instead of editing it in place, it will copy it to varlib werewolf overlays and your copy will live there. And in this way, if we upgrade werewolf, it will always only upgrade this system version of it and your copy will persist. But there is a an unfortunate side effect of this that has come up a number of times now that I at least am still uncertain how to help people with outside of just raising awareness.

If you have done in the past the thing that we don't want people to have had to do, if you modified one of those monolithic overlays in the past, then when you upgrade werewolf, it will see files that were not modified as part of the package and it will remove them and then put the new version in the new location. But if you had modified a file, that file alone, the modified version or if you added something to one of those monolithic overlays, that will remain as a site overlay in Varlip werewolf overlays. And so sometimes people have made like one small change in the werewolf init overlay, which does still exist.

You see it here. This is required for werewolf to be able to boot your node. Uh but if there's a shadow version of it living in varlib werewolf overlays with just one file in it uh then it breaks the rest of the overlay. So after your upgrade I recommend looking in varlib werewolf overlays and see if there's any overlays in there that are accidentally shadowing the the now preferred system location. And uh and if there are, compare it against what's in the new overlay and see if there are changes. Maybe to fix a bug that we've already fixed, maybe to implement a feature that we've already fixed because that's happened a number of times.

Um or maybe it's just something that you do need to port into a new overlay with a new name. So that's just a quick gotcha to be aware of. But the next thing we want to demonstrate werewolf pro and deploying that in our new 464 environment. Werewolf pro requires at least 463 I think and uh and so we're we're able to do that and we used um depot to provide access to the werewolf pro packages before. We're coming up on time here so I'm going to try and do this as quickly as we can. It shouldn't be too bad. We we we should have enough time.

Uh so the first bit here is this werewolf cockpit package. We're using the uh the cockpit server web interface. It's a generalpurpose kind of pluggable web interface. It's already included in Rocky Linux um to as a base for a web interface for Werewolf. So we're installing Cockpit and we're installing a Werewolf Cockpit plugin here. And this makes the Werewolf uh Pro web interface. And once that's installed, we will enable it. Uh but it'll just take another moment here. Did I Was there any question? Anything? Yeah. Okay. I should have had this running already. We were talking so much. >> That's okay. I >> I I know what we can do.

I I can take this opportunity. And this is an issue. Unfortunately, there aren't um there I can't do multiple windows shared. Uh, but maybe while this is going, I can switch over to this one and talk about what's included in Werewolf Pro. So, I think I'm on this window. Yeah. Um, and now I'm getting Slack notifications. So, Werewolf Pro is Werewolf builds for Werewolf for Rocky 8 and 9. this web interface that we're installing right now, some pre-built uh node images and overlays that go with them. So, we're currently targeting primarily Open HPC, Slurm, and Open PBS as just turnkey cluster solutions that you could install and not have to make changes to the image.

You can just uh enable an overlay and then uh and then set tags or set resources to configure how they should behave and do that all in the web interface. And then uh we are packing in appainer support with that as well because we wanted to be able to bundle appainer into those node images. We're uh strong believers in tainerization for HBC. And then we also are bundling in access to um this uh training curriculum that we put together. We we will sometimes do trainings for our customers but we we make the curriculum for it available. It walks you through doing a cluster setup and using the werewolf probits and installation.

and a lot of what we're going over here, although I don't think we have a a strict upgrade guide. It's more expecting a green field deployment. So, that's all in there. Uh, but my terminal is now done. So, I'll switch back over to that. So, we've we've installed cockpit and then we need to enable cockpit is a little bit different. You don't start the service. You start a socket and then uh systemd uh starts cockpit when it sees connection on that. Oh man. Uh I I would switch over to the web interface to show it but we'll finish doing the configuration here and then I'll do that at the end.

So uh in order to use cockpit or sorry in order to use the werewolf web interface we need to enable the werewolf API. It's disabled by default and we do that in Etsy werewolf werewolf.com. >> So you're not sh you're not sharing anything. Do you think you're sharing something? No, I don't see anything. >> I'm sharing it, but it's not. >> Okay. Well, you're looking at it. No one else is. >> There it is. >> Thank you for that, Rose. >> Yes, you're welcome. So, so yes, we are uh in uh Etsy werewolf werewolf.com and uh we just need a new top level thing in here that is API and then I think it's just enable is it enable?

No, enabled enabled true. And then one other thing we can do while we're in here, a new feature I think in 464 is that werewolf control configure can restart werewolf itself, but it won't do it by default. We need to tell it what name to use. So I'm going to go ahead and do that. This is not required, but this just tells that werewolf control figure command how to uh uh how to restart the werewolf server. And then uh we're going to use this make password command to generate um the hashed version of our API key. This this is again a little bit uh manual right now, but we need to uh set an API key in a new configuration file to allow us to log into the um the werewolf web interface.

And I'm just entering a password here. This is the format of that file. And we are using the make password command here to hash the password that will go in here. I'll it ends up looking like this. That goes in etsy werewolf.com. So well with sudo uh we just need a file there that says who should be allowed to log in. We've created a username admin and given it a password but that password is hashed. So I typed it in here when make password asked for it and it generated this configuration file for us.

With that we can just run uh werewolf control configure minus a all of them and among other things it is restarting the werewolf server for us because we told it how uh we will do one more thing here so I'm not switching back and forth as much at this point we could log into the werewolf web interface but I will also install our overlays And we see right now uh we have overlays that configure it. It says appainer. This is doing um what are they called the the subuid and subg uh user ID mapping. It's something you need for rootless appainer to work. So we've packaged it in an appainer overlay.

It's configuring that for you. Um we're also including a crony. This is a a network time damon client uh configuration. This lets it makes it easy to tell your cluster node to get its time from for example the werewolf server. Um and then we also have an overlay for fuzball substrate. I didn't mention that uh along with PBS and slurm and uh and open HPC in general we also have turnkey options for deploying fuzzball in a werewolf environment here. But then there's also overlays for PBS and slurm that configure these nodes to be uh PBS and slurm nodes. So those are installed. I think that is the last thing that we will do here at the command line.

So we will switch over to the web interface one more time and I will I will risk uh showing behind the curtain for just a moment um to have a bare browser open. I want this one. I think it's that one. Is it that one? Yeah. I guess it doesn't really matter. um because we're going to go to this URL which you're seeing in my procedure now. Uh but this is 9090 is the default port for cockpit and then werewolf demo one and I have uh multiccast DNS support set up so that I can get it by name. But we'll go here. I'm logging in here with my server user if I can type my password properly.

And here we're just in cockpit and because I said /warewolf, we're on the werewolf pro version of that web or side of that web interface. We don't have our API key configured yet. So we're prompted to enter that. We can just enter the username and password that we entered earlier when we were setting it up. And now we have the web interface running. We can see our node that is configured and the default profile image that we have available here. and the overlays that are available. You can inspect these over Oh. Oh, you should be able to inspect these. Why is that not working? I'll reload.

Okay. Oh, you know what? It's because I picked a file that is a sim link and we have discovered a bug. I just happened to pick a bad one. Uh that's a bug that we need to address. We will look at an actual file instead. Uh this here uh shows you how a template is you know can be rendered on a node. This is what it looks like when you re well you wouldn't normally render this one on a node. This is from the host overlay but uh you can see here and then the last thing I'll demonstrate is over here in the images tab. So we have this image here but we can also import an image from uh from depot.

And so we'll we'll tell it that it's an open HPC image. We're going to give it this URL. So we're going to pull it from a Docker registry from Depot. Um and then uh Werewolf Pro here. And uh you know, this is the location that you would get from depot. So if you find the uh the image, I'll just go there. We have just a few more minutes left. This will be fine. So, if we're in Werewolf Pro and we look in node images, we can find the image we want. And this is the one that I'm using here. It's this Rocky Linux HPC image. And scrolled over here somewhere.

If my window were wider, there we go. Uh, I can get the image from there. And then we would paste that in here. Uh if you visit and I'll show again here. Oh, is this the right place? Yeah, if you visit uh this location portal.ciq token, you are told what your username and password are. Again, I'm not going to expose that here on the stream. Uh but the username is is ciq portal. That is I think a generic one. And then uh I will put my access token in here. And you can import an image from here. On this server, this takes a little while. So I I don't think we should wait for that to be done.

But this is, you know, showing how easy it is to go from a 458 environment through some steps uh to a 464 environment and then, you know, install a few packages, add another config file in the place, and and you can be doing this with a web interface. Oh man, whirlwind. and under the gun there. But oh yeah, you have uh your password asterisk as well. >> How are we doing? Any any remaining questions, Ian? Any questions here about the web interface or anything that you saw here? Any Actually, this is a better question. Any suggestions for improving this process? What do you think? >> I It's pretty straightforward.

Um, you know, all the all the gotchas you've outlined pretty well and I don't think there's any way to avoid them based on what's going on under the hood. >> I really wish we could find a way to to fix the issue with the overlay split. >> Yeah, but I mean the thing is it's a lot of a lot of parts, a lot of moving parts under the hood. It's a complicated process. So, trying to simplify it as much as possible. And I I think you've outlined that the steps needed um you know it's it's it's a straightforward process with the the caveats provided and >> as long as you know what the process is >> right check your work right proofread proofread proofread.

So yeah, every time a file is modified automatically by anything, you should always look and see what's going, you know, make sure it's looks like what you expect it to look like. Um, you know, I, you know, I'm sure we've all done it before. It's like reading license agreements. Yeah, whatever. Check. Okay. You know, you you let it go and then it comes back and buy you later. So, especially with the 45 to 46 because of the significant changes made there, definitely look through the config files and make sure everything is where it should be. Um, >> starting in door six, uh, we also started generating pretty full release notes for each one.

And so, you'll see them here, uh, at werewolf.orgdocs. And if you just go there, you can navigate. It's pretty straightforward from there. Um, uh, I I highly recommend checking out the release notes between whatever you're upgrading from to whatever you're upgrading to. Um, and there's a lot of a lot of good information in there. >> Yeah. Thorough. Appreciate it. We got a question from Jeff. Go ahead and put that up. Thank you. >> Oh, hey Jeff. Any guidance on upgrade steps from previous versions to get to 458 first? Yeah. So there's there's two parts to that question. If your goal is actually to get to 458, that can be it's manual there.

There's there's, you know, we can help with it. If you want to come into Slack or if you want to engage with CIQ, we can help with that. Um, but if you're going from you don't need to upgrade to 458 before you upgrade to 46, whatever. Um, you can go directly to 46 from your current configuration. And uh the upgrade scripts that are available or the upgrade commands that are available in 46 can help you upgrade from that older version. You don't need to upgrade through 458. Um if you're already on 45 something, upgrading to 458 should be safe enough. Um but the 44 to 45 upgrade was a little bit uh was was a bit of a struggle which motivated these improvements.

So, um, I don't recommend if if you don't if you're not running four or five something today, I don't recommend going to that. I recommend just going to 46. >> Good point. Um, another one from Jeff. A student asked about that in our class. Yeah. >> Oh, yeah. >> Yeah, that makes sense. >> Great. Any other questions? >> I think maybe we're good. Um, oh, >> okay. How to create a recoverable backup export before each upgrade as rollback plan. >> Great. So, this is something that probably would be worth us actually supporting first class with some kind of a a backup or export. Right now, it's a little fragmentaryary, but I'll talk about what you should look into doing.

And I'll you I said we wouldn't go back to the command line, but here we are. So, if we can get that back up here. Um, so the things that you need to back up is this directory, Etsy Werewolf, and then varlib werewolf and frankly probably just varlib werewolf overlays and maybe varibwolf uh trroots. This is where your images are stored. Um, and this is where your overlays are. Uh, this is a temporary location. It's where the things are generated. So you probably don't need to back that up. If you really wanted to be thorough, you could just grab this whole directory. But all werewolf state will be in varlib werewolf and then all werewolf configuration will be in Etsy werewolf.

So if you grab those two directories, you should be fine. And the the biggest space in in in that backup will be depending on how many nodes you have. It will either be the images which are large individually, you know, depending on how big your image is. And then if you are going to up this provision directory um that can be large if you have a lot of nodes. Each individual built provisioned overlay thing in there is not well I guess provision is large uh because the the built images will be in there also. Um but uh your overlays can be large if you have several thousands of nodes.

But you probably don't need that. You probably just need uh at most uh true roots and definitely overlays. Awesome. Well, I think that we are at time. We have been chatting here and getting some good information from Mr. Jonathan Anderson. Thank you so much. And thank you, Ian, for showing up too and hanging out with us and sharing your expertise. You guys can find them both in the werewolf Slack. So, um just a couple of you who we we saw the names. I was looking, not all of you are in the Slack. So, if you have more questions or if you have things to share, you know, like um uh the IPv6 stuff or or anything that you're doing that you think might be interesting to other people in the werewolf community, please, please, please go to What are you writing here?

Werewolf.orghelp. >> I think that's where it is, isn't it? >> Yeah, there's a link to the Slack there. >> Yes. Yes, we have links all over the place to join the Slack. We would love to see you in there. Please come and join us. The technical steering committees are also available for you to join. Uh we have a a user group now werewolf user group a wug on the second Wednesday of the month. Very exciting. So we've already done two now which is awesome. Um so come and and and join us and and continue to uh provision with werewolf. So we're so excited. Um also if you need help CIQ right this is uh the company that you know myself and Jonathan Anderson work for.

We do provide training and support. Um, and of course you get the wonderful UI that Jonathan and the team have been working on behind the scenes to kind of ease the transition, help you get going. So, we would love to chat with you, ciq.com. You can reach us there, fill out a little form. We'd love to chat with you more. And again, thank you so much for your time, both of you, and everyone who's been here. Appreciate it. >> Thanks, Rose. >> Have a great day. Thank you.