Research Computing Roundtable - Turnkey HPC: Warewulf & OpenHPC
Up next in our Research Computing Roundtable series, our panel of HPC experts will be discussing Turnkey HPC and focusing on Warewulf and OpenHPC.
Don’t miss this exciting and informative live stream!
Webinar Synopsis:
Speakers:
-
Zane Hamilton, Vice President of Sales Engineering, CIQ
-
Gregory Kurtzer, Founder of Rocky Linux, Singularity/Apptainer, Warewulf, CentOS, and CEO of CIQ
-
Jonathon Anderson, HPC System Engineer Sr., CIQ
-
Forrest Burt, High Performance Computing Systems Engineer, CIQ
-
Glen Otero, VP of Scientific Computing, CIQ
-
Gary Jung, HPC General Manager, LBNL and UC Berkeley
-
Misha Ahmadian, Research Associate, Texas Tech University
-
Jeremy Siadal, Integration Engineer, Intel
Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.
Full Webinar Transcript:
Zane Hamilton:
Good morning, good afternoon, good evening, wherever you are. We appreciate you joining us again on another CIQ webcast and doing our research around the table. This week we're going to be talking about Warewulf and OpenHPC. We have a fairly large cast today. Have some new faces, some we last saw a while ago. I'm going ahead and doing introductions real quick. I'm going to go around the screen the way I see you. Gary, please introduce yourself again.
Gary Jung:
My name is Gary Jung. I manage the scientific computing group for Lawrence Berkeley Laboratory. We manage the institutional HPC program for the laboratory, and I also wear two hats. I also run the HPC program for UC Berkeley.
Zane Hamilton:
Thank you, Misha. First time. Welcome.
Misha Ahmadian:
Thank you very much for having me. My name is Misha Ahmadian, and I am from the High Performance Computing Center of Texas Tech University.
Zane Hamilton:
Thank you, Jonathon.
Jonathon Anderson:
Yep. My name's Jonathon Anderson. I'm an HPC Engineer with CIQ.
Zane Hamilton:
Jeremy, welcome back.
Jeremy Siadal:
Hey there. Jeremy Siadal, with Intel Corporation. I'm an Integration Engineer at Intel and the lead liaison to the OpenHPC project at Intel. I also represent the OpenHPC project.
Zane Hamilton:
Outstanding. Thank you, Glen.
Glen Otero:
Hi, Glen Otero. I'm the Director of Scientific Computing at CIQ.
Zane Hamilton:
Greg.
Greg Kurtzer:
Hi everybody. I'm Greg. Open source guy and CEO, now.
Zane Hamilton:
Thank you, Forrest. Welcome back. It's been a while.
Forrest Burt:
Thank you, Zane. I'm Forrest Burt. I'm an HPC Systems Engineer here at CIQ.
Background of OpenHPC [00:01:58]
Zane Hamilton:
Thank you. Let's go ahead and dive right into it, guys. First, I want to open it up and ask somebody to tell me the background of OpenHPC, where it came from, what it is, and where it is today.
Jeremy Siadal:
Sure, you want me to take that question?
Greg Kurtzer:
I was going to say, I know just the person for that.
Jeremy Siadal:
Let's see here. I haven't tried this before, but let me share something. This might be helpful. As I said, I've not tried this before. We'll see how this works here. I'm just looking for where we can go. Make sure I got the right one.
Suppose everyone gets a chance to see that. This gives you a quick background on OpenHPC. It's a reference collection of HPC software. Specifically, it is intended to be open source and architecture neutral. I also like to view it as a collection. Overall, I like to view it also as a software distribution. It's not low level like Linux, but what it does is on a basic Linux system. It's going to sit on top of it. It will provide a full collection of HPC software, including the provisioning system, resource management, and most other components that are needed. I am trying to remember if we've got the timeline. I need the timeline handy. But just a quick background. It was originally launched in 2015 with 1.0. The most recent or major update was in 2020 when we launched two 2.0—supporting at the time CentOS 8, SLES 15, or openSUSE 15. It has now been updated to support Rocky 8. But that's a quick background.
Zane Hamilton:
That's great, thank you. One of the pieces that are involved in OpenHPC is Warewulf. Greg created Warewulf a long time ago. We've discussed that before, but how does Warewulf fit in the OpenHPC world? Greg, that's a question for you.
Greg Kurtzer:
Well, one thing I wanted to mention even before that is early on, like when OpenHPC was starting, I was on one of the technical, I think it was a technical steering committee. Jeremy, you may remember better than I do where in the organization. But I was brought in as a department of Energy and Berkeley Lab representative. Interestingly, Gary was my boss and approved my time to be spent on OpenHPC and to help with the project. I go way back with the project as well. I was invited because of the interest that OpenHPC had in leveraging Warewulf for that provisioning. It's been fantastic to be part of the project early on. Then again, watch it take off and continue. Very, very cool. Now, Zane just remembered what your question was.
Warewulf Working With OpenHPC [00:05:36]
Zane Hamilton:
How does Warewulf fit in with OpenHPC? Like what role does it play in that stack?
Greg Kurtzer:
I accidentally just touched it. Warewulf is an operating system, provisioning, and management platform specifically for clusters. It was built around high performance clusters, making it efficient for system administrators and engineers to build and manage their systems. People use it outside of HPC, but HPC is its dominant use case. OpenHPC has leveraged Warewulf as the primary or one of the primary provisioning solutions that it uses within OpenHPC. It's been focused on Warewulf version 3 for the whole life of OpenHPC. More recently, we've started building towards Warewulf version 4. In this conversation, we'll touch on Warewulf version 3 versus 4, the benefits, and maybe even the OpenHPC direction. That's where it's focusing within OpenHPC, OpenHPC as a plethora of different libraries and applications and middleware and solutions to make Turnkey HPC as easy and reproducible as possible. They've done a fantastic job at that. Again, Warewulf is just sitting on how we help provision and manage those images on compute notes.
Warewulf Working With Node Images [00:07:17]
Zane Hamilton:
I think one of the things we hear quite often when we talk about Warewulf is images. Quickly, Jonathon, how does Warewulf handle the portability and flexibility of node images?
Jonathon Anderson:
This has changed and, in my opinion, drastically improved even over other previous versions of Warewulf. Historically, you've created a special directory, a true directory that you've installed packages into and handled manually upfront. Then you could shell into that image and make changes to it. Similar to how you would if you were SSHing or logging into a system and making changes to it, then distributing that image to other nodes. Warewulf 4 refactors this through vocabulary and a little integration on top of the semantics that has come up through the Open Container Initiative through the Docker project and similar things. Now, rather than having to, you still can, but rather than creating that node image from scratch by hand, you can initialize it from an upstream OCI container through a registry or through an image you bring with you.
That makes the whole thing make a lot more sense. You can start from that and then continue to shell into the image and modify it the way you used previous versions of Warewulf or other similar systems. You can keep doing that. Or you can do what I prefer and never change that image. Just modify your container definition file, and then you can keep that in Git or another revision control system. In that way, track the version history and the changes to that image over time without having to keep track of this big binary blob or special directory that contains your image.
Difference Between Warewulf 3 and Warewulf 4 [00:09:16]
Zane Hamilton:
That's great. Thank you. Misha, I will pick on you next because you're going through this right now. You're currently a Warewulf 3 user in your environment, and you are going through that change to 4. The question is, how different are they in your mind?
Misha Ahmadian:
Warewulf 4 has been great work compared to Warewulf 3 regarding managing the provisioning. Warewulf 3 had some issues for us. One of the big issues was the database. It would help if you kept everything in the big database, and you have to keep those blobs in the database, like gigabytes of blobs. Even extracting data from them and shooting those images to the nodes was a pain for us sometimes, but not always. The new design is a game changer at this point because we're not going to have that big database. We'll have a new way of seeing the containers, which used to be a VFNS thing in Warewulf 3. Moving towards containers, it's a whole better idea as Jonathon mentioned that you could keep track of things that you have done inside your container.
The big problem we currently have is too many images for different systems for provisioning. After a year and a half, we didn't know what was happening in those images, so we didn't have good documentation. Sometimes people keep adding stuff to the image, fixing things inside the image. There is no track of them. Having a good documentation recipe and then keeping it somewhere for your organization or your institute and trying to get that recipe for you every time is the best way you can track it. The next person can pick up and work on it if you are gone. Now, we still need to do it. We still need to move to where both 4 and we're still working. Jonathon is helping us at this point. We are likely targeting the end of this year to fully move to Warewulf 4 and all the nodes in our cluster provisioning from Warewulf 4.
Zane Hamilton:
Thank you, Misha. Gary, I've seen you nodding your head. I know that you've had experience with Warewulf 3 as well. What do you see as the differences, and how is that impacting you?
Gary Jung:
We haven't used Warewulf 4 yet, but we are at Warewulf 3 and a half. Thanks to Greg, since Greg was at the laboratory, we saw a lot of the early thinking for Warewulf 4 in our version of Warewulf. Someone mentioned, for example, how you would CD into this directory and build your VNFS. Early on, we realize that only the person who did all that work understood what was in that build. We needed to find a way of reproducing it easily. When Singularity came along, Greg invented it too. Then we started using Singularity to manage our build. It was like an early form of using containers to manage our VNFS builds. We still do it that way. It's a precursor to what we'll be seeing in Warewulf 4. We're looking forward to that.
Zane Hamilton:
That's great. Thank you. Jeremy, from the OpenHPC perspective, what do you see the difference or how have they impacted you? Oh, I can't hear you anymore. Sorry, Jeremy.
Jeremy Siadal:
The first major change occurred when we were doing internal migration to openSUSE 15.4. There were some changes in the way openSUSE 15.4 compresses the modules. It's not compatible with Warewulf 3. Warewulf 3 and its underlying structure require this intermediate boot layer, and that boot layer has to, by design, be able to support the operating system it's booting. Warewulf 4 is, it jumps directly into booting the operating system. We can bypass all of those requirements, and everything works great. Internally there was a move and a push even before it was available in OpenHPC to use Warewulf 4 because it has many more capabilities.
In the recent TSC two weeks ago, we discussed Warewulf 4. It'll move into the factory first. I sent a new PR for an updated build yesterday to include the 4.3 sources, plus a few patches from commits. I expect that to go up to be available relatively soon. But the answer to the question is there are two options that we have if we continue supporting SUSE: either we have to update Warewulf 3 or move to Warewulf 4. The clear channel is moving to Warewulf 4.
Zane Hamilton:
Great. Thank you. Glen, you've been playing with this lately in some interesting ways and getting back integrated with it and playing with it. What are your thoughts?
Glen Otero:
It's a big step up from 3 to 4. Having a Docker file for your container instead of just creating a true. There's no record of what people have put into the truth, for example, to know what's in there. That's a huge step forward. As Jeremy mentioned, too, there's no shim. It just boots up your kernel and your file system right away in a stateless manner. It's more agile, has quicker booting, and is more streamlined. It's a really good step in that direction, and I've been able to use it with Ubuntu, Debian, and a bunch of distros. It's been really good.
Zane Hamilton:
That's great. Thank you. Forrest, I'll come to you last on this question. They've been playing with this for a while as well.
Forrest Burt:
I've looked into it. For the most part, I'm excited about what Glen is similarly excited about. The reproducibility of the container is a central focus of it now. Beforehand, as you mentioned, with truths and stuff, it could be difficult to keep track of exactly what went into that. That's something common that we talk about with those sandbox type containers that you could have some original definition. Still, as you modify it, those could be lost. One of the main goals of containers and that type of thing is reproducibility. I'm most excited about the ability to integrate just containers from definition files directly so that they can all be tracked a little bit better.
Overlays [00:17:16]
Zane Hamilton:
I think we've touched on quite a bit about what to be excited about, but Jonathon, I know there's one interesting thing and that you touched on it a little bit or someone did, but talking about overlays, can you explain that a little bit deeper?
Jonathon Anderson:
This is one of those things. The fact is, I had heard of Warewulf before I came to CIQ. I was familiar with Warewulf 3. It was something that a past colleague of mine had mentioned, but I'd never really used it. My first hands on it were with Warewulf 4, and I needed to figure out how to imagine running a system without the overlay system from Warewulf 4. Then even 4.3, I came in right before 4.3 was released where you could take... I'll back up a moment. Overlays allow you to bundle up configuration files. They get compressed into an image that gets shipped across to the node to customize your node's state at boot time and then separately on some interval. By default, it's once a minute.
You can have different overlays for different nodes and different groups of nodes. In 4.3, you can have multiple overlays that get layered and concatenated together rather than having a single overlay image that goes to a given node. This makes configuration management, such as in an image-based system, just really, and Glen said earlier, really agile, really straightforward. It just does what you want. It's been great to have access to it. I know there was a way to do this before, but the one in Warewulf 4, especially 4.3, is flexible and easy to express the kinds of things you're likely to want to do in a clustered environment like this.
Kernels in Warewulf 4 [00:19:13]
Zane Hamilton:
That's great. Greg, the kernel question comes up quite often about kernels and different versions of OSS on things. How easy or difficult is it to deal with kernels in Warewulf 4?
Greg Kurtzer:
I'm going to start actually with Warewulf 3. Warewulf 3 in every version of Warewulf previously handled the node provisioning in two ways. There was the operating system, like the runtime user space portion of the operating system. And then there was the bootstrap. Now the bootstrap is basically the kernel, initial ram disk, and initial kernel drivers that you need to get the stack up and running. But that was all tied to everything. That was the kernel and that bootstrap. Now you have to play this mix and match game between what kernel you are running. Determining what drivers are there and what your user space will sync up to or match again. This gets really complicated when you start thinking about things like GPU stacks and thinking about luster and other things.
Whenever you have a kernel component and a user component that has to match, you have to do this by hand in previous versions of Warewulf. Warewulf 4.0, 4.1, and 4.2 did it the same way as everything did in the past. We had a bootstrap or kernel, and then we had our operating system user space component. This made it very, again, difficult to go and just install like GPU drivers because you had to install the user space component into your user space portion of the stack, the container, or the VNFS and do it in the kernel as well. Again, very disjoint. Warewulf 4.3 changed that to the point where the entire kernel stack exists within that container.
This is important as we start thinking about, especially in my mind, things like OpenHPC. OpenHPC includes GPU drivers. It includes luster. It includes other facets to ensure you're managing a close relationship between the kernel and your user space. Now with Warewulf 4, we've talked about this a little bit. There are some key differences. If you're already used to playing with Warewulf 3, and you've been using Warewulf 3 or the previous version, there are some key differences. One key difference is that we've swapped out the notion of the virtual node file system, the VNFS, for a container. This means, and we've touched on this a little bit here, so we can take a docker file, a Singularity or an Apptainer recipe or whatever, build a container, then import that into Warewulf.
That container build process could be manual, it could be true if you want, or it could be coming out of a CICD process. Where you're constantly managing and controlling who has access to that, what's going in there. And you can share it between many systems: OpenHPC and one of the visions I originally had with this. Jeremy and I have talked about this now for years at this point, Jeremy. How do we take OpenHPC, a custom image, or a piece image that somebody can expand? How do we take that image and make it as easy as possible to get it out to compute notes? With Warewulf 4, you can import the OpenHPC container. It includes the kernel. It includes the kernel drivers. It includes your GPU stack. It includes luster if you wish. Everything you need, you import directly into Warewulf. Then you start saying, okay, I want this node to run OpenHPC, this OpenHPC container. I want this node to run that, this node to run it. Or do I want these thousand nodes to run it and change it in just one configuration? Quickly, the configuration for Warewulf has also changed. It used to be in a database. This has now come up, at least once or twice, that we've alluded to. Now it's in a Yammel configuration file. We've been able to scale out that Yammel configuration file to the north of 10,000 nodes to ensure that it was a scalable interface. Now, if anyone's running 10,000 nodes with a single Warewulf controller on a flat broadcast domain, your network team would probably like to speak with you.
Zane Hamilton:
Why? That sounds like a great idea.
Greg Kurtzer:
Until it's not.
Zane Hamilton:
Exactly. Misha, I know you're going through this right now, and as Greg said, you're moving from 3 to 4, and things are different in trying to make it easier. What parts of that are you looking forward to being able to do other than just getting away from the database? Are you going to have those multiple images? Is that something that you're going to utilize?
Misha Ahmadian:
A couple of things are very important at this point. Migrating from Warewulf 3 to 4 is a very big process. Your system is actually under production, and you cannot have Warewulf 3 and 4 on the same system and even under the same network. I learned recently that there is a conflict between the DHCP on two different nodes if you want to run Warewulf 3 and 4, and you should be able to manage that. Now, we are looking forward to coming up with the right design and implementation of the new Warewulf 4 on our system. Then we have to start building new images for different systems. We need different images because we have different networks and sometimes different architectures. Because of that, you have to install some of the drivers for OmniPath, differing from Infiniband, for instance. That requires us to have a plan ahead of time and migration from Warewulf, where 3 to 4 is a tedious job to do. We have to ensure we are doing everything correctly with all services on the system at this point.
What to Expect From OpenHPC [00:26:00]
Zane Hamilton:
Thank you, Misha. Jeremy, on OpenHPC, I know many things are coming now, but what else can we expect to see come out of OpenHPC or are you focused on right now in that environment or in that project?
Jeremy Siadal:
I'm focused on a couple of things; the main one is Warewulf 4. I'm also looking at updates overall to support the latest openSUSE builds. Regarding what is available, that would be a good discussion. I wish I had some slide decks talking about some of the things that are coming out. In the next builds, we'll typically be updated to support the latest operating systems. One of the pushes I'm certainly making is to support openSUSE 15.4 because it has been out for a while. We also had an agreement to move the tool chain from what is currently GCC 9.4 or 9.5. That will be moving to 12.1. Some updates are coming out.
Security On Warewulf [00:27:22]
On a side note, one thing I want to talk about when we're still talking about the migration from Warewulf 3 to 4. We haven't touched on it, and that is security. One of the key elements we added with Warewulf 4 is the ability to secure your provisioning system. This has always been a sore point, with Warewulf 3 fairly open for other systems to get in. But some specific features were added to Warewulf 3, including the ability to wait for you to store. Greg can explain this better. You can store the ability for Warewulf 4 to look for specific information coming from the node before it sends an image down to it. One of the things that we're looking forward to is using Warewulf 4 because of the additional security features.
Zane Hamilton:
No, that's great. Greg, if you want to dive in a little bit and talk about security in Warewulf 4, that is something that I just glossed over and didn't even bring up. That's important.
Greg Kurtzer:
This has been a general problem of provisioning systems forever if you can get pixie boot a node. That node can start pulling all of the necessary bits, layers, information, and configurations from a control system, then what is stopping a user from doing that? If the user can do that, is there any material we are sharing with the compute nodes from the control node that could or should be controlled? Anything that might give them the ability to gain insights or data on exploiting and gaining privilege within that system? Warewulf 4 does a couple of things fairly differently. People typically manage this. By the way, they have an out of band provisioning network and use it just for provisioning and management.
When the node operating system comes up, that network is unavailable; it's either V land off or not configured. You have to get to the root to set up that network. If the users are not root, they can't ever do that. You end up with this level of security that requires root to get root. Usually, that's good enough, but not all systems and architectures can build out of band provisioning networks or fabric. We wanted to do some things that further enabled the security and made it. Hence, the only way you can get this data, get the provisioning data configurations, and whatnot for your node is if there's some secret key we can leverage on the hardware.
We ended up using asset keys for this. Most platforms support asset keys, and you can put in a freeform string in your system's firmware for an asset key or leverage a random asset key for that. Add that to the Warewulf configuration. Warewulf will only hand out those bits if those asset keys match. Now a user who's SSHing and coming into the system without physical access cannot get to the bios of those systems and doesn't have root won't be able to get those asset keys. This gives us a level of protection, which, in provisioning, we have not had before. At least not in Warewulf.
Zane Hamilton:
No, that's great. I want to tell Gary I saw you shaking your head again on some of that, and I want to allow you to talk about, from your perspective, the security around provisioning.
Gary Jung:
We're glad to see that addressed. We weren't all that concerned that somebody, if they knew that obscure path, could get that data off the master node until we recently produced a secure computing and research data platform for the university, which will handle PII and HIPAA data. Then we had an outside firm come in to evaluate the certification to 800-171, a NIST standard for sensitive data. Then it became an issue, so we had to figure out a workaround. The fact that we would not worry about that with Warewulf 4 is going to be great.
Zane Hamilton:
That's great. Thank you. Jonathon, I'll pick on you again because you've been playing with this lately. How have you been dealing with it and playing with it? Can you give me what you see coming, and I don't know, open it up and let you guys talk about whatever you want to? How are you? What has been fun for you to deal with?
Warewulf 4 [00:32:35]
Jonathon Anderson:
I don't know. That is broad. Warewulf 4 is great, and I've been trying to get more people to use it. There is broad applicability in the HPC space, especially the academic one where I come up. I don't want HPC provisioning with a different tool right now. It is the best of the breed. I see opportunities for it outside of that as well. I've been thinking a lot about provisioning storage clusters, particularly with Warewulf. I need to remember if it was mentioned here, but we've been talking a little about what it would look like to provide Luster systems with it. I have experience with BGFs in the back as well. I'd like to spin up that type of cluster with Warewulf.
Also, just putting up libraries of starter containers and so the work to create that container vial is minimal, especially working off of the examples already available in the Warewulf docker hub space, account, or whatever. You can see how we did that with Rocky and replicate those experiences with your container. There's no reason why we couldn't start putting up different versions of containers. I have a small pile of containers to support, like the Mellanox OFED and InfiniBand, using the built-in support with Rocky and supporting OmniPath. The ability to replicate those experiences goes beyond just your ability to replicate what you did locally. Once you can replicate it, someone else can too. As the community grows, that'll only happen more quickly when Warewulf 4 shows up in OpenHPC. More people will be using it, and there will be more opportunities to share containers with which people have had success.
I'm hopeful we'll find a way to do that with overlays. There's nothing in Warewulf for that right now, but as we see that system become packages of things that can be composed together, we might see a use case for being able to grab, export, and import them too.
Greg Kurtzer:
I have something I want to jump into real quick. I didn't mean to cut off the question, but I am excited about what this looks like now to provision systems, OpenHPC systems, or anything. One process you could use is installing whatever OS you want as your controller's main operating system. You install Warewulf into that at some point. We'll get into Epple and other things as well. It's super easy to do, and then you start importing containers. Imagine if you want to build an OpenHPC system. You go and import the OpenHPC container from their documentation. Here's our string, and here's the container. It's pre-made. If you're running this architecture, here are all the different variants of OpenHPC that you can choose from, whether you're doing an IO node, whether you're doing a controller node, or whether you're doing a compute node. You can now select the images you want or use other images that other sites may have been basing on, and you can leverage that.
You can import that into Warewulf. You can configure your nodes at the Warewulf command line at the WWCTL command. Configure things like your IP addressing your nodes. Any other information you want with those nodes, et cetera. Configure your overlay if you need, the default overlay will probably work fine for almost everything you need to run, and you turn those nodes on. Within minutes, you can set up and run your cluster with a quick how-to. Now we've done this at Berkeley Lab with a number of clusters, and there's one story, everybody, who knows me, knows I'd like to tell stories.
There's this story of how we built this geophysics cluster. This was a while ago. We built up this geophysics cluster, and I am trying to remember how many nodes it was, but it was a good-sized system for the day and age. It was like six or seven racks or something to compute. It was a good size. We were integrating this, and we had Dell there, the NCSA, and all these people there and whatnot. We first validated using the standard image that Dell and NCSA wanted to leverage. It took us two days to get the software stacked up to get everything running. We validated it so we could get the system approved. Then we said, okay, let's go now, respend this using our supported OS. Everyone's like, well, we can't stay for too long, so we got to rush.
This will take a little time. It's like, but our plane leaves, we have to leave here in another hour. It's like, yeah, this will take less time. Within the hour, we not only had the whole system reinstalled and reprovisioned using our standard stack that we're using everywhere else in a completely supportive way, but we ran the scientific code and validated it. The funny thing was, we got a better performance. You put this whole thing together, and from that point on, we had a great relationship with Dell and the NCSA and everybody else watching this. But that's the experience we want everyone to be able to see and be able to leverage. Imagine being able to say, I want an OpenHPC cluster, or I want this cluster, or I already have this cluster running, but you know what, I'm going to go check out OpenHPC.
Imagine that you can do that within minutes and reprovision your fabric. Test out different node images and different operating systems for upgrades. We're enterprise Linux 8.6 here, but 9 just came out. We're going to try Rocky 9. You can import that image. You can add all your overlays and everything you want on top of that image. It's okay, well, I'm going to reboot these five nodes with it and start testing some jobs with those five nodes, and you can start testing and validating this in a very sane way. That's some of the benefits that Warewulf offers. The synergy between Warewulf and OpenHPC will make just the idea of TurnkeyHPC, which is the whole thread of what we've been talking about in these research computing round tables. I think it's going to make that whole Turnkey approach and Turnkey aspect so easy for people to deal with and get solid, reproducible systems that have everything you need on it that you can run with. Anyway, here's my big spiel. Jeremy, was I totally off on what we could do with OpenHPC? I know we've talked about this.
Jeremy Siadal:
We can do quite a bit. We have an open HBC running. That was when you talked about the speed at which Warewulf 4 can operate, something I never thought of doing. But we've tested it internally and have systems set up where we templated the image. We have a base node image that operates as a template. When a job is submitted, it takes that template, copies it, updates it, and sends it to the compute nodes. When that job's done, the nodes are cleaned off, and the entire node image itself is wiped out. But you can replicate and update that node, which happens in less than a minute.
Gary Jung:
I wanted to jump in with something about Greg's story. You know, I was there. The whole thing happened exactly as he said. The interesting thing that I just wanted to emphasize is the reason why we did it twice, once with a Dell stack and once with Warewulf is because a lot of people when they buy or procure clusters if you say, well, I want this hardware, I want it this way. But we want to run our software stack when we want this level of performance. If they know what your software stack is, the vendors will likely be reluctant to give you or do it and agree to an acceptance test using your stack because they need to become more familiar with it. In this case, we said, fine, why don't you qualify the hardware with your stack, and we can ensure that it works out fine. Then we're going to roll our stack on it, and then we can make sure that it works with ours. That was a reason for the back-to-back test, but the fact that you can provision both right next to each other makes it so that you could do your acceptance, and then you can go on from there.
Zane Hamilton:
That's great. We had a question before we threw it up, Jeremy. We've alluded to different container definitions. Is OpenHPC going to host some that you guys have as defined? This is the base that does this, and then you host those so people can download them.
Jeremy Siadal:
I don't have an answer to that. That's something we should look into. There is an OpenHPC container project, but that's a good question. I don't have an answer to that.
Operating Systems Provision on Warewulf [00:43:04]
Zane Hamilton:
Great, thank you. Now we get to Sylvie's question. The question is, what operating systems came from your provision with Warewulf? I know we've touched on several of them, but Jeremy, I will tell it back to you.
Jeremy Siadal:
I have provisioned CentOS, Rocky, Red Hat proper, RHEL, and OpenSUSE. You can provision Ubuntu with it. Question back, I wonder if anyone has experience provisioning older versions of CentOS and CentOS 7. It should work. You can go quite a ways back. Because of how Warewulf 4 is written, it should have broad support for most Linux Distros.
Zane Hamilton:
That's great. I know, Glen, you've been touching some, not typically what you would run across in a normal environment type operating systems lately.
Glen Otero:
If there's one thing I'd like to see the community do more of, it's post containers for Ubuntu, Debian, or whatever distro they're working on. A place where we could gather all those because the Warewulf containers are limited to CentOS 8.7 and Rocky 8. We want to get some more up there, but because adding the kernels to different distros, whether it's Ubuntu or Rocky, are slightly different, and getting the modules in there is slightly different. Just those things to smooth out those speed bumps for people would be helpful. I want to try tiny Linux or Pi, piCore, or whatever. All sorts of others, really different things to see if they would come up. Just for fun.
Greg Kurtzer:
We learned something interesting as part of this whole thing of trying to boot random containers up like Docker hub. Most base containers have been unable to boot out of single user mode. There's a core util single user and a bunch of system D stuff that has been turned off and optimized for container use. If you go and look inside the Warewulf source code, we got a directory right off the route called containers. Go into the Docker directory, and you'll see, look at the Rocky 8 one. You'll see how many things we had to change regarding the default or the extreme base container to get it to boot properly. There are a couple of interesting things there. The theory of pulling any container out of Docker Hub and running that container on bare metal is pretty neat. Again, there's so much you can do with that piece. Anyway, that was the point I wanted to bring up there.
Any Reason to Hold Off on Warewulf 4? [00:46:13]
Zane Hamilton:
That was great. Thank you. This is a good question. I know, Misha, you touched on this earlier, and one of the things you are working through and looking at right now is the state full in version 3 versus stateless and version 4. Other than already running version 3, are there any reasons to hold off on a Warewulf 4? Something like that, I'll let you go ahead and answer that, and then I'll throw it up to everybody else.
Misha Ahmadian:
A few days ago, I got a good answer from Jonathon that we don't need a state full anymore. I'm confident that stateless is going to work for us. As I said, there are a couple of things that you need to consider when you move from Warewulf 3 to 4. If you currently have a management node for your Warewulf 3, you cannot install Warewulf 4 simultaneously on the same node. It would help if you had a different node, and you have to ensure that you will not mess with the network because the way that Warewulf 4 is handling the DHCP is different from Warewulf 3. That's what I have learned so far. Now, we have this on-and-off option of DHCP when you want to provide a few notes for the test.
Once you make sure everything is fun, you have to move fully to Warewulf 4 to take care of the things for you. Meanwhile, you must ensure you can handle the DHCP on both server nodes for your Warewulf. They're hosting the Warewulf for you. A couple of other things will happen when you move to Warewulf 4. One of the things is that the public and private keys that Warewulf 3 generates for all the nodes will be changed. Did it used to be DSA or RSA 1024? It's going to move up to RSA in 2049, which is a big change and more secure. Your previous keys are going to fail to work. You have to ensure that you have a brand new key for the Warewulf 4, and that's it now. But there is one thing that you can do, Greg. You can correct me on that. If you currently have any images you shoot, make changes, and then create VNFS out of that, you can still import them into the Warewulf 4. It doesn't have to be exactly the container. You can bring your old images into the Warewulf 4, but again, you may want to move to the containers at some point anyway.
Greg Kurtzer:
With Warewulf 4, you can point it at the root directory or VNFS directory and import that into Warewulf 4. It doesn't have to come from a container. It could come from an actual trurot.
Misha Ahmadian:
But you have to ensure the kernel is installed on the sheet.
Greg Kurtzer:
Ah, so have to is a strong word. You should. You don't have to because Warewulf 4 supports the ability to do kernel overrides. This means that it can operate the same way that Warewulf 3 and previous versions did by overriding whatever kernel you may or may not have inside your container or VNFS. It'll override that and say, ah, you just imported a kernel into Warewulf 4. You configured that for a node. I'm just going to use that. It's very helpful for doing testing of different kernels, especially because you can import a kernel from a different container. Let's say I've got container A and container B import it into Warewulf. Container A has a newer kernel that I want to test. I can take the kernel out of container A, import that into Warewulf, and then apply it to container B. It's very, very, very flexible. There's a lot you can do with it.
Zane Hamilton:
Sorry, Jonathon, go ahead.
Jonathon Anderson:
I wanted to give one more point: whether there is any reason to hold off on Warewulf 4. From a functionalist perspective, I don't think so. I've had to go back for another reason and install Warewulf 3, and going back has just been difficult. The one place where Warewulf 4 has some deficiencies right now is in its documentation. There have been many new features, and a lot of new functionality added, even just between the different point releases of 4. It's been easy for the website documentation to get out of date. We're actively working on this, and I, individually, am working on it today and yesterday. With that, the Warewulf and development communities are good to get involved with. I have never touched GoLang before. Warewulf 4 is also a rewrite of the project in GoLang. But the code base as it exists is pretty approachable. If you're good at Googling what different programming semantics do to find what you're looking for in the code and offer some poll requests or help write some documentation if that's a way you can contribute.
The Warewulf Community [00:51:43]
Zane Hamilton:
That's great. Thank you. Sorry, I got a little sidetracked looking for questions. Jonathon, I know you said you've been playing with it quite a bit, and I've picked on you a lot today, but I know when we look at the community and people giving back to it. There's the question right there. How is that important? How should people go about it?
Jonathon Anderson:
There's that. There's Warewulf.org. You can get involved by contributing to the open source project, which is hosted on GitHub under the HPCng account. There's also a Slack team for the HPCng project where most of the community Warewulf conversation takes place.
Greg Kurtzer:
I was going to say the same sort of thing. The community is extremely friendly. It's very helpful. We talk about a lot of things, even more than just both. We talk about OpenHPC. We talk about how to build clusters and how to deal with clusters. The easiest way is to join Slack. Just participate in the conversations, join in, introduce yourself and go from there. The one I did want to mention, I wanted to get props out as well to SUSE. SUSE has just done such a remarkable job in terms of joining the community and being such an active group within the community. It's been super helpful. When we were talking about operating systems and whatnot and support, OpenSUSE and SLES are very well-supported inside Warewulf. It is not specific to a standard enterprise Linux, Red Hat Enterprise Linux type ecosystem. It is SUSE, and SLES is absolutely supported, and again, they've done great. Christian is one of the developers over there and is now one of the Warewulf committers like Jeremy, myself, and others.
Zane Hamilton:
One of the other things I want to go back to, I have, sorry, go ahead.
Jeremy Siadal:
I also want anyone interested in joining the OpenHPC project. OpenHPC.community also check out our GitHub page. The Wiki is fairly extensive, and you can join the project. If you have any OpenHPC open source components you'd like to see added, there's a formal process for making that request. We consistently add new components to the system or the distribution. For example, we have other projects like our summer internship project every summer. We would appreciate it if you are interested in joining and becoming a developer for openHPC.
Zane Hamilton:
That'd be great. And Jeremy, we'll add those links.
Jeremy Siadal:
Okay, thank you.
Stateless or Stateful Processing [00:54:53]
Zane Hamilton:
There they go—one of the things I wanted to go back to before we ran out of time here. We talked about stateless and stateful, which comes up a lot in my conversations. It's important to clarify stateful stateless, which doesn't mean dislist. Whomever, I'll let Jonathon or Greg because they're the two that we've been talking with the most on these two topics.
Jonathon Anderson:
First, the definition's what we mean by a stateful provisioning system. The one I'm most familiar with is a system called Foreman. I will usually automate a Red Hat kickstart, or I forget what Devin calls that, but something that automates the usual operating system installation process to lay an OS down on local disks. Like you would if you were installing it by hand, just automated. Warewulf is an image-based remote stateless provisioning system. You have that process, creating an image and then shipping it over the network at boot time and booting it. There are sometimes desires to have a state with the node, and this is where I like to differentiate the idea of that provisioning process being stateless in that the thing that was provisioned is stateless and happens every time you boot the node from the node itself is stateless.
One of the things we hear is that people want node logs to be retained on the node. If they're not centralizing them through other means, they may want them to be retained on the node. If a node crashes, it can bring that node back up and see its logs. They didn't get thrown away when the node rebooted. Nothing prevents you from having a partition on a local disc on the node. Part of what your stateless image does is mount that into the operating system so that logs get written to a local location. An even more straightforward use case in an HPC context might be having local storage resources available for high IOPS workloads that don't need to be shared between nodes.
You can mount some local storage onto a scratch path that users can write into that retains the state on the node, even though the provisioning process is stateless. But then you can even consider a stateless node with disks. One of the concerns with stateless provisioning is that the operating system that comes across the wire is entirely held in memory. A concern is eating into your node's compute kit capacity because you need that memory for compute workloads. If you have disks in your node or an SSD, you can create a swap partition and mount that swap partition. As that memory is required, it allows the operating system to be flushed to swap, which is an inverted process from what you would normally do, but it ends up in the same state that bits of your operating system are in storage. Bits are in memory, depending on when it was last used. All of the memory is returned for active use to the running workload. There's a lot of flexibility, and it doesn't have to look like putting an operating system down on the disk.
Zane Hamilton:
Thank you for that, Jonathon. We are actually up on time. I will give everybody an option to give their closing thoughts or anything you want to add. Gary, I'll start with you again.
Gary Jung:
Really looking forward to Warewulf 4. We're already making plans to do that.
Zane Hamilton:
Great. Thank you for joining. Misha.
Misha Ahmadian:
We will have a good result by the end of this year by showing progress in moving from Warewulf 3 to Warewulf 4. We're excited to look forward to that.
Zane Hamilton:
That's great. Thank you, Misha. Glen.
Glen Otero:
Just semantics, stateless does not equal disless, and images do not equal containers.
Zane Hamilton:
That's very true. Thank you. Jeremy.
Jeremy Siadal:
With the work we put into Warewulf 4, I'm looking forward to its inclusion in OpenHPC, which should happen fairly soon. If you're attending Supercomputing OpenHPC, we'll be at the OpenHPC booth. Warewulf also has its location, but look for us anyway.
Zane Hamilton:
That's great. Thank you, Jeremy. Jonathon?
Jonathon Anderson:
I talked a lot about Warewulf today, but I'm also excited about OpenHPC and being more involved in that community, which will only be bolstered by Warewulf 4 being part of it. I've gained a lot of value from using the hard work that's come out of that project and not having to replicate it on my own. Thanks to everyone in that community; if that one interests you, take a look at it. They're doing good work.
Zane Hamilton:
Thank you. Greg.
Greg Kurtzer:
I don't have much to say that hasn't already been said. This was a fantastic conversation. Thank everybody for being part of this. Thank you.
Zane Hamilton:
It's great. And Forrest, sorry we made you be quiet in the corner. I'm sorry.
Forrest Burt:
No worries, Zane. I'm excited to see Warewulf 4 come to wider adoption in the market. I'm excited to see how it'll improve the capabilities and capacity of OpenHPC in the end and how it ends up improving the capabilities of the next generation of high performance computing that we're all working on building. Very excited to see how it works.
Zane Hamilton:
A great way to end. We appreciate you guys joining. Thanks to the panel. Misha, thank you. Glad you were able to join for the first time. Looking forward to having you back again. Thank you for the interaction today. Thank you for the questions in the chat. Feel free to hit us up if you have any more questions or if there's anything you would like help with. Go ahead, like, and subscribe, and we will see you again next week. Thank you.