Warewulf with Gregory Kurtzer: History, Containers and future HPC uses
Rose Stein interviews Warewulf creator Gregory Kurtzer about the project CIQ supports. Kurtzer explains that Warewulf is a software implementation of the Beowulf architecture HPC has used for almost 30 years, which is why the name is spelled the way it is. He created it in 2001 after being asked to build several HPC systems while already stretched thin, and the goal was to manage hundreds or thousands of nearly identical compute nodes from a single image.
The conversation covers Warewulf's stateless design, where a node loads the entire image into memory at boot and forgets everything at power off, with disk persistence coming to Warewulf 4. Stein and Kurtzer discuss how Warewulf 4 leverages containers: you build node images with Docker, Apptainer or other tools, import them, and provision them out. CIQ adds support, customizations, training, help migrating from other provisioners, and ready-made turnkey node images, with more node types such as I/O and visualization nodes planned.
Kurtzer notes that as other HPC provisioning toolkits fade, Warewulf is becoming the leading option, driving new interest and uptake. It is a concise introduction for anyone new to Warewulf or evaluating provisioning for an HPC environment.
Key takeaways
- Warewulf is a software implementation of the Beowulf HPC architecture, which explains its unusual spelling.
- Kurtzer created Warewulf in 2001 to manage hundreds or thousands of nearly identical compute nodes from a single image.
- Nodes load the whole image into memory at boot and forget it at power off; disk persistence is coming to Warewulf 4.
- Warewulf 4 node images are built with container tools like Docker or Apptainer, then imported and provisioned to nodes.
- CIQ provides support, customizations, training, migration help from other provisioners, and ready-made turnkey node images.
- With other HPC provisioning toolkits going away, Kurtzer sees Warewulf becoming the leading provisioning system for HPC.
Questions this video answers
Why is Warewulf spelled with an 'a'?
Because it is a software implementation of the Beowulf architecture: ware from software and wulf from Beowulf. Kurtzer says the Beowulf model has been used in HPC for almost 30 years and that Warewulf was designed to answer the practical question of how you actually build one.
How does Warewulf use containers?
Warewulf 4 relies on a node image that is pushed out to all compute nodes, and it leverages containers to build and maintain that image. You can use Docker, Apptainer or another tool of your choice, import the resulting image into Warewulf, and provision it to your nodes. CIQ also offers ready-made turnkey node images in a supported way.
What support does CIQ provide for Warewulf?
CIQ helps maintain the open source project and offers support, customizations and training around it. It provides turnkey node images that solve common problems, helps organizations migrate from other provisioners, and plans additional node image types such as I/O, compute and visualization nodes over time.
About this video
In this video Gregory Kurtzer speaks with Rose Stein about the creation of Warewulf, Warewulf history and some of the many uses for Warewulf. We discuss the Warewulf open source community around the product and future success with containers in the product.
This video is part of the Warewulf Pro playlist. Browse every CIQ video by product and topic.
Transcript
hi Greg thanks for being with us today sure thing so I have a couple of questions for you first off thank you for creating a product that we support called Warewulf many years ago uh I've find a lot of questions on Where that name came from so first off tell me like what is Warewulf and where did this name come from the name actually comes from kind of what it is it is a software implementation of the beow wolf the B wolf is an HBC architecture that we've been using for almost 30 years now it's been incredibly impactful uh but how do you build a
Bay Wolf uh that's what Warewulf was was designed to do that's awesome okay so we got we for the software and wolf for the B wolf model yep and this is why I cannot spell the actual monster wear wolf to this day because it is spelled w r wlf right that's exactly and I absolutely love it and we actually have the coolest logo in all of life our Warewulf logo so hopefully we'll be able to share that with you real quick okay so the next thing Greg if you would be so kind as to tell us why you developed Warewulf and what it is so
Warewulf originally came because I was tasked with building a bunch of HPC systems and well I was already kind of busy so didn't have a lot of time to to go and build these other systems I wanted to figure out is there a way that we can more optimally manage these HPC systems cuz an HPC system could have hundreds or maybe thousands of compute nodes in it when each compute node is a standalone serter for all practical purposes but every one of them is pretty much identical to all of the others so how do we take that and scale that uh that was the whole
point of Warewulf and uh Warewulf now allows you to take a single image or or a group of images and assign and out to hundreds or thousands of compute resources and it always was built around the idea of uh volatile volatility in the operating system or non-persistence which means that soon as you turn it on it gets the whole image and you turn it off it forgets who it was it has no context because everything is written to memory nothing's written to the hard drive now we've extended that to the point where you can actually write things to the hard drive uh were 4 is
View full transcriptHide full transcript
going to be getting those features very very soon Greg you basically created Warewulf back in 200 and one and now that we have moved forward all of these years it's 2024 Warewulf has its own open-source thriving community that is absolutely amazing and ciq as our company your company that that we work for um is helping maintain this project and we providing support uh customizations uh trainings and everything and anything else that people need around Warewulf one of the super interesting things about Warewulf is that it relies on this this this notion of a image that you're going to blast out all all of your compute
notes uh where Well for now leverages containers for this so you can use Docker you can use Apptainer whatever you want to maintain and build your container images and from that you can now import that into Warewulf and then provision that out to your compute notes so ciq is adding a lot of value on top of Warewulf in in a variety of different ways as you said custom features uh support uh but also the ability to provide uh readymade TurnKey node images that gives you the ability to immediately just have solutions to a variety of different problems in a supported way yeah but that's really
awesome is amazing to me that you know uh a problem can be solved so many years ago and then iteration upon iteration upon iteration and support from the community Warewulf is an amazing thriving Community with a lot of different companies that are are are providing resources and and valuable uh engineering hours to continue to move that project forward and here we are now in 2024 looking into this you know vast expanse of what is possible in the tech world and Warewulf is at The Cutting Edge so another thing that we do is we provide um an opportunity for customers to come in and get some
help vibrating so if they're on a different provisioner or they're doing something something else and they're like hey you know what Warewulf solves a lot of my problems that I I would like to utilize this then we can help with that as well absolutely uh as you alluded to Warewulf beenin around for a really long time partially because I'm just really stubborn and I don't give up on things and well as a result Warewulf is very longstanding in uh uh solution within the high performance Computing ecosystem and at this point we've seen a lot of shift in this in this part of the the area
of the ecosystem where a lot of the other provisioning systems toolkits and and solutions are kind of going away which means Warewulf now is becoming uh the lead uh if not the only uh real provisioning system out there for high performance Computing and we're seeing a huge amount of interest and uptake as a result and as always ciq is here to help and uh we believe in open source we believe in making sure the community has everything that the community needs to be successful but if you need a hand if you need any additional capabilities uh we're here to help so reach out to us
absolutely we'd love to talk with you
Built for scale. Chosen by the world’s best.
2.75M+
Rocky Linux instances
Being used world wide
90%
Of fortune 100 companies
Use CIQ supported technologies
250k
Avg. monthly downloads
Rocky Linux
9
Enterprise products
Spanning the kernel to the orchestrator
Have questions about your infrastructure?
Talk to a CIQ engineer about Rocky Linux, HPC, and AI infrastructure.