Warewulf: HPC Cluster Management and Provisioning Platform

also have to do things like are there any custom configurations on particular nodes does that include network configurations that include service configurations does that include user configurations and some of these can be provisioned or managed after the node has been provisioned but some of them actually have to be before the system uh runs s been init and spin in it is the parent of all processes with pid number one so in some cases we actually have to do customizations to each node that's being provisioned before we even start pid number one so there's some tricks that we have to be thinking about in terms of

provisioning but you're exactly right even though i think of provisioning as being that bare metal kind of base operating system that base operating system includes a number of different facets uh such that the cluster can work properly and again that includes networks service configuration user and user configurations and whatnot great question so you actually use an interesting word there you're saying configuration so whenever whenever you're talking about configuration management and provisioning are are they different are they the same thing are they paired together oh that's come up a bunch of times in in high performance computing especially as we're talking about and thinking about provisioning

of the operating system excuse me so werewolf and other cluster provisioning systems will typically do some basic rudimentary configuration management but that rudimentary configuration management as i kind of alluded to a moment ago occurs in many cases before the operating system has booted so you're basically making these configurations like like your network configuration for example uh you want to have your network configuration um in place before you call s been in it and then aspen and it goes through systemd and and starts all of your services starts all of your your devices you need to have that network configuration already there same thing with things

like fstab right you need to have your your file systems already configured you need to have you know any any you know service configurations configured before systemd runs those services so there's a side of this configuration management which has to be very kind of lightweight and and engraved into that provisioning system such that you're getting all of the nodes are basically getting their appropriate configurations whether that be something as simple as just a network configuration and a hostname configuration or that be something as complicated as a whole different operating system you have to be able to manage that at the provisioning layer and werewolf does

all of this for you but it does not replace a more typical and much more fully fully fledged configuration management solution um so if you need something that's going to do much more complicated configuration management tasks that's something that we would say well provision your base system with werewolf do your base configuration with werewolf but once that node comes up run your configuration management services so then you can actually then take over with your cm uh platform whatever you're choosing excellent thank you so i think one of the other things that i hear quite often the conversations i have is stateful or stateless so when

we start talking about werewolf how does how does that come into it come into play so this this kind of even goes down to the foundation of of werewolf and kind of some of its basic tenets of of what differentiates and has differentiated werewolf all the way since the beginning um so if i go back in time um let's see it was created in 2000 2001 um yeah i'm i'm way that old uh basically at that point um high performance computing systems uh were traditionally as we were starting to make the bridge over into linux and the beowulf was was all the rage at the

point at that point we were basically trying to figure out how do we manage the operating systems on again hundreds or thousands of compute nodes um and to kind of again put this into perspective many organizations say well you know for every system administrator you know maybe it's you know if it's a high you know high load uh you know high complicated you know systems a system administrator may be able to do 50 maybe 100 um separate servers if they're really good or if the systems are fairly similar you might be able to get you know hundreds but an hpc system could be thousands and

we're not going to have you know we need to come up with better ways of scaling uh scaling the system administrators time scaling how we're doing this so there there came to be toolkits on how to manage this and there's there's various different ways of doing this so um one that was very big at the time was called oscar oscar used a system um or provisioning uh tool called system imager and um system imager basically allowed you to basically make a a template of your operating system and then it would blast that template out to all the nodes um when you boot it and would

write it out to the hard drives and from that point on all of your nodes are kind of provisioned and then you you manage them as a cluster of workstations in a matter of speaking um there's other tools that leverage things like kickstart which is anacondas of the or the red hat originally the red hat installer uh mechanism for automating deployments of of resources so uh there's tools so the rox cluster system not to be confused with rocky similar name but rox uses um basically a you know it's a turnkey system that basically uses kickstart uh to basically provision out all the nodes and um

werewolf though approach this from a completely different perspective instead of facilitating the installation of a massive amount of compute nodes let's completely remove the necessity to install any compute nodes let's just boot directly in to a runtime operating system and all of a sudden that kind of shifts the paradigm a little bit right so every time a node boots our model was let's just put it let's just dump a real you know a runtime operating system onto that we're never doing an install and then it runs directly from being booted it runs kind of almost like a live os or a live cd and um

every single time so so when werewolf boots a node every single time that node boots it gets a you know the the operating system byte for byte as it was intended by that node and all of the nodes above and all of the nodes below and it makes it very scalable it makes it very easy to maintain and so when we talk about stateless in high performance computing provisioning we're basically saying that the operating system is not written in a persistent way to the storage so this means that you could reboot that node at any given point and as you reboot that node it's always

getting a fresh version of the operating system now stateless is usually confused with diskless and a diskless node i mean obviously you'd have to run that stateless but if you have a disk full node you can choose if you want to run that stateless or or stateful you can also choose do you want to write that operating system to um memory and do you want to run it directly out of system memory or do you want to run that operating system off of that spinning disk there's a lot of different kind of options in terms of how you can choose to provision that resource uh

in in you know that operating system to that node werewolf the thing that's really differentiated werewolf over the years is the fact that it really uh focused on stateless so there were other systems out there that were focusing on stateful and i kind of i mentioned rocks and oscar but werewolf really focused on stateless operating system management and again there's there's a lot of benefits in doing that when you're when you're dealing with a large system at scale a story that i like to tell is you know at some point we had a compute node in a very large cluster we had one compute node

that would always seg fault on one application and as we would run this you know this this application across all the nodes all the nodes ran it perfectly except for one node and it was but it was a seg fault and um you know typically when you see a seg fault you usually are going to assume that there's something kind of weird going on with with the the software stack or uh you know something something's going on but it's usually software related but because we were using werewolf and i knew with fact that the nodes above and the nodes below are running perfectly but this

one node is having a problem we knew it was a hardware issue so i packed up this node and i sent it over to the hardware vendor and they said what's wrong with it and i said i don't know it's it's seg-faulting the software and they're like yeah that's the hardware you sure yeah i'm pretty sure it's the hardware and um you know they they look through it and they didn't see anything wrong with it they wanted to send it back i'm like nope nope there's something wrong with it finally they get around the checking the power supply under load and it turned out that

the power supply under load was not providing enough juice for the motherboard in the system and that's where um the the i don't know how i'm not a hardware guy but that's how it somehow was interpreted by the kernel you know something ended up coming up a little bit short on power and it was interpreted as a seg fault not a hardware failure and um they put a new power supply in the system system ran perfectly from that point on so it kind of changed the paradigm in terms of how do we maintain and how do we think about maintaining these large these large systems

and especially when you consider mean time between failure is actually not that bad unless you're looking at you know a thousand plus different different nodes then your mean time between failures is actually it hits quite often so how do you manage things like this how do you mitigate that again with a stateless operating system provisioning system you can basically just take out that node put a new node in change the configuration to say okay the hardware address of this system is now changed just turn it on and it has now completely assumed all of the responsibilities and roles of the previous node that was there

very easy to deal with so stateless versus stateful so this sounds just using something that a lot of people are talking about today is a very popular topic this sounds a lot like containerization so how is this or how is it or would it be related anyway to containerization so that's a great question so um when werewolf was first uh conceived and very similar to oscar as i mentioned before using system imager it had the notion of a golden image this this base template image of what you're using for the template to provisioning out all of your compute resources and all of your compute nodes

excuse me so in werewolf we did it a little bit different uh we basically had a directory and it was a root directory if there's any you know engineers or admins in the room uh it was at your root directory which basically allowed you to have like another root file system in an in a directory location and werewolf leveraged that as a template uh to make the operating system make the bootable operating system for all of the compute nodes now we didn't know it at the time but what we were actually using was basically a container and i know it to root and a container

is very similar already but that's basically what we were using we called it something different we called it the virtual node file system and we manage this virtual node file system using tools that were well quite honestly very similar to containers today so one of the big switches that we just made with werewolf v4 is that we have basically just embraced the container ecosystem directly so instead of us kind of having a separate version of this that we were basically using as a vnfs we basically said well it's not a big jump to just say well let's just start using containers this has the effect

that all of the existing tooling and everything that people are doing today with containers to build containers to secure containers to validate containers well we can just leverage that directly in werewolf now so that was one of the biggest change that we made with with werewolf v4 is it now is basically an oci compliant provisioner and this means that we can basically you can use any of your your docker recipes your docker files you can use singularity apptaner you can create your container you just point werewolf at wherever that container exists whether it's in an oci registry or whether that's on your local file system

or in the local daemon running and you basically say i just want to import that into werewolf once you've imported that into werewolf you can say okay i want these thousand nodes or however many nodes you have to boot on this container and what werewolf does is it actually not only blasts you know the the operating system out it's actually blasting a container out to all of the compute nodes so you're basically running this container on bare metal and as a result of it again we we get to leverage all of the existing tools um and uh and capabilities that people probably already have today

with regards to their containers and how they're managing containers and so on and so forth so all that can be basically directly directly imported into werewolf and then you can provision out those containers it's awesome so i think we've talked about kind of what it is today i think a lot of people are interested to know how did i mean 2002 it's 20 years ago how did you get started and what brought werewolf to life what what what story did you have or what were you going through at the time that made you decide you had to go do something else or do this obviously

it's solving a problem so how did it become so um my background is actually in in biochemistry and i made the jump the leap into linux and open source um as i was trying to solve problems computational problems for biochemistry and i got so excited over um being able to do this for you know in open source and i was so enamored with with what we've been able to do in linux and open source and whatnot that uh it pivoted my whole career on that as a result of that i ended up landing you know going through some startups but i landed at berkeley lab

at the department of energy and i was joined appointment to to uc uh berkeley um and i did projects for ucop and uh and whatnot but when when i first started this i went from not only scientific computing but we started you know the advent of you know 2000 ish was kind of you know hpc you know cluster computing linux cluster computing was starting to get become a really big deal and so one of my first tasks when i joined berkeley lab was to basically start you know building up some big hpc clusters and um at the time as soon as i walked in the

front door i was basically tasked with um an existing you know existing load of projects and i was completely taxed out like i was at 100 utilization if not more and uh they they basically said okay now we're going to basically add you know some pretty big clusters to the mix and we need you to maintain this and i started looking at well okay there has to be some way to help me with this because i am not going to you know physically install you know 100 computers and uh basically exactly exactly i mentioned this was a government position too right um and uh so

i i was basically you know which which means we're already overworked uh by the way so this basically means that i'm trying to figure out what is the best way to manage these systems to install them to to build them and then maintain them over time and so i was looking at different tool kits and what are people using and and just honestly none of them really spoke to me in terms of the way i wanted to manage this so uh i developed a prototype very similar to it seems like many of the things that i've developed over my career i develop a prototype which

was close but at the same time completely wrong at exactly the same time um so singularity started the same way singularity version one those of you who followed singularity know that um it was it started off basically being almost like ubuntu snaps and then ended up morphing into what you know today as being singularity and apptaner in this particular case i worked at a previous organization called linuxcare and we we were very well known for creating these little bootable business cards that you can put into a system there's a little rescue cd and i left there thinking all of the world's problems could be solved

with a little bootable business card and uh if everybody just carried these around you can you can have everything you ever wanted uh obviously computing wise maybe more i don't know uh so uh i started werewolf actually with werewolf actually created iso images for all of the compute nodes so they were stateless it's just i wasn't using the network to transmit them and uh i went to linux world and i think it was 2000 and uh i i went and i talked to people about this and i presented this and the feedback that i got is wow we really like the management paradigm of what

you're doing but good god what are you thinking with these isos that is just such a horrible idea and um and they said you know there's a booth over there called ether boot which was the predecessor to pixie in in many regards it says go talk to the ether boot guys and like ether boot oh cool so i went over and talked to them i learned everything i could learn about ether boot in a few days and it just took me like four days to port werewolf to now you go use ether boot and ether boot was basically a little firmware that you can either

flash to your your network card um or you can boot on a floppy that just had this little piece of firmware on there which basically set your network card as a bootable device by basically loading an option rom into the bios to do that and and then the network card would actually do the pixie booting well it wasn't pixie at the time but it would do something like pixie and that was then the real that's where werewolf really then took off is once i got that built up and integrated and then it seemed like organizations all over the place uh started adopting werewolf started going

to super computing and everybody was talking about werewolf uh for quite some time that's awesome i'm not going to say that i've ever used the linux cd because i have rescue rescued some stuff so i know that yeah you've we've talked about werewolf i know a lot of people are here because they want to see where we'll so i know you've prepared something for us let's let's see what you got yeah absolutely all right so i put together some some slides um real quick um you know just so you can kind of see um we're doing more than just werewolf obviously as a company and

i'm not going to go into this i don't want to sound sales or marketing wise but if you're interested in any of these things definitely please let us know and werewolf is a big thing that we're going to be talking about so when when we talk about werewolf we usually describe it as um you know even the name comes from you know it's a software implementation of the bao wolf and that's why werewolf is spelt totally wonky as a matter of fact if you ever ask me to spell werewolf as the big you know monster human dog thing um i can't it actually it doesn't

feel right to me it's it's there's something wrong with it um so i werewolf this is this is how i've it's just engraved now at this point um but basically werewolf is a software implementation of the beowulf and the beowulf is a specification for how to build a cluster using commodity parts using linux using open source and it's an architecture and this architecture is roughly drafted as you see below right you got your user workstations and users ssh across the network and they basically get into a a control system that control system may have a file system attached to it directly and it's sharing it

out or this could be a some sort of network attached storage and it's sharing it out but then you have a switch you've got a private network where all of your compute nodes reside and these compute nodes may actually even be on another network that may be a high performance data network something like infiniband and this is basically kind of like a you know very basic beowulf first and early versions of werewolf were designed to solve this but as we got more and more complicated and we were looking for more features in what these systems look like we started adding to it and we started

adding to it in such ways that again users would basically be sshinging in we'd become basically directing them through a reverse nat in our firewall to land on our interactive nodes and the compute resources are in typically groups now and they're in groups because you may have different vintages you may have different people that own different sections of them and whatnot but you have these different groups of compute nodes you may have separate networks kind of dedicated to each one of these groups and so on and so forth um werewolf and um or provisioning and scheduling are usually done in some sort of redundant um

uh resource and again you've got your shared storage and typically by the way you just you usually have like a a home directory storage and some form of parallel uh scratch storage that's very efficient now werewolf now needs to be looking at this not only as just a completely flat system you know like you may see here but multiple flat systems and werewolf now is is was built around the idea of how do you maintain you know not only these these simple or kind of flat systems but how do you maintain systems that have additional depth and additional complexity and when we built these systems

at berkeley and berkeley lab as well as uc berkeley we basically have there's no state in the entire system except for the werewolf master so even the interactive notes ran stateless the entire architecture was stateless and this meant that every any system could basically blow up and we can just replace it and we didn't lose anything there was no context that was ever lost so this was how we were thinking about building systems and this is kind of how werewolf is envisioned today and you're going to see in just a moment how you can have node configurations and then node profile configurations that are completely

cascading and it really facilitates anything from again a more simple kind of turnkey model going all the way up to something that may be a lot more have a lot more need for flexibility so if we take a look at so i've already set up werewolf and i've basically built a node on this and it's a virtual node so we can we can hack around at this a little bit and take a look uh i'm right now and i installed it to user local just because i didn't install it via package and user local is what i do so it's in user local and inside

of etsy werewolf you'll see you got a few files and if i take a look at the nodes.conf you'll see two major sections now this is yaml uh and we've got two major sections in it we got node profiles and we've got nodes now nodes are pretty easy to imagine right this is a single node right this is the configuration for a single node we've got some information in here that is maybe specific for this node and if you have hundreds of nodes you may want to consolidate some of that information and configurations to profiles now every node by default gets a profile called well

default and that means you can actually add your configurations to the default profile so for example in this in this no default profile we have a container name which is rocky8 and we have a kernel version and i'll show you basically how i got to these how i got these into the system here in just a moment but basically what i end up doing is because node 0 is part of this default profile and this profile has these values node 0 has those values and i can supersede those i can have multiple profiles that supersede other configurations and it's extraordinarily flexible you can see in

terms of network devices we have a default network and what you can see is basically i have the netmask being kind of globally defined but i have the ip address and the mac address being defined specifically on just this one node so obviously i don't want to put my ip address or my hardware address my mac address in a profile because then all nodes would have the same ip address and same network configuration so that i specifically put into this node profile so this kind of encapsulates what an entire system architecture would look like and it gives us the ability at a very high level

to actually do very kind of cool things now i'm showing you this nodes.conf and you can edit and work with thisnodes.com directly if you want or you can use configuration management but most people use the tool ww control now ww control is basically the primary interface for werewolf so you can do ww control for example node list and i want to see all attributes so i'm going to use the minus a and you can see now if we had many nodes configured this would basically be a giant list but you can see in node 0 we have all of these different fields or keys that

are associated with this we can see all the different values for them and we can see where did we get that configuration if it says default under profile that means we basically got it from this default profile so for example container name and kernel version came from the default profile and that's exactly what you see here this is listed here because there are some configurations that may have many node profiles and in some cases it's best to see you know where is a particular value coming from so if we may have let's say a default profile but on some nodes i want to test a

new kernel i can have another profile for a new testing kernel and then add that profile right here and we'll basically be able to have you know a certain group of nodes testing that that newer kernel it's extraordinarily featureful and i don't want to get too bogged down on this but i really wanted to explain this because it is extraordinarily capable with regards to configuration of node so as we're looking at building systems that may be more complicated like this it makes it very easy to group these nodes together with a profile and then define all the attributes for this profile werewolf gives you that

functionality very very easily okay greg real quick i got a question for you whenever you i mean you said you just installed this what is required to install the the werewolf controller ah how complicated is that good question and thank you for bringing me back to this i meant to show this but um i totally forgot as i just wanted to jump into the tech um if you go to werewolf.org um you can click on docs and from docs you can click on quick starts and we have a an enterprise 8 enterprise linux 8 quick start and as you go through this i'll quickly run

through this you can kind of see what how this operates the first section is just installing it from git um you can install it from packages from the releases etc but this is just from git and it's basically you just you install the dependencies you do a git clone you do a make all and then sudo make install super easy install excuse me once you've done that you basically just run these commands to configure the firewall appropriately this is the werewolf.com file you're going to edit this and just make the appropriate changes probably going to need to edit your ip address maybe your netmask and

then you probably also want to take a look at your range start and range end for the most part everything else is going to be you know pretty standard um then what we're going to do is you can create some if you want system d to run this um you basically just create the needed users and whatnot and um and then basically just start werewolf d from systemd and at this point we now have werewolf installed now what we're going to do is we're going to run a ww control configure all what this will do is this will basically go through all of the system

services that werewolf needs for it to boot so for example it's going to need things like pixie it's going to need thing well pixie excuse me has certain requirements like dhcp and tftp it's going to need nfs to be running if you want nfs to be running via this configuration file it will set up dhcp as i mentioned make sure tftp is running properly and drop the appropriate files in there so this will just kind of configure your underlying system and this has been tested and works on enterprise linux variants on soos variants um as well as debian and ubuntu from here now we're going

to start basically working with werewolf directly i showed you that i had a container called rocky8 this was how i got it i just did ww control container import and i pointed it at a docker registry you can reach this from anywhere this is public on docker hub and you can then import your rocky 8 image we're going to name it rocky8 and we're going to set it as default what's setting it as default does is it basically adds it to the default profile as you see right there then what we're going to do is do the same thing with the kernel we're going to

ww control kernel import and we're going to give it our current version name of the kernel that we're running and we're going to set that as default what this is going to do is it's going to take a snapshot from your host you can get the kernel from a variety of other locations if you wish just look at the help output for kernel import but what this will do is its simplest form is take the kernel running from your control server so if you're running gpus if you're running infiniband make sure those drivers are currently installed on your host kernel or whatever kernel you're importing

from and werewolf will automatically pull all of those kernel drivers and support for that kernel and then import that into werewolf and again the set default then puts it right here now once we've done that oh and then if you didn't use the set default this is how you would set your profile to basically be exactly as i just showed ww control profile set we're basically going to use set the default profile this just basically means assume yes set on the default profile we're going to set the kernel and we're going to set the container the next thing we're going to do is we're going

to set on the profile again we're going to set a basic network configuration and again this is exactly what is what has already been done as you can see here i didn't set the gateway though i guess i was slacking on that slack has a whole new meaning now doesn't it i can't use that word anymore okay easy [Laughter] and um lastly we can now add a node and you can do www control node add give it a give it a node name and um then we're basically going to specify some network information and we're going to make this node discoverable this means that when

the node boots it will automatically be added to the werewolf database and it'll be automatically configured so it will automatically put in this mac address onto whatever network device you have set as the default so it will automatically do that the next thing to talk about a little bit about is werewolf overlays what what we've just done so far is we've taken a container out of docker hub and we're going to blast this out to any number of of of hosts um that actually would work perfect if every host was identical but we do have some some node-specific things everything from network to um to

service configuration user configuration as we were talking about earlier so some small amounts of data and configurations may need to be changed on a per node basis and that is done via overlays and i'll show some of these overlays features here in just a moment but overlays is basically a way of adding information to a to a provisioning system using uh text templates and i can show that here in just a moment all right so that's basically what i did to get this system up to speed and that's where we kind of ended up so i'm going to start off by showing overlay so if

i do a ww control overlay you'll basically see all of the sub commands for for overlay if i just type in ww control you'll see all the primary commands so we're going to do ww control overlay from here um let's do a list let's see what exists on this system and you'll see i've got two overlays one is called generic and one is ww init to show how this relates to the node i'm to do a node list minus a again and you're going to see we've got two overlays here system overlay and a runtime overlay the system overlay we're calling on is ww inet

and this and excuse me yeah and the runtime overlay is generic these are in parentheses because they are not a configured attribute in the node configuration which is up here so that means they are defaults if you set them in the node configuration the parentheses um will go away and it will look like everything else and it'll be whatever you set it to but if they're not set those are the defaults so the wwe and net and the generic overlay um are can be found here now if you want to see what's inside of them you'll see for example the generic overlay has seven files and the wwe knit overlay has 46.

so let's do this and we're going to do a list this shows us all of the different files their permissions and ownership in the wwe init overlay and you can see all the different files that are basically going to be provisioned with this ww init overlay notice some of them have the ending or the suffix of dot ww this indicates that this is a template file and just like auto conf with the dot in n as soon as this gets resolved for a node the dot w w will disappear and the contents of this file will automatically get replaced with whatever templates is in there

so i'm going to show one which is this one so if i do ww control overlay edit so the first thing i need to do is tell it what overlay i'm going to be editing a file in and do that and when i do that you'll see this looks pretty much like a regular if config file except for it's got these template macros in here and these template macros get replaced for every node that this is getting provisioned for so this gives me the ability with a single template or with a single overlay i can actually have any number of nodes booting with unique configurations

for each one of them and again the the text templating language is actually very featurable so if there's anything that you want to include anything you know conditionals loops ranges it supports all of that so it's very easy to customize this um you know with very large granularity like there's a lot you can do so there's a few other pieces in here i can demonstrate and i'll show you this one real quick so ww control container because we talked a little bit about containers so ww control container and we're going to do a list and you'll see i've got one container imported called rocky8 it's

it's built and it's ready to go and it's configured on one node so we could do something like this container shell rocky 8 and you can see i'm now sitting inside of that environment all right so if i do cat etsy os release i'm actually sitting inside of this operating system this this this vnfs or this container and i imported that container just out of docker hub if i exit this it will tell me it's going to rebuild the container automatically but it skipped it because it's current uh it quickly did some time stamp comparisons and realized it doesn't need to rebuild anything if you

want to install something into this you can do something like this i'm just going to demonstrate it with the etsy but the host etsy directory i'm going to put into the container's mount directory and so if i do this you'll see i've got all the files there so if you want to install something like the mofed user space component you can put that you can get that into the container by using a bind and then doing an install just as you would with any other container or you can have that built um via whatever ci pipeline you're using to manage your to manage all of

your containers and building your containers so i've kind of walked through the general interface of ww control i don't think there's anything else to really talk about in terms of specifics that i haven't shared already uh the the goal of werewolf is to be very simple it's to be lightweight it's to be simple it's to be easy to use and very flexible so um the commands that you'll see here they should be fairly self-documenting right the configure command is basically just to manage the system services as we did at the very beginning container we walked through kernel i demonstrated note i can show that a

little bit more detail we can add nodes we can use ipmi and at some point we're actually working on some redfish integration so you can get a console on that you can look at sensors you can set status and and whatnot and when you do for example node status you're going to see we have this node but it hasn't booted yet so there's been no last scene so when we boot the node we're going to actually watch this and see what happens and see we also have profile which is the interface for managing profiles different profiles we've got power this is basically for power management

when you do things like you want to reboot the entire cluster it will automatically manage a parallelized fan out with an appropriate number of concurrent tasks so it does not completely overload your network um you know little things like waiting you know half a second to boot each node as you're going through that loop actually has a profound effect in terms of booting efficiently little things like that also dhcp is very not scalable and then server is basically the werewolf daemon that is running actually comes from this binary but you can you can definitely interact with it here but it since we're running this through

systemd it's better just to to let this be and let systemd kind of manage this and you can see so system control um and you can see it's in there and it is properly running so what we're going to do now is we're going to actually test boot a node and we're going to add the w option to the which is to watch this update and if i come over here i just so happen to have a virtual machine that we are going to boot and this is going to this has been configured to pixie boot by the way doing pixie booting on vmware fusion

as you can see that i'm doing has gotten incredibly difficult because now everything is using max dhcp services and whatnot so um if anybody wants to know how to do that and they're get they get bogged down with dhcp issues send me a message um i wrote a quick script that just blocks mac's ability to do this um so when you basically boot this up you'll see we're doing an efi uefi boot and it's basically going to start with pixi and you got to watch fast because it it moves pretty quick um but you can see now we're basically sending it the container which is

happening down here at the same time next we're going to send it the different overlays here's the k mods overlay system overlay runtime overlay once it's got that it now is managing a little bit on the back end now it's doing some basically booting through werewolf and knit and we're now done it was super fast as you can see there was no operating system installed on this node whatsoever everything just came over the network and when you're actually booting on real hardware it's actually faster so granted i'm using a fairly small and basic container so if you have a bigger container will take a little

bit longer but it is a very fast provision um and again it's designed to be very scalable uh yeah for lots of nodes booting parallel and um the default update and i didn't talk about this before so i'm just going to mention it real quick there's multiple overlays as you saw and i can even show you the different overlays so if we look at we're going to kind of get into the weeds a little bit right here but if you look at the default ipixi script you're going to see that we've got four pieces that we're pulling into in a nit rd and what you're

going to see is we got our base container we've got our kernel modules we've got our system overlay and we've got our runtime overlay what we're actually doing is is overlaying all of these overlays on top of the container and what that gives us the ability to do is to start with a base and then add in the appropriate necessary bits that we need for this node to operate properly the system overlay is only provisioned once at boot at provision time the runtime overlay actually happens um reoccurring so if you have user accounts and we can look at what was inside of that so if

we do the overlay list remember the generic so if we do ww control node list minus a you'll see that the runtime overlay is set to generic so let's take a look at what's in there overlay list generic minus long and you'll see we basically got user information in here so we've got our groups hosts file and password as well as you know roots ssh authorized keys but basically this gives us the ability for root now just to drop into that node easily but basically what we have now is we have things like the password file and that password file is going to be automatically

updated as users are added into the host system so if you add a user into this host system because of this overlay default configuration within one minute all your nodes will automatically be configured to have that user account added because that's the update interval for for the runtime overlay and you can see as soon as this hits basically 59 you're going to see another message another request to to get the update from this node that's booting so um you can also sort by the way this file or this output based on uh you know what what nodes have not checked in you saw it turn

yellow there for just a split second um and that's just because it hit that maximum time for the overlay uh request and it said well if we don't see it in in the update interval that we're supposed to see it in then that's a warning so it basically turns to yellow and again you can search for all of those sorts of things as you are managing your node so if you've got a thousand nodes sometimes it's hard to find the one node that didn't provision all the way or the one node that didn't reboot via its ipmi command which does happen or there was a

dhcp error again i kind of alluded to it dhcp seems to be one of the most limiting factors of scalability of parallel booting of hundreds of compute nodes at once so you end up with dhcp conflicts and whatnot so you can quickly see what nodes have been provisioned and what nodes have not using this tool i think i've pretty much gone through everything that i planned to go through um are there any questions i haven't we do have several questions uh alan allen has some good questions one of them we actually covered we talked about containers he was asking if uh if you can integrate

tools like boot to container but he he said that wasn't answered so this one's got a lot of parts to it greg and i'll i'll try to read it up there we go perfect um so is there an ability to have tight integration with baseball baseboard management controllers as well as data center layout switch configuration related topics let's start with bcm is it possible to do essentially everything that werewolf wants to do through the pre-os polling configuration file preparation by integrating with bcms tools for each node with open source software such as ipmi you kind of hit on ipmi but i think there's a lot

of that question so yeah so um the problem with using bmc's is it's not a standard interface for every vendor so everybody's done their bmc differently and even many organ many many vendors have actually done ipmi differently uh depending on you know what what additional features are trying to add now now at this point ipmi is kind of stabilized but the bmc controllers are very vendor specific um we do not have a direct integration into the bmc controller with one major exemption which is to help secure the provisioning process you probably noticed when we looked at where's my mouse there we go we looked at

overlay list www oops generic uh we looked at the files you'll see files like the password and group file are there now granted there's no passwords in the password file um which is you know we changed we stopped doing that like 20 years ago more than 20 years ago at this point but there's still user account information in that password file and you may have other things that you put into these overlays which you may not want users to find because we're using tftp to download all this stuff well excuse me to download the first stage then we're using http via pixi it doesn't give

you the option of having a whole lot of security so what we've done and if we look at the default ipixi file you're going to see that we're using things like the asset key and the asset tag so the only thing we really inter interface with the bmc with today are things like the hardware's uuid and the asset tags and if you specify did my voice just crack there that was funny if you if you specify an asset tag on that hardware it's interesting because users cannot get to that it requires root to be able to get to that asset tag using something like a

dmid code but this it gives us the ability to leverage that asset tag is almost like a a key or a token that unless that token is properly passed to werewolf werewolf will not hand out any of this information to anybody requesting it so it's actually a way that we can gain some security optimizations um through this booting process which of course whenever you're talking about security through provisioning you have to have a route of trust somewhere and this gives us a very easy way of kind of managing that route of trust which is directly on the hardware via the bmc and i don't think

this answered your question directly alan aside from it was a very fancy and long-winded way of saying no so alan i know you you've sent a link uh about some things that you're working on as well i know you and i have talked about redfish before um it's certainly something that greg and i can come back and follow up with you on and and have the conversation uh let's see one of the other questions we have is uh for maria asking about uh i've used what types of functionality would not be possible to provision through werewolf like what is not a good use case or

instance for werewolf uh i i gotta say it um windows werewolf will not do windows [Laughter] um that's the big one uh there are a few types of operating system images that um are difficult with werewolf for example if you have very very large operating system images uh it can be very difficult to deal with the provisioning of those so it's more optimal and especially just because of you know building a good you know big large scalable system you do want to have small operating system images and if you have more complicated requirements containers is actually a better way to manage you know large dependency

stacks and whatnot than putting it directly on the base operating system but if you have a very large operating system image uh it does it it's difficult you know from a scalability factor of getting that out to all of the nodes um in a in a in an efficient way there's also the second piece which is we are consuming memory for those operating system images so if you have a big operating system image hopefully you also have a hard drive that you can set up swap on if you want to set up a swap space on your hard drive you simply put that into your

fs tab and werewolf will or rather the kernel will automatically swap um uh the the space we're using in memory as it's needed by applications uh so this whole operating system image and memory will end up being swappable so as long as you do have a physical disk on your system just set up a swap space and that mitigates the issue thank you herman i believe your question was kind of in relation to what allen had posted i i could be wrong but i think you're kind of referring to what allen had talked about that's something that would probably be better if we followed up

with you so we'll get with alan and then we'll get with herman real quick i could i could also say that the um you can use ansible playbooks um or any even configuration or system management toolkit in werewolf there's two ways that most people think about it the first way is kind of after provisioning you use it to take over on the configuration management instead of using werewolf for that which to be blunt werewolf is very uh basic with regards to how does it do configuration management it's very simple and um you can't do a lot with it but it gives you that core functionality

a pre-running espion init that you would need um but the uh can you put that question up now i'm blanking out on uh answer oh yeah yeah so um the second piece of how people use werewolf and ansible is if you have multiple werewolf control nodes or managing your werewolf control node uh it's it's funny as as a jab to hpc people like myself i've heard a number of people from enterprises and clouds saying um what do you mean you use ssh for for nodes we don't even want our administrators using ssh anymore everything should go through something like ansible and so we've heard that

a number of times you actually can manage your entire werewolf control server using ansible or other configuration management and you never actually touch it so that's the other way i hear of people using ansible with with werewolf so here's the question now you're gonna love craig it's from kenneth thanks for the question and he's asking about is there a tool that will help you lay out groups and nodes in a visual gui mode it makes configuration easier so not at this point but it is something we've actually talked about quite a bit um because we're dealing with straight yaml uh building a graphical environment around

management of that configuration file actually is not not difficult to develop it's just something that has to be developed so that thought is actually been broast we've we've we've considered it and we're actually planning on doing it we just haven't done it yet and kenneth great to great to see you again alan that's all the questions we've had come in and if you guys have other questions post them quickly the biggest clusters i can i can just blab for a little while while we're waiting for questions um uh some of the biggest clusters that i know running werewolf people always ask me one of the

frequently asked questions is how many nodes can you boot on a single werewolf control server and the number um that i usually recommend is i say you know it's good for you know a thousand to 1500 nodes but i know people doing much more many more than that the biggest system is a that i know of is about 3 500 nodes on one werewolf control server um without any redundancy and without any load balancing i don't recommend doing that but the nice thing about the provisioning system is is it does not have to be high availability once that node has been provisioned you can actually

take the entire provisioner offline and if it goes down all of those nodes are still provisioned you're not actually affecting anything so they do not it does not have to be high availability in some cases you may want it to be load balanced if you're trying to boot thousands of nodes at once and i have seen rfps that says all nodes in the system whether it be a 100 node system or be a thousand plus node system the entire system needs to be booted and provisioned in less than five minutes from scratch and werewolf actually generally speaking is very good at that granted you may

have to build up your control server with a nice you know very fast ethernet controller and or split and bisect it between two separate broadcast domains so there's different ways of thinking about that uh different ways of handling it but you can definitely boot very large systems with werewolf very very quickly it's fantastic greg i i don't see any more questions coming in and i think we're a little bit over so uh greg thanks for the time it's always great to hear the stories of how this stuff started and be able to see it in action so really appreciate it looking forward to what else

we talk about later absolutely thank you everybody for coming join us again next time guys appreciate it bye thank you