Warewulf discussion with Gregory Kurtzer and Rose Stein

share that with you cut that part out no Snips no Snips we're going we're going to keep all outakes forever yall can y all see us how we really are totally not fun I love it I love it so where were you exactly like what was going on in your world when you thought hey we need Warewulf I need Warewulf so I'm going to create it so this started when I was at uh Berkeley lab and I was hired into Berkeley lab as a system administrator uh and uh my first my first kind of group that I was working with in Berkeley lab was the

earth Sciences Division and started off being system administrator really kind of focusing on Linux and I did a few other unixes and whatnot and then um but but they had a need for high performance Computing and that was one of the reasons they brought me in is because I have a s a background in science and already started doing some work in high performance Computing so uh the first big cluster that they that they wanted to build was you know it was it was it was not gigantic but it was a good siiz system now I'm the only system administrator in all of Earth Sciences

so it was you know and I was already doing a lot of work so now all of a sudden I think I had at the time like 50 or 60 separate servers that I was I was maintaining and and and and whatnot uh and those were mixed by the way of Linux and Unix and Unix always being kind of a higher effort of of Maintenance than than than Linux uh but so I I had a full load I was doing a lot already and then they asked me to go maintain you know you know a significantly sized cluster I don't remember exactly how many nodes

it was but it was pretty big and it was using uh just to kind of date this a little bit it was using mirette uh for the networking fabric um and uh it was it was kind of a you know big system and I was looking at it and I'm like that's a lot of computers to maintain I'm already maintaining pain in a lot now you want to like triple the amount of work that I'm doing amount of systems like there's got to be an easier way and so there were there is there were some tools already out there um three main tools that um

that I thought would be super interesting to kind of look at and evaluate the first one was a system called rocks another one was a system called Oscar and lastly there was one called skilled and I looked and evaluated these three systems and each one of them kind of had benefits and and it was doing cool things and maybe you know a couple cons at least in my mind of things I'd like to do do differently or do better uh so I I I basically would was looking at these I said you know I think there is a better way of doing this or a

way that i' actually prefer now I I came from a company called Linux Care by the way you didn't say this had to be a short story so sorry no we're we're going with that this is the whole story so I came from a company called Linux care yeah and it at Linux care one of the things that the company was was really known for was creating this really cool uh little cdrom image that was called the bootable business card and on this bootable business card and it was called a business card by the way because it was a a single CD ROM so if

anybody here remembers CD RS uh I mean I it's kind of old I guess at this point I don't know I remember eight tracks so the gray and the beard my first car had an eight track that's awesome I was going to say the gray and the beard I don't dye it to be gray so it actually is gray so I'm old so um anyway uh the CD ROMs you know most people think of them as like the 5 in size there was a 3in side size did they have like a flat side and then two round and then flat so the CD ROM singles

were the 3-in and if you have any old CD R players when you that have the little tray that ejects out and and what not you put your CD on there you may notice if you look at look at the old ones there's two rings there's the ring for the CD ROM that most people have there was a small ring right the smaller ring is for what was called the cdrom single and you're exactly right the bual business card took that CD ROM single and sliced both the edges off of either side so that little sing Single CD ROM could actually fit in your wallet

so it was called the business card size so it was a bootable business card and on that that CD ROM we put a uh an ISO image that boot just like an installer would boot or anything would boot but it booted a live version of Linux now today this is nothing fancy we're going back 25 years ago where this didn't exist so I promise this actually relates to Warewulf not just like reminiscing and just having a great old time so this little bootable business card was was was uh super Innovative and super interesting and and there's a lot of cool stories I can talk about

regarding the bo business card that team that did that is was fantastic like that was one of the coolest projects that uh that existed in my mind uh and it and it later gave rise to things like knopic and and the compressed loopback file system squash fs and and other other types of file systems like that so uh going back to the to the story now I believed when I first started you know building these clusters I'm like you know all of the world's problems could be solved all of them all of them with a bootable cdrom business card type note image okay the first

version of Warewulf that I made I coded this whole thing up to basically say okay in all my computers uh I I can make little bootable not little but they were full siiz but they were bootable cdrom images of the node operating system and the way that it worked is you had this template of your node operating system and uh from that template it would say uh you you would basically tell it go burn node one and it would actually go and burn it to the CD ROM they say you insert dis your blank CD ROM insert it and it'll go right to it and

then you take this and write node one on it go over to your node one put it in there turn it put it on and turn it on and now you have node one and it's running your whole operating system there's no hard drive or or anything necessarily in there but it's everything's running off that cdrom and you do that for each one of your nodes which works great if you have like 10 nodes 15 nodes because every time you update you have to burn a new cdrom image right and then you have to burn it for each one of your systems so it's like

yeah that's a lot of CD ROMs to go constantly be burning those of you may remember you know again the grads and whatnot that you used to be able to buy these in 100 disc Stacks yeah I bought a lot of those and um uh so I went to a Linux world to show this off Linux World used to be a thing I we were talking about that today bring Linux world back right you heard it here we're going to do it let's do it bringing it back so anyway continue back into the story so um I'm thinking this is the coolest thing ever all

the world's problems could be solved with a bootable business card or cdrom and uh and I'm going to show this off so I show it off and a bunch of people really liked it and what they liked about it was what we're what we've done is we've centralized the system administration to a single image so you have one node virtual node image virtual node file system that all of your compute nodes can can leverage and can scale from and so I brought this brought this to to a bunch of peers a bunch of people and they're they're looking at this and they're like we love

that part you did you nail this like the system administration side of this is so cool but CG seriously what the heck are you thinking with these CD ROMs like come on that is what a I mean what a waste of plastic among other things like and a pain in the butt like haven't you heard there's the ability to net Boot and there's this ether boot uh booth just right over there ether boot was the predecessor to pixie for everybody who's curious about what the heck ether boot is so it's predecessor to pixie so uh a couple of people brought me over to The Ether

boot booth and introduced me and they showed me how it works and and what it does and it was so similar in in boot process and methodology that it literally took less than a week for me to change transform all of Warewulf to use ether boot now the thing about ether boot is ether boot at this time was not there was no pixie boot like you today you have a network card or a network chip and you could just put Pi use pixie Boot and just bring it up on the network it didn't happen like that with with ether boot uh ether boot had this

ability that you can Flash the ROM in your neck so you can actually like burn it to your phys iCal like device but the way I always did it is it also came with bootable floppies again kind of dating myself here but you can actually take the ether boot image for your Nick and boot it put it onto a floppy and then put a floppy into your drive and then that will load an option ROM into the BIOS which makes that Network device bootable and then it boots and then it downloads the image and now all of a sudden we have net Boot and then

that was the first actual released version of Warewulf Warewulf 1.0 and um what that now gave me this is like the longest answer to a question that I think I've ever done but with that said that's actually saying something because usually I have long answer I think this is the longest so what this now gave us is this ability to uh net boot any number of images at massive scale so doesn't matter if you have you know 10 nodes or 15 nodes or 100 nodes or 150 nodes or even a thousand nodes we can now Boot them all over the network at scale and if

you need to change your image change your image and literally just reboot and that was what made Warewulf initially so super special the entire system was stateless and what this allowed me to do is to maintain a large cluster without having to uh really scale my time like you know linearly so let's just say the system was 500 nodes a lot of systems maintain from a software perspective it's only one it's only one image it's all the same right yeah so that was quite a long time ago right was at the early 2000s you talking about when I started the story exactly I think it

was about the 2000s when we first started talking you done yet um yes so when was this Linux World conference with the discs and all stuff 2001 okay so that was quite a long time ago and then what happened with the project like there wasn't GitHub back then right like what what what where did the project go how did you share it what how so there's a couple things that people are going to poke fun at me about okay and please go right ahead the list is long the list is long thank you I really appreciate that um I love you too so uh yes

the list is long actually uh so there's there's um a number of things that that you know 25 24 years ago like was kind of standard mode of operation so uh I hosted a sub version server which is the uh kind of the successor to CVS um and uh but used sub version and it was coded in and now you're going to really judge me but it was coded in Pearl and um uh I knew two languages really at the time which was C and Pearl and I'm like yeah doing this in C will literally take me a decade to write so I'm like yeah

I'm going to code this in Pearl much much faster so I coded it in Pearl uh post it to sub version and uh the way the project got out there and got well known is there was a website at the time and again all of you gray beards out there are going to um or or gray if you don't have beards gray hairs whatever um are going to remember that there was some websites out there there was sourceforge I didn't use Source Forge for this um but there's another one that was within the same kind of family which was something called Fresh Meat and fresh

meat was is an open-source site where people can publish projects that they're working on updates and and and you have different you know classifications for these updates and whatnot so you can kind of subscribe to I want to know every time there's a release in like HBC or I want to know be notified every time there's a I can follow the Warewulf Project without actually using uh Slash do or anything like that so uh so I posted a fresh meat and I learned the open- source Mantra release fast release often the reason why is every single release fresh me will post it to the top

of their list and then it gets it kind of goes down over time as other projects so the more often you release the more often you get kind of top of page front page headlines at Fresh me okay and everybody looked at fresh meat at the time so I would release fast and I would release often and uh lots of Point releases um probably did about a point release every week and a half to two weeks and um and getting up up you know updates out there I was part of the Bay wolf.org mailing list I was part of other user groups and whatnot so

talking about this working with other people uh at some point um uh there was a magazine called cluster World magazine and the very first issue of it and there wasn't many of those issues but there the very first one of it was um All About Warewulf do you have this no oh but I know where to get it oh Doug if you watch this Doug Doug you know Doug okay Doug I want to see this I'm going to call you this this needs to happen and be somewhere and framed and whatever okay oh Doug and Jeff okay they will know if they watch this like

And subscribe oh man that was good I like that okay so now that has happened and then you like when did you did you continue to work on this on a regular BAS is this release fast release often like how how long were you really involved cuz at some point you kind of stepped away from it and I where's that happening so um didn't step away but what I did was um like many projects that I worked on it was weekends and evenings it was not technically sanctioned as part of my day job there was not a a you know was at Department of energy

there was not a grant or allocation of money to go and to make Greg's life easier like they're just there wasn't so um to make my life easier I had to do it in the E evenings and and and weekends right so I actually did a bunch of Open Source projects in the evenings and weekends excuse me and uh uh Berkeley lab um UC Berkeley and uh Department of energy were always really helpful and supportive of that so uh so I coded this in the weekends and and evenings and uh over time it just kind of started taking off it started building up some speed

and at some point I remember Berkeley lab uh was was was looking at how we're maintaining our systems and they they wanted to do some publicity around how we maintain our systems cuz we got really good at this in in the HPC group that I was in and we got really good at this and um uh my position changed and I I I I became senior and then I became lead architect of uh of the HBC group at Berkeley lab and uh so so we started kind of doing more of this and and other sites started looking at how we're doing this and whatnot and

so the project kept growing and growing and growing to the point where uh another lab Los Alamos National Laboratory was building a system called roadrunner and Road Runner is fairly well known because it beat a whole bunch of the the the the records out there and interestingly it used the cell chip which is the same chip in the first version of the PlayStation if I remember correctly if not you can I'll comment below and tell me how much I don't know anything um you will anyway you don't need my permission I know well anyway so uh you can also tell me I'm right that would

be totally cool that as fun though yeah nobody will do that um so anyway sorry I'm I'm interrupting myself and losing track uh so uh they made this out of the cell processor and the cell processor at the time was super fast and it was one of the first kind of real accelerator accelerators that people were using outside of a CPU and uh but it required all these programmers to go and develop uh differently for that cell processor but it turned turned out to be again one of the fastest systems in the world and it uh was the first system if I remember correctly to

beat the petaflop or to surpass the peda Flop barrier and uh which is which was awesome and so when they were building this so IBM was building this and IBM was planning on using a system called uh uh xcat to build this system and xcat is another provisioning and an operating system management system but I believe Los Alamos or IBM somebody uh decided that they wanted the system to run stateless something that were is again really really good at so uh uh the xcat team uh or lead specifically Egan Ford hi Egan if you're out there uh awesome uh brilliant brilliant engineer and and architect

uh we started working together a little bit to to get Warewulf and ex-cat to play nicely together I believe Egan used to call it wear cat and um which is kind of funny um and uh so wecat uh really ran Los Alamos um Los Al was a supercomputer Road Runner at the time and not a lot of people know that it was actually running Warewulf underneath now later on xcat you know kind of continued development at the time it wasn't open source uh then it became open source and they implemented their own stateless system within xcat so many people who have been running xcap more

recently know it to be its own stateless system but it didn't start off that way started off leveraging Warewulf uh and um so I contributed to Road Runner in some cool way kind of you did okay so at what point so you said that you were always kind of involved with the project there was a point at which the project cuz now it's matured in a totally different way so what role did you play in the rewrite of Warewulf into go oh 100% uh for the rewrite okay um uh so you asked about if there was like a Hiatus and I guess actually I said

I didn't step away the project but I definitely had other things going on that kind of leveraged a lot of my time uh container system called Singularity is something I wrote as well um while I was at Berkeley lab initially and then took it out of Berkeley lab uh as I became an entrepreneur and um but so uh also Warewulf had some interruptions through the creation of chaos Linux and even sentos because sentos happened after that I remember a few people actually in IRC trying to uh battle to get some of my time to work on the different projects that was kind of that was

kind of fun really great for my ego by the way so um so uh what was the question the rewrite yeah go so um so Warewulf one uh the first version of Warewulf uh again Pearl and uh it did things in its own way back then I learned a lot from Warewulf one Warewulf 2 was pretty much an entire reimplementation of Warewulf it did things different configuration file syntax was different everything was different but it had the same like kind of major premise and and how it worked it's it's kind of it's it's set up how everything is um configured uh it's ideology Etc so

everything was kind of the same from from that perspective but it was implemented very very differently so system administrators and whatnot uh it didn't translate like where everybody using Warewulf 1 it was a kind of reinstall reset up for Warewulf 2 now Warewulf 2 continued to increase in terms of popularity and got lots of feedback and lots of people who who uh had a lot of opinions about how things should be done because the open source Community has you know we're not shy on opinions uh there's a lot of them well said very delicate so uh what we did was I I asked around and

I said uh various sites and people that I know were either using Warewulf or interested in using Warewulf or interested in the provisioning system and I asked and I talked to a lot of people and I said so if you know if if you wanted this provisioning system if you wanted to use this like how would you do your Prov your provisioning like would it be over HTTP would it be over uh a file system like luster would it be over like all these different sorts of ways and what do you want your backend data store to look like what do you want your configuration

syntax to look like and and how do you want all this stuff to kind of work and fit together this might be surprising everybody said something different so so yeah at the time not totally surprising everyone's a unicorn right every environment they like to think so yeah we are all completely unique totally unique everyone is complet we all we have that totally in common anyway um so uh what's this no it's it's you are totally unique just like everyone else that's what it is I like that so uh anyway uh everyone had their own opinion about how this needs to work what does it need

to look like uh Etc so I spent a huge amount of time trying to make everybody happy in Warewulf 3 and I I I set it up such that every part of it was modular and in Pearl it's not the easiest thing to do so I created all these abstraction interfaces inheritance designs and whatnot uh so I can so I can so everything can be modular and people can have their own subsystems their own events and all these sorts of things so so set it up so everybody can do it internally how many people do you think actually wrote any external modules or any actual

configurations to have it do anything different three none none zero not a single person but I tell you what it did give us did give us a maintenance headache to try to maintain this code base that is has so many layers of abstraction and indirection and whatnot such that we can do anything yeah for in terms of provisioning uh at some point I actually did use this because I was investigating different ways of doing scalable provisioning so uh one was you know just standard tftp one was HTTP one was a bit torrent actually did use bit torrent for doing provisioning another one was something called

xget which used the Plan 9 file system implementation to do parallelization of of threading I did multicast I did so I tested all these different things a dolly ring I tested all these different mechanisms by the way the dolly ring was was was the best but it required everybody all the nodes to boot in complete parallel and Unison which doesn't happen as most people know uh but that was the fastest of anyone was ever curious so um but I tried all these different things um oh and xget was the was the most robust but it wasn't it wasn't fully maintained um at the time and

it was still kind of a a um an experiment so anyway sorry side tangent um what was the question again so the different vers there was a different Focus for each version now you're on version three where you're trying to please everyone you know okay oh and coding y so Warewulf three kind of remained for a long time it had a very very long life and open people still using it actually oh there's a lot yeah in the Warewulf slack Channel there is a Warewulf 3 y subsection and people are thriving in there yeah and open hpcs it was actually using it up and even

until just recently like last month this month even maybe actually no just a super Computing I believe they announced that they're now moving to yeah okay so now okay uh a few years ago I said okay you know what let's move on to Warewulf 4 and a couple things kind of became totally obvious and apparent the first one is that a vnfs the virtual node file system that we talked about doesn't have to be managed specifically through Warewulf it's a container and as long as you think of it as a container the entire almost the entire container ecosystem all of the containers in Docker Hub

everything can theoretically translate to a Warewulf node image okay so I spent some time kind of reintegrating and rethinking about how are we going to pull all of this together the container ecosystem Warewulf booting all of this stuff and so uh that's what we started kind of building and thinking through and um uh that was kind of the the main gist of of Warewulf 4.0 now the whole container ecosystem including Singularity not the first versions of Singularity first two versions of Singularity written in C um but the third version and I wrote the first two and then uh Cedric led the development of version three

and translate he translated that to go at this time I did not know go and I said I am too old to learn yet another programming language so I didn't want to I'm like no I don't want to do it grumpy and so uh but then came time to move Warewulf to this and I said you know what I'm going to learn go and so I sat down and started coding it uh I first actually created a little User Group around next version of Warewulf and we we had anywhere from you know 15 to 25 people that would join off and on we had weekly

meetings and whatnot to kind of identify what are the features that everybody needs and whatnot instead of taking the same path as Warewulf 3 where I said I'm going to do everything to make everybody happy I basically listened to everybody and then I made a bunch of decisions and said here's how it's going to work and uh Cod didn't and go as I said and uh uh and then release that I think I wrote as a matter of fact I'm I'm 99% certain I wrote uh almost a 100% of the code in go at for the first release of Warewulf 4.0 from there started getting

more people from the community involved open HBC team in got involved uh Susa has been in has been a fantastic contributor to Warewulf love the Susa team they've been so awesome and uh we've been now in in in you know typical open source fashion we've all been coding on this we've all been contributing to this and uh others have now taken on more of a lead leadership role within the project so I'm just now uh uh just kind of an overseer and um uh which is a technical term for complainer just so everybody knows um and and I basically just kind of watch and I

see how things are going and I'm there for for advice and whatnot but the team has been doing fantastic in terms of taking it and continuing the development and bringing in new features new capabilities and things that I I either wanted to do and and didn't have the time or things I didn't even think of that are so much better and bigger than I than I even considered so Warewulf has continued to develop uh and Warewulf 4 is out there right now uh open HPC is now standardized on Warewulf 4 as well and uh and and Warewulf's doing fantastic uh ciq is now doing a

bunch of additional development around Warewulf uh and just a forec forec forecast um foreshadow foreshadow sorry uh words are hard sometimes that was fun that for catter for catter I feel like that should be a word I know at some point I actually said docility thinking of Singularity and Dockers and I actually did that then a public for at some point um so anyway um there was a uh um the the I lost my train of thought uh I'm doing that a lot you know I think I need some coffee that's probably accurate but I need to get to the airport actually so we're going

to finish this um little interview code word for Greg you talk way too much and no this is actually really fun we're getting to the part that we kind of wanted to get to but like all of this leadup and understanding the backstory is actually really interesting and really helpful for me and I think for the community as well like sometimes when you know why and where something came from then it kind of it gives it a little bit more context you go oh yeah that is something that I can use in my environment so where is Warewulf at currently right now oh and I

do remember by the way what I was going to say so the plane can wait okay um I'll tell them just just tell them the hold this way you'll be right there uh so uh ciq is now putting in uh some additional features uh so the foreshadow there you go um CQ is putting in some additional features right now around 8 apis for Warewulf you can actually communicate and do things through Warewulf via an API and we're putting in a lot of that effort because some of the other things that we're building like Fuzzball and whatnot it's going to Leverage The Warewulf API and we're

building a goey that's going to go on top of Warewulf and that's something that has been asked for for so long so we're doing a graphical environment I've seen pieces of this this integrates with uh co-pilot with your existing kind of uh system configuration uh and It just fits right into that and now you've got a gooey for Warewulf that will be part so the API of course is completely open source that's going to go with the project the guey is going to be part of CIQ's value ad so if you're interested in a guey for Warewulf uh come give us a call uh

we're also working on here's another little bit of foreshadowing we're also working on uh and going to be releasing here very shortly a turnkey HBC stock that is completely compatible and compliant with open HPC and Warewulf V for or with a gooy wow so when you say TurnKey what exactly does that mean um like well imagine if you had download something boom your cluster is up and running almost almost it is a uh everything you need for an HPC system uh it'll include the installers it'll include the repositories it'll include the containers because remember Warewulf operates on containers now so it'll have the access to

the repositories that have the containers you'll be able to import those containers that are pre-made node images which are fully compatible with the rest of the the rest of the image over time we're going to have different types of node images so we can have IO nodes we can have compute nodes we can have visualization nodes all sorts of cool stuff that you can Import in a Warewulf and then provision out to your system so lots and lots of cool stuff that's really exciting so what about this debate between the you okay um totally okay I was trying not to trying not to like cough

okay um what about this debate between the stateless and stateful from what I understand that once you understand how Warewulf actually works and what you can do and you know in your environment what it is that you want to do in your environment then people kind of calm down about that a little bit that's a question I get often so for the last 24 years that Warewulf has been in existence people have been debating are you sure Greg that stateless is the right way of approaching ing this yes I'm sure now you may not believe me I look forward to having that debate with you

uh let's talk about it uh and you may win uh there's lots of cool cool ways of thinking about provisioning where State actually has some some benefit so we have been thinking about how to do stateless stateful excuse me as a matter of fact there are some ways to do stateful in Warewulf today so we you can there's just a little extra manipulation that you got to do is something with overlay yes yeah we can do it we can but even just because we can doesn't mean we should and I think I have a feeling that a lot of the people uh end up wanting

stateful because it feels more comfortable for what they're used to used to managing systems that have hard drives that are have an operating system you can ssh in you can you can manipulate your operating system and that's a persistent action so people are comfortable usually with what they know uh and completely fine but I will say that there are a lot of advantages to using stateless there are some disadvantages as well um uh I'm diluted in many many ways but not everything so there is some advantages and there's some pros and there's some cons so you have to be able to balance the pros and

the cons and let's figure out what makes the most sense and if you want stateful let's figure out how to do stateful using Warewulf or something else doesn't matter to us like we're here to help so whatever you need you know almost almost almost can't help you with a haircut funny in my mind may maybe we we can okay maybe I mean give me some scissors you know I don't know if you'll like it but I can cut it for I have cut hair before so just saying just saying like the Zohan oh that's amazing uh you know so there's probably a lot more questions

that have come up um even though we probably answered a lot we gave you a lot of history it was really thank you so much for coming in and all the good work that you've been doing and being pulled in all a million directions and I actually didn't know that you were still so involved with where with communities that actually makes me really happy thank you for coming out here I mean I didn't have to fly to here you flew here I did I mean you didn't fly well not with my arms okay you guys we're having way too much fun here so definitely and

I'm out all right we're we're being cut off our our directors producers are now saying it's time to cut it off we talked too much that's on me sorry Rose as always thank you yes pleasure bye let's see okay recording so there's actually a really super interesting story so uh at some point uh Berkeley lab was was setting up the system for geo geoh physics and as part of this they were uh it was this is the first infiniband system that that Berkeley lab actually built and uh we bought the system from Dell and Dell had a great partnership with the NCSA and so the

NCSA uh and Dell said let us let us work with you to optimize and build this infiniband cluster and to show you how new this was ofed wasn't even ofed at the time and we had to check everything out from a head of CVS like this was like totally not um uh you know uh production release software not even a little bit so what we did is well we we built this up and and the NCSA said well we know how to build a system and I don't remember what system that they just built with this infiniband so uh they like let's just emulate the

whole software build that they did and so we got the hardware we rolled the hardware in and we went through the Dell merge Center so everything was already ready to go it was awesome put up all the racks stitch them all together uh built the infin ban fabric uh into a fat tree big old fat tree and uh and then started doing the software install now the NCSA took I mean they did it pretty quick and they were happy with it and for all practical purposes I think they did fantastic it took about if I remember correctly about 4 days maybe five to get the

full software build going and get uh the software application validated and everything done so didn't take a long time but it it took a good amount of time to kind of do it uh and get everything going but we built it using the same system image and everything that they did at the NCSA which was they based it on rocks and uh rocks a great provisioning system great management system and whatnot uh and and I I I feel like it it did really good on this but we don't use rocks at Berkeley lab we wanted to use Warewulf so it took almost the entire time

uh in the afternoon right before on Friday right before everyone was ready to go uh everything was qualified software application was working everything was good to go and um everyone is like Yay it's great let's go have dinner and celebrate and whatnot and I raised my hand I said hold on we said we were going to run this with Warewulf not rocks and so everyone's like you're joking right we are not going to reinstall this whole system right now with Warewulf so well I was like no actually not joking like we're going to redo this all with Warewulf right now so uh in order to

just make it so we didn't have a whole bunch of wasted work just in case Warewulf didn't work we put a new hard drive into the control node so we can always go back just in case although I knew it wasn't necessary but I wanted to make everybody there comfortable that all the last few days of work that everybody did isn't completely wasted so what we did is we we we we put in a new Drive reinstalled the system inst so I don't remember what version of sentos it was it was a long time ago so it was probably sentos three or four or something

like that and um uh and and rebuilt a system with Warewulf now from start to finish with Warewulf I think it took 2 hours uh and then we were qualifying the the the the jobs qualifying everything and everything worked so we were able to stand up the whole system in a couple hours it might have even been faster than that I actually don't remember but it was it was within a couple hours for sure and uh and the NCSA was the people that were there were so impressed with how fast and how quick and how easy Warewulf was to use at the NCSA actually at

that time started uh evaluating Warewulf for them to use internally as well and uh we were able to rebuild that whole system and get everything working super fast uh there was also a very interesting slight performance increase with using Warewulf now uh it should be noted we didn't rebuild all of the binaries either so we were even using the same binary the same Geo physical geophysics binaries we're using the same binaries that uh ran on on on rocks before and uh I believe CU this was a fairly big system I believe that the difference is because Warewulf just didn't install a bunch of stuff like

by default it was very very very minimal node images so there wasn't any other services running there wasn't any um uh Damons system Damons or processes or anything else that was running so it was less operating system noise and and whatnot so I think it just ran super efficiently now I I don't know if it would have had that much of a difference but we definitely saw a little bit of a speed increase and that speed increase was uh very welcomed by the Geo uh physics scientists who were there helping to qualify the software so at the end of the day uh everybody was super

happy with Warewulf we were able to do it still make all of our timelines and we still all went out to dinner together so it was a great experience