Migrate Hosts from CentOS 7 to Rocky 8 with Ascender

pieces absolutely and one of the things that we harp on specifically when we talk about a sender is if you automate this stuff it makes that transition easier later so a lot of people have done a lot of configuration management and they've done a lot of automation but it's still surprising how many people weren't or are not prepared for something like this so I think this is a fantastic topic this is a really good topic I just had a crazy question come into my mind but maybe I'll write it down and ask later so we can kind of jump into it do it yeah okay

so I'm I'm driving it down uh but I think that we have okay so what exactly are we talking about I think you guys have got it right so it's like migration yeah migration specifically from sento 7 to Rocky 8 and I actually had a call come in a question come in today of like hey can can we don't want to move from Seven like what then I'm like okay well I can't really answer that question for you so we are rocky 8 and beond it's a much more complicated question to answer uh yes you could continue to run it if you so chose just

know that there's no Arata there's no security fixing there it it will be stuck in time stuck in time and I unfortunately see far too many people that have just done that they've stuck themselves in time on some very old Enterprise version fours version fives we're talking 16 17 year old versions it's not uncommon I've seen not a great place to be not uncommon oh yeah I saw four recently yes Mission critical 6,000 machines for oh man you're making it tough for your own self it's terrifying terrifying job security okay so let's bring Greg on I know that we have an there he is is

everybody a little sound a little rough today oh my goodness horse did you wake up with the allergies going on around no this is day three of fever so I'm not really sure if you guys are actually here or is this still just the fever dream so it's dream dream just do a really good job either way doesn't matter well this is actually really powerful because if you can use a sender to help you automate even when you're sick this is powerful stuff right so this testimony to the power yeah I'm I'm buttons you'd be surprised we say that cuz somebody like are we if

you're sick and you can still do your job you don't to take sick taste so that might not make everybody happy but the admin would be happy automate my job away that's kind of I can be sick and not tell anybody they'll never know fantastic I love it well I mean really to me it's uh is there a difference can you tell if I'm sick or not because I think I do this amount of work either way so all right with tell tell us all about it show that this tiny bit amount of work that is going to save a lot of people a lot

of headache a lot of time and a lot of money yeah yeah let me do the uh the screen shared thing I think um it is the appropriate caveats in here too absolutely it is pretty interesting um after effect um is everything sharing we we looking good no I can't see anything we're not not yet right this one fellas and fellet uh let me try one more time it says there it there it is there we go it was just thinking on it it was sharing somebody was somebody was thinking in the background excellent we're going we're good namess yeah yeah you uh you kind

of hit the nail on the head there Zan some caveats uh we definitely need some um uh some asteris around this stuff I'm not saying you're going to be able to transition every one of your machines so I have been doing some testing using the Elevate tool made by our friends over at alal Linux they uh they created this awesome tool called Elevate that kind of tags on top of the leap tool so it's like a little bit meta right we're using several tools that call A Tool uh but the uh Elevate tool will take say a Cento 7 box in this example is what

I'm be working with here and it will migrate it over to a lot of different things and one of those things is Rocky Linux 8 so I'm doing a Linux 8 now I am doing a fairly simple build it is just a simple little web server you can see this amazing web page that I made yes it is the testing web page so I I didn't even bother to put an actual HTML page in there I just used the testing page but um if you are doing a simple machine right it um using kind of standard packages there's nothing to chocolate in there right vanilla

is going to be kind of plain normal chocolate's going to be customized so if you're uh if you're pretty much a vanilla install uh so far I've had pretty decent success using the tool and uh it is about a 20 minute process with about 10 15 individual steps so it's pretty nice to automate it let me show you how simple it can be so this is I think we're not gonna go too much into sorry I forgot to turn my dinger off of my watch not going to go into the tool as much in this one is how to automate the tool so not as

much showing how the actual tool works just want to yeah yeah yeah for sure yeah I'm not going to go down that rabbit hole and in fact we have a blog post I think yeah this right here uh a blog post that goes into nice full detail on all the steps we took right breaking everything down so if you really are curious how all the look at this look these beautiful screenshots I made so if you're curious like how to reproduce all of this there is a stepbystep in addition to a video um because different people learn in different ways so yeah if you do

want to go further down the rabbit hole we talk more about kind of the tool itself in there but we're here to see some cool stuff happen right so in my Ascender my automation platform here right this is my Grand Central Station where I run all of my automation type Centos and find all my job templates associated with it so I could have just one Playbook right if you're familiar with anible we create these things called playbooks which are just our automation scripts that anible consumes and performs in interesting actions um each one of these listed as a job template uh really is referring to

a Playbook right so these are all just little playbooks now you can see I have a whole bunch of them and that's because I have them do very discret operations instead of one big thing to rule them all I uh I break them into little bite-sized chunks and uh as uh I think the uh the famous famous famous Justin berdine once told me he uh he likes to do that because it makes of Lego bricks and so I've stole that line uh for myself as well so they become these little things you can snap and use in different workflows and so uh I'll hit the

CTO 7 migration workflow visualizer right here and as any good uh network engineer likes I've got these little pictures that I can follow in diagrams right it soothes my brain I I love seeing it in this fashion but each one of these squares is a different little Playbook that it's going to run so I'm doing a migration on a centos7 box it's actually one that's important man theoretically not really I don't care about my test page but theoretically I care about this box you know it's probably something in production or maybe it's in some Dev environment where it's actually important so I can't just try

and migrate it and hope everything works right I need actually some assurances here so what I do in my environment is I'm running VMware and the first thing I do is I create a snapshot of the VM right that's nothing very complicated I can go in there and I can right click and say create snapshot but I can also do it an automation fashion so the very first thing that happens is creates a snapshot once that's done it connects into my playbook right the centos7 migration Playbook and that actually uses the Elevate tool to um perform uh well the Elevate tool first it has to

do a pre-upgrade sequence where it's going to look for everything it might need to make an adjustment to spits out this file well the automation then goes through that file and looks for any recommendations it has and it fires those Against the Machine then it completes the upgrade cycle after it's done I have a playbook in here that will test the web server right this could be test your web server this could be test your database server this could be uh test your DNS server whatever it is you can have any kind of test controls baked into here so trust but verify right it looks

like it upgraded let's go ahead and verify that it upgraded well if this migration process fails follows this red line right here that restores the snapshot remember I told you this was like an important machine maybe in production Maybe not maybe in some special environment well if it fails I'm gonna go ahead and roll that snapshot back right I'm going to revert back to a known good State and we can try and recoup recover from there later but in this instance it's going to work of course it's going to work right I wouldn't do this live if it wasn't going to work um it's going

to hit the web server testing again if that fails restore the snapshot and in my demo environment if it everything succeeds I'm going to go ahead and remove this snapshot now I'm going to tell you from doing stuff like this in actual life I would not not have this remove snapshot step there I would have um I would have a reminder Somewhere In A system that says two weeks later if everything is still working right go ahead and delete that snapshot because uh oftentimes there's Gremlins you know somewhere in the system that you don't realize are there for a few days even so it's nice

to uh always have kind of that backup there I think it's important to point out this is much much easier on a VM and that's probably going to be 90 plus% of the use cases but if you're doing this on bare metal make sure you test your backup solution before you start I've actually run into that more than once where backups weren't necessarily backing up or we thought we could recover from a backup that we couldn't get back so delay yeah absolutely absolutely nothing is 100% foolproof well this uh migration I told you you know even though I'm three days into my uh fever dream

here I can still click a launch button and it's going to work now this is a 20-minute process so we're not going to stand here and stare at this I'm going to jump into the job section and I'm going to look at the one that I ran yesterday so we can talk through that and I'm going to click on the workflow right here and you can see all the steps that happen inside that workflow visualizer right so even the output looks just like it did when we were actually building the workflow and I can click on any individual step and it'll take me into that

individual job and I can see all the details right I can see the output oh I did the snapshot right there's nothing really too interesting there let's see here I'll jump into the workflow really let's go into the migration that's the interesting one that's what anybody cares about in here now remember any time I have a job run it produces all this awesome information and um it is a bit overwhelming to look at all this stuff but really if you break it down it's showing who did what when and where and what was the outcome all of this information uh that we're producing here can

actually get shoved off to a logging ser server as well so I like to give a plug for Ledger um it's custom designed just for pulling in logging messages from a sender kind of any system that's derived from Upstream awx it's really made to take and parse all that stuff great for compliance great for reporting I can even run a report on anything that changed right so I'm changing a bunch of stuff here so it would be a good uh a good uh tool for using that as well but I can come to my output and I can see all of my beautiful output here

I start by um doing a uh quick Centos 7 version check right I'm just building it so that at the very end I can show a little message that says this was the version before this was the version after so I'm grabbing the OS version I'm installing the Elevate tool I grab all the packages for said tool now there's not a whole lot of detail that we see right here but say for example I want to understand what happened here I can click the changed part I can click the Json output and I can see the detailed output of everything that was installed right so

I can see the exact package that we pushed to that machine so all of it's here and available for me to take a look at well it's going to do the preupgrade it's going to generate a report it's going to have me go through I do some standard pre-upgrade fixes but then also anything that was in that pre-upgrade file I run all of those you can see I'm doing that right here next I do the upgrade and that joker takes a while after it does that it goes into the the reboot sequence is what I call it because it has to reboot it'll do some

configurations it'll reboot again do some configurations reboot again then it's up and available after it comes up I pop back into it I check the OS version and I spit out this glorious message the version before when we started was centos7 core version after running Rocky Linux 8.8 so I don't know that's probably not the coolest thing you've ever seen um but the fact fact that it went through this 20-minute process and then did all the verification to upgrade me from Cento 7 now I'm Rocky 8 and I'm rocking en rolling to me that's actually pretty dope it's fantastic so one of the things you

talk about in the blog Greg is making sure you don't do it against every server in an inventory at the same time correct I um I'm an extremely paranoid person I came from environments where we had 100% slas on literally everything so we came from a data center environment where uh and I was in charge of the internet and I had to guarantee 100% internet wasn't going to go down right so being in those kind of environments really um it breaks a person let me just say that right uh no it it really changes the way you look at the world honestly and so F

Dream It's fever dream residual no I'm a belt and suspenders kind of person right so I I uh check and I check and I make sure that everything's going to be okay so one of the things you can do in your your playbook and in my playbook is you specify the host you want to operate against inside your playbook and if I pop into mine right here you can see I'm saying operate against all hosts and that's cool unless I accidentally run this um against a really important inventory and then it converts all of my stuff maybe I don't want it to convert everything you

know it could be dangerous that this could be um delete a drive off a server and then I run it against everything that could be really scary right so what I do is I bake into my playbooks while I do have hosts all right here I also um have the very first task check to make sure that I'm not running against host all without adding the limit option so in a job template that I'm going to run in a sender I can actually there's a little field down there called limit and I can f that in to uh further narrow down who I'm going to

actually operate against inside that inventory so it's just kind of again a belt and suspender way that I'm making sure I'm not running against everything but I can also limit things down so that is definitely kind of my safety valve I bake into these playbooks isn't it glorious all of this stuff and this is also open source in a public repository you can grab it you can use it um I am always open to feedback if anybody has any questions or comments or they would adjust it they would add this cool thing in here uh do a do a push request Rose you said you

had a crazy question which if it's c to you I mean I don't even know if it's going to am might be able to handle this I don't know I don't know I so I have three questions actually that have come up as you were talking oh we're adding to the list it was one now we're there was one I mean but I think that probably this happens to you as well right it probably happens to everyone it's like when someone says oh hey you can do this and it's like but can you do this can you do this can you do this right it's

like where does this ever end so this is my question so we'll go back to the very first one just to kind of wrap wrap up here can you okay so we were talking about like those old sentos boxes right like the four five six seven can you like and I say this I know that we're talking about a you know a sender and so this is a little bit of a different tangent but we also as ciq we also support obtainer like can you containerize those old systems and just let them hang out there like is that what people do like how do you

make sure that it is here gotcha well I think yeah zanen touched on something really important is that if you have an old system and it's been deprecated it's not getting updates anymore right so any security related stuff I mean while there is a way you could you could segregate it off and keep it alive over here and I have never met anybody that does doesn't have this special little area in their infrastructure where it's like we can't touch this thing but we absolutely have to have it running you know those things live over there and you try and protect them as much as possible

so really if it's something you want in your production environment especially if you have any compliance requirements or security audits you really don't want it hanging out on something that's end of life you don't and I agree but we do have customers who have taken old machines like that old apps that are running on I know we've done it with Enterprise 5 and actually put that into an Apper container and then you can execute it on newer machine so we are in some ways enabling Tech rot it does happen and it it does work in a lot of cases it's it's perfectly acceptable to go

do that if you want to have that conversation of how we can help you enable more Tech we would love to but we would prefer if you would just upgrade yes keep up with keep up with what's going on but there there are ways to do that uh it is a lot of ways it is elegant because it makes it simple and it makes it portable but it also is not a great idea so if you're this kind of goes into my next question I mean if you're taking this and I know that this is not the only way to do it right using leap

isn't the only way although this is kind of what we're talking about right now so if you can use a sender and leap to help you upgrade from major version to Major version of an OS can you also automate and do one at a time for like regular dnf updates like on current versions oh so like uh package management is that what you're referring to like I don't know what you call it absolutely yeah yeah yeah yeah sinder that's one of the um I would say one of the first use cases folks think of is like how do I make sure my infrastructure out here

is up to date on various packages so it's a very extremely extremely common use case for it so you can uh do rolling updates is a fun one right so you don't update all of your infrastructure at once you do a little chunk here a little chunk there and uh when I'm saying updates I'm not just talking about your Linux servers I know most of the people we talk to are are uh our Linux admins but for you Windows folks that are out there you can stage your Windows updates for those of you that are network Engineers you can do your firmware updates that's a

lot of um really good I would say almost low hanging rout uh as far as that stuff especially when you're talking about like Network updates usually any device you're going to perform an update on is 15 to 20 minutes of human time per device and then on top of that that's going to be you know you need to do some testing some checking on top of that and so people going across their entire fleets can save so much time and have so much a more reliable experience using automation it's very exciting I have one more question but if you guys have questions put them into

the chat right there we've got people all over the place looking and hanging out and so we'll throw your question up here and we'll make sure that you get it answered um but this does kind of go into my last question here okay so can you update only specific parts or packages within the operating system and not the whole thing you can yeah so in Linux uh say for example we'll probably be using the dnf module and a module in anible is just like a pre-written chunk of code that I don't have to read I don't have to understand all I have to do is

look up the documentation and say uh dnf module let me fill in the blanks on the things I need so uh say for example I was doing a web server if I just want to update Apache I would say call the dnf module name so what's the object I want to update and I would put httpd because that's what they call Apache instead of calling it Apache uh and then I would say State latest and I I would run it and it will go and it'll connect to that machine it'll make sure Apache is at the latest state or I can even specify a specific

version number I wanted to upgrade to so you can be very uh granular in your updates and all of this can actually go back and to plug what Greg was talking about earlier ledger so from a compliance audit and compliance standpoint you could push all the what you did to Ledger and have the auditing compliance people go look at it themselves without ever having to come ask you for the data so it's a nice getting people off of your back audit know how fun that can be yeah oh goodness Ledger can even um schedule reports to send so it can just automatically show up to

them if you want it to to do that for all of your infrastructure not just Linux to Greg's other earlier point that was awesome thank you I'm sorry went so fast I know this thing we're supposed to drag this thing out no we don't need to drag anything out do some show tunes my voice is like three octaves lower right now so I could do some Berry White maybe or maybe is your voice gonna crack because that's what's gonna make it fun 100% it will I guarantee it it's awesome all right so I'm gonna go do we have questions guys question from the folks questions

from internal external have Jonathan making comments that's it Jonathan making comments why of of just like admiration and aw all problems in computer science can be solved by another level of f direction that reminds me of um uh today's bug is a direct was it a a direct correlation from last year's bug fix or something like that so yeah so unfortunately true though you're going through and writing stuff where that happens a lot to cover from the from your blog no iing trying to think I think the one that I was really key on was the uh being able to do multiple machines but make

sure you're not doing them all at the same time rolling rolling through yeah for sure well uh there are some um like serialized features inside of a sinder we can do so the idea being uh I could do batches so I can say say I have 30 hosts I want to upgrade well you know say they're all performing similar functions I can tell it all right run against a single host first and it will operate it'll make it way all the way through if that complete successfully I can have it right just in one run I could say do this and then it'll come back

and it'll say all right run against five hosts and if that complet successfully all five of those I can say then run against 50% of what's left and then 100% so I can actually kind of slowly batch it through um that is especially important for say networking elements uh you know like uh I have push configs that uh break things and so it's good to make sure that you know if I'm doing like some new firmware upgrade and it breaks stuff in the field it will stop before it breaks anything else so I can just kind of have my canary first in the list make

sure everything goes through clean and then you know go ahead and start batching through the rest of my infrastructure it's fantastic and I just thought of another question now I lost it where did it go I don't know I must have Greg's fever dream brain so I I think probably a really good point to make right now is that anyone who is familiar with automation probably either knows something about the capabilities of what it is that we were talking about but maybe are a little bit overwhelmed with when they go to support because most of the stuff is is open Source right they can go

and play around with it and figure it out and command line and do all that jazz and figure out automation make a Playbook and all of that but if they want support right if like this isn't like their their jam like they want to automate stuff so that they can go and do other stuff or if they're kind of unfamiliar with it getting support around that can be super crazy expensive correct excuse me I'm hanging in there okay we're live are we're close man we're close yeah for sure for sure yeah I mean you hit the nail on the head there um Rose and um

uh support for me has always almost been a four-letter word like like you have ultimately failed if you have to contact support you know like uh if uh your home internet goes out you're probably going to have a bad time calling support right and I've had opportunity to work with multiple vendors that made me feel that way um which is unfortunate and definitely um the word justtin likes to you is reimagine um but uh for me it's it really it's simpler than that it's like what kind of experience what I want to have when I contact support I want to not feel hinky about it

I want to not feel bad like I'm about to lose a chunk of my life and that's kind of how our support operates right you get in there you get with a tier three Tech like right off the bat and you get your problem solved and you can actually get a human on the phone which is for me sometimes I don't want to talk to a human I totally get that when other people are in that mode but sometimes when the world's on fire you just want somebody on the phone to uh to help you get through it as quick as possible um as well

as you know maybe you want somebody to look over your shoulder and you know literally like in a zoom call look over your shoulder and help you work through it so to me it's been um kind of a night and day different experience here at ciq versus a lot of the other places I've worked with yeah I remember my question and Michael asked another question I think you already touched on on it but you put all these playbooks out for consumption right MH yeah they're in um public G repositories all the links are in the uh blog post as well in the blog that's what

I thought just making sure yeah my question you actually sorry go ahead no I was just saying then the blog post is on the website I mean that should be obvious but you know I feel like they linked it to this as well oh yeah my question whenever you said internet goes out that was actually my question not if internet goes out but can you run this not just this but can you run the actual Elevate or leap tool without an internet connection I'm going to bet you can um part of the migration is it wants to make sure that your centos7 um server is

updated you know it needs like the newest packages on it first um so that could be a project for you to take care of ahead of time you don't trust me you probably don't want to do updates and then do this migration because if the updates broke your system before you did the migration you're going to be chasing your tail for a while so I would say make sure everything's updated first and then let's worry about migrating um but there is just a handful of packages that are required and you could have those stored in kind of a local repository that you could pull from

uh so you could do it um disconnected I mean it doesn't have to be online I'm doing it that way because I'm lazy and it's easy but you don't necessarily it's fantastic and I we do work with a lot of people who do not let their servers talk to the internet this why I ask and I think it's also a good interesting point that in the near future you would be able to put these Centos repos into Mountain so you could do it an airgap environment so have have it pull from Mountain what you need and go apply what you need for Rocky yeah for

sure I think that's a good uh a lot of our very security conscious people that's kind of a best practice for them well I hate saying best practice it's a good practice right should be yeah like we do have another question oh from Jimmy Jimmy cona what's up man thank you for watching so upgrading between versions is a good stop Gap whereas the end goal should be to automate the building of the servers from scratch as then you build your application out on any new version that is true that is definitely one of the things that we recommend as a best best practice but sometimes

don't want to I would love to to see more people go this route I know we've been yelling this for years but it's amazing how many people are still not proficient at it always said like cattle versus pets kind of conversation where if your machine is something that you have to lovingly tend to all the time right you can have problems with it whereas if it's something that you're not married to you can just destroy it build something up in its place and you know you just pick up and I mean that's a lot of security people like that too if a machine gets compromised

you just pull it offline and you put something something else in its place that way you can actually look at it figure out what went wrong um kind of uh do some after actions on those stuff I think that was actually a direct attack from Jimmy to me because uh like my personal blog that thing I've been limping that joker along or the last 12 years from one version to the next just upgrading and I it's probably on such an old version that I can't even use this technique to upgrade it either so I I definitely need to get that taken care of point taken

Jimmy I get it point taken point taken yeah I I I mean I understand that is definitely our our recommendation and so it is a little bit weird to kind of you know being like well you know if you don't want to listen to us or get our help around that then you can use this thing over here so here's this cool thing um but we're also really highlighting a sender and the power of a sender and all the different things that you can do with a sender which is open source and thank you uh Jimmy and Greg and Michael and Justin and all the

other amazing people who have contributed to that um but as a company ciq does offer the support and we wanted to do it a little bit differently and instead of having you count nodes and what are things connected to and how's this going and like charge you for all the different things we're like you know what we just really want to support the admins you know we kind of started our conversation today about that where it's like you get more free time right automate things we love you sh the bosses aren't listening wait they have to pay for it yes okay you know everything's gonna

be wonderful we're all G to save time and money which is true and so in supporting the admins it makes the whole world open up and all of my crazy questions and all the questions that you have and like can a sender an automation help with this and help with that and do and you know automate all these different things the answer is go try it and yes and we're happy to help so if that is something that is inspiring to you myself jib Zane all the rest of us at ciq would love to chat with you more about what that admin model means it's

kind of new and different and we kind of have to explain it a little bit in many different ways to people to really kind of like comprehend and understand the value that we're giving there so we're very excited about that and we'd love to chat with you so get a hold of us absolutely we do have one more question Greg from Wally he actually ask are livert snapshots as good as a VMware snapshot I'm the wrong person to ask that question to trying to think I don't remember ever taking a liver snapshot because I usually just blow them away and start over I would assume

the answer is yes while and I also know the second part of his question is did your playbooks do the snapshotting the answer is yes you had a snapshot Playbook as a part of the workflow yeah I have a A playbook that uh does all the portions of the snapshot and uh it uh whether it's created it restore it delete it and it just depends on what variable you pass in what you want it to do so make them as flexible as you possibly can that's usually what I go for so you could technically decide which environment it's running in decide if it's a VMware

cluster you could do a VM snapshot or you could do a lib bird if it was running on a leged Linux machine yeah easy peasy KVM I don't know why I can't get KVM out sorry or if you're using a system like proxmox that's a open source one that a lot of folks use coming up a lot too yeah it'll work just as well yeah I used to be a big user proximas big fan nice again seeing a big upd in that one also SE a lot of questions around over it again something must be going on I wonder what I don't know an open

stack open Stack's making a comeback shockingly speaking of all the amazing different uh companies out there I just wanted to pop in to and say that we're going to be at sc23 this year c i Q we have a booth it's 1855 we're going to be in Denver Colorado so come and see us um we're also happy if you're going to be there in person we'd love to see you in person meet you in person and um yeah you can reach out to any of us just via our website or on the chat and get more information about that and how to schedule a meeting

with us sorry absolutely pop that in I just remembered as you guys were like talking about all these different companies and services and things great it's like heck yeah all right so I think we're complete for the day do you have anything else that you want to add Zane or Greg I don't I appreciate it Greg awesome appreciate it I know thanks for for showing up even though you're not feeling a 100% And so drink tea drink plenty of water hyd mud water please don't go plug mud water not I did fresh clear revitalizing Le water worri all good in the hood all right you

guys well thanks for watching uh update your systems and if you need to migrate gently using our our our um our you know beautiful Ascender plus leap Services we' happy to chat with you make sure that you like that you subscribe that you reach out to us and we hope that you have a really wonderful day we'll be back here next week same time same place bye by thanks everyone bye [Music] thanks