Working Rocky Linux Migration Best Practices

from the head nodes so how's that going what does that look like i mean can you kind of help us from a best practices standpoint yeah so essentially in any high performance computing cluster you're going to have more or less two different types of node there are other things that you can look at um some places you know have dedicated nodes for data movement dedicated nodes for you know things like grafana that kind of stuff but primarily the two things that you're going to be looking at that you're going to have in most hpc clusters are going to be head node or a head node

or multiple head nodes and the compute nodes that are going to actually run your computations zane as you alluded to there the compute nodes are typically pretty simple to be able to update from this um compute nodes are typically just uh you know kind of nodes that receive their configuration through something like ipixie booting um so typically your head node will have some type of provisioner something like that on it that i will kind of discuss a little bit more in depth in a bit um typically your head node will have some type of provisionary something like that on it that essentially allows you to

all at once affect what's going on with the configurations of those compute nodes essentially meaning that because those compute nodes don't have anything typically stored on them they're basically just for providing resources for computational jobs those when you're migrating those from centos to rocky linux those are typically a pretty simple operation a lot of places probably already have a system image or something like that that is spooled up that is kind of their generic compute node image that is served out to compute nodes when those need to be reset or something like that so migrating your compute nodes on a cluster is basically as simple

as kind of changing those images that you're using for those different compute nodes there i know that a lot of places have different types of compute nodes you know potentially have ones that are just cpus potentially ones that have gpus on them ones that maybe have a lot of ram for high memory jobs um and so there's you know obviously different considerations there you'll have to make sure that you know your gpu drivers and stuff like that are all set up correctly for rocky but uh in general um the compute nodes themselves are pretty easy to kind of reconfigure and to be able to serve

out kind of a new operating system and new stuff too zayn as you kind of alluded to there it's where you start to get into head nodes that things get a little bit more specific and for anyone not aware the head node of a cluster is essentially the controller node for it a lot of clusters will essentially have one head node where you're running things like your job scheduler your database services all that kind of thing um sometimes you'll see configurations where there's two head nodes one of them acting as a failover for the other so that if the first head node goes down um

it can fail over and immediately pick back up on that other head node um sometimes you'll see two head nodes to kind of run you know one's running uh you know for example i'll take the scheduler slurm as an example sometimes you have one head node that's running um you know slam service on it and another head node that's got your cirm database servers and stuff like that on it um so headnotes can be kind of an interesting configuration they're definitely you know very specific to each site depending upon kind of how you've configured things um there's a lot of different kind of considerations there

if you're migrating your head node from centos to rocky first off you'll want to look at you know you know obviously backups and things like that um making sure that you know your slurm databases are all backed up your um you know just gen general databases for things maybe like your um you know monitoring stacks and stuff like that are all backed up um it's a pretty specific configuration so it's something that's best planned out uh you know kind of in advance uh because it's you know kind of a drastic change to move a head node from one operating system to the other um so

in general uh my comments on that is like i said plan that well in advance make sure that all the data you absolutely need in order to rebuild that server is ready and kind of stored away um you also make sure that you fully understood you know where things are stored where you know slurm is moving things where like i said your monitoring stacks could be putting things where your databases are putting things um there's a lot of sensitive information uh that you know needs to be preserved through that um and wouldn't want anything to happen on migration um so as far like i said

as head nodes go they can be kind of a lot more of a complex case um they can almost be kind of down to the um server itself like i said every cluster is a little different um there's some commonalities in the software and stuff like that but um as far as having those head nodes go that's something that's really um best planned well in advance and um it's best to make sure that everything that needs to be understood that that head node is doing um is understood and uh made sure that that will continue when it's moved on over um there won't be you

know any kind of interruptions in service there because of the migration so force have you seen have you seen challenges in that in those air gap environments where you're trying to install rocky how are you going about dealing with those or you're going to have to pull down an iso or you need to get the packages that are outside does that become a problem or is that something that you worked around pretty easy um do you mean an air gapped environment in general or kind of an air gap high performance computing environment either one high performance seems to be the place that we see that

the most often it's used to enterprises have a little more uh flexibility when they can connect things to the internet versus the hpc environment that they seem to be a lot more secure so is that become a problem is it something you're running into um i myself haven't seen that too much um i know that for example in clusters it's a pretty common configuration for your compute nodes to not have internet access at all um so obviously from the compute node standpoint uh you know serving out new images and stuff like that to them will pretty much uh you know involve an offline operation anyway

um but with your uh you know head node potentially that's an air gap environment that becomes a little bit more complex um while i haven't quite migrated something in that environment i would imagine that there are pretty easy ways to get those files that you need from you know rockies different uh you know places the iso stuff like that out there um throw those onto you know just kind of the standard boot drive or something like that and be able to move that stuff onto there so my best practice there is probably just standardly what you would do um just uh yeah moving that stuff

over making sure you have those offline copies of repos and stuff perfect thank you i think neil has been able to join as well so welcome neil hey thanks for having me absolutely so just to add on counter to foreign instance that we're using uh rocky's what supports private mirrors so if any organization wants to have their a private mirror that only they're using and redirect all of their hosts to use that automatically that's something they can go and sign up on on the mirror manager to to subscribe a private year or two thank you yeah so i think there was one thing that you

were you brought up to me yesterday that you wanted to update everybody on uh around the script the migration skip yeah so uh yesterday or a few days ago now i guess on january 31st centos had in accordance with their end of life policy removed all of the centos eight content from their mirrors and uh i've been seeing it cause some some people some problems in their ci environments and elsewhere where they're expecting those images and um files and art repos to be available and uh it actually affected the rocket migration script as well because we were making an assumption there uh but we did

have a little bit of foresight there and pulled down a copy of the centos vault to some of our servers and have switched over that script to update your system to the latest centos version using the copy of centos 8.5.2111 that we pulled down and just orchestrate the whole update and migration process from there and it's working pretty well so far so that was a great addition that one of our community members added to the script yeah so if you downloaded that script before like yesterday make sure you go get a new copy of it because there are changes in it um you can go

back and look and see what the differences are and add it yourself or just go download the new one and make it work for you so for sure absolutely appreciate it neil so neil we were talking about migrations and i i've done a lot of cloud migrations i've done a lot of data center migrations over the last 20 years uh it seems to be something that most of the enterprises in the last 10 years have kind of figured out how to go about doing that so they've got their operating systems they've got their standards when it comes time to move they're a little bit further

along or more mature but if you look at small and medium businesses a lot of times they just don't have the resources they don't have the time i mean obviously you run into constraints and you have an operating system that you've built out uh small business i was talking to for example they've got a closet full of gear they don't really have a full-time i.t person they're actually downsizing their their environment and that rack has to go somewhere so it is old they're looking at moving those centos servers to something and they're trying to figure out what that looks like and if they look at

moving to the cloud they want to move to rocky but they see that as that's not something we've built before that's actually built by someone else kind of uh what do we do how do we go about doing this and i think there's a lot of a lot of things that are missed over time as small and medium businesses grow they're not the time to use so the configuration management is missing um automated deployments of applications are missing maybe they've gotten just detected things have gotten really old and it becomes a problem so as a rocky community what can we do to help with that

or what does that look like if you're going to do something like that yeah it's a great question i think that you know the proliferation of the cloud over the past 10 years has just been something that has really affected anyone and to your point with with a small amount of servers like they're you you do need sort of a critical mass for it to make sense for your your business to move over to a configuration management standpoint where you're not just configuring things by hand it requires specific knowledge on different tools that are being used whether it's ansible or puppet or chef or any

of those sorts of things you have to have the knowledge there to make it work and often there's a pretty big learning curve on making these tools work for you instead of kind of against you i faced that myself while learning a lot of them and it can be the best practices kind of around how to use them how to ensure that if your server goes down and you need to rebuild it you can just do that uh it it's tough and it uh it takes a lot of time too and especially for small and medium businesses there's there's likely not a lot of time

that that person has that's not a full-time ip person to be redeploying vms and make sure that they are working and to check their whole disaster recovery plan if that's even something that they have and so i think the cloud is something that's very useful but you have to be kind of careful with it too because especially transitioning out of that capex opex model which you can talk to your finance team and see if they believe you that the cloud is uh is opex but the the whole idea kind of of using the cloud is that it will save you money and and unfortunately in

a lot of the clouds there are just ways that you can spend way more money than you have to because it's easier to click those boxes so i in some ways there are different problems that are the same that you have to solve in the cloud for making sure that you're architecting your cloud environment properly and not over spending on cloud not spending 50k when you could be spending 5k a month for example and those are not easy things for small and medium business to do for the same reason that configuration management isn't easy for uh small and medium businesses to do they require domain

knowledge of that specific tool whether you're talking about aws or google cloud platform azure um they just you have to know the ins and outs of them and it can be scary uh to try and look at that especially when you don't know how much you're going to be spending and everything else but the good news is that rocky is available on pretty much every major cloud provider as a free image that you can just go down and pull and deploy that out into you know on aws it'd be a bbc and if you have a web application or something running that you need your

users to connect to a lot of the clouds really are adopting this zero trust network access um format to access all your services um remotely in a secure fashion and that also comes down to the architecture of the the programs and applications that you're running in themselves and to their credit aws gcp azure they have a lot of great documentation on how to architect those things in a cloud environment so again you have to take that with a little bit of grain and salt grain and salt because they may be the best recommendations but they might not be the cheapest recommendations in some cases you

you may need to go and look at uh different options that maybe you have to architect the high availability or disaster recovery yourself but save a lot of money in the end and that's where these sorts of tools really come into play i think sure so i think one of the other things that that becomes a challenge and a problem is a lot of times whenever you've licensed an operating system to run on-prem it doesn't necessarily transfer into the cloud so you can have some licensing issues or just because you have a license with someone or a subscription with someone doesn't mean you can run

the crowd cloud and still get support and i think that's something that that we've done a little bit differently uh being able to support the community it doesn't matter where you run it or where you installed it from but that really gives us a little bit of a different view uh love to hear your feedback on that neil forest kind of yeah licensing is i think a constant problem for for everyone i've seen it myself in ci environments not being able to use rel for example because it's uh it's hard to subscribe the systems in a convenient manner when you're running jaws multiple times a

day and so red hat has added some stuff like some content access to to help with that but um really the what i like about rocky is being able to use it as a uh parallel or an analog tyrell and run those jobs in my ci environments that i need to and uh not have to worry about the licensing in that way so whenever you deploy from the marketplace in any of the big providers amazon azure google that image that you deploy from the from the marketplace we would support it doesn't matter it doesn't matter yeah yeah there's you know it's not like you have

to go and subscribe to system or whatever else um yeah perfect so for my for my consulting days i know it was always a challenge of finding the time to actually do those configuration management pieces and to go through and make the effort of automating deployments so that you could move if you had to it always came down to i have to do something now and now is the time to invest the time but getting someone to actually spend the time to do that automation was always hard and i think it became a conversation of but if you don't do it now you're gonna run

to the same thing in two or three years and you're gonna have to invest a ton of time and money so i think if you're going to make this move now is the time to start doing that configuration management and i think it's important to realize that all of the the tools that are out there all of the tools but there are a lot of the tools the big tools that are out there today are supported on rocky correct i mean if it's puppet chef salt ansible all of those work on rocky today yeah and not only do they work on rocky today but it's

it should be an experience that you can simply redeploy the server using rocky and use that same configuration management because it is a one-to-one compatible red added veriz linux build which is what centos was as well so uh you know there may be a couple of tweaks that you need to do and there's there's definitely some pitfalls which we should definitely put an article together on to make sure like the original version is updated and that kind of stuff but um the the rocky community has done a great job of upstreaming those changes into the different community configuration management tools that the rocky users are

using so they have a vested interest as well in making sure that chef puppet ansible are working properly on the os as well so a lot of that stuff is fixed at the tooling level and thankfully a lot of it's already been fixed at this point excellent so i think one of the other questions that we've seen quite a bit today or up to today has been what is going to be different and what's going to keep rocky from going away like what happened with centos i run into that you know you're not going to call day before yesterday where that was a very interesting

topic that was a lot of passion behind it and yeah i think it's important that we describe that so if you don't mind yeah no absolutely i'd love to um i think that is something that everyone is still very uh sensitive to of the subject because of the change last year and you know i looking back at it a year later i feel personally that it it wasn't the best thing to do and i think that there's a lot of you know other people out there including folks from from red hat themselves that admit that the change midstream of centos 8 2 centos stream which

was a terrible sentence uh is was not the best move and that they wish they had been done different and i think that's kind of where i see it now too right like it i wish it would have been done different um for the change for centos uh to send to a stream um but in terms of what happened to sento centos being eol and change the stream i'm i'm a bit on the other side of that i i'm i'm really excited about the changes that stream has brought and will enable us to help the enterprise in this community and submit things and contribute upstream

to those um jobs that are working and in terms of it going away or being bought i i enjoy the structure that we as a community have come up with for rocky linux in that it's uh it is very much led by the team um and there's a nice team of uh people that are providing guidance and uh and expertise on how to set up the governance of that project and make sure that something like red hat something like what happened with centos doesn't happen again for rocky and that's something i think that all of us on the team are very committed before so neil

i think it's also another even you bring up an interesting point when you talk about sense stream and kind of what that is i don't know that everybody has a great sense of what what that actually is and how that gets contributed back to i think there's a lot of misunderstanding of is it something you contribute to is it something you can't what does that go to fedora first like what does that flow look like and how do you actually contribute back to stream so historically the the flow has been kind of uh you spray something at fedora and hope that it makes it into

a rail release and that was never really something that was helpful for enterprises or people that wanted bugs fixed or wanted to get their their code contributed up into things it was often a multi-year process to to be able to have those changes uh integrated uh what what stream has done is kind of flip that around on its head and there's a little bit of nuance here with the centos stream eight and centos stream nine because the the contribution model just because of the change midstream is a little bit different so the way i can describe it and uh there's a great tweet by uh

carl george from from red hat about explaining this uh the process here but centos stream 8 is technically a rebuild of realm technically centos stream 9 flips that around and it becomes that red hat enterprise linux is technically a rebuild of centos stream nine so the contribution models are a little bit awkward for eight but for nine there's a very clear path for making contributions to the the centos github for centos stream or gitlab sorry and creating tickets or merge requests to have those merge backed in and in centos 8 stream it's a little bit different you have to kind of open some bugzilla requests

but generally the contribution process is the same and it allows us as engineers that are working on problems in rocky linux that we see customers using to upstream those changes into centos so that they'll become up they'll become packages that are just defaultedly available or fixes that are defaultly available in the the upcoming release of whatever that operating system is so uh in terms of the contribution it's a little bit finicky and you got to wrap your head around it a little bit but it's it's something that i am genuinely excited about to be able to to shape the future of enterprise linux excellent so

how do you see that playing out with what do you see happening to fedora where does that you don't have to go all the way to the top anymore do you see going back up and going down i mean how is that going to look so ultimately yeah that's that's a great point like you have to upstream those changes again from from centos stream into fedora because centos stream itself is a cut of fedora and that's a great point it's the release cycle i believe i'm not even going to say it because i'm going to say iran but the release cycle of fedora centos stream

rel and then the derivatives thereof is very clearly defined now where it it it's not that it wasn't before but it was maybe not as public as as uh most people would be able to see and so there's that path knowing where a centos stream nine rel nine build is cut from from fedora whether it's you know 28 which was for centos 8 or 34 for fedora 34 which is going to be or is upstream of uh red hat enterprise linux 9.

those those paths to contribute are fairly well established excellent so do you see from the rocky standpoint the rocky community i mean it's it's massive already for the asia it is i mean having 15 000 plus people in that community it still blows my mind but but enabling them to quickly give back to sense stream at an enterprise level and not have to go up all the way through what kind of impact do you think that's going to make on on enterprise linux and kind of how that's going to shape rocky i think it's a that's a really good question and in a lot of

ways sent to us stream the changes there were shaped by uh hyperscalers like facebook that are were looking to get their changes integrated faster into um interred hat that they were using and or sent to us that they were using and didn't want to or didn't have the time to wait for those changes to be pulled down all the way from uh from fedora down into rel eventually and so the it's exciting that there's a lot of work happening in centos stream not only by community members but by developers as well and i think it's a really awesome opportunity for the rocky folks to contribute

that to contribute to that if they want to um but also have an area inside rocky linux to contribute things that are specific to them that not that they don't want to bring up stream but that are are more specific that have to be uh held inside rocky and i think that's that's something that we'll be seeing more of this year as we deploy a new build system over there excellent you touch on hyperscalers for a moment um and i'll jump in really quickly about kind of the hpc side of things um one of the big focuses uh one of the big you know kind

of consumers of rocky linux at the moment that we're starting to see is super computing sites um it's very well established in hpc that one of the kind of standard operating systems that's used across clusters uh was centos 7 for a very long time there's a lot of clusters out there that were running it for their head nodes compute nodes um a quick you know cursory glance over the top 500 i'll show you tons and tons of systems on there that are using it and that in a high performance computing environment you know typically you have hundreds of users on at any one given time

trying to you know share the resources that are available across the cluster so it's all so with that you know job schedulers and databases and all that it's already a sensitive enough environment and it's difficult enough to get those environments kind of ready for a maintenance period um that having to move from os to os in your cluster is not desirable that takes a lot of time that takes a lot of effort that takes um lots of planning to be able to do something that major with all of your high performance computing architecture and so for one the move from scent 7 to scent 8

was already something that you know i don't think a lot of sites were really starting to think about um because sense 7 is pretty like i said pretty well established as a reliable and solid um hpc operating system and with the end of life for that being still a ways out there was really a lot of reason to switch over there but what i think is great about rocky linux is that now that that sent eight that clear upgrade path for some of these high performance computing sites is no longer so clear um with centos 8 being end of life i think that one of

the great things about rocky is that it provides that necessary alternative for a lot of these high performance computing sites that don't want to add the complexity of having to manage red hat licenses and you know across potentially dozens hundreds thousands of nodes at once what rocky provides is not only um you know a free alternative to that um but a way to avoid having to like i said go into all that kind of specific met licensure management stuff like that play um essentially management that places haven't had to do before but that would add a lot of complexity to their operations to have to

you know scale out and manage and so rocky really provides a great upgrade path for anyone in high performance computing that's looking to move off of you know centos or even red hat if you're looking for something that doesn't have that type of management um i know that you know managing software licenses for all these specific pieces of high performance computing software is already difficult enough so yeah one of the big draws of rocky i think to the high performance computing community is that it provides that community alternative that was uh a lot of places to rely on and have relied on for a very

long time um to run their high performance computing clusters on the first in those types of environments going through that i mean a lot of that stuff's been around for a while they get very solid very stable performance so they just don't touch them they leave them alone for long periods of time it's been around kind of like some of the smaller enterprises or even enterprise where you have legacy tech that and things that you just don't know how to rebuild anymore are you seeing people start to adopt more configuration management automated deployment type infrastructure or type processes into that environment or is it still

trying to catch up um honestly it kind of depends upon the site that you're at um you know we talk about small to medium businesses and kind of how they uh you know are working with configuration management and automation the first thing i think of is you know small to medium high performance computing sites because it's a very similar thing um there are a lot of places out there that are basically just running a couple of clusters you know there's the universities for example that are kind of running you know a couple of smaller clusters that are meant to serve out some resources to their

users at their um institution but uh you know that don't have a lot of people or you know a lot of cases a lot of time to be able to you know look into that kind of configuration and automation type of stuff so i think that especially as we kind of see these things move into the cloud we kind of see a greater proliferation of these type of tools across hpc high performance computing has traditionally not been a field that the absolute latest technology gets into very quickly a good example of that is containers which for a long time until there's um until singularity was

kind of built now obtainer of course but originally singularity you know for example there wasn't really a good way to deploy containers in a high performance computing environment that suffered from a number of security pitfalls and kind of other issues with that specific multi-user environment and so we see kind of a pattern in high performance computing as i said of these technologies slowly making their way into that and i think that especially with kind of the move of a lot of sites to wanting to more manage cloud resources as opposed to actual very large on-prem racks of hpc servers i think we're seeing kind of

a wider proliferation of those tools coming about as a lot of these hpc people start to realize that you know these um provisioning tools and being able to scale up um let me think as you know being able to scale up automatically and load balance and things like that become you know needs as our uh you know science advanced and our simulations become more intense and we want to you know run for example you know just more just an example of a use case you know we want to run more gpus on our machine learning models um you know as we see the scales get

greater and greater on things i think that we'll start to see more and more people move to the cloud and kind of start to look into those automation and kind of configuration management tools because just the necessity uh you know just where the technology and the science is going will necessitate larger clusters and more power to be able to um you know be easily put to the task at once even just for smaller sites just like i said do the nature of how things are advancing um so yeah i think that we'll definitely see more of that type of usage um as it kind of

starts to pick up here over the next couple of years but there are definitely some places out there that don't have that infrastructure and don't have the time to set up that infrastructure yet um but like i said as there's more demand for the cloud that'll kind of hopefully even help to ease some of those tasks that go along with management of high performance computing clusters on-prem um and you'll free up time to explore those technologies and maybe see greater proliferation from kind of across the hpc community of that stuff into hpc i think something that uh just to tack on there like coming from

the enterprise space you can wait especially with a pandemic now like years to get servers and switches and then so there's there's definitely a uh cost benefit analysis that you can do there for for how much you're spending not only for your power and space and internet usage and everything else but uh how much time you're spending on on human resources making sure that everything's configured and patching all those systems and the switches and everything else and uh one thing the cloud does especially for those the smaller side of things that maybe don't don't haven't reached that critical mass to have a you know team

of people that are responsible for just you know the networking for those servers or whatever um one of the things that the cloud offers is you don't need those people and that's not to say that those people aren't necessary because they are they're they're just in a lot of ways needed in these cloud environments and i can go back and forth and probably spend a whole hour talking about the the centralization of everything to these clouds and you know you see outages too so there i think there's there's a lot of things you have to consider when you're moving to the cloud it's not necessarily

as simple as i got 10 servers and that's going to be this many cores on some ec2 instances you have to kind of think about your disaster recovery and especially with the cloud i think is a really great time to think about that configuration management and to take the time uh you know as you said there's still um a few years left on the centos 70ol take that time to plan your migration think about how you can do these things in a an ephemeral manner where it doesn't really matter too much if your server just falls off the roof or if us east one goes

down yeah i think that's something that we don't i mean from a consulting background i we ran across people who didn't take any of that into account they would think just moving a server to the cloud made them where they would never have an outage or a problem and they would also just leave them on all the time and i think that's where people run the cost problem is not actually identifying does this resource need to be on 24 hours a day or does it not need to be and being able to turn things off and doing it in an automated way you can save

a lot of money if if you identify those different types of use cases and workflows that you can you can do that i think it's important and i think it's interesting for us that we start looking at hpc in a hybrid model where you're having not just that on-prem hpc environment but being able to scale into cloud that's new and exciting different and as you both touched on there um you know there are definitely a lot of complexities to setting up these things in the cloud you know it's definitely uh you know it's already a complex enough task to set up you know an on-prem

supercomputer if you're you know standing up uh high-speed networks like infiniband you're standing up uh you know management networks there's all kinds of different considerations that you have to take into when you're building something on-prem um and to be frank it's the same way in the cloud you know i know that some of the major cloud providers have different types of you know different kind of hpc stacks that you can deploy on them um i can't speak to the maturity of them i haven't like deployed some of aws's different kind of massive gpu deployments but there's definitely a lot of complexities that are involved there

that are very similar to what goes on on prem um it's you know setting up resource managers databases for those resource managers setting up um you know the mpi stacks that allow these instances on these clouds to be able to you know talk to each other figuring out what even instances can support you know the high performance networking that's necessary to make a compute cluster work so there's a lot of considerations that go into this a lot of it translates pretty simply like i said um you know you're already probably used to working with infiniband networks in your on-prem hpc um there are definitely ways

to get those in the cloud providers um but it's definitely not as simple as just spinning up you know some large compute node like instances and you know starting to run stuff on them there's definitely a lot of complexity and a lot of you know as we talked about configuration there that can be automated and um you know should be automated for the ease of people's time and you know getting things up and running quickly and some of that you know obviously kind of comes from the ability to scale and stuff like that there are specific you know configurations that are useful in a cloud

environment that even beyond just a you know simple cloud cluster would be useful like i said a good example is auto scaling a user submits a job to your cluster that's far bigger than you have capacity for being able to automatically scale up the nodes there you know that you're running in your cloud cluster is obviously something that you can't easily do on-prem and so there's a lot of complexities there that uh have to be kind of taken care of as well in similar ways to how they would be on-prem and the hype and the hybrid cloud model is very interesting because we kind of

see that coming up a lot um you know as institutions and different hpc sites have these very large clusters that they've expended huge amounts of you know time and money into over the years that's definitely a big concern is being able to still take advantage of those resources um and you'll not just have to you know essentially waste money in the end by mothballing them um so there's definitely a big need for kind of hybrid cloud solutions in high performance computing especially those that could say do something um you know be able to serve things out to clusters in the cloud clusters on-prem at once

um there's a need for that kind of thing at the moment so um yeah as i said as things kind of move to the cloud in general what kind of things are is the hpc world looking at for to solve some of those inherent data gravity problems that come from a hybrid cloud model you know where you have maybe petabytes even of data that you you want to run on uh workload in aws and as well as on-prem or maybe gcp or something do we have good solutions i'm not sure quite yet and that you know this is kind of the error that we're starting

to get into of big data i mean we have um you know it's really only been within the last five to ten years that we've seen you know mass proliferation of for example one of the biggest consumers of big data machine learning um so i think that in general there's a lot of kind of solutions that are out there for that type of data warehousing and that um large-scale storage of data but as far as moving them between the cloud and um kind of these on-prem stacks where there may be you know high-performance computing clusters that have large petabytes with the storage attached to them

that obviously can't be easily moved up into the cloud um i i am not 100 sure exactly uh you know what solutions are being uh explored there but i imagine that we're seeing a similar proliferation um you know with how we're seeing you know all the specialized hardware that's coming out to process this big data that we have you know different accelerators and things like that um so i think that if it hasn't already started we'll see a very similar proliferation in the big data sphere kind of like for example as i mentioned what we're seeing with all these different specialized hardware accelerators and things

like that coming out because uh you know perhaps before we weren't as concerned with the data because we were running you know small models we were running things that weren't difficult to move around um but definitely now that things are recently starting to get into you know petabytes and petabytes of data that one you know model might need like i said use the machine learning model example um they'll definitely need to be tools there that allow for that kind of easy movement of data between cloud and on-prem there's definitely cost to be concerned about there too absolutely that's why i'm wondering if sorry force go

ahead i was just gonna say because you know obviously that type of um ingress and egress is pretty difficult um you know even if you're using some of the um you know kind of archival storage solutions i mean those have pretty high egress costs to them so i think a lot of it is just kind of figuring out at your organization what your budget model is and what you know fits that if you want to you know you know if you're using if you're going to be using the cloud extensively do you want to spend the money that it's going to take to move all

of that you know those petabytes of data and store them in you know s3 or something like that so you can ingress them quickly into your aws cluster i think in the end it kind of comes down to what solution works best with your organization and what works best with the budget because there's multiple different ways of going about it you know even like per cloud there's multiple different storage solutions so i think it comes down to what you think your overall cloud utilization is going to be how much are you going to have to deal with that data gravity over time um i think

that could kind of guide a little bit of the decisions on where you move that data off of these on-prem systems and into uh into the cloud and i started thinking about the way that you i mean whenever people started using cloud and being able to actually direct connect a network connection from your local environment into the cloud so that you had that transparency i wonder if it's going to become that way with data as well where you'll i mean in some places you'll just physically have your personal infrastructure sitting right next to the cloud infrastructure and some sort of connection between the two i

mean for speed and scale it seems like that would be the next progression if it hasn't already happened mike probably has but seems like that would be kind of where we're headed there's a lot of people in ashburn great conversation guys i think the last thing that we wanted to talk about was pk exec i know that's something that's kind of been hot on the internet lately and everybody's talking about pk executing the vulnerabilities so i think now's a good time neil if you would you'd like to chime in on that yeah absolutely yeah that was a kind of kind of weird bug that's been

sitting around there in the open source world for 12 15 years 12 13 years yeah uh interesting i think to say the the least about that and to look at the changes that have been made and discussions that have been having in the open source community in the linux kernel uh for example there's a patch upstream to uh i don't know if it's been accepted yet but to actually fix this at the xve level where it will just error out if you try to supply these things um and there's you know some considerations there too i think that we're in for a bit of a

rough couple of months here as people find more uh suid libraries that are going to be affected and for those people that don't know there was a vulnerability uh last week in pulkit which provides it provides a layer between the the operating system and um how things are supposed to be executed basically so there's this function called pk exec which was uh able to be exploited to send it some bad environment information and essentially any user could compile a script of clc program and just get root and so it kind of points towards the importance of having a strong security model and and also it

lends itself really towards things like rootless containers where you can't actually escape those even if you try really hard um and so i think that that is something that uh greg picked up on and he that day he was asking me hey the script isn't working and why isn't it working on rocky the one that the exploit script that was all over wasn't working uh it turns out it was just kind of a gcc flag that needed to be set on it but uh he went through and ran that on i think he has a youtube video on it um on app tuner uh formula

singularity uh and showing that the model there with the with the zip files and how the container run time is orchestrated uh it doesn't allow people to escape from that shell even with that exclamation so that was uh that was pretty neat to see i thought especially with um you know kind of the vast proliferation of containers at the same time across everything um you know a lot of these uh container runtimes are not exactly built with the similar security model that's unique to apptaner and so we were pretty pleased to see uh you know that being able to take care of that and actually

stop that exploit because of the security features that obtainer just has built inherently into it um in its run time and in how containers are executed and how users are able to interact with those yeah i think that's a great point and you know there are there are secure more secure distributions out there like alpine linux for example wasn't affected by this at all because they don't use glibc they use musil as their compiler and our interpreter so you know there's definitely other things out there that are possible and alpine has container images that are great i use them myself um but having a container

runtime that is really thoughtful and insecure and uh is something that i think is really powerful and important absolutely absolutely and that's not just me that readies and everything else aren't good getting there and working on it but uh i don't know it should be noted that obtainer was specifically built for those type of secure use cases um this is a really common type of thing that you would see in high performance computing that need to have that level of isolation between users inside of the container um or the need to have my apologies that kind of privilege isolation um there are other ways that

apptaner works to more kind of integrate than other runtimes the capabilities of the host and the container and there are high performance computing reasons for that um but yes it's it was very neat to see obtainer able to just stop that wholesale it's excellent well guys we we've reached the end of our time uh you'll be respectful of everybody's time and i appreciate neil boris for coming on uh thanks for having always always wanted to talk to you guys i appreciate your time and looking forward to uh part three of our conversations here in the next couple of weeks so thank you again go like

subscribe follow us follow along with us on this we appreciate it smash that like button yes thank you zane and thank you for having us thanks everyone for watching thanks guys everyone