Rocky Linux HOWTO Series: Building a Rocky RPM package

and then it would work but maybe it wouldn't work for somebody else and that's that's no good and mock um it kind of gets rid of that idea it's very nice so what we do now instead of hey saying Zane's going to build his RPM on his system or Jonathan's going to build his RPM on his system what mock does is it will create a a Siege route or container it does both um but the idea is to to spin up very quickly a very minimal Rocky Linux or other Linux system within your current box right and then run the build inside that minimal system

using just things that are available in there the build can't see anything else and produce the results collect the logs and you're done and it's actually it sounds simple it sounds stupid but it's actually way easier than setting up RPM build and doing all of this stuff yourself and we're going to demo it we're going to show off uh we're gonna do a little build of Our Own and I think the one of the first things I need to talk about when I talk about RPM packages is where do they come from exactly because a lot of people don't you know when you do your

dnf install or whatever under your Linux box you're still using your gum command I understand love that muscle memory um the RPMs effectively are compiled from Source RPMs Source RPMs come from code effect you know if there are there's three components to a source RPM we have a usually a zip file or tarball of some kind with source code in it so we're going to talk about bash we're actually going to go pile The Bash comp or The Bash shell and everybody knows and loves well most of us love and um so in inside and we're gonna you know what we're gonna demo it uh

if you don't mind right now I'm gonna can I present here can I can I share my screen certainly because it's easier to show you than to than to tell you about this so share screen I'm going to click on this one and here I've got a rocky VM running but we're not going to use that just yet we are going to move this over and we're actually gonna go to one of my most visited sites the raw Rocky Linux download pages and I just happen to know so you can you can browse these These are also added automatically in your dnf big files you

can go to Etsy yum.repost.d show you that in a minute but we're going to go into here and uh Rocky Linux just like just about every Linux distributes Source packages with their binaries right so there are here so aha Ash this is the bash Source package and we're going to look at it real quick because I just want to show people so there's three components to a source RPM there's a spec file looks like this bring it over here it's a recipe for doing a build right it's got you know the name the version things that it depends on make it bigger for everybody we're

not going to get deep into this right now though it's it's you know a little complicated there's a lot of documentation about it but this is just a how do we build this you know what commands do we run whether it's a make and a GCC or whether it's a python thing or you know whatever right and it's got requirements here that we need in order to build this things that it provides things that the package requires that sort of stuff and it's got you'll see bash has quite a few of them it has the second component of the source RPM is patches and you'll

see these are all the patches most of these are issued by Red Hat because our Upstream provider is red hat from our code and this is just stuff that is added on to the main bash source code that are oftentimes they're back pores they fix the little bugs security issues maybe they change Behavior to a way that's more conducive to Enterprise Linux whatever right and then the final component is as I said the zip file or the tarball and that's in here and this is literally if you go to um I think it's Savannah is it gnu Savannah I'm not sure the Upstream it's actually

probably in here we look at uh Source zero aha so this is where the Upstream Source was got from so if you go to you can use FTP server to this location right this base version our base version is 5.1 it's this file right and you can you can book it you can they should match exactly and that's all we need to compile our Bash so skip if you don't have the source file will it pull it down from that URL automatically or it's a not no it has to be bundled so the The Source RPM it's expected to be able to compile without any

internet access right so we we kind of we want to make sure that we have everything um bundled together within our source our source RPM right and once we have the source RPM it's really as we see here kind of demystified it it's just a bunch of source code and a spec file to as a recipe to you know how to build that source code and we're going to take it and we're going to build it um and I will show that I think right now is that okay with you guys absolutely practical awesome awesome let's go let's go all right first thing we need

is a rocky Linux system doesn't necessarily have to be a right Linux system but as a member of the rocky links team you can imagine I have a bunch of VMS I've got a bunch of all kinds of things going on ready to go so got my thing running we're actually going to load up a terminal an SSH into it because it's going to be easier than using the VM screen but we're gonna that's the one we're going to be using and let's see if it lets me in here excellent okay and the first thing we're going to do this is very easy I think

we're going to use four commands here to set up our our Mock and the first one many people likely have already done this is um dnf install Apple release spell it right skip and Apple for those that are not familiar is an excellent collection it's a third-party repository stands for extra packages for Enterprise Linux has a bunch of stuff including this mock program that we're about to install um but it's another repository maintained by the Fedor project has a bunch of extra packages extra stuff you might find interesting compiled for Rocky Linux or red hat Enterprise Linux and now that we have our Apple Powers

enabled we can dnf install mock very um like I said four letters very simple simple package here it's written in Python so there's going to be a few python dependencies but um as you'll see here it's uh it's actually quite small surprisingly small for what it does um this package is you know less than 200 kilobytes here and it's got some support stuff that um it needs to install so we're going to do that go go a terrible time for any mirrors or anything to uh to fail yeah that would be uh that's Murphy's Law in action there okay cool so now we've got mock

here I'm just gonna yeah I'm not in the mock group excellent so we're going to do that too so we don't want to run maca's root I mean you can but I don't want to on the system it's not how I work in general you can run it as normal user you should users in the mock group are allowed to run mock field so we're going to do is we are going to add Skip that's me to the mock group here very simple his um and then we're going to re-log in because we want group membership to be updated and I will check and we

see aha now I'm a mock user I have build powers awesome the next thing we're going to do is we're going to scroll back over to our vaunted bash RPM Source RPM that we saw earlier we're going to grab it here and you guys can read this I don't know if I don't know how visible that is but that's good it's good go check it out and we're gonna we're gonna download let's go good old W get there we go now we've got a source file I'm gonna I'm gonna create a temporary folder to work in here just because I don't want to clutter anything

up with logs and now we have our source RPM we've got our mock build component right or our mock built program very simple um I'm going to time the build time is a command that you just you put I do it all the time for my builds because I enjoy seeing how fast the system works how fast compilation happens it's very useful to uh well for a lot of reasons but especially during builds very useful for benchmarking and so forth um V verbose we like verbose and let's see I'm going to say result or is home skip cup there we go that's the current directory

and then we're just going to say hey um at source RPM home skip RPM awesome and I'm going to press enter and go go and we're gonna while this is happening it's gonna take a few minutes we're going to talk about what's going on here so um and then we'll pause for some questions for some discussion because I don't want to talk forever of course um What's Happening Here is this the mock program is actually creating a very minimal Rocky Linux system right in my virtual machine here inside a CH root it's actually a container environment and it's installing you'll see there there's some packages

here that are really the bare minimum of things you need to get going like you know just so you can you can um you can have just so you can run dnf to install more packages effectively and once that step's done and it's going to complete soon here um this is going to look very familiar to anybody who's ever installed packages before then we're going to check out it's going to read that bash spec file that we looked at over here and it's going to say hey what what are the requirements that I need in order to build this package right and then it's going

to install those packages but only those packages nothing else and then it will proceed to read the it will proceed to call RPM build just like you would but inside this minimal system and perform a build and that way we only we're doing it in a place that only has the bare minimum needed to build bash and no extras no uh no optionals I have questions about this small environment yes yes the environment tell me more about this you said earlier a CH route or a container how how does this work what is it doing underneath is it really a container platform is there something

else I've got to do yeah so there's uh there's two options in mock the default uh which we're using right now is called um systemd in spawn container you can read all kinds of technical wonderful details about that it's a systemd feature that effectively allows you to create containers not with your traditional Docker or podman or whatever it's actually system D will talk directly to the kernel to c groups and create a container for you with storage and networking and everything and it's um it's very nice because this program you don't want to you could use Docker to run RPM build but we don't we

don't want that overhead right we don't need a whole Docker installed to do this and we're interested in just spinning up the bare minimum environment and then we're going to discard it at the end of this like this is all you know once we have our build and our resulting RPMs then we don't um we don't care anymore like we just you know the environment discarded and we'll wait for the you know the next time we call for a package spin up a new environment and um so yeah we keep things clean there is also an option if you don't like um in fact it's

actually very popular is some originally systemd in spawn is kind of a newer in Enterprise Linux terms a newer thing you can run it with just a simple CH root right where no containers at all just good old you know CH route developed in the 1980s still around wonderful tool and it will do the same thing the technical details are a little different right the the container and the CH root inside it they can they can see different aspects of the kernel sometimes that in when you get to Advanced builds that will affect things occasionally but here for bash it's a pretty simple package so

we're just going to uh we're just using the default okay and then whenever it lays this down we're talking about being clean and simple does it actually clean this up for you when it's finished or is that something else you've got to take care of after the fact is the environment yeah so by default it cleans it up um but you can there's actually uh and again if you get into mock building I encourage you to check out all the options because there's a million of them and you can say you know especially for troubleshooting hey when you're done do not clean this up right

go you know leave leave the directories lying around so that if something went wrong we can actually go into the CH root and check out like you know what was going on like what kind of files were produced right if there was a you know an error or a file that was missed or something we can explore that and while this is happening I know there's a lot of scary text there but it's it's and this is what happens when you compile Bash like I can show that we can because it's very uh um it's pretty straightforward which again is why I like this tool

like uh no mystery so all this is um in fact I'm going to become root for this because I believe I need to be rude to explore this and there's when you installed mock there's a mock here and we'll see there's a couple of folders we're going to go into the mock config your configuration for mock sets this we'll go into that in a minute and you can see here's root and aha aha this is if you look at my system here I'm inside here but this looks like a root directory this is where bash is being built even as we speak and I'll bet

if we go into Builder here we can yep there's all kinds of stuff going on inside this wonderful little uh lib directory in here and so we can if we wanted to we can we could explore the GCC you know what it's pulling as it happens I'm not going to get into that right now but um yeah this is this is the CH root or the container in this case I don't want to mix up my terms but yeah so if you want to explore it uh there's a mock there there's an option to mock I think it's called dash dash no clean something like

that don't quote me and um it'll say hey mock do not remove this stuff whenever you're done right keep it keep it around keep it intact so that skip or whoever can can look at it because something went wrong foreign skip we've talked a little bit about the benefit of of kind of listing all of your build requirements in that spec file um how normal is it for the some of the packages that come as this minimal base system to be omitted from that list or specified does the package make assumptions About That Base system and only Define requirements that would go on top of

it that's an excellent question and um yes so you're uh um what you're talking about the initial set of packages that was that First dnf Command that was run inside our mock thing that is what we call a build root so and it's it's there to basically um bootstrap a a real minimal Rocky system and you're absolutely right you're touching on um your your requirements and the um I'm trying to put this in a in a good way but um your RPM build requirements are only as sound as how minimal your build route is because if you throw say Java into your build root which

is a terrible idea don't do that and your your RPM actually requires Java but does not specify that then yeah it will have Java and it'll be a problem because somebody who doesn't have Java it's not a requirement will experience issues right and um it's a hard question to answer I think um the defaults for mock are pretty darn good is all I would say they are about about as minimal as you can get and still have a functioning system um there's if really if you take stuff away and it's it's really if you let's look at our log here and I'll show you what

I'm talking about because oh look at that fun bash RPM and you know I always love to test my RPMs so let's do something kind of dangerous here let's take uh this new RPM that I just compiled and overwrite we're going to just um we're gonna remove the current flash which I'm actually running right now and install the one we just produced and it should be just fine because I think it's the same version installing yep awesome Yep and so now we are running strictly speaking even though they're identical we have bash installed from the one we just built we just built our own uh

shell interpreter fantastic so yeah back to your question so let's check out really when the when the system begins and we're going to I'm going to make this a little prettier here root.log by the way is the um is mock setup process for creating the container right this is the stuff that it needs to install before compilation is actually done go down here we see that this this is the list of packages here right and really there's not a whole lot of you can't really take much out of here and still have something that's going to work if that makes any sense and so I

I think um you could add on to it if you're interested in that but again careful so I think every RPM package no matter what should expect these to be present like as as a you know you you can still specify that you can say hey I need grep or I need gzip it's good practice right but I think it's safe to say that just about every system you're going to encounter is going to have this stuff and its dependencies which you can see here installed hope that answers the question properly I don't know if I'm wandering here yeah maybe the the point of reference

is I I don't think I've ever seen for example a package say that it depends on or build depends on g-lib C and is it acceptable in that case to have there be just things you don't have to say in your spec file it really depends on what system you're building for so like Rocky Linux packages they come from you know we have a tightly controlled system that's based entirely on red hat Enterprise Linux and if you're running Red Hat Enterprise Linux you are running glibc right there's no ifs ands or buts and so it's not normally specified however you I can imagine a situation

via container or via something else crazy where you have an RPM based distribution that you know maybe has mucil libsy or something and that would need to be specified within your your package right so it might be a good practice in that case to you know it really depends on your RPM's uh intended audience I guess and tend to you know build distro because they're very different if that makes any sense yeah thanks we're very fortunate in Rocky Linux we have again Red Hat Enterprise Linux a great product we have a very high quality Upstream there's still problems because when you build you know 3

000 plus packages to build a distro there's it's not trivial there's all kinds of weird things that can crop up and need debugging but in general for most run-of-the-mill packages like this one they're they're pretty solid so do we have any other questions looks like we built it I want to actually we were you know I'm gonna go off on a little tangent here because we were talking about other distros and it's um it's important to note if you're not familiar with mock or or this process Mach is a fedora tool and it's very it's used a lot across the Spectrum right this is not

just a rocky Linux thing so if you go into slash Etc mock configuration directory very interesting here there are a lot as you can see of distros and the power of this tool is that remember when we create a CH root and begin our build process in there it doesn't have you know this is a rocky 9 system we just built Rocky 9 bash but it doesn't have to be that way we could specify hey we want to build you know Synthol 7 and takes you know A bash design for that and build that right and we could do it right here on the system

or a fedora one or an Open Source One or whatever it's uh the only limitation is that you have to have an art you have to be an RPM based distro yeah there are a lot in there yes there are there's there's an awful lot of derivatives and distributions and all kinds of things and I think we'll uh we'll explore a little bit um so you'll see the one that I was using is the default is Rocky nine it's default on this system because they kind of assume that if you you know the packages of mock say hey you're using Rocky 9 you probably want

to build Rocky 9 packages which is a fair assumption and you'll see this is really just a a python config file here it's it's a bit of python code if you're familiar with it this is a python dictionary and it's really cool how the the settings you know that's how the settings are set and let's check out this template file because this is the real interesting one here and we'll show you kind of um high in the curtain here with mock very interesting if you've ever edited yum or dnf config files you will be familiar with this right and it's it's fascinating because you have

to you have to remember when you're building an RPM package it's the the location that you grab its dependencies from matters a lot like it's hugely important and so it's really cool because in mock you can manipulate this to be whatever you want it to be right so right now like yeah this is this is pretty default but you can point this you know you can you can change the dnf uh priorities to say hey I want my own repository that I hold on to to I want you to get your GCC or your dependencies or whatever from there first and you can manipulate the

versions that you're building against right so let's say um common I wouldn't say common but issues we have buildings sometimes where like hey this Ruby package will not build because new openssl breaks it so let's build it against a slightly older open SSL at build time so that it compiles correctly right then there's all kinds of all kinds of fun things you can do to inject your own dependencies to change your build route um there's the sky's the limit here because you can literally you know this is you customizing your uh system that you build or that you build your RPM on so very cool

very cool stuff but easy to edit too like it's literally just some text so what's the process like for creating a new one of these good oh boy so it's um it's really not bad um you you uh I'll give you an example here and when you say new one you mean a new config file right well a new config file but also like you know you and the team uh just put out a whole new distro in a sense of Rocky 9 and not only is there not a config file for it but there's also not repositories for it to drop and it's true

you have to start somewhere right and um you can there's several ways to do it um fortunately for us because Rocky by its nature is a downstream distribution we have excellent upstreams in Red Hat Enterprise Linux 9 and uh cintas9 stream further Upstream and um so what we've done uh to get started is um we have bootstrapped rocky Linux 9 off of red hat 9 beta right which was released oh gosh I want to say it was like the December or January December last year January this year and you can see I'll uh kind of show it off here because I'm heavily involved in this

um I have a um a web server that was kind of um the project allows me to use here and we'll go into we'll look at the arch 64 here this is Remember by the way in in real life Rocky linuxland we have to do this for I think five different architectures we compile for the arm processors like Raspberry Pi we compile for uh 32-bit I'll explain that in a minute we compile for Intel 64-bit this is your default you know laptop desktop server processors s390x which is the IBM mainframe system Z processors and power PC which is another IBM derived processor line and so

all of these are generally they're fine right they're very consistent but occasionally we'll have a problem where oh gosh there's a you know it compiles for everything but s390x and we have to figure out exactly why and there's there's a lot to that but um yeah you'll see this is this was the this repository here is huge there's like I think 8 000 plus packages in this repository and this was hey skip needs to go through and begin a effectively a complete rebuild of what's in Red Hat Enterprise Linux 9 and some of cintas9 stream in order for us as a build team to get

going and so yeah this is how we start and we we effectively You Know download source code put together Source RPMs and begin this mock process over and over and over again and there's there's tools and automation to help us right so it's not like I'm just sitting here you know can't buy Candlelight you know pressing you know oh gosh I'm gonna I'm gonna build this and then build this and three thousand more times or whatever now it's it's we have tools to you know make the machine do the work but uh yeah it's a lot of effort though and it's a lot of time

but so you see and these are free to browse you can see how rocky Linux 9 was bootstrapped and it's uh it's pretty crazy I hope you have a lot of memory in your web browser because these are some big listings so once you save bootstrap once you have that set of packages that you get from somewhere and you you build you build brachiolinux packages against them do you then go back and rebuild all the Rockets packages yes so this is only you'll notice here this this directory stage one right it's so there's there's more and that brings us into uh Peridot which I think

we'll talk about just briefly um let's go ahead and load it up this is a public site Peridot dot build.resf.org um I'll put it in the chat here and if anybody everybody wants to Pace it up you can it's very uh you said public site anybody can browse this not everybody can import and build stuff but everyone can browse it right and let's just check out Rocky Linux 9.

see how we're doing here and let's go into builds let's just we're going to pick out a couple recent ones here and we'll see node.js hey that's a cool one right everybody's favorite JavaScript server I'll be honest I don't understand it very much but uh that's yeah some people seem to like it a lot all right so and you can see here in this there's a bunch of tasks that happen and these are the interesting ones here it says build for x86 build for i-686 these are the different architectures that we're building for and if you 've gone to build is rather long so we

got to get all the way up here give me a second I gotta scroll having trouble scrolling here I don't know what's up with my browser and those people are watching I guess uh let's go Ben let's pick another one here yeah we hey bind everybody's favorite DNS server all right a new version of bind okay and if we go up here boy this is a lot all the way to the top here usually I click home for this but I'm not sure why that's not working right now there we go okay and you'll see here you'll notice that this text looks awfully familiar right

and that is because when the this Peridot which is our build system I guess gooey it's a lot of things API GUI important build system all it does and uh Koji which is the Fedora Centos build system um all they do they are very organized elaborate ways of calling Mock and RPM build right and so like this this text out here is almost identical to what we were just doing in our if I can back into my build directory here you'll see a little clearer here like this X here like we built we compile her we uh we install a bunch of initial packages then

we install a bunch of packages that the the RPM we're building requires and then it executes RPM build just like we just did right and so this is an excellent way of doing local testing or local builds and you know if you set your mock config correctly you can make it completely compatible with what the real Rocky Linux project or even the real Centos or or red hat Enterprise Linux project are doing and um uh actually I see that question there yeah got a question mark can you cross compile no uh and there's I'll explain why um RPM in general does not natively support cross-compiling

um you can but the the logic to do it has got to be built into the spec file I've seen ones there's some qmu ones in particular that are like this but there's really most packages are not going to bother doing that because they they assume that if you want to perform a cross compilation you have access or able to you know you're able to have a system that also has that compiler um and if you know for for bootstrapping initial systems it gets I'm not that's kind of beyond the scope of this talk but it gets into like well you have to you're gonna

have to cross compile things manually enough to get RPM and some basic packages going let's say that if you completely new processor architecture comes out and we have to build software for it right there's going to be some more fundamental work before you can get to RPM land and so and that and the the goal should be really when we begin to you know we begin porting or a system for this the goal is to get enough stuff to get to this build route right this initial set of packages that we have inside of mock once we have that we can get rolling we can

say okay we'll we'll build you know we'll build very slowly I guess some basic packages get our C compiler going etc etc right so there's a um that's a very not going to get into that process very deep very complicated but the short answer is no what about uh i-686 packages on an x8664 system yes so that is fine um mostly and same ditto goes for um um the ditto for uh if you think about arm 64 uh you can compile as long as the processor supports it so we know x8664 processors Intel processors support 32-bit mode um not all of them but many arm

64 processors are also backwards compatible with arm32 in the same way and yes apps because what we do and we can you know let's check it out here actually let's look at the i-686 build presumably you'll still need like a 32-bit bootstrap to or environment to build against but other than yeah and that and that makes the whole difference right because as long as you as long as you have some RPM macros which mock does for you set to be set to the proper processor architecture and then you have a proper build route and build environment that supports it then you're golden right and we

can look up here um way up here and I'm not sure normally I just press home and it takes me to the top of the log I'm not sure what's going on today but the exact same time must be here we go so yeah you see you'll notice that when we're installing here this entire little system were installed is all i-686 there's no 64-bit stuff going on here right and that's because well you know uh it's 32-bit packages do not need 64-bit packages in fact they they actively don't want them in order to build so um and going beyond that a little bit because I

I can sense your next question which is um it's story time for me um uh the Y build 32-bit packages because Rocky Linux and red hat Enterprise Linux does not release a 32-bit version my question was actually where is my 32-bit ISO yep so I can I can read your mind and the answer to that the reason that we build 32-bit packages is because um if you look at your Rocky Linux repositories and and red hat Upstream Red Hat repositories there are certain packages like openssl several others that do have they include 32-bit variants bundled in so if you want the 32-bit version for whatever

reason you can you can install that and they're for compatibility sake for for various you know a lot of commercial software they might you know distribute a 32-bit version still God Knows Why and they need 32-bit open SSL or whatever to work right and the rocky team uh in the beginning said oh sure this is going to be fine right we're just going to compile you know these you know 20 32-bit packages right we'll compile open SSL 32-bit Oh look The compile open SSL 32-bit we need these three other packages okay well that's fine we'll compile those three other ones oh look these three other

ones require 10 more packages for 32-bit right okay we'll do those we'll do that and once we got to about 150 we were like wait a minute actually we're compiling everything and and yeah that's the conclusion and you know uh really the conclusion that synthos before us came to is in order to build these 32-bit packages these few we effectively have to build 32-bit for everything and um it's uh um I'm not going to go into it right now but if you connect to the Peridot dnf repo or we build eight and Koji still the Koji repos the 32-bit packages for literally everything that's supported

are in there there are some packages I think Firefox comes to mind that will actually refuse to build with 32-bit they're intentionally excluded but most of them are there and so yeah there was um we it was a sad realization we thought it was going to be super easy to throw these you know 20 32-bit packages in and then we found out you know we did a couple days of building and we're like oh God there's the dependencies of dependencies of dependencies we have to build we might as well build everything and so we did do you do a 32-bit kernel also or is that

no that's one of the packages that is not that excludes 32-bit and that's a big reason why we don't have a 32-bit ISO right um it would be you I mean hypothetically this project idea for some anybody who really wants a 32-bit system you could take the 32-bit builds that we've done for all of user space and figure out a way to build a compatible 32-bit kernel you know package put them together and produce an ISO or an image or something and you know I don't see what reason why it wouldn't work it's a lot of work though I'm not sure yeah I'm not sure

it's worth it any uh 32-bit you know old Hardware aficionandos out there it's becoming increasingly irrelevant I'm afraid so it's just not something we're interested in doing sure so the the last big thing that we haven't really touched on in terms of things that kind of cross the boundary and aren't part of that isolation is the kernel itself so even though it's a container presumably as I understand when you're running mock you're doing the build with the kernel of the host system no matter what you're you're targeting has that ever come up is that an issue usually in in in ways you'd maybe not wouldn't

think about right so um good example is uh during the rocky Linux 9 building I believe there was a ruby package that um it required um we were so back up in uh at the time and I think even still now we're we're building Rocky Linux 9 on Rocky Linux 8 systems right so these have you know they're running they're running containers but they have a rocky Linux 8 kernel you know it's a little little older but still well supported and there is a ruby package I can't remember which one but it's related to Ruby that for its compilation um it requires uh it runs

some testing and it requires a kernel to support I think it's um not only temp FS temp FS the temporary file system feature which we have in our Rocky 8 kernel but oh there's a a certain feature of temp FS called like o underscore Temp and I think it's for I I can't remember exactly the feature it provides but it's a new thing in the nine kernel and the Ruby tests would not pass um you know the tests are done at the end of compilation for an RPM make sure the binaries you produced are working properly and yeah we ran into a problem we actually

uh um in order to make sure everything was okay I believe we had to for a couple of our build machines we had to upgrade the kernel on them so that it would support this new temp FS feature and that's just an example of how that can come up have you ever encountered a situation where it works in the other direction where you have a newer kernel but something you're trying to build requires an older one my memory is very fuzzy on this I want to say yes but I don't have a specific example in mind because we've God knows we've run into an awful

lot of problems like that that we've had to investigate and fix if the question is has a problem occurred then the answer is almost certainly yes at least once a broken thing so many ways you make your head spin but again that's usually people are like oh it's you know it's so easy you're just calling the RPM builds over and over again and well yes and if all of them completed on the first try yeah my job would be simple and you know I'm I'm very I'm happy to give it up to anybody who wants to do it but um the problem is when things

don't go right and we you know you'll have to spend you know in some cases hours some cases days of trying to figure out hey why the heck won't this work and why did it work before and there's a lot that goes into that oftentimes it has to do with again the versions of things that you're bringing in and building against because they sometimes a new version comes out and it it break it will break a at build time it will break something in an unexpected way yeah so it's very how did this ever work before is the worst question in systems engineering oh comes

up more than you would think I'm building yes especially again this is building in mock here is great if you want to build you know two or three four or five packages but when we're talking about you know industrial capacity and we need to build five thousand packages or whatever stuff like this comes in handy right we can do Mass builds and we say Hey you know we we're not going to do this one at a time we're going to you know kick off a million or whatever so skip are you running across users out there that are trying to take some of these packages

that are just a default do something to them and then like you said earlier put them into their own repos so they have something specific that's required for their company their environment that brings me to another great point is that I mentioned that um we already know that we can get our source RPMs from you know our download site or again they're they're available via dnf command right if you check out your your um uh inside of or sorry yum dot repos dot d here it is um Rocky Linux and other Distributors come with Source repositories and all they do is point to these Source

RPMs however these Source RPMs have to come from somewhere right and I'm just going to go into that real quick get.rockeylinux.org we want to put that up on the screen it's another great site I'm here a lot quite frankly and so let's go into staging RPMs and then the sort the source RPM name so let's talk about Basher right and it's my fault I misspelled it typo using RPMs and Bash without an apostrophe at the end and you'll see here we've got some branches and branches are funny when it comes to distros here this is Rocky 8 Rocky 9 we never merge branches really these

are just different versions that we keep track of and you can see you know when when things were imported I think um yeah we the latest version of bash for Rocky 9 was imported five months ago and built I think it hasn't there hasn't been any updates from Red Hat since then but you'll notice here sources and specs let me pull up my old uh if you remember this guy here this is the source RPM that we got from here and if you browse it this Source here I'm going to put it on top here so that we can kind of see them side by

side you'll notice so specs bash.spec is right up here and if you go into sources here you'll see all these patch files and here they are here they should match exactly right and that's where you want to know where these Source RPMs come from check this out all right and you can this is open open to everybody you can clone it you can Muck with it you can do whatever you want with it and you can build your own you can you know don't want don't want our bash bill you know make your own fixed bash like put your own patches or whatever and and

um yeah so this is um and our sources in turn come from uh git.centos.org is where uh which is where Red Hat publishes their official source code for and we'll see go into branches here just like us they have sentos branches you know you want the synthos floor bash here it is all right we're not really interested in that we'll check out there is no cintas9 really but um they still call it C9 but yeah this is the this is the red hat source code and this is exactly where we got it it gets pulled down and tagged into our git here so you can

kind of see the flow of where these things come from and if you're interested in modifying it it's pretty easy well I won't say it's easy I'm no C programmer right but it's it's easy to get at anyway see you again never mind I got a question sorry come here nope nope no question I thought we had a question but I was I had a question okay all right good stuff yeah so let me let me check my notes over here real quick but I think that's um that's about what I wanted to cover just uh please if you're building RPMs please consider mock the

old especially if you're trying to use just plain old RPM build it's it's hard and it's outdated and I just I used to hate doing this you know when my company back in the day would be like oh skip we need a customized RPM build for such and such a program like oh no all right give me a day and I'll see if I can figure it out you know like this makes this much much smoother right we have a if we kind of Mark is done for package building especially RPM package building what really containers have kind of done for developers we have a

standardized base that we can build on and you know we can make sure that when I build a package in my mock it's going to be the same as a package built in Jonathan's Mock and Zane's mock and as long as our configurations are the same we should get the same result and it's really excellent for that please consider it if you're still using raw RPM build to build packages nobody should be doing that because it's it's rough like quite frankly never have to configure RPM macros ever yes I'm not saying that but for most things yes and and configuring RPM macros are are just

configuring mock right like it's inside that mock configuration file you can set macros just like you can do everything else and so it's much easier than having the muck about on your file system with God knows where wherever RPM macros are kept right oh that's great I don't think we have any more questions guys I really appreciate you coming on skip thanks for putting this together it's great to see you glad to see you like I'm glad it went well because I don't uh whenever I do demos Things Fall Apart it's uh not today it's been great we really appreciate stuff appreciate it so if

uh do not anything else I don't see any other questions I really appreciate you if you want to uh join us next week and like And subscribe we appreciate it see you next week all right thanks guys foreign