Skip's School for People Who Can't Package Good and Wanna Learn

February 9, 2023

Webinar Synopsis:

What is Rocky Linux?
RPMs
Source Code
Requires and Provides
BuildRequires
Spec Files
Can Bison Be on the Same BuildRequires Line?
Where Does Rocky Linux Come From?
Git Structure
Do We Only Mirror CentOS?
Do You Copy Packages From One Location?
Compilation Process
Mock
Controlling Versions of Build Dependencies
Version GCC and Building Bash
Mock Config File
Does the Mock Environment Retain What You Install?
Capability of Mock
Arch 64
FWTS Hands-On Demonstration
Fedora Full HASH vs Rocky Short HASH
FWTS Hands-On Demonstration Continued
Issues With Install
Can A Second Package Interfere With Local Repo?
Limitations to Using Mock
What Skip Does For CIQ

Speakers:

Skip Grube, Senior Linux Engineer, CIQ
- Linkedin

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Skip Grube:

And we'll wait just one second. Meeting is being held. Okay I got you. Yeah, cool. That thing came up for me. It probably came up for you guys too. All right, cool. So yeah, we'll start talking here. So first of all, a little bit about me, if you guys don't know me, many of you do, I'm Skip Grube. I am a Rocky Linux volunteer, contributor, dev guy. I'm a package head, basically. I started working on the project from the very beginning. And now I work for CIQ. And I get to do Rocky Linux and Rocky Linux adjacent stuff as part of my job, which is really cool. I'm very happy about that. And so something about this whole talk or whatever, I like being interrupted. So if you have questions, if you don't want to, like, interrupt me talking, you can do that too.

I don't care. But if you can chime in with questions or you can use the chat here. I have the chat window up on the side of my screen here. So I'll see your question as they come in and I can find the end of sentence and pause. And I can talk about whatever you guys want to talk about. So, cause it's, there's all kinds of twists and turns. And I don't want to get onto too many tangents, but it's, this is important stuff to cover questions. So we're going to go into some of my notes here, which I'm pulling up. And so the first thing. I think what we're going to do here, we're going to talk or we're going to talk. Have questions and so forth. And then we're all going to do a hands-on exercise, right?

What is Rocky Linux? [1:49]

With our own systems, I'll guide you through that. And it'll actually have a tangible result. So it'll be really cool. We're not here just to, it's not just practice, it's actually something that is useful. So look for that after the talking. And so first of all, it's important to know we're all here. We all know, at least, something about Rocky Linux. And I would say, when you ask, "What is Rocky Linux?" Like, what does it consist of? It really, yes, there's a giant community and people and we all love each other, but there's two tangibly though physically Rocky Linux consists of packages and it consists of images, right? And images can be ISO's or containers or whatever, right? And we're not going to talk about images today.

Not a lot. We're just going to do packages. But yeah, this is really the bread and butter of like, what it is, right? Like without those, you don't have a Rocky Linux, there is nothing, right? So that packages are super important because they make up the distro, they define what versions of stuff we use. They have all our cool things, our goodies, and it's important to know where they come from and how they're made, right? And it's really useful to be able to make your own too. So yeah. So when we talk about packages in Rocky Linux we talk about RPMs. We are an RPM based distro. It's what we do. We come, you guys are all familiar with this. We come from Red Hat Enterprise Linux. We're basically, we're a clone.

Exactly. And ultimately way upstream before it gets to Red Hat Enterprise links, we come from Fedora, which is also a popular RPM based distro. So, but we're going to talk about RPMs specifically because I think a lot of people are not real, not, don't have the strongest picture about what they are precisely, right?

RPMs [4:02]

So RPMs. I'm going to actually, I'm going to flash up my first visual aid here. Hang on a second. Let me share my screen. This and this. I'm going to use stuff from the GitLab repo that you guys all got linked to. So if you. I'm going to . Actually, I'm going to put it here so you can. Because we're going to use content from there. Cause I've been uploading stuff. And one of the things I have here is a picture. Hang on a second. I'm going to open in a new tab and I'm going to share my screen here and we can look at my picture.

Awesome. So yeah, so RPMs, they come from source code somewhere, right? They are, they consist of source code for something. In this case. I love using Bash as an example, cause it's a really good basic package that lots of people use and love. And they consist of source code, and they consist of a spec file, which is a kind of recipe for how to build the source code into something tangible that you can use. And I got my talk here, hang on. So does everybody understand that? First of all, do you guys? Here, let me pull this back up. So we go from source code here and we hold our source code in Git. We don't have to use Git, but generally that's what we like to use. And it becomes a source RPM, which is just a, effectively, a compilation of the source code and this spec file.

And then there's a process which we're going to talk about in a second, which turns the source RPM into binary RPM's. And these are the things that you install on your end user system and so it's important to note that the RPMs you install are not always compiled, right? So this one here, Bash, is because it's. Bash is written in C. It's a. You have to compile it in order to make it work. But we have another one. For example I can go to. I'm just going to pull this up real quick here to show you guys.

It's here. Yeah. So there's a rocky-logos package. You'll probably be in, if you have a Rocky Linux system that has Gooey enabled, you'll probably have this on your system. I think you have to actually and this is just a collection of pictures, right? This is like, I think it's backgrounds and things, but yeah. And it does not have, I'll give you an example of the spec file here. This is the recipe that builds this one, but there's no compilation to be done here. It's just, you just copy the pictures into where it goes, right? So we say compile our source RPMs, but they don't. Compiling means different. It doesn't necessarily mean make a file and we're going to do all this. It could be anything, right? There's all kinds of ones that don't do compilation

They just. This one's just pictures. You just copy the pictures in the backgrounds or whatever. And yeah, there's a bunch of JPEGs and PNG files in it. So yeah, so we don't actually mean. Compiling means a broad thing here. And it's also important to know that, so we have source code. Source code and source RPMs are one-to-one. They have a one-to-one relationship, right? I'm going to also demonstrate this here. So every freaking. We're going to go to our Git here. This is where our RPMs are. So in RPMs, I don't think it has a count, but there are thousands. You can see how many pages we have here? There are thousands of packages right in here and I'm just going to pick a random one here.

Yeah, I need one that I know is in. Oh yeah, exorg-x11-server cool. This is this. Ha. If you guys aren't familiar, this has to do with what ruck drives your Gooey or part of your Gooey, sometimes in Rocky Linux, sometimes you have Wayland as well. And, but this package right here corresponds exactly and we're going to go. We're going to browse, I don't know if you guys have seen this, but download.rockylinux.org is where we publish our packages along with the mirrors. This is where the mirrors get cloned out from. And if I go into it, it's an AppStream, I think. So I'm going to go into sources where we keep these source RPMs. You'll see there's an xorg-x11. Sorry, there it is. Yep. Right there. And so every time you see a source RPM, it corresponds to a Git repo.

They are one-to-one, right? So if you have the Bash source RPM, there's a Bash Git repo. Xorg-x11 source RPM, there's a Git repo for that too. And so but when we go to compile it, go back to our diagram here. Once we go from source RPM to compiled, what we call binary RPM, this is the stuff that you actually will install on your system. It's a one-to-mini, right? Like over here we have, there's a Bash-devel. A Bash, this is the actual program. Here's the documentation that you can install. Like it pro, it can produce many, right? An infinite really, it's sometimes if you compile a source RPM, it will just produce a single binary RPM. But I would say in most cases for at least for official packages that we have it is, it will produce more than one.

And sometimes it can be insane. Like I will show you. This is our build system. We're going to cover this in a little bit, but a little preview here. Glibc is a core C library that's installed on all your Rocky Linux systems. Covers a lot of things. And if you look here, here's the source RPM, right? There's just one thing. There's a glibc source, but here are the binary RPMs, right? And there are supposedly language packs, but there are a bajillion of them. And this is just the nature of it, right? You can, one source RPM can cover the source for a lot of little sub-packages that are, they're all related, right? They're not completely random. But you know, you can divvy up your binary RPM and do as many pieces as you need to, right?

Because not everybody needs to install like all these language packs and not everyone needs to install the dev, um which has like header files and things you're, if you're interested in building. So yeah, so it can be, it can get kind of crazy. I'm just using that cause I know glibc is an extreme example. So that's an important concept to know, right? We go from Git, which is source code to source RPM to binary RPMs. And then these binary RPMs are ultimately what gets shoved into our repositories for and that's what you install when you do your DNF install. You get these binary RPMs that are produced. So I'm going to. Let me take some tea here. Any questions or anybody, everybody understands that you, probably a lot of you are probably, I know, already understood that before I even started this. But so everybody is on board with this idea? I'm taking your silence as you understand completely.

Source Code [11:49]

So good stuff. So, now, yes, we're going to talk a little bit more. There's lots of parts to RPMs. Let me go back to. We'll pull up Bash here because I'm familiar with that one. And again, it's easy. We're going to look at, so really the source code. I don't want to say it's boring, but source code for these packages can vary wildly, right? Like we mentioned, Bash is all C code, right? And it's all real low level. There's Python packages that have Python code in them. And there's the rocky-logos package, which doesn't have any code in it. It's just pictures and things. And so but they all, so the source codes are all over the place. But the recipe for building these things, while it's different, it contains different things. They all have this .spec file in them.

And you'll see, you can go to any of these by the way, I encourage you, rockylinux.org, staging RPMs, and then package name. And you can browse any of our packages. We have, like, 3000 of them here. And yeah, the spec file is the recipe for putting this thing together. And I'm not going to, I don't have time to cover the entirety of RPM spec files and all the different ins and outs of them. It's actually quite a complicated topic. But one of the things that we do want to focus on is the concept. You say, there's a bunch of, like here's a source code for it. There's a bunch of patches that get added as part of the build process. We'll talk about that in a little bit. But I want us to talk about the concept. You know what? I'm going to, I think I'm going to copy this. I'm going to get into a syntax highlighting state because GitLab does not highlight the syntax very well.

Let me make it big, everybody can see it. And I think highlight, mode, RPM spec. Here we go. Okay, everybody sees that okay? Me too. Okay. So yeah, you can and you, like I said, these are not, they're kind of code, but you can figure out what's going on here. Even if you have no experience with this. You see we've got some information about it. Here's the name, right? Bash summary of it. Here's the license, it's under things like that. And there's a version and everything. This is where the source was obtained from, right? Like this is where the actual GNU project. Yeah, I'm pretty sure everyone here has at least heard of that. They developed Bash, right? And this is where they published their source tarballs for it. And this is where it was originally obtained from, right?

Requires and Provides [14:44]

So that we have information like that in here. We're going to skip past some of the sources and patches. I want to talk about this concept of "requires and provides," right? Because it's super important, especially for what we're going to be doing. So RPMs, one of the huge benefits of them and other package systems is that they are interdependent on each other, right? So like, if I go in and I install Firefox, I need the packages that provide the graphics libraries and the other things that Firefox needs to operate, right? And so, again, I'm, I know I'm stating the obvious for a lot of people here, but I'm going to, for every. So everyone is aware that RPMs provide and require things. And these are literally. It's important to realize these are arbitrary, right? Like they are very, we have standards and we have conventions I would say that we use to try to require things.

But like, I could go in here and edit this and I could do, hey, right? And like I could. This could be anything, right? And if, as long as another RPMs will not install without another RPM providing the thing that they require, right? So it's a little, basically DNF your package manager matches, provides with, requires. That's how it works. It's got a big list of each of them. And it says, "Oh, when I go to install Bash, I need a file system greater than or equal to 3, right?" And I'll, it has a giant list of RPMs and there is an RPM and Rocky Linux that provides file system 3.5 or whatever, right? That says, "Ah, that's the one I need, right?" And so these are carefully constructed to be interdependent on each other

And then there's other packages that will require #!/bin/bash, right? And so it says, "Oh, #!/bin/bash, we need to install this one, right?" And this is how your system operates. Like this is how it's put together. And it's, again, completely arbitrary. Like there's, if I had a package that required blah blah kit blah and then this Bash package provided that, that package would install Bash, right? In order to fulfill its requirement, right? Doesn't matter. Like it. They require and provide, they don't. I want to, I don't want to say they don't matter, but they're not. There's conventions, but they're completely, they're arbitrary. They are. You can add whatever you want in there. I wouldn't. Don't add this in here cause it. If you have like, it requires this and nothing requires or nothing provides it. It won't work. It won't install.

So hang on a second. I got to. My meeting chat just broke. Just a second. Okay, good stuff. Yep. So sorry guys. So anyway, yeah, we're not really. I'm not going to mess with this too much, but yeah. But you can kind of see here, like here's a description about what it is. What this package is all about. What's inside it and everything. The other thing we're going to talk about. So requires and provides, this stuff matters to the end user, right? So when you go to DNF to install your package, we're going to look at it. Or DNF rather, not us personally, but DNF looks at this stuff.

BuildRequires [18:29]

Now there are. There is. This is more interesting here. This BuildRequires: this means that stuff that's required to build Bash, right? In this case we're looking at Bash. This is stuff that's required to build but does not. It's not relevant though at runtime, right? So what we're talking about is when we go from our source RPM to these different binary RPMs, we need to have this stuff, right? And I think, I would say it just assumes GCC is there, that's interesting. So really, if you guys aren't familiar. Bash is written in C, and requires a C compiler. C is a programming language.

So I might. To make it better. So it definitely is. We'll talk about why we assume it's there already anyway, like later. But yeah. This I would say, this is definitely part of that. So, but this is. But like this stuff here. As you know, autoconf. This helps, this autoconf thing helps when configuring your package for compilation. But when Sharif or Mike or whoever goes to DNF install Bash, they don't care about autoconf. They don't need it, right? Because they just want to use Bash. You don't need autoconf to use Bash. You need it to build Bash. So it's a super important distinction there.

But yeah, so the. You need things to. You need pa. And sometimes these can be the same. Like sometimes you'll have some packages that BuildRequire this thing, but they also require it at, in, at runtime as well. That's allowed, you know. But yeah, important to keep those two distinct. And again, you can see that you want to drive home the point, these provided here do not matter what they are. They can be files here. Like this one provides this file. It can be whatever you want. The the important thing with the

Requires and Provides is that they are all interdependent. They all are. They all depend on each other, so they have to be accurate, right? We can't make a mistake and say #!/bin/bash here. Because then if we go to something that requires #!/bin/bash, nothing will provide it, right? So it has to be included. So anyway yeah.

Spec Files [21:05]

So we're going to move on. So everybody is kind of familiar then a little bit. These spec files are super important. They offer and there's all kinds of things that we're not going to get into the ins and outs of how exactly they work, because again, that's a whole, like, course by itself. And we want to actually do some stuff. So we're mostly. We're not going to be mucking with RPM spec files too much. We're mostly just going to use them as is. But yeah, you can see there's stuff in here that configures. If you're ever familiar with compiling software you'll recognize it's a little odd here to have this percentage sign. But configure_make, make_install, that's what it's doing in here, right? And there's scripting to make sure that the different files that are produced get put in a special place and then they get captured and put into your binary RPMs.

So you'll see. If you look at any of these spec files, you'll see they're all in here. And you might be familiar, probably familiar with some of these commands and what they're doing, right? And it's just. It's just scripting in here that does all this. So it's very, I said if you're familiar with any kind of Linux scripting or file manipulation, you'll be able to look at this and figure out what's going on. So let's go to the next section then, Alex. Surprise everybody's understanding. This is great. So I guess the next part I'm going to talk about is more. So this is, by the way, this stuff that I'm, I've mentioned so far with RPMs and everything. This is not just a Rocky Linux thing, obviously this is every RPM distro that does this, right? Whether it's Fedora, openSUSE, Red Hat, CentOS or CentOS Stream. There's and there's dozens of other ones, smaller ones. I'm not even going to mention, but let's.

Can Bison Be on the Same BuildRequires Line? [23:01]

Oh, hey, that's a good question here. So why can't bison be on the same BuildRequires line? You know, that's a. Let me see here.Yeah, it totally can. I'm not sure. In fact this is actually a good example here because it shows the different things. You can separate them by commas, you can separate them by spaces. And yeah, so I think, I believe, again, anybody who knows more about spec files than I do, tell me if I'm wrong, but this. If I changed it to this, it would also be valid, right? I could just have them all on the same line. But I assume they've done this for human readability here. Cause I think text info and bison, I believe they're kind of related a little bit. And yeah, thanks Greg. Completely a hundred percent valid. Yeah, you can totally. I accidentally picked a good example, I guess. But yeah, but it, I mean if you. Especially if you get to, like, 10 BuildRequires, you probably don't want to string them all back to back to back to back. It's hard to read. So I think autoconf and gettext and texinfo and bison are. They're kind of related to BuildRequires. Like they're related to each other. And so it's set up like that.

So, but, I. You'll have to ask, probably Fedora. Actually, that's where these spec files, most of them, the meat of them, originally comes from. We'll talk about that in a second. So I did not originate this package, so I don't know. But now we're going to talk about the left side of my diagram over here and go back to my visual aid. Hang on.

Where Does Rocky Linux Come From? [24:48]

So we've talked about, we can take our Git stuff, we turn it into source RPMs. We'll show you how to do that in a second. Then it becomes multiple binary RPMs and we can install them. That's great. But in terms of Rocky Linux specifically, it's important to know where this comes from, right? And everybody's like, "Oh, well it's Red Hat Enterprise Linux, right? It's Rocky's a RHEL clone." So it totally comes from that.

But, like, how? Right? Like you could just say that. Sure. But like, we're like, "Show me, right?" And I will do so. Cause it. This is super important to understand. So we have and I'm only going to go back as far as the RHEL source code. If you want to, you can track it back further because it gets imported into RHEL from Fedora and now CentOS Stream. And sometimes it gets modified in RHEL, sometimes not. Sometimes it's just the straight up same old package that got thrown in there and you. If you go back to Fedora and CentOS Stream, you can go even further back to the actual mozilla.org or gnu.org or wherever the heck that particular package is developed, right? So we're not going to go back that far. We're going to stop at where, kind of, focusing on Rocky Linux here.

So we're going to show where it comes from in the sense of Rocky Linux and I believe if you go to the Git URL that I have, it's in the information down here. I put a bunch of resources. Yeah. So we're going to start at Rocky Linux, right? So we have in here git.rockylinux.org staging RPMs. And then I'm just going to use Bash because, again, it's a good example. It's really basic. So there's a Bash here. We're going to put that in the meeting chat here. Here we go. We change this, everyone, there we go. Put that. I put that up in the meeting chat. This is where the Bash source code is and you'll see there's a little note here. It says, "Hey, we imported Bash, this version four weeks ago, right?" And it's like, "Well imported. Where the heck from?" Right?

And so we look at our. We look at our package Git locations here, RHEL 8 and 9. The RHEL source code, as we always tell people, is hosted git.centos.org. It follows a very similar pattern to ours. And that's by design.

git.centos.org/rpm/package. So if I take this, I'm just going to copy that bit there, and we're going to get a new tab, rpm/bash, right? So that's it. Because that's the name of our package. And here it is, this is the actual RHEL source code, right? This is what Red Hat Enterprise Linux builds from. And I'll put that up in the chat as well so people can just click on it.

Git Structure [27:51]

Yes, and so we're going to talk about Git structure a little bit here. Let me find my notes again. So the. You'll notice the if. If you're familiar. I think a lot of people here are probably already familiar with Git, which is great. You'll notice that there are branches here, right? And package storage is a little different from traditional development where we don't have, generally speaking, we don't do merges. And I know people who have worked in development before, they're like, "Okay, I'll create a new branch and then we'll merge that branch into main." There is no main branch here, right? There is a master or whatever you want to call it. There are these differences and you can probably guess what these are for. c8 is the convention for CentOS 8. CentOS 8 is no more.

It's actually RHEL 8, right? So if you want the RHEL 8 source code for Bash, you go to this branch, right? And this is so and conversely, if you go similarly. If you look at the Rocky Linux source code we have r8, Rocky 8. Hey, easy, right here's r9, Rocky 9. And so if you wanted to, for example, audit Rocky Linux, right? You could go to each of these repositories and let me go back to branches here. So like the c9 branch here should match exactly with the r9 branch here, right? We literally just took the code from git.centos.org branch c9 and we brought it into branch r9 here, right? And you'll see, like I say, we have a little note here in bash-5.1.8-6 and over here, I'll bet in Commits.

Yep. 5.1.8-6, right? So this was done 17 days ago here. And these are the Commits on the. This is when Bash was changed in RHEL and if we go down here, where's the? Yeah. History here? Yeah. Two weeks ago. It doesn't have, I think if you hold it over, maybe it'll, yeah. January 23rd. Is that 17 days ago? I think it is. My math is right. So yeah, basically what we do is we constantly monitor this area, right? For Commits. And when a Commit happens, we copy it over to here and then we can build it. And we'll cover how that happens in a little bit. But yeah, so. And if you're curious about the other branches here, there's some funny looking ones. So like for example where, okay. So c8s here, CentOS 8 Stream, right?

That's where you want if you're running CentOS Stream, you need the Bash source for CentOS Stream. This is where it's located, right here. So as you can see in the extra information from my git page, we have the four things I'm concerned about: Rocky, RHEL, CentOS Stream, and Fedora. And this is where they're all hosted. So you can just. Any source RPM that you care to name. If you go here and swap out packages for this, you'll get to their Git repository. That's where they are. So yeah. So a quick pause for tea. Any questions? Any fun ideas? Is everybody, I mean, it's pretty straightforward I think, but source RPMs, Git repositories, one-to-one and they're hosted in these places. And again, if everybody's quiet, I'm going to take it as they completely understand. Right? Don't all speak at once. Okay.

Good stuff. So yeah, great. We've covered Git and where stuff is stored and now we're going to. I'm going to talk a little bit about building RPMs ourselves. Like individually and then we'll, yeah. Then we'll be able to do it, like for real, right?

Do We Only Mirror CentOS? [32:11]

Oh, hang on, I now have Jonathan's question. Oh, multiple questions. Nice. I'll go in order here. Okay. Do we only mirror CentOS? Yes. Rocky Linux is. The main distro is hyper-focused on bringing the RHEL source code in, compiling it, and building into distro based on the exact same as RHEL. Fedora, I can say individually there may be some Fedora packages that they're not related to the main distro that might get imported at some point, right? But that would be one-offs and little things that we would like. For example, I could. Maybe I have a little project that enhances the Raspberry Pi Rocky Linux thing, right?

And it wouldn't be anything in RHEL cause RHEL doesn't work on the Raspberry Pi. And so maybe there's a package in Fedora I have my eye on. It might work well. So I'll import that package or I'll copy it myself. It won't be part of our big actual project. But it may end up in Rocky Linux Git. But it probably won't be in the main area here, if that makes sense.

Do You Copy Packages From One Location? [33:22]

So do you actually copy packages in one location? Okay, so yes, Git clone. Yes. So this, right now, this process that I'm talking about, let me go back to my map here. So we're doing it here, right? So it's just to Git clone, right? We haven't produced any RPMs yet. We're not doing it. We have not even sourced RPMs. We haven't built those yet, right?

We are just cloning from git.centos.org/packet or /rpm/package to git.rockylinux.org, right? So this is pure just to Git clone, Git transfer, and commit to our area. So no package building has happened yet, right? I mean it has obviously and Rocky Linux proper, but in this stage of whenever there're become. There's a new thing in RHEL: it gets cloned first before anything else. Yeah. So we do not copy any Red Hat binaries because even the source RPMs are binary. Like they contain source code, but they're binaries in a sense. We do not copy or clone any of those. We don't need them.

Are All Those Other Repos Needed to Build Rocky Packages? [34:28]

So any who. Hopefully that answers the question. Oh, ah yes. So good. Good question here. The question is, are all those other repos needed to build Rocky Packages? And that answer is complicated, but I'm going to try to simplify it a little bit. This one for sure is not. I just. I'm noting this here partially because this is going to be part of the exercise that we're all going to do in a little bit. But yeah, generally speaking, we clone from git.org into the Rocky packages and that is enough to build all of Rocky Linux. I'm going to put an asterisk there because this changed with Rocky 9 and I'll explain that later. Rocky 9, actually, git.centos.org is not enough. You actually need some CentOS 9 stream stuff in order to build it. Obviously you don't need it in order to. It's those that BuildRequires versus requires thing again, right? So in order to build some Rocky nine packages we need some, a couple of CenOS stream packages. We don't need them in the distro, but we do need them at build time. So hopefully that answers the question.

Compilation Process [35:48]

So cool. This is great. Good questions by the way. Thank you. It's excellent. Love questions. I love interruptions. So, all right. So now we're going to talk a little bit about what we've covered. Let me see. We've covered this end of the proceedings here. We go from Git to Git. We know how that works and we're going to talk a little bit about the compilation process over on the right side here. So we know there's a spec file and we know there's source code that you and we're going to say, "how does the spec file turn in and that source code turn into stuff that you can DNF install, right?" Because it's got to somehow. And there is an old tool called RPMbuild. It's creatively named, I guess. And it's been used, a long, for a long time.

It still is used, but in a different way these days and I'll explain it. But effectively what it does is it reads your spec file and/or you feed it in a source RPM. It reads the spec file and then it will produce your binaries, right? It'll actually go through, go read all of this stuff, parse it, go through, and perform these steps, basically. Check out what source you have or it will extract your source. It will do all this configure stuff. In this case, it's going to run GCC, the C compiler, and make all these binaries for Bash. And it puts. Basically put it together and it's great. It's a good tool. It's a very old tool. It's been used since the nineties, I think. And it works well.

There's a limitation to it though and that is whenever I run rpmbuild, it is using your system as the build environment, right? Like your, literal, my, your. So if I have a laptop and I want to run rpmbuild on it and say it's running Rocky, if I run rpmbuild on Bash, I'm going to need this stuff installed in my system, right? This texinfo, and ncurses-devel, and all this stuff. And that's fine, it's cool, but I don't really. I don't necessarily want all that installed on my system, right? Like it's and if I'm building many RPMs, I'm going to have a really big messed up system in the end, right? Because I'm going to need all this stuff. And there's another issue that in fact, I'm going to illustrate it here when I was playing earlier.

So if I compile Bash like this, right? And let's say that I don't have one. I need GCC in order to compile Bash and let's say that I already have it on my system. I was compiling something else earlier and I already have a C compiler no problem, whatever. So let's say that I have so Bash is like, "Okay, well you don't and if I build Bash here it will happily compile like this, right?" It'll use GCC because I have it installed. But what if somebody else wanted to build Bash as well and they don't have GCC installed, right? It's going to fail because it's not listed here, right? And so it's. We have a. My system, your system problem, right? Where I can build stuff, but everything that I build here is dependent upon what's installed on my system.

Even if there are, sorry. Even if there is hidden stuff that I already have installed on my system that I just assumed, right? Maybe that's not a good assumption. Like Kyle's laptop, or Dave's laptop, or whatever. They may not have GCC on it or some other thing that's needed. And so it's. We have an odd problem here and I know Greg has told me in the early days of CentOS. What they would do is they would literally have a script that, like, installs just these packages on a system, do a build, and then uninstall everything and try again, right? And that's like and it works. It worked well, I assume, but it's kind of clunky, right? Like you have to.This is and it's. It may not. It's not great for everyone because you have to assume that you have the build system, right?

Mock [40:31]

Yeah. Okay. Yeah. So second. Okay. Thank you, Greg. So yeah. So we've and Greg was saying he did it in a chroot or a fake. You're not familiar with the chroot fake system underneath a directory that's got like a. It's a root file system, but it's held somewhere else. And that's what we do now too. It's just a little more formalized and we're going to talk about that. So there's this tool called Mock and Mock is really neat because what it does is this little thing written in Python and its whole job in life is to create a fake system underneath a directory. It's a. It's such a chroot. It's actually a container sometimes, but we'll talk about it in a second. But basically what it is. So when I go to compile a Rocky package and I use Mock, Mock will create an entire Rocky system underneath this one directory on my actual laptop or my desktop or whatever. And inside there it will install just a tiny, minimal Rocky system. Just the bare basics of what you need to get going, called a Buildroot. And then it will execute the rpmbuild. This process is inside there. So I don't have to muck around with the packages. I don't have to mess up my laptop just to build a package.

Controlling Versions of Build Dependencies [41:56]

All right? Oh yeah, we got, this is great. Yes. So how do you control the versions of build dependencies and the build system? Does your build system always? Yes. So David, that's what we're going to do. I promise we'll cover that more in detail in a second because that's super. Dependency control when you're building is super important and one of the things that we do with Mock is that we have a whole DNF config where we point to different repos that we want to pull from and just those repos. And DNF does DNF things, right? It will pull. It will pull the latest from those repos. And we're going to, again, we're going to talk about that. Version GC, sorry I'm reading. So Daniel did run it in a Docker container. Yes. I think the idea is that this doesn't have a Docker dependency.

The only thing that, I mean, it could be done that way. Sure. Like or just a chroot or whatever. I believe the reason Mock is set up how it is, is because it depends on system D, which provides nspawn containers. Which is what they use. But we're actually going to use it, not even that. We're not even going to use containers. But it works the same way. I'll show you in a little bit.

Version GCC and Building Bash [43:19]

A YouTube comment: any issues with the version GCC on system being used to build Bash differently for the versions? Okay. So yeah. This is so because we're using Mock here and yeah. This is. These three questions are actually kind of related. And it's again, something I harp on a lot. Super important to control what versions of things are being used to build your packages.

And yeah, cause it's. If you use the. If you use the wrong things at build time, sometimes you can have issues, right? Like maybe it'll build, but you know you've injected something from Fedora in there and now it messes up because it really wants the thing from Fedora to be there or whatever, right? So yes, being used to build Bashes, yeah. So generally speaking, most of the packages and we are fortunate in Rocky Linux in that our distro is very well defined, right? There's not nearly as many packages and crazy things as in something like Fedora, like we have 3000 packages or thereabouts. But that's actually a small number believe it or not, compared to certain other distros. So we're kept pretty tight. So generally speaking, the other thing is we are extreme-stability, right?

So I think in Rocky 8 for example, we're looking at the Bash from Rocky 8 including GCC 8.5. I think somebody might want to correct me on that, but GCC 8.5 or the GCC 8 series will be throughout the lifetime of Rocky eight, right? So it is. So when we build Bash, we assume we're building with GCC 8 and it's. That makes it easier on us cause we don't have to suddenly switch over to using GCC 11 or something. So other Distros, again, like Fedora might have this issue where they have to have a giant jump in version numbers for a build tool and they have to compensate by. Maybe there's some flags or something they have to go in here and set because it doesn't work anymore, right?

So let me see. I've got more stuff. Yep. So, okay. And so we're going to talk about it. So because we're using it. I see the YouTube follow up here. So I didn't actually remove this. I. So this BuildRequires GCC wasn't even there. And I'll explain why that is in a second. But yeah, this. The nice thing about not having a standardized Mock build system is that I can point to my, and let's look at the config. Actually let's pull that up here.

Mock Config File [46:15]

So you'll notice again in the teaching GitLab here. We have. It's up here. Yep. There's a bunch of files in here and one of them is Rocky 8-x86_64, right? And this is a Mock config file and we're going to take a look at it because it's not that intimidating. It's about a hundred lines long. But there's a lot of cool stuff in here that is super relevant to what we're talking about

Yeah. That's all right. Okay, so the first thing up here is. And I put comments in here by the way so that you guys can follow along if you download this file. So this chroot setup command here. This is the stuff that gets installed no matter what, right? I remember I said that, "Mock is going to create a chroot and it's going to populate it with a minimal Rocky Linux system." And you got to ask, "Well, what is a minimal Rocky Linux system?" And it's this, right? And this is configurable. You can. We can make this whatever we want, but in general, this is accepted. Like this is what we're installing, right? And these are all. These are all provided by stuff in Rocky Linux. And you'll notice here there's a GCC C++ and I think over here. Yep, there's a GCC here, right?

So we are already starting with GCC, right? So that's why it's not in the Bash. That's not. That's why it's not here. Because we are assuming that every time we build something, even if it's the logos package, right? We are just going to have this basic environment. And so the nice thing about this is that it is long. If I'm building something on my system and I hand it, maybe I'm running Rocky Linux, and then I hand it over to Mike who goes to build something on his system and maybe he's running Fedora, right? As long as we have Mock and the same Mock config file here our builds should come out the same, right? We are using the same dependencies and the same stuff. Thank you Jeremiah. Yes, this is the config file.

We're going to actually use this config file in just a second here. But yeah, I'm going to just explain a few things. So these are. This is some standard stuff that, again, you can. If you're really into it, you can muck with this on your own time. But yeah, I'm not going to explain absolutely everything here we see if we're using DNF as our package manager. We can add macros, our own macros into our builds, right? So macros are things that you can control. They're things that get past the spec file. The spec file can substitute them in for text. You can see. You probably saw in the spec file, there's a bunch of percentage signs and there's. Bash has a bunch of them here. Yeah. So like, this is. These are variables, basically they're. You can think of them as variables, macros.

Again, I'm not going to go into the meat and potatoes of RPM stuff in depth. And then anybody who's administered a Red Hat or Rocky system is probably going to look like, this looks familiar, right? And this is because this is totally just a DNF comp file, right? We don't have repo files because you can. If you wanted to on your system. You could on your Rocky system, you could add the repos instead of in those .repo files. You could stick them all in one in the configuration etc.dnf.conf file and it's valid. And we do that here because we don't want to break this up into multiple files. It's just a variable. Another cool thing about this, if you guys. If anyone here is a Python programmer, you probably recognize some of this notation. And that's because this is literally just Python variables that we're setting.

Like they are all these Python variables. Like if you ever set a Python variable, these are dictionaries or whatever. And they are all, yeah, they're settable like that, right? That's all you're doing. So, which I think is kind of neat. I like that because you can be pretty sure about the format and everything because Python has a pretty clearly defined format. Hang on, I'm looking at where to catch the Mock building, right? So you'll only add to it, okay? So when you say only add to it, that's. You want to talk, David? You can talk. It's okay. So when, like, I don't.

Does the Mock Environment Retain What You Install? [50:52]

David:

Yeah. So I was wondering if we could keep what we've already installed into a Mock environment so that the next time we bring it up it already has the legacy of what you had in there.

Skip Grube:

So in general the, only for this stuff, right? And this is I'm oversimplifying when I explain this, right? So what Mock actually does is it installs a minimal system with just these packages in it here, right? This list. And then what it does is it copies that, what it did over, and then it will look in your RPM spec file and say, "Oh, I need this stuff to build too, right?" So I'm going to install this stuff on top of it, right? But we still have the original basic thing. And by default you can change this. But by default Mock will cash, excuse me, it will cash your original container. I'm going to call it a container, but it's a chroot. So that you can. You don't have to download these over and over and over again. It will just. But you do not want to do it completely additively. You don't want to save this one and use this one, right? Because remember the whole idea is that we want an absolutely minimal system that we then add our special BuildRequires onto. So, like, if I'm compiling Firefox, I don't want the BuildRequires from Bash, right? That might be extra stuff. And maybe I want only what I need, right? So basically the thing. This is recycled, but this is not what I'm.

David:

So that's more for repeatability, I guess. Then so that anybody building this independent of the whole collection can build it successfully with what is contained in the spec file. But I'm just wondering for efficiency on a build server, if you might want to just keep on aggregating these over and over. So, okay, I already added bison or I already added autoconf.

Skip Grube:

So DNF cashes the downloads, but no. That's something you actually do not want because you'll run into the problem we were talking about before where if I go to build Firefox and it requires it. Maybe I'm using Firefox as an example cause that's just what came to mind, but maybe Firefox actually needs texinfo, but it's not in its BuildRequires for some reason, right? And then I go, but I already have a thing. I've built Bash already, right? And now I'm going to build Firefox and this is already installed, right? And so that's actually actively bad because now Firefox built for me no problem. But it won't build for somebody else, right? Who doesn't have texinfo on it.

David:

Yep. That's and it's the legacy of the developer environment. You don't.

Skip Grube:

Yep.

David:

You don't necessarily want everything that is on it so you can prove that you can build it. It makes sense.

Skip Grube:

And that's why we, again, we cash where it's necessary. Like cause obviously you don't want to download this over and over again cause it's. They're not huge. But this is a fair number of packages here. But yeah for sure. We actively, when we go to do the build, we actively want these gone. When we do the next package we want to build. We want to start with the same basic system defined here every time no matter what, because otherwise you run into problems. So hang on, we got some more stuff here.

Capability of Mock [54:27]

Yep sure. Rephrase is a good point here. Yep. So you can and we're going to talk about that in a minute too. When we get to the hands on. But yeah. So by default, Mock will clean up for you. But you can tell it not to clean up, especially in case of errors, right? So like if Bash. We're building our Bash here and it ran into an issue and we want to check it out. What was going on in there? We can tell Mock, "Okay, build, but leave everything there. Don't clean it up and we'll look at it." Right? So, but generally speaking you don't want to do that. It's useful for debugging. It's not so we actively want to erase the stuff. And so yeah. Mock is really cool because you're effectively building a new system each time you build, right? And it's just. It's the way of the packaging. It's the and it's important to note too. Mock is a standard tool. It is not just a Rocky thing. It is Fedora, CentOS, SUSE, obviously Rocky. All these RPM distros use Mock and in fact, I'll show you something cool. I'm, look. I'm going to go ahead and load up my virtual machine here. Will be one of my, you can say I have a few.

I wasn't kidding when I said I'd do a lot of this wacky work here, but look that up there and I'm going to. I'm just going to hop in. Look at the Gooey later here. But if you and if you can follow along too with your systems at home if you are, if you have Mock installed, you can see I love looking at this. So you go to etc/mock and do an ls. You'll notice there are an awful lot of options for you, right? Like we have a Rocky 8 one that I posted here, but this is just like. These are. Some of these I haven't even heard of, right? But there's all the Fedoras here. So if you want to build against a Fedora 35, you pick the Fedora 35 config. If you want to build Fedora 36, you pick the 36 config and it will change what you depend on and all of your. All of these settings based on which one you're picking, we're kind of using a cut. A Rocky one that I've customized a little bit cause I think it's easier to read.

But yeah, there's lots of options. So you can. So we could build openSUSE packages on our Rocky system, no problem. Like it's useful like that. And you'll see here, obviously very important. Anybody who, again, has configured DNF will know what this looks like. And we are just using, right now, this one's pretty basic. We're just using the basic live production Rocky packages to do our builds. But you could get crazy with this and we're going to show you how to do that in a little bit. You can. It's super important though like this is incredibly important to point to the correct things that you want to build against. Because if I had this but then I threw a Fedora repo in here, my build would come out completely different, right? Because I'd be using a bunch of Fedora GCC and glibc and all kinds of cool Fedora tools to compile my thing. That may not be what I want, right?

So this is just so important here. I can't emphasize enough how important how your build time dependencies influence what comes out at the end, right? Like it and they have a huge effect, right? We could build for open source by changing all of this to the open source repositories, right? But obviously I don't want to do that, but you could though. So yeah, this is super. We're going to talk about this in a minute because we're going to need this. But this is a. You can build locally, just like this is a DNF config file. So you're allowed to point locally. No problem. Talk about that in a bit.

Arch 64 [58:27]

And yeah, this is just ours. Like our architecture, I'm assuming X-86_64. If anybody has. Raise your hand if you are, do not have. Like if you have an Arch System or something like the Arch 64 System that you're currently using. That you want to do this with, like, right now. Because otherwise I just assumed Intel for everybody. So we're doing this. But yeah. If I wanted to change this to Arch 64, I'd probably just go in here and swap out anywhere I saw x-86_64, swap it out for Arch 64. I believe it would work. I don't think there's any. But it doesn't sound like we need that. So yes. And that's the other thing you can build for. Based on what you point to and what your macros and stuff are. You can build for the different processor architectures. So, all right, let's see here. So let me go back to my notes cause I know that there were a lot of questions and thank you so much though. I love, again, I love questions.

Okay, so I think about what I've been doing. We have been talking about Mock a bunch. And so I think before I talk about other tools, because again, Mock. We use it and it's great. It's very nice for building packages, especially on our own. But after we're.

FWTS Hands-On Demonstration [1:00:03]

I think we're going to do our hands-on thing in a second here. So everybody get your engines ready. Get your PCs ready. And I think we will talk about mass production of packages. And after we do this hands-on thing. So I'm going to switch over to other notes here. Hang on a second. And so, little background here. So basically this is thanks to, shout out to Mike Carifio, and hang on. Jess says, I have a YouTube comment. Arch Linux. I think that was just a joke.

Arch Linux is not a joke. It's a wonderful system, but I want to say Arches. I mean processor architectures. But anyway, okay. So we have a request from Mike Carifio and we have. There's a package and it's called FWTS. And I looked into it and it's a neat little thing. I don't want to call it a, and Mike, you probably know more about it than I do, but it's like a hardware checker. Almost like a Bios and a CPI checker. Is that how you would describe it? Kind of verifies that your Bios is intact and it's running okay and everything's working. Yeah. And yeah. Mike has a need for this package, by the way, and it is not in Rocky Linux. And so let's bring it to him.

Oh my. Michael. Now it does not. As long as it's a Rocky system, it does not matter, right? So I have a VM right here. I am. I pull it up and I'm actually logged into it here. But yeah, it's. It can't be a container. That's the only. You can't just do Docker run Rocky Linux here cause you can. But it requires a setup that I'm not going to get into. But yes, as long as you've got a Rocky Linux system, internet connected, right? Should be good. So we are going to look at this FWTS package and again, it's in Fedora. So the first thing we're going to do is we're going to go and examine what we're after here. Eye on the prize, right? So we remember from our stuff here listed, that our package Git locations.

So it's in Fedora. Src.fedoraproject.org/rpms[packaged]. So we're going to go here and to Michael. Or yeah, Mike says, "It's FWTS." You see, I've cheated. I already visited it and here it is. Cool. They got their suite. And it looks like, yep, it's here. Firmware test suite. Nice. Cool. It's fancy and what we're going to do is we will go ahead. We're going to download some stuff. So I'm going to hop on our system here. Again, if you have your. If you have Mock, Git, and Curl or Wget installed. That's what you need. First thing we're going to do is we're going to git clone. I'm going to paste these commands in the meeting chat so you guys don't have to type stuff. Cause I hate typing stuff. So first thing, we want our code. Get me the code. Here we go and now we have an fwts on our system here, just threw that up there. I'm going to go into it, cd into it.

And so, oh, actually. Before I, okay. Backing up, before I do that. The other thing we're going to want on our system. What downloads here is this config file that I had. Where is it? And I think somebody posted a link to it in the meeting chat above. I want to. You can't just download it from GitLab, unfortunately. So if you click into it. Then, there you go. What I like to do is just copy file contents. Basically I want this file in your home directory on the system, right? So I'm in my home directory here. I'm at home, skip, and I'm just going to. What's it called? I don't know. We're going to rocky8-x86_64. Sure. You could name it anything .cfg. It really doesn't matter. Thank you. Good. Dave. Yep. Dave's got it. You can run wget to grab it as well. I'm just going to paste it in. Show another way to do it. But basically get this cfg file in your home directory. Don't care how you do it. There it is. I'm going to sip my tea and give everybody 20 seconds to do that.

It's good stuff. Yeah. In fact, give me five seconds. We'll turn this on. The window's no longer good enough light. The rain's coming. So. Okay, so now we've got our source code, we've got our Mock config file, and you'll notice there is a problem here and hang on. Got more messages coming through. Yes, I for sure would definitely use mine and not, like, just the straight EPEL one and I'll explain why in a minute. But yeah, so there's. We have an issue and that is you'll notice there's some files in here. I'm going to actually, that'll be easier to read.

So we've got a patch file, right? Which is like a code correction or whatever. We've got our spec file. There's a README and there's a source file, which is kind of funny, right? But you'll notice there's. Where's the fwts source, right? It's not here. This is garbage, right? And we're going to explain why that is in a minute. So this is what. So if you look at that source file, it looks kind of funny. And this is it here, right? This is what we want, but it's not really. This is not it though. It just has this big long SHA512 checksum. And explains how this works... So this file is not in this repository, it's what's called a lookaside. And Rocky Linux and CentOS also use this. And what we do is, instead of ho. This is. This all comes down to Git. We don't. If you've ever used Git before, you'll know files like this are not welcome, right?

Git is all about text, text comparison, doing patches, and inserting lines of text, removing lines of text. And this file is this big giant blob of archive source, right? And it does not belong in Git. It should not be in Git. And so what do we do? This source is a pointer to it, right? So we have. This is the name of the file and if anybody, I'm sure some people have already looked ahead here, but if you go back to the teaching repository. You go down to the useful references. Here are lookaside locations, right? And so we're doing the Fedora project. Let's take this pattern I have here. So we have some things to explore, right? So basically what's happening here is. Let me piece this in.

What is happening is Fedora stores this stuff in a special web directory off to the side. That's why it's called lookaside because it's kept aside, right? So it doesn't. We don't have to pollute Git with these giant tarballs or whatever. And it's also a useful way to. It allows us to. No, let me just go to the. Give me a second. Yeah. It allows us to separate what Fedora has done to the project from the original, right? So Fedora, for example, we have this .patch file here. This patch file was added by Red Hat or Fedora people, right? You can actually see Benjamin Burg, he did it. I don't know who that is. But basically what they're saying is that this Tarball file came from the original place where fwts is developed, right? I don't actually know where it is.

I think it's actually originally an Ubuntu thing. But, and then anything extra that Fedora, or Red Hat, or Rocky Linux has done to it is kept in patch files. So we. It's a nice way to keep it clean, right? So we don't have to. It's very easy to see like in Bash. For example, we have a Bash tarball that has the source files and then Red Hat patch files, right? And so we can keep them completely separate so that we, if we wanted to use the original Bash or we want to. We're very curious about a Red Hat or Rocky specific change made to Bash. We can easily find that by looking at the patches. We don't have to sift through the source file and compare it to the GNU one and figure out, like, the source file is pure, right?

Our source is pure. And so we're going to go fishing for this here. So this is source Fedora project or repo packages. I'm going to start here, all right? And we will, this is a warning, this is a big directory because you can see here, these are all the Fedora package tarball sources and other files. There's not just source tars, but there are, if a package had images or whatever, a lot of times those would be in there. Whatever. So we're going to look at it. Looking at our, let me put this on top here. So we are going to look for a package name. What's our package name? Fwts here. There it is. And then file name. And I've already clicked on it. You guys can see, but you'll see this is not even the file.

This is just a directory. But this is the one we want. I got it from. In fact, go back to our. Put this here too so we can. Okay, so this is what we're after here. File name. Yep. And then you'll see the file name. There's SHA512. Which I already clicked on. And then HASH. This is the HASH here. It's a big long string. And then the file name, right? And this is the way they do this to avoid collisions. So you can't just say slash file name because what if there was a file with the same name, right? That was different for whatever reason, cause that can happen. And so we have to use a unique hash value. And so what I'm going to do is I'm going to copy this and I'm going to paste it into the chat here.

Big long URL. So you guys copy it if you need to. And yeah, we're going to just grab it. You good? There we go. Okay. So now we have. And that. This file here can have multiple things in it. By the way, this source file, this is. This particular project happens to have just one, but there could be 10 things in here possibly. Depends on your project. So we've got our sources. We've got our stuff. Let me follow in my notes here, diligently. Got to pull my notes back up. Here we are.

There's that. There we go. Okay, I'm back. We've got our RPM Git. We've got. So the next thing we're going to do is you'll notice that there's a flat file structure here. And this is something that Rocky doesn't do, but Fedora and CentOS Stream have recently gone to do it. We are going to make two directories SPECS and SOURCES. I'm going to keep pasting this stuff so you guys can, again, follow along. Yes that is true. My guess, 10 hardballs would mean 10 lines in this thing. Yep, exactly. Yep, and so we're going to create our folders here and what I'm going to do is the spec file goes into SPECS. Everything else goes into sources. And I'm doing this off here.

Fedora Full HASH vs Rocky Short HASH [1:13:38]

How come Fedora stores the full HASH and Rocky only stores the short. Well, I want to say full HASH. That's the full SHA512 HASH. Rocky Store is the full SHA. I think it's for Rocky 8, it's the sha1sum. And for Rocky 9, I believe it's SHA-256. But it's not it. They are all full HASH's. They just use different algorithms, different lengths. But yeah. And, but each distro, as you can see, has different possibilities for patterns for these lookaside. And that's what makes it confusing. Honestly, I don't like that. But like Rocky, we are very simple. Like we literally just put the HASH in here, right? And you just download the HASH, like the file is named the HASH. And then you just rename the HASH to whatever file name was in your line there, right?

So obviously I know RHEL 8, 9, they organize it by package and branch. And then the new hotness is the Fedora way to do it. Which CentOS 9 Stream also does. Has this package name SHA. The SHA type HASH, like it's. I will share with you guys later an easier way to do this. Which, but for now I'm doing it the long way, the hard way, because it's important to understand where it comes from, right?

FWTS Hands-On Demonstration Continued [1:15:01]

So basically what we're going to do now is we've moved our spec file, our .spec file into the SPECS directory. We're going to move everything else into the SOURCES directory. I'm actually going to. Yep, mv * sources. I'm just going to keep pasting what I've been doing and then because we moved. I'm doing this clunky, like. We moved x into sources accidentally, we'll move it back here. So right, this is. Sorry, I did not paste properly.

There we go. And so now when you're left, it should be nice and clean. We've got a SOURCES, now we've got a SPECS, right? X is literally just a spec file. SOURCES are everything else. So now. So back to our chart here, pull this up. So we've got our. And in this case it was from Fedora, but we have our source code, and it's organized, and we've downloaded it. Now we have to make a source RPM, right? And this is pretty straightforward. Once you're in, again, you can follow along. We're in fwts, right? We're in the. We're inside the root, I guess of the RPM. We're going to run rpmbuild. If you don't have that, you can pseudo. Just going to have it already, but if it's not there, that's how you install it. I don't know why the actual package has a dash in it and the command it's just rpmbuild. Don't ask me. But yeah.

And we're going to bs, which is a build source. This means we want to build a source RPM. We don't want to do compilation or anything like that. We just want to take our spec file and our SOURCES and put them together for a source RPM. We're going to define a couple of things. I'll explain this in a minute. Slash macros we're defining, okay? And then I will paste this command for you. So here we go. So rpmbuild and we can look at the man page layer for this. But bs, build source, we want to build a source RPM. We're not interested in compiling at the moment. - - define, topdir, this is a special macro that says, "Hey, when we build our source RPM, we want to do it inside the fwts folder." The default is to do it in your home directory, which we don't want.

It creates these different sources, folders, and other things in your home directory. I don't want that. I want to keep everything in my fwts area here. Just something I use an awful lot. And then - - define 'dist dot .el8, right? You'll notice a lot of Rocky packages, Enterprise Linux 8. Lots of. In fact all Rocky packages will have this or something like it at the end of them. This just identifies that, "Hey, we're going to do this for Enterprise Linux, not for Fedora Linux." Fedora Linux would be like FC or could be just F37, F38, whatever. So they. It's used to identify which distro this is for, right? And then we point to our spec file. And if you do that, you should have a very simple rote, SRPMS and fwts and the ls here. You'll notice we created some more folders and this is part of why I did that topdir thing. I want to keep this stuff here. I don't care for it in my home directory. So hang on.

Yeah. Oh cool. Good job, Mike. Yep. Mike, your output is a little weird. Funky. I can read it though. I can read it. Yeah. Freaking Zoom, man. There's no way to, like, space it out. So yeah, cool. So now and you'll notice here it says "Wrote SRPMS, so. If we look at SRPMS here. Look at that. Hey, we have a source RPM and it's got our. This has our source and our specs in it. Everything we need to build, right? And so now what I'm going to do is I'm going to go into the capital build directory and I'm going to. Actually I don't have time. We'll just do Mock. I want to type some stuff in and then we will go over what it's about in a second.

Okay? I'm going to copy and paste this and we're going to examine this real quick. I think I'm going to throw it up. We'll throw it up here so it's a little easier to read. Got a scratch pad. Okay, so Mock, that's our special command that does this stuff. -v verbose. I like to see all the output from it. -r and then we pointed to our config file, right? I'm not sure why it's -r, but this is the. It's -r and then that's a special config file that we downloaded. This defines how we're going to build it, right, is our Rocky config file resultdir ./ that just means, "Hey, any RPMs and logs and stuff you make, put it in the current directory, right?" Because that's where we want to see it. Isolation simple, this tells it.

Do not use an nspawn container. Use just a basic chroot and that's what we want. Because Rocky, CentOS, and I'm not sure about Fedora, Sharif may know, you know Sharif? Yeah, I don't know. I know that Rocky and CentOS and RHEL all use simple chroot. They do not use containers for their build and that. Just that little option just makes it that way. And so this. We're trying to build a compatible for Rocky. So that's what we're doing. By default, if you don't have this, it will spit up an nspawn. The results will not look any different unless you have a complicated build problem. It's why we use the simple chroot. But anyway, and then you point to your SRPM wherever it is your source RPM that we made. So it's in the back one directory and that SRPMS folder there. And so everybody understands roughly what that means. Questions? Comments?

And you can go ahead and press enter. If you're doing this along with me. We're going to let it go. There's some text, but what it's doing here is it is creating a minimal system for us. Just like I said with those basic packages in it. It's going to install those basic packages and then it's going to turn its attention to our source RPM and attempt to build it after installing its BuildRequires. And I'm.

Issues With Install [1:22:40]

I cheated. I've already looked into the future. We're going to run into a problem and we're going to look at it and it's going to happen pretty quickly. Here we go. Look at this. And it's not even like a build problem, right? If you've ever had DNF issues, right? You'll be familiar with this. Look at this, "No matching package to install: 'dkms,'" "No matching package to install: 'libbsd-devel.'" So this package has BuildRequires that are not in Rocky Linux. We need them, right? And one of the places that I like to go fishing for packages, cause it's super. It covers a lot of distros and it's super useful is pkgs.org. Which I'm going to go to now.

This is a really cool site here. And what you'll see, what. It has got a lot of Linux distros here. All kinds of RPM based, well actually not even RPM, but there's some. There's Debian and Ubuntu in here too. And what you do is you enter, in this case, two missing things. I'm going to scratch the pad here. We need dkms and we need this other one here, libbsd-devel. And you can enter package names, or require, or provide into here and it will find them, but it'll find them across distros. Not just your current distro. Dkms for short here.

And we're not really interested in ALT Linux. Looking around here. So Enterprise Linux 8. So this is it. We're building for Rocky 8, by the way, because that's what our Mock config says. And no, there's a lot of. I don't think we need them. Hang on. Yeah. CUDA, no. I don't think we're interested in Nvidia. EPEL has it. Look at that. It's an EPEL dkms. Now and the other package we're looking for. I'll, spoilers, is also an EPEL. I'm going to talk a little bit about why we're not just going to enable EPEL. Cause we totally could, right? Like we could go to our Mock config wherever it is here and we could say slap on. We could slap on EPEL here at the end, right? And just. And we would have. Our build would be given special EPEL powers and we would. It would find these things in EPEL.

I do not want to use EPEL for builds and that is because it's, I don't want to say it's not reliable, but it's liable to change, right? Because just because a package is an EPEL today does not mean that the Fedora maintainer of that package will decide, "Well, packaging for EPEL is too difficult, I'm not going to put it in there." Right? And then you'll have problems because you will not have EPEL and will not have the thing that you need, right? And so, generally speaking, in the package world, at least in the Rocky world, if we are going to build a package, you're going to build if you need dependencies for that package. You have to build the things it depends on as well, right? And it's packages all the way down. And so what we're going to do and this other package here. You can see it's called libbsd. I should have done that first cause it's going to be much shorter.

But you'll see, let's search here. It's down here. So you see it's not in Rocky or RHEL or anything like that, but it is in EPEL. EPEL for those who don't know is Extra Packages for Enterprise Linux. They have. It's Fedoras stuff that they compile for Rocky, and RHEL, and those things as bonuses, right? But again, it's completely volunteer maintained. That's all. You could. Tomorrow, whoever's doing this, could be like, "You know what, I don't want to put libbsd in EPEL anymore. I'm going to take it out." Right? So in order to make sure that our package is built with the strong foundation we are going to. We can use the sources from EPEL here, and import them, and compile them, but we're not going to rely on EPEL, right? It's just. Generally speaking, it's a bad idea.

And also it gives me a chance to show you how to do this. So like if I were if I. If this were an actual SIG in Rocky Linux, right? And I was actually. I wanted to provide this package as part of my special interest group, so I would be doing this. I would not rely on EPEL. And that's what we're going to pretend here. So we do need these packages. We are going to take a little shortcut here because we're testing and we're just doing a test build here. So what we're going to do is. So we know where they are and if you go over here back to the pkgs.org, we need there. We need the source to these guys, right? And if you go down here, for example, here's a package. Yeah, here's the source package here. So there's a downloadfedoraproject.org. We. They already have. They have a source RPM published here and we could go to Fedora and actually clone this just like we're doing this current thing and build it and everything. We're just going to, for testing's sake, we're going to use the source RPMs that are already built for us of these dependencies. We're just going to build them. And so if I go to this URL here and you give everybody the URL. See we need this libbsd package.

Yep, there it is. Libbsd src.rpm. And here's something cool. So what I'm going to do. I need these two packages. So I'm going to mkdir libbsd and I'm going to mkdir dkms. These are the two guys I'm going to. Then what I'm going to do, I'm going to run Mock again with the same config file. I am. I want libbsd so I'm going to resultdir is going to be libbsd. Not my current directory. I'm going to be simple. And so Mock is cool and so is RPM. You can. You don't need to download anything, you can feed it the url. So I'm going to paste this in a second here. Let me just go back to. I can just be like them. I don't even need to download this. Just give it this. We're going to let it spin for a second while I go up here. This one's probably good. Yep. Awesome. Uh-oh. So for those keeping score at home. We're going to try building with libbsd. Didn't work. We're going to go even deeper, but hang on a second. We're also going to do it. Go into here, into the b's dkms. Send me in here somewhere. Use the Fedora RPM sources here. There it is.

Let's try that first cause I'll bet that one doesn't have dependencies. I'm going to paste my command here. So dkms and this time change my result directory. I want it in dkms. And we're going to let this go for just a second here. So let's, while this is going. Actually we're going to map this out. Because this is a common task. And so we want fwts. Fwts relies on dkms and libbsd-devel, right? So it looks like we just built dkms. It. Everything looks good. Here. I'll give you the command to build dkms. Here we go. I know we're rapid fire in here because I've still got a little bit extra to show. Yeah, so we totally built dkms. Oops.

RPMs. Hey, awesome. Look at that. Bam! So we built one thing, but if we go back to what we did before with that libbsd stuff. Build again, have failed again. We'll look at them. Look at what happened there and fail. Good. And now we have another problem here. No matching package to install Libmd-devel. So back to our map here. Okay, so this one. So ftws, I'm indenting here you can see requires these two. This one indent further requires here. Like, so, we need to build this, then these two, and then we can build this. So go to. We're just going to go find libmd in our Fedora packages and it'll be in the L's. Oh, I got the L's here, libmd. Here it is.

Well our. Not our EPEL packages, but same difference I suppose. And then now we're going to do the same thing. Make a libmd and then I'm going to fire up another Mock command. Have, yep, give it a different url, give it a different result area. Libmd for you guys and go. Hopefully we don't have to go deeper, right? But you can see how this can get complicated, right? And it's. Ha! Look at that. Well, it looks like we're compiling here. That's good. Success. Yep. Whenever you get to this part, it says. Now it's spitting out requirements and things for the built RPMs. You know, you're golden. So now we just built it and yep. So we have our libmd. Great, awesome. Now we are going to. So let's check our map again. Libmd done. Now that we built libmd we should be able to build this one. So let's go back to our build the libbsd command or the libbsd, right? I'm going to press it here and we're going to have a pop quiz. Pop quiz. It failed. Same error with no libmd-devel. Does anybody know why it failed? Any ideas?

David:

I know

Skip Grube:

Why did it fail, Dave?

David:

I think I know. I might be wrong. I think that it's not an in a. It's not a repo. You got to put it in a repo.

Skip Grube:

Exactly right? So we're using this Mock configure, guess what? We. Every time we build, just because we built something does not mean we get to use it, right? We are still using this same store of RPMs in order to build our stuff. Nothing is. We have not put this in the base os rap stream. Nothing's changed, right? And so in order to do that, you'll notice down. Actually I'm going to look at your config file, which I'm going to edit now. So this is the Mock config file in your home directory that you. That we downloaded from my little Git repo. You'll notice I have a little local repo here. Look at this and you'll notice enabled=0. That means that it's not enabled, right? So what I'm going to do, enabled=1. We want local repo powers, right? And what we're going to do is it has to exist, right? So I'm going to create a folder there. Yep. Because we have to. Yep, we have to use our stuff here. Password here.

I'm going to change it to myself. I'll paste that here. Please don't use skip, use your own name, but username. But now I can copy stuff into it, right? And so what I'm going to do is I'm going to copy kkms into /opt/localrepo. I'm going to copy lobmd into /opt/localrepo, right? So also pacing everything I do here, great. And so now up localrepo. I've got these things, right? But that's not good enough. The last thing we need to do is to turn this collection of directories and files into an RPM repository that DNF can read, right? Because it's not automatic. And so what we'll do. I like to do createrepo. This command createrepo was installed whenever you installed Mock. It's actually its own package, but it creates RPM repositories. It's very simple. It's actually. It's got a lot of options, but we don't need anything fancy.

We are just going to create this local repo. And v is verbose. And you'll see, look at that. It says adding packages. And it added all this stuff that you have done so far to the local repo. So now, because I have in my Mock config, I have added local repo powers and I've copied my stuff and I've created a local repository with my requirements in it. Now we're going to do it again. Going to do the libbsd build, because now we have. If you go into our local repo, we have that libmd that we built should be available now. And let's see it. Let's see it happen here.

We're going to build libbsd again. Go! Yep. And looks like we're. I can already tell I got further. Look at that. Go! Now we're doing it like a make. Like a. Doing some compilation here, compile stuff. Go! And that did not take that long. That's awesome. And yeah, looks like the proof is in the pudding. If we libbsd, right? There it is. Great. So now we're going to do the same thing. We're going to take that libbsd and we're going to put it carted off to the local repo, because we need that too. Remember?

Oops, I did not. Yeah. There we go. Yep. And now. And again DNF does not know about it. We got to create a repo again. Every time you do an update, just yeah. Create a repo. And now you'll see some of our libbsd stuff was put in there. Great. And now we've done all this. I'm going to remove the log files here and now we're going to go all the way back to our original RPM build thing, right? So boy, we're going to go. You know what? I'm just going to create it again. Mock -v -r. Really good at this. After a while, it's like muscle memory. And then we're going to go back to SRPMS. Fwts awesome. We're going to build fwts. But now with special powers, because we have built the requirements for it. Our BuildRequires should be fulfilled. And there is. And the fact that we're actually installing stuff enough here tells me that it's probably working. Yep. Now we're going to build this one.

Go! So yep. We're taking the package from Fedora and we're using our Mock config setup with Rocky Linux repost to build it for Rocky Linux. And I'll mention that this one, this particular package, is pretty straightforward. We're lucky. You're lucky, Mike. Cause you want this. There are ones in Fedora that I can, about, guarantee you will not compile properly without code changes or without some kind of spec file change or something. But I investigated this one beforehand and it actually works really well with just building as is, but with different build tools. So Dave, actually, Dave, no, it's not. So here's the thing with local repo, it's like, this is, it's really. I should have called it something different. This is really our results repo, right? So if we were. If I was interested in publishing my work here, right? And making fwts available to everyone, like, in Rocky Linux via a SIG or via an extra repo or something, I would want to take the things I put into local repo and put them on the internet, right?

Because I want people to have mine. These are. This is what I produced, right? This is my product. I've. If you want to think about it that way, like I have you know, I'm doing my compilation and because those other RPMs are inside that repo, other people could build my same fwts as long as they're pointing to it, right? So, no, it's you. Definitely the Mock build root gets blown away each time. And I'll show that. But that happens inside of our lib/mock here, right? And actually I probably should have shown that, but yeah, if you take a gander.

Can A Second Package Interfere With Local Repo? [1:42:27]

David:

Sorry. Well, I guess my question is more like. Then if you decided after you got done with this that there's another package that Mike wants and we go and we build that second package is there a danger that whatever we built in the local repo could interfere with?

Skip Grube:

Depends on what you're building and what it is. In this case and in most cases, I would say not really. Because if a package required any of them. If the. If a package required any of those three packages that we just compiled, right? We obviously. We can't get them from Rocky. We would need to build them again from the local repo. And so I think in this case it would be pretty safe. That's not always the case. But in general though I think you'd be fine. And there's also a question of separation, right? Maybe this is one project and maybe we wanted to go do another project, right? And that would be a completely different repo we would set up with dependencies and so forth, but maybe not. Maybe we start an extra repository, right? With just extra stuff that we want. We just throw it in the same place. So it really is. It's kind of a situational thing. But no, that's not inherently a problem. I wouldn't say. But again, it depends on what you're doing, right? But yeah.

You'll see in /var/lib/mock/ there are. That is where your chroot is. And so like here, the 8 localbuild bootstrap, you'll see there's, like, a root folder in here and this is the temporary system that was set up the remains of it. Anyway, that was set up to do your build if you ever want to look in there. It's not that interesting. It's pretty basic to root whatever. So now this is awesome. We have fwts as well. Let me just. I'm going to put fwts in a folder too. Fwts and just like before, cause I'm going to just demo something here. A copy fwts off the local repo. Just like I did before, createrepo there. And then I'm going to do a little rough and tumble publishing here. Hang on a second. I happen to have a home server here and bear with me for a moment.

So now I'm just going to do it. This is just to demonstrate something that I think a lot of people forget is that I'm going to, in fact, I'm not going to do that. Okay, so now I'm going to give you guys the URL for this. This says skip Rocky. Hey, there it is.

Yeah, sorry. Yep. So this is the stuff that was just made here. And you can see there's. Here's the folders that we made our RPMs with along with we copied the build logs too. You don't have to do that, but yeah. And there's a little folder here called repo data that when we ran that createrepo command, this what it created, right? And so like, if I were to take this URL and add it to my DNF, right? It will work. I can just DNF install it and we can, in fact, let's do that or we can. You can just install the local RPM as well. But yeah, let's just do this old.

I'll paste this as well. They're not signed, so I'm going to do gpgcheck. Bad idea in general, but this is just for testing. It's just a fun thing. I think this should be good enough here now on my real system. Not the Mock system. Hey, look at that. Yes. Hey, let me give you guys the old, let's see. Awesome. And then now I have my own little cottage industry here, right?

So now I have fwts. I can look at this. It's got all kinds of cool stuff here. It's doing, like, signatures and I don't even know. Like all the tests and stuff it runs. But I assume you have to run it as root for a lot of this to work. But, oh, test fan status, load system, check the fan status. This is awesome. Freaking A! What have you dug up here, Mike? Awesome. Yeah, so this, but this is how. And so really you can kind of extrapolate a little bit if we were to keep going, right? If we were to build pick a package and start building dependency after dependency, eventually you've got Rocky Linux. And this is how it was originally started, right? Because you got to go real deep, but yeah. And there's tools we have and we're not. I'm out of time here, so we're not going to talk in depth about them, but there are tools we have to do this on an industrial scale, right?

Limitations to Using Mock [1:41:39]

Because we've got 3-4,000 packages and we can't just. I'm only one guy and Mustafa and Louis and the other people on the build team we're people and we ain't got time for Mock this and Mock that. We can't do that. So we have large systems that are meant to build many packages in a row against each other, right? And do all of this stuff that we've been doing manually that will make it more automatic. And I'm going to show off a little. So there's a. You can see links to them in. Hang on a second. I don't believe I. Wow. Okay, I'm going to update this with links to. In the references where the build systems are because I thought I put that in there and apparently I didn't. But yeah, Rocky 8 was built on Koji. You can find it. You can go to koji.rockylinux.org and this again, you can kind of see our build if you could click on builds. Here. Let me put the thing in here for you guys.

This is a system for, like, what we are doing, mass building of packages. Literally underneath the covers, it just calls Mock, right? Just like we are. So yeah, it's got its own Mock config that it uses for Rocky 8 and it just. It builds it or it builds packages like this. And you can see, actually, you'll notice that in our build thing you know, we have these log files here. There's. Here's build.log. Which is all the compilation stuff that happened. And there's root.dot log, which is. Here's the DNF stuff that was set up. If you go to Koji and I don't know, click on anything. You'll look here, tigervnc, right? This is the vnc package that Rocky comes with.

You'll see here's our RPMs that were produced and you'll notice, "Hey, this is exactly, this is what we were doing," right? It is just done on an industrial scale and it produces, like this build.log. It looks like the build for tigervnc exactly as Mock produced it, right? Or exactly if the RPM was built. So yeah, you get the exact same output and you can emulate what it's doing, right? So if you want to build stuff on your own, you can point your repos, get it set up with Rocky repos and so forth. And you can build just like real Rocky's doing, right? So a really super useful tool. Especially if you want. If you're like me, you have to do a lot of troubleshooting on your own system.

You can yeah. I can look at what the error was. Sometimes in here I have to look at, "Like oh, we had a build error, what happened?" And I'll look at the log file in here and I'll be, "Okay, well I'm going to go into my system and with my Mock config and set it up and see what happens." Stuff I get the same error. Maybe I'll fiddle with the way the dependencies are done. Maybe I'll. There's all kinds of things in that Mock config you can do. Point to a different repo perhaps or fiddle with version numbers and yeah. And see if it works with something different, right? So yeah. Was, oh, good job, Mike. Yep. Yep. That's the version we got from the Fedora match. Should match perfectly cause we didn't change anything.

So yeah. So we got Scott, we got Koji here. I'm going to show Rocky 9 is built with our new system of. Engineered by Mustafa. Do you guys know Mustafa? He's great. Paridot and this is where SIGs and Rocky 9 are currently built. We want to get Rocky 8 here. But porting it over is a project. A large project. So I haven't got that done. We got two systems really, but the Distros are completely independent of each other. So it doesn't really, it doesn't matter that much, but Peridot, a little broader than Koji. It actually will import stuff as well, like from git.centos.org, like we were talking about in addition to building. So and you can see again, I think some of you have seen me show this off before, where you have builds and peridot here.

This is. You can tell oftentimes Lewis does a lot of this, but I'm in here too, a bunch. And like we can. You can see if you see a string of red marks or red and yellow marks probably means Lewis or Skip is having a bad day. So be nice to us. We're. Some update came in from Red Hat and it's fried for some reason or whatever, right? And we can't build it and we're probably tearing our hair out. But we. So but yeah and again, just and peridot, just like Koji, just like we are doing, if we look at let's look at a nice small one zlib, that's fine. Zlib's fun, right? We look at some of these build logs for architectures and you'll see go all the way up here.

Because it has this, like, web interface that you look at. So you'll see this is just Mock output here. This happened. We've got. We're building our minimal system just like we did with our Mock. And then it goes down there's a bunch of DNF output that happens here. These are the exact things that are being used to build this project or this package here and then we are and then here is the compile output, right? So if you want to look at exactly what happened, what went on here. This and this is mostly useful when there's an error or something that we have to fix. But this is open though. This is like. There's no login or anything required for this. I encourage you, if you're interested, go check that out. Maybe if you want to build a package yourself and you've seen that we've built it or something. Go see it, right?

You know? Go check out your build logs and look at our build logs. They should be pretty similar. So yeah. So I'm about to talk now and we're about done. But I'm going to hang out for a little bit if anybody questions or comments or anything like that. I hope you learned something new. I know a lot of people are plenty experienced at this kind of thing already, but try to start from the ground floor. Make sure everybody can have a chance to do this. So sir. Awesome. Thank you. Cool. Anything else? No? Well, I'm going to stick around and drink tea for about three or four minutes here and anybody wants to chat? Good stuff.

David:

This is really cool, Skip. Thanks.

What Skip Does For CIQ [1:56:20]

Skip Grube:

Okay. Yeah, good job. I just want to show examples of this. The power of this Mock config stuff, right? So one of the things that I'm responsible for at CIQ is I go and I. We have an offering where we support Rocky Linux 8.6 which was retired in November. Last November. And we have a long-term support where we're saying, "Okay company, pay us some money. We'll build Rocky 8.6 packages and security updates and stuff for you." And so what I'm really doing is I'm taking patches and sometimes actually oftentimes the new 8.7 package and I'm going in here and I will point this to the vault. Right to the or the 8.6 retired packages are all the tools and things, right? All the. If you guys are familiar with the downloadrocky.linux.org. There's a 8.6 vault and I'll compile it against 8.6 to ensure compatibility. And that's how this works, right? Like, that's sometimes there. Sometimes I have to fix it because a change made in 8.7 won't compile against 8.6 anymore. So we have to do a little actual work. But that's really what I'm. That's all it is though, right? Is that hey, we need to compile with the stuff that was in Rocky Linux 8.6, but we need this new version with this special security fix, right? So and just like we took the Fedora source and compiled against Rocky. I'm my chain or I'm going much smaller. 8.7 to 8.6 is much different than going from Fedora to 8.7. But it's the same kind of deal.

When we have more questions. Yeah we have more questions, I like that. That's it. You will for sure. Check this out on the chat.rockylinux.org. I hang out a lot in a bunch of channels, but the development channel is probably the best place. And it's not just me. There's all kinds of people that know more than me, quite frankly. I'm a junior to mid-level packager, right? And just here to share some basics, but we've got some real experts involved with this project. Quite frankly, they are awesome at what they do. So I'm going to put that link up there. But yeah our Mattermost and this is also if you're into IRC. It's also. I think it's the bear chat. I believe that's the. Yeah, there you go. So Jeremiah, you can see the WeChat a lot there about a lot of things, quite frankly. But yeah, if you have packaging questions, I think that would be the place to go. I'm there most of the day. Sometimes I'm there too much, I run my mouth off and chat instead of working. Right? Everybody does that. But yeah, that's happy to answer questions whenever I get to them.

Oh, this is great. I'm glad. Oh is that a. Oh no, that's not a hand. That's a. Clapping now. Okay. Thank you. Silent clap everybody a silent clap. Thank you. I appreciate it. No, this is yeah. This is really nice, but again, I'm happy to talk about this stuff, but yeah. Go. You can go and you can build your own packages now. It's awesome, right? There's. We just scratched the surface. There's a lot more to it. We dive into spec files, dive into the ordering of repos and stuff, but yeah. This is. It's a good way to get started though. So cool. I guess we're over now. So I'm going to go ahead and.