Secure Boot: Getting to know your frenemy: All Things Open 2025, Michael Young

Get it to where we could build it, distribute it. But Secure Boot was one of the things I believe in the first release was not available, correct? One of my buddies here. So, it was not available with the first release. Why? Because there was a process to be able to support Secure Boot. And we'll get into that a little bit later. But it was a lot of learning, having to figure out how to Secure Boot work. What's the process for somebody who wants to distribute an operating system? What do we have to run through in the Linux community to be able to support Secure Boot? And we started distributing Rocky, and the community was like: but I need Secure Boot!

I need Secure Boot. To be able to switch to Rocky Linux, I need to have Secure Boot on this. So, the community was very much anxious to get Secure Boot. So there was a lot of work put into - the community put a lot of work into getting it, going through the process to be able to get Secure Boot. And we'll go in a little bit later as to what that process was and why it takes time sometimes, and it's one of those pain points that we all have. Another story: I work at CIQ. We help support Rocky Linux, but we have customers that say, okay, we love the community version of Rocky Linux, but we need some changes.

We want something different. Okay, great. We'll help you out. But what we provide requires us, for instance, to rebuild a kernel or provide a driver. Now, all of a sudden, Secure Boot isn't working for them. Well, as a vendor, what do we do? Well, we had to go through the process of learning how Secure Boot works to be able to support our customers. So you have two different environments: you’ve got community; you’ve got customers. We all need Secure Boot. So these stories are to help us understand why we don't like Secure Boot - it's a pain - but some of us are required to work with it.

And so hopefully throughout this talk, you'll get a better understanding of how it works. It might help you to further your opinion that Secure Boot stinks. Yeah, my buddy Skip here, he's already going eh [thumbs down]. Okay. But we have to work with it. Skip and I, we have to work with it all the time. So let's get into what is Secure Boot. There is a good majority here that didn't understand how it works. So hopefully this gives a bit of a history lesson, but also gets into where we are today. So what is Secure Boot? Going back to 1970s, there was a firmware on the machines that was programmed to look for a bootloader.

And where was the bootloader? It was on your storage. It was stored in the master boot record. Right. And so great, that's how we boot up. But we all know what happens with time. With time, some bootloaders that we didn't want running, bootloaders we didn't trust, were making their ways into the master boot record. And so then when you boot up that system, you're running untrusted malicious code. So that was an issue that needed to be solved. Also, as computers advanced, the BIOS started to have its limitations on what it could do. So it wasn't until the early 2000s - 1970s to early 2000s - when there was a specification that was released.

UEFI is the acronym and it stands for Unified Extensible Firmware Interface. And this was to tackle the modern computers and get rid of those limitations that the BIOS had. And it was in 2006 when what was added to the spec was this feature that we all know today as Secure Boot. It was added in 2006. What was its goal? Its goal was to try to prevent untrusted code from loading at boot time. And how did it do this? Well, it does it by using digital signatures to validate the authenticity, the source, and integrity of that code that's being loaded. All of this - the goal - remember the goal here in the specification is to prevent malicious code from running.

That was the goal that was set out by putting out the specification. I jumped ahead. So there it is on the slides. Okay. Now, how does Secure Boot work? This is going to be a little complicated, but my next slide is going to hopefully make it a little simpler in terms. Okay. But every machine that we receive, every server, every computer, they have the firmware already loaded on the system. Inside that firmware, you have a set of keys. Now, these are the technical terms. You have a platform key. You have a key enrollment key. Then you have databases, right. And those databases contain the DB1 - has okay, this is okay.

This is whitelisted. DBX, though, these are bad. We don't want anything that matches these hashes. These signatures do not allow it to run. Okay. These are the certs or the firmware itself or the bootloader itself. All of that's on your system before you even get to whatever bootloader you're deciding to use, okay? So that's on every computer today. So the database here that's already mentioned during each stage of the boot process, the digests of the firmware, bootloader, kernel, kernel drivers and other boot chain artifacts are calculated and compared. And that's what it's doing before it even gets to run your bootloader. It's trying to say: hey, I'm going to set myself up, be ready, and then check this bootloader, make sure I'm allowed, I should be running it.

So this is the slide that hopefully simplifies a little bit if we think about: computer is the OEM, the manufacturer, right. So they have on their PK key. So they have their key on there. That then is a start of the rest of these keys, right. And it's allowed to update the KEK, okay, or in this case, it's a Microsoft key that's on our systems. And then you have your keys that are allowed, keys not allowed. Hopefully this puts in a little more terms that we're more familiar with. So you have OEM’s key, Microsoft’s key, alright. Pretty much that's the way. But now you might be going: why Microsoft?

Why is there a Microsoft key in here? Well, this has to do with back in 2011, Microsoft announced that with the release of Windows 8, any machines that were going to be certified to run Windows needed to ship with their public key and have Secure Boot enabled. Those of us here in Linux, think that went over too well? No. Right. So it was not. People were not happy. That all said. And now every machine with Microsoft, I can't run my system. We'll get into how this works with Linux then. So, it was clarified afterwards, after the uproar in the community, that systems that were certified needed to enter or have a custom mode where you can change the keys or you can just disable Secure Boot, which a good majority here, just turn it off.

But this is how all computers and servers started to have, by default in their firmware, Microsoft's keys, and that works great for Microsoft. It works great for Windows, right? And running their bootloaders, everything just works. And the end user would not know, right? But that brings us to here to Linux. What about Linux? How does this work for us? So we have here: same setup. But what the key is to getting it to work with Linux - because we're not going to have a bootloader signed by Microsoft, right, and there's different bootloaders out there, different distributions - we're not going to have anything that's going to pass that chain of trust - so what we have is called a shim (in the green box).

We have a shim, which is a small binary piece of code. And what it does is it actually gets signed, that shim gets signed by Microsoft. And because it's signed by Microsoft, now it's part of that chain of trust. Now, what is the shim? I already mentioned it's a binary, but inside that shim there is a certificate of authority. And because that certificate authority - so think of it, the shim is signed by Microsoft. It loads that certificate authority that’s inside the shim. Now it becomes part of that chain of trust. Right. And anything that's been signed by that certificate authority inside the shim, such as your bootloader, is now able to load.

And there's different drivers that are signed. For instance, in Rocky Linux, we have, our grub, our bootloader; we have kernel and the kernel drivers, and we have the firmware update, fwupd. So that all needs to be signed. But it's signed by what? It’s signed by Rocky's certificate of authority. And because that certificate of authority is in the shim and the shim has been signed by Microsoft, now it just works, right? I'm making it sound too simple, but that's the idea. That's the idea: that shim is what we need in Linux land in order to load our operating systems and keep Secure Boot enabled. So next question is: when does a vendor need to get a shim?

Notice I said vendor, right? On an individual basis, you could probably do a self-signed cert. There are ways. I'm not going to get into it here, but I've done it in testing, where I can - because, remember, there's a custom mode; you can load your own self-signed cert. So there's that too, which I'm going to get into. There's the machine owner key, but there is a way to act like you’re Microsoft: put your own key in self-signed, and then the rest of the way, it's good. It's that custom imogen that I mentioned that's available in the firmware. You could do that. But trying to convince a customer to do that across their fleet - that doesn't fly, that doesn't work.

So when you need to distribute something to your fleet, to your customers, you need to go through the process of having a shim so you can keep security turned on. And like I said, it's becoming a requirement in a lot of environments. You need to support Secure Boot. So, that was to answer the question: when do we need one? But what's the process? How do you get your shim signed by Microsoft? And there's a bit of a process. My friend Skip here is already laughing because we've been through it. So, how do we do this? Well, you first have to join the Microsoft hardware program. Okay?

You have to go sign up because, through that program, that's how you're going to get your shim signed by Microsoft. But when you sign up, you also have to go get your own EV code signing cert. Why? Well, because what you submit to Microsoft needs to be signed, and that's how shell is going to confirm that you are who you say you are, who the organization that you say you belong to. Then you have to go through - so that's just setting up, getting into the program on Microsoft, right? Then you have a whole process called a shim review. Now if anybody's interested, they can go to this URL that I have here on the screen.

It's on GitHub: https://github.com/rhboot/shim-review. There's a committee there. They are approved to review the shims, approve the shims, and you have to go through this process before you even submit it to Microsoft to get signed by them. They have all the documentation there, what’s required. They ask a lot of questions: they want to ask about how you're storing your keys, how you're doing your key signing, you know, all the good stuff. It's good because it's confirming that they're just not signing anybody's shim, right? Again, the goal of Secure Boot is to try to stop malicious code from running, right. So it's all down to trust. And we're trying to build this chain of trust here.

So this shim review, which is quite extensive, it can take months. And this is one of our pain points that we've had that I want to highlight on the community side and on the commercial side, on CIQ side, it can take months because it's just volunteers and there's only a select few that are allowed to actually - you have some that will help with the review, but they don't have the final signoff. You need someone that can do the final signoffs, especially if you're a brand new vendor. If they haven't seen you before, you're going to go through extra scrutiny to make sure you are legit.

So, that's a pain point in this process. So first pain point: it's all depending on Microsoft, right? Second pain point is that there's a whole review process you have to go through. Once you get through that process and it's approved by the committee, then you can submit it over to Microsoft. And I’ve got to say the signing process, maybe it takes a max of one week. It doesn't take too long on Microsoft's part to actually sign, but to get to that point is the pain point for many. To submit, you have to create a .cab file that has your unsigned shim, and then Microsoft gets it.

They confirm that it's been approved by the committee. And then what you get back is you receive a shim that’s signed by Microsoft. Now you can install it. Your boot chain works. Your kernels that you've signed with your keys, your drivers, everything, as long as it’s all part of that chain of trust, now work, and you can keep Secure Boot turned on. How about our auditory modules, our drivers? What do we do?

Well, if they're not signed by a trusted CA in the database, the keys, it will not load, if you have Secure Boot turned on, and that's also a big pain point for many, many who want to have their Nvidia drivers running or some other driver running, and I have Secure Boot on - why is my driver not - a lot of the community gets tripped up over this - they have secure root turned on and they go: why is this not loading? Why? It's not like - you have to keep it on. Yes, you do. So it won't load, right? But what can you do? You can create your own signing key.

And then that's where MOK comes in, that was mentioned: machine owner key. You can load your cert into that list. It's an NVRAM and it's persistent, so upon boots. But that feature is because I'm the owner of this machine, and I am telling you, I trust this key. Please load it into this list. Right. But as a vendor, I need to be able to run this on any machine. That's where, you know, the shim MOK doesn't quite work from a vendor perspective. But this is great on an individual perspective, right? Or get your vendor to be able to provide drivers that have been signed, right? Then you don't have to worry about this.

But for most of us here, I think we're playing around with this stuff, trying to understand it. It's okay. You can create your own signing key. You'll enroll it into what's called MOK manager - machine owner key manager - and use this same key to sign your kernel drivers. But there is one gotcha. If you go and update your kernel or your drivers, you don't re-sign it? Now you're back to it not loading again. So you have to remember, if you're using your own self-signed key to re-sign, and anything that needs to be part of that chain of trust, that's where maybe some here - I don't know if anybody here has heard of DKMS?

And it's supposed to try to help, right? To make that a little more automatic. It's been improved over the years. Nowadays, if a module supports being built by DKMS, it will look and see: Hey, do I have a key? If I don't have a key for signing this module, it will create a self-signed one for you and then sign the module. But you have to enroll it. You have to make sure not to forget that step of enrolling the key that DKMS created for you into MOK manager. But once you do that, when you have kernel updates, it'll automatically rebuild your module and sign it as an end user.

You don't have to think about it. So that's one benefit that DKMS provides. So that's what this next slide is talking about here. So there's been different solutions trying to ease the pain of running with Secure Boot enabled, especially for those of us that are required to keep it turned on. I'm going to attempt to do a demo here and, the point of this talk was not only for those who already know, a lot here don't know how Secure Boot worked. So this is, I think, a good demo just to kind of show how to get into it, how to play with it, how to test it.

A lot of times people will go: Oh, I need to have bare metal machines so I can enable Secure Boot. I've been doing most of my testing with virtual machines, and a lot of times folks get tripped up over one little detail. Now I have a Mac here, but it also works on KVM as well. What you want to do is let me see if I can push this over to the screen. I'm going to have to stop my slide. Okay. Bear with me. Okay, so on the Mac, I'm using UTM, right. And if I'm going to create a new machine, right, go through all these steps.

But what I want everyone to pay attention to is how to make sure, at least for UTM, how you can do this, and have Secure Boot enabled, for you. Over on, for instance, one of my favorite distros, if you haven't noticed, is Rocky Linux. Over there, if you're running KVM - I'm sorry - Cockpit, right. If you're using Cockpit to create your virtual machines, the process is going to be similar, but of course, it's a different layout, different terminology. But the key is please keep in mind when you're setting up your virtual machine, don't go create right away. Select the option to be able to go in and change the settings before you start up that machine.

And in Cockpit, when you're looking at the virtual machine setting, there's firmware. It defaults to BIOS. You want to change that to UEFI. If you switch that to UEFI, then you're going to have Secure Boot enabled in your environment. You can play with turning it on, turning it off, playing with the keys. So I just want to point that out for those that are getting started and want to learn more, dive into this further. On UTM here. It's as simple as - don't want that big - and I'm not going to run through the whole, I'm just going to show you guys the setting. Okay. So this option here: open VM settings.

So when I go save - Now I want to point out: so UTM in order to enable Secure Boot, you got to turn on TPM 2.0 and that's it. And then when you go to do the install, you'll have Secure Boot turned on. But Secure Boot does not depend on TPM, right. I think there has been a change that for 2.0 - I just want to make it clear it's not that they depend on each other, but they do work together very nicely. But for UTM to get Secure Boot turned on, you have to turn on the TPM module there. But just a tip I wanted to share with the community here.

For those that want to get into playing around with Secure Boot. Now, I already have a machine up and running here, and I was going to get into it, if I can get my mouse. Because I want to show you guys some things that you can do to play around and get a better understanding. So some commands, some basic commands that you can run: mokutil --sb-state tells me this virtual machine that I'm running has Secure Boot enabled. Now, if you're like, remember earlier I talked about keys? Maybe some of that kind of went over our heads because I went through it a little quick. You can actually see, using this utility, you can see - remember we talked about Microsoft's key, right?

So we can see this key here. Right here, there's Microsoft's keys. Right. So you can check out what actually is being loaded. And there's a bunch of Microsoft keys. There's, if you want to see the DB, so this is again, you have Microsoft’s keys in here, their root CAs. I believe in here is the UEFI CA right here. So it's a good way to learn and understand a little better what's being loaded. Now I talked about MOK keys or, let's say, Rocky keys. If you go: list enrolled - you're going to see in here: Rocky Enterprise Software Foundation. So that's how you know: oh, this is what allows my distro, my distribution to boot up with Secure Boot turned on.

In the case of CIQ, you would see our CIQ key has been enrolled in here. So I just wanted to show that, expose everyone to the fact that you can play around with this without having to have hardware dedicated to it. You can do this all within a virtual machine, run those same commands. On this one, because as a virtual machine, if I ran the PK, the OEM one, you would see UTM’s key is a start of this. So hopefully that was beneficial for y'all to see. All right, let's get back over to my slide. Okay, so that was the demo. So trends: we're seeing the trend that in the enterprise space, it is being required to keep Secure Boot enabled.

It doesn't pass compliance. Your security folks will be like: no, I need this turned on. Customers in highly regulated industries or government spaces, right off the bat - they won't talk to you if you don't support Secure Boot. So we have to support Secure Boot. Cloud compute instances, which - before we price it - I don't need to worry about it; I'm in the cloud! Well, they are now supporting Secure Boot because that's becoming a requirement of their customers to be able to support it. How many here have heard of confidential computing? Okay. I was expecting not, but there's a handful. If you haven't heard about it, it's something that's actively being supported, worked on, and improved a lot.

We're having to learn about it ourselves at our company. And it is very new, but it is becoming a requirement. We have a customer - that's when we dived into it - that was our requirement to switch over to using Rocky Linux. They wanted to use Rocky Linux on Azure, but they required this support. So we had to go and figure it out and add support on Azure to do confidential compute. It's using hardware-based Trusted Execution environment. So what it's doing is taking many different components. It's taking Secure Boot, but that has to be on. It’s taking TPM. That has to be available. So they're using virtual TPM.

And then you have the hardware side, the CPU instructions, to help keep any instances up in the cloud separated from everyone else that's using that machine. It's all encrypted and your storage is encrypted. How do you unseal your storage? Well, it relies on TPM. It relies on, even for Secure Boot, you got to have, I won't go into it, but it uses a unified kernel image, and - well, that's another subject. But all of that ties together. So Secure Boot: we might not like it, but it's becoming part of these components, part of this process. So we have to use Secure Boot. So what's the takeaway I hope everyone gets from this talk this morning?

Is that our frenemy? I still say yes, it's our frenemy. It's not my friend, but it's not quite my enemy because I need to use it. I have to understand it. I have to work with it. I have to support it. And so, hopefully, for the rest of you out there, giving you something to think about, something to understand for your different use cases. Is there a better way to do Secure Boot? That's the question I want everyone to think about. Is there a better way? My answer is probably. Unfortunately, I don't have the answer. I don't have the solution for it. But we need to really think about it and say, is there a better way?

Before this talk, I was out there talking at our booth and somebody came over, and as soon as I mentioned, somehow it came up, and I was like, well, I have a talk today on Secure Boot. He was all: It’s such a pain in the butt. I'm like, yep, I agree. And we started talking about it and it's just like, man, that there's got to be a simpler way to to fulfill the goal. Right. There was a goal. And please remember UEFI Secure Boot is just a specification. They specify this is how it should work. But the implementation - that's my next point here - we don't know yet - needs to simplify the management and not allow one entity to control it.

And right now we have one entity controlling it. And therefore the processes are slow. And we're dependent on this one entity. So is there a better way? I very much think there is, but we need to come together and start thinking about how can we solve the problem that they set out to solve, but do it in a much better way and it will get much wider support. That's it. Hopefully those were the takeaways. I thank you all.