CIQ

Rocky Linux and Security

February 24, 2022

Webinar Synopsis:

Speakers:

  • Zane Hamilton, Director of Sales Engineering at CIQ

  • Neil Hanlon, Solutions Architect at CIQ

  • Gregory Kurtzer, Founder of Rocky Linux, Singularity/Apptainer, Warewulf, CentOS, and the CEO of CIQ


Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Good morning. Good afternoon. Good evening. Wherever you are, we appreciate you joining us. For those who are coming back, thanks for coming back. For those who are new, welcome for the first time. We hope you like what we have to talk about today and if you do, go ahead and like and subscribe. My name is Zane Hamilton. I am with the Solutions Architecture Team here at CIQ and today I have Neil Hanlon and Gregory Kurtzer with me to talk about Rocky Linux security. Why don't we jump in, guys, and talk about it.

Do you want to introduce yourself, Neil?

Neil Hanlon:

I'd be happy to. I'm Neil Hanlon. I'm a Solutions Architect here at CIQ and I'm also involved with the Rocky Linux team.

Zane Hamilton:

Perfect. Everybody knows Greg, but say hi.

Gregory Kurtzer:

Hi, I'm Greg.

Secure Boot [00:52]

Zane Hamilton:

Perfect. One of the things that I want to talk about first is secure boot. I know that getting secure boot enabled and ready to go was a big deal when moving into Rocky Linux 8.5. Tell us a little bit about secure boot. What is it? How is it beneficial? What are things to watch out for?

Neil Hanlon:

Absolutely. Secure boot is part of UEFI, which is the next generation of the legacy BIOS boot. It was developed in response to malware rootkits in particular that were able to replace that first stage boot loader in your environment. It is a cryptographic way of ensuring that the operating system that you're trying to boot from is not a virus. That's done right now through a central authority through Microsoft. They have a portal where we upload our certificate and have them cross-sign it so it is valid. That's loaded into a piece of hardware that's soldered onto your motherboard if you have a new computer called the TPM or a trusted platform module. That trusted platform module enables secure storage of encrypted secrets such as certificates and keys. This allows us to have a secure boot environment where Microsoft has validated that the boot loader, built by Rocky, is not a virus. It's also built in a secure, trusted environment that we maintain on the Rocky Enterprise Software Foundation. This step allows you to ensure that when you're booting Rocky, you're getting Rocky and not something else.

Zane Hamilton:

Is there anything special that you have to do to enable secure boot whenever you're doing this with Rocky?

Neil Hanlon:

As long as you have relatively recent hardware, and by relatively recent, I mean since 2013. We know there's systems out there that are older and still able to run stuff. If you have some older stuff, there may be in compatibility with the shim due to hardware design issues early on. I think those are few and far between most  of the time. If you're using Rocky Linux 8.5 and you have a new laptop, you can just boot up and a secure route should be enabled by default.

Gregory Kurtzer:

I have a funny story in regards to this. I just got a new laptop by the way. The new Dell XPS 15 with an OLED display is really awesome. It almost totally works with Rocky out of the box, but the audio had a little bit of an issue. I patched the kernel, rebuilt the kernel and tried to boot on this kernel. Secure boot came right out and did exactly what it was designed to do: to stop viruses and people like me from injecting things into the boot loader. It actually blocked me and then I had to go into the BIOS to disable secure boot to run my kernel to test the audio patch that I put in there. It worked. Secure boot is totally cool. It does give that validity on the operating system that you want to be running. It stops you from getting viruses and having weird wannabe kernel hackers like me playing with your system.

Custom Kernel Modules [04:30]

Neil Hanlon:

That's a great point, too. In regards to secure boot, if you are relying on custom kernel modules, third-party drivers from ElRepo, or using a different kernel, these will not have Rocky Linux secure boot. If you're using those custom drivers, then you may also have to disable secure boot or load the certificates that those are signed with into your TPM, into the secure view, trusted storage that you can boot off those. You can do that with the kernel that you built, too. You'd have to generate your own keys and load them in yourself to validate your own computer that you're building the current kernel. Thankfully, not a lot of us have to build kernels every day.

Zane Hamilton:

Except Greg.

Zane Hamilton:

Go back through and make your kernel work. Did you go through and make your own keys?

Gregory Kurtzer:

No, I didn't. I didn't make my own keys. I just disabled it, but the kernel fix for audio does work. I'm now doing what I can to bribe the kernel folks on Rocky to let my patch in. It’s already upstream, it's not like I created it, but it works. It's awesome. Now that laptop, the XPS 15 has an OLED display touch screen. It's seriously awesome.

Gregory Kurtzer:

Everything works perfectly right from the install. I didn’t have to make any BIOS changes until I put my own kernel in there. Everything works well.

Zane Hamilton:

Have you found yourself going and poking your other laptops as you're trying to work on them? Do you forget that they're not all touchscreen?

Gregory Kurtzer:

You know how many times I've done that?

Zane Hamilton:

I've been sitting in meetings with people who are used to using the Microsoft Surface and they'll have somebody else's machine. They will sit there and poke other people’s monitors and knock them over. It's hilarious. 

We got a question: how's the validity of a runtime kernel module validated as part of secure boot? Does the kernel verify them on insertion?

Neil Hanlon:

That's a great question. I'm not one hundred percent sure. I can circle back with our folks that are more well-versed in the specifics of how that works to get a great answer. I imagine that they would have to be verified on insertion. If you try to load one that isn't signed while it's running, it would taint the kernel and then cause a possibly recoverable error to come back. I'll have to try that out. I think we can certainly get an answer for exactly how the runtime modules are validated.

Gregory Kurtzer:

Neil, I may be able to get you a malicious kernel module or two, if you want.

Rocky Security [07:34]

Zane Hamilton:

Sounds fun. When we start talking about and looking at Rocky security, I think it's important to talk about from a community standpoint, what is Rocky security like? Is there anything special that we do around that?

Neil Hanlon:

Security is something that we were focused on from the very beginning while building out Rocky Linux over a year ago now. It was the foundation for the access to systems design which determined who has access to keys. For example, secure boot, among other things, is not different from anyone else. We only give people access to things that they absolutely need access to. That's something that we believe is also coming from the transparency side of what the organization is doing.

All of our security updates are coming from RSS feeds from upstream and notifications. We have a coming site that's going to be tracking when those packages are imported and released to the public mirrors. We can see when they're released in Red Hat and when they're coming out, when they're imported into Rocky's git, and when we're releasing them as well. In the past few months, we've gotten some more integration around tooling. That's commonly used in the enterprise environment, such as SIS for scanning as well. We recently had a CIS work bench or CIS community created around Rocky Linux so that we can get a workbench in for validating the Center for Internet Security benchmarks across Rocky Linux as well.

Gregory Kurtzer:

I can jump in on that, too. For example we've been working on FIPS. CIQ is funding the development and certification of FIPS 140-3 for Rocky Linux 8.5 that is currently underway and that will give various organizations confidence with our crypto implementations on what we've done with Rocky Linux. There's one more aspect that I'd like to touch on real quick, which is if you look at most of the critical CVEs that are out there, one of the most common mechanisms or vectors to exploit hosts and applications is buffer overflows. 

In a buffer overflow attack, they're taking some sort of input data or input stream and they're overloading a buffer that was not being properly checked within the source code. They will then overload that buffer to the point where they know another function can basically come in and actually execute a portion of that memory stream, thinking that it's the function or another piece of that software. They buffer overflow it and then they put in a malicious piece of code as part of that buffer overflow. This is one of the most common ways that hackers and malicious users are using to run malicious code on systems. We're working with a technology right now that was originally designed for embedded systems, but we're applying this to the entire operating system.

What it will allow us to do is create a moving target defense of that memory segment of that memory of the application. At a high-level view, it will randomize all of those different entry points for raise, for variables, and for functions. When the application is started, it will randomize that. It won't stop a buffer overflow, but it will make them completely useless for utilizing a buffer overflow to run malicious code through the application. We're going to be applying that to the entire operating system. What's really cool about it is that it doesn't change the API at all. It doesn't change the API and it doesn't add any performance impact to running this. It is some build chain modifications that we're making on our side.

We're going to be adding value on top of the community open Rocky and CIQ is going to be offering hardened packages and hardened capabilities on top of Rocky that will mitigate and block these buffer overflow attacks. If I were to take a guess, they are 40% to 50% of all of the high severity CVEs that you have. This means that we're going to be able to block past, present, and future exploits moving forward. We're really excited about this. Again, this is moving target defense.

Zane Hamilton:

Great. You said one thing that's important is we do it for the operating system, but is there the ability as a software developer? Can I extend that onto my software as well?

Gregory Kurtzer:

Yes. We're figuring out how best to do that, because that's part of our build toolchain that we're going to be using for building some of these packages. We will figure out how to make that happen. It's probably going to be licensing that toolchain to customers who want to compile their software with that toolchain. It is also important to mention that this entire moving defense model exists in userspace. It’s a runtime feature, not a kernel feature. This means we can apply it to a container and wherever that container goes and moves, that container will also have this moving target defense model within it. It really is super powerful. It's very easy to move around and do different things with. We're really looking forward to bringing this to market and people should be seeing this in Q2, so pretty soon.

Zane Hamilton:

Awesome. We talk a lot about the Rocky build system and how it's different from some of the others, and how we're trying to make it more open. How does the build system we have offer additional trust to the resulting packages that come out that make up Rocky Linux? 

Neil Hanlon:

Just to clarify, Rocky is currently using the Fedora build tools, so the Koji module build service, and those artifacts to build the sources for CentOS 8. Those tools are awesome. They take a lot of time to wrap your head around, but once you get them up and running in an environment, they work great. What we found when we were learning about these tools is that they didn't integrate well into the toolchains that we wanted to use and wanted to try and integrate into our stacks in a more cloud-native way. 

One of the reasons that a lot of the tools like Koji MBS work really well is because the folks that are developing them are the same ones that are using them. They have the expertise and the knowledge to make those services and tooling work really well. They have it all built into their existing stacks and services. What resulted from all of this was a build system that we are calling Peridot. Mustafa, one of the software engineers here and one of the community members at Rocky Linux, did a presentation on it at FOSDEM a couple of weeks ago. One of the things that we’re doing with the build system, Peridot and the previous build system, is to provide better transparency into everything that is going on. This applies to sources that we are ingesting upstream from git.centos.org to the artifacts we published at dl.rockylinux.org. These build systems will continue to be used for the 8.x train for Rocky.

We want the pipeline and its process to be fully transparent for you. This way you can validate the entire stack from where the artifact was pulled, what patches were applied, where it has been, and have a softer bill of materials for the operating system. Peridot allows us to integrate plugins so we can add things like security, we can integrate with CI/CD pipelines, and notify people. It’s a very extensible platform that will enable us to be more transparent and hopefully enable better security for us in the long run.

Zane Hamilton:

Peridot is really going to be for Rocky 9 moving forward, right?

Neil Hanlon:

That's what we're targeting right now, as well as for special interest groups that might be looking to add value on top of Rocky and release packages there.

Errata [17:40]

Zane Hamilton:

Very good. What about Errata? How do we handle that?

Neil Hanlon:

It’s kind of the same system. We are pulling those sources from upstream and validating them and reissuing them as Rocky Linux security updates to errata.rockylinux.org. You can go directly to that website and see a list of the security vulnerabilities that have been released and what their IDs are. It has an API with a version 2 coming soon with some bug fixes that will allow anyone, including security vendors such as Nessus or Tenable, to integrate with that to find out about the security vulnerabilities as they're released. I believe the V2 of that API is also going to come with an RSS feed so that users can subscribe to that if they want to get notifications.

Security Patches Available [18:39]

Zane Hamilton:

Very nice. I know a lot of people just run YUM update, and move along; I don't really pay attention. What’s available for security patches without just blindly doing what I do?

Neil Hanlon:

That is a great question. I think the best way is…

Zane Hamilton:

… to not do what I do?

Neil Hanlon:

There are a bunch of different ways. There is the errata.rockylinux.org website. It has the API and soon there will be an RSS feed for it. The second way would be the Rocky Linux mailing list. We send out updates as soon as we publish images to the mirrors and they can start pulling them down. That happens 24 to 48 hours after we release a patch to circulate to all of the mirrors in the network, but it'll be available right when we send that email. The patch will be available at download.rockylinux.org. You can subscribe for notifications about these patches at lists.resf.org. We have a list there called Rocky announce that has announcements about all of the packages and any updates that might be coming in. So there are two ways. As for the third way, I have a short little demo here on a container that you can use DNF to see exactly what updates you need on your system right now.

Zane Hamilton:

I can't get used to DNF yet; I still use YUM. It’s the default. It's the way my brain works still.

Neil Hanlon:

Let's see, let me share my screen here.

Gregory Kurtzer:

While you're doing that, Zane, don't feel bad. I know many very large enterprise sites that do a YUM update in Cron every night. Even professional sites are doing that. I'm not suggesting that it is the best way of doing it; as a matter of fact, I would suggest the opposite. There are probably better ways of doing it, but don't feel bad.

Zane Hamilton:

No worries. Neil and I were talking earlier about living through scanning for different entities that need to do scanning and our pain through PCI audits and FAA audits. I'm sure you've never been through one of those, Greg.

Gregory Kurtzer:

Not an FAA audit.

Zane Hamilton:

That was fun. Every year, we went through an FAA audit; those were interesting.

Gregory Kurtzer:

During my tenure at the Department of Energy, there was one point where we had a training course on how to deal with being interrogated by intelligence officers.

Gregory Kurtzer:

That was fun. If anyone is curious, most of these interrogations revolve around the idea of awkward silence. They'll ask the question, you'll answer, and then they'll keep staring at you as if they're waiting for something else. The amount of people who will just willingly give up, keep talking, and keep sharing and blabbing about all sorts of information is actually how they get most of their info. You have to get used to awkward silences. When you’re done answering, be quiet and stare back at them.

Zane Hamilton:

I love it.

Vulnerabilities [22:34]

Neil Hanlon:

I got my screen shared. It was difficult to get my screen shared. I've got a container here running Rocky Linux 8. I just pulled this from Docker Hub. We also have them available on quay.io. They are also available on the Amazon container registry which now pulls in Docker Hub images. This dnf updateinfo command has a bunch of different options. One of them can just list out what vulnerabilities are currently open for the system that you have installed. It lists here the vulnerability ID or the advisory ID, and what type it is. These are moderate security vulnerabilities. There could be bugs instead of security updates, as well as the name, and this is the name version architecture; just string something here.

There are a couple of different ways that we can install updates from here, if you don't want to just see them. The first way is we can show information about all of those and get more information like the CVE number. In fact, let's just go ahead and grab that CVE number. We're going to go dnf install –cve and not install, but update. Then we're going to say yes, and yes again. That command there—we took the CVE number and passed it into DNF, and it went and grabbed that using the metadata that's signed on the download.rockylinux.org repository mirrors. We can do that again with an advisory ID if we wanted to.

Zane Hamilton:

I know Greg's laughing at the Vem one, too.

Neil Hanlon:

I'll grab Vem, let's update.

Gregory Kurtzer:

Don't mock the Vem. You don't want to talk bad about Vem.

Zane Hamilton:

That's still my favorite text editor. I can't convert. I am too old!

Gregory Kurtzer:

Is it still POSIX standard?

Zane Hamilton:

I believe.

Gregory Kurtzer:

I think it is, too.

Neil Hanlon:

I also use it.

Zane Hamilton:

Yes!

Gregory Kurtzer:

Then it has been validated; three out of three senior webinar engineers choose Vem!

Rocky Linux Security updates [25:06]

Neil Hanlon:

As well as doing it with the CV number, you can also pass advisory numbers and advisories. As you can see here, there are multiple packages associated with them. If you had to pull in any  one of these they'd have to be pulled in. Anyways those are dependencies. The final way that you can update your system and make sure it has all the security availabilities patched is by saying dnf -update security. That's going to go ahead and install the other two advisories that were available. If we try to run that again, DNF update digest security and should say nothing to do. Likewise, if we do a DNF update, info is empty. We can see all of the security vulnerabilities that our patch on our system.This will bypass the dnf updateinfo - - list - - all. That is at least three ways to make sure your system has all of the Rocky Linux Security updates.

Zane Hamilton:

That's better than the way I do it. That's awesome. Thanks for sharing.

Peridot [26:10]

Gregory Kurtzer:

I want to talk about the Peridot build system. I want Peridot to incorporate all of this as part of the build system. It is slated to be open source here very soon if it's not already. This means that organizations can actually leverage the Peridot build system and they can have some that they maintain by themselves.

Neil Hanlon:

Absolutely. Collaboration and people using the system to help accomplish the goals that they want to accomplish is very much one of the principles we want out of it.

Zane Hamilton:

We talk about sharing. We talk about being open. What's keeping a contributor from putting something into this that is malicious?

Transparency [27:02]

Neil Hanlon:

The biggest thing preventing malicious packages or wrongdoing is transparency. All builds are auditable. You can see exactly what went into them. You can get the artifacts because we signed everything with their GBG keys. It's obviously not impossible. Security is a net, there will be things that you can’t block completely. It would be difficult to get something in. We use PR's and we take security very seriously in terms of not giving people access to things that they don't need access to. This comes down to making sure admin accounts aren’t being used for daily drivers and having approvals on code. Being merged in that will affect the building of packages and everything else.

We took a hint from Fedora, Debian, and OpenInfra among others in other Linux distributions with how they perform transparency and mentorship.We also watched the auditing of resources that come out of there. We don’t think anyone could insert something malicious into a package without some alarm bells being raised. Though humans are infallible, we hope someone will notice. That's part of a bigger discussion in general. We saw that last year with some comments that were being made as part of a research project for Linux kernel.This is another challenge as a distribution of packages as well as you have to not only worry about the safety and security of your distribution, but also what's going into it too.

Gregory Kurtzer:

I was going to bring up that point as well. This is the same for pretty much every open source project that exists. As long as you have a process for how you are verifying and maintaining every bit of code that's coming into the project through transparency, auditing, merge requests, PR's, or whatever your policy is. However you're bringing that in, and keeping the people that are contributing accountable. Who are the people that have committed access? Those people that have committed access, how do you make sure that you have multiple people doing reviews? All of these sorts of processes are designed to not only facilitate good code contributions to the software, but also to catch if somebody is doing something malicious. Over time you will trust more people so they can be part of the project.

The goal is to increase the breadth of people who want to be part of that project, but also how are you managing and mitigating security risks through that? I want to mention this is not specific to Rocky Linux. It is pervasive throughout all of open source. We are doing the same sort of things that every open source project would do in terms of transparency auditing. The infrastructure is designed in a smart way, putting security as one of the primary tenants. The Rocky project itself has a very active security team within the project. Not only are infrastructure development contributors thinking of security as a number one facet to the project, but we also have a number of people that have a high degree of say, regarding that infrastructure and regarding how we are creating policies around security. 

Security is one of our number one priorities, along with community, and creating a very large, diverse, inclusive open source project. Another priority is balancing the two and coming up with the infrastructure necessary to be able to allow contributions from casual contributors and including them in a way that does not compromise security. It does not bring up any security issues. That's really what we've been doing. As Neil mentioned before, the infrastructure was based on Fedora and Koji with all the other pieces coming from the Fedora team. They've done a remarkable job pulling all that together and creating a platform for building operating systems.

Peridot is taking that and turning it into a cloud native architecture. Because of these changes we're going to be able to get more transparency and more auditability into that environment. Basically persisting every artifact that we can out of that and making that publicly accessible is going to be hugely important. Just to reiterate what Neil said, the software bill of materials, the SBM, is really critical. We've actually gotten a number of large organizations that are very security focused asking for that. It is a huge priority.

Zane Hamilton:

Very good. I feel like we should just sit here and do a long pause and see if people start asking questions.

Gregory Kurtzer:

The awkward pause.

Zane Hamilton:

If you guys have any questions please send them to us. We'd love to answer them. We've got some time here. If not, we can make Greg tell us more stories. You've got to have more securities stories, Greg.

Gregory Kurtzer:

I've got a really good one. Only if nobody asks any questions. I hope people ask questions because it's more fun to answer questions than tell stories.

Zane Hamilton:

Not yet. Let's have a story.

Zane Hamilton:

Love it.

Gregory Kurtzer:

I may get in trouble for this one.

Zane Hamilton:

The best stories do.

Story: Lock Your Screen Terminal [33:51]

Gregory Kurtzer:

A couple people from the Rocky team and others have heard this story before. When I was at DOE we had an intern that would always leave his terminal logged in and would then leave his computer. We gave this individual a lot of crap saying things like "you gotta lock your screen when you walk away, this is a government facility, you gotta lock your screen!" He just wouldn't do it. He wasn't trying to be malicious, he just wasn't used to doing it, so he'd walk away. One day he walked away and I came over to his computer and I saw not only is it logged in, but he's got a root terminal open. I quickly changed his password and root password to the system. I locked the screen and  walked away.

I walked away from his desk into my office. I hear him come back and I start listening. I hear him trying to get into his computer, getting frustrated and concerned. After a little bit of struggling, I go out there and I say, "What's going on?" And he replied, "I don't know, my computer's not working. I can't get into it. I can't log in." I said "Hmm, you want me to try?" “What are you going to do?" he said. I responded, "Let me just try it real quick and see what I can do." I'm sitting at his login prompt, typing something, hit enter, and it logs in, and he's looking at me like, what did you just do?

I said "Oh, it's no problem." I said, "I put a backdoor into every CentOS system in the world." Then I said "Yeah, there you go. Cool. Make sure you fix your password." I walked away. Apparently this person contacted the security team and told them that I have put a back door into every CentOS system in the world. That did not go over very well with the security group who very quickly came into my office, asking me all sorts of questions. I was able to use that awkward silence thing to my benefit and maintained that. It was a little nerve wracking for a few minutes. Then, I shared with them what I did and that I did not in fact put a backdoor into every CentOS system in the world. Everybody lock your screen terminal, especially if I'm around.

Zane Hamilton:

Cruel. That's the only word I have.

Gregory Kurtzer:

But this person learned the lesson.

Zane Hamilton:

They did, and they never walked away with it unlocked again. We do have a question that came in asking us to repeat the FIPS compliance status.

FIPS Certification [36:59]

Gregory Kurtzer:

In terms of status, that is currently in progress. The process of doing FIPS certification is quite lengthy. Especially considering that we just went from 140-2 to 140-3. There were some changes that are going to be necessary as part of that compliance. We're expecting that we're going to have to be making some code changes to get that certification. At best, we're looking at probably another six months. At worst, we're looking at another10 months. It's really hard to say. We do have an organization that we're working very closely with on that. It's in progress. The goal is going to be for Rocky 8.5 and I think that's the high-level gist of it.

Once we get that FIPS compliance, we're also going to go after additional security accreditations that are dependent upon that FIPS compliance. Then what we're going to be doing is having specific point releases. These will target updates on FIPS compliance. We won't have FIPS compliance on 8.5, 8.6, 8.7, and so forth. It’ll probably be 8.5, and then it may be 8.7. We may skip one and then it may be 8.9. We're going to have to wait and see kind of how long it takes and what the lay of the land is at that point. The goal is to have multiple versions. As you may know, FIPS compliance is a certification of a particular operating system patch or release set. Once you've updated those RPMs, those RPMs are no longer compliant because it validates a particular binary release of a package. We will have to continue doing FIPS updates long term. We are planning on doing that.

Rocky 9 Beta [39:16]

Zane Hamilton:

Very nice. I think one of the other questions that I've gotten asked quite a bit lately, as people move to Rocky 8.5, is when can they start playing with a beta of Rocky 9?

I get that question a lot.

Gregory Kurtzer:

I've gotten that one as well, too. We are currently in the process of building, testing packages, and building processes. We are doing this in conjunction with the new build system. We kind of have two things that we're doing simultaneously. The goal is once 9 is out, it'll be all built on the new build system. That's delaying the beta, but we don't think that's going to delay the release at all. As a matter of fact, we're hoping to have the release as soon as absolutely possible. Neil, I hand it over to you to give more concrete information on 9.

Zane Hamilton:

Awkward pause.

Neil Hanlon:

We're in the middle of planning. We're not planning, but executing and planning. A lot of it can be followed along on the Mattermost chat for rocky at chatbot.org. There's a planning board essentially in the development channel that we're tracking the tasks for Rocky 9 and the roadmap that's going on there.

Zane Hamilton:

Excellent. Thank you. Jonathan has the following question. “Will FIPS compliance support the use of Rocky Linux in NIST 800-171 and/or CMMC environments? Or are those mostly separate concerns that will require independent effort?

Gregory Kurtzer:

You caught me, I'm Googling those right now. I'm going to need a little bit more time, but I will get back to you Jonathan. I don't have an answer for you specifically. Google was not fast enough for me or I was not fast enough in reading it. I'll get back to you on that, Jonathan.

Zane Hamilton:

We have two questions we need to come back to. Perfect. If we don't have any more questions, we are pretty much up on time, I appreciate you guys coming again. Like and Subscribe, stay in touch with us. If you have any other questions, feel free to reach out, hit us up, let us know what you're thinking. We appreciate it. Thanks for the time. Thanks Greg. Thanks, Neil. I'm glad you didn't ask me for another story, Zane. Out of time. Sorry. Thanks everybody. Appreciate it guys.