Set It Free: Up Close with Rocky Founder Greg Kurtzer

March 29, 2023

Editor’s note: This article by Joe Casad originally appeared in the Rocky Linux Focus Guide, published by Admin Network & Security magazine.

Rocky Linux is new, but it draws from a long history of community-minded volunteers focused on keeping a free Linux option in the enterprise space. Rocky founder Greg Kurtzer tells the story.

Tell us a little about yourself. How did you get started with computers – and with Linux? What was your motivation for launching CentOS 20 years ago?

I’ve always been interested in computers, even at a very young age. In elementary school, I spent lunch time using the Apple IIe that was in our classroom, learning to program. But for me, I always considered working on computers more play than work.

In school, I studied biochemistry and started my career working in pharmaceutical research in the mid-90s. In doing genomic analysis, I began to use more computational resources to quantify and annotate genomic data as the bioinformatics field was just emerging. This was where I was first introduced to Linux and open source, which I became enamored with and thus ended up pivoting my career. From then on, everything was Linux and open source coupled with science, so HPC and research computing was my perfect niche.

I ended up at Lawrence Berkeley National Laboratory within the US Department of Energy with a joint appointment to UCOP/ UC Berkeley. It was there that I started my first big open source project (Warewulf, a cluster operating system management) and, shortly thereafter, cAos Linux, the cAos Foundation, and then CentOS, which came out of my work with cAos.

The motivation to build CentOS occurred when Red Hat killed off their freely available Linux platform called Red Hat Linux (RHL) in favor of Red Hat Enterprise Linux (RHEL), which was only available commercially. At that point, we were still bootstrapping and basing cAos Linux Core off of RHL, so this affected us greatly, and we decided to rebuild the RHEL sources as cAos- EL, which then became CentOS Linux.

Where were you when you heard that Red Hat was repurposing CentOS? Was it a total surprise, or was it something that you could see brewing for a long time?

When I first heard that Red Hat was acquiring CentOS Linux, it was because I had a number of contacts from software vendors who were concerned that one of the biggest operating systems used by their customers would be under the control of the same company that pretty much sold the same bits. It seemed like a conflict of interest, but my response was to see how it played out.

Later, when I heard that Red Hat was being acquired by IBM, it was because I had a number of contacts from hardware vendors who were concerned that their competition now owned the two biggest Enterprise Linux options for their customers (RHEL and CentOS). Again, I didn’t want to have a knee-jerk reaction, and instead I gave them the benefit of the doubt, because Red Hat had been doing well balancing the conflict of interest so far.

I created CIQ based on a vision for the next generation of computing software infrastructure, and we were planning on building out and supporting a full software stack. Our stack was to be built on top of CentOS, so we did some diligence and saw the writing on the wall that something was going to change. We created some contingency plans, “just in case,” and even though we were somewhat prepared that something was going to happen, it was still surprising.

Because the changes to CentOS affected us, it affected our customers and the community at large, so I quickly responded, and within hours, Rocky Linux was born via my comment on the CentOS blog post.

CentOS was supposed to be an open source project, with its own governing board. How did it end up disappearing upstream?

My role in CentOS was limited to only the first one to two years of the project’s foundation. CentOS came out of another Linux distribution project called cAos Linux, for which I created the cAos Foundation, a nonprofit organization that was the legal entity and decision-making authority behind cAos Linux. Rocky McGaugh, who was an early community member and developer, announced the name change and that he was 99 percent complete with the first version of CentOS. CentOS was under the cAos Foundation for only the beginning of the project’s life, as Lance Davis was allowed to control some of the assets personally and thus take over the project. Lance continued to hold the CentOS assets for many years until this “Open Letter to Lance Davis.” The project only survived due to the dedication and integrity of the developers and contributors, to whom the CentOS community owes a debt of gratitude.

Over time, Red Hat was able to acquire the project by hiring the remaining CentOS team and gaining control with a new board structure that they defined, led, and controlled. We put in a number of protections to protect the projects under the Rocky Enterprise Software Foundation (RESF) from this sort of thing.

How do we know the same thing won’t happen with Rocky? I understand that CIQ is not Red Hat, but how did you approach it differently this time? What steps did you take to ensure Rocky’s permanence and independence?

The longevity and stability of Rocky Linux has been our goal since day one. I used to believe that the best path for an open source project was to roll the project into a well-funded host company that can contribute ongoing resources (like CentOS, Fedora, Ubuntu, Elasticsearch, etc.), but I was wrong.

When an open source project is under a profit-motivated corporation (especially when the company is productizing the project), it runs the risk of alienating outside contributors, and company-sponsored development fills in the gaps. More strategic engineering conversations start happening solely within the company, not in the community, thus the community suffers. I believe this problem is unmanageable unless the control structure for the open source project exists outside of a corporate entity.

“If you love something, set it free.” That is how I feel about open source projects, and this applies to my own company as well. That is why CIQ does not own or control any of the open source projects we’ve created or contributed to. We “set them free” to the community. This also forces us to build an ethical business model by adding value, not controlling or holding the project hostage.

Companies, vendors, and people come to us when they need help beyond what the community provides or if they need a product that delivers more value to customers.

In short, we set up the RESF so that no company, not even CIQ, can do the same thing to Rocky that Red Hat did to CentOS. The RESF is also available to host other open source projects that are looking for neutral ground and collaboration across company barriers.

The Rocky Enterprise Software Foundation is actually a public benefit corporation (PBC), with you as the owner. Why did you choose a PBC instead of a conventional nonprofit corporation?

About 10 years ago, it started becoming difficult for open source projects to gain charitable organization status from the IRS through a conventional 501(c)3 nonprofit structure. 501(c)6 is what some open source projects are heading for, but specifically that type of a nonprofit focuses on supporting commercial activities rather than the general public benefit. Either way, tax-exempt status is no panacea guaranteeing that the organization will protect assets properly or even avoid succumbing to corruption. Being a nonprofit is not a recipe for integrity.

So what is the recipe for integrity? We think that the answer is accountability. At the RESF, this accountability starts with making promises, sharing visions, doing the right thing, and defining a structure of checks and balances. All this gets codified with bylaws, a charter, publicly defined principles, and transparency, all providing absolute accountability for our actions. We are a self-imposed not-for-profit organization specifically for the benefit of the general public, and we ask for the community to hold us accountable.

The RESF bylaws and charter will be ratified in the next few weeks, and we are very excited to share with the world how this vision for accountability is unfolding. Editor’s note: The RESF bylaws and charter have been ratified.

Describe the Rocky development process. How many developers work for Rocky? What percentage of them work for CIQ (versus volunteers)? Rocky has several other sponsors in addition to CIQ. Do other companies commit full-time coders to the Rocky project? Did a lot of former CentOS volunteers follow you over to Rocky?

Great questions…. First point, no developers work for Rocky or the RESF proper. From that first blog post, every single person, and every contribution came from a volunteer. In less than two months, we had over ten thousand volunteers join to help, coming from all corners, including CentOS and Fedora.

And to date, CIQ has hired fewer than 10 people from the Rocky Linux community. That means Rocky development, engineering, and infrastructure is controlled directly by the community, which manages all resources. This means all companies are welcome!

Rocky and the RESF have many sponsors, partners, and contributors who provide resources to the project (CIQ is one of them). Having many organizations associated with the project is critically important for longevity.

A great example is Google, which not only joined and sponsors Rocky Linux, but also supports Rocky Linux directly to all Google customers and offers a version of Rocky Linux that is optimized for Google Cloud.

The Peridot build tool received some attention in the press as an innovative open source tool from the Rocky project. What is Peridot and how is it important for Rocky’s evolution?

Peridot is an absolute game changer for all Linux build systems, not just Rocky! To date, build systems have been purpose built to run on dedicated servers, most of them project or company specific, not well documented, not easy to replicate, and very difficult to fix or customize.

One of the first things we realized was that creating the infrastructure necessary to build the operating system was actually more complicated and more important than building the operating system itself. It is critical for the build system to take advantage of modern capabilities around cloud native orchestration. That is what Peridot is, a completely cloud-native, stateless, build system that can easily ride on top of Kubernetes.

But wait, there’s more! Peridot is designed to help with the important goal of allowing the community at large to take part in the core development, as well as enhancements of Rocky Linux via Special Interest Groups (SIGs). Peridot also allows someone to “fork” the operating system to make specific versions or builds of Rocky Linux for custom sites, appliances, or other value-add builds.

I’m so excited about the innovation that Peridot brings to the table for Linux distributions, optimization, extensibility, and flexibility. Peridot helps make Rocky Linux the Enterprise Linux for all use cases.

HPC is clearly an area of emphasis for the Rocky community. Are there other sectors Rocky is targeting? CentOS used to run on a lot of ordinary web servers and file servers. Do you see Rocky fitting into that space, or are you aiming more at the enterprise, cloud-native market? OK, I get that Rocky Linux can do everything RHEL can do, but I guess the question is more about where Rocky is putting its energy.

HPC is an emphasis for Rocky Linux, just as HPC was an emphasis for CentOS. Members of the early CentOS team (including Rocky McGaugh and myself) were coming from an HPC background and motivation. Aside from that, there are many areas of emphasis for the Rocky community.

The project is focused on several areas: 1) The base Rocky Linux as a bug-for-bug compatible and completely stable Enterprise Linux variant; 2) the infrastructure necessary to build, maintain, and enhance Rocky Linux; and 3) development of special interest groups, such that all areas of the ecosystem can enhance and build targeted versions of Rocky Linux (e.g., cloud, HPC, hyperscale, etc.).

This makes Rocky Linux a perfect general purpose operating system for all use cases and stable environments.

Lots of Linux distros are out there. What makes Rocky different?

Rocky is filling the immediate pain point that CentOS left behind, so it is less of “What makes Rocky different” as opposed to what makes Rocky Linux the best replacement for CentOS. Rocky is specifically focused on the enterprise community of professional organizations and individuals. This niche includes engineers who are ultimately responsible for the servers and infrastructure that power their organizations.