CIQ
HomeCIQ Webinars

Uncovering the Hidden Secrets: Linux and Open Source Projects

April 6, 2023

This webinar will explore some lesser-known facts about Linux and open source projects. From the early days of Linux to its current widespread use, we will cover the history of this powerful operating system and the open source movement. We will also discuss the benefits of using open source software and how it has revolutionized the software industry.

Webinar Synopsis:

Speakers:

  • Zane Hamilton, Sr. Vice President - Sales, CIQ

  • Jeremy Allison, Engineer, CIQ

  • Rose Stein, Solutions Engineer, CIQ

  • Neil Hanlon, Infrastructure Lead, CIQ

  • Arthur Tyde, Sr. Vice President - Global Business Development, CIQ

  • Gregory Kurtzer, Founder and CEO, CIQ

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Good morning, good afternoon, good evening, wherever you are. Thank you for joining. My name is Zane Hamilton with CIQ. And at CIQ, we're focused on powering the next generation of software infrastructure, leveraging the capabilities of cloud, hyperscale, and HPC. From research to the enterprise, our customers rely on us for the ultimate Rocky Linux, Warewulf, and Apptainer support escalation. We provide deep development capabilities and solutions, all delivered in the collaborative spirit of open source. And today, I have my co-host with me, Rose, what are we talking about today?

Rose Stein:

Hello. Hello. So thank you so much for being here. I'm excited to be here as well. So today is super fun. We are going to be uncovering Hidden Secrets, the things that you never knew about Linux and open source projects. So this topic is broad and inclusive. So at any time, if you have something where you're like, Ooh, I wonder if they knew this, pop it into the chat, and we would like to comment on it as well. Thanks for being here.

Zane Hamilton:

Absolutely. So we'll go ahead and bring in the panelists. I think this is interesting now because I still think of Linux as being new, but it's been around for a long time now, guys, it's not new anymore. So let's go ahead and have everybody introduce themselves. I'm going to start with Jeremy.

Jeremy Allison and Samba Open Source [6:27]

Jeremy Allison:

Hi, I'm Jeremy Allison. I'm one of the co-creatives of the Samba open source software project, which actually just about predates Linux. It was created around the same time. And yes, I remember when it wasn't open source. It was free software. I've been around a long time, but that just means I'm old

Zane Hamilton:

Free software. I love it. Mr. Tyde, welcome back. It's been a while.

Arthur Tyde's Open Source Background [6:58]

Arthur Tyde:

Hi, Zane, Rose. Yeah, this is my first webinar. So thanks for having me on. So I'm Arthur Tyde, I've been involved in open source since maybe as long as Jeremy. It's hard to say. We're both getting up there. Wikipedia generously credits me as the founder of Linux and the open source services industry. So I don't know how correct that is. That's from Linuxcare. Back in the day. Look what I found in the closet.

Zane Hamilton:

Oh, you still have the shirt? That's awesome.

Arthur Tyde:

And it fits. Although today it's all about Rocky. But that said, so I founded the Silicon Valley in the Bay Area Linux users groups back in the mid nineties. So actually the Bay Area Linux users group in San Francisco has been going strong for like 28 years. So, they're still meeting in Chinese restaurants in Chinatown. It's crazy. But yeah, back in the day I started this company called Linuxcare. We supported 21 different Linux distributions on nine different hardware architectures. We were actually the first company to do that for the enterprise. And we held the initials. First of all, we were all supportive of Red Hat and SUSE in North America. We also supported All Red Hat and Dell, which made them unhappy for a very long time. Apologies guys. And then we were also sort of the money and the founding sponsors behind really important things like the Linux Professionals Institute, which is sort of the open training for Linux and open source stuff. So that's LPI.org. Their chairman is Joan Hall. We also co-founded the free standards group, which evolved into the Linux Foundation. I was the first CTO of that. And then we put money behind literally dozens of open source projects, both in the kernel space and the user space. And that's where I had the honor of meeting Jeremy Nelson back in the day. I think you were working for VA.

Jeremy Allison:

If you remember I had the choice between working for you and working for VA, and you wanted me to commute up to San Francisco. So I said, no.

Arthur Tyde:

We love you. Anyway, we had your brother from another mother, so it kinda worked out.

And then Greg, currently my CEO. Right? Which is proof that you need to be nice to the people on the way up, because you never know who you may meet later in life. Greg was an engineer in Linuxcare Labs and wrote all of the initial Linux certification code. didn't know that, that I think we stole. Yeah, actually, I can show you his code if you're interested.

Jeremy Allison:

Show me the code.

Arthur Tyde:

Anyway, but I've said enough. Thank you. Thank you Zane. Okay.

Zane Hamilton:

Absolutely. Neil, welcome back.

Neil Hanlon's Experience With Linux At CIQ [10:29]

Neil Hanlon:

Hey, I'm definitely not going to take as time as Arthur, but that's okay. Little bit of shade. I'm Neil Hanlon. I have been working with Linux for I guess probably about 12 or 13 years. And I'm a Linux engineer here at CIQ and I am the infrastructure lead over at the Rocky Enterprise Software Foundation. And I support all of the Rocky projects and the stuff that we do over there.

Zane Hamilton:

Thank you, Neil. Last but not least Greg.

How Did Gregory Kurtzer Get Started With Linux At Linuxcare? [11:07]

Gregory Kurtzer:

No, definitely least in this crowd. Let's see. So I worked for Art. and it was a while ago. And Linuxcare profoundly impacted me in a very positive light. And I've told this to Art before. It was the first company that I could actually find that was actually working on Linux. I wanted to work on Linux. I got hooked and enamored on Linux in about 96, 1996. And I pivoted my career from biochemistry and genomics into Linux and open source at about time. And I wanted to go work with Linux. Couldn't find anything initially to do Linux in the mid to late nineties. I even was a Windows SysAdmin for a little while.

Let's not ever bring that up again. It was very traumatizing. But then my wife saw an ad in one of the Linux magazines that we picked up at Barnes and Noble for this company called Linuxcare, which was in the city. And I applied that day, that night, a guy named Michael Perry responded back to me and Mike Perry was excited to reach out to me. We talked a little bit over email. I think it was either the next day or the day after I went in for an interview. And the day after that, I had a job. I was now working at a Linux company doing Linux. It was the coolest thing ever.

Jeremy Allison:

Did you do the Tridge's mini-CD test?

Gregory Kurtzer:

Not a test, but this is actually a really funny story. And so in the lab we're working with Duncan and Seth and there were a number of other people, and they were working on this little tiny business card, which I'm assuming that's what you're talking about, this little bootable business card. And if anybody doesn't know what the bootable business card is, you gotta look this up, because this was the first rescue operating system with Linux that I knew of at least. It was a little ISO, a mini ISO, a three-inch ISO with a side cutoff, so it can fit in your wallet as a business card. And you put it onto a CD tray. And it's funny, most people don't even know that CD trays could do this, but CD trays could take CD singles, which were the three inch, and, and every CD-ROM you have a little cutout, little intention for a three inch single CD . 

And so it fits on the three the three-inch single CD, and you put it in there and it boots Linux off the ISO. Now, back then, this was the late nineties that didn't exist anywhere else. Like this is the first time somebody was thinking of booting an operating system off of this little bootable business card or the little CD, right? The compressed loopback file system didn't exist yet. This was the creation of the compressed loopback file system, which has now given way to SquashFS and other types of compressed file systems. And it all started at Linuxcare in this lab when Seth Duncan and others were showing off what they've created. And sitting across the desk from me at the time was Paul Russell, Rusty. As he was very well known and we were showing this to Rusty and his response was, oh, this is so amazing. Wait, where is rsync? And I remember Duncan and Seth like hanging their heads for a moment, there's just not enough room. We only have like 50 megabytes of space. Like, there's just not enough room. We can't put anything else in there. And Rusty's like, what file system are you using? How are you doing this? What are you doing? And he is just, oh, it's easy. Just compress your file system. And we're like, that's easy? Like, what are you talking about?

Jeremy Allison:

Well, it's for Rusty.

Gregory Kurtzer:

It is for Rusty. And he tried explaining it to us. And then he gives up and goes, I'm just going to show you, I don't remember how long it was in my mind. I think it was like 30 minutes, maybe it was more. But he created the loopback file system, the compressed loopback file system right then and there.

At a folding table in the Sony building off of Townson in San Francisco. And we then we then all of a sudden had like an additional 35 or 40 megabytes of space. And at that time, that was so much room, we were also able to put x11 in there with a window manager. So not only did we get rsync in there, we got x11 in there .

Zane Hamilton:

I thought So you got Emacs in there?

Gregory Kurtzer:

Oh, I actually don't even remember because I'm not an Emacs user. Was Emacs in there? Art?

Arthur Tyde:

Oh I don't recall. I'm a nano guy, so.

Zane Hamilton:

Guy. Was that not an interview question or something? Wow.

Gregory Kurtzer:

It is now. Wow.

Zane Hamilton:

Yeah, no kidding.

How Gregory Kurtzer Got His Job At Berkeley Lab [16:59]

Jeremy Allison:

Eventually, I think Tridge used those bootable desks. He told me that he created basically a test for people they wanted to hire where they would basically just say nothing and hand them the desk and say, okay, figure out what to do with that. And if you could bring it up and boot it and do something with it, then it's like, okay, you are worth talking to as an engineer.

Zane Hamilton:

Being a Windows admin, have you ever used one of those to grab a password file because you lost the administrative password?

Gregory Kurtzer:

No, this was beyond me doing Windows at this point. But there is a second follow up to this story, which is that's what got me my job at Berkeley lab, at Lawrence Berkeley lab. I probably was doing okay through the interview process until they asked me, one of the Windows engineers asked me, who's still at the lab? Awesome guy. So Jay, hi, if you're here, if you're listening. And he asked me, he says, so what would you do in the situation where you got a Windows server and the hard drive goes bad, but you have a RAID and you need to get the data off of that RAID. And I said, what file system is that? It says Windows, so I guess it's NTFS.

And they said NTFS. So I said, well pull out my wallet. Like, so I'd pull out the Bootable business card, I'd put it in the system, I would boot it up and it boots up as a compressed loopback file system off an ISO 9660 file system. Again, I'm just, at this point, I'm just trying to get them excited, just throwing a bunch of names and buzzwords and name dropping at them. And then I said, then I do the Trivial net setup, which brings up the network and whatnot. I'd use the SMB client, part of Samba, to copy the data, and mount that file system using, at the time it was an experimental NTFS kernel module. And I'd mount that up and I'd copy the data off using SMB client and send that over to another window system of your choice. And their jaws are like, what did he just say? What, wait, what is that thing you just pulled out of your wallet? And I explained it to them and it was just like, mind blown, what is that? How does this work? And, and what was really funny was then I said, well, I'm really curious. I mean, how would you do it? And they said, well, we take the hard drives out of the server, of course, and put it in another server.

Zane Hamilton:

Without breaking raid?

Gregory Kurtzer:

I said, that's horrible.

Jeremy Allison:

So Zane you mentioned the recovery. Get the passwords out of the Windows box. Did you know I wrote that utility?

Zane Hamilton:

No, that's fantastic.

Finding Hashed Passwords in Window Registry For Linux Migration [20:05]

Jeremy Allison:

I actually figured out how Windows stored their hashed passwords in the registry and then wrote a utility that would basically pull those out. And because I needed some way for people to migrate from Windows server to Samba and the one of the sticking points was always the passwords were stuck in windows. No one could get them out. So I figured out how and where they were stored. And so then you could repeatedly, don't turn it into a Samba password file and quietly migrate your Windows server to a Linux box in the background.

Zane Hamilton:

Well, thank you for that. Because before everything was connected to a domain, that saved many servers.

Jeremy Allison:

Well, no, even as a domain, passwords are still short in that place. Sure. Well, not in AD, not in active directory. Back in the old menu. That's fantastic.

A Brief History of Linux and Open Source [20:56]

Rose Stein:

That's fantastic. All right, guys, so we need to backtrack here a little bit. And Arthur, well, you're over here for me, wherever you are, for everybody else. If you're on Wikipedia, you must know everything. And so I would really be interested in you explaining to us the history briefly. So Neil, keep him on track here. A brief history of Linux and then the open source and the key players are there.

Arthur Tyde:

Well, okay. I'll do my best and I'll try and keep it short. It all starts back in the seventies, right? With Ken Thompson and Dennis Richie.

Well, actually, I got to meet one of the luminaries at SCALE 20x down in SoCal. Just a few weeks ago. I got to meet Ken Thompson, which was a tremendous honor. But yeah, so it all started with Unix and AT&T.. And then you kinda fast forward BSD sort of happened at Cal Berkeley in about 77, and that's when the Unix lawsuit started, which motivated a guy named Richard Stallman to create the new project in 1983. So Richard, there's a favorite quote of mine from Biloxi Blues, a play by Neil Simon that says, you should never underestimate the stimulating value of eccentricity, which I think applies to really everybody, especially on this call.

Why Was Linux Created? [22:44]

But Richards was a very passionate guy about free software, and his goal was to create a whole operating system. And he did that. He basically recreated all the tools of Unix including a kernel called Hurd, but nobody liked it. And then you fast forward again to 1991 where you had this guy named Linus who created the first Linux kernel, which was actually released under a commercial license. A lot of people don't realize that it was only after the Linux kernel kinda got wedded with the new tools that everybody took a step back and said, let's make this thing free software. But at the time, there was no term for open source. The term open source actually wasn't coined until I think 1998. It was in a conference room at VA Linux.

It was Larry Auguston, Chris Debona, John Maddogg Hall, Todd, and I'm spacing on his last name. Eric Raymond was there, the guy who wrote the Cathedral in the Bazaar which is an amazing paper that was published in 98. And if you don't know, it's a foundation read. But the actual term open source itself was coined by Christine Peterson in that meeting, who's the former executive director of the Farsite Foundation or something like that. One of these think-tanks. It was a nanotechnology thing, I think. Thank you. Thank you. Yeah. So, that's where the term open source came from. That's when it started. I remember because that morning, I went to Transmena to bring Linus Torvald's prune Juice and a gift. But that's a separate story. Yeah, Linus was addicted to prune juice.

Neil Hanlon:

I mean, who isn't?

Arthur Tyde:

It's so good. Hey, regularity is important.

Neil Hanlon:

That's the next webinar.

What Was Neil Hanlon's First Linux Distribution? [24:49]

Arthur Tyde:

Yes, absolutely. So that's sort of what got us to Linux at open source is a thing. And then, along the way, distributions, like my first distribution, was IG Drizzle. Then I moved to Slackware and then Slackware evolved into Seuss. So I was like an early Seuss guy. And I'm sure everybody has just like, what is your first computer? Everybody has that, what's your first Linux distribution? And I'm very interested in knowing what Neil's first Linux distribution was, because that was after a lot of them fell out.

Zane Hamilton:

So it's probably going to be something like CentOS, watch, it's going to be one of the newer ones.

Neil Hanlon:

It was, it was Ubuntu so get your laughs out. But it was, I think it was Ubuntu 12 or so my brother brought home an Ubuntu CD from college. And yeah, that was that when I started my descent into delinquency. I mean free software.

Arthur Tyde:

And so as a Ubuntu guy, did you dive into Debian and go into the whole, did you explore all the derivatives and so forth?

Neil Hanlon:

yeah, a little bit. I mean, I moved on to Debian fairly quickly, I think. And wanted just sort of a more pure experience per se. But I didn't really, honestly, I used CentOS and rail and such in my day job, but outside of that really was more of a Debian in person up until starting to work more with Rocky Linux and taking a step forward into working with the ecosystem there. But yeah I didn't do a whole lot of experimenting. I had a little bit of a journey into Windows at one point and did some Windows admin, and then they found out that I knew Linux at all and I got quickly pulled into doing Linux. So, that was that's, that's kinda been my history anyways.

Jeremy Allison:

Everybody's done some windows admin. It's like I did my time.

Zane Hamilton:

Yeah, I'm ashamed to say I actually got my certification and I never used it. I moved out soon thereafter. So, let's see. Let's just wait and see if we post those. Why does Unix sound like eunuchs, is that like related or something?

Rose Stein:

Oh my God.

Neil Hanlon:

What a question.

Arthur Tyde:

There's so many places we could go with that, but no.

Gregory Kurtzer:

So, I have a story. So I was telling my family that I was getting really interested in Linux, and they said, what is Linux? And I said, it looks very much like Unix. And the response that I got from my family at that point was again, mouth drop, like whatnot. They're like, who do you know that are eunuchs?

Where Did Unix Get Its Name? [27:57]

Jeremy Allison:

There was a Multiuser operating system called Multex. And I think Unix was a pun on that because it was essentially a single user version, even though it's actually Multiuser, it was considered a single machine version of Multex. And that, I think that's from Bell Labs?

Arthur Tyde:

Yeah. That is some ancient, ancient wisdom there that is going way back.

Gregory Kurtzer:

Is anyone else having flashbacks of history of the world, part one about Unix?

Linux and Debian at SCALE 20x [28:38]

Arthur Tyde:

Well, I think one of the things that I'd certainly like to say just in general is whether it was, when I think about my history, I think I owe a huge debt of gratitude to the founders of this industry because really, if it wasn't for visionaries like Stallman and Linus sitting up all night in his dorm room and stuff like that, I probably wouldn't have a job today. I don't know, I'd be working on the railroad or something, who knows? But, which would be cool. But yeah, there's this rich history whether it was the formation of this free software open source thing to the early Linux companies, really, because you had SUSE in the beginning and Red Hat and Slackware and Debian and I do want to give a shout out to the Debian guys because at SCALE, their booth was right next to us and they were roasting, grinding, and brewing coffee in their booth.

So we were like shoveling dark chocolates their way and they were sending coffee our way and what an amazing crew. So thank you guys. But yeah, you know, the growth of Linux was really an inflection point in the industry. I mean, I still have a memo somewhere from IBM where they sent to me informing me that I was at risk of being dismissed for loading free software operating systems onto IBM hardware. And I like to think of how things have changed.

Rose Stein:

You're a rebel. A rebel from the very beginning.

Arthur Tyde:

I'm a fossil. I'm a fossil with a cause I guess, I don't know. But like I said, delinquency.

How Did Linux Become So Popular? [30:40]

Jeremy Allison:

Does anyone remember there was actually, let me just post it in the internal chat. You can maybe make that available externally. There was actually a presentation with, this was the first time I met Linus actually, was a presentation in 1998 called The Future of Linux. I was looking to be invited to that. And it was the first time this was in Silicon Valley, it was in the South Bay. It was done by Mountain, I think arranged it, VA Linux. And I thought it was just going to be this typical Silicon Valley Linux user group thing. And then Intel said they wanted to be part of it. And who else was on it? It was just an amazing group. And then of course they said, oh, and we have Linus there as well.

And it was interesting. I turned up about 30 minutes beforehand or an hour or so beforehand, and they were already queuing around the block to get in. And that was when I realized this is going to be big. This isn't just a hobby. People are genuinely interested in this. And the guy who arranged it, Mike Masterson from Mountain, he basically said I want you as the warmup man because you're funny and you can tell the jokes until the serious people come and talk about it. And so I walked up and I was really looking at the time I was walking up to the building, and Linus just happened at the time I was walking up. I said, oh, hi Linus, I'm Jeremy. And we walked up and Mike knew me, but he didn't know Linus. So I said, hi Jeremy. And then he turned to the line and said, Hey, where are you going? Stuck hand in his face. And I turned to him and said, Mike, he's the reason that people are queuing round the block. This is Linus Torvalds. Oh, I'm so sorry. I'm so sorry.

Zane Hamilton:

That's fantastic.

Jeremy Allison:

Yeah, that was great. I think that was when IBM, at that point, IBM announced they were putting a billion dollars into Linux and Oracle said they were going to port the Oracle database to it. It was at that point that things really started to take off interestingly.

The Demographics Of Linux Users Is Becoming More Diverse [33:02]

Zane Hamilton:

So you guys noticed, I'm trying to think of the last time I went to a Linux users group. It's probably last year, earlier this year, it's been a little bit, but in the past, I'm just going to say 20 years, when I started going to them, I've seen a significant change. And the people that are coming, it's no longer just the hardcore Unix/Linux admins. It seems to be that there are more hobbyists and just people who are wanting to get familiar with the operating system and younger and younger kids getting involved with it as well. Have you noticed that? I know Neil, you spend quite a bit of time with these too, just what have you seen from the changes of who's actually coming into these things now?

Gregory Kurtzer:

So I'm not sure if that question was for Neil.

Zane Hamilton:

All of you.

Who Used Linux in the Past? [33:45]

Gregory Kurtzer:

Real quick. When I first got into Linux, there were no suits at any of these conferences. Like, it was coders, it was kids mostly. And I remember as Linux World 98, 99, 2000, 2001, it started actually changing, the demographics started changing to the point where there were more and more suits involved. And I think now it's going the opposite direction. Again, I'm starting to see it at SCALE. I wasn't there, but I heard that there were a lot less suits than are typically there. Sorry, Neil.

Who Uses Rocky Linux Today? [34:23]

Neil Hanlon:

No, absolutely. Go for it. Yeah, I think that it's an interesting group too, and we see it represented as well in the Rocky community. There's a variety of age groups that are participating and I think that that's a good thing. And there has been a change, I think, similar to what Greg was saying. And part of that, I wonder if it's just like wearing less suits, right? But the people that we're seeing come from all across the globe and all across different walks of life. And that's represented too, I think in the user groups that I've had the pleasure of being part of or going too. I was at SouthEast LinuxFest last year as well as Ohio Linux Fest back in December.

And the Rocky community has been invited to participate in some virtual Linux user groups as well, both in EU and the US. And so it's really great to see all of these people still coming together even virtually after all of this time that are participating in this free software idea project thing, trying to keep it alive. And, I think you see people from everywhere, from teenagers to people in their seventies, eighties, nineties contributing what they can, how they can. And I think that's super awesome.

Arthur Tyde:

I think at SCALE, SCALE was the most diverse event I've ever been to hands down. You had old guys, young guys, all different races, all different identities. It was very, very broad whereas when I think back to the early Bay Area Linux group, we used to have our average meeting size for a number of years was like 300 plus people. It was busy, but this was before Reddit and I mean, really sort of Facebook and online forums and so forth. We had IRC, that's what we had back in the day. And it was pretty bad. It worked. Yeah, it's like, ugh. But whereas today it's such a different crowd. I think that the user group seems to be smaller, but the Linux events seem to be getting bigger. Like, I was surprised at how big SCALE was. And I wish I could have joined you Neil, in Ohio for that. For the Ohio Linux. I'm trying to remember what they called it.

Neil Hanlon:

Fest, the Ohio Linux Fest. Thank you. Yes.

Everyone Is A Linux User Today [37:08]

Jeremy Allison:

The interesting thing about who's a Linux user though, is everybody is a Linux user now, and that's something that I think most people don't recognize. And I know you're going to say phones, Android or whatever, but hey, many people use iPhones. They're not Linux, but every single embedded system.

Neil Hanlon:

My goodness.

Arthur Tyde:

This is called a boot floppy for you, for Neil.

Zane Hamilton:

Neil. Have you ever seen one of those?

Neil Hanlon:

I know what the save icon is.

Jeremy Allison:

But every single TV display, every single TV manufacturer is now an embedded Linux system. I mean, almost certainly the monitor that I'm watching right now is running an embedded Linux. Yeah. Yeah. It has become the Universal OS really. And even Microsoft who fought against it for the very longest time, I remember reading a statistic that over 50% of that cloud is now a Linux machine. Yeah.

Rakuten's Support For Linux and Open Source Protection [38:25]

Arthur Tyde:

Well, in Telco infrastructure, I mean, what Rakuten did, amazing. You guys should, Greg, Neil, you guys should mention the Rakuten announcements. So if you haven't seen it, you should look it up. But Rakuten not only went with Rocky Linux, which we're obviously a fan of, but they really took a stand on what open source means to them as a corporation, which was the right thing to do and brave so kudos.

Rocky Linux' Open Source Origins [38:57]

Gregory Kurtzer:

It's more than even just open source. Because so much of what we use on a daily basis is open source. It was the separation of commercialization from open source. It was having a true community. We've seen a number of times now where open source projects, which are controlled by a single company, have done negative things to that community in order to favor a business model or business agenda. And when we started with Rocky Linux, I mean, just to be blunt, I was thinking about it from CIQ, we'll go and hire some developers and we'll create CIQ Linux. Right? That was my first thought to be completely frank. 

Arthur Tyde:

Linus's first thought, I'm going to make it a commercial kernel. Yeah.

Gregory Kurtzer:

But what we heard back from the community was, please don't put this into a company. Keep this separate and build something that better associates the community's needs to business interests. And when I say business interests, I mean professional interests, like professional enterprises, cloud, hyperscalers, the whole ecosystem of professional users, right? They need to know that somebodies got their back in a way that's much more than hobbyist and community. And thus the Rocky Enterprise Software Foundation was created.

Why Was Rocky Linux Not Built With GNU? [40:31]

Arthur Tyde:

Sorry, can I ask a very uncomfortable question? Of course, yes. Why Rocky Linux and not Rocky GNU Linux?

Gregory Kurtzer:

So I will give my idea.

Arthur Tyde:

Only, the old timers are going to get that. Yeah.

Gregory Kurtzer:

I'm going to give my quick 2 cents on this. So, okay. The Rocky Enterprise Software Foundation has nothing against GNU, Copyleft, et cetera, nothing at all. But I personally have now been in two situations where entanglement with GNU stopped a business process from flowing. And the easy one to understand is company acquisition, right? If there's GNU software tied up in that project or the product in any way, shape, or form, and they do full massive audits during a company acquisition, in many cases that will actually create enough risk or concern with the acquiring company that it'll either block or other architectural changes will have to incur if there's time. So from the Rocky Enterprise Software Foundation, again, focusing on the balance between community need, community leading, community interests, and how we also better support the enterprises and rights. So we're not against GNU in any way, shape, or form, but when we write software and we release software properly under the RESF, it is a non-Copyleft license.

Arthur Tyde:

Thank you.

What Are The Benefits Of Open Source? [42:20]

Rose Stein:

Greg. Can you speak a little bit more to the benefits of open source? You were really giving some good history and background and benefits. Are there other ways in which having things be open source? Because quite frankly, I mean, you have your hand in several open source projects that are used widely throughout the entire world. Warewulf, Rocky, Apptainer. So, what have you seen, like why, I mean, to a lot of people, and you were just saying, and thank you so much for your honesty. It was like, well, at first it was like, okay, this is going to be awesome and we can create a great product from it and like make some money and have a business and all that wonderful stuff. And then you decided to go open source. What are the benefits of that?

Gregory Kurtzer:

Well, I can tell you my side of it, and I'm really interested also in Jeremy and Neil's perspective as well. Because they too have their own experience and whatnot in this that may differ from mine. But from my perspective, I see open source as a development model, it is a community engagement model, and it is a way of creating software in a way that, to my perspective and experience, is much more supportive of creating a better solution. And it is that way because you have more eyes on it, you have more people involved in it, you have more intention, more companies potentially involved in it. You have a more altruistic perspective in terms of creating something that matters. And there's a lot of open source projects out there that don't have a community behind it.

It's a single individual, and those are incredibly valuable as well. But the beautiful thing here is, as more and more people start to use that open source project, naturally you're going to have more and more contributors contributing to that open source project. And as that continues to grow, there's going to be a critical mass that gets hit at some point where now all of a sudden it's more about the community than the project. And when we first saw this happening from when CentOS was the end of life, it was the end of life in such a way where you can tell from the community perspective, there was not a huge amount of support for end of life in CentOS. Right? CentOS moved to something new, CentOS Stream, which is a great project. I love what CentOS Stream is solving and providing to the community.

But in doing so, they also end of life what everybody knew and loved and whatnot. And that wasn't a community decision, right? And you can tell that very easily because of the number of people that were against this and upset over this. So the last thing that we wanted to do, even though my first intention was this will be a great asset for CIQ, that's the complete wrong way of thinking about this. And within, I think it was the first six hours after putting forth that comment to the blog, which caused Rocky Linux to basically get kicked off into action. It was within hours. It was like this is not the right model. We're creating a community distribution. And a month and a half later when we had 10,000 people join, and we've announced some of our goals and whatnot, we had not only the community grow, but we had companies clouds, like AWS, like Google, like Azure, like VMware, who all jumped on board and they said, we love this vision.

It needs to be a community that has alignment with the commercial interest, but a balance that it's still controlled and managed by the community. No single company can come in there and muck with it.

Arthur Tyde:

Well it's interesting because when I think about how the Linux Foundation got started very much in the early days, it was all community driven. And the guy who's the chairman of that, Jim Zemlin, has done an incredible job with the Linux Foundation over the last 20 years. It offered a home for Linus to be, because back in the day you remember it's like, oh my god, what? Like at Linuxcare we were all like, oh my God, when a VA hires Linus we're screwed. You know? And it's like, what if IBM hires Linus? It's like, where can that go? But by putting certain key open source projects under a neutral space, all boats in this particular hardware get to rise. It's really, it becomes a community of innovation.

Gregory Kurtzer:

Yeah.

Arthur Tyde:

So I, sorry, I didn't mean to.

Jeremy Allison:

So, open source, I think this is something that we've figured out in the early 2000's, is the scientific method of creating software. And I've heard Linus say that.

Gregory Kurtzer:

Boom.

Open Source Is Not A Business Model [47:19]

Jeremy Allison:

Yeah, that's what it is. People made the mistake in the early days of thinking, Hey, this is a business model. No, it isn't. Linus can prove that. Sorry, it's not a business model. Wow. Sorry, I was a little mean, I apologize.

Arthur Tyde:

So service and support isn't a business model first.

Jeremy Allison:

No, it is, but the idea of it is.

Arthur Tyde:

I'm just poking the bear. I love it. Jeremy, no matter what you say.

How Has Copyleft Disrupted The Tech Industry? [47:56]

Jeremy Allison:

Step three something, then where's the profit? Yeah. So another minor point before I shut up and let Neil talk is you're saying that Copyleft software has disrupted things. So, my perspective on it, because my project Samba is copyleft. So the thing I've discovered over the years about that is Copyleft is a great license to disrupt an existing market. So Linux being Copyleft allowed it to disrupt and essentially replace the Unix market. Samba being Copyleft was going into a proprietary dominated Windows file server and Novell file server. It enabled us to disrupt and make a space for ourselves in that market. For greenfield stuff, for new sites, for new software, Copyleft isn't necessarily the right way to go. And I think it's very perceptive there. Using an Apache license or something more open when you're creating something new that helps bring on more collaborators who aren't worried then about being forced to contribute when they don't want to. So, from my perspective, Copyleft is great when you're trying to disrupt an existing market. New markets, Apache style licensing works better.

What Is Copyleft? [49:28]

Rose Stein:

So, sorry Jeremy, I was just looking this up as you were talking. All of you guys are nodding your heads, but there might be somebody else who doesn't know this term. Copyleft?

Jeremy Allison:

Oh, sorry. So, Copyleft was, I mean this was an amazing hat by Richard Stallman, the founder of theGNU Project. What he realized is that the standard Copyright license, that, when you write something, it's yours. It gives no rights to anybody else, it's yours, you can show it to people or whatever. They have no rights to copy it. They have no rights to use it. This is basically what the entire publishing entry's based on what Stallman realized was that you could use that restrictiveness to force openness by saying you have no rights to this software, but if you use it, you can use it on the condition that if you change it, you must give those changes to everyone else who you give the software to. So we actually used the restrictiveness of copyright licensing to essentially force openness. And like I say, when you're disrupting an existing market, that's just a perfect thing to do. So it's obviously a lot more complex than that, but that's the 10,000 feet view, I think.

Rose Stein:

That's awesome. Thank you. And I was just looking it up, so like the C goes the other way, I don't know which way. Yes.

Jeremy Allison:

That's why it's Copyleft.

Rose Stein:

Copyleft. Yeah. Wow, thank you for that. That really explains a lot.

Gregory Kurtzer:

Jeremy. I like how you described it about disrupting an existing market and I think you're totally right about that. I haven't thought of it from that perspective before. That makes a lot of sense.

Zane Hamilton:

It is. I'll let Neil go and then I've got my Copyright story.

The Benefits of Community Controlled Open Source [51:15]

Neil Hanlon:

Yeah, sure. I think it was just sort of what Greg was talking about. I mean our goals for the Rocky Enterprise Software Foundation and having things owned by and for the community, it goes into a recent news sort of thing and decision that Docker had made and walked back regarding their Docker open source and free teams which affected Rocky Linux because we have images that were part of this free team tier. And thankfully they've walked it back and also extended their open source program here. But I think it's not only like a lesson in messaging about policies and changes, but also about having things that are controlled only by and for the community and having a way to establish that.

I mean, Docker Hub has become the de facto, or is for a lot of people the de facto place to get containers from. If you're running Docker, that's where you're going to be holding containers from. And other projects like Podman and some other container run times have flipped this on its head and have some other ways to discover it. But by and large Docker Hub is still that location for where people are expecting that. And to have that be owned by a company that is able to make decisions that unilaterally affect a lot of organizations, a lot of people, and not just from a monetary perspective, but if organizations chose to leave Docker Hub, what happens to those name spaces? What happens to those people that are using those programs that aren't going to go and look for images that are coming from other places that the security updates, the everything else that comes around with that. So I think it just really outlines the importance of having organizations like the Apache Foundation, Enterprise Foundation, and other organizations that support these projects and are able to protect from unilateral attacks.

Why Commercial Open Source Does Not Work [53:04]

Zane Hamilton:

Thank you Neil. Yeah, early on in my career I spent a lot of time with what now people don't call it J2E anymore. Now it's just JE but, there were only a couple of versions of real commercial job application servers that you could pay for. And those companies were not interested in listening to what a customer wanted. They were building what they thought you wanted. So being able to go out to the community and have something built that was scalable and actually incorporated the features you needed because we would actually find bugs in the proprietary versions and go back into the code and find where the problem was and go back to the vendor and tell them, this is broken. We need this fixed to move on. And they'd be like, well, we'll see, we can work it in the next release by then. It was easy enough for us just to go to the community, find the same bug or whatever was going on in Java there, fix it and move on. And that was something that we spent a lot of time doing was migrating people off of those proprietary tools onto open source projects. Because we actually could go make a change. We could fix the bug ourselves and not wait. Exactly. So it became very beneficial from a scaling standpoint to be able to be a part of it instead of just waiting

Jeremy Allison:

The ability to control your own destiny and say, this software is close, it doesn't do exactly what I want. But when you have the source code and the rights to use it, then the power is to say, well I'm going to make my own version that does what I need it to do. That's yeah.

Zane Hamilton:

It's amazing how often you are not the only one looking for something there. Or a lot of people or a lot of companies looking for that same thing but are just not being heard. Yeah. Being able to be a part of a community where you can actually be a group of people being heard is important.

How Does Copyleft Ensure That Companies Do What Is Best For The User? [54:45]

Gregory Kurtzer:

And going back to the Copyleft discussion, I believe I saw a comment from Snoop who said that the conditions that Copyleft is putting out there is exactly from a goal like blocking exactly the point that I was bringing up earlier regarding it blocking an acquisition. But that's part of the goal, right? To make sure, and I think that's what Snoop was saying, and Jeremy you're saying this as well, that's part of the goal of that license to ensure that companies are always doing the right thing. Right. The problem occurs though, depending on how many people you have at the table in terms of copyright, because copyright still exists and if you own 100% copyright, you can actually even change licenses or pivot the project.

And so it's more even than just Copyleft in terms of protecting this is one of those times in which you want to have a large number of people being part of an open source project. And the last thing I'll mention on this is, I've actually contributed to a project that, I'm not going to mention the name cause I don't want to badmouth it, but it's a project, put it this way, that every Linux and every Mac uses, and I contributed to this project back in about 2003, 2004, fixing some scalability bugs and the author could not accept my contribution even though it was an open source license and had to rewrite it in a somewhat clean room method to ensure that they still own and control the copyright of that software. Right?

Jeremy Allison:

Right.

Gregory Kurtzer:

And it was unfortunate for me because I put some time and effort into coding that and maybe my patch wasn't perfect, maybe he did better, but it would've been cool to have a patch into that software base.

Jeremy Allison:

Don't feel bad. Greg, my very first patch, open source was to Stallman for GCC and he said, kindly does what you need, but it's terrible code. I'm not taking it.

And, at that time I just went, he replied to me!

Rose Stein:

Oh my god.

Gregory Kurtzer:

Oh, that is awesome.

Jeremy Allison:

Yeah, it's code. But he sent a reply. I was so happy.

The Future Of Linux As a Server Operating System [57:13]

Zane Hamilton:

So Steve has a good question here. With the rise of containerization in cloud native technologies, how do you think the role of Linux as a server operating system will evolve in the future?

Gregory Kurtzer:

That's a great question. Wow. Go ahead. Art, go ahead.

Arthur Tyde:

Yeah, I'll be real brief. I think the trend for Linux on the server for a long time was to get bigger and bigger and bigger and bigger. I mean now it's bigger than a DVD. Whereas I think with hyperscalers hosting containers, I think we're going to see a return to smaller, tighter, highly efficient Linux builds. And I think that'll probably be the predominance of operating systems. If you think about operating systems right now, I mean Android phones it's small, they have to fit on a device. And hyperscale, same thing. It's like you have a small Linux operating system running KVM or some other hypervisor and then a bunch of stuff. So yeah.

Less Code Created More System Security [58:25]

Jeremy Allison:

It also comes down to security as well.

Arthur Tyde:

Pardon?

Jeremy Allison:

It comes down to security as well.

Arthur Tyde:

Very much so

Jeremy Allison:

The smaller it is, the less code you actually put in a container or in a system that you're deploying, the more likely you are to have security. And that's why provenance software materials become incredibly important for creating products using Linux.

Zane Hamilton:

Yeah. So there was a good statement. Yeah, yeah. So there was a statement that was made. I just looked, and I didn't see a copy of it anywhere, but the statement was, containers are Linux, choose wisely. So even though it's going away from an operating assistant perspective, it's still Linux in the container. So I don't think Linux is going anywhere. Oh, Greg's got his hand up.

Arthur Tyde:

Neil.

Zane Hamilton:

Oh yeah?

Neil Hanlon:

No, I was just going to say Linux containers.

What Is The Difference Between The Kernel Space And The User Space? [57:17]

Gregory Kurtzer:

So I get very pedantic about this when we start talking about it, because I have some background in containers as well. When we talk about an operating system, especially with Linux, we're talking about two major areas, right? We're talking kernel space and then user space. And the kernel space is Linux, period. Yeah. The user space is Linux only because it's compiled to be compatible with that kernel, but it is a user space that sits on top of the kernel. When we talk about containers, we're talking specifically about user space, right? When we're talking operating systems, we're talking Linux with a user space. So how are we seeing things shift? We're seeing things shift to the point where it used to be kernel space and user space was tightly aligned and it came as one thing, right?

And that's how we normally, again, think of an operating system. What containers has done is said basically we're taking the top half of this stack and we're basically just going to use the libraries and other user based components out of that. And that now becomes a container. So the notion of your operating system has actually now greatly expanded because we've broken the operating system into these two chunks, these two pieces. It always was, but now we're further decoupling those and to the point where you can now distribute the entire user space of your operating system, as Neil was saying, through something like Docker Hub and container run times can very easily leverage that, right? It used to be called a chroot, to all of the old people here. It used to be a chroot, right? But now we've extended the notion of a chroot with name spaces, with c-groups and other things where we can run that chroot as an entirely separate operating system. I'm blanking out now, cause it's been a long time. So I touched Unix. But since I've messed with those Unix, like this has already been a thing in other Unixes.

Jeremy Allison:

Yeah, Solaris had, what was it?

Gregory Kurtzer:

Jails, right?

Jeremy Allison:

No, BSD has jails. Solaris had something else.

Zane Hamilton:

Zones. I filled way too many zones. That was not fun.

The Future Of Kernel Level Virtualization With Linux [1:01:45]

Gregory Kurtzer:

And this is the different types of virtualization as well, right? So now we're talking about kernel level virtualization versus operating system level virtualization. So how is the operating system changing? Well, it's changing because it's now no longer just running bare metal. It's also now used in other ways to really help distribute and manage applications and services. And so I see that the operating system market, Rocky has become more important because it's not just the operating system that we're installing to bare metal, it's the operating system that we're also running on top of cloud, on top of virtual machines and inside of containers. So that operating system stack the user space is incredibly important.

Jeremy Allison:

Yeah. And with the new code in the kernel, the kernel is getting better and better at supporting containerization. I mean, as you say, the c-groups, name spaces, the mount name spaces, all of the things. Many, many years ago I remember people asking us, oh, can we run multiple versions of Samba on Linux, like slurry stones? And the answer was always, no, I'm sorry, the OS doesn't allow it. We're binding to one port. That's all you get. You know? And now of course with the expansion in the kernel space to add those features, I don't know what's missing. I'm sure there are things missing there, but I think that's going to be a massive thrust in the future to make that not only work better, but easier to deploy as well. Because it's no good having all this complexity if nobody can use it. I'm saying, you said using zones is terrible. Yeah. We need to make this stuff really easy to use because the complexity is hard. I realize, by the way, I neglected to mention in my initial bio, Hey, I just joined CIQ! This is my first win.

Zane Hamilton:

Glad to have you here!

Jeremy Allison:

I feel like someone has just connected a fire hose to my brain.

Zane Hamilton:

That doesn't stop for a while either, Jeremy.

Jeremy Allison:

I know.

Zane Hamilton:

I've been here 18 months and I still feel like that

Jeremy Allison:

Yeah, the containerization stuff is probably it.

Arthur Tyde:

I haven't seen talent like this since back in the day, so it's been amazing.

Rose Stein:

Makes me want to bust out into song. Back in the day when I was a kid. I'm not a kid anymore, but sometimes we're kinda coming to the end guys. This is like an amazing hour with amazing people. Thank you all for being here.

Zane Hamilton:

Absolutely. Thank you guys very much.

Neil Hanlon:

My webcam decided to die at a good time.

Zane Hamilton:

Oh, it died.

Neil Hanlon:

Linux, you know. What can you do?

Arthur Tyde:

Next webinar. Rocky Linux, the musical.

Zane Hamilton:

There we go.

Neil Hanlon:

I think they already have that?

Zane Hamilton:

Rose and Art. I can see it now.

Rose Stein:

I'm a down man. I got the tunes.

Neil Hanlon:

It'll be our traveling show. The Rocky Horror Picture RoadShow.

Gregory Kurtzer:

Rocky Horror.

Richard Stallman Receives The Linus Torvalds [1:05:06]

Jeremy Allison:

I have one more Stallman story, which is fun. And I remember this was at the Linux Foundation. I can't call those events in New York. And they, because Stallman's there, right? He's close by. They invited him to give him the Linus Torvalds Award, which did not go down well. So he stood on stage holding this thing up. I don't know why it was some golden penguin or whatever it was. And he said, you do realize this is like giving Luke Skywalker the Han Solo award.

Zane Hamilton:

Oh wow.

Jeremy Allison:

Oh dear. Fantastic. Well people move on. Newer people come up and they take the stuff that you've built and they do better and more amazing things and that's just the way it goes. And that's a good thing.

Zane Hamilton:

Absolutely. It's a great one. Guys, we are up on time. We really appreciate you joining us again this week. Welcome back to all of you Jeremy. Welcome to CIQ. Glad to have you here. Thank you. Good to see you. Thanks for all the stories, Art. I actually thought you had been on after SC last time, so sorry for the welcome back. It's my first time. Thank you. See you again soon. So thanks everyone. If you like and subscribe, we appreciate it. We will see you next week. Thank you. Thank you. Have a great day.