Apptainer (formerly Singularity) 1.1.0 Has Been Released

We are excited to announce Apptainer 1.1.0 has been released! Join us as we walk you through the new features and updates.

Webinar Synopsis:

Speakers:

  • Zane Hamilton, Vice President of Sales Engineering, CIQ
  • Forrest Burt, High Performance Computing Systems Engineer, CIQ
  • Dave Godlove, Solutions Architect, CIQ
  • Krishna Muriki, HPC Systems Design Engineer, KLA
  • Dave Dykstra, Computer Professional, Fermilab

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Good morning, good afternoon, and good evening, wherever you are. Thank you for joining us for another CIQ webinar. My name is Zane Hamilton. I'm the Vice President of Sales Engineering here at CIQ.

What are CIQ and Apptainer? [00:24]

For those of you who are unfamiliar with CIQ, we're a company focused on powering the next generation of software infrastructure, leveraging capabilities of cloud, hyperscale and HPC. From research to the enterprise, our customers rely on us for the ultimate Rocky Linux, Warewulf, and Apptainer support escalation. We provide deep, development capabilities and solutions all delivered in the collaborative spirit of open-source. Today, we've got a really exciting announcement to make, or we're going to announce the release of Apptainer. And I want to bring on our panel, I think there's more. Perfect. So I'm really excited to have Dave Dykstra here, and I know we've all introduced ourselves except Dave. So I'm going to start off and have Dave introduce yourself and tell us who you are and how you're involved with Apptainer.

Dave Dykstra:

Okay. My name is Dave Dykstra. I work at Fermilab or Fermilab National Accelerator Laboratory in Illinois. And we support high-energy physics software including doing all the research, doing a lot of the research. And we have used Singularity for a number of years. So I'm the support person for the high energy physics and also the high Throughput Computing community that we represent.

Zane Hamilton:

That's fantastic. Thanks Dave. Mr. Godlove.

Dave Godlove:

Oh, it is going to get confusing with the multiple days.

Zane Hamilton:

Your microphone is really loud there, Dave. Wow.

How Dave Godlove got Started with Apptainer [01:56]

Dave Godlove:

Okay. Yeah. Good to see everybody. This is going to be a little confusing as we say, "Dave, Dave, Dave". But this has been a problem, well not a problem, but a great aspect of the Apptainer community for some time. Lots and lots of Daves. I'm Dave Godlove. I've introduced myself several times, but I used to be a neuroscientist at the NIH. I got interested in high performance computing, and joined the Apptainer, which was then the Singularity community shortly after it began. I've been working pretty closely with the project for a long time, and I'm glad to be here. And I'm also glad to see Dave joining today and Forrest and Zane.

Zane Hamilton:

It's awesome. Forrest, welcome back again.

Forrest Burt's Introduction to Apptainer and HPC [02:46]

Forrest Burt:

Thank you, Zane. Hello everyone. My name is Forrest Bert. I'm an HPC Systems Engineer here at CIQ. I originally started using Apptainer, of course, when it was Singularity. While I was working at Boise State as the student SysAdmin on their high performance computing architecture there with their research computing team. I was looking for ways to install packages for researchers, and install software for researchers. That was something a little bit different than what was out there. And ended up finding Singularity, used it to deploy a lot of workloads. And now I'm the container guy of sorts with regards to some of these things. So I've been a user of Singularity, and Apptainer for a while. And I'm really pleased to see Dave and of course, Dave, two major figures in the community on here as well.

Zane Hamilton:

That's great. And so Dave Dykstra, whenever Dave Godlove refers to you, he calls you Dr. Dave. But when I see Dr. Dave, it's really both of you, because you're both Dr. Dave. So I'm going to refer to Dave in this context. It's going to be Dave Dykstra. Godlove, I'm going to call you Godlove, because that's what I call you around here because we have so many Daves. So let's start off and talk about what we are announcing? I know there was a new release of Apptainer, so let's talk about what the release is and then dive into it a little bit. So I will go with Dave Dykstra. Tell us about the new release.

What are the Big Changes in Apptainer's 1.1.0 Release? [04:03]

Dave Dykstra:

All right, so the big change in this release is that it's now rootless by default. So we are not installing this UID root component. And this is possible now or practical because we are using unprivileged username spaces in more places than before. Most especially the mounting of SYF files can be done completely unprivileged. And then it also does sometimes use features of mounting of Fuse2fs, mounting the ext3 files, images, and also overlay, which most of the time Apptainer does not need to use overlay, but sometimes, if you want to extend, especially if you want to extend an image, you can do that with overlay. And then the other big thing this time is the unprivileged build. So it always used to be that you had to be root in order to do the build, and now it can be done as an ordinary user.

Zane Hamilton:

That's fantastic. So, Dave Godlove, what does that mean for the end user? How is this going to help in some of those contexts of security and workflow?

Dave Godlove:

Well, the building aspect that Dave just talked about is really huge for users. So in the past it used to be that if a user wanted to build a new Singularity container or Apptainer container, they would be required to have their own Linux environment in which they were root, right? so to do that they would either have to be able to spin up a VM or maybe have a cloud instance or maybe have a dedicated resource that was their laptop or something like that. And that's a, that's a significant hurdle to an average HPC user who maybe the only Linux environment that they typically have access to is that within the high performance computing system.

And obviously they're not going to have roots there. So this is going to allow people to build right in the production environment where they normally work. And that's really huge. Also as far as not needing to be root to, I mean, not having that SUID, or root owned SUID part of the workflow, I mean, oh, hey, Krishna. That's also huge from an admin perspective. So now we're leveraging username spaces, which were included I believe in Red Hat 7 and are on by default in Rail 7 and Rail 8 and Rocky 8. So they're enabled by default. And so now, it's a Linux feature, which is already there.

And I think that one of the cool things about that too is that it extends something that Singularity and Apptainer have been doing for a long time, which is using Linux Kernel features and file system features and things which are already there, and not doing anything exotic as far as trying to put security in. But just utilizing these features which already exist and which are already well known and well used and well trusted to enable security, I think it continues that tradition, which is really cool.

Zane Hamilton:

Great. Thank you Dave. Welcome Krishna.

Krishna Muriki:

Hello. Hi everyone.

Zane Hamilton:

Would you like to introduce yourself and tell us how you were involved in the community for Apptainer?

Krishna Muriki's Background with Apptainer [08:04]

Krishna Muriki:

Hi. So, I'm Krishna Muriki. I've been working with Greg for more than a decade now. I was employed at Berkeley Lab and being involved with this project from the early days of Singularity. And over the years the project evolved more and I tagged along with it. And when, recently when Apptainer was forming Dave was looking for volunteers on various activities related to it. And I put my name in. I don't think I'm doing enough even though my name is there. But I hope to contribute more. And right now I am listed as the official release manager, although Dave has been doing a lot more than me right now. Other than that I'm employed. Right now I'm employed with KLA Operations and we make vapor inspection products like powerful microscopes.

In other terms we're looking at vapors at nanoscale and identifying defects. Those are the products. We make microscopes with huge cameras and a lot of data coming in and crunching on Linux clusters. And we are, as a company, as the products that we are in, we are interested in and adopting containers and the organization, and Apptainer is the right way to do it. And that's why we are interested as an employee, we're interested in contributing to the community and being able to leverage the product that's coming out of the community. There is a huge interest in Apptainer, and we have been keeping eyes on all the features that are getting released with the current 1.1 release. We want to use the products that are about the ship in the next few years.

Zane Hamilton:

That's fantastic. Thanks Krishna. Dave, I think you had some, something else you wanted to add to or talk about? When it comes to security?

Dave Dykstra:

Yes. Well, first I wanted to say Krishna is also on the technical steering committee with me and a few others including Greg. So yes, I wanted to add, you used to be able, even before this release, used to be able to do unprivileged builds but only with the fake root option. And that fake root option required system administrators to set it up with SubUID and SubGID. So a lot of people haven't had that experience when their system administrator had set up that mapping for their userID there. And so now in this new release, the fake root option is extended. So even if you don't have those things set up, you can still build, you can still run the fake root option. And it does that by using unprivileged username space and then also using the fake root command, which fakes out the build and package installs and things and thinks that it's running as root when it really isn't. But it's sufficiently so that for building packages it works great for that.

Zane Hamilton:

That's great. Thank you, Dave. So how have we actually gotten to where we are? So, I know it started off as Singularity, a lot of things have` been put into the product, the project, and then over time it moved into where we are today. So Dave can you tell me that story of how he got here?

Dave Dykstra:

You want to know the whole history or you're in particular about this release?

Zane Hamilton:

No, let's hear the history , and then we'll talk about this release further. Like, how did we get to where we are today and why is this release important to us?

Dave Godlove:

In the beginning there were two roots.

The History of Singularity and Apptainer [12:26]

Dave Dykstra:

Well, in the beginning Greg wrote Singularity 2, written in C and then it grew a lot in popularity. In fact, it grew a lot in popularity already at that point in our community and the Throughput Computing community. We saw a lot of potential in going, especially we needed to be able to use it as unprivileged users who were running pilot jobs at sites. And then those would then run user payload jobs. And so we needed this capability. We, from the beginning, saw that things were soon then moving to totally unprivileged; it wouldn't need a SetUID root. And so we thought that would be a way to go. And so some people from our community, especially Brian Bockelman started contributing at that point.

And we started using it early in Singularity too. And then Greg left LBL and started the CyLabs company. And, they rewrote it in go and made a whole lot of new features. During that time period, we started using it in our community, started using it unprivileged, and we were able to do that because we use the CERNVM file system to distribute our containers. We don't need SYF files. So it doesn't need to have to mount anything. So we, for several years, have been running unprivileged. But it's long been my personal goal to be able to have it be, by default, unprivileged because we know that that's the way the Kernel support was going.

And because that's where other tools have gone. And so, I've requested a few years ago, Hey, let's make it totally unprivileged and. So well then Greg left CyLabs and took the project with him. And then after about a year CyLabs decided to make a fork of the projects, I call it Singularity CE. And that was very confusing to the community to have a Singularity and a Singularity CE. So then Greg led us to the Linux Foundation. And then the Linux Foundation required a name change. So that's how we got to the name change of Apptainer. And then, since that time, I've gotten more deeply involved and I thought my top priority for Apptainer was to make it totally unprivileged. So I've been putting a lot of effort into that in the last six months.

Zane Hamilton:

That's great. Thank you.

Dave Dykstra's Role in the Apptainer Community [15:14]

Dave Godlove:

And Dave, you haven't really said it yet, but I mean everyone should know that I think that it's not maybe official, but you've pretty much assumed the role of the Community manager or the Community Liaison and also of the Release Manager. So Dave is the one who's been running the meetings, the community meetings that occur, two times per month. And he's also been the one who's been setting the milestones and figuring out what releases are going to contain and what they're not going to contain. And prodding people to get things done. And so he's really jumped in and filled that role that has become empty. So thank you Dave for doing that.

Previous Security Concerns and How Apptainer Solves them [16:00]

Zane Hamilton:

Absolutely. So before we go into that more, Dave Dykstra, when we start talking about being able to run things unprivileged and we talk about security, what use case problem does this solve? Or what does it enable for an end user that is important and why were you pushing for this for so long?

Dave Dykstra:

Well, SetUID root is notoriously difficult to secure. I think CyLabs, the developers there and Greg took a lot of care to try and make sure that was secure, but it's still notoriously difficult to secure. And there can be a lot of attack vectors on that. And so it's something we try to try to avoid when we can. Now on the other hand, the unprivileged user name spaces have had quite a few CBEs in the Kernel where exploits come along, which can be exploited through the use of unprivileged user name spaces. And so in our community, and also we've now recommended to the whole Apptainer community, is we advise turning off network name spaces because almost all of the CBEs in the last several years that can be exploited through privileges and name spaces have had to be done in combination with network name spaces.

So if you can avoid it and it is used by other container tools by default, but if you don't need them or if they also can be given an option to not use it. They're also used a little bit by SystemD, but those can also be turned off since various packages of SystemD use network name spaces. But you can disable that and we get instructions on how to do that. So we think it's a good security trade-off. There's also a difference between a small number of people looking at it versus a large number of people. Whereas a large number of people, a huge number of people are constantly going through the Linux Kernel looking for holes, whereas hardly anybody's doing that for Apptainer. So it's a question of security by obscurity versus security through experts. I'll say we want to have many, many eyeballs on things because that's the way you make it ultimately secure.

Zane Hamilton:

Absolutely. So Forrest, how do you see this changing? Because I know you've spent a lot of time building containers in Container and then executing them in another place, but how is this going to be different for you?

Apptainer's Increased Freedom in Building Containers [18:41]

Forrest Burt:

Well, in a lot of ways, things are somewhat similar as was mentioned to what you could do with that fake root option to be able to build things unprivileged, but this just provides a lot more freedom with how the user can be able to build their containers unprivileged in an environment. This gives users a lot more capability to build their containers perhaps within the context of a cluster. As we mentioned, it makes it easier if they don't have to spin up a root environment or anything like that to be able to get on and get working with Apptainer now. So in general, it just makes it a lot easier for the users to get up and running with creating their own containers. There's no concern about root, there's no fake root has been expanded and stuff, so it can do more now.

And just in the end, it makes it easier for the users to get their workloads containerized and up and running on their architecture. And there are some technical considerations that we're exploring with different things you can do with a container now that it has some of these capabilities around CICD, things like that. So we're looking into that, but primarily the biggest thing I see is that this makes it a lot easier for the user at the end to not have to be concerned with the security model they're dealing with and just be focused on getting their application containerized.

What is the Biggest Challenge to Making an Apptainer Run Without Privilege?

Zane Hamilton:

Great, thanks Forrest. So we do have a question from Sylvie. So what is the biggest challenge to making Apptainer run without privilege? Leave that one up to anybody who wants to answer it. I think that's probably a fairly deep question.

Dave Dykstra:

I can answer that. So well at this point of the Kernel development that really, there was nothing stopping it except for time to develop it. Because they were already available. I mean, you can do mounts. So the mount is the biggest part of what was needed to be able to mount a container. So you want to be able to mount a Squash file system, for example, and that was a privileged operation until unprivileged Fuse was allowed in the Kernel. So that allows from inside and unprivileged user name space, you can then do Fuse mounts. So there was already a Squash Fuse available. So what's left to do is tie that into Apptainer and then there's a lot of different combinations to test.

Now it did turn out, this is something we didn't find until late once people gave us benchmarks to run. It turned out that Squash Fuse in parallel mode was a very poor performance when there's lots and lots of files to be opened. So you're running in lots of cores, they're all opening up lots of files with the same single Squash file system that turned out to be extremely slow. But that was fixed in the final release by using a multi-threaded patch for Squash Fuse. So that was the one that was the potential killer to the whole thing but we are very fortunate to have that multi-threaded patch ready and it worked really well.

Why is Privilege Needed? [22:12]

Zane Hamilton:

Thank you, Dave. So Dave Godlove. So for those who are less experienced in all of this, can you walk me through the step-by-step of why privilege is needed?

Dave Godlove:

I'm really glad that you called on me when you needed a less experienced viewpoint. That's, I mean.

You did the right thing . So this is a great question. So, as we were talking here, I started to think to myself there might be people wondering right now, why is any of this privileged at all? Why do you need this? So there might be even people who didn't even know this was a privilege, I guess. So as a user of an HPC system, when you run the Apptainer command, what you're doing is you're leveraging, there are parts of the workflow or were parts of the workflow. Up until recently, up until now, that was running an executable, which was owned by root and had the SUID bit set, which was basically making it run as root. You were becoming root even though you didn't know it for some of the operations that you were running and those operations were running on your behalf.

Now that's not that unusual of a thing. There are other programs that do that on your behalf that allow you to safely do things within the context of a particular program that normally only root would be able to do. But as Dave alluded to before, that's something that's pretty hard. It's hard to lock that down and ensure that nobody can leverage that for nefarious purposes. So in the case of Apptaine what really happens behind the scenes is that you've got a file system sitting in a little image somewhere typically within a SYF file. And inside of that, within a SquashFS image. Apptainer locates that file system and it mounts it to your host file system. And that is one of the parts that up until recently, up until the work that we're discussing today, needed to be done as root because mounting a file system onto the host file system is something that requires escalating privileges.

Then after that occurs or perhaps before, I'm actually not totally sure of the order of operations, but Apptainer allows you to pivot into that new file system within a newly created namespace, which presents that file system to you as though it were the root file system. And so those are the basic operations which are necessary. There's a lot of other things that occur, but those are the big things that occur to set up an Apptainer container. And so that mounting operation, that's a privileged operation. And so now with these new developments, there are ways to be able to mount a file system without using privilege. And these are the things that that Apptainer is now leveraging.

Does the Building Process Also Require Privilege? [25:32]

Dave Dykstra:

Thanks for that, Dave. And then in addition to building containers. So when you build a container, you are installing packages. So it's like you're doing a YUM Install, that's usually a privileged operation. It tries to write into your written file system and such. So that is a privileged thing as well. So that's why we needed to either give it a fake root or a real root and in order to do that.

Dave Godlove:

It might be worth noting too that the building operation is not inherently something which requires privilege. And a long time ago when we were first coming up with that, one of the things that I worked on too was a long time ago helping to put together the build commands to create new containers. And while I was doing that, I sort of had the idea that building a container, and I talked about this a couple of weeks ago, building a container entails the operation can be lots and lots of different things. You can build from lots of different sources, and you can end up with lots of different things depending on what you want to build and what you want to build from.

And not all those things have to be privileged. And even currently within Apptainer, if you want to build a sandbox you can do that unprivileged, right? Because all that requires is taking an image and just dumping it out essentially, or filling up a directory. So for a while I thought, well, maybe we won't require you to be root, or we won't require you to build with privilege, and we'll just see how far you can get and maybe some things you might be able to build without privilege. But in practice it turned out that almost anything of consequence that you want to do when building an image, you have to be a root user. You have to be a privileged user to do that. And it got confusing to just allow users to try to build things unprivileged as we were doing testing during development.

The Risks Associated with Building with Full Privileges [27:52]

So we basically ended up putting in a check to see are you root when you go to build something. But another interesting topic to bring up, or an important topic to bring up, is that this came up several years ago as well building a container and in doing so with full root privileges is actually a pretty risky proposition. You risk the integrity of your host operating system, your host file system when you do that. You can, I touched on this a little bit a couple weeks ago too, when I was, when I did the whole thing on, on building containers. But you can make mistakes using the wrong sections, and you can make changes to your final system.

And that's something that you can pretty easily guard against just by not using the sections, but there are more nuanced and complicated things that can get you into trouble. Recently it became possible to bind mount directories from the host system into your container at build time, which was previously always disallowed. And I have a few times now had a container binder environment variable set and forgotten that I had it set because I was running containers and went to build containers and said, oh, I don't have privilege to write here. Why don't you, or what's going on? Because I've used fake roots instead of real roots. But if I built those containers as root, then I, oh man, suddenly I'd be writing to my host operating system.

And even if you don't make those mistakes, you almost always build a container from another container. And if somebody decides that they want to do bad things to you or bad things at least to your computer, it's possible for them to craft containers that very quietly do things to your operating system and not in the container. Just with very, very simple commands during the bill process, if you do so as root. And so that's why in the past we always suggested that people throw away VMs to build their containers and then just discard them when they're finished with them. But now using these unprivileged ways of building containers, it really makes it a lot safer and a lot nicer to build your containers.

Zane Hamilton:

Dave, I take it you've never messed up a host system before?

Dave Godlove:

No.

Zane Hamilton:

Krishna, I think you had something you wanted to add?

Why Not Needing to Root is so Important in Building Containers [30:45]

Krishna Muriki:

This ability to build containers and not needing to have the root. When I was in the role of supporting the research community at Berkeley Lab, that was a big hurdle in container adoption by the user community from non-CS domains, like people like the domains without a lot of computing knowledge, if they have to even build up a discardable VM for building containers, that's too much of an ask for them. And that was a big hurdle for all of them to adopt containers, especially Singularity, was to build everything from scratch. So they would just go and do it using a Docker container, and try to convert that from Docker to Singularity, which can be done without operation, as they may. That's the part they were taking, but they are limited by the docker containers they have access to.

So they need to have additional processing, additional steps. They have to get everything working in Docker, do the conversion to Singularity, and get onto these huge shared machines where only Singularity was available for them to make use of container systems. So this, being able to create Apptainer containers without the root privilege is really a game changer for situations like those. And this is something that we have been waiting for quite some time and thanks to the community and Dave for staying behind it and pushing it out, making it available. Thanks. Thanks, Dave.

Zane Hamilton:

Thank you, Krishna. So Dave, what else is in this release of Apptainer? I mean, I know a lot of it is focused on security and being able to do things we've been talking about, but what else is a part of this release?

What Other Changes and New Features Were Added to Apptainer 1.1.0? [32:55]

Dave Dykstra:

Oh, there's so many miscellaneous things. I don't know if it's hard to say what's more important. We've touched on the most important ones. There's a bunch of bug fixes, and a bunch of minor new features. You also did import many of the fixes that CyDev put into Singularity CE. I did want to mention also on the build topic that a lot of people talk about, well we want to be able to build a package to optimize for particular HPCs. And so they want to use the compilers from the local HPCs. And so that's another thing that's helpful to have, to be able to build on the machine where you need to run the containers. Nesting. Okay. That's a,

Can you Have Containers Inside of Other Containers? [33:55]

Zane Hamilton:

That's a great question.

Dave Dykstra:

That that is another,

Zane Hamilton:

I feel like we've, we've got into this one before and it becomes very interesting. I know Dave and Greg, I think you talked about this too, but go ahead, Dave..

Dave Dykstra:

Yes. So that's another big advantage of the way that this was implemented in, in 1.1 and that just about everything can be done nested inside of other images of the containers. So even though Apptainer runs things with no new privileges set, so once you've started inside this privilege, you can't do anything set UIDs, anything set Cap, you can still do all the same operations inside them now? Well we can still do the fake root build that is with this fake-fake root, I call it thing, the real fake root is with SubUID, SubGID mapping, that one you can't do inside because that's also a privileged operation. So this is a lot of times there's a whole ecosystem called the rootless containers.

One thing that they also do still require some increased privileges and a little bit of increased privileges, is that program that runs it, that reads from SubUID and SubGID, that's a little increased privilege program. So we can't run that nested inside of another container because we don't allow that. So, but this one, this fake-fake root can be done inside. And all the mounting of Fuse file systems is done in an unprivileged user name space. It does not require the, normally, you can actually run most Fuse programs straight from outside of a container. But what that does also use is a set UID root assist program, the Fuser mount.

But that Fuser mount doesn't get used if you are running in an non-privileged username space at the first level when you first started an unprivileged username space, you have another fake root, but your one user ID is running as root there. You can do a Fuse mount, but then we always switch back to the original user ID once the user starts executing the user code. But you can then run another, open another user on privileges and namespace and do some more mounting. So everything can be, all the new features at least can, can be nested.

Why Would You Want to Put a Container Inside Another Container? [36:26]

Zane Hamilton:

Thank you for the question, Rose, maybe I'll show a little ignorance here. What would be a use case? When would I do that? When would I want to do that? To nest a container within a container?

Dave Dykstra:

Somebody else wants to answer that?

Zane Hamilton:

Oh, go for it. Forrest!

Forrest Burt:

I was just going to say that it has a lot of usability within, like, the context of CICD systems. Docker, for example, has a system that's somewhat similar called Dockering Docker, that allows you to use a containerized build environment to automate your container builds. So you can set up within the context of a container, an environment that can build containers in it. So the first thing I think of is that being able to build Apptainers inside of Apptainers is massively useful for being able to get a lot more automation around how you can build and test those. Like I said, especially with CICD systems like those they can get through on GitLab, GitHub, BitMarket, places like that.

Krishna Muriki:

And in commercial, one use case where I am looking right now with my employer, there are multiple teams developing various applications, which are interdependent. And if the application stack is too deep and there are dependencies and there are various teams developing them, Apptainer is giving them the flexibility to say, okay, this is our latest container, and build your application stack on top of this container. It's giving that flexibility and freedom to various things to do on their own and also declare what their requirements are and how they can run easily in this nested approach. That is what we are adopting here internally. So the new approach in hardware is adopting GPUs in our computers, in our Linux computers, within this way for inspection, which is right now with GPUs having the base OS stack separate and the NVIDIA stack separate. And on top of that, having the framework the, application base framework and then having the application, all of these things, the various teams were doing each of these stages in this application stack, and all of them are releasing their own versions of containers. And we are doing it nested, running one container on top of one other container. That's another example of how we can leverage it.

Zane Hamilton:

Oh, that's great. That makes a lot of sense. Thank you.

Krishna Muriki:

Especially when the application stack, it's too deep and there are a lot of teams working on it, not just one single team who are all sitting around and talking and developing that stack, but various teams geographically distributed and they're developing their own piece of the puzzle at that point. It gives a lot more freedom and there's less communication that needs to happen and less moving everybody together will be difficult. So so making

Zane Hamilton:

It's very modular.

Krishna Muriki:

That's one example of how we are using it here at KLS.

Zane Hamilton:

Thank you, Dr. Godlove. I think you had something you wanted to add to this too.

Apptainer's Support of Using Containers Recursively [39:48]

Dave Godlove:

I mean, so we're having a little sidebar chat here where we're talking about, I mean, this is basically recursion, right? This is, using containers recursively. And as anybody with any programming experience will tell you, recursion is very useful in lots of different contexts. So when you're talking about containers, basically anytime you want to manipulate or have some sort of some sort of workflow or whatever, which depends upon containers, it might be good to containerize that thing, which is dealing with containers. And anytime you're doing that, you're talking about nesting containers. One concrete example, I have a silly little container scanner that I've got sitting around in various places on the internet, and what that does is it's in a container and it's got some vulnerability scanning tools, and it asks you to point it to the container that you want to scan, and then it grabs that container and ingests it into itself. So the container goes into the container and then it starts scanning it inside the container. To do that, you've gotta be able to do nested containers, right?

Zane Hamilton:

So scanning it in terms of it actually executes the container and starts scanning it against ports that are open or that actually

Cyber Security Applications of Recursive Containers [41:10]

Dave Godlove:

No, it's a lot more civil. So this thing is actually it's scans for just, it's not the way that people typically think about scanning for vulnerabilities within containers. It doesn't look for CBEs and stuff like that. What it does instead is look for actual viruses and rootkits within the container. And so the way in which it does that is that it's got some scanners built into the container and then it's pretty silly actually, in that it just uses the container machinery inside of it to convert, to pull the thing out of the SYF file and convert it to something like a bear directory that it can just walk through and scan. Like the quickest and easiest way I could do it. And it uses nested containers and does what it needs to do.

Zane Hamilton:

Thanks Dave. Dave Dykstra, I think you had a topic you wanted to bring up about Apptainer and Podman.

What is the Difference Between Apptainer and Podman? [42:12]

Dave Dykstra:

So this is a big, let's say, philosophical difference, different applications than that Podman and Apptainer. So Apptainer is really focused on being able to run higher level applications, let's say number crunching applications, right? Whereas Podman is designed to be more, well, it's compatible with Docker, it's trying to do system services, it's trying to be able to, trying to pretty much emulate as closely as possible an entire whole Linux system. So that difference allows us to be, we can do a difference, since it's a different goal, we have different types of approaches that we can have. So like the whole, it's very important to Podman to have this SubUID, it's a SubGID mapping, so that each user gets a whole range of let's say 64K user IDs that they can work with.

Whereas most of the time we don't care, we only care about one user ID. We don't care about emulating the network because we can just share the network. So there are a lot of things and simplifications that we can do in Apptainer. I mean, it's also, in terms of scanning goes, security wise, I care very little, very few of the CBEs that come out that will apply to Apptainer because we don't run services, we just don't run services. So the only thing that it might apply to is on outgoing or your clients or out your clients, let's say a TCP client. And there might be some possible way that you connect to a malicious server that might somehow come back up through your connection. So there's a lot of simplifications that we can do with this different goal. And so I think it's worth having the two different container run times that are in use and different purposes.

What Does It Really Take to Run an Open-Source Project? [44:14]

Zane Hamilton:

That's very helpful, Dave. Thank you. Because I think that what gets brought up quite often is what is the difference between the two and that makes a lot of sense. So I appreciate that. So we are getting close on time, but before we do, Dave, I want, as someone's sitting on the outside watching open-source, I don't think that I myself don't really have the perspective of what it really takes to run an open-source project. What does the build process and release cycle look like, and how do you manage that?

Dave Dykstra:

Hmm. Oh, it's a big open-ended question.

Yes. I mean it's, it's really about using and having the right tools for one thing GitHub makes it really quite easy. And a way to be able to have, anybody can contribute and you can review everything very carefully. So you can run a whole bunch of tests. Every time you make any small change, it runs a large suite of tests automatically, let's say we have it set up and GitHub makes that quite easy. I mean, then it's also a matter of also getting volunteers, and managing people. That becomes a big aspect of it. And I mean, since CyLab split off from our community, we did suddenly get a big drop in the resources that have been being put into it.

And so I've stepped up now. I'm hoping that there will be more people that will maybe take over from me some of the stuff that I've taken lately. But also I felt it's important to have a community-based project and one that's not focused on the needs of a for-profit company. And being in the Linux Foundation is a big help for that too, with requiring at least five people to be a committee that's in charge instead of a single individual. So that's another big aspect of it.

The Importance of Developing a Strong Community [46:40]

Krishna Muriki:

Git surely helps. I mean, this is the role where I learned a lot of Git more than any other roles that I have in that. It is a good opportunity to learn really important skills and being in the volunteer position here volunteering for an open-source project like this. There would be a lot of issues that communities put in. And looking at those issues is another way of learning what's happening in the tool and understanding how the tool works. That would be another aspect of running an open-source project like this. Issues that come in, monitoring and keeping an eye on them. Following up on them, that is a little bit of work that's involved in running a project like this. And for anyone listening and interested in learning I would recommend starting there.

Look and keep an eye on the issues. Most of them probably would give some answer to them. You see how an issue is coming in and what kind of responses, what kind of back and forth happens, that's a quick way of learning. If you're interested in contributing to the project. And then if you have a lot more time, you can join the bi-weekly, two calls per month. Joining those calls and at least listening to what is being discussed, how decisions are being made when the scheduled discussions that we're having, what features do we want to include in the next release, what we want to push for a later time. Listening to those discussions that happen, those calls, that's the next step that you can get involved in. I would say

Zane Hamilton:

It's fascinating to listen to from my perspective. So I sit through those and find it very interesting. So there's another question I was going to ask you, Dave, and now I've lost my train of thought here.

Security Challenges Associated with Open-Source Projects [49:01]

Dave Godlove:

I was going to say, well you already done it, so I don't have time, but I was going to say, while you regain your train of thought, I always thought that one of the most challenging things about doing the open-source development and helping to guide the open-source development is really security related issues. That's where things become really complicated, especially with an open-source project like this because the development is all supposed to happen right out there in the open, and it works the best when it happens right out there in the open. But then if you do have a vulnerability, especially one that you don't immediately have a patch for, things become complicated as far as like, well, how do you make sure that the parties that need to know about this do know about this and make sure that nobody else does as you shepherd this thing through and get a patch done and get it released and do all that.

And that's a complicated complex thing to do in an open-source community like this.

What Can You Do to Get Involved and Help? [50:08]

Zane Hamilton:

For sure. We haven't had stuff lately. Yeah. Most people don't ever think about it until you're involved with something like this. So I think it's really interesting from the outside. I remember what I was going to ask you, Dave, is when you're having these community meetings and you are seeing that you need help, where, so the people watching this, where do you need the most help or where is something that from a viewer's perspective, where could you really use help today?

Dave Dykstra:

Well, it's just to contribute. If you run across a problem or something that you think could be improved, just try to work on contributing the code. You can ask for some help. I've benefited greatly by being able to communicate with the people who did the original implementation via Slack and you can ask them particular questions. That's a big help. But definitely, if you have an idea, please submit bill requests. And when I started working on this, I didn't know Go at all when I started, before I started seeing it in Singularity. So learn the language. You can hack things without learning the language too. You can. Google is a great help for that.

How to Start Working on an Open-Source Project [51:27]

Krishna Muriki:

The community does have a lot of nice people who just reach out. As you look at the comment logs or the traffic in the issues, you can see who has been responding on a specific topic. You can find the same person in Slack. And if you do your private message or in the general chat, community members do respond. They respond and give you helpful hints. So more Prs if community members can submit PRs fixing for specific issues as Dave said. Surely helps. And also as new issues come in if you are familiar with that and you know the answers for that, any help you can give any giving quick responses to those issues. Also, there's another idea where we can, the project, we do help from volunteers.

Dave Dykstra:

Yeah. I mean, it's also a help to give us an issue. If you run across a problem, please let us know about it.. Instead of just saying, well, I could work around that instead and not let us know. I also, I did have a plea for quite a while. I said, please, people test out, give performance testing on this. And it didn't happen until very late. And then it was like, oh, no, . And we did only get one person to give us some benchmarks. So we would still love to see more people run their benchmarks, run their code on Apptainer 1.1.0 and with both, set UID mode and not set UID mode and see if you can detect differences and if they see any differences, or even if you don't. If you just do the test, let us know. Give us some results.

Dave Godlove:

That's a great way to start. If you want to begin being part of the community and you don't really know how to do it, that is a really, really great way to start. I mean, that's the way I started as well, is that I was watching development and every time Greg would do a new release candidate, I would go through and just test everything. I mean, it was possible to do that at that point in time, but I would just go through and test everything that was in the CLI to the best of my ability on my local resources. Then I would ping Greg in Slack and be like, here's all the stuff I tested, and here's what worked and what I'm not sure about and what didn't work. And so that's a great way to just get started, get moving, and it's definitely something that's really needed.

Zane Hamilton:

That's great. Thank you guys for that. So we are running up on time. I want to make sure Dave has enough time to get to his next meeting, but I am going to open this up to final comments about whatever you want to say. Apptainer, this release. Plea for help, whatever you'd like. I'm going to start with you Krishna.

Krishna Muriki:

Try it out. 1.1.0 came out with really amazing features. Try it out and use it and open more issues if you can. If you find andy and come and join the slack channel monthly calls, and contribute to the code if you want. If you have seconds available.

Zane Hamilton:

Thank you, Forrest.

Forrest Burt:

I would just say that here, a year after we moved into the Linux Foundation and switched from simulator to Apptainer, I am really pleased to see still such changes being made to the software and such long running developments being worked out, so it's great to still see it. There's a lot of activity there and Apptainer still has a lot of active development and a lot of great goals that are being implemented and left to go. So it's great to see it still active.

Zane Hamilton:

Thank you Forrest, Dave Godlove.

Dave Godlove:

So I would just say that I feel like this release of Apptainer, the 1.1 release, it feels like a new milestone. I mean, it really feels like a new era opening up within Apptainer. Just the way things are progressing and obviously as a community effort, a lot of people deserve a lot of congratulations, but I also want to just call out, once again, Dave in particular, Dave Dykstra in particular, and thank him. I think you had this idea a while ago and you stuck to it for a long time and it was a lot of work to see it through from start to finish, so congratulations on getting that done.

Zane Hamilton:

Absolutely, Dave.

Dave Dykstra:

Thank you. Thank you.

Zane Hamilton:

Closing thoughts?

Dave Dykstra:

So this tool has been really, really important for the community that I support, the I Throughput Computing community. And so, it's really important to us that it has a long life. And so that was, even though we don't really need this, some of this functionality. I thought it was really important for it to be there. So I thought I could do this with a little bit of help, mostly from Cedric. So tell me what needs to be done. And this could be the same way for anybody who would like to contribute to it. This is a community project and we're part of the community and every user is a part of the community.

So please contribute what you can. And let's continue to have a long life. I mean, I see people having presentations that say, oh, we should all switch to Podman. Well maybe if it changes quite a bit so that it can do all the things that Apptainer can do, but why should it. Why not have two run times? I'm not sure about that. I think it's important for it to be active and being under active development.

Zane Hamilton:

Absolutely. Thank you very much. Well, we thank you guys for joining us today. We hope you come back next week. Again, Dave and Krishna, thank you for joining us from the outside and Dave, Forrest it's good to see you again. We really appreciate it. You guys like, subscribe, and we will see you next week. Thank you.