Getting involved with Apptainer Development
Webinar Synopsis:
Speakers:
-
Zane Hamilton, Vice President of Sales Engineering, CIQ
-
Brian Phan, Solutions Architect, CIQ
-
Forrest Burt, High Performance Computing Systems Engineer, CIQ
-
David Godlove, Solutions Architect, CIQ
Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.
Full Webinar Transcript:
Zane Hamilton:
Good morning, good afternoon, and good evening, wherever you are. Thank you for joining us in the continuation of our Container Education series. My name is Zane Hamilton. At CIQ, we're focused on powering the next generation of software infrastructure, leveraging the capabilities of cloud, hyperscale and HPC. From research to the enterprise, our customers rely on us for the ultimate Rocky Linux, Warewulf, and Apptainer support escalation. We provide deep development capabilities and solutions, all delivered in the collaborative spirit of open source. Like I said, we're actually going to continue on with our container education series, and we have a great group of guys here to talk about it today. I appreciate it. Pardon any background noise. There's something going on behind me. Might be a little loud. I hope you can't hear it, and I'm sure you can now. So I'll let you guys introduce yourselves and I will mute. Brian.
Brian Phan:
Hi everyone. Brian Phan here. I'm a solutions architect here at CIQ. My background's in HPC administration and architecture. I've been a user of Apptainer for a while, but have not had experience developing the tool. So I'm excited to learn from Dave today.
Zane Hamilton:
Forrest.
Forrest Burt:
Hey everyone. My name is Forrest Burt. I'm an HPC systems engineer here at CIQ. I've been a longtime Apptainer user as well. I used to use it to deploy workloads out for academic and national lab researchers at my previous institution. I've used it for all kinds of stuff these days, but much like Brian, I've never been a developer of it either. So I too am very excited to learn from Dave as to how that setup all works.
Zane Hamilton:
Thank you. And if you are not familiar with Dave, if you are in the community, I would be surprised that you have not run across Dave before, but Dave, go ahead and introduce yourself.
Dave Godlove:
Hey, everybody. I'd like to start off by saying that I am not actually an Apptainer developer, but I have played one before on TV. So my name is Dave Godlove. I've been around the Apptainer community really since it started pretty much. I have developed Apptainer, although it's been a long time since I've been an active developer of Apptainer, really not since it was written in C and Bash. I've also been the release manager for Apptainer for a while. And there helped to manage development even though I wasn't one of the ones who was actively developing it. So hopefully I can say a few things that are interesting and useful about the community and the developers, even though I'm not currently one of them.
Zane Hamilton:
Yeah, and you pretty much can't go through any of the documentation for Apptainer without running across something that Dave has written. He seems to have written almost all of it. I know there's a community. I know a lot of people do, but Dave definitely has his fingerprinted on a lot of things in that documentation for sure. So Dave, why don't you give us a little bit of background on the community and where it came from, where it is now, how it got there. Just tell us the story.
About Apptainer [08:07]
Dave Godlove:
Yeah, absolutely. We thought that we would spend this time to talk a little bit about, not so much like developing containers or developing workflows and things like that, but more developing Apptainer itself because I mean, this is a great opening welcoming community, and I think this is a great project if you're interested in getting into some sort of like software development, but you don't really know where to start. I think this is a great project to dive in because the community is so open and organic and everything. Apptainer has been around in some form for like six years or something like that at this point in time.
And it used to be, Greg started the project when he was working at Lawrence Berkeley National Lab some time ago, and he sent out some announcements. He created a Slack channel, he created a website, and people just sort of organically got involved and started to mess around with it. And I think that the project has really been organized mostly around GitHub as a lot of projects are today. And we've really used GitHub to great advantage to organize the community and organize the project and everything. So it was singularity for a long time. And I believe that you can still find the repo, I believe it's under HPC NG at this point in time, but you can actually find the old Apptainer repo, which was renamed from Singularity.
And so you can go back on that all the way back basically every commit all the way back to the beginning of the project almost. And so now the way that the project is structured, there are a group, and unfortunately we couldn't get any of them to join today, everybody's pretty busy, but there are a group of people that are associated with the Linux Steering Committee, and they are the ones that have commit access to the repo. But you can go to the Apptainer repo and it's now just on GitHub, it's Apptainer slash Apptainer, and that's where the code is, and that's where you can get involved and raise issues and create PR's and do all that kind of stuff.
Linux Foundation [10:59]
Zane Hamilton:
So, Dave, I know that it moved into the Linux Foundation, but what has that done for Apptainer? What does it actually mean for the project? How does that help? What view has it given it?
Dave Godlove:
Yeah, so the Linux Foundation is now an official sponsor, an official host of the project. I, myself, am not one of the members of the steering committee and I haven't been deeply involved in that, so I can't speak about that in depth. Maybe somebody else, maybe Forrest or Brian can also comment on this as well, but I know that we're always looking for ways in which to sort of make sure that Apptainer remains open source and remains community driven and remains something which is not really dominated by any corporate entity or so on. And I think that helps us to be able to make sure that container stays open source.
Zane Hamilton:
So have you seen growth in that community since it's been moved over to the Linux Foundation? Have there been more people coming in to help actually do development documentation? I mean, I feel like there are already a lot of users of it, but it seems like maybe there's been an uptick in that as well. Is that something you've seen on the project side?
Community Growth [12:22]
Dave Godlove:
Yeah, I think so. I mean, I feel like for a while there when once again, maybe somebody else can comment as well, but for a while when I was watching, I tend to watch issues and PR's and the different things that come into the GitHub repo, I don't really comment on them, but I at least follow them and lurk and watch them to some extent. And I feel like for a while there, it was pretty much the same kinds of names I would see over and over and over again, creating prs and creating issues and stuff. And I feel like recently I've started to see a lot of new names on a lot of new faces. I don't know if the actual volume of issues and PR's has increased.
And I think that partially that's because Apptainer's gone through several different phases of development as like a big, long, ongoing project will, and one of those phases of development it went through was, we built out a whole list of different features that we wanted to add to Apptainer, and we organized all those into different milestones for different releases. And this is when we had like a dedicated development team, and we went through and we prioritized stuff, and we just sort started adding stuff. And during that period of time, obviously the number of PRs and issues was just huge. The volume that was going through. Apptainer has now matured, I believe, as a project. And this could always change.
But right now I think that the user base is probably, and I'll let Brian and Forrest maybe speak to this too. I think the user base is probably pretty happy with the feature set that Apptainer has. And so now we're more in a mode of just sort of trying to make sure that we fix bugs and we keep things up to date as things on the underlying system change and things like that. It's more of a maintenance mindset versus like adding features mindset. So I don't think we've seen a really dramatic uptake in pull requests and issues, but definitely a lot of new faces recently for the ones that we do have.
Zane Hamilton:
Great. Forrest, I saw you come off mute a minute ago.
Forrest Burt:
I was just going to say with regards to the community, what was the question that you just alluded to us?
Zane Hamilton:
We were just talking about moving over to the Linux Foundation and what that's look like for the product itself?
Forrest Burt:
Apptainer is a really, really mature technology. It, as Dave has said, has been in development for quite a while now. So it's definitely at the moment it's been deployed over like 20,000 or 25,000 sites worldwide, so it's very well known, just like Dave says, it's at this point a very mature technology, and it's in a maintenance mode of just making sure that CBEs are applied to, and bugs are fixed. There were some major developments not too long ago with bringing Rootless capabilities to it, which was pretty neat. But overall moving into the Linux Foundation was also meant to represent the goal of being able to cross pollinate with some of the other projects there and starting to see how Apptainer can fit in with those. Because again, it's a very mature technology and at this point a lot of the work there is on just maintaining it and making sure that it has the same level of quality for users at all levels.
GitHub Repo [16:11]
Zane Hamilton:
So Dave, let's talk a little bit about the source code for the project in GitHub. How is it structured? I mean, talk to us about that.
Dave Godlove:
Yeah, so let's see. Maybe I could just share a window really quick.
Zane Hamilton:
Can we please just have it shared on your screen to a camera again?
Dave Godlove:
Yeah, let me do that. All right, at this point in time, I think most people in the scientific community and definitely in the development community and so on, have probably they're probably intimately familiar with GitHub and how it works and everything. The way that the GitHub repo is structured here used to be that there was a main branch, and then there was a development branch, and then there were a bunch of tags. And so the main branch was, at that point in time, it corresponded with whatever the latest tag was, which was the same thing that you were supposed to get whenever you pulled main.
We've since changed that so that main is development, and I think that this seems like a small change, I think. But it took us some discussion to come to this. It was a big question mark as to whether or not we were going to do that. And I think that the reason for that is because GitHub is serving multiple different purposes and multiple different communities here. One of the communities that it serves is obviously the developers who write the code for Apptainer, but another community, historically, that GitHub has served is the end users. So GitHub was used not only as a place to develop code, but also as a distribution platform. And I think that earlier in the project, we were concerned that when end users download GitHub which maybe has a hard learning curve, being a little bit harder, especially for non-technical users to figure out.
And so I think that we were concerned that some of our less technical users who just wanted to use the software might not realize that if they download and install main they're getting stuff that might not be stable. It might be development stuff. And so we wanted them to be able to install the main branch and then have that stable. This has been several years ago. Since then I think there's other distribution methods that people are using, like DNF and repos and things to install Apptainer. So I think that that concern is a little bit less of an issue now. So anyhow, there's the main branch, and that's development. And then we've got a couple of release branches here. We're currently working on the 1.1 release branch.
Maybe we should have a 1.2 here as well. I don't think there is a 1.2 release branch, but there probably will be pretty soon. So, and then 1.0 is probably deprecated, because I don't think that we're currently working on 1.0, I don't know if there's some major bug fixes that went into 1.0, maybe we would backport them. So these branches here just have to do with some automated bots that we have running in the background. And then tags, this is how we do our releases. So, you know each one of these tags corresponds to some release, and of course you can also go over here and see the releases. So that's basically the way things are structured now. And the way that it works is that whenever somebody creates a new poll request, which is merged, it will get merged ... I believe the way it's currently working ... Dr. Dave is the one who's currently pretty much running this, so I'm guesstimating. I used to run all this but for some time now he's taken this over and he's been running the releases and development and stuff.
But I think that the way that it works currently is, when you create a poll request you create that poll request against main, and it will get reviewed and merged into main, and then we use cherry picking to poll that poll request, just the commits, from that poll request over here into, I guess it would be the 1.1 release and get that ready so that we can we can release the next point release here. And so you'll see that this is just 1.1, even though we're currently on 1.1.6. So the idea is that whatever the latest release is, that's what is going to be 1.1.7 in this case whenever 1.1.7 comes out.
Zane Hamilton:
Well, Dave, you did mention one thing. You were talking about
Epple [22:16]
Dave Godlove:
Yeah, there's a technical aspect of it and there's also a social aspect of it. And so that's a place where Dr. Dave has really had a major contribution for a long time. He has been the long running Epplenet maintainer for years now of the latest release of Apptainer and previously Singularity. There's the whole just trying to package it as an RPM and trying to do all those types of things. But there's also the issues of, if you're going to package it for Epple, you can't just create an RPM. There are a bunch of conventions that you have to stick to, and you have to stay up to date on what's going on, and there's more to it.
That's why I said there's a social component to it. And Dr. Dave, for people who don't know, that's Dave Dykstra who works at Fermilab. Dr. Dave has stuck with that for a long time and has been in that community and has been a part of that and has really stuck with that for years. And I think more recently we have Yoshiaki Senda who works here at CIQ, who's another solutions architect. He has been working with Packaging Apptainer in Apt as a Deb package. So I'm not quite sure where that stands right now, but I think that he's also working with that and I don't know, Brian, Forrest, what repo is that currently in?
Brian Phan:
I think it's in PPA, but that's live now, so you can get Apptainer today if you wanted to. And I think we also have a blog post out for that as well, so, yeah. Cool stuff.
Zane Hamilton:
So, Brian, I know sometimes packages in Epple or in some of the other WN ones that they're behind significantly from what's in GitHub. How quickly does this follow a release cycle from the time that it's in Git till the time it's in Epple? Is there an average timeframe or I know different things take different amounts of time, but?
Dave Godlove:
I I think the timeframe is like a week, maybe. It's really fast. And I think we benefit greatly from us being lucky enough to have Dr. Dave both packaging this stuff for Epple and also being the release manager, because he of course knows exactly when each new release comes out and what's in it. I think he actually gets the RPM into Epple pretty much like within hours or less after the release actually happens. But then there is a process by which users have to show that they're downloading and using and testing. I think it's called Karma within Epple. But so it has to accrue so much karma before it's said to be generally available for everybody. But yeah, that usually happens pretty quick and I think that's why there's usually a small delay between the time that Dr. Dave puts it in Epple and it actually becomes available.
Raising Issues [26:18]
Zane Hamilton:
Interesting. Thank you, Dave. So let's talk about raising issues. So if somebody's working with the tool, how do they get involved and start raising issues?
Dave Godlove:
Yeah, raising issues is super, super easy. And once again, this is something that most people might know how to do, but just in case there's a few that don't, or just if you're new to this space or whatever. It's really easy to just go over. So once again, I was just at the Apptainer or main GitHub place, and I just hit issues and went right over here. And so oh, I gotta sign up for GitHub to do that. So basically you just click new issues, I don't know if I want to sign in right here in front of everybody. Hold on.
Zane Hamilton:
I was going to say, if you're going to do that, drop your screen.
Dave Godlove:
Yeah, let me do that. So I'll sign in real quick and then I'll show how you can create issues. Yeah, maybe somebody else wants to comment a little bit while I try to figure out my username and password.
Zane Hamilton:
Forrest, have you submitted issues to Apptainer?
Forrest Burt:
I can't say I ever have myself.
Zane Hamilton:
What about you, Brian?
Brian Phan:
I have not myself, but there is an issue there, that is a feature, that I would like that has to do with templating out definition files that would make reusing your definition file for different versions of a software easier to build.
Zane Hamilton:
Interesting. So are you going to raise that one, or do you want to ask whoever's watching to go do that for you?
Brian Phan:
Maybe I'll put in some PR's as well after learning everything from Dave.
Dave Godlove:
Yeah, so the thing that I wanted to highlight here is when you create a new issue we have pre-populated and marked down, so you write here, and then this is what it ends up looking like. We have prepopulated all these different questions, like what version of Apptainer, what did you expect to happen, what actually happened, all this kind of stuff. I will say that some people basically just start off by deleting all this and then just writing whatever. The thing that I wanted to highlight here, the ultimate goal of this issue, if it's possible. And it's not always possible because it's not always possible for us to replicate your environment, but the ultimate goal of this issue is to give us enough information that we can take this issue and we can run it locally and we can replicate whatever the bug or the feature request or whatever it is that you're looking for.
So do think about going ahead and filling in all these details about what's the operating system that you're using, and how did you install Apptainer, and what are the exact commands that you used to create this bug or what did you expect to happen or so on. Something like that. So that's what I wanted to highlight there.
PR's [29:36]
Zane Hamilton:
Thank you, Dave. So I think the next topic we wanted to talk about was prs, like going through that process.
Dave Godlove:
Yeah. So once again, if you're already a Go developer or something, this is probably pretty basic but it's a little different for different projects. And so I just wanted to go through the steps of creating a poll request here and the steps of getting it reviewed and how that all works. So it might be a little bit different if you're used to working on a project where you've got the ability on the main repo to create a new branch, because if you see a new feature or you see a bug or something and you want to create a PR to address it, the very first thing you want to do is you want to fork the repo.
What that's going to do is it will ask you, where do you want to fork this thing to? I've already got a fork, so I'm not going to do this. If you're part of multiple different organizations or whatever, but you want to fork this to your private space. And then once you've got that forked, you can go over to your fork and you can clone that. And then the next thing that you want to do, which I always forget, is you want to create a branch locally on that forked clone that you have. And so then you're good to go. You can go ahead and make whatever changes you want, and then you're going to push that to your fork, and then through the gooey or whatever, you can make a polar request from your fork directly back to main.
And so that's the process. If you're new to all that, if you're new to GitHub in particular, or if you're new to this project in particular it's not like you just create a branch on the main repo and then you just do a poll request from that branch to main, because you don't have the ability to create a branch on the main repo, you have to do that fork first and then create your PR from there. Another thing that new developers just getting started who maybe don't have a lot of experience, maybe don't really realize, how do you edit things? So you're going to get a review, and in that review you might be asked to change a bunch of things within your code.
If you just simply push more commits directly to your fork, the branch that you have set up in your local fork, then those will automatically go up and become part of the PR as you push them. So you don't have to worry about, what's the process for making sure that my commits go from this forked branch over to the PR, that syncing just all happens automatically. I should show you too, so we got a few different PRs open here, so whenever you create a poll request we've got some continuous integration which runs in the background, and it runs all these tests.
And that's another thing that you'll want to check out. The tests do everything from a bunch of basic actual functional tests where they actually download containers in VMs that are running up on the cloud and do things with them to just lintering your code. And so one thing that you'll sometimes you'll find is that tests fail, and it's just because whatever editor you were using didn't have a go linter installed. And so because of that, there's stuff that still works, but there's stuff that's not correct syntactically, doesn't look right. And so this linter will catch that. And so sometimes you'll have things fail and if you're new and you're just starting out, that can be frustrating to not know what's going on and why things failed.
And so just make sure that you're using a linter. Another thing I think is that the project now requires that you verify your commits. And so that's another little gotcha that if you submit a PR and your commits aren't verified, then you'll get nabbed by one of those tests there. And it can be confusing and frustrating to try to figure out what's going on. So just make sure that you verify your commits and there's a pretty simple way to do that. I can't remember off the top of my head how to do that, but I was able to figure out pretty quickly how to do that when I needed to. And then I jumped into here real quick because if you do have tests that are failing, if you see something like this, then you can just go in here and have a look and it can be daunting at first to try to figure out what's going on, but you know I don't know if I'll be able to figure out what's going on in this example.
But usually you can parse through this and you can figure out, okay, all these are running successfully. Okay, here's the part at which, oh, lint. So this might be an issue where once again, like I was talking about before, the linter caught something which was not correct, I'm unsure. But yeah, so this is the method of going through and figuring out if you've got something that's failing, why it's failing, and then you can get an idea of how to fix it. So after all your tests passed, another thing that has to happen is you have to have a review. And those reviews are usually done by the folks that are on the Linux Steering Committee. So those would be the people like Dr. Dave, Ian Kaneshiro, Cedric, other folks like that that can actually do your review. And I think actually anybody can review. But I don't know if those reviews actually count toward making it so that this PR can be merged yet. So that's something that if you want to get involved in, I think that anybody can go and review PRs. So you could go here and if there's anything that looks interesting to you, you could go through and test it and check it out and look at it and then add your review to it. Once you pass all the tests and once you pass reviews, then your PR will be merged, hopefully.
Cutting a Release [36:54]
Zane Hamilton:
Thank you, Dave. So let's talk about actually cutting a release. How does a release manager actually go about cutting a release?
Dave Godlove:
Yeah, so the way that's happening right now ... Let's see if I can find the milestones. Let me go back into the poll request for a second. So we've got milestones set up and this is something that Dr. Dave does. So right now we're targeting release 1.1.7, and 1.2.0. We follow semantic versioning to an extent, but not completely. And so the micro release here, like the 0.7 here, should denote bug fixes and changes that don't change the behavior. So changes that don't change things having to do with the CLI or anything that the user's going to touch. The minor version here, so the 0.1 here or the 0.2 here, those are going to denote things that might change behavior as far as the user is concerned.
So like you might have a flag that you were using or an option or an argument that works in a previous release that's going to change in this new release. And historically, we have not changed the major version release unless the actual format changes. And that's why I say that we kind of follow semantic versioning, but not necessarily exactly. So in the past, because we went from using ext3 images to SIF images, and we also had a major code rewrite where we went from C and Bash to Go. And when we were with Singularity, we went to 2.0. So the release manager then can put together these milestones, and these milestones have a list of different PR's that we expect to be merged or issues that we expect to be fixed before the next release is cut.
A lot of times as the release becomes nearer and nearer, you'll see that people sort of start to clamor for PR's and features and things that they really like to be added in. And that can become difficult because I don't know how much we're doing it now, but sometimes you'll release a release candidate, and the idea behind a release candidate is that, okay, we're now freezing development. We're not going to add any new features. This candidate is what we expect to release ultimately to be, and we're going to release the release candidate and we're going to expect the community to install that and test it and see if it's got any bugs, basically, see if it's got any issues that we couldn't catch via automated testing.
And so at that stage, sometimes you'll see that people are like, oh, no, but there's this new feature that I really, really want to put in. And it's like, well I'm sorry, but technically we need to wait until the next actual release, because otherwise we have to put out another release candidate and we have to start the whole community testing again. And the community's only got so much testing in them. Sometimes you get more or less testing out of a release candidate. So I think what we've moved to now, especially since the project has become a bit more stable, is the actual releases themselves. Those are what goes out because we don't expect there to be as many huge breaking changes or bugs or whatever in the releases themselves.
And so I think we're getting away from release candidates to some extent, but the flip side of that is that we might have more micro releases, right? So you might have a bug in a particular micro release and that's okay. And so you might do another one to take care of that. And so then the actual process of doing a release itself is just making sure that all the changes are in, they've all been cherry picked inappropriately. Sometimes when you cherry pick a set of PR's in, basically what you're doing is you're taking a set of commits that were made for one branch and you're pulling them out and you're messing with the history and pulling them out and putting them into another branch.
And because of that, you can get conflict, you can get like the same file was changed between these two branches, and then you're trying to add more changes to that. And so those have to be manually resolved. Somebody has to go through and actually look at that stuff and figure out what's going to work and what makes sense. And it used to be that the release manager, basically me at that point in time, would do all the cherry picks for the release myself. And then if there was anything that I could not figure out how to resolve on my own, I would go and contact, hopefully the developer who actually created that PR or one of the other developers and sit down with them and try to figure out what was the proper thing to do. Now I think that the way we've instituted it it's incumbent upon the person who created the PR to do the cherry pick.
And that's good because they're usually in the best position to be able to look at the way the code has changed and figure out the best way to resolve any conflicts. So anyway, the release manager will go through and make sure that all that stuff has been resolved properly and is all in there the way it ought to be. There's a few little housekeeping things that they have to do. They've gotta bump the version number and a few different files, make sure the change log is up to date and then basically just create a new tag, cut the new release, and document all the stuff on the releases page as to what's changed and everything. There's usually a few emails that have to go out, a few announcements that have to go in different places.
And then I don't know if we're currently doing this, but it's a good practice afterward to take that release branch and merge that back into main. So what you've been doing is you've been taking everything from main and you've been copying that as needed into the release branch, you might have some changes that ended up in the release branch through resolving conflicts and doing stuff like that, that you ultimately wanted to go back into main so that you can keep those two as synced up as possible as development continues. So that's the last thing that'll usually need to be done.
Zane Hamilton:
Thank you, Dave. And I think we had a few questions that got thrown up before we move on to the next topic. While you have GitHub window open, Dave, can you show us the "get involved with development" part?
Get Involved with Development [44:40]
Dave Godlove:
Yeah, certainly. So once again, one of the things that you can do is you can go over here to the issues and you can create new issues or, I mean, you can just hang out and read through these issues. This is a great place too. People sometimes have issues that are actually bugs that are actually things that need to be changed in the code in order to work properly, but sometimes people just need help, right? This is a place where people go to get support. And so this is a great way to get yourself more familiar with the project at a higher level, because it's one thing to just use it for your own purposes and to fix problems that you have.
It's a totally other thing to try to get into somebody else's head space and figure out what is their environment and look at what they're using it for. And you learn a lot by going through issues and trying to figure this stuff out. And another place, like I said, you can go through these poll requests and you can look at what looks good and what doesn't. You can review the code. If that's something that you're interested in getting into, if you want to start writing code for Apptainer, then a great place to start off is by reviewing PR's and having a look at them and seeing what code changes. So yeah, this is a way that you can get involved. Another great way to get involved is to join the Slack channel. Apptainer has a Slack channel.
Zane Hamilton:
I think we've already posted the link, if I saw it correctly. I think Forrest put it out there.
Dave Godlove:
Oh, cool. Thanks Forrest.
Zane Hamilton:
Yeah, it's out there.
Dave Godlove:
Yeah, so that's another great place that you can get involved, is just sort of watching what's going on in Slack and helping people out when they need it, asking questions, that kind of stuff. And it's announced in the Slack channel, but there is a twice monthly community meeting, and that is a really, really great place for just an hour twice a month if you're interested in what's going to go in the next release and what are the current problems that the developers are thinking about and what are we looking at trying to incorporate into Apptainer for the future? That community meeting. And Dr. Dave announces that in the Slack channel because he runs that community meeting right now as well, so he announces that in the Slack channel. And that's a great place to go to just keep an eye on what's being developed and what's new in Apptainer.
Zane Hamilton:
Dave, is that also a place where people could go if they wanted to hear what things needed to be or where people were looking for help, what they need help on?
Dave Godlove:
Absolutely, yeah. Or where the developers need help too, so it's like we've got these changes that we'd like to see incorporated into Apptainer, but we just don't really have anybody to work on this stuff right now. I mean, that's the kind of thing that gets discussed quite frequently in those meetings.
Apptainer Bugs & Issues [48:02]
Zane Hamilton:
Thank you. All right. As a user, what issues have you come across? It's a good question for Brian and Forrest.
Forrest Burt:
I honestly can't speak to any specific experiences that I've had with bugs in Apptainer for the most part since I started using it. It's been pretty much rock solid for the use cases that I was using it for. Honestly, I've always had a great experience using Apptainer. It's been one of my favorite tools since it was Singularity. As soon as I found it, I started trying to make a lot more things containerized back in the day as far as my HPC software deployments went. So I would say I can't really think of anything too specific. It's been a rock solid experience for the most part.
Brian Phan:
For me, also, no real issues. More of just like a small pain point on just managing multiple files for different versions of the same software, but we can solve that with a little bit of templating, so yeah.
Zane Hamilton:
That's great. Thank you. And I see someone ask if we could post the Slack link to LinkedIn as well. Appreciate that. I think one of the other questions that was asked was for Brian and Forrest as well, what uses have you seen or what have you seen people using Apptainer for?
Forrest Burt:
So like I said, my original experience with Apptainer was using it to deploy software in an HPC environment. So we had a pretty standard compute cluster that you might find at most universities in America. A nice fifty, a hundred fifty, two hundred node cluster, something like that. I don't remember the exact count, but a nice mid-size cluster. Like I said, when I first started I was doing deployments using standard module files stacks and that type of thing. I found pretty quickly after I started looking into Apptainer that the ability to, for one, be able to build and rebuild a piece of software to do iterations on getting into build correctly as both having to constantly be deleting and recreating a build environment on the cluster. Honestly, that was one of the biggest things that I found at the time, not having to go and do this kind of CM shared apps, CM shared module files shuffle to give my user permission there and just to do that type of software deployment safely with that.
I think one of the most memorable things that I ever put into a container was a large stack of software for a bird research or an avian research agency called the Peregrine Fund. One of their researchers was doing large scale research into the movement of Eagles through North America. And so they were using a containerized environment I put together to do mass data analysis of that. And then we had another guy who was using that same containerized environment to do analysis of flight paths as birds were flying towards wind turbines. So you could try to come up with models of how to best time like turning off wind turbines so that groups of birds can pass by without being hit by them. Like I said, all that research was done out of a singularity container that I put together for them. And that is the coolest specific example that I can point to of something that I've seen deployed with Apptainer that I've personally done. But that's just one of many examples of other times where Apptainer proved to be the best option for deployment.
Brian Phan:
Yeah, for me very similar experiences to Forrest. I've mainly used Apptainer to get customers up and running as fast as possible on the HPC system. So this ranges from containerizing, maybe a single application to run like a physics simulation to maybe even like a suite of tools, for example, to run a bioinformatics application that calls multiple tools.
Getting Familiar With Code [52:17]
Zane Hamilton:
Thank you, Brian. Thank you, Forrest. Dave, I think you also wanted to give us a few tips and tricks here for getting familiar with code.
Dave Godlove:
Yeah, I wanted to do a little demo. Hopefully, hopefully people find this useful. Let me see if I can present. Once again, I always have trouble figuring out what screen to present. I got too many things going on here. I'll give you my coding background a little bit. My coding background is kind of silly in that I used to be like a scientific coding person, meaning that the majority of my coding background was done in an IDE, in an integrated development environment. Now people have mixed feelings about that. Some people like me really thrive on that and really like that.
And then some people think that that's a crutch and are not so hot on IDE's, but if you like an IDE and you're new to Apptainer and maybe you're new to Go and you want to get involved, but you don't really know where to start, I wrote a little blog post that came out yesterday in which I detailed the steps. So you could search like CIQ blogs, if you do that, you'll go to CIQ'S website and you'll be able to see our blog posts. And the most recent one that came out yesterday is this one about debugging Apptainer and actually other Go projects using Delve, which is the like the major Go Debugger and integrating that with something like VS code.
And so what you end up with is the ability to step through your code, line by line, set breakpoints, interrogate variables at runtime, and do stuff like that and kind of see what's going on. And so I just wanted to show you, so the steps to set this up are here in this blog post, I've already got this set up here on this machine. And so I just wanted to show you what that looks like and how you could use that if you're just starting off to get a good idea of what you should be looking at. So the way in which I've set this up, you basically run an Apptainer command at the terminal here, and then that command kind of pauses and waits for you, and then you can attach to that command using VS code. Let me show you what I mean. So I'm just going to go ahead and run a silly command. Just using the lol cow container, which is up on Docker hub. Oh, maybe I won't. I got a typo here. What's my typo?
Unable to handle the urI, maybe I'm not attached to the authorization required. All right, let me see something else that I've got that already works.
Here we go. Okay, so I've got a typo up here somewhere. I'm not sure exactly where it is. All right. So let's say that I wanted to debug that command. I want to step through and actually see how it runs. So I've got here an alias set up because I can never remember the entire command. And the alias is DB or debug dash app and it resolves to this entire long command. And so what it's going to do is it's going to start a delve debug server running headlessly, and listening on a particular port. And then it's going to end with these two things that say, okay, I'm done giving you options. And that's going to allow you to give this command whatever other options and arguments that would normally go to Apptainer. So after I've got this set up, this little command here is just going to replace my Apptainer command up above. Okay. So the way that that works is I would do DB app. Wow, install. Oh, okay. You know what, I'm sorry guys. Let me, that's not going to work right off the top of my head. Let me go off screen and try to figure out why that's not working really quick. I messed with my dot bash RC right before this and I think I messed some stuff up. Sorry about that.
Zane Hamilton:
It's always fun. Hope you made a backup.
Dave Godlove:
Yeah.
Zane Hamilton:
Eh, no. Speaking about BashRC, I had an interesting conversation yesterday where someone was trying to do some things in Bash and they were using Chat GPT to create their BashRC, it's fun to play with and try. Gave some interesting ideas. Forrest, you should try it.
Forrest Burt:
I think I have a terminal open in the GPT 4 at the moment just for the record.
Utilizing Apptainer in Different Environments [58:01]
Zane Hamilton:
Do some really involved syntax highlighting and spacing for Python. That was an interesting conversation. I still wish we could do this, where we could turn Dave's screen around again. That was still one of my favorite moments. That was so good. So Brian, in the university space, I know you've been spending quite a bit of time there lately. Have you seen them from a non-containerized environment? Have you seen them starting to adopt and really pick up and start looking for how can they start utilizing Apptainer in that environment?
Brian Phan:
Yeah, they have. So in the environment that I'm working with right now, they used to have a lot of software installed on an NFS share and we are currently working with them to containerize both some commercial applications and also some open source stuff as well.
Zane Hamilton:
How's that going with the commercial side of it? I know we've had several conversations around containerizing some of the commercial software and licensing issues. Are we starting to see as an industry that that's becoming something we can do a little easier, that vendors are getting a little more integrated into that type of environment?
Brian Phan:
I think it varies from vendor to vendor as of right now. And yeah that's how I see it.
Zane Hamilton:
Yeah, it's interesting whenever you start looking at places that do a per socket, per core, per host, it gets really interesting with containers. Yes, yes. Especially in something where it moves stuff around all the time. Is it going to work, Dave? Do we have any
other questions?
Dave Godlove:
Yeah, I'm getting there.
Zane Hamilton:
Alright.
Dave Godlove:
I could share again if that's useful.
Zane Hamilton:
Whatever you want to do. I could start asking Forrest other questions. It almost sounded like it was raining earlier, Forrest.
Forrest Burt:
It is here, it was hailing through my door earlier cause I need to fix the seal on it, but yes.
Zane Hamilton:
That is not cool.
Demo [1:00:24]
Dave Godlove:
All right, so yeah, so hopefully this will work this time. So I ran the same command that I ran previously, but I replaced it. Instead of Apptainer, I ran this DB dash app command and then all this nonsense scrolled by on my screen. But what it's doing is it's basically waiting now for me to connect so what I've done now is I've downloaded the Apptainer source code and I've got that in this repo and I've opened up this entire directory here. And within that directory you can see that I'm in this main entry point for Apptainer, the main package, this CLI dot go and I've set this break point here. And so what I can do now that I've actually, I'm here, is I can go ahead and press F five and that's going to drop me down into the code where I can actually interrogate and see what's going on.
Okay, so now I'm actually running here and so I can step in, I can step over, I can step out, I can do things like this. And so now I can actually start stepping through the code and see what happens. So the first function, which gets hit is this execute Apptainer function. And so I can step in and go through this, and I can see that this variable is set to ARGs and this variable ARGs is set to the following and these are the arguments that get input to Apptainer whenever I run it. And so you can step through the code using stuff like this and go through piece by piece and see what's going on.
I was going to demo it, but then I had this little hiccup and so I don't think I've got enough time right now, but I'll just go ahead and say a couple things. Number one, this is a little clunky to get set up, and there's a little bit of a hump to get over there. And so this isn't for everybody. Some people are going to be happier to just use Vim or something and just debug things with print statements and go through and look at things like that. But if this is your cup of tea, this can be a good way to get started with the code. But it's a little hard to set up.
There's a little bit of a hump to get over there, so unless you're going to really sit down for a little while and try to understand the code and devote some time to it, this is probably a little too much startup cost to get into. That's one thing. Another thing is, the way I just showed you, so this is the demo I was going to do, but I don't have time for now. But the way that I just showed you, starting at the beginning and then reading through the code, that'll work for smaller projects, but for Apptainer, that's probably not the way in which you want to familiarize yourself with the code I would think. I mean, there's a lot to Apptainer. If you just start at the beginning and try to work your way through, it's going to take you forever to try to figure out what's going on.
So I would suggest, say you're interested in the code that downloads containers from Docker hub for instance, but you don't really know where that code is, you don't really know how to get started. You don't really know how to look at it. Run Apptainer once with the dash dash debug option to give you a bunch of verbose output and find within that verbose output a message that indicates that the container's being downloaded from docker hub or whatever, as you're doing it, grep in the code for that message. And that'll give you the file that is important for that particular operation. And then you can go in and you can stick some breakpoints in that file, and you can run the same command again. You can pop right into that file and you can start looking all around and figuring out, okay, how do these variables get set?
What the stack as far as where am I at as far as what functions are being called? And from there, you can start to really interrogate that section of code and start to figure out how that works and if there's a bug in it, you can try to figure that out instead of going through the entire code. I feel like that's a really good way to get started as a developer and to just get your feet wet.
Zane Hamilton:
That's great. Thank you Dave. So we are actually up on time. Dave, I appreciate it. Thank you very much, Brian, Forrest, it's always great to see you. Thank you for joining the conversation. Join us again next week for a round table and then we will see what is coming the week after that. But thank you for the conversation. Thank you for watching. We will see you next time.
Forrest Burt:
Bye.