Power to the Users! Unprivileged Operations in Apptainer

Godlo. Yeah, I'm Dave Godlov. Um, I am a solutions architect here at CIQ or at CIQ. CIQ is not here, but I am a solutions architect at CIQ. Um, and I've got a a long history with Singularity Appainer as well. And I'm also really happy to be uh demoing some of the features that um primarily uh Dr. Dave next to me uh has worked really hard and championed and um developed. Thank you. So I'm going to start off with you uh Mr. Godlov. I go ahead and go and go back for those of you who are maybe new coming into Singularity Appainer, but let's talk about how Appainer's really been giving users more power to do things over the the past seven years.

Cool. Yeah, this is a topic that I really really like talking about because it's something that I've been personally involved in quite a bit and it's it's um really kind of the legacy of Singularity and Appainer uh as a project. So I I would say that um Appainer really was started with the idea of giving more power to users. Um if you think about just the the uh technology itself, the technology allows um a user on a on a high performance computing system basically to install anything they want. Thank you, Rose. Basically to install anything they want uh on an HPC system. Basically to install it in a container and then grab it and pull it over to the HPC system and run anything they want.

Um, and this is this is great because, you know, not only can they just install anything they want, I mean, I guess maybe they would have been able to do so anyway, but it allows you to install things using tools like yum and dnf and aptgit and just, you know, package managers. A lot of times when there's a new scientific software that you want to run, um, it'll be hosted on GitHub or something.

You'll go to GitHub and there will be some instructions for installing it and the first instructions will be install all these dependencies and it'll have a list of um aptgetit or yum commands assuming that you're on some you know particular distribution and then it'll say and then you know I don't know pip install after you install all these dependencies the software or something like that and you'll be like oh well that's great but I can't do any of that because I'm not in a system administrator in my HPC environment and so you know it it used to be that you would have to contact then

a system administrator and say do we have any of these dependencies how c can I talk to you about maybe trying to install them in a non-standard location and you know now you just say oh I'll just spin up a container and it doesn't matter that we're running re on the cluster I'll just spin up an iuntu container and do all this apt get stuff that you want me to do install all these dependencies you know pull in the program drag it over to the system and boom I'm done so it's all about you giving users power and you know handing them the ability to do stuff.

Um, and you know, I'll I'll talk a little bit more too. I'll try to, you know, I know I'm excited about this, so I'm going to talk a bunch, but I'll try not I'll try to cut it short. But I I will say that there's been lots of times in Appainers development too where the developers have gotten together and said, "We could let users do this, but we're not sure if we want to because users might end up shooting themselves in the foot." And those discussions almost invariably have come down to Let's go ahead and give the users the power to do whatever feature it is that we're thinking about letting them do.

Let's go ahead and trust the users and let's go ahead and, you know, give them the ability to do whatever. And this extends to the ability, I'm going to call it, to be weird, to do things that are not necessarily best practices or what we expect or how we think users are going to use the the uh the uh software, but actually to get into the internals, into the guts of the container and change things in there and to do things that um you know to allow their containers to work even if they're not things that we necessarily expected users to do. So this is this is really what Appainer's always been about is giving more power to users, I think.

It's great. I know Forest, Brian, you guys have come from outside environments where maybe you were using other containerized platforms. How has this given you guys a little bit more power or the ability to do things that you couldn't do with the other systems or other platforms I should say? Boris, let's start with you. You came off mute first. You win. Um, as far as containers go, uh, my background has primarily been mostly with the singularity obtainer project. Um when I was working at my previous position as a system administrator in the academic sphere of HBC um I needed a solution that allowed me to install software in kind of a more flexible way and in kind of a more standardized way.

Um and so I turned to kind of looking into containers as a solution for that. Um and I found out that some of the other containerization solutions out there were unsuitable for HPC. And so Singularity at the point at which I started using it was basically the deacto standard. Um so while I've kind of tinkered with other solutions around um my background is primarily with singularity obtainer um because I've mostly been an HPC um dealing with HPC focused containerization problems. Thank you for Brian. Uh so obtainer has um enabled me to deliver um software sol in a previous role um it enabled me to deliver software solutions very quickly to um a customer.

So, we had this inbound customer who needed to get a workflow running urgently and uh usually we would have to wait for the entire like release cycle before we deploy our updates. But since they needed it really urgently um I was able to use Appainer to quickly whip up a container for them and they got their machine learning workflow running uh the next day and uh they they met their deadline and they were super happy with um the solution I was able to deliver them. Great. Thank you. Dr. Dave, I know you've been I mean for me now you've been around labs for a while and you've seen a lot of different things take place over time and kind of moving towards that containerization.

What has this allowed you to do? I'm the uh um support person for the high energy physics community and uh we run we do some HBC but we also do what we call high throughput computing where we run on instead of big computers just lots and lots of them hundreds of thousands of computers all over and um this uh we uh our community got involved in using Singularity early on because we wanted to be able to We start uh we we start jobs at at all the all the sites first from from the from the uh the experiment that's running. They they first submit a job in the and uh just to allocate a space first and then it goes and runs the user's computer.

the users um code. And so we wanted to be able to um to start the uh start as one unprivileged user start a container from another unprivileged user and keep them isolated from each other. They could even on the same machine have be running from different users. So we were very interested in the isolation features and and we were very much interested in unprivileged and that's we we've been using unprivileged for for quite a few years because our code uh distribution is not through CIF files. So we have our we have uh the CERNVM file system distribute already unpacked and then and we could use the unprivileged features uh earlier they as they were available in singularity and didn't need the the un the mounting of unprivileged SIF files.

So that's that's kind of our back my background. That's great. Thank you Dave. So I we talk about unprivileged users. I know that's something we really want to talk about, but I think there's been a lot that has come out in the last the last at least two releases that have really been driving towards what's going on now with within the Appainer community. But let's talk about what what that means and where we're adding features and what those are. And I'll leave it open to anybody wants to take it. I know both Davees probably have a lot to say on this topic. So, Mr. Godlo, I'll let you go first.

Well, yeah, maybe it'll be good to um recapitulate some of the history um and some of the some of the need for privilege and so on. Um just to kind of level set a little bit uh to answer this question and to get everybody. So um you know for for for a very long time now uh appainer has required some elevated privilege to run and um because of that so if you're not I don't know you know we might have viewers who are not like um you know super uh up to speed with everything Linux based. So the way in which that works is that uh your an administrator will will install uh appainer um and that will have what's called a set UID bit installed with it.

It used to be I think uh you know correct me if I'm wrong but I think like a really long time ago the entire executable had a set UID bit. What that means is it sets the user identification to somebody else. And so when you have a binary which is owned by somebody else um you can run as that user even if you're not that user and that be that becomes handy because you can own the binary by root and then you can allow an unprivileged user to run it and they run as though they were root. Well I think a very very long time ago I think I could be wrong but I think that the entire binary was set UID.

Uh pretty quickly in into that development we changed that. So there was one binary which was set UID and then um Appainer would go through and elevate privileges only for things that needed to be elevated during the the startup of the container. And so you know what do you need elevated privileges for? Well, for instance, you need to mount um you need to mount a file system to your file system. And that's that's usually a a privileged operation, right? So you can't really just do that on your own and we can't just give everybody who we want to use obtainer root privileges. So we allow in that one specific circumstance very you know carefully we elevate privileges just long enough to mount the file system then we drop privileges.

Okay. So set UID bits um bit binaries are notoriously difficult to write properly. um we've always had very very smart people working on the Singularity Appainer um project and we've had very security-minded people working on it and so because of that uh we've been very careful and we've had a you know a pretty good track record as far as uh you know making sure not a perfect track record but a pretty good track record as far as making sure that we you know don't have vulnerabilities in this system but the fact remains it's a really difficult system to administer so along come user name spaces in the Linux kernel several years ago.

Um, Linux namespace, usern name spaces within Linux have actually been around for a while. Uh, what the user namespace allows you to do is the kernel allows you to enter a new um, kind of walled off area, a new namespace, and it maps your UID on the host system to a different UID inside that namespace. So, we can use that to map whatever UID you are to zero inside the user namespace. and that gives you the ability to have privilege inside that namespace um even though uh you're you're not privileged on the host system. So several years ago um we started to play around with you know using this to give you the ability to be root inside the container and then the username space started to become adopted more and more.

it started to become more standard in Linux kernels to the point where now it's it's it comes standard and it's on by default and so now it's quite easy to use that to be root inside the container. Um this also allows uh you to be able to build things inside the container that would normally require root privileges. And then I think the um kind of the clincher that now allows us to be able to do everything that we want to do even to be able to mount um and then you know Dave you can jump in and kind of correct me if I get this

if I get this wrong but there's now uh new technology using fuse which is an unprivileged uh implementation of it's kind of like a general um implementation of lots of different file systems but using fuse there's another um there's Squashfuse now file system which allows an unprivileged user to take a Squash FS file system and mount that onto your file system. And that's cool because guess what uh Appainer uh containers are saved as Squash FS files. So now we can leverage that technology to mount the the uh file system to your file system unprivileged and that pretty much removes for for like 95% of the um the operations that you would normally want to carry out with Appainer that pretty much removes any need to have elevated privileges.

And so that's you know where we came from to where we're at today. Yeah, I'll add that that's that's the most important that's the most important thing. But there was also at the same time we added the ability to to mount uh fuse 2FS ext3 ext34 file systems which is used for overlays. Sometimes we use those for for overlays when we want to make it appear writable. And then also um fuse overlay FS when we when we want to have different layers it will use that um and um that is uh uh or or it can or it will use there's even the newer kernels uh you can actually run the kernel overlay FS unprivileged inside a user name space.

So that that's an even more recent feature than than the mounting of fuse. So, uh, those things and then of course also the unprivileged build features with fake root that were added in in this release which you you were probably going to talk some more about. Yeah, I was just about to ask that David. I mean, if you want to continue talking about that and tell tell us why that is important. How does that help a user an end user? Okay. Yeah. So, so Dave did mention that we could use uh do some building even even before the recent obtainer releases. It was possible to do building I'll first back up a little bit.

Um so so when you're building a new container from um from scratch using like a definition file um you're doing these you know privileged operations of aptget or or yum and so it requires it to appear privileged inside and it not only that it it uh it really normally wants to have multiple user ids there's root you files but it also has other things like bin or other other per package user ID. So it wants to have multiple user ids and the uh unprivileged user name spaces by itself.

the totally unprivileged really only gives you one user ID you're allowed to map and um so but we have had for for quite some time this d- fakeroot option in um in singularity and obtainer uh and that would allow you to do a build as fake root but that but it's not still it still requires system administrator intervention because uh it requires setting up this uh mapping it's not in a totally unprivileged operation it requires is setting up system administrator to set up something called SLTC subuid and subg to say which user ids you allowed to map into in your your namespace and that actually uses a a privilege escalation program to help with you the new uh new UID map and new GD map and that's what podman and other container rootless container systems are are using.

So, Singularity had support for that. You had to add the option when you did the build to say D-fake root and you also had to make sure the system administrator set it up. So, what's new in obtainer 1.1 is that if those are not set up, if those are set up, it'll just use it by default. You don't even have to say d- fake root when you say do a build. But if it's not set up, it instead runs a fake root command. This is different from the fake root option. as a fake root command which makes it appear as if you have multiple user IDs but you really don't.

It it traps the system calls. Um, and so if something uh tries to do a ch own to another the user ID, it uh it says, "Okay, I changed it even though it really didn't on the host." And then it also keeps track in in a in memory how which files it has changed. If you go later ask and say, "Is this file owned by who's this file owned by?" It remembers that other user ID. So through that, we're able to do builds as well, even without any system administrator setup at all. And so this this can be very useful especially on HPC systems. People need to want to use the the locally optimized for that HPC system to be able to access to to be able to compile right on that system is is a really big help.

That's great. Thank you, Dave. And then Mr. Godov, I know earlier in the week we were on a call and you started talking about the ability to install Appainer without having to have root and that that was news to me. So I think that's very cool. Why don't you tell me a little bit about that? Yeah, and that is that is one of the new things that um that Dr. Dave over here over here uh has has introduced. Um so yeah, I'd actually just like if it's okay with everybody, I'd like to just show that to you. Um absolutely. Okay. How do I No mute settings.

I'd like to share my screen. Okay. And I'm going to share It helps if I speak while I do it. Um, window conscious train of thought. Yeah. Sorry, I'm not seeing the window that I want to share. Should have tried to do this before. All right, let me um It's live demo. It's what happens. Yeah. All right, let's see here. Um, maybe we could talk a little bit about something else while I try to figure out why that's not working. All right, I'll I I'll talk a little bit about what the script does. So, what the script does. So, so uh it actually installs the program itself uh without having to compile it from source.

You could always compile it from source, but this takes existing binaries that a system administrator would install, but instead of requiring a system administrator to install it and use like uh yum to install it, it just uh reads the packages from the servers where they come from and unpacks all the files into your own local directory wherever you want them. And also uh obtainer has been updated to be able to run that way to be able to uh detect where it's running from and then and re relocate itself that way wherever it is. As long as it's not in it's not a set UID installation, it will uh uh then it's not allowed because that's that that can be dangerous um that set UID installation.

If the set UID uh starter isn't there, then it will uh just use them. So Dave, is this kind of like Python virtual environments where you at some point as Appainer moves on, you could actually have a user have a specific version they want to be running and have multiple users on a system using different versions installed kind of locally for them. They certainly can do that. I'm not sure exactly how Python virtual environments work, but yes, they can you can install whatever many however many different versions you want in your own directory there. Something else is being shared. Um, not seeing anything being shared yet, Dave.

There we go. Or something coming up floating. Uh, it looks like it's just a gray screen. So, I was asking uh yeah, we're having a little technical difficulty right now. Um might be because I have got multiple uh multiple screens running. So, let me uh let me see if I can fix that. Sure. So, Brian and Forest, have you guys had a chance to play with this yet from an unprivileged perspective? Um, I haven't had a chance to play directly with the features that Dave is going to demo, but I've definitely been enjoying just being able to type obtainer build, um, you know, star.sip, star.deaf, and having it, uh, you know, build without fake root or anything like that.

I'm definitely not taking advantage of, uh, that type of capability. I have not uh been able to play with these features yet, but I am really looking forward to uh Dave's demo. If you can get it to work, um if not, we're going to have him read it line by line as he goes. Maybe I'm really sorry, guys. I uh I have the options either to share windows and it's not showing me the window that I want to share or I have the options to share screens and it's just showing me black screens. So um having a hard time trying to figure this out. I apologize.

Let me uh continue to work on it. Drop out and come back to you and do that too. So I mean while we're waiting Dr. Dave, what other things are you now currently working on in Appainer for like what's what's coming? Yeah, the main the main thing that we're still trying to do um is um to also support uh encrypted file systems unprivileged. That's the one the one significant uh feature that still requires set UID route. We didn't get around to that to that yet. So, we're trying to work with a uh existing fuse file system that does encryption and uh uh we're getting another someone else else to work on that.

I'm just just uh consulting with them on that. So, that's the other other main main thing that's that's still still missing without uh set UID. Uh yeah, with the current release of unprivileged. Very nice. So I guess the other question I got for you Dave since we have some time appainer now being in the Linux foundation have you found more people to help to test to write code is how has that been going? Well that that's that's still been an issue that we that we need could could certainly use some more some more development help. Um uh so the Linux Foundation doesn't doesn't really bring development resources.

They bring they bring legal help really to to uh to help you uh do things and and advice on and and keeping things open and that type of thing. But um yeah, could could use some more help. I believe some we do have some more people now recently been starting to to step up to to uh to do some more volunteer for for some more pieces. So that's good. Oh, that's great. And I mean I guess if we could go ahead and post kind of the links to those communities along with this so we can kind of advertise and start asking for some help. Are there specific areas that you need help in?

I mean is testing is always going to be one that seems to be a place where people could contribute. But what else do you need? Yeah. I mean it's it's really you know what people whatever whatever features that people find important I think that they would be interested in is there certainly uh you know this is this was the unprivileged was my big thing and that's why I I did a big push. I've spent, you know, dedic paid by my employer because I felt that this was a really important feature. And so, um, I don't expect to be being quite, you know, quite so active anymore because I do have like four other jobs I'm also trying to do.

So, of course, yeah. So, so whatever whatever people feel is uh important uh for their for their use cases that please uh please jump in and help. I think kind of going back and looking at the Oh, somebody's got a question. Good. Is there anything that does not work with an unprivileged installation? Yeah. And that's that's the um uh the the encrypt and decrypting of of the of the file systems. That that's that's the one that ROS does require. Um there there are a few other minor things that things that were uh hardly ever used. uh things like uh uh going with uh different uh user ID inside.

I think maybe even that might be might be available soon. Um um I forgot how that ended up but but but mostly it's it's the uh decrypting of file systems on the on the fly. Thank you. Thank you. Um, there is it it is a slight difference in the uh if you if you start up a SIF file and you look in slash LS-L SL you'll see a different um set of user ids.

So that that's a different so so when we run with when obtainer runs with sewid mode set UID mode then you see all the user ids get mapped they get mapped straight through um so you would see something owned by root whereas actually when you're running in unprivileged mode it it's owned by you and the files that were owned by root look like they're owned by you inside the and a lot of other files are just owned by nobody because you have two two IDs there's you and there's nobody that's what you get with the unprivileged username spaces. Um uh unless you're using fake root. If you're using fake fake root then then you get them all again.

But um so that's a difference but in general it doesn't u doesn't make much difference. Sometimes programs are very picky and uh then it does make a difference but uh uh so that that's that's another thing but it's so in general no one you hardly ever notice that. Thank you. Any better luck? How's it going? D, not great. Um, I'm going to try to move everything to my first screen and then just share the entire screen and see if I can do that. Okay. One of the other questions that I have that maybe people are are wondering about. If you're used to some of the other container platforms, I know we've talked about Podman being able to run rootless to some degree.

Why is that important for especially for systems administrators and really for HPC? Why is it important to not have to have root other than you don't have to go get somebody to install something, but why is it why else is that an important thing, right? So, um the uh basically we're talking about security security considerations here. I noticed that was in the in the uh in in the in the abstract about today's talk. we haven't got to it yet. Um, so I'll I'll talk about that while while Dave is busy. Uh, so yeah, so as as Dave said, well, he did already talk about saying that, you know, set UAD root is notoriously difficult to to secure.

Um, so we tried to do a good job of it. There is in fact one known um there's one known uh serious uh risk that's that's always been in the case with with uh with singularity set UID obtainer set UUID mode um and that is that um uh allowing the user right access to the container so the this is which is basically a file system, right? It's a file system inside it. Um, this gets mounted directly in the kernel uh with and um the kernel developers have said this is a risk because uh allowing users write access to the underlying bits while the kernel is interpreting it is a risk that that uh the kernel is not designed to uh to protect against that case.

So let's say the user is twiddling with those bits and then the mean right in the middle at the same time the the kernel is is interpreting them and the kernel they said the colonel file system drivers are not designed to to support that and so um so it's always been a risk. There's no there's no one has priv has has uh published a known attack using this, but it's always been a risk that we've lived with and because it's been so convenient to use and um so uh the uh so when when you run with unprivileged and and the file systems are not running in the kernel at all.

It's a fuse file system. So this is in fact this is the reason why the kernel uh um owners have only allowed fuse file systems. They don't want they won't inside this unprivileged user namespace because they're worried that any other any other uh file system that you allow to be mounted inside this unprivileged user namespace where the user is running as a pseudo root. um that that uh while they have you know to allow them to mount other file systems while they have access to the to the underlying bits is is too much of a risk. They don't allow it. That's why they only allow a fuse.

So uh so that's that's the the risk that we've been known risk that we've been living with all these years with singularity and and obtainer set UAD. So on the other hand there is tradeoffs because um when you are using unprivileged user name spaces um that exposes the system administrator to anytime the kernel comes out with a CVE that might of something that might be exploited through unprivileges or namespaces that means they need to upgrade their kernel and boot reboot and that can often be a big pain for uh system administrators. Um there can also be uh you know you say oh it's it's a risk that maybe some of these things are running undiscovered maybe maybe uh uh people with a lot of resources have discovered these other holes and and they haven't given it out to the kernel developers.

Um uh so there's that uh disadvantage. Now it it what we have put in our in our advice is that um turns out that almost almost every CVE for the last couple years related to being exploitable with with unprivileged namespaces have been in combination with network namespaces because it exposes I guess a whole big another big piece of the kernel code the network handling and um so uh we recommend commend turning off network namespaces if you're if you can if you can live with it.

So that means if if you don't mind sharing the host network with all the containers which is what obtainer does by default uh it's not what docker and podman do by default they by default give you a separate network namespace uh you can give an option to it to say use the host networking um it's also uh there are also some systemd few systemd services just a couple in EL7 and and more in EL8 um do use network namespaces by default, but we give instructions on how to turn those off uh so that these services aren't using a network namespace if you if you want to do that.

But that is that is the way that we in the high energy physics community, we've been for a couple of years have been running singularity unprivileged using not using CIF files. that we didn't need squash fuse and we've been telling people you know enable your own unprivileges. Well, uh Dave mentioned that it's it's it's enabled by default but that's also in EL8 and later and also in I think most of Debian it is enabled by default but it's not in EL7 yet. So we say those on EL7 enable your user name spaces and everybody disable network namespaces if you can and that way you're going to be exposed to a lot fewer risks in the kernel.

So, thank you for that, Dave. And we do have some questions coming in actually. So, if we can throw up Le's question first. I see. Back in November, I tried to reallocate and unpacked Appainer RPM for some operations. It still looks for a hard-coded central path. 114 changed that. So, if you were, and that was I think since November. Uh, no. Uh, I think I think uh you can look it up, but uh you probably weren't yet using 114. So at the same time we enabled the un unp install unprivileged.sh script it had more uh relocatability. If there's if it's still any hard-coded paths please make a make a GitHub issue and we'll look at it.

Great think of that. Now from Peter is there a problem using a shared file systems like luster with rootless podman. I guess there is an issue with with podman. It's the same thing true in appainer. Ah, well, not with the SIF files. See, that's the difference. Um, if you try to use sandbox mode on Luster, you'll have the same issue. And and Podman basically is only it's basically only uh sandbox mode. Um, now they do actually have um the um Podman has a feature, their latest I think of 4.0 Podman, they added a feature. Well, they can read a SIF file, but all it does is unpack into a sandbox before it starts.

So, it's it's really not helping you. Uh you still have to I guess if you unpack it into another file system, not Luster. And then there's also the Luster that's uh the the other issue with with having unpacked files on Luster and any worked file system is that it becomes a big uh performance bottleneck. Um so like I said we didn't have that problem in the high throughput computing high energy physics world because we used another file system this certain VM file system which also um did every all the all the metadata operations are done on the client node and the same way but the

problem with when you use luster or network file system like that all the metadata operations have to go to a metadata server and when you start up your jobs on hundreds thousands of nodes all at once and and everyone is reading thousands of files, your metadata server gets really bogged down if not if not killed outright. So, so uh especially in a pythonbased container which has tens of thousands of files I think um u so uh that's a uh another issue with that. So yeah, so as long as you're using a sif file on luster it can work. Um does that answer your question? Um abs I think so.

Thank you Dave. Yeah and u in fact we've done uh did some did some performance. Let me say a little story. Um uh that the uh when we released one or just before actually just like just we were about to release 1.1.0 and somebody finally gave us a benchmark that we could run with Squashfuse and it turned out to be super super slow. we this benchmark and it's a Python-based benchmark and it was running on 64 cores and squash fuse was single threaded and so that kind of killed the performance. Uh but very fortunately there was already someone had written a multi-threaded squash fuse patch and so uh we turned out uh when we applied that on this benchmark it turned out to be almost identical to the kernel uh squash FS.

In fact uh measurements showed it to be slightly faster although I don't I didn't I thought it was within the margin of error. So uh so that so but that's really uh still uh we didn't in that particular test we didn't compare it against unpacked sandbox on on on luster but other other people have done lots of measurements in the past and showing that's why that's why in fact that's I think the whole reason why um why singularity took off in popularity became the deacto standard in the first place is because system administrators found out that it actually performed better than raw files and raw, you know, software on on Lester.

It's because the the packed files the uh single file system that gets mounted onto the client node moves all those metadata operations over and and so it was it was actually performing better. So there's been a lot of studies about that in the past. Yeah, that's fantastic. So I see we lost Mr. Godlo again. Oh, he's back. Maybe I should I'm back. I'm trying, guys. I'm trying. The only thing it's letting me share now is tabs. All right. So, um Jonathan, uh you know, uh Jonathan had a good idea. He suggested why don't I just point the webcam at the screen. So, I love pixelated visuals here.

Let's see what happens. I don't know if this is going to it's going to work or not, but let me um let me mute and see if I can get a a something that'll sort of work. I could also try to share if you want and then it just breaks everything. It's just not dayto day and you're on mute. This is like traveling with kids, Dave. You never know how it's going to go. Could be great, could be a mess. Well, when it's me, it's pretty much guaranteed to be a mess, right? Um, where's my There we go. Hey, at least you can see Dave's setup.

Can we make Can we make Dave's screen bigger if we all go backstage except for Dave? Yeah, we drop all of us. We make Dave big. That's good. Good call. Wow. All right. I don't know if we want to try to. Can you guys see anything here? There we go. I don't know if you can still hear us, Dave, but go for it. Yeah, I can hear you. Oh my gosh. Nah, you're not gonna We can We can see that. That's awesome. All right. Well, uh, if I do, okay, I'm just going to show you really quick the unprivileged installation if I can. If I do pseudo, who am I?

Uh, you can see that I'm in trouble. This incident has been reported because I am not a sudoer. Okay. Um, so but I do have get. Okay. So, what I can do is I can I'm gonna CD here to temp and I'm going to do um get clone https github.com appainer appainer I'm just going to copy the uh the appainer um the appainer git repo uh you know directly in my computer here. And now if I CD into Appainer and I CD inside of Appainer to this directory called tools. I don't know if you can see that there but if I CD there um and then I print what I got I had there's this uh script written by Dr.

Dave called install unpriv unprivileged.s SH, which is really cool. So, I'm gonna go ahead and um get help off of it. I don't know if you're gonna be able to see anything. Um yeah, let me So, the help is that you run script and then you can change the distribution. You can change the AR arch. Um this is for the uh version of Appainer that you're going to install. um you can change the version of Appainer and then you give it an install path. So just really quick, what I'm going to do here is I'm going to say um I'm going to make a directory in which to install this.

Uh it's my home directory. I'm going to say at Tainer and then 1.1.5, which is the version I'm going to be installing. You can you can change this to whatever you want, but that's what I'm going to do here. Okay. So, I've created that directory and then I'm going to call that script. I'm going to say I want to install version 1.1.5 and I'm going to give it that location that I just created as the place to install it. Now, what's going on here is actually really cool. Um, because what it's doing is it's downloading RPM packages and it's extracting those RPM packages. So, normally when you're thinking about like RPMs, you're probably thinking about installing things using something like DNF or Yum, which are obviously privilege operations, which you can't do unprivileged.

Uh, what Dr. Dave has done and he can correct me if I get this incorrect um is that I I think that he has basically said I'm going to go ahead and just download those RPMs from where they are up on um you know the their their repositories extract them and manually install them in the directory of your choice as a non-privileged user which is really cool because you're still using RPMs right uh I I I it's funny that you think this is funny name um so anyhow It's uh it's going ahead and it's it's extracting these things. Now, I don't know why, but I've noticed that sometimes this takes a little bit longer and sometimes it goes a little bit shorter.

Um, this demo is really completely gone belly up today. Um, so I wouldn't expect this to to actually complete in a reasonable amount of time. It's probably going to take forever, which, you know, is part for the course today. I haven't had a demo go this badly in a really long time, so I'm overdue. Um, okay. So, it's going to sit there and take it time. Usually this just takes a minute or two. Uh but um you know this time maybe it's going to take a little bit longer. So maybe if you guys want um we could we could talk about a few other things while this is uh completing.

We do have another question Dave from David. So Dave, so I thought you I heard you say this code was coded in Python. If he heard that right, will it run on a Raspberry Pi instance? No, I I you can hear me, right? Um um it it's not in Python. I I talked about a container being made having Python that's actually in in uh in Go. Uh the the most the main program is Go. This is a bash script that is that that's hanging right now. I would suggest you just kill that and restart Dave. Maybe it maybe there's a problem with that server. Um but the u uh Yeah.

Right. So it's written in Go. It should certainly run in the Raspberry Pi. I don't know uh anybody who did. Now the install um yeah actually if if you are even this install on privilege might work there. Uh if depending on what operating system you're running on it's um what see notice that the um uh you're on what what what operating system are you on there Dave or is it a Fedora? You're muted. Yes. I've lost my mouse. I've lost my mouse now. So, it takes me a few seconds to to um unmute. Yeah, like this. I think my house is going to catch fire next.

That's going to be like No, the only thing that can can uh get worse. But um No, no, I'm actually uh this is Rocky 9 that I'm running. It is on a Rocky 9 system. Okay. So, it's it it read it was reading on Rocky 9. But the script actually uh is set up to install it's the uh from EL78 or 9 onto any actually Fedora will also do Fedora will take a native ones but if even if if you're on Ubuntu or Debian it's still going to take the corresponding uh Red Hat or Eel or or uh or Fedora. Actually it only does Red Hat Eel.

Um, and it then it it unpacks the um unpacks it there. So I see David, you said in your note that you're running Rocky on PI. So certainly it should it should just work. Um, and it it and and it will be a ARM architecture, but that's supported there whatever architectures are supported in the um by the upstream provider. So yeah, just try it. In fact, uh Dave showed you doing a get clone of the whole um uh of the whole thing. You don't need that. All you need is that one script install obtainer.sh uh which the uh instructions and the uh uh obtainer.org um documentation obtainer.org docs um and the uh administrator and the administrator guide the install um it shows you the full path.

You just run curl and download that one one file and pipe it into into bash and and it will uh that's all it takes unless there's a glitch in your networking. Right, Dave? Right. So now we've got Appainer uh running. So um I'm I'm moving a bunch of cords and scroll back a little bit. We missed it. Can you scroll back a little bit? What did it do? You just started it again. Did it eventually finish? I Yes, it did. Okay, you didn't kill it. Did you guys see this? Yep. Cool. Okay, so I exported the path Appainer uh you know 1.1.5 bin path to my to my path and then I I went ahead and just ran Appainer like so.

And now we've got obtainer running. Um in this demo today, I was going to go over building containers and a bunch of other stuff. I don't know how far I'll get. So, I've been moving everything around on my desk and of course I've been pulling on cords and doing this and that. So, I've lost now my mouse and my um keyboard and I have to do everything on my laptop. Um so that's that's an additional problem here that's going on. But uh that's cool. I like the extended reach typing. It's making it better. Yeah. Yeah. So we can roll with it.

I mean whatever we'll do the the other thing that this install script does is install all the dependencies and that's that's kind of been the motivation for the whole script is because there are now in 1.1 there's squash fuse you need there's there's fuse overlay fs there's fake root all these extra commands which if you're not the system administrator and they're not installed they're they're they're unhandy to get so so this uh script gets all those dependencies as well from from NRPMs and installs all of those. Okay. And I'm going to run the uh well-known hello world of Appainer if I can. So, this is just going to prove to you that not only can you run help, but you can actually do stuff.

Um, I was gonna say useful stuff, but of course this uh con particular container isn't useful at all except for just verifying that you've got a working working um installation of Appainer. Who built this container, Dave? Some jerk. I don't know. It's been around for a while, hasn't it? It's been around for a long time. Awesome. Okay, so uh yeah, and this is built by by me. So I I'm not just calling some random person a jerk. I'm calling myself a jerk. Um, okay. So, uh, yeah. So, it works. So, unprivileged installation works. Um, if you guys I'm gonna try my best to write a quick little definition file.

So, I don't know if you guys want to riff a little bit more while I try to do that, but um yeah, I'm going to try to write a definition file and just show you that you can actually build unprivileged uh with an unprivileged installation of Appchainer, which is awesome because this basically means that you know at this point in time, even if you you happen to be on some cluster where the administrators for whatever reason, you know, they haven't heard of Appainer or they're suspicious of it or they just, you know, don't want to mess around with containers, they haven't installed it, whatever, you're still not out of luck.

You can install Appanner yourself in your own in your own space on a cluster where you don't have administrative access and actually even build your own containers um on that place where you don't have administrative access. So this is really really awesome as long as you have like a computer that works which is kind of you know and as long as it has unprivileged user name spaces enabled. Yes. And that's an important that's an important thing. So thanks for bringing that update. So, so if if you're an administrator and you really really do not want Appainer running, don't fret. You can still turn off unprivileged namespaces and you can still, you know, uh lock things down appropriately.

But um yeah, so while Dave does this, if you guys have any more questions, go ahead and shoot them this way. But I'm going to ask I'll go back to Forest and Brian because they're being quiet again. Where could you guys see this being useful in things that you're working on right now? Start with Brian. in this time. Um so uh so this is useful in the sense that it helps me prototype um I guess uh software solutions pretty easily. So in my current work um I'm currently working with uh open foam and uh currently trying to get that working with uh EFA uh which is like a AWS product and uh basically uh this has enabled me to quickly rebuild open foam so that it's compatible with that instance type and uh I basically can get it easily running um on that.

Uh yeah, pass it on to Forest. Um I basically just take this back to the user perspective. This is going to massively benefit high performance computing users across the sphere of high performance computing that are interacting with uh app tuner and using it in their workflows. Um this really you know just to bring it back to the theme of our our webinar today. Um power to the users. This gives them a huge amount more flexibility in how they're able to deploy and manage obtainer within their own research their own uh you know industry tasks they're working on. Um so this is uh just massively interesting from a user perspective.

So, as someone who used to, you know, do a lot of direct user support type stuff in a CIS admin like role, um, I can see where this just gradually lessens the amount of work that I would have to do to manage this for users and just gives them ultimately u more of the capabilities that they were always looking for around reproducibility and kind of um the ability to integrate these within their workflows.

I want to say that that actually what motivated me to write this script the most was uh I in I I distribute a version of uh or an installation of obtainer in the CERNVM file system which is accessible then by anybody who's running CVMFS and uh and by the way I should put in a plug for there's something called cbmfs exec which is another one of my tools you can google for it you can run that as an unprivileged user you use CVMFS as an unprivileged user and and it also uses unprivileged user name spaces and you can run it's a CFS is a fuse file system but you can run it all unprivileged.

So with there's a command you can run the CFS exec and then from there run obtainer because it's all all available uh in inside the CFS namespace and um but I wanted to be able to support multiple architectures because I was starting to get requests for this on on Power PC and so I wanted to be able to okay well I don't happen to have a Power PC machine to compile on but oh but those things have already been built for for Apple. So all I need to do is grab those binaries and install it. So the this script actually you can install multiple architectures at the same path, the same directory and it will it will it will run whichever ones you've installed based on whichever one uh host your host architecture is when you run it.

That's very cool. Thank you. You ready, Dr. Godlo? Yeah, as ready as I'll ever be. So, I built I I created a very simple little dev file and uh you know, I'm bound to have made some mistakes in it because of the you know, I'm trying to just recapitulate it from memory and um you know, I'm typing behind my back. But, um I'm I'm going to go ahead and and press enter and see what happens here. Uh if I can highlight the screen again. Here we go. All right. So, make Dave big again. Sorry, Dave. There we go. Okay. So, it's it's running kind of slow.

So, in addition to all the other stuff that I'm having, I'm probably having some issues with my network, but it's it's doing a DNF update and it's doing a DNF install of a bunch of stuff right now. And it's actually installing. So, what this tells you is that who I am root inside the container. So, um, you know, once again, just to bear in mind, I have no elevated privileges here, right? But I've been able to install Appainer, run a container, and build a container with a bunch of DNF commands inside of it. Um, all as an unprivileged user on this system, which is really, really groovy.

Yeah. And I'm getting a question to show the dev file. So, let me oops. Oh, Vim is not installed. Um, and I can't install it because I'm not privileged, but VI is installed. So, never fear. It's not like anything will go wrong. There we go. Very simple. But I've installed event. So, I have I've got Vim actually inside the container. I was going to play around with that and show that uh I didn't have Vim on the host system, but I had it in the container, but you get the idea. Installed Python 3, Python 3 pip. And then I installed um a little program, a Python program that I like to play around with called Askinma, like ASKI cinema.

I a lot of people are probably familiar with it by now. It's been around for a while, but basically allows you to record asy text sessions in your terminal as though they were movies and then play them back as text, which is cool because you can put those things up on websites and things and then you can like pause them and just copy the commands out of them and run them at your terminal. But I was going to show that a little bit. We don't have any time right now. You can check it out on your own. Um, but yeah, and then if you wanted to run like fake root inside the container, um, that works too.

I don't know if you guys want me to try that real quick or if we're over time already. Yeah, one more thing, David. Throw it. Try one more. All right. Real. This is And this is basically I mean I was gonna go a lot further in this demo and show a lot more stuff, but this is basically the meat of it. The three things I wanted to show. You can install it. You can build containers. And then let me um jump in here as fake root really quick and just kind of show you. So [Music] there's the command. I'm moving my hand so you can see it.

And then I'm going to jump into testif. Okay. Um I'm in the container as root. If I do who am I, it shows me as root. My ID is zero. If I created files, they'd have a UID zero here. Um, you know, so now I can do like things like DNF update. I can change things in user local bin or user bin or whatever. I can do except it's read only though. You have to run it with with a with an overlay if you want to make it writable. Yeah. Right. Right. In the longer demo, I was going to what I was going to do is I was just going to dump this out into a sandbox and then jump into it, install some more stuff using DNF, but I mean we're over time.

I think most of the users hopefully or most of the viewers actually, you know, get the idea at this point. So, sorry that was Rocky uh Rocky sorry that was a a bit of a rocky demo. Um a different different type of Rocky demo. Um but you know, hopefully we were still able to do all the things that we wanted to do here and um you know, can still kind of we still got it done. Absolutely. And if we need to, we can come back and try it again later. Yeah, we are up on time, guys. Dave, I really appreciate it. Thanks for fighting through and reorganizing your office.

Now, I know you can spend the rest of your afternoon trying to get it all put back together. Dr. Dave, it's always good to see you. Thank you very much for being here and for all the information you provide and all the work that you do. We really appreciate it. Uh, thanks to the community for being there as well, uh, for all the off camerara help today. Really appreciate the the ideas and the comments and troubleshooting live. Brian Forest, always good to see you. Thank you for joining, guys. Go ahead and like and subscribe and we will see you again next week. Thank you. [Music]