Continuous Integration and Delivery with Apptainer

running uh workflows jobs specific applications things of that nature um it's been really heavily picked up within the high performance Computing um uh space it's been around for about six years it's a pretty mature product apptainer itself is the successor to the singularity project aptainer is the Linux Foundation uh like uh variant or the Linux Foundation approved version of the The Singularity project so that in a nutshell is what it is um so apptainer integrates really well with uh gpus with other types of Hardware uh it's really great once again for you know running jobs it integrates very well with um the host file system

and with other things it basically makes a lot of things that are usually difficult with containers a lot easier excellent thank you for that I see your videos a little bit behind so I think you're maybe that's just me all right so I know today Brian you want to talk about CI CD and I feel like that's a term that we hear a lot nowadays from everywhere within it not just HPC maybe it came out of it but kind of I think it gets overused to some degree and I don't think everybody knows exactly what it means so talk to me a little bit about

cicd and then tell me why I would like to do that without Tanner sure uh so let's just start off by formally defining you know what CI is right so continuous integration is the automation of integrating code changes uh made by multiple contributors to a software project uh so in the context of obtainer and HPC um as the solutions architect team and with working with fuzzball we obviously work with a bunch of different software and we build a lot of containers and obviously these containers need to be tested uh so let's let me jump into maybe an example so uh my experience has been mostly

in uh genomics so let's say I am some software developer I'm working on a new alignment or I'm making changes to an alignment software and uh and these changes are going to be made for let's say performance purposes right um because this is a genomic it's a scientific piece of software ideally what continuous integration does or what it should show is that when I make these code changes I should get consistent results so if I'm aligning once I make these code changes uh the the result the result file should be the exact same as the file uh that was created without the code changes right

and this becomes really important in the context of uh clinical analysis right because it would be very bad if uh you ran analysis on a patient's genome and uh certain variants are called and you make these code changes and uh you know now you're getting different variants and someone has already made a diagnosis on the patient and they've already given them medication so there are very there are a lot of real world consequences that can uh happen with uh not getting your integration tests and having consistent results so um you know I would like to pass them out to the team and see if you

got see what your experiences in you know and the environments that you've worked in with containers and testing them and whether you've like automated this type of testing in the past so I can speak to this when I was at my prior institution and I was starting to deploy Singularity at the time containers Alpha researchers for the most part I didn't really do a lot of like cicd with it I was basically just kind of figuring out even just how the technology itself worked much less trying to apply it to something like this so my workflow was basically build out my def file on my

workstation build that container on my workstation send it up to the cluster and then kind of do testing there even that without obviously as you've elaborated on Brian there's a lot of external benefits about automated testing even without automated testing I was um finding that that was still a fairly effective workflow to get stuff out to users um just because of the external benefits that containerization gave to me as a CIS admin and freed up my time to you know start looking into how I could do things like automation um but yeah in general this is uh some of the um kind of first work

that I've really took a deep look into that you've been working on here Brian um with them doing this type of stuff with containers so very excited to see uh specifically with um obtainer so very excited to see how this integrates Dave do you want to add about there you go yeah I was just gonna jump in real quick yeah so I I don't have um so yeah so this would this would be great too you know so my background um is uh you know some of my background is that I used to be at the National Institutes of Health and I used to create

containers uh for applications that scientists would ask for and um the way in which I would do that many other staff members would do that is to create new containers with new versions of the applications came out and then to do some you know manual testing if we had uh you know some example data and we had some example workflows that we could use to do the testing and then to push those containers up um but you know sometimes it's the case that you don't test everything or you don't test um enough things or you know whatever um and then a you know a researcher

will come back and say hey this this used to work and now in the latest version it doesn't and so that kind of stuff would be great uh you know for those types of situations I don't have experience with CI CD within the context of containers uh it's kind of funny because I do have experience with cicd actually with aptainer itself so the workflow that you described is actually um you know exactly the kind of workflow that uh the developers of app tainer have set up um you know to to test and and and and uh push out the software itself as new versions come

out so it's it's widely this this workflow is you know it's widely used and it's pretty much it's like ubiquitous yeah it's amazing I I hadn't thought of having cicd for whatever reason I when I started looking at HPC and and research and something I didn't necessarily put together but I'm glad to see that it is something that I just missed and it seems to be growing a lot of the conversations I have now are people integrating that into the life of their cluster it's not just about the the people writing software but how are they actually getting the software onto the cluster and getting

the researchers going faster so it's very exciting and I think Brian you had something you wanted to show us as well right uh yeah yeah but before I show that I guess maybe we can just talk really briefly on just uh continuous delivery and just kind of defining that uh so in our context like we're working with uh you know obtaining our files uh and basically these are being these uh definition files are being built into containers and what continuous delivery what it means to me is that once these containers are built they get pushed to a registry where uh our our customers can consume

them or we can cons we can start leveraging them within our fuzzball product uh I don't know if the rest of the team has anything else to add to that uh yeah I have a friend that worked at a very large hyperscaler and she was running a large development team and so they were pushing via their cicd process code to the hyperscaler about every 11 seconds so the amount of speed and velocity that you can get out of this type of process is amazing to me I just got the very tail end I think I understand uh the context here but one one thing I

thought would be valuable in kind of an end user application CI CD pipeline as like a service that an HPC Center might provide to their end users is not only in reintegrating and retesting when there are changes to the application but when there are changes to the underlying infrastructure something that we would hear all the time from our end users when I in a past life as a cluster admin was they wished that they could just hand us a suite of their applications with a test case so that we could make sure that their applications still uh functions properly and performed the same or better

when we made file system changes or when we made scheduling changes um and that that's something that I think as as we see more people containerize their applications than in so doing making it easy to build and test them in an automated way that might be become more and more feasible at different sites that aren't able to do it already foreign thank you Jonathan all right Brian all right then I guess we can uh jump into our demo for today uh so basically I've set up um some bitbucket Runners um with within that bucket and I am uh and for today's demo we are going

to be building uh Promax uh we're gonna be building a container with robax uh so let me start off by sharing my screen real quick and perfect do that okay um I basically have uh the pipelines page pulled up uh within our bitbucket repo so I'm just gonna build a uh definition file that's on our main branch uh the uh pipeline that I'm running right now is going to be a build and push uh pipeline so it's just basically just gonna take the definition file uh build it and then push it with whatever tag I specify here so uh let me just go and do

that so I'm just specifying this grow Max file right uh definition file right here uh I will jump into uh I'll jump into that uh after we take this uh build off and for the purposes of this demo I'm just going to tag it with uh CI demo so all right let's kick this off and Twitter off so uh as you can see uh this build has kicked off and what it's doing right now is building my container so uh let me just jump into what exactly we are building so as you can see here uh this is a this is basically a definition file

for gromex uh we're building this uh using a rocky Linux 8.5 Docker container as our base um at the top here we have LD Library path and uh path Set uh in our environment and uh and basically what this will let's just make sure this just ensures we have access to the correct libraries and uh binaries uh that are in our path and then at the bottom here we have uh this openmpi variable that uh allows us to run openmpi workflows as root and this is because within fuzzball uh our the your jobs are run as root but it is not a privileged user so

uh let's jump into I guess the post section of this file as you can see it is such in front yep can you make that can you zoom in a little bit it's a little hard to see oh yes yes for sure it's just me but there we go thank you no problem so uh as you can see we are uh installing some dependencies uh just through dnf right here uh in this section here we are installing uh some NVIDIA drivers in Cuda and then in this third section here we are installing um some AWS uh EFA stuff and basically this will allow us to

leverage um the instance types that have EFA on AWS so an example of this would be um the c5n uh instance type and then finally uh we are pulling down the gromac source and we are building it against um EFA cool hey Brian I got a quick question for you yeah doing a chromax what is chromax so Promax is a molecular Dynamics uh software uh it runs through um using MPI and is embarrassingly parallel uh I'm not sure that that is uh my understanding of it I'm definitely not a uh power user of this software but uh Forest if you have anything else to add

to this uh yeah by all means I was just going to say uh that basically covers it grow Max is a pretty standard molecular Dynamic Suite it's really commonly used very popularly in the um kind of pharmaceutical field for doing uh drug Discovery and that type of stuff so like take 500 candidate uh you know pharmaceutical chemicals uh test them against you know certain tissues and types like that and um you know do simulations of that performance comes out in like nanoseconds worth of simulation a day but essentially one of the biggest use cases for this is pharmaceutical drug discovery cool cool thank you Forrest

all right so uh We've basically covered uh what we are building so let me jump into I guess how these bitbucket Runners are kind of set up so uh still the type of bitbucket Runner uh I'm currently using is the Linux shell Runner uh I can jump in and these are configured using a ansible Playbook uh that I wrote uh the bitbucket runner uh is basically just this Java file uh it's a jar file so I'm installing Java here uh on my instance uh I'm also installing apptainer creating a bitbucket runner user uh and then finally pulling down the bitbucket runner and installing it basically

uh let's see what else do I do so uh down here um I've basically turned this bit bucket Runner into a service uh basically did that just because uh in case our server gets rebooted um ideally I just want the service to come up come back up on its own uh setting up a uh Docker config Json here uh the reason for that is uh we are going to be pushing containers to our Google artifact registry so um within this Docker config file I am basically uh configuring apptainer to use the Google Cloud credentials Helper and then last but not least uh I installed the

Google Cloud CLI so that I can authenticate with gcp and actually push the built container to our artifact registry and Brian I just want to make sure I understand here so this configuration here this bit of ansible this is configuring a conceptually arbitrary Linux machine to be a bit bucket Runner this isn't something that is executed every time a a an integration happens is that right uh yes yes that is correct so typically you would only need to run this once uh so how I've set up the host file is that I give it the I the IP of the host and I also pass

it on some variables that bitbucket gives you so uh these are things like oauth client IDs that are specific to uh each bit bucket Runner so after you've set one up uh in after you set up your Runners within the bitbucket UI you can populate the host file with these variables and then run the Playbook against uh the instances um ideally like with the oauth client IDs and stuff uh currently they're in like plain text for demo purposes but in a production environment I would like I would ideally like to access these with something like Vault for example cool so uh last but not least

um we can jump into uh the actual build and push script that's uh actually doing the work uh most of the work for this uh CI CD pipeline so uh this is basically a bash script uh it takes a file and a tag as a string and basically does a little error checking here just to make sure there's no spaces in the tag uh make sure that the file actually exists uh and once uh and if it actually exists uh we basically using the file name uh figure out a image name and give it an image path to build to and then start the build

uh and then once you're in it once the image is built if your container file has any tests that are specified within the tests section of the definition file then it'll run those uh if you didn't specify a tag uh I just tag it with uh the git commit hash uh and then finally uh I basically authenticate to uh Google cloud with um our service account that has uh write access to the artifact registry and then finally the image gets pushed uh yeah so that is uh the script that we're running today uh yeah foreign can you give an example of the kind of tests

that like are there any tests on this gromax one for example or or what how what might someone do to test their their container so um in terms of HPC software uh my ideal test would be like the smallest possible test case that uses some form of input for the actual piece of software but um if you're trying to do just a quick sanity check just to see if your application actually installed correctly uh I would just call the binary and try to get a help message out of it sure yeah we do have one question yeah Brian sure from Sylvie oh different one do

you have an opinion on containers and Commercial isvs should the iscs build a container or does that have to or does it have too many security consequences um I mean I think most isv in my opinion from the isvs that I've worked with in the past I I'm a firm believer that their software can be containerized and can be used as a better means of Distributing their software to the end user um not sure if the rest of the team has any strong opinions on that but uh yeah I'll pass the Mike TR they're definitely considerations especially in 2022 around like the software bill of

materials so knowing the exact provenance of where your containers come from um is important whether that's you know something that the company has built is giving to you or something that you're building and distributing out to your users um maybe I'm finding out the obvious but I would say that um some commercial software vendors it's going to be very difficult I agree I think most things can be containerized I think that is the future there I think one of the biggest problems there is just going to be um being able to get access to the proprietary software and all the build instructions and things like

that to do it yourself versus having that pre-packaged from ISB I'd say that if you're going to get that from a software vendor they should probably be looking at you know like their own software build materials and stuff just like you should be so just making sure that you know if you are getting containers from them that they're you know doing their due diligence necessary here in the modern world of containers to ensure that you're getting a safe product um which ultimately a lot of that stuff is stuff that you should as well if you are containerizing software and serving it out to your users

whether that be open source or proprietary solutions that you actually have the means to be able to build yeah and I think I'm on this on board with the other people you know what's been already said it would be good if more isps Upstream were publishing containers for their software alongside that if there are any security concerns what I would want to see it as a user of that container or of that software is just the container file that was used to build it and even if your application isn't open source like the the container build file whether it's in apptain or build file or

an oci build file or anything it's really just a script that would be akin to the installation instructions for that software but then you get to see exactly how they built the environment that they run their tested environment in or run the test for their application in and that would let you rebuild the container yourself presumably uh even if their code is proprietary and not open source and I would add to that uh you want to make sure also that the container that you get this is again this might go down a rabbit hole but just real quick the container that you you know you

want to make sure the container that you get was actually the one that was built using that definition file so this is a great place to um ask the vendor to uh to sign to cryptographically sign the container and to make sure they communicate to you the fingerprint of their signature so that when you have it at your you know shiny little terminal you can go ahead and verify it and make sure that that same fingerprint comes up and um then if you like you can use a obtainer inspect dash dash def file and start going through that definition file and see how they came

up with what they came up with because if they um you know signing it is great but if they signed it after they built it from an untrusted Source that's not so great so you want to also make sure that you know Providence you you trust the author and you also you know you're happy with their sources where they got where they started the container from thank you Dave we do have one more question I believe so uh what improvements would you make to your cicd pipeline oh that's that's an excellent question uh so uh I was actually pondering about this this morning uh one

of the uh big limitations uh with the uh bit bucket Linux shell Runner is that um I can only run one build per instance and I was actually thinking about uh how can I be able to run multiple builds uh using the Linux shell runner on on a single instance uh so uh the way that I thought of to approach this was to basically install the bit pocket Runner within apptainer and then have update and start start it as a obtainer instance and then that way uh you can in theory I should be able to run multiple builds within uh an instance so uh you

know that is definitely something uh I'm going to test and if I do get that working hey that'll be um a future webinar that we'll be doing very cool all right Brian we're gonna run this and show us something else uh yeah so let's let's go back to um our build and let's just see where this is at we could make that a little bigger if you don't mind too there you go and we are at the point of uh oh okay perfect so as you can see um we've basically run through the chromax install here and uh down here you can see it's creating

um the Sif file and then finally uh the zip file gets created uh in the grow Max uh definition file that we are looking at there were no specific tests that were specified uh so it basically didn't run any tests and then finally uh we authenticated with um Google cloud and pushed our uh built uh image to um our artifact registry cool and yeah that basically concludes uh My Demo so I will sharing my screen and yeah we can we can start wrapping up that's great I mean if you guys have any other questions start throwing them in there but I kind of want to

go around a little bit and ask so I mean we talk about or the big bucket Runner what exactly is a bit bucket Runner so um a bit bucket Runner is basically a uh it's basically a service that waits for um build jobs to uh build jobs within bitbucket so let's say I I do I kick off a manual build uh bitbucket runner will uh listen for one of those tests and if there is a task present it will pick it up and run whatever uh build you've specified and these builds could be like building a container or um in different environments this could also

deploy to like your production environment for example and then maybe maybe this is a question for a different topic but how does that relate to some other tools like Cloud bees or Jenkins um so it's basically like how my understanding of it is basically it's bitbucket's version of like GitHub Runners for example okay yeah yeah yeah so so Jenkins would be like a more genericized third-party one of these that isn't tied to any particular Source repository but as as the source repositories have become a greater and greater source of Truth for all of these things GitHub was one of these bitbook has one of these

I think yeah I get get lab has one of these if you're doing self-hosting or using their services and that's the one that that we used in a past life of mine foreign thank you so James asked my environment is strict security requirements and containers need to be encrypted how do I achieve this with abtainer it's a great question for Dave yeah um so uh there's two different there's two different um kinds of site you know encryption kinds of things within apptainer and sometimes they get confused a little bit but I think the one that you're asking about is actually encrypting the containers that it

can't be read um and so the the way in which you do that is that you generate um a key uh and there are commands within apptainer actually because apptainer is a really full featured program there there are commands uh you know helping you to do that there's there's actually um some documentation online too on on the app Tanner website for how to create actually um I I'm I'm I'm wrong you don't create the con the key for encrypting the container within obtainer itself you you uh uh you know you create your key outside of that workflow but any case once you've got that key

you can go ahead and encrypt your container um using the public key material and then whenever you decide that you want to run it you can decrypt it using your private key material um and so the way that this works uh is that you're you're con you're so a Sif file a singularity image format file is basically a squash FS image um with uh some other partitions in the file with some other you know metadata associated with it um once that's encrypted uh those different partitions become encrypted so that they can't be read when they're sitting on disk and in fact they remain encrypted uh

even when you run the container so that they're only decrypted within the new Mount namespace um so that you know they they remain encrypted uh on disk or obviously in transit so the um maybe we can post in the comments after the webinar we could uh put a link to that that documentation but it's it's pretty well documented within the app tender documentation how to how to do that encryption and decryption workflow yet and handling that encryption key wouldn't be that difficult um for a cicd workflow because like you pointed out Dave you you encrypt with the public key and then you use the private

key to decrypt but if you do want to do uh signing which you you might want to do alongside your encryption uh or even separately you would do that with the private key material but I I think Brian that as part of like pushing up to Google Cloud uh you're already handling secrets and I know you you mentioned this or you touched on this as you went on could you talk a little bit about how uh storage of Secrets works and how you might store a private key for yeah yeah for sure for sure uh So within bitbucket uh with your soul within bitbucket you

can set uh for your pipeline Runners you can set uh what are what they call secure uh variables and basically uh once you set these uh within the U why within your build job you can access these variables and if they're configured as secure variables even if you try to Echo that specific variable in your script uh it won't actually Echo it and it'll just Echo the variable name so uh that's how I've been handling our gcp secrets uh using that method and I think I've seen just a randomly note that you can do that same thing on like GitHub CI CD just in like

one random thing that I was tinkering around with this so that's common to all those platforms to be able to secure those variables like that thank you guys uh so we asked how our consumers of my container able to trust the container they are running I think this kind of goes back to what you were talking about a little bit before Dave but this is the cryptographic signing right yeah so so this goes back to signing and verification and so maybe I could explain a little bit the entire workflow so let's say that um yeah so Sylvie you have created a container and you are

sending it to another user or another um scientist or uh you know whatever and then they need to run it and they need to be able to trust it so the the workflow for that is for you to um go through the workflow of creating a a key and then signing your container and then sending you know in in like out of band in a different method sending your Downstream user um the fingerprint to that signature and so then you know let's say I'm the person I'm the downstream user who gets your container okay so I get this container I pull it down it's completely

untrusted I have no idea who created it what's in it you know whatever so um the first thing I want to do is I want to verify it so I would run something called a singularity verify which is going to do two things it's going to check um it's going to check that the the container actually is signed and then it's going to actually tell me the the fingerprint of the a key that it was signed with and so once I do that then I can compare the finger so that tells me it's signed once I do that I can take the fingerprint I can

compare it to the fingerprint that I obtained from you out of band and if they match up that tells me it's signed i u so now I know that it's not only that it's signed but it's also that it's signed by you um and so that lets me know now that the this this sip file that I've got is a bit for bit for a bit reproduction of the one that you created now the next steps um have to do with whether or not I trust you right and so this comes down to um you know and this is why the aptainer model of of

letting anybody sign and anybody else verify the container it sort of it sort of crowdsources the trust out right instead of saying we have a central repository somewhere where we're going to store all like official images and we're going to assign them as a team and then you just trust us in in this crowdsourced version you decide whether or not you trust the person who originally created the container so and then trust also comes down to two things number one do I think that you're malicious or not hopefully that's a pretty easy one hopefully I know you and I don't think you're malicious but then

the next one is a little bit more complicated it's like do I trust that you know what you're doing um at a very high level in fact because it can be very easy and it is in fact the the usual use case to grab um a container when you're building it from an existing container somewhere you know up on Docker Hub or somewhere else so do I trust that you know what you were doing when you built that container and when you grabbed it from the sources maybe instead of building it from another container maybe it would be nicer if I can look in the

definition file and I can see that you actually built it from the Upstream operating system mirrors because those you know have a higher level of trust uh implicitly than existing containers which are sitting up on a registry somewhere they kind of have to because all of us who are running Linux anywhere need to trust those mirrors right in order to just run Linux so okay so that's like hopefully I went through all the different aspects of it and there's a lot to it um you know it's kind of complicated but you know it's it's necessarily so thank you Dave good first just yeah just a

building that I wanted to mention um I think I used this term earlier but I'm not sure if I defined it very well um one thing that this kind of starting to come out in the world of containers for a long time we've all kind of had this good faith assumption that the containers out on the docker Hub are not malicious the container you know it's not going to have a crypto Miner hidden them one of those layers or something like that um there are starting to be more cyber attacks and things like that they're starting to be more concern in general uh about where

exactly as Dave was touching on like the software that's building these base images that are out on Docker Hub is actually coming from um so some of or one of the things that's starting to become very big in this field like I said I think that's necessarily but I'm not sure if I defined it very well and it's what's called an s-bomb or software bill of materials much like how you know if you're building a jet fighter or a tank or you know anything you would have a full list of where everything was sourced from and you would be able to trace back into every

manufacturer every you know seal plans every you know bolt manufacturer or whatever exactly where every piece of that machine came from with software you know software systems reach a similar level of complexity at times um in their own ways but we haven't especially as related to Containers had that formalized where did this entire container that we're using you know to run our multi-billion dollar website and that we have you know a million nodes sitting out they're all running at once you know what like literally I mean you might have you know thousands tons of thousands of machines all running this one container at once which

is some incredible scale to have that much metal working on something um and so ultimately it's like well what is that actually running and so one big concept is starting to come out here around like I said seeing you know if that software came from the mirrors and it came from the you know the repositories that type of stuff um is that software bill of materials that gives you the ability to trace back the exact provenance of all the software that went into building your container and know for a fact that it was built by a trusted source and it's not just coming from somewhere

random so that is kind of um really big and I don't have a specific like thought that I can say about like um a tenor and integrating that inherently it does all of its signing and stuff like that that's very useful to hypothetically you know for your custom applications you could have someone on your team sign off on them build them like at a very very basic level um but I know that uh those are things that we're very interested in exploring so we are definitely up on the you know the software Bill materials in the provenance of software um wave that's going on right

now within security and it's one of the things that's absolutely bolstered by just the process of building your containers as part of a CI CD pipeline if you're building containers manually and someone's having to manually construct a bill of materials of what's in that container and sign off on it with the key uh that that's an error-prone process it's very detail-oriented and finicky and it's easy to miss something when you're building the same container over and over again even if you're only talking about the released containers uh the the goal with doing something with a CI CD pipeline is that all that's automated and the

same process that's building the container in the first place is the process that is then cataloging what went into it and signing off on that is a holistic process that's great thank you Jonathan so guys I think if we're we don't have any more questions if we're up on time uh go ahead and wrap up well I have one more question I actually have one more question if it's cool for Brian um so you you demoed uh you know uh so this demo was great and it was like focused like at an admin perspective I think of how do you set up this uh CI

CD pipeline what I'd like to know is you know just a little I mean you don't have to like show but if you just talk a little bit about what is the look and feel of this for users so let's say somebody's already set this up you know I've got experience with cicd pipelines that are set up in which what it looks like for me is that I I push some change to some repo somewhere and it you know ends up uh because of the pipeline which is there um that kicks off a build process which creates an artifact in this case the artifact is

the um is the container and then that lands somewhere is that fundamentally pretty much what this looks like or is it are there differences there uh for the end user I guess it really depends on what you're doing if you're creating like a scientific application I think it would actually follow that type of process but let's say um I'm a if the end user is more like a bioinformatics user that runs workflows using multiple containers and if you were to have like let's say a fuzzball yaml for example uh that's pushed to a repo where you're making changes ideally you would have a CI CD

Pipeline with that where you have a some input file and then you run your workflow on it with all these updates to all of these different pieces of software and you should have some expected output that should be consistent and uh that's basically CI CD from like an HPC users perspective yeah how about like one more follow-up oh sorry go ahead Force I was just gonna say what about like for parameter Suites would this be something where for example you could kick off your CI CD pipeline to um take maybe an example use case of yours and try to find on your specific architecture you

know it runs you know 2 4 8 16 32 64 cores different configurations tries to determine what's most optimal and then like reports that back out to you so you know what the most optimal way to run it on that architecture is for CI CD I probably wouldn't run it within like a CI CD pipeline but I think that would be a really good use case more like a fuzzball workflow where you can have just a single job uh just multiple like different uh resource configurations within that OnePlus ball yaml and then you can just fire on all of those jobs at once I like

that yeah that's a great idea and then one more quick uh like uh ask a follow-up so it um you know this is another like potential use case could could this be possible with this pipeline so you use the the the example of gromax let's say I'm an administrator who's responsible for making sure that the latest and greatest version of grow Max is always up to date on the cluster could I uh set this up in such a way that it's triggered on a a major release of gromax within you know their repo somewhere and then every time there's a new major release um it

automatically builds a new container and you know pushes that someplace that I can check it out and then pull it down to my cluster or I think it definitely is possible I think the biggest challenges uh in doing that is just monitoring uh for those actual grow Max releases uh with the installation of the software so insta the installation process across different versions is usually pretty similar um I was actually thinking about it would be really cool if um obtainer had like some templating functionality where like you would just have the install script and all you would need to do is template out the version

and then you boom you have a new contain the latest version of the container and yeah uh yeah well those are my thoughts on that thank you Brian see we have one more comment that got made up again throw that one up there we go uh he just goes back to when Dave was talking so he points out the third third point they could have been thrown out uh can I trust the container to be updated and vulnerable or vulnerabilities get discovered and dependencies later there's definitely quite a matrix to keep track of with all the component softwares versus all the CDs that are out

there this goes back to kind of what we were just discussing with making sure that you know you're keeping things up to date you're still doing your due diligence on what's out there and keeping up with you know the industry and things like that um I don't know like what type of mass CDE matching schemes there are out there to like apply like I said all the software in this container to the possible cdus for it but starts to feel like a super Computing task at times with how much with the how big that Matrix could possibly get but I have seen you know like

for example after log for Shell came out for example a lot of different places added verification like Docker Hub I think added some type of verification specifically just for that bug and so or for that invulnerability and so a lot of containers now have like no you know log for Shell CBE detected so I know that it can be done automatedly and in some places I've already kind of looked at doing that for some of the major ones that have come out um but I can't speak to exactly what like I said doing that like Matrix between all the CDs all the software it looks

like because that um could be very very uh complex to be able to like build that entire tree out but I'm sure it can be done I I'd like to take this opportunity to pull on the pull on a thread a little bit and talk about something which uh is a very kind of nasty topic and um once you start to get into it and I don't think that the security like the security Community is really clued into this um or I mean I think they are clued into it but it like what's the solution there's not a good solution is the is the problem

at least not presently and this is that a cve is not just a cve across the board there are vulnerabilities within operating systems um that are reported as you know really uh high high magnitude vulnerabilities which are you know really critical to patch which don't amount to a hill of beans within a container depending on the container environment that you're running them in and vice versa there are things that pop up in containers that you know uh no big deal if it was on your operating system but because of the containerized nature of where you're running this it becomes a real problem and scanners and

things aren't you know they're not they don't differentiate between this they basically just go check databases and say oh well you've got this file or you know whatever and so that means you're vulnerable um and so I mean if you really really want to do your due diligence and figure out what's going on you have to do something like use a tool like um sift or whatever to create your your s-bomb within your container and then use something like grip to flag all the cves but then the next step which is hard which I don't think a lot of people go through is you actually

have to go through those cves and do some research and go you know go to miter or whatever and look and see you know what they're all about and whether or not they actually um are a problem for you within your container and you know you you may be able to get rid of you know just like you know delete files and things but at some points you might not be able to do that and at that point it's good to know like if you even care if this cve is uh is is even um something which is going to affect you within your container

the flip side of that which I said I don't even think is being addressed at all which is there can be cves which can affect you within containers um which are not being reported at the operating system level because they're not problems within within the OS level so I don't know uh I mean I you know send me send me a nasty mail if if there is something like this but I'm not actually aware of scanners that are they're finding stuff like that right now um and that's a it's an incredibly difficult task because those would be all specific to The Container environment which you're

running them so you need like a different scanner for every environment so this stuff gets really really messy and really ugly really fast and um you know I I don't know I think that it's not being addressed because there's not really like great ways to address that kind of stuff as of yet oh God thank God for us just say one more thing um about the the first side of that you said not bugs existing retainers no exist on the operating system um within obtainer itself there are a number of different capabilities that have been built into it over time that like Dave said you

kind of uh outmatch an entire class of vulnerabilities that can be done within a container just because of how it's built there um I'll just point out one example that I have on my mind because I think it's on our YouTube channel there was a PK exact vulnerability earlier this year that we found almost immediately you couldn't even do in a containerized environment which is natively blocked and didn't work from inside of it um so that's definitely an interesting side of it to see like everything that's been built over time into this to ensure security but I hadn't thought of that as well what can

actually affect the container that can affect the host I would imagine that some of that would end up coming up as bugs on our end um but it would be interesting to explore that concept more exactly what the differential there between yeah that's interesting especially listening to open the containers sorry got them while listening to Dave talk about well and really just hold this whole part of the conversation cve detection or vulnerability detection within the container it's it's actually changed my mind about the way I thought about containers a little bit I've in the past half jokingly been cynical about containers as Reinventing static linking

uh and just part of the continual go back and forth between Dynamic linking of applications versus static and we now we have statically LinkedIn or entire OS together into one file um but where that's wrong where I was wrong to kind of dismiss the concept that way is the the way the container packets it all up together you do still have the full understanding of what component parts go into this full stack so you get to treat it like this statically linked thing that you can pass around as a single sift but when it comes down to actually scanning it for vulnerabilities you can as

though it were not all packaged together look at the packages that went into it the versions of the libraries even if it's some crazy non-distribution thing that just has files in there you can still scan the libraries that are present and so while while it is work I'm glad to realize that the the way it works at least makes it possible oh thank you Jonathan so I know this is this seems like a topic that we could go down for a long a rabbit hole for a long time it probably turned into its own webinar bring in container security and scanning experts but it'll do

that at some point so I'm gonna go and wrap up because we are at the getting close to the top of the hour and I really appreciate Brian you putting this together and showing us and walking us through that it's always good to level set on what things are and make sure that we're all using the same vernacular so I really appreciate it guys I appreciate you being here today guys if you would like and subscribe uh and we will see you again next week for a round table thank you [Music]