CIQ at PEARC 2025: Highlights and Takeaways from the HPC Community

So what is it that you do at CIQ though? So just just tell everybody kind of like what So to come out of behind the scenes into the front of the scenes was kind of fun. So yeah, tell us a little about what you do. >> I am the um engineering security um I'm the lead of security products for engineering. One of my favorite things about bringing people like that have never been to a conference like this to a conference is just how someone's specific knowledge and experience immediately becomes useful and you're like oh man I'm so glad this person was here and that was

that was me at our booth with you Jason just every time someone came up and asked about a security thing about Rocky Linux and I would have like you know handwaved a little bit and talked at a high level and immediately you were like belineed a deep dive with them about stuff that I didn't know and I was really happy to hear. I mean, we talk about RLC hardened, for example, but just the work that that goes on with the FIP certification process and the hardening process and the STIG process. It was really great to hear you speak with such authority about those things cuz >> it's something people want you to be authoritative about.

Turns out >> Yeah. No, it was great to hear that so many people had such a high interest in security. Normally, you only see that in like security conferences and events focused on it. So, it was refreshing to be in the same place as Jason during the event. For sure. Absolutely. I learned a lot. >> Yeah, agreed. So, thanks for coming along, Jason. >> Yeah, of course. >> And how about Has Chris been on one of these before? Do people know what Chris does? I can't remember. >> Who are you, Chris? What do you do? What do I do here? I run the open source program office here at CIQ.

So for me, going to an event is like, you know, getting getting past just like the sales team and talking to customers, right? Like it's actually allowing me to talk to the engineers that use the open source bits and the commercial bits hand in hand. >> Yeah. Were you able to get to any of the the talks or the tutorials or any of the workshops? I only went to the Warewulf bath, which we'll talk about a little bit more, but um maybe we'll just go around. I'd love to hear your experience. I think Jonathan, I think you went to a few, right? >> Yeah. I I first of all, I've just switched my microphones around.

Are you able to hear me still? >> Sounds good. >> Okay, cool. >> Yes. All right. Um so, yeah, I My focus at the beginning was the Warewulf buff. Like you said, I was really glad that it was on Tuesday, so we got that done and I could kind of relax for the rest of the conference having gotten the the specific thing we were participating in done. Uh, but I also got to attend the open HPC buff, which I I hadn't even noticed on the schedule. Someone just happened to be walking past our booth and mentioned they were headed to it and I just attached myself to them and went out the door.

Um, so it was good to hear from from Chris and Tim over there. um and a couple other people, you know, that were speaking about their use of and uh interaction with Open HPC. Hoping to do more uh integration work there, especially as we bring more of Open HPC in scope with our Warewulf Pro product. Um but also just to take advantage of their their testing infrastructure. Um but I I also went to in a couple I like to go to buffs. So I went to uh the bof for uh open on demand which is an HPC focused um dashboarding system and the bof for cloudfront which is a user management like project and allocations management system for academic HBC centers.

Um there were a couple of uh user reports from the CloudFront buff that both it was two different presentations that were talking about their very extensive and positive experience of CloudFront. Both of which effectively entailed rewriting every file in the project. So that that was an odd thing to hear that apparently integrating it into your environment means almost rewriting it. But uh you know it was very cool to hear about that in the community. And then I went to one of the uh actual like paper presentations. They were using a large language model to install software on their HPC cluster. So if you if you do a traditional HPC environment, you install a bunch of software usually from source and there's config flags and things for how those get built.

And when you upgrade your base operating system, you might need to rebuild a bunch of software. They were trying to see if they could automate that process by having a large large language model um actually do the builds and then uh test to see if the build succeeded and if it still ran properly. And then they had um a set of four different models that were all doing the work and they were doing kind of a bake off to see how good different models were at doing it. They they were talking I think about 60% success rate on the positive side which I guess if it means 60% of the work you didn't have to do and you only have to go back and do 40 of it because you can kind of you just know if it failed or not it's not maybe it's wrong somewhere.

So maybe that's good but it was at least really interesting work. And then I went to one other presentation about um about OS upgrades or OS I think they were moving from like an enterprise Linux to an Auntu. Um they they were talking about OS upgrades in HPC environments and man it just sounded like a nightmare and there were other people in the the environment that were like having the same nightmare and they were just commiserating and talking about the pain that they were going through and I I did my best to just listen and not raise my hand and say what if you did it a different way.

What if you did it with Warewulf instead and it took a day and you didn't have to worry about that. But it, you know, it's interesting to see just the breadth of the ways people run uh advanced research computing systems as the uh conference title suggests. So I I went to a few other things as well, but that that's most of the highlights. I also checked out the the poster presentation. There were some really good stuff in there, but maybe we'll get into that in a bit. >> Yeah, agreed. I I had a great time talking with one of the guys who actually he I think he was on his way to go put his poster up >> and stopped at our booth and talked for a little while and then was showing us his poster.

So we kind of got to like you know chat with him a little bit more in depth about what he is doing and um I'm really excited to collaborate more with him as well and support the work that he's doing bringing I think he's he's bringing compute resources and like actual like hardware and then knowledge of how to run um you know the compute environments to areas that don't have it specifically in Africa I think he was saying. So >> yeah I I I took a picture of that because I I loved that moment. uh it was immediately like a a an impromptu poster presentation in our booth.

Uh he and his colleague were talking not just about the research that they were doing but the particular challenges that they have kind of attracting and and uh retaining talent in their region and then also just understanding and having a community of best practice there. They feel pretty isolated and so it was really great they were able to come to the conference, talk about the work that they're doing and then hear from other people doing the same work. But yeah, I I love it when someone is so excited about the work that they're doing that they're like, "No, let me show it to you. It's right here." >> Yeah, it really resonated with me because I've worked with a couple uh colleges in previous roles in Africa and they had some of the most unique problems you could possibly imagine.

like you know the stream ran dry so our hydroelectric plant is down and uh yeah the davis is off >> until it you know gets water again and starts flowing pretty wild >> until it gets water again that that could be like months >> right exactly >> that's wild >> it's wild like but you have to remember um the the continent is huge >> and the internet is relatively new. There's a lot of infrastructure development that's happened in this century there around you know internet connectivity and you know trying to boost compute capacity for all of the African nations. So you know it's really cool to see someone else working in that arena and you know sharing that same problem space.

>> Yeah. >> Yeah. This this was the uh the research their company was called research data communication technologies and they specifically do they support biioinformatics research uh in Africa and they were very opinionated and and uh passionate about trying to build up a community of practice and expertise there in the region. It was great to hear from them. >> That was awesome. Big shout out to those guys and girls, I'm sure. Beautiful people. Um cool. So, Chris, where did you make it to any of the um >> I didn't sadly and of course I heard about the open source bof after it finished and that would have been a great one for me to go to and just be a fly on the wall, but uh maybe next time.

>> Yeah, I mean the conference was small enough and I kind of liked it was funny. I was actually talking to um another one of the people who had been there last year and part of one of the the complaints about the way that it was set up last year is that um you know the the vendors and where that area was was so far removed from you know what was actually happening that it was hard to get people in you know and this was completely different like they had to walk past our booth to get to where they wanted to go the presentations and you know the student stuff and and all of that I mean it was a big building there were other things going on too but we were part of it like the food was there, the coffee was there, right?

It was like a lot of the things where people wanted to be was right where the booths were. So, we got a lot of action and um activity there, which I I really appreciated. So, we didn't really have to go much of anywhere, right? Like I I went around and got to chat with people at every booth uh without having to go to, you know, a a various presentation. So, that was that was kind of cool. I like the way that they did that. >> Uh Jason, did you make it to any of the talks? No, I I did not. >> Yeah, you were hanging in the booth with us.

Yeah, it was good. Yeah, it was very good. >> How about any of the other vendors? Did you Did you check out the >> I had a really great conversation with someone from Access and how they just um basically are just talking about all the other um clusters and resources and how to manage it. And um I talked about fuzzball with them and they were they were big this person was a huge fan of fuzzball and I could just see how fuzzball could work you know when I know access is just an entry point into all the other systems but you know fuzzball could just

I mean I'm pretty sure you could talk more about that Jonathan but just how it could be utilized throughout that system you know >> yeah it might be worth giving the context of what access is and and how it relates to the Perk conference. So before PERK was PERK, it was the exceed conference. That's XD and uh it it stands for something terrible, I'm sure. Uh but uh it it's it was kind of a a long-term NSF funded program um for bringing compute infrastructure from different NSF grant sites and I think other government research sites. Certainly DOE participates in it too I think um to to let people use the resources at other institutions.

So you could go to exceed and you could get a grant for exceed resources and then that grant could be realized at a number of different uh HBC clusters around the country. Um that program has wound down and has been replaced by the access uh process which is a bit more decentralized and access is more about just the the userfacing allocations management part of it and less about the infrastructure and each site is a bit more independently um independently responsible for the infrastructure locally. But um in order to let the conference survive, they they took it out of the exceed program and made it perk. And so it of course is still very um you know very related to similar academic research uh in general and I think uh perk is an ACM conference now.

It's kind of like how Apptainer was put into the Linux Foundation. So it would would live beyond and kind of have this entity over it. Perk was taken and put into the ACM. Um but yeah, Access is very much interested in giving the general population of US researchers access to um to HPC computing power wherever it happens to be which of course is a big part of the mandate and and vision of Fuzzball too. So, it would be very cool to to work together and and see where we might be able to support each other or what fuzzball could do to be more applicable to that kind of a a heavily distributed research computing environment.

>> Oh, I like it. >> I really like that. Jonathan, does that just like make you so excited? There's like some other uh like collaborative research data centers. Is that maybe what you would call it? like they they have like multiple clusters in different areas and researchers from all over the country kind of dip into it to utilize those resources. >> I think that one I don't remember the actual name and maybe you're maybe you're right maybe that is it. there was like a three-letter acronym they were using for it and it's um I think that's explicitly a a decentralized Kubernetes environment and that was a bit more like an exceed or a terror grid in that they were more opinionated about how your infrastructure should be configured so that they were more explicitly interoperable.

So yeah, that that kind of looks a little bit more like what a fuzzball federated deployment would look like. Um but more on the service side of things than the batch compute side of things. >> Yeah. Yeah. I think that's a different webinar. I think that's a different conversation going in that direction. But >> there's more there's webinars in the air. There's just >> they're all over the place. Exactly. But I mean I think that that really highlights the excitement of that particular conference. There was a lot of youth there, a lot of students, a lot of excitement. there was um just you know people doing wild things all over the place just because you can just to see if you can um which is what I heard you speak to a little bit as well Jonathan.

Um yeah that was really exciting. Okay so is there is there more that we want to talk about just of like the the conversations that we were having? Um I know that we talked about the the the one guy that came over with his poster and that was really fun. Um, what other conversations kind of kind of came up that you would like to to share and make sure that we jot down and make a note of and follow up with? >> Well, I of course came very opinionatedly there to uh spread the word about Warewulf Pro and what we're doing with that. And so the the the bof was a community buff.

So it was we we mentioned what CIQ is doing with Warewulf Pro, but that was more the the core of of the project. Um, but there were a number of HPC integrators there, some that I, you know, I have previous relationships with, and it was really great to get, uh, frankly just our solutions brief for what Warewulf Pro is, what our vision for it is, and what the value ad is for our customers and our shared customers with an integrator. And I was very pleased to see just how much immediate interest there was for that specific vision that we're casting. and and even more so since the conferences have been kind of putting finishing touches on on the product side of it.

You know, putting it in the box as it were and seeing just how easy it is to stand up a cluster uh using, you know, the pre-built node images and the overlays that we're putting together. It's it's I as an engineer have been excited to see how good it is. Um, and I'm looking forward, as you mentioned, having follow-up conversations with uh potential integration partners on that front because I think they will be pleased as well. But yeah, spoke with Aspen Systems where I have some friends there, Microwave, Dell, and there were a couple others that I I am embarrassingly forgetting. But yeah, it was great to to share space with them.

And then went and checked in with the uh with the BGFS guys. I always like talking with them. It's usually the same couple of guys at a conference like this. And um one of them called out a uh a an article that I had written on HPC networking several years ago. I he suggested that it might be linked from the BGFS docs which was news to me. Uh but but that was kind of nice to see. Always good to catch up with them. >> That's awesome. >> That is Jonathan. You're out there, man. You're like the the the face of many things. You know, you you do this stuff long enough and it just ends up happening.

>> That's fair. Yeah. The um Oh, sorry. I was going to go in the the Warewulf bof direction, but is there more that you want to talk about, Chris, of the conversations that you had and kind of the excitement that was built up? >> For me, it was kind of my first bath in the HPC space like >> I have not had a lot of experience there. I know it's very awesome technology and very big compute clusters. So for me coming from the Kubernetes community where I've been for the past few years, it was very interesting to see other ways of thinking about managing and orchestrating containerized workloads, especially in the you know some of these systems are massive and rank well in the like top 100 compute systems on Earth.

So it's very fascinating for me. I think a lot of the conversation about Linux and how interesting it was for me to see that a lot of people are still using like version 8 of enterprise Linux distributions and how much uh interest we got in the fact that we supported FIPS. Jason, did you want to touch on that a little? Yeah, I had um FIPS certification for validation for the Rocky cryptographic modules. It's something that I lead here at CIQ and we put significant effort into bringing um the EL8 cryptographic modules up to 140-3 specification. And um because we knew that there weren't a whole lot of companies in the community supporting uh EL8 and um just talking to people that they were very surprised like oh you have support for EL8.

I'm like yes we do. and um just talking about how we added FIPS features for uh TLS 1.3 um so our customers could be compliant with TLS 1.3 and just all the other um additions we had to do to keep it compliant with the new specification and um I you know I I work with it every day. I I live in that and it was just a surprise that other people didn't know what we were doing. Um so but yeah so being able to talk about that and touch with touch on that and um them being so surprised and happy that they had somewhere to go for someone to support them on that.

So that that was really cool. >> That is cool. Thanks for all the work that you're doing with that. Um you know people that that don't know they don't care but the people who do know they care a lot. a lot >> a lot. It is very detailed and a long process and yeah, thank you for for doing that. Appreciate it. >> Cool. So, let me think about some of the conversations that I had. I mean, I think most of them we we um talked about already. Um I think it was really cool too, Chris, for you to be there because there are so many of the projects that we uh support that are open source, right?

I mean most of them right so we've got Rocky Linux and all that that comes with that we've got a Warewulf of course open source great project um Apptainer of course the Linux foundation is kind of taking care of that but we do the support and the training around that um also ascender which is our you know anible automation platform downstream of of AWS um and then Fuzzball is our proprietary software and we you know we talked a little bit about fuzzball and the awesomeness of that but um being our open-source guy here at CIQ. Like, how cool that you got to dive into and talk to so many people that are like >> using the projects and the products and then also many of them were our customers already, >> right?

I mean, and some of them weren't and some of them were very, you know, um, enthusiastic about their use of Rocky Linux, too, right? Like it wasn't just an operating system. it was very much a vessel for research. Um and and that with Warewulf in and Apptainer talked about at the same event was very very refreshing for me >> as someone who you know works in open source every day like I don't get to see and talk to people like that intimately in uh a given conference about things and you know there are still challenges in open source that can be solved and especially in the HPC space I I think the cost of the systems will always increase but you the use of open source really does help democratize some of these really expensive efforts.

>> Yeah. And it's also nice because especially in the education world where there's lots of students, right? they're using open source products and projects, but they don't they didn't even know that there's maybe training available, that there's extra resources available, that there is support available. Like oftent times you're just running around kind of using what you use and you might settle on one, but even just understanding that there is a company that does support these these um these projects is like that was also refreshing for a lot of people because they don't really want to run things unsupported, right? especially depending >> like having been in their shoes like at companies that are exclusive open source consumers and only bought support contract for the things we didn't have expertise around like I get it.

It's sometimes you're the authority just because you have the most experience with it or have solved the most problems in a given space or with a given project. So the the amount of like interest that was, you know, around our booth, you know, folks came up because they knew the Rocky Linux logo or the Warewulf logo or just they they understood what CIQ can do, but also they they also figured out who the people were they could go to to kind of not necessarily get like unpaid for support, but just these people know this and we were all there in the booth together, right? All of us have our unique skill sets there and it was it seems like all of us use those skill sets during the conference.

So, it's really great mix of folks we had and talking to the students was just like invigorating to an extent because yeah like a lot of them have to use open source because anything proprietary would just be cost prohibitive. So, >> yeah, >> it it was really awesome. >> Yeah, it was awesome. I I heard several people comparing um Perk to like a small FC, right? So, it was like a lot of the same Yeah, a lot of the same people, but just uh you know, slim down to where you could actually talk to everybody. I think we probably at least saw, smiled, waved at, talked to just about everybody that was there.

And so it's a bit more mellow and relaxed and and intimate probably is a a good way of saying it too, but in that wonderful way where it's not it's not as, you know, intense and overwhelming. So if you like SC, but you like it a little bit toned down, I would definitely recommend going to Perk next year. And I I'm putting it on the list, you guys. Anyone behind the scenes at CIQ, we are going next year. And I would really love more to be part of the what the students are doing. >> Yes. >> Yeah. They were looking for like student mentors and things.

Um there was opportunity for individuals to sign up for that. I I think I started to and then just ran out of time and was like I'm not going to do a student uh justice by >> half half doing it. So it'd be great to put time into it next year. >> Yes. Exactly. And I went and talked with um some of the students and they gave some great suggestions on the ways that we could um interact with them and what would be uh valuable and beneficial and even I could help the students. It was so cool because I'm not technical like you guys. So I I am always referring to you.

That's why I love going to conferences with you guys because I'm like, "Oh yeah, yeah, yeah. I can smile." And then oh, you need any like actual details like right over here. But one of the things that they said would be so valuable to the students is like what is an open source project? How do you get involved with an open source project? Where do you go and what do you do? And then also just like those soft skills like hey it's okay to walk up to somebody and ask them a question to shake their hand to ask them where they work. Like all these little things that we just kind of assume as we get older like yes of course do that.

But when you're a student >> you don't know how to do that. It's like so overwhelming. Yes. >> Yeah. So, that was really cool. Definitely be putting in there to to help the students along the way. Those are definitely our people. >> Cool. So, is there anything else that you little little wrap-up maybe final final comments, Jonathan? And maybe just like we we've talked about um you know partners and customers and things, but it was also just great to hang out with you guys. Great to hang out with people that we >> work with in the community all the time that we don't get to see very often.

The conferences are always a great uh opportunity to just be present with people. We spent a lot of time with our our friend and colleague Ian Kaufman from uh SDSC, right? Work with him and the the Warewulf community. spent a lot a decent amount of time with Tim Middlecoup also from Warewulf and then just a number of other people that you know you see at these conferences. It's great to catch up. >> Yeah, agreed. Final thoughts, Chris? >> Um, yeah, like after talking to several of the students, I really want to get more involved there because like students are the future. Like they'll be sitting in our shoes 20 years from now.

So it would be nice to boost them and enable them to do more good in the world and that could be through their jobs or through open source or however like any amount of help we can give those folks because a lot of them were for or not from the US >> um talked to a group of folks from Uganda I think if I can remember correctly that were there you know part of the student program there. So yeah, that's work that I can help with in the future. For sure. >> Yeah, agreed. And Chris, they're smarter than us. It's not going to take them 20 years to get here.

>> Hopefully. Yeah. >> No, it's because they will have invented time travel by then and they'll literally be right here. >> Let's just slide over as they slide right in. >> Oh, that's awesome. Thank you, Chris. some it was really good to see you in person and hang out too. Enjoyed that. Okay, Jason, final thoughts. >> Um, no, I had a great time. I had a great time talking to the students. Um just what came up to my mind is just you know a a guide to open source and different projects and you know they're they're they want to help and a lot of times they you know everyone has time but but here's a guide to how to step in, how to learn, how to get involved in the community, you know, and because they want good things and everyone wants good things for them.

but how do they get their foot in the door? And um I think that's huge for everybody and that's where I would like to help and just say I'll here I'm here to listen or this is what I know about the community and um yeah next time I I attend I I I would really like to be in be a mentor and um but also you know the community is always there just people don't always see it and they're not always it's not you know you don't want it to be out of sight out of mind. >> So They need some kind of guide. >> I may or may not have a talk at another conference that is an open source survival guide of sorts.

>> Nice. Wow. >> Yeah. >> Open source survival guide. Is that the the name of your talk? >> Yes. Yes. >> Are you surviving using open source or are you surviving open source? >> Yes. All the above. But it's a it, you know, it's a it's a paradigm shift from what we're, you know, used to. I'm doing this because I'm passionate about it versus I'm doing this because I get paid for it kind of thing, right? Like obviously I'm very fortunate to be on both sides of that equation, but not everyone else is. So, >> I think it'd be good to share the knowledge and experiences I've had.

>> Yeah. Agreed. Agreed. Well, yes. Uh I I thought that that was a wonderful conference. I'm so glad to be there. I'm excited for next year's conference. Thank you everybody who is involved in um you know producing and being a part of perk. There was a lot of people that came up to us all throughout and at the end you know asking for their you know our our suggestions and our feedback how to make it better. And I just I I really like that. I appreciate the iteration on continuing to improve and that was uh wonderful. I mean we talked to customers, prospects, people in the community, open-source, um, our partners, integrators.

It was it was a full full conference and we're so glad to be there. So again, thank you so much all of you for attending Perk and for being there and uh we hope to talk to you soon. All right, if you have any questions about CIQ and and the support that we provide from, you know, all of these open source projects, Rocky Linux, Warewulf, Apptainer, Ascender, and then of course our own proprietary stuff, Fuzzball. We'd happy to chat with you. Just go to ciq.com and we will see you there. Okay, have a wonderful day. Thanks, guys.