CIQ
HomeCIQ Webinars

Storytime: How Did You Get Into HPC?

March 30, 2023

The field of High Performance Computing (HPC) has experienced rapid growth in recent times, leading many to wonder how professionals in HPC got started. This unique webinar offers a rare opportunity for people to learn from experienced HPC professionals about their journey to success.

Join us for an engaging and educational session in which a panel of experts will recount their experiences, their entry into the world of HPC, the obstacles they faced, and their strategies to overcome those these challenges. Our panelists come from diverse backgrounds and have different perspectives on what it takes to succeed in the HPC industry. Whether you’re a student, professional, or just interested in HPC, this webinar will provide valuable insights and tips to help you navigate your journey into the HPC field.

Webinar Synopsis:

  • How did Jonathon Anderson Get Started in HPC?

  • New CIQ Webinar and Podcast Set

  • Gregory Kurtzer's HPC, Linux, and Unix Background

  • Zane Hamilton's Experience with Linux and Beowulf

  • Why does Gregory Kurtzer Love HPC?

  • How did Gary Jung Enter the HPC Sphere?

  • Rose Stein's Beginnings in Linux, Command Line, and HPC

  • Where Did Forrest Burt Start His HPC Journey?

  • Why Don't University Students Use University HPC Systems?

  • What Got Brian Phan Started with HPC?

  • How John Hanks Got Started with Linux and HPC

  • Dave Godlove's HPC Background Through MATLAB

  • What Are the Various Types of HPC's?

  • HPC Versus HTC

  • How did Alan Sill Get Involved with HPC?

  • Do Younger People Have HPC and Linux Easier Than Before?

  • Will Scientists Ever Not Need Technical Computing Knowledge?

  • How will AI Affect HPC and Computing?

  • Glen Otero and Biological HPC

Speakers:

  • Zane Hamilton, Sr. Vice President - Sales, CIQ

  • Rose Stein, Solutions Engineer, CIQ

  • Gary Jung, HPC General Manager, UC Berkeley

  • Gregory Kurtzer, CEO, CIQ

  • Forrest Burt, High Performance Computing Systems Engineer, CIQ

  • Jonathon Anderson, Solutions Architect Manager, CIQ

  • Brian Phan, Solutions Architect, CIQ

  • Dave Godlove, Solutions Architect, CIQ

  • Alan Sill, Managing Director - High Performance Computing, TTU

  • Glen Otero, Director of Scientific Computing, CIQ

  • John Hanks, HPC Principal Engineer, Chan Zuckerberg Biohub

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Full Webinar Transcript:

Zane Hamilton:

Good morning, good afternoon, and good evening wherever you are. Thank you for joining us. My name is Zane Hamilton, and as you can see today, I'm actually joined live with Greg. So at CIQ, we're focused on powering the next generation of software infrastructure, leveraging the capability of cloud, hyperscale and HPC. From research to the enterprise, our customers rely on us for the ultimate Rocky Linux, Warewulf, and Apptainer support escalation. We provide deep development capabilities and solutions, all delivered in the collaborative spirit of open source. So this week we're actually going to have, how did I get involved in HPC? And we have a nice big crowd to join us, bring everyone in. We'll start saying hello. There we go. So I'm going to start in the top middle. Rose, you want to introduce yourself?

Rose Stein:

Have to unmute. There you are. Okay. You know what? Slowly but surely I'm learning the tricks of the trade here.

Zane Hamilton:

There we go.

Rose Stein:

Hey you guys. I'm Rose Stein. I started with CIQ in October. I am super excited to be here. And I guess that's it. I'm Rose

Zane Hamilton:

Gary, welcome back.

Gary Jung:

Thank you. Yeah. My name's Gary Jung. I manage the HPC, institutional HPC for Lawrence Berkeley International Laboratory. And I also run the HPC program for UC Berkeley.

Zane Hamilton:

Looks like Gary's in a different place today.

Gregory Kurtzer:

Gary is in a different place today. Yeah. Gary, what, where are you and what is the picture in the background? Yeah.

Gary Jung:

Oh, I'm with my in-laws down in Southern California.

Zane Hamilton:

Ah,

Gregory Kurtzer:

Are you having better weather than we are up in Reno, where we are still getting snow.

Gary Jung:

Oh, it's, it was raining on the way down last night. It's not. Great.

Zane Hamilton:

Welcome Gary. Forrest. Good to see you again.

Forrest Burt:

Yeah, thank you. Can everyone hear me?

Zane Hamilton:

We can.

Forrest Burt:

Good. All right. Good morning everyone. My name is Forrest Burt. I'm an HPC systems engineer here at CIQ. I've worked for CIQ for a little under two years now. And before that I worked as a SysAdmin in the HPC space in the academic National Lab sphere. So happy to be on the webinar.

Zane Hamilton:

Thank you, Mr. Anderson. You're in a different place too.

Jonathon Anderson:

I am. This is my new location, but we'll see how long it sticks. Ah, yeah. So my name's Jonathan Anderson and I've been an HPC systems person for a while now in the academic and higher ED space mostly. And some national labs, but yeah, with CIQ, systems engineer and solutions architect that kind of thing.

Gregory Kurtzer:

So, hold on, hold on, hold on, Jonathan. Hey. You're known throughout the HPC ecosystem.

Jonathon Anderson:

I almost, for this one, introduced myself as an HPC mascot and just was going to leave it there, but yeah.

Gregory Kurtzer:

But you gotta tell the mascot story.

Zane Hamilton:

Well hold off, let's wait. Let's get to Brian. Okay. We'll get everybody else going. Welcome, Brian.

Brian Phan:

Hey everyone. Brian Phan here. I'm a solutions architect at CIQ with backgrounds in HPC administration and architecture. I have a little bit of workflow experience in the genomics and automotive industries.

Zane Hamilton:

Thank you. Brian. John Griznog, welcome back.

John Hanks:

Hi John Hanks Griznog. I am an HPC principal engineer usually in life sciences, but I've dabbled in a little bit of everything HPC-ish over the years.

Zane Hamilton:

Thank you. Good to see you again, Dr. Godlove.

Dave Godlove:

Hey, I'm Dave Godlove. I'm one of the many CIQ solutions architects here in the meeting today. My background is in primary science, biomedical sciences, and I've been in HPC off and on, I guess for about six years or something like that.

Zane Hamilton:

Thank you, Dave. Dr. Sill, good to see you. Welcome.

Alan Sill:

Hey so yeah, Alan Sill, management director for the High Performance Computing Center at Texas Tech University, and a co-director of a multi university industry University Cooperative Research Center in Cloud and Autonomic Computing which we do a lot of work with data center automation and where I try to spend all my spare time not taken up by my day job.

Zane Hamilton:

Thank you, Alan. So, let's go back to the middle. Let's go back to Jonathan and hear the actual mascot story. How did you get involved in this, Jonathan?

Jonathon Anderson:

Well, I mean, that story is not quite the same as how I got involved in it, but maybe.

Zane Hamilton:

We know both. That's fine.

How did Jonathon Anderson Get Started in HPC? [09:51]

Jonathon Anderson:

So, I didn't know anything about HPC or what it was until my first job out of college. But a guy that was a year or two ahead of me in school for his senior project built a Beowulf cluster out of old IT equipment that he got from the IT department. So they had a terrible, the worst case scenario of a completely zero ventilation, closed closet with baker's racks, and 16, I think Dell optiplexes just jammed in there. And I just knew it existed, and he was always in that closet working on it. And either before that or around that, he had also done an internship with the summer undergraduate laboratory internship program of DOE Suli. And he hated it. And for some reason I thought that meant I should do it.

And so I did the same internship at Argonne National Lab for a summer and worked on some of their academic clusters there and had a great time and went back there for my first real job out of college. And so as part of that, I was with the mathematics and computer science department there first, just doing the existing institutional cluster stuff. But it's one of these memories that I don't understand in retrospect what happened, because I'm pretty sure I didn't have a job there, and I just drove up there and wandered onto campus and stayed until they gave me a job. I think this actually happened. But as part of that, they were hiring people for their new leadership computing facility where they were building a big IBM Blue Gene Computing system that was supposed to be a big, I was going to say first peta-flop.

That's not true. It was a half peta-flop system. And at the time was the number three system in the world, which was cool, but that meant there was press around it. And so now we'll get into the story that Greg wants me to tell. So there was press around it and they were taking glamor shots of the machine, and the photographers happened to be there on Veterans Day. And I, completely unrelated, was singing with a couple of guys in a choir there on campus and was dressed up for a Veteran's Day thing that we were doing, we were going to be singing later that day. But because I had a suit on and was a crazy new hire who would do such a thing, the photographer pointed at me and suggested that I should stand next to the computer and use it while they took pictures.

And so there's a picture of me with my hands inside the guts of a BGP rack that has become quite well distributed on the internet because someone, I've tried to figure out who did this, but, someone put it as the the first thumbnail on the supercomputer page on Wikipedia, which, because of the way Google works means that when you Google supercomputer, you get this picture of me standing next to a Blue Gene. And that means that anyone who wants to do any PR about supercomputing has a pretty strong likelihood of using this picture also, because it's completely open licensed so you can use it for anything without any fees or whatever. So I first started getting notifications of this picture because an Illinois Senator, I think like a state senator, was using it in a political ad. And that was kind of interesting . But the pinnacle of it for me is that when IBM Watson was competing on Jeopardy, IBM used it in an interstitial about supercomputing and such. So I like to tell people that I was in jeopardy as well.

Zane Hamilton:

So I've heard you tell the story about the, in the picture before, but did you know that Jonathan was a singer?

Gregory Kurtzer:

I'm not sure what I should say.

Zane Hamilton:

I feel like this is something we should have known already.

Jonathon Anderson:

I have to explain. It's not on my LinkedIn. So Maybe that's why no one would know.

Zane Hamilton:

But you've also never told me that before.

Gregory Kurtzer:

He's not just told you.

Zane Hamilton:

Okay.

Gregory Kurtzer:

He's told everybody in the world, wow. Who's going to watch this?

New CIQ Webinar and Podcast Set [14:20]

Zane Hamilton:

Everybody. That's fantastic. I love it. So, Greg, because we're sitting in a different place ourselves I'm going to let you go next, but where are we?

Gregory Kurtzer:

So we're at the CIQ office and this is the first time using this for doing a webinar or podcast, but we actually have a pretty nice setup. What you can't see is just like every new sound area and whatnot at the moment. There are wires everywhere that are outside of the camera view and it is not safe. Oh, there we go. There it is. There we go.

Zane Hamilton:

Look at that guy in the suit.

Gregory Kurtzer:

It's awesome.

Rose Stein:

This is the best. Thank you. Whoever found that and is viewing it right now.

Jonathon Anderson:

It's not difficult to find.

Rose Stein:

You should have gotten paid for this man. You should have gotten paid.

Gregory Kurtzer:

Jonathan, I was going to say the same thing. I can't believe you didn't get royalties.

Jonathon Anderson:

I think that they're not allowed to pay you for it when you're a government employee.

Gregory Kurtzer:

Did you, did you sign off on sharing your likeness.

Jonathon Anderson:

No, I don't think I did.

You know what, it's all worked out okay.

Zane Hamilton:

Don't see a lot.

Jonathon Anderson:

I Hope that the clothes are a bit better.

Gregory Kurtzer:

Well, how long ago was the picture?

Jonathon Anderson:

Oh, man. That would've been like 2008, 2009, something like that.

Gregory Kurtzer:

Oh, see, I was going to say if it was the eighties there's a video that the way the clothing fit is very similar to in the eighties. I think it was Rick Astley.

Jonathon Anderson:

Could be. Yeah. Maybe. I'm sorry to do this to you, Greg, but if it was in the eighties, I would be crawling next to the rack instead of standing next to it.

Rose Stein:

Cool. Also, did racks look like this in the eighties? Come on, Greg. Come on. You know that this is.

Gregory Kurtzer:

Oh man. Oh man. You all are harsh.

Zane Hamilton:

All right, so we are, we're in Reno.

Gregory Kurtzer:

We're, we're in Reno, we're in our office, the CIQ office. We have an audio room that we have now set up. We can do podcasts, we can do all sorts of cool stuff from here. But as I was saying, you can't see the ocean of wires that are on the other side of the camera. Because we haven't actually done any wire management yet, but on this side of the camera it's pretty neat. That's nice. We just need a sign or something behind us. Yeah, it says something cool. Maybe a neon sign.

Zane Hamilton:

And thank you for all the work behind that side of it too. Yes. Fantastic. People putting this together. Really appreciate it.

Gregory Kurtzer:

Yep, yep.

Zane Hamilton:

Go for it. Yeah.

Gregory Kurtzer:

Where are the guitars? Sorry.

Zane Hamilton:

So yeah, to Alan's point, it'd be great if you guys would tell us your story point, throw those up in the comments and let's hear how you guys got started. But I kind of looked over at Greg. How did you get started in HPC?

Gregory Kurtzer:

Oh, it's a very short story.

Gregory Kurtzer's HPC, Linux, and Unix Background [17:24]

Zane Hamilton:

Did you say Greg? With a short story?

Gregory Kurtzer:

Yeah, it doesn't happen. So I started off in biochemistry. I got my degree in biochemistry, my undergrad degree in biochemistry. And I decided I wanted to take a little break from school and I wanted to go work and do something different for a little bit. Cause I spent my whole life up until that point in school. So I wanted to go do something different for a little while. Now it's funny, so around the university, somebody near graduation, as we were approaching graduation time, there were a bunch of signs that said, new biotech startup looking for junior level scientists. Now anyone who's ever worked in biotech or like a startup would not be looking for junior level scientists. But my wife saw this and she wasn't my wife at the time, but she saw these flyers, she ran around the whole school and tore all of the flyers down and then gave me all of them.

So I was the only one who's calling. So I started calling up this startup, and the receptionist is like, we are not looking for junior level scientists, so no thank you for calling. And I'm like, okay, well let me send you my resume anyway. And just like Jonathan, right? I wasn't physically hanging out there, but I started calling back like every week, like every single Friday I would call back, oh, I've got a new change to my resume. Let me send you my new resume. And this lady was getting so tired of hearing from me. This went on for well over a month. She's getting tired of hearing from me. I'm getting nervous because here we are nearing graduation time and I really wanted to find a cool place to work and that she's getting tired of me.

So she gave my contact information to the CEO who responded to me and said, Greg thanks for reaching out. We're really not looking to hire a junior scientist, so please stop bugging us and whatnot. And I said, okay, okay, well thank you. I hung up and I was so excited because now I know the CEO's contact number and how to bug him so weekly I would bug him and to the point where he finally felt bad enough for me that I ended up with a job offer. But it wasn't to work at this new startup. He said, I'm creating another startup and I'd love for your help on that. You'd be the first employee. I'm like, okay, awesome. And so I joined this and I started, I literally converted his five car garage in Hillsboro into a genomics laboratory.

I mean, it was awesome. Like, I'm laying tiles, I'm cleaning cobwebs and whatnot, and turned it into a sterile lab. And at some point he is like, we need to do a lot of we need to do a bunch of genomic searches on the internet. And at the time, in the mid nineties, well 96-ish, he's like, the only place you can do this is over the internet. The NCBI is hosting a bunch of genomic supercomputers. And they're running Blast and other things, which wasn't even open source at the time. And he says we can't give novel sequences away over the internet for free. Because somebody can get that novel sequence. So we need to figure out how to build systems in our lab, in the garage and start doing this.

And so this was my first introduction to Linux andI I thought he was pronouncing Unix weird and because I never heard of it. And then he explained to me what it was. We had somebody come over and start doing some development and we got genomic searches running on-prem as fast as the government supercomputers using huge amounts of memory clustering systems together and whatnot. And this is the first time I saw any of this and I was just like, this is the coolest thing ever. Right? You can go download software off of our 56K, 28K modem, whatever it was at the time. I don't remember. Put it on a couple floppy disks and go install it on a deck alpha, just to put the timing on right.

With half a gigabyte of memory, which costs well over $10,000 at the time. Oh, I'm sure. And we built a little mini supercomputer out of that. But immediately, I'm like, this is so cool. I just became completely enamored with Linux and Open source. And so I started following that side and went to work for a startup called Linux Care in the city. But after that was like, what do you do with somebody with a science background and an interest in computers? You do high performance computing and you support scientists. I landed a job with Gary who's on this call at Berkeley Lab and I started off in the earth sciences division and they wanted to build some, not only did I have a bunch of just Unix support they wanted, but they also wanted to build some clusters. And so that's where I started and Warewulf came shortly after that in terms of how do I build clusters in a more sane way? And yeah, that was how I got into it and I told you a really short story. Yeah.

Zane Hamilton's Experience with Linux and Beowulf [22:57]

Zane Hamilton:

That's great. Fantastic. So I will go next just because I have an interesting one. I realized the other day that I've been on Linux for as long as I can remember, the last 25, 26 years since Linux was free and came out. But I didn't realize until we were talking to a university the other day. I had a friend who was getting his doctorate there and I actually helped him build a Beowulf cluster and it completely, I just never thought about it until we were talking to them and having that conversation. And once we got it built and working, I never heard from him until he had his PhD after that. But that was my first entrance. You built the Beowulf.

Gregory Kurtzer:

What year was this?

Zane Hamilton:

2002. 2001, 2002.

Gregory Kurtzer:

What did it do? What kind of research?

Zane Hamilton:

He was doing medical research. He was building a, what was he doing for that one? Something with Pacemakers. He was actually working on the heart and was crunching data for pacemakers.

Gregory Kurtzer:

Was there any form of interconnect? Back then it was probably MERACOM was the most common one. I just know the stuff he was struggling with were all drivers, so it was, it was fun. I remember working all day at my regular job. I get paid for helping him overnight. It was fun. That is so cool.

Zane Hamilton:

But I had not thought about that story until we were talking to those guys the other day.

Why does Gregory Kurtzer Love HPC? [24:12]

Gregory Kurtzer:

So, we were talking actually just yesterday as well about this with some other people that pivot for me going from science to now I'm actually supporting scientists. It is actually a very dramatic change and it ended up powering.

And I've heard Dave Godlove talk about this a little bit in the past. It ended up powering my why. Why am I doing everything that I'm doing? Why have I done this whole career path? And it really came down to, I mean, I was probably doing good things by doing science myself and so, but I could do much better by focusing on the operating system, focusing on the computational side, and actually supporting the scientists who are much smarter than me and let them actually focus on the science and then I'm able to support them. And that became a why for me, for my entire career so far. It's great. It's fun. I appreciate it. So since you mentioned Gary, we'll go to Gary next.

How did Gary Jung Enter the HPC Sphere? [25:20]

Gary Jung:

We have a shared beginning there and it's just fun listening to it. But when, when Greg showed up it was in around 2000, what was it, what year you started Greg?

Gregory Kurtzer:

Yeah, 2000.

Gary Jung:

Yeah. And most of the scientists at the time were using Big Iron, so they were using like, we had a national computing center at our site, and they were just, the outgoing system was a CRAY T3E and they had just got a IBMSP system. And it was an interesting time because the supercomputing guys looked at clusters as toys. Oh, those things are unbalanced. Great on processors, no interconnects, low storage and on and on. And so it was an opportunity for us because I was part of a working group where the scientists wanted to figure out how to do HPC on the cheap. And so I ran the Unix group, and at the end of the working group, which actually lasted for over a year, we were given a mission to build 10 clusters.

And so that's where Greg and I really kicked off the cluster computing at Berkeley Laboratory. And so those were fun times. And just because there weren't very many tools there. I remember him and I going to a Linux users group meeting in San Francisco at the Empress of China. And just to give you an idea how early this is, the speaker was this guy named Peter Brams, and he's talking about this intergalactic file system that he hoped to get funded, which later became Luster. So, that looks pretty cool, that file system. But those are early days. We had a lot of fun. I was with Greg when we thought of the idea and he did it, but the idea of doing CentOS came up in downtown Berkeley at Jupiter Restaurant in the back. We were like, this is ridiculous. How can these guys be charging for open source software? So lots of fun. Back then.

Zane Hamilton:

Thank you Gary. So I think Rose has a little bit of a different path into HPC. So Rose, tell us how you got involved in HPC.

Rose Stein's Beginnings in Linux, Command Line, and HPC [28:04]

Rose Stein:

Well, I was just living my life, minding my own business. And Greg called me up. I mean, the last time I saw a command line was in fifth grade when everybody got computers and we all sat at this giant computer and there was like a little green command line, and I think it was called DOS. Flash forward many years, never really thought about it. Greg calls me up and says, we're going to do some really cool things, Rose, I want you to be a part of it. Go learn Linux. I'm like, what? What is that? What are you even talking about? And so what I found out after jumping in and meeting all of you and working with all of you is, wow, this is just kind of like how it happens.

I think that there is some consistency in people's stories when they talk about how they got into HPC because it's like, I don't know, man, I was just living my life and then boom, and I went in that direction, all of a sudden I'm over here. And so that energy though, that mentality of jumping in with both feet, like, let's do something new. Let's do something cool, let's make things happen. Like, Greg, that was such a cool story that you just shared about, no I want this and this is what I'm going to have, and so I'm going to go in this direction and I'm going to bug whoever I need to bug to get where I need to go. I mean, it's interesting because there's confidence, but there's also just a passion and a desire to know and to learn and to understand and to be helpful.

And that is what I have found with you guys. And I was telling my brother this the other day, I'm like, man, in my mind, I flash back to myself in fifth grade sitting at that computer. If I'd known how amazing all of you are in the HPC world, I'd have told myself back then, follow this, go in this direction. I mean, I know I had my own path, but it is really cool and really inspiring to see all the incredible work that you guys are doing. And anything that I can do to help with that, I am so grateful to be here.

Zane Hamilton:

Fantastic. Rose, we, we appreciate you being here too. It Was a lot of fun. Thank you. Thank you. So I think we should go to the other part of the middle row and save the actual scientists at the end, not actual scientists. Nothing against you guys. I just know that they started their life as scientists. So Forrest, not computer science, science science.

Where Did Forrest Burt Start His HPC Journey? [30:40]

Forrest Burt:

Well, in my case, computer science.

Zane Hamilton:

I know that's what I'm saying, that middle row, like the computer science row.

Forrest Burt:

Yeah. Yeah, so my story of getting into HPC is intertwined with my story of getting into computing in general. Essentially I've always been interested in technology, computer stuff like that. I was born into the digital generation. So I was born into the tech and molded by it, you could say. My first experience with computing in general was in like 2011, 2012, 2013 or so. At the time you could run Minecraft servers, that type of thing, all in the cloud. And so I was running, for me and my friends, a cloud-based Minecraft server doing all this stuff. Managing backups of different images for different mod configurations, this type of thing. Resource configurations for the node it was running on, that type of stuff.

I got really interested in tech. And from there, I had a few other interesting formative experiences. I was really early on tinkering around with, I don't know if anyone remembers, like cleverbot, that AI thing from the early 2010s. I tinkered around with that. I had someone who introduced me to Bitcoin and Raspberry Pis and stuff like that around the same time. So I've been interested in computing for a long time. I tinkered with VMs in high school, all that thing. And in the providence of what I was doing in high school, which was being the manager of this portable trailer that had a bunch of printers. The standard printer tech entry and we'd haul this trailer out to like the middle of nowhere and we'd do all the printing for the Forest Service on wildland fire base camps.

So we print out like the information packets, the big huge maps on plotters, that type of thing. While I was out doing that, I got the chance to interact with a lot of their IT people and got to see the servers and stuff like that. The hardware, the setups that they would bring to these really remote sites to run the computers for the camp. That got me even more interested in the IT side of things. And somewhere around that same time, senior year of high school, I heard about the San Diego Supercomputing Center. Now I had no idea what a supercomputer was. I had no idea what the San Diego Supercomputing Center was, but I was like, oh, wow, that sounds fun. So I was starting and in college at that time, and I made it my goal, while I'm here, I want to investigate two things, working on supercomputers and being an orientation leader.

Shortly after I did the orientation leader stuff, I found out through a research project that was there, this was about two years into college, I'd kind of bounced around in majors a little bit. Ultimately it settled around this point on computer science. But as I mentioned, I've always had an interest in systems and stuff like that. So I was trying to explore different options that were not directly related to software development, but was just seeing what else you could do with CS. So around that time I ended up involved with a research project that company Micron, one of the big semiconductor manufacturers, and a research lab on campus were doing. It had to do with basically using DNA as computer memory was a part of this.

They asked me randomly one day, oh, do you want an account on the university's supercomputer? And I was like, the university supercomputer, what? I had no idea that universities had supercomputers up to that point. I just hadn't looked. I had the goal of being in supercomputing, but it hadn't quite found me yet. So I was like, oh, absolutely, sign me up. Yeah, I went on the supercomputer. So I started poking around there, seeing what I could do. About a week or so after I got my account on it, I was like, I was bored at my office assistant job that I was doing at the time. And I was like, okay, well there's a supercomputing center on campus. What can we do to get in on that?

So on a whim, I wrote up an email. I pinged them and said, Hey, I'm a student. I'm really interested in supercomputing. I'm curious if you have any positions for students available. And just like a couple minutes later, I got a ping back that said, oh, fantastic timing. Our prior student just left, or one of our prior students just left. Here's the job posting for it. And so recent to my email, had the job posting been put up, that it's still said, posted a few seconds ago on it. So just like the same time they posted this job was almost the exact same time that I pinged on me, do you have any positions for students? So they were very excited to send me the link to that.

I was a little apprehensive about applying for it because they wanted C experience, FORTRAN experience, they mentioned MPI and they're posting different stuff like this, and I'm like, I don't know anything about that. Someone encouraged me, called them up, and asked them exactly what level of experience they wanted there. So I called the office. I didn't reach them at the time, but I just left them a message saying, Hey, this is Forrest. We've communicated over email. I'll be on the level. I don't know a ton about these requirements. I'm interested in what level of on-the-job training you're willing to provide. The response back was, absolutely we're expecting to train someone up from nothing. I ended up interviewing, I parlayed the Linux systems class and stuff that I was taking the next semester and then I'm going to be learning about this stuff soon and end up getting the job.

So I was a student at HPC SysAdmin there on sites like R2 and Boise State for about two and a half years. Did all kinds of stuff with the International Lab, et cetera, et cetera. It was a blast for a long time. It was just me and the chief Sys Admin working on things. So I got to do a lot of stuff that you probably wouldn't expect the students Sys Admin to get to do normally. Eventually I got to see IL's data centers, all kinds of stuff like that. It was an absolute blast. I was interested in continuing with HPC after college. I applied around at a few different places. Got to the end of a couple interviewing processes. A few weeks, but nothing that hit.

Couple weeks after graduation, I got a LinkedIn recruiter Ping from Greg saying, Hey, I have a startup called CIQ, Control IQ at the time, that I'm building. We're looking to hire basically a research computing professional. I'd be interested in seeing if you would like to apply. So I put in, I went ahead and did a couple interviews with CIQ. Greg showed me through like Fuzzball and some of our systems like that, and I was instantly hooked. I was like, this is the same type of thing that I had researchers asking me for at Boise State. This is the exact type of cutting edge HPC tech that I want to be working on. So I was very thrilled to have that all happen. Shortly after I got my job at CIQ. And I've been here for, as I mentioned, a little under two years. So that's a little bit long-winded, but how I ended up here, HPC.

Why Don't University Students Use University HPC Systems? [37:53]

Zane Hamilton:

That's great for us. Thank you. Do you think that's a common problem, though? People don't know that universities have HPC environments? Because I certainly didn't know.

Gregory Kurtzer:

Yeah, actually Gary and I have heard this a number of times, I think from UC Berkeley, as well as some lab people. There was one point in which I was talking with some laboratory scientists who are running their Python code, their scientific code, their Python code on their Macs or whatever laptop it was. And they don't like even shutting it down when they go home. I met one person who leaves the MAC open so it doesn't go to sleep, runs to work and takes it home, and walks to the car with it open. Puts it on the seat next to them and then drives home and then plugs it back in before the battery dies. So the computation can continue. And when I met the scientist, I'm like we've got a HPC system, you could just run it on. And yeah, it was so yeah, a lot of scientists don't know. At Berkeley Lab, we spend a lot of time reaching out to people. I remember at some point we even had, in the cafeteria at the lab, tents, little tabletop tents, advertising that we have an HPC system at the lab.

Zane Hamilton:

Do you think part of that is people don't even know what HPC is? Or they didn't even know that they should be using it.

Gregory Kurtzer:

There's probably, there's probably a good chunk of that.

Zane Hamilton:

Yeah. Interesting. Thank you. Brian, how did you get into HPC?

What Got Brian Phan Started with HPC? [39:26]

Brian Phan:

So I got into HPC I guess after my junior year of college. So in my program I studied computer engineering. In between our junior and senior year, we had the option of taking 12 to 16 months off to do an internship. So at the time I thought I was going to be a software developer, and I did a bunch of those interviews and then I realized I really sucked at coding. So I found this one, and at the time I was studying computer engineering and minoring in bioengineering. So my ideal outcome would've been like something tech and healthcare related. I found this job on a job board for an HPC admin role at a children's hospital. At the time, I didn't know what HPC was, just ended up applying for it and got the interview and yeah, interviewed for that role and ended up getting that role.

And that's where I learned all of the basics of HPC and supporting like and in that specific role, I supported genomics and medical imaging labs. And then when I went back for my senior year, all my friends who were software developers, we took a distributed systems class. And my friends who had software developer roles during their internship, they needed my help on some of the assignments that we were working on. So that's when I realized like, oh, maybe I finally know something that the other kids don't know. And I've just run with that ever since. And yeah, and I've spent the last nine years of my career at HPC

Zane Hamilton:

Oh, that's really cool, Brian. Thank you. John, how did you get to HPC?

How John Hanks Got Started with Linux and HPC [41:13]

John Hanks:

It weirdly parallels what Greg went through. And even right now to the timeframe. I was a biology student. I finished my master's degree. By the end of my master's degree, I realized I really suck at wet lab work and field work, and I better find a different way to be a biologist. So I started working on a PhD in computational biology. About midway through that I got drafted to be the Sys Admin for the biology department I was in. And then later, while I was doing that, I got money from a faculty member to build a small cluster that she was going to run a PALP on. And I can't remember what PALP stands for, but it was one of the first, it was a phylogenetics application that they happened to release a Linux version for. And I convinced her that she could run it faster than doing the thing you guys just described of running it on the Mac and never shutting the Mac down.

so I built this little four node AMD cluster out of basically desktop systems. She was running PALP on it, but then a friend of mine was doing his graduate work. He had a model he was running for his graduate work and he was running it in the Sun Lab, the very expensive Sun Lab that another department had. And it was taking him forever to finish his computation. So he found out I had this little cluster of AMD machines and asked if he could run his model on it, and I let him run it. And so one afternoon I got him on the system, showed him how to login and run jobs and everything. We compiled this code, got it all working, it started, and he went home thinking nothing about it.

We showed up the next day, he showed up and said, okay, all that stuff finished last night. These things are like 10 times faster than the Sun machines I'm working on. Can I keep using them? And he went upstairs to tell the Sun people that he was going to move all of his work down to these 4 AMD machines. And the Sun Admin said, that's impossible, that those crappy AMD machines are faster than my Sun machines. And when he told me that story, I was immediately hooked because that played right into my anti-capitalist agenda that I've had my entire life. And I was going to take down Sun with cheap desktop machines building Beowulfs in the corner of the lab. And so yeah, that's what kicked it off everything since then.

Zane Hamilton:

Yeah, I've noticed Solaris Admins didn't like being told their stuff was slow. I've come across that before.

John Hanks:

It upsets them.

Gregory Kurtzer:

And you probably didn't realize it at the time, but the way to get rid of Sun was just to wait. Yeah. Wait.

John Hanks:

You can view it as you just need to wait. Or I can say I won.

Gregory Kurtzer:

Yeah.

Zane Hamilton:

My favorite is one of my Solaris Admins who used to always tell me that Linux was cute.

Gregory Kurtzer:

It's like, oh, that's cute. Oh, in Berkeley, you might imagine there were a number of free BSD developers. Who thought of Linux as the kid operating system and yeah, I think you said earlier like, isn't that cute? Like, yeah.

Zane Hamilton:

Oh, thank you John. Yep, Dr. Godlove.

Dave Godlove's HPC Background Through MATLAB [44:36]

Dave Godlove:

Yeah, he always cracks me up when you whip out the doctor and say, Dr. Godlove.

All right, well I think that I have a theme in my career that basically when I get good at something it's time for me to leave that career path and go do something that I'm bad at. And so that's a theme that has kind of been throughout my career path.

Gregory Kurtzer:

Dave now, what's going through my head is, as you get good at something at CIQ man, I gotta pivot you right away.

Dave Godlove:

Yeah, I still suck at my job right now, so I'm good for now.

Gregory Kurtzer:

I don't agree, but I will agree with the point of this conversation. Yes, Dave?

Dave Godlove:

Yeah, so I won't go all the way back to like, I've done a lot of different jobs, but I think that my path toward HPC begins as an undergrad. So as an undergrad. So I was, for reasons, really interested in neuroscience and the place I was going to school at Radford University didn't have a neuroscience major. So I was basically a psychology major with almost, but not quite a minor in biology. I was like putting stuff together to try to make my own neuroscience course. And I was working in a lab, on campus doing all kinds of wet lab stuff. I was doing a lot of pipe heading and histology and cell counting and staining stuff, and some animal behavioral work and all kinds of stuff like that.

Almost no computer science. And so then it was time for me to graduate and go to grad school. And so I applied and went to a few different places and everywhere that I went people were like, well do you know how to program? And I was like, I mean I'm a neuroscience person, why do I need to know how to program? And everybody was like you gotta be able to program to analyze data. And almost everybody used MATLAB. And so I dove in and took some crash course MATLAB stuff and figured out how to start programming in MATLAB and I figured out I had a knack for it and got pretty good at it. So in grad school I did a lot of programming in MATLAB.

Now MATLAB is kind of funny because it's this programming language in which you learn like all this programming and all this computer science and all this stuff, but it's all within the confines of this one thing, MATLAB, and most of the skills that you learn are not transferable to like, so you don't like learn MATLAB and then you know how to do Python or you know how to do like something else. Like you learn MATLAB and you know how to do MATLAB and that's it. The skills don't really transfer that well. So I went through my graduate school career rather than MATLAB. And then I got into a postdoc at the NIH. And up to this point, I hadn't really used high performance computing.

The lab that I was in, in grad school, I had some lab mates that were using HPC at Vanderbilt. But I always dogged them and sort of laughed that I was able to optimize my code to the point where I could run it on my desktop, whereas they had to use this supercomputer to get their work done instead. But when I got to the NIH, I actually started doing experiments where we might gather like 30 or 40 gigabytes of data within a single experiment within a single day. And it was my job to take all this data and do signal processing on it, to pull stuff out of it. And I just couldn't do that anymore on a desktop computer. So I started to use Biowulf while I was at the NIH which is the big intramural cluster there.

And it was really, really awesome. I loved it. The whole idea of using the super computer andI started doing stuff where it was like I would do the calculation. I would figure out if I do this on my machine here in my office, it's going to take me like a year and a half from start to finish to get all this stuff done, and I can do it overnight, on the cluster. It was amazing. I like showing all my lab mates who actually aren't using the cluster, and I like bringing them on and giving them some helper scripts and stuff and showing them how to do it. And just got really, really interested in it. It was my first experience with Linux.

I installed a Linux box at home. I installed a bunch of lab computers with Linux and started using it a lot. And from there, so I was a postdoc, and it was like, okay, what's my next career step? Either it's to become a PI and start running my own lab, or it's to become a staff scientist. And the PI thing, I didn't really like a whole lot the idea of, because to me that was like, I was going to be doing all the work that I didn't care for, like writing grants, writing papers, and directing other people to do the work that I actually thought was fun, which was writing code and collecting data and doing all that stuff. Staff scientists would've been a little bit better. But I was suffering from that problem of like, I was getting good at my job and if I just continued to go on as a staff scientist, I'd continue to be good at this job.

And why not change to something? I was not good at it , so I was looking for something else. And the NIH Biowulf cluster, they were hiring people and they had some positions posted. And one of those was for a staff scientist to sit in between the scientists who were using the computer and the supercomputer itself, and help people to use it. And it's just like Forrest, they had this job listed and they had all these requirements and stuff, and it was like, I don't know any of that stuff, you know? So I went to the interview and they're going through and I'm like, well, I mean, I have Linux installed on a box at home, and I mean, I administer it there. But I was like, but I know MATLAB. And they were like, oh, we don't have anybody on staff who knows MATLAB. And we get a lot of questions from people using MATLAB that we don't know how to answer. And so I became the MATLAB person on staff and one of my buddies from grad school actually joked with me. He was like, you found like the one MATLAB job that was out there.

And so from there, I became the MATLAB guy. I administered MATLAB for a while. I wrote some hacky programs to allow people to better use like the MATLAB compiler and wrote some stuff to better manage the licenses and just a bunch of stuff like that. And then this thing called Linux Containers came along and some of the scientists asked us to install singularity on the cluster. And it fell to me just because they were neuroscientists that wanted this software. And so I had the neuroscience background, I was talking with these folks, I got the issue, I started to investigate Singularity. And so then I became the MATLAB guy and the container guy. And that's pretty much how I got into it.

Zane Hamilton:

Yeah. And now your name is all over the documentation for Singularity and Apptainer. It's everywhere. Thank you, Dave. I can't read it.

What Are the Various Types of HPC's? [52:26]

Gregory Kurtzer:

What are the various types of HPCs we, depending on various technology and deployment models, am I missing? Where, what are they, sorry. It's again, nobody knows on this side we're looking at is actually far away , so we're both Yeah, I'm nearsighted.

Alan Sill:

Yeah, the answer is yes.

Gregory Kurtzer:

Yes, yes.

Alan Sill:

There are lots of types of HPCs. Yes. I'm not sure we can go into it. That's a good topic.

HPC Versus HTC [53:02]

Dave Godlove:

Well, one big difference I didn't appreciate until recently. I mean, I appreciated it, but I think the terminology I've recently heard is HPC, that is high performance computing, versus HTC, high throughput computing. And so that's a big way that you sort of bifurcate the two and, and split 'em up. Basically just like, does your application depend on running one application across multiple different nodes, or is it embarrassingly parallel where you're just, you got a thousand files that you have to run some analysis on to just do it on a thousand different computers.

Zane Hamilton:

Thanks Dave for feeling that one. Appreciate it.

Dave Godlove:

Yeah. Then of course you can differentiate further after that. But that's, I think, one big place where it bifurcates.

Zane Hamilton:

Good. Thank you, Alan. I saw you posting the gopher, and that made me laugh when you have Forrest talking about his foray into Supercomputing in the cloud. So that made me chuckle. Yeah. Tell us about your story. How did you get involved in HPC?

How did Alan Sill Get Involved with HPC? [54:14]

Alan Sill:

Well I, fortunately thank you, had opportunities in previous podcasts to talk about some of this. I guess the thing I want to say here is I probably had about a dozen different ways I got started in computing stories before I ever encountered anything like HPC. Probably my first scientific programming class was in 1974 folks. So as an 18 year old at Lewis and Clark College, where we would go into the basement of the oldest building on campus and use an IBM computer. That day's job was to run the payroll. And then at night we got to do scientific programming on it and a card reader and a pen plotter and stuff. And I could probably tell a dozen such stories, one involving a general Nova with 4K of far core memory that I found in the corner of a lab.

And that's probably the oldest computer that we've gotten a scientific publication out of. That one requires a beer, that story. But it probably ran on two different cluster environments, a VAX cluster and an Alpha cluster before one day. And I have told the story on this webinar series, encountering a friend who said, we built a Linux cluster. And I asked, why did you do that? And that was the first time I veered into this. So I had sort of gone through the tenure track career path here assistant professor, associate professor. All of these stories are unified by the fact that I was always just using the computer to do science. One science or another. It just happened to be that I could get that computer in science, the computer.

And when we finally did the stuff that I've talked about before where we turned these Beowulf clusters of Linux machines into a worldwide resource and used it to find the top core that probably started my path into full-time computing. I became a research professor. I was spending all my time on funded research, and the universities had a project with state funding and a deadline to demonstrate the value of grid computing for things of economic value to the state. And they didn't have anyone to run the project. They made me a very nice offer, and I realized I didn't have to write research grants to fund my summer salary anymore. So that's what finally tipped me over. And we ran those projects for a while, and then basically what happened was the guy who ran the high performance computing center here retired and his successor ended up having health problems, and I stepped into the breach.

So it was a very accidental path into HPC. And basically at that point, it was through the management track. So the interesting thing about this, and maybe inspiring for some of your listeners is that I never learned MPI programming or the classic things that you have to learn to become an HPC expert except as necessary to do their jobs. And it's still the case that, every time I turn around, I'm learning something new but fortunately or unfortunately, expertise in this field has a very short shelf life. If you're an expert in something three years later, that thing is going to be obsolete, right? So you might as well start learning now about the next thing that's coming and that goes back to the question that was asked earlier, what kinds of HPC are there? And my answer is yes. Well, I think that's really a valid answer. It's limited by your imagination and the evolution of technology, and it is still a really fun time to be doing this.

Zane Hamilton:

That's very true. Thank you, Alan.

Gregory Kurtzer:

And knowing that it's always changing means that Dave Godlove will always have things to learn. It's true. And will always keep him, we'll always keep him engaged.

Dave Godlove:

I'll just keep sucking. I'll just keep sucking at what I'm doing forever.

Gregory Kurtzer:

Happy. That's not at all what I'm saying.

Zane Hamilton:

John, you had something you wanted to add?

Do Younger People Have HPC and Linux Easier Than Before? [58:47]

John Hanks:

Yeah, I have a little lead up to a question. I was thinking, as Alan was talking about, on the previous of these, we've shared old man's stories or old admin stories, I guess. And I was thinking about all the painful things that I went through that have scarred me, that led me to a lot of the beliefs and views that I hold now. And just to pick one at random, my very first cluster I provisioned by sitting up one machine and then using DD and swapping drives out one at a time to DD that image onto the other drives to carry them to the machines. So I can look back and the reason I went through that, that and NFS route and any other horrible, horrible ideas for revisiting a machine, the reason I went through that was because I had to struggle because there wasn't any documentation. Like there was no place for me to go to find out how to do this right. You know? And so, what I'm wondering is, looking back at all the scars I've accumulated for the younger people in this, if they look at what they're doing today, what do you think is going to be a scar when you look back in 30 years?

Gregory Kurtzer:

Brian, Forrest?

Alan Sill:

Yeah. My advisor used to say that you spend all of your career checking the things that went wrong in your thesis experiment. Those scars persist.

Gregory Kurtzer:

So I think what you're asking is Brian and Forrest, what are we doing now to scar you?

Forrest Burt:

I don't have an answer for that one at the moment. Maybe in a few years since this is all settled.

Alan Sill:

You could turn that into a ringtone, Brian.

Brian Phan:

I mean, for me, I imagine a future where HPC workflows just work and there won't be back and forth with the users in just trying to get their workflows up and running. And I imagine users submit their job and it just runs. Yeah, I think so.

Will Scientists Ever Not Need Technical Computing Knowledge? [01:00:34]

Gregory Kurtzer:

Brian, Brian, you're onto what I was hoping you guys would say, which is the Beowulf, I mean, we've been using the Beowulf for the last, what, almost 30 years now, right? Is it possible we can do better? I think so. Is it possible that scientists don't need to know SSH how to compile their code, how to optimize and profile their code, copying and management of the data understanding how the scheduling queue works for every single system that they have access to and everyone being potentially different, which is usually the case running the job and then having to copy all the data off me. Couldn't we optimize that somehow? Probably. I think there's probably a good way to do that. Yeah. It's a little fuzzy though. I'm still figuring it out.

How will AI Affect HPC and Computing? [01:01:44]

Alan Sill:

I am a big skeptic and critic of the so-called AI technologies that we have, which are basically an exercise in predictive statistics, right? The most likely thing to come next is X and then a bunch of sentences of X get put together and people say, oh, it sounded amazing how much the computer has learned. No, it's just predicting patterns of text, based on other patterns of text that are seen. However, Greg, this is exactly relevant to the problem you described. We can, I think and should be teaching AI to predict the next right thing where it's possible for their next right thing to be predicted. This has actually been my design criteria for a software for decades. The computer should do the right thing by default, where it's possible to know ahead of time what the right thing is.

And so I actually think at the risk of losing my skeptic hat, that the chat GPT like technologies have applicability in systems programming in taking the rough edges out of these gotchas where the user experience is, I didn't type that obscure command. Exactly right. Well, we can teach the machine what you meant. So maybe I need to become more of a skeptic, but I actually think we can use these technologies to do API programming and stuff like that.

Zane Hamilton:

That's quite enough that way. That's pretty deep. Thank you, Alan.

John Hanks:

ChatGPT as a shell prompt would be fantastic. Some significant portion of my life has been spent backspacing over mistyped commands, which should have been trivial to figure out.

Gregory Kurtzer:

I actually think that there is a shell that is now based on, I can't remember what it is, but I've seen it. There is an AI powered shell.

Zane Hamilton:

Yeah, it was open source. And see you can go download it and put your credentials in.

Alan Sill:

I think this is not as crazy as it sounds. It probably will become valuable quicker than the rest of that crap that's out there.

Zane Hamilton:

Did he really put that up there?

Gregory Kurtzer:

Forrest? Okay. You, I, I wish we could share the internal thread so bad of what everybody says. So Forrest, I'm going to call you out.

Zane Hamilton:

No, thank goodness.

Gregory Kurtzer:

Because you didn't speak before when we, when we asked about it. Chat GPT plus Emacs, when?

Forrest Burt:

It's gotta be coming. If there's already an AI powershell. It's inevitable.

Zane Hamilton:

One's already posting it. Just sounds terrible.

Jonathon Anderson:

I just hope that the plugin was written by ChatGPT.

Zane Hamilton:

Do you think AI's going to be smart enough to just tell you you shouldn't be using Emacs?

Gregory Kurtzer:

Actually that's the next question I'm going to ask ChatGPT.

Pirates are Ninjas and VIM versus Emacs.

Zane Hamilton:

That's a good one. Yeah, that's a good one. So I know we also have Glen in here. Do we want to quickly?

Gregory Kurtzer:

Is Glen on?

Glen Otero and Biological HPC [01:05:01]

Glen Otero:

I don't know why my camera's not working, but yeah quickly. So Glen Otero, director of Scientific Computing at CIQ and I have a similar biological path to HPC. When I was a postdoc working on HIV molecular biology and thinking about my next steps, I was not really enthralled with academia anymore. So I struck out actually was interested in trying to build computational models of the immune system responses to things like HIV, work that Allen Persson at Los Alamo's had pioneered ended up working with Alan probably about 30 years later. But my next step was the San Diego Supercomputer Center, and it was downhill from there. So that's where I picked up Linux and I was actually first working on Sun machines there.

Got access to some of the clusters and supercomputers at SDSC. And then I started wanting to spread this information to bioinformaticians everywhere. So I went back and forth consulting and working for different companies, whether it was Double Twist or my own company, where we created like a bio brew, bioinformatician role for the Rocks provisioning tool back in the day. And worked for some others, worked for Dell as a life sciences expert when it came to HPC helping their customers pick the right parts for their cluster. I designed the first one stop cluster shop at Dell with the genomic solution there. And then I moved on to TGen as a scientific director there helping them bring new technologies like GPUs and compressed Technologies for Life sciences data. And then I moved to CIQ and here I am.

Zane Hamilton:

Thank you, Glen. And to answer your question, Rose, if it doesn't say over it's wrong. Are you okay? Just making sure. All right. Well, I think we're going to have to wrap up in about an hour. It's been fantastic. It's fun doing this in person for a change, and I really appreciate everybody joining us. I like this format.

Gregory Kurtzer:

I do too. It's fun. I do too. I think we should just bring you over to Reno every week.

Zane Hamilton:

Every week. It might not be feasible. It's fun to get people rolling through here.

Gregory Kurtzer:

Though. Yeah, absolutely. Love that. We're building a nice little podcast webinar room and we can completely Joe Rogan it. I like it.

Zane Hamilton:

Be awesome. That would be awesome.

Gregory Kurtzer:

Joe Rogan for HPC.

Zane Hamilton:

That's very cool. Well, thank you very much for joining the panel. Thank you for joining us and telling your stories. We really appreciate it. If you guys would like to subscribe, we would really appreciate it. So thank you very much. It's good to see you and see you again next week. Bye.