HPC Roundtable - Turnkey HPC Solutions

way over the years very cool thanks john and and for everybody i'll typically go back and call john grisnogg or john depending on if there's other john's since there is a jonathan you may hear me call him grace dog old habits die hard gary hi my name is gary jong i work at lawrence berkeley laboratory and i also have an appointment at uc berkeley i'm i lead the scientific computing group and we run the institutional hpc for both institutions um i've worked with greg for for many many years and he and i built the hpc programs at both institutions very cool thank you gary jonathan

hey my name is jonathan anderson and i've worked as an hpc assistant at a number of sites in the u.s national lab and broader academic space and today here i'm happy to be representing ciq as an hpc systems engineer and solutions architect thank you jonathan trevor hi yeah my name is trevor cooper i'm an hbc systems programmer um with the data enabled scientific computing group at san diego supercomputer center um i've been here um or working with with greg for a while now um in in the rocky project and i'm here to kind of lend lend my uh my input on on this panel excellent

thank you trevor forrest hey everyone my name is forest burt i'm a high performance computing systems engineer here at ciq i do a lot here at ciq but primarily kind of with our fuzzball system working on hpc 2.0 and that type of thing i come out of the academic sphere i graduated college about a year ago and while i was in school worked as the student sysadmin at the institution i was at and also a long kind of related that got to work a little bit in the national lab space with regards to high performance computing so very excited to be here and uh having

a blast at ciq so glad to be on the panel as well thank you forest and thank you everybody um you know we had a dry run yesterday where we all just kind of got together yesterday evening and talked about things and i wish i could have paused that dry run and just kind of superimposed it to what we're going to talk about here because there was so much information there that was just so fantastic that i i really wish it would have been live quite honestly so what that told me is we are going to have this group back again multiple times i think

and we may even make this more reoccurring that we have panel discussions like this because it was it was so cool to hear so we're going to try to replay some of that and focus on some different topics so everybody's fresh with it and have and bringing you know bringing their ideas you know like they didn't just say it the day before uh so we're gonna start off kind of talking about what problems do we need to solve for a traditional kind of sweet spot hpc system when i say sweet spot system a lot of us have experience with very large institutional uh level systems

but that's not the primary use case of most people out there in the community most people out there are going to buy a 50 to 100 node uh system from a vendor and they're going to be looking at how do we easily manage this this system in many cases these are uh post-docs these are grad students these are undergrad students we joked around a little bit sometimes these are these are you know the the most junior member of the team the newest person to come in or whoever lost the bet on who has to maintain the hpc system and install it and whatnot so it's

our job to kind of think about take let's take our experience and figure out what do we need to be thinking about and providing to make their job more successful and we have so much experience collective experience in this room uh virtual room how can we help with that so let's start with talking about some of the kind of just general purpose jobs that we all have experience with in high performance computing so let's talk a little bit about kind of those requirements so i'd like to hear from uh from the panel on what what types of jobs are we seeing i mean this historically

used to be a lot of mpi and tightly coupled jobs is that still the case or are we starting to move to a more broader diversity of jobs at this po this point and maybe less mpi what are people's thoughts on that in my arena it really depends on the the software vendor that we're relying on or even in some instances the the specific uh task that's trying to be accomplished because we we have i i've seen a lot of mpi and a lot of non-mpi jobs and a lot of jobs that are mpi jobs that are not explained by the vendor that their mpi

jobs which is always fun um because we use a lot of commercial tools in my industry and a lot of those uh have a lot of underpinnings which are not fully disclosed or explained and you kind of have to use a crystal ball to figure out what exactly they're trying to accomplish on the back end um and then uh profiling it and figuring out well we have to uh we have to really optimize for specific tools in those environments and so it's really a mixture um of of mpi based stuff and and stuff that that really wants to to grow to encompass all of one

physical machine um but you can sometimes lie to it and and hide that you're on one specific physical machine and grow beyond that um but um then you get into licensing issues which again i'm not sure everybody here has to deal with that but that's one specific thing in my arena that i have to deal with a lot is vendors tend to juggle the license requirements quite a bit to optimize their their cash gain and so that can really change what you have to provide the software um you know sometimes if you're if you're utilizing a gpu let's say uh you're going to pay a

more significant price if you're using multiple sockets you're going to pay a different price if you're using multiple cores that sort of thing so it's it's constantly a juggle to meet the needs and demands of um at least my customers [Music] what percentage of the applications that you're dealing with in eda patrick uh are kind of traditional mpi versus uh let's say serial or embarrassingly parallel um i'd say only about 25 percent actually utilize some form of mpi in the back end um and you know sometimes that's that's openmpi sometimes that's intel sometimes that's um you know just whatever implementation of mpi is required and

again it's it's not always disclosed that that's actually what's happening um a lot of vendors like to package that and then hide it as i said uh but from my experience it's about 25 use mpi and then others uh it's a it's a it's a it's really a mix outside of that um what what you end up with i don't quantify it really is that is that about a normal uh percentage in terms of what we're seeing in other scientific disciplines what about bio for example the graphic that trevor had yesterday pretty much sums up what what i see on my systems and what i've

seen throughout all my life science related clusters which was almost everything is single node and a lot of it single core yeah yeah i think i'd agree with that i you know we talked about this yesterday a little bit too and i just want to reiterate a big change that i've seen in the few years that i've been working in this area is the size or the number of resources on a per node basis that are available right so when i first came to sdsc our biggest system had i believe it was 16 uh cores per node and i think it had 32 or 64

gigs of ram i'm installing nodes in our system today that i have 128 cores and have 512 gigs of ram each and those aren't the large memory nodes like the large memory nodes have a tera have terabytes of ram and so on our old system we would actually like run software that would build virtual uh it was called smp virtual systems that had large memory that appeared to the users to have a lot of memory but you know nowadays we don't need to do that we can just deploy nodes in the cluster that have more cores than say 10 or 20 nodes had in our

old system like not even 10 years ago and so the resources on on the nodes now kind of eclipse what has been available before and so there isn't really a need to support something like mpi generally for the vast majority of our users at least um and so we have codes that actually bundle npi in inside under the hood and and users will bring those and they used to run on multi-node before but they can run those codes with mpi locally on the same machine now and so they don't change anything and it looks like it's like a 32 core 64 core job that used

to take four or eight nodes before now it all runs within a single node and they don't do anything they just run the application it's running mpi under the hood but just because that's the way that the developers coded it right it's a really interesting point me you know when when i first started getting into hpc there was not multi-core socket chips and as a result if you wanted to get up to a hundred processes 100 cores you had to have 50 nodes generally speaking you know dual socket systems it's a very different situation right now uh gary i know you run some pretty big

systems um at berkeley what are you seeing in terms and you support a lot of scientific disciplines what are you seeing in terms of kind of tightly coupled mpi versus uh per node systems or jobs yeah you know it's a really um interesting uh question and so i actually pulled some numbers last night and looked at it and so as most people would surmise um it really depends on the domain so we we see people like the climate people in material science science guys doing a lot still doing traditional mpi multi uh node jobs uh but you know at the lab all together at the lab in uc berkeley we manage about 80 000 cores and it goes across you know many disciplines but the average uh number of cores that uh a job would use uh i was surprised to find out that the the number is 15.

so it's they're not just single core jobs uh but you know it's kind of a mix um but but that number is kind of what i thought was interesting is that we're talking about fat nodes and uh you know we've always been buying fatter and fatter nodes um but maybe that's not what we need to do or that or to get better utilization we're going to have to move from scheduling from exclusive nodes to shared nodes um so but but that's kind of what we're seeing the large majority if you look at the job number by by numbers of jobs then then they're all like

single node jobs got it so maybe this is maybe this is a little bit discipline specific gary i heard you say that the climate scientists and whatnot they're still doing very kind of tightly coupled parallel mpi focus workloads uh and and i know we have we have some additional kind of realms here that we could be talking about but i'm gonna wager and say that it's really kind of maybe specific to the different scientific uh the different types of science that's going on uh within a system to the point where maybe that the system that we're talking about in terms of a turnkey system may

actually be a little bit uh dictated by the type of research that they're doing so let's expand this a little bit right we talked about mpi which kind of leads right into uh the interconnect and the interconnect fabric that you need right if you have tightly coupled mpi going multi-node you may need a higher performance interconnect fabric and if you don't need that the internet the inner um the fabric may not be quite as important but what about gpus are we seeing a trend still of increasing gpus yeah if if i would say um yes so we're looking at uh i think additional uh certainly

more instances of hybrid systems traditional mixed with gpus um mixed with scale out to alternate platforms such as extending jobs out to the cloud as needed and i was also going to bring up one thing because i don't have in my personal job i don't have a lot of visibility into the end user applications other than what i see through openhvc which is very mpi focused traditional hpc focused but with the other groups that i work with there's certainly a lot more folk focus on visual computing and hybrid uh traditional hpc and ai if you look at it you know you look at an organization

you don't have infinite money to go buy multiple systems that are tailored for each discipline you want to run and so you look to get a good hybrid system that can cover both and greg i totally agree in the you know traditional npi fabric is going to be you know extremely important but i will also say fabric is important even in any type of large data set applications where you're going to have to run interconnected storage over that same fabric in in my current environment infiniband is primarily a storage network um and it's we're rolling it out kind of aggressively at 100 and 200.

mainly for a storage network but also as a hedge for if and when the day comes which we really when we really are doing multi-node gpu jobs and we actually need that bandwidth so mystic knight is mentioning any insights for geosciences such as oil and gas imaging so it's funny um gary and i actually have quite a bit of experience working on uh those sorts of workflows uh at berkeley uh as a matter of fact some of the bigger systems and the early big systems that we created was specifically around oil and gas and imaging uh in terms of the insights that i have and

i'm going to ask gary to to jump in as well as well as anyone else jonathan you may have experience in this as well but typically what i've seen is for that sort of imaging they're going to typically break that into a lot of smaller uh three-dimensional pieces that they would basically run over mpi and typically those sorts of jobs actually scale extremely well over large systems using mpi uh gary jonathan any any additional comments or thoughts on that i i think a lot of the the geo codes work well on mpi we geophysics geochemistry those guys still do do that type of thing and

um a lot of the climate people have gone on to also use like national systems leadership class systems but we still see a lot of the other people in that domain using what we're calling the sweet spot hpc systems yeah my experience is that you know a domain like geoscience and oil and gas these are these are hpc bread and butter applications these are the applications that grew up and co-evolved with hpc as a medium and what we're seeing with the prevalence of of these single core single node non-mpi workloads are really from a new entire set another a new community of of users who

are are more likely to have grown up in cloud resources where things like parallel computing wasn't as much that's just not how their applications were developed they were more developed around the the cloud paradigms and so we're seeing that get re-ingested into what we've thought traditionally of as the the beowulf cluster style hbc work uh of you know the fabric with a low latency fabric and an mpi and i think that's that's a big part of where the the impedance mismatch comes from their applications that evolved in different environments that is a fantastic point um and and speaking of the fabric uh again mystic knight

great questions um predictions of cxl and i would actually even expand that to base basic just pci switching and leveraging the pci fabric directly and running message passing over that fabric instead of using something to come down further further down the stack with with an infiniband interface could we just do pci directly and and do that messaging i've seen a little bit of this personally um i'm curious if anyone else in the room has has seen some uh you know some uptake of this at this point and maybe see some some cool ideas coming out of this space i've seen some examples i've never seen

anything in the wild but uh the the you know the the science projects i've seen with it are pretty neat um but they're just that they're science projects because they're not practical due to cost generally i mean the cost of something like that at this time is so astronomical that i mean you're really only going to see it in my in my opinion you're only going to see it in in some of the the show pieces that are out there um you're not going to see that boiled down just yet it's not it's not quite ready for prime time and it's a there's not an

economy of scale to bring the cost down to where it's feasible to to implement at this time but what i am seeing is um and i've seen quite a bit of this recently um is push back on traditional speeds of interconnect um so like there's a huge push to get away from um 20 and 40 gig links to go to to 25 hundreds and 400 gig links um and that's it's coming in different directions it's mostly coming from hardware vendors that are sorry i don't want to don't want to name names um coming from from certain hardware vendors and it's it's causing problems because the

switch vendors want to go a different direction um and they want to stay towards the the 40 gig and the 80s and the 160s and and that sort of thing and it's like there's competing technologies at play and i don't know if it's due to supply chain shortages or what but it is causing havoc uh with getting nodes built out that i'm experiencing so i'm hoping cxl helps with a lot of that by creating a standard interface there but uh yeah really good points and regarding scale i think that having smaller cxl or pci switching fabrics and then clustering those smaller pieces those smaller fabrics

may actually help kind of achieve an economy of scale in a different way than just having a very large single flat network like what we see typically with infiniband or ethernet so i think there may be ways to to help manage that but what i'm really hearing is that you know there may not be one size that fits all as a matter of fact i kind of even pushed back on that notion maybe this is going to be a little bit more about you know if we're thinking about some sort of turnkey hpc system it needs to be fairly flexible about what needs to be

part of that and um so let's start moving into that direction like what is needed and jeremy your experience sorry was there another thought well i was just going to say along those same lines i mean if we're thinking about um sweet spot sized as you said you know 50 to 100 nodes i mean we're talking a lot about um npi and what sorts of workloads use mpi and things of that nature um but really i mean uh a lot of the the geosciences the the the types of applications that we're talking about the material sciences and things um those are applications i would contend

that probably aren't going to run very often on the sweet spot kind of uh you know size machines that we're talking about when we're talking about those we're talking about you know individual labs or maybe you know small groups of labs that are buying this hardware and you know acquiring it running it themselves um you know they're not going to be you know big oil and gas is not going to be running uh workflows on on these these size machines right so that's a that's a really interesting point and i'd be really curious on uh jeremy and forrest take uh on that um where are

you seeing like in terms so jeremy i know openhpc focuses a lot on kind of the turnkey you know distribution of this or what kind of use case are you seeing are you are you still seeing a lot of mpi uses of course i mean and i mean the mpi the mpi applications are out there and people are using them like i said it's a you know in certain in terms of what i'm seeing in terms of what i'm supporting um at intel is more of the move into the hybridization of like i said it's it's adding additional components like gpus uh as well as

being able to support uh you know different workloads like a you know ai and machine learning is a major um is a major move you have you have a lot of systems available um so you're looking for these uh being able to at least dual purposes between mpi and and other you know running pi torture uh other jobs but um you know there's a lot of background discussion going on right now about ethernet because you know in terms of turnkey in terms of what i still think most people want is is ethernet it's there it works and for most jobs it's fast enough especially if

you're talking about coarse grains applications they're not going to make use of high-speed fabric you know that's that's more money put into faster systems instead of higher speed fabrics yeah for a system this size that we're talking about like this sweet spot ethernet is has evolved to such an extent where you know you can put that in and if you don't have any cross node parallelism great you just have a network but you know a couple steps later let's forget about supply chain for the moment just pretend we can buy what we need um you know you can get a rocky compatible switch and you

get low latency too and maybe it's not as good as the the best infiniband right now but it's a pretty good middle ground for this size of cluster and doing pixie booting over ethernet is much easier than doing it over infiniband and that's where you come in where sometimes the best approach is multiple layers where you have your command and control layer you've got your your your general interface layer and then you have your your your background disk layer necessarily of networking um you know sometimes you want to segment that out there's not the whole concept of this uh i hate to use this buzzword

because it's painful to me as always hyper converged um where you know that that that's kind of it doesn't work sometimes when you've got narrow pipes so being able to break it out into multiple segments is sometimes very useful where you have you know it adds to it administrative overhead potentially but with the commonality of ether is so common and so inexpensive for the most part these days well you've got you know four four four parts minimum per node in most instances well why not you know lacp two of those off for a management side and interface side and two of those off for storage

um you know or something along those lines and you can come up with a fairly elegant solution where you you get it to work without too much management overhead so that's a really good point as we're starting to think about again kind of you know i know it's not a one-size-fits-all but if we were to kind of think of it like that maybe what we're starting to lean towards is simpler fabric maybe more robust more capable nodes is just the evolution of nodes is growing um it sounds like gpus is still going to be kind of a major aspect of this we're going to need

to be able to support gpus on some percentage of those resource nodes may not be all um but now i'm actually kind of curious one other area is oh and we do have a question a question from insane duck and jeremy i think that this is going to be aimed at you yeah what's the state of icr well yeah it's good because uh i've i actually uh started with the icr program quite a ways back um so intel cluster ready is not a program at intel anymore however it was replaced with the hpc platform specification which encompasses i would say much of what uh intel

cluster ready was trying to uh accomplish which was a standardization of a um base platform for hpc to make to provide a a lot of what we're looking for in a turnkey system uh in terms of how it's built and uh the applications i said the software and middle layers that are installed on it um the other thing that um came out of the intel cluster ready was intel cluster checker and that is uh still very much a product um and again when you get into turnkey systems having a tool that allows you to rapidly check that it's working correctly is is definitely helpful and

of course the benefit now is all of the intel tools you can get on directly from online repos so um i wanted to can i just jump in here for a minute um oh if um i wanted to jump back to that thing about the uh simpler fabrics is would this be a good time to do that because i want to put in a i we're seeing researchers um where the sweet spot system that they have is uh one of several so they typically have a lot of people have a portfolio of computing resources that they can access and so the uh the sweet spot

system that we're building is noddling for their use uh but you know the some of the one of the other things that they use it for is for on-ramp purposes to other systems in their company or uh you know national systems if you're in academics uh world um so being able to on-ramp your application onto other places or even the cloud then perhaps you want that fast fabric so that you have a similar environment we even get people who off-ramp on some of these because the the high-speed systems maybe have architectures that are no longer suitable for their codes that's a very good point um

one thing i wanted to come back to is we're talking about applications and use cases i'm kind of curious uh you know i have a i have a really big um perspective and and lots of opinions around containerization as you you might you might all know uh what percentage of users and users applications are we now seeing that uh supports containers uh if we were to just take a guess and just say you know approximately you know 50 percent of the workloads are on on containers or 20 or 80 percent i'm just kind of curious i'm going to jump through the room real quick and

just tell me in your experience if you were to wager a guest kind of what that would be uh trevor let's start with you yeah i can say for us i mean ucsd and sdsc was one of the first hpc national systems to put a containerized solution for users into the general space for them to use and so we've got kind of a long history of supporting um initially singularity and now other other solutions as well and what i'll say is it our users are are typically of two flavors one um they're an investigator doesn't have a huge amount of resources to do building of

containers or computer software development support or they're they're users that we call users but they're really kind of gateway service providers and those users typically do have full-time staff that can work on these things and so what we've been seeing is a number of the the gate the gateway type user cases are deploying their software stacks with containerized solutions and so some of our biggest users are actually gateways that support hundreds or thousands of other users outside of our system we don't count them directly on our machine but they get counted in the big pool and so you know we've seen cases where um they're

doing that um so that they're more able to move their software stacks between systems so they may have allocations at multiple national systems um this is all grant funded stuff and and also they're starting to branch out in and kind of working on a pay for use model where they're having their gateway customers come in and provide funds and they're able to run either on our systems or in the cloud as well and spin up resources on behalf of those users and so you know those kinds of cases are growing it's not typical for i'll call them small time users or long tail users to

be doing that because they don't generally have access to systems where they can build their containers or have the expertise to do that our user services people build containers for them and provide them on our system so i think it's on the order of like 20 to 30 percent of our users are running containerized on our big systems now got it um forest what's your take on this um while i was at my previous institution what i saw we were a very small university um like i said we were associated with a national lab and a major um kind of energy company for a state

um so we had a few kind of outside institutional users that were also kind of interacting with our clusters but what i by and large saw was that we were basically doing standard module file type management of applications um while i was there when i kind of started around 2019 or so we were starting to look into solutions like singularity spac that type of thing and that was where we were kind of starting to make our first forays into like actually using containers to deploy stuff out for users so i would say that for the most part what i saw was just traditional module file

type stuff it's probably on the order of maybe five percent of the users that have containers and stuff it was mostly um things that i was building out that were just kind of you know custom environments for certain researchers that would be difficult otherwise to represent within a module file system so yeah in my experience at least from kind of a small institution uh it's mostly still on the module file system i would say it's uh zero to five percent of people that are uh deploying containers regularly in that type of environment um as far as users go all right got it thank you um

just to kind of run through the rest of the group here real quick uh i'm gonna ask you just just to spit out the per the percentage if it that you would that you would guess on so jonathan uh percentage of containerization that you've seen on systems and in terms of actual running workloads i don't know what what i started seeing though was when new workloads came in and when uh when you know my last academic appointment when new requests came in for help getting software set up our support guys had really turned the corner on instead of trying to fold all of those dependencies

into a central module stack it was just way easier uh to fold it into it and build it up as a container and hand that to them and so i'd say well over half maybe even 90 percent of the workloads that people were bringing in if it was more than just make install in their home directory they were putting it in a container got it very cool gary what about berkeley you know i don't know the percentage of users um but we see a lot down at uc berkeley for a lot of people making the transition life sciences off their laptops um and then we're

also seeing it on the other side um you know we're just recently started hosting the computing for the joint genome institute and um they're containerizing their work also as uh trevor mentioned so they can make use of multiple um resources and so it's as almost like a type of high throughput computing the way they're using it so that takes up that actually uh uses up a lot of computing resources now they kind of have like a whole automated thing going got it very cool um grizznog i'd say less than 10 or 15 and probably the vast majority of that containerization driven by container being the

best way to install a specific application or the application being delivered as a container got it uh dave yeah you know i mean as you know the nih was also an early adopter of containers i would say as far as um so i think it's it's good to uh separate out uh container users between like consumers versus producers and so um i would say maybe 30 percent of users or more um are either creating their own containers or downloading and running containers that are pre-built uh from you know docker hub or some other registry and then the other thing once again as as others have

mentioned is that um staff have really uh you know at the nih at the nih staff supports around a thousand and growing applications um and they've really started to install uh applications primarily as containers and and to do so in such a way that that users and users don't even realize they're using containers and they use these applications so that's that's become like i don't even know maybe maybe 70 80 maybe even more of new applications which are being installed on the system by staff members are being installed as containers within l mod underneath of l mod and you know you know still as part

of the module system got it so it sounds like a similar use case to what jonathan was saying as well um patrick how is the adoption of containers in the eda space just percentage-wise percentage-wise there's only two major vendors which are adopting containers aggressively um and that's only because they've been really forced to by the industry um the the the eda industry in general really wants containers because the overlapping needs requirements are so uh painful with the software um so it's it's hard to give a an accurate percentage simply because it's really it can depend project to project um chip to chip what you're going

to be pulling in you know process to process what you're going to be pulling in for a software vendor perspective so that can change but i can say from administration and architectural side there's a huge push towards containers again for that same reason of um eda traditionally has to keep ancient unix systems running for years and years and years well past their expiration date and that's simply be to support or to have a fully supported from the vendor perspective environment and if you can do that through the usage of a container then that's amazing because then your your your actual compute environment can actually you

know proceed and move forward with performance enhancements and and security fixes and things like that and then you know you have the the the container to to hold the the antiquated os uh inside of it um and and that works great the only the only problem is is a container to container communication can sometimes uh be a problem um which i'm sure we've all kind of experienced to some extent when you've got a product from one uh needing to be consumed by another uh a piece of software and it's how do you have a container that calls another container and then feed back to that

first container especially when you have an interactive ui inside that first container and the only way to truly do that well is through uh you know either using a job scheduler and and network pipes or something along those lines or or stacking containers on top of each other with with singularity stacking i mean there's there's some weird ways to get around that sorry not to delve too far into the the nitty-gritty of the problems with with adopting containers but i mean that is kind of what it comes down to is that from uh from architectural administrative perspective it's a difficult thing to adopt when you

don't control the entire code base um and that's again with with homegrown applications it i think it's uh fairly easy and uh highly recommended to adopt containerization um simply be you know for for repeatability and reusability and infrastructure as code you know needs but then when you have to deal with vendor software that's when things get interesting because then you have to really push those vendors to make things work right inside of a container and that that's a kind of a mixed bag depending upon the vendor once the industry fully pushes really hard towards that i think we'll see a change kind of come over

the software vendors and over the last i want to say two to three years that's really started to happen in the eda space um some of the really big names are starting to wake up and realize that oh we better really kind of start doing this container thing because it also enables hybrid cloud on-premises off-premises when you've got containers that work regardless of what the underlying operating system is or has that's when things you know become a lot easier when you're doing that um so sorry not to delve again too deep into that but no i think you nailed it a really important point which

is so much of this is dictated by software vendors and i'm hoping over time that they see that containerization is a very good mechanism for transporting their applications around and and then supporting reliable um all of the interfaces that they need underneath it so um it can be used as a solution to kind of enable bitrot in a matter of speaking but in a good way jeremy what are you seeing on the openhpc side well openhpc's had i think container availability for a while in terms of putting it before its its software stack um as well as intel's made containers available for a number of

applications as well as its tools and if i was to take a wild guess at a number i would say probably between 10 and 20 percent right now of containerized application um but i will say i'm also supporting a team that's that's building a large werewolf four open hpc cluster and it will be 100 containerized and that is the way they keep the compute nodes relatively clean that they expect people to come in with a containerized application and run it and then clean up after themselves got it so i was i was hoping that containerization was going to be the magic pill um to really

help enable turnkey hpc but it seems like we still have a little ways to go before we're going to get there so we're still kind of still going to be thinking about locally installed applications and and how are we going to manage that tools like spac and easy build merging with things like environment modules and l mod i think are going to be really still kind of critical uh tools that we're going to need to be leveraging so i'm going to close and and i know that we're we still got like another 15 minutes ish but i'm gonna close with oh and there's a great

question by the way so before i get to closing i think we may even have a few more questions um this kind of touches on uh patrick i think it was you who kind of mentioned um going hybridization with containers from on-prem to cloud and so this is an example of it's kind of a specific question regarding efa on aws i'm not incredibly familiar with with use cases running on efa specifically in aws and i don't know if anyone else is more familiar than i am uh i'm gonna call out forest because i think you've been working on some of this but if not tell

me to be quiet and pass the past the mic um so yeah for anyone not a uh familiar the elastic fabric adapter is something that aws provides to give rdma capability to their compute nodes and kind of doing operations over there um you can get fast networking but by my understanding the efa adds the remote direct memory access component of it and leads to like a lot of the performance gains that you get expecting that you expect out of it um it's not like infiniband under the hood it's a little bit of a different setup that amazon has put together but by my understanding it's

still presents to a system like it is based on kind of an infiniband interface like with verbs and stuff like that um and so my initial i would this is still something that i myself am kind of like i said working on here i would say tentatively that most applications that can utilize an infiniband interconnect would be able to use the efa because on a programmatic level i believe the way they're accessing that is very similar and what i've seen is like for example um i'm working with lamps at the moment um all it took to get lamps up and running within a container in

an environment on aws with efa was basically just installing the um efa into there and then building against uh you're building against that and i've done something similar with gromacs as well um and so yeah there's uh the bigger consideration i think with regards to this with containers um yeah is making sure that this is the network stack that's accounted for there like making sure that you have this cloud version with the efa installed in it um and this kind of touches on something that i've started to find with existing container registries out there there are a lot of existing containers that are made for

hpc but these containers are built with basically just infiniband stacks in them and so there actually becomes modification that you have to do in order to get the efa running inside of some of these pre-built containers that are available um and in a lot of cases because they've compiled uh their applications against like the open mpi and stuff like that that uh you can install with the um sorry a lot of times because they've you know they've already built their applications against a different version of openmpi for example than the elastic fabric adapter will want to install and build against um a lot of the

times like i said it becomes necessary to either modify these containers or more reliably just rebuild the applications from the ground up uh in kind of a configuration that um expects that they're gonna end up being run in the cloud with the efa and stuff so there's a lot of considerations there um but yeah in general i would say most applications can support it it's just a matter of installing them in such a way that that's what they're going to be expecting to use as their interconnect and kind of network stack does ofed support efa i'm not sure actually okay i'd have to look into

that a little bit more all right uh and another question uh thoughts on on-prem versus cloud hpc solutions um there's two questions actually the first one and then the second one will there be a point where it'll be cheaper to do hpc in the cloud versus on-prem um i'm gonna actually i want to take a stab at the second one and then i'll open it up to anyone else who wants to jump in uh at berkeley in my role at berkeley and it's funny with my previous manager sitting sitting here on on the panel with me but upper management uh i would say going back

about 15 years ago we started seeing a lot of requests from upper management to figure out what is our cloud strategy specifically for high performance computing and this has come up a number of times over the over the years and i've heard many people kind of describing this uh that this continues to come up there are times in which it makes sense to do hpc in the cloud versus on-prem and typically in my experience on the models that i've seen and run it mostly has to do with when you're not running uh dedicated resources if you're running dedicated resources it makes more sense to do

that on-prem but if you're doing bursting or uh inconsistent high-performance computing use cases in many cases that makes more sense to do up in the cloud because then you don't have your your equipment purchases and and management of that equipment that you have to maintain so you can you can run your hpc for for one month and then for the next 11 months you can you you know you're not paying anything uh so that's my my take of it um are there any other points i mean does anyone else have a of a counter opinion to that so i don't have a counter opinion but

i can say we we had a a similar kind of top-down mandate to investigate um uh whether we should be doing hpc in the cloud and we did kind of the the financial modeling for it and you could kind of make an argument for it with compute but it was always storage that just made it completely untenable the minute you started talking about having as much storage as we had available on-prem for long-term capacity as well and in particular scratch uh it just didn't make sense um what i like to say is that cloud is really good not because it scales up there are reasons

for that especially in business use cases but it's it's good in this realm because of how it scales down and you get the full benefit of a crazy production environment even if you're only running a single core job and you didn't have to deploy any of the support infrastructure for that so exactly like you said burstiness is is a great use case for it but the minute you start talking about high utilization uh even on compute it it's always going to cost less to deploy locally whether you have the staff for it and the the floor space for it and everything that goes into it

it's a different question and i touched on the first part of that question too which is uh the data in many cases the data is going to be the limiting factor as to where you're going to run you're going to follow the data uh grizznog i think i i accidentally cut you off there please i have a i have a long list of biases against cloud but i'll pick my absolute favorite one and i think the one that gets the most overlooked and that is every dollar spent in the cloud is a lost opportunity cost to have something that you have locally that you can

use later um and that that ties into um the thought process behind the cloud so in our in our environment specifically and i'm involved in a big migrate back from the cloud right now that's super painful um the biohub pays and does a good job recruiting the best scientist possible to do research i'm involved in a big data migration back where i realized after listing all the files that we have in amazon i had just spent probably a thousand dollars just listing files and we have a lot of data up there and since then everything i do no matter what i do in this migration

move a file bring a file back from glacier everything i do has this back of the thought back of my head context of how much money did i just spend how much money did i just spend when you're trying to get the best researchers possible to do the best science possible you don't want them devoting their brain power to wondering how much money they just spent you want them focusing on the science and the minute you move into the cloud because of the cost model whether it was cheaper or less you know doesn't matter cheaper more expensive whatever you've now made that a a thought

process that everybody who uses your resource has to go through it's it's why i hate cost recovery in general for scientific clusters for clustered support research some of it's necessary i get that but just in general using the cloud means that everything is done in the context of how much money did i just spend can not should i do this experiment can i afford to do this experiment and spending that money on premise you put the hardware in the closet corner of the lab whatever you can run on it for the next 10 years and it'll still do what it what it did when you

bought it and you don't have to ask yourself how much did i just spend on that experiment when you run it it's an interesting point looking at it from the uh from from the scientists perspective in terms of managing costs and gary um i didn't mean it in a bad way at all gary was the best boss to work for that's fine it's fine but i definitely never you never you never forced a cloud mandate on us but we we definitely had to deal with this together yeah i just wanted to chime in that you know i uh you know because it's been a while

since we've done that cost analysis i had updated about six months ago and uh you know we have 20 years with it with the data center so i know all the investments chillers you know cost of the maintenance people you know all the hardware that's gone in there and so you know just costing our workload in the cloud but you know i was going to give it to you in round numbers you know including academic or you know uh institutional discounts you know it's it's roughly about three times as much to run in a cloud and if you you know then we took you know

we modeled it you know taking into account things like uh reserved instances and every kind of trick you can do to get the discounts by committing more uh and then then you can get down to maybe about two times as much and and then if you really really you could there's a paper out there fermilab people look for it they you know they did like running you know when the spot prices dropped low enough and you could see that the best that they could do with even preemptable jobs you know was like one and a half times so you know so but that just kind

of reconfirms what most people on this know um but it's not to say that we don't use the cloud because our use of gcp has tripled in the last two years um so there's a lot of tools that people a lot of easy to use tools which attract people and it works well for for those things i'm not against cloud is all i doing is just stating the cost so we oh great question from brent i actually haven't tried to run gaussian in a container yet uh as anyone else okay brent we're gonna have to get back to you on that one i can't see

any reason why it wouldn't run in the container um my only concern would be is if it's licensed to run in a container there you go um one thing i wanted to make a point of with regards to to cloud push for hpc is and and i think uh grizznock really kind of hit that point home about data movement things like that that's where and and and others also brought up some really good points regarding this i think that's where a hybridization really matters because being able to run your workload when you've got available hardware and available instances and available licenses locally is great but

it's those burst instances when you really need oh well shoot i need an extra 10 000 cores well what's the infra cost on that what's the supply chain on that what's the how long will it take me to get that how much power needs do i have how much cooling how much um do i have racks open do i have you know all of the logistics to getting that additional 10 000 course well when those additional 10 000 cores aren't gonna be needed three months from now that's when it really gets makes sense for cloud usage is for that burst need and then you if

you have the facilities for hybridization set up whether it be block level or on-demand block level data transfer between on-prem and off-prem um that's really that's really key to kind of fixing i think the the data movement issue uh you know essentially local scratch and then output is then uh block level transfer back to your environment uh on you know essentially on demand um that's where that really takes uh kind of makes cloud work for you in an hpc sense rather than um kind of against you is that you know you don't use it for your day-to-day jobs you scale your on-prem capacity to meet

your day-to-day jobs but then you use cloud to meet your burst needs your burst functionality because i think all of us probably have had that um you know instance where uh developer in my instance in my case it's a designer in in many of your cases uh i'm sure it's a scientist who says well have this thing and it's going to take this amount of resources but it's really important and i've got to get it next week and you know how do you get that how do you take care of that well not bursting your budget for the year on hardware and then having hardware

sit idle um and that's to me that's the worst is when you've got beautiful infra that you've built that's just amazing and then it's sitting there idle uh or licenses that you're paying millions of dollars a year for that your designers or developers aren't utilizing those licenses but you're indebted to and contractually obligated to those vendors to hold those licenses and if you give them up getting those licenses back on the next next fiscal year is so painful um you know that's when burst matters a lot but then the the downside is is that how do you get vendors to license that intelligently to you

so that you can do that you know without breaking your bank because again yeah those additional ten thousand cores you can easily spin them up in the cloud but then how do you pay for the licenses to run the software on those ten thousand uh when you're on a six month or a twelve month license cycle with a with a software vendor you know that's that's that's that's that's again one of the other um things to bear in mind and is how do you get the vendors to work with you on on burst capacities but great points um comment from stanford uh yeah so having

a researcher say i'm too scared to ask a question about my data because it'll cost too much uh grizznog this is a hundred percent exactly i think what your point was right there and um yeah i completely agree not a great way to do research and to encourage research but the patrick's point and gary's point i think there are good ways that we can be thinking about how do we leverage cloud resources for high performance computing and what did we learn over the last hour well first we learned i'm not a great panel moderator because we're over time and we didn't even get to like

a quarter of the questions that i actually i wanted to get to and where we wanted to get so but what we learned a lot of other stuff really as well um this is a this is a bigger topic than just one panel discussion so what we're going to do is we're going to extend this this is going to become a multi-part series and we're going to work with everybody on this panel and see if we can have a few more follow-ups on this uh on this topic we kind of got to what problems are we trying to solve kinda got through that we still

got a few more aspects of that but um let's continue this discussion and maybe it'll be next week maybe it'll be in two weeks from now whenever everyone's schedule aligns but we're going to work on that uh and i'm gonna channel zane here so like subscribe follow keep track of what it is that we're doing and be notified on the update so you're so you are notified on the updates for this zane will hopefully be back very soon because he's much better at this than i am and um uh but until then i want to thank all of my panelists and everybody for uh for

watching this the huge amount of awesome information that we went through and i'm expecting lots more to come with this group so thank you very much everybody and uh keep track we'll we'll see you next time bye