HPC in the Cloud (vs. on-prem)

one of the things that most people want to talk about we talk about cloud is cost so I kind of want to save that for a little bit because I think that could be probably a webinar in and of itself but let's start off Gary how do you guys leverage Cloud today um so it's part of our portfolio for providing Computing for the researchers and so one of the things that we had done as we had put in um place a master payer agreement so that to make it easier for researchers to use use cloud and because of that we did that a few years

ago then we've had quite an uptake on on cloud usage so I right now we have maybe about 200 users slash projects on a cloud and and this represents maybe about 15 percent of our scientific workload that's interesting that's actually higher than I was assuming you were going to save Forest Jonathan I know we run across this quite a lot and that's kind of one of the reasons that we exist so whenever we're talking to our customers what are you guys seeing so kind of what I've seen on the space especially in the academic sphere is that it's really a bell curve in terms of

people's utilization of the cloud um some institutions are in the Enterprise level the cloud is you know essentially ubiquitous there's not a large Enterprise out there today that isn't trying to use the cloud in some way or shape or form but as far as you know for example one of the other major sites where a lot of HPC happens Academia like I said academic institutions form a pretty nice bell curve here some institutions are really really involved with the cloud to the point where they're deployed on multiple clouds they have um you know for example in AWS maybe they have like lambdas and stuff like

that that they've set up they're really really involved with the tooling that the cloud provider provides to interact with our platform um so on one end you have places like that that are super involved they've got you know like I said multi-cloud they're interacting with the cloud tools themselves there's a great middle where there's maybe some Cloud operations is going on they've got you know maybe some GPU bursting and stuff like that going on to provide more capability for people around Ai and stuff like that um the other end of the spectrum and this is kind of where I originally came out of in the

academic sphere is where there's not really any Cloud activity going on everything is basically traditional on-prem compute you have a couple of big super computers at your data center somewhere else and most all of your researchers are using those so from what I've seen kind of like I said in Enterprise the cloud is essentially ubiquitous there's nobody that isn't using it but across the academic sphere it forms very nice bell curve with some people that aren't using it at all a lot of people that are kind of starting to get into it and then some people that have major Cloud operations going on Jonathan where

have you been seeing cloud so mostly again it's similar to Forest my background is is typically in kind of the academic side of high performance Computing and there I've mostly seen cloud computing used where it's been in an academic field that wasn't traditionally kind of co-evolved with HPC in the first place uh and mostly these are are a lot of them are our GPU accelerated because a lot of the complexity with an on-prem HPC environment comes from the distributive memory multi-node you know fabric part of it so if you can get by as a lot of you know Big Data workloads can with increasingly dense

single node workflows then I see a lot of those come up scaling up uh in a um in a cloud context and then uh you know many of them tend to be what we would otherwise call embarrassingly parallel and they can be spread out that way so uh I see it more as as a a community thing if you're coming from a community of a practice or an academic uh field that uh let's say is in Humanities or or big data or even genetics and genomics stuff these days um all of that that's new to what we would call high performance Computing they're coming up

through the cloud and uh less you know in the traditional HPC environments they see the benefits of a cloud computing environment uh but you know there's there's a difference of of technique between the two and Crossing that boundary is sometimes difficult in both directions thank you Jonathan I would like to just point out before we go move to John to let him introduce himself and answer the first question uh we would love to have you guys tell us your views on cloud ask us questions let us know how you are using Cloud out there today so John welcome hi um briefly I got here late

should I give a history of my use of the cloud absolutely yeah so I um I've been doing Life Sciences stuff a long time and my first introduction to the cloud was in 2008-ish um when we tried to move a sequencing Pipeline and an entire sequencing operation into the cloud and it did not work it failed miserably and we burned through our introductory credits in a very short time and since then my entire career interaction with the cloud has been migrating things back on Prem that should not have been put in the cloud in the first place so I have a I have a long

history of experience bringing things back home from the cloud not a lot of experience putting things into the cloud well I think that's going to lead to an interesting point wherever we start talking about cost because that always seems the biggest driver of why things come back to the data center of the cloud in my my experiences but before we get into that topic when we look at cloud and why people do it I think it's always a question why I mean we've talked a little bit about it Forrest Jonathan you touched on a little bit of where people are coming from today what they're

comfortable with but from an Enterprise perspective or from a business perspective why are people even looking at putting things in the cloud Gary um you know uh you know the cloud provides a lot of bells and whistles and convenience um and a lot of a lot of things are readily available so um so we've done a few things to encourage use of cloud as I mentioned we have a master payer agreement so that our the researchers at our institution don't have to put up their credit card to do this and so the institution first makes it easy um the other thing that we do is

that we realize that you know there's just tons of services available so even though it may be easy Once you know what it is you don't know what it is and so we do have Consultants that provide uh Consulting help to get users onto the cloud so just to make the choice or figure out you know just ask them what are you doing and then and sometimes the cloud is The Right Use um and then we're seeing a lot of researchers who are younger who have used the cloud in college or in other places and they come with that knowledge already so they're they're really

interested in just continuing using the tools that they have already used so we have a number of reasons why people move to the cloud and then and then there's other reasons that having to do with like resiliency uh that maybe you may not be able to get as good of an uptime uh on-prem because you're not multi-site for example sure so I I want to ask Gary a quick question so the last time I was working with Gary as we were talking about clouds and how to integrate with clouds um we were talking about kind of the pragmatic side of it like how do you

actually do it right how do you integrate with the scheduler how do you how do you create a uniform environment between on-prem and cloud and how do you decide what jobs are are best capable sounds like you you've you've come a long way since since I've worked with you on this and I'm curious kind of how did all of that land like what is the in what does the infrastructure look like and what kind of jobs are running there versus on-prem and data and how do you handle all this you know that's an interesting question that's that's more you know of your what you're describing

is more of a hybrid environment where you're using both and what we're doing in our case is is a lot simpler is that some researchers are using the cloud exclusively or you know the amount of um of um integration between on-prem and and cloud is not what you would want it's because it's a very difficult problem I I think that you know there are solutions that are coming out but they're they're either expensive or they're complex or you know it's it's not an easy thing to say oh you know my on-prem I I'm just going to burst to the cloud you know people say that

all the time I I think it's uh that oh we'll just do that when we uh when we need to use the cloud and I I think that's a harder problem to solve than than people make it out to be and so so to get to your question then then our our usual cases are are more like we'll pick a certain use use cases and stick them in the cloud entirely and um they may be that somebody wants to do something and and you know automl may be the right tool for them to do something quick and easy with that and so we'll do that

and then it'll run entirely out there is it a completely kind of it sounds like then it's a completely separate instance or Cloud HPC system that you kind of built did you did you build a Beowulf style system up in the cloud we've done that too yeah so people have asked for that or people have um you know some some things work well in a cloud some people um like our physics physicists or or you know there's a lot of domains where they're generating a lot of data and they want to make the data available to for download to other people and you know that's

kind of like a variable workload that kind of works well with a cloud because you can Auto scale for that and then when you're not you know you do the data release uh you know a couple times a year and you need a whole bunch of compute power for that and the rest of the time you hardly need anything so uh you know so there's there's things that we will choose to put out there and they seem to work really well for the cloud on the cloud cool thanks no no I've got a question about that whenever we get to the cost part of it

Gary that's an interesting it's an interesting problem that we see a lot with cloud is when we start talking about egress data or having people move data out so I have a costing question about that but Dave the bonus actually did point out a good point people move back to the data center out of the cloud not only for cost but also for control so John I'm assuming that's a little bit of sum of why you're doing it people like to call it Shadow I.T I'm assuming the shadow HPC as well yeah most of the stuff that well everything drive and what I do to

migrate stuff back is always around cost it's never around control or capability or anything I mean at the end of the day a job as an application an application runs on a computer where that computer is is irrelevant right it doesn't matter how the application got started as long as it runs uh so the control thing I don't really understand in the direction of migrating back on Prem I do think a significant amount of work makes it into the cloud strictly because that's the place where the person trying to do the work can avoid their local I.T organization or their local HBC group that is

not getting their stuff to run correctly um that is a more Insidious problem to me because that's of stuff that go in the cloud that doesn't need to go to the cloud because it's just getting around local politics and I I've I've ran across numerous examples of this over the years of people who were using the cloud only because due to policy that's the only place they could get a machine they could actually control and make work that makes sense and that's what I've seen as about me from the Enterprise side it's always getting around policy that's always been the thing and you end up

having data that's in places that it probably shouldn't be or is not supposed to be somewhere like the club on the on the coast side though maybe somebody here can explain this to me because this one blows my mind um at my previous employer we were given a go all Cloud directive uh to move entirely into the cloud and we very easily and conservatively showed that it was going to be a 10x cost increase to go to the cloud and people whose titles start with c um the answer their answer was we don't care it's Opex not capex so I want to know out there

in the world why is Opex free money and capex is not free money because I've asked this question many times nobody can explain it to me but this is repeatedly come up as a reason to go to cloud is because you can pay for it with Opex I don't have a grade for you answer for you but I've seen that as well and I've only seen it last for short amounts of time usually within that first year they realize they're paying a lot more money and even though the topics I've seen that stuff come back time and time again just yeah I mean it inevitably

boomerangs but I don't understand the thinking that gets us to throw that boomerang I've I've never been able to grasp the accounting side of it so I my impression and experience of this from having that same conversation in the past uh is that uh one it's easier for the finance people to budget for so if you're doing cost recovery in an academic situation or if you're rolling your costs into a profit margin that it's easier to see the incremental costs of those things and apply them to whatever you're getting money back in from and the second is that you can stop paying for an Opex

at any time whereas if you pay all up front for a capital expenditure that you stop using that money is sunk into that resource and you can't recover that as easily so I mean it to your point John it is some accounting magic they use 10 attempts comment absolutely but what I've seen on that is people have their budgets and they talk about thinking budget one of the cloud migrations that I've done before they budgeted and I think we talked about this before about 800 000 a month when they got their first bill and it was almost 3 million so even budgeting was off and

they were okay with it for a very short amount of time because saying earlier John and then they pulled it all back in it didn't even make it a year before it was oh my goodness we can't even budget this because it was way higher than what we had anticipated and spent a lot of money just trying to figure out what we were going to have and we were wrong and to be clear when I say easier to budget I don't mean actually I mean to write the numbers down absolutely absolutely Gary I don't know if that's the budgeting part of it you know I

might say something that might uh not come across well to people who are in the position of making decisions but I you know I I feel it is just a a tougher thing to do politically to try to get a large funding for a large Capital project and so I uh you know if that could take sometimes years to do and a large workup to do that and and um uh so I you know I I think going to an Opex is more of a like a a solution where you're kind of backing into it where you're you don't do the don't do the right

thing you do that default thing and that's just my my opinion yeah I think certainly a lot of the drivers I mean I'm I'd kind of tongue-in-cheek mentioned getting around a bad I.T org and controlling your machine but I suspect a lot of cloud drivers are political and policy based not technical right that you you're not going to the cloud for technical reasons you're going to the route the cloud to get around exactly what Gary just said if you propose to build a cluster on a campus you will be inundated by people who know how to build a cluster better than you who want to

build that cluster instead of you right like this is a bike shed problem of all bike shed problems and you can avoid that by just taking your money to the cloud and spending it one time and you know walking away with your result Forrest is laughing I'm assuming he's never seen that before it would seem to be more psychologically palatable to people to pay some small amount for disposable instances in the cloud every you know to pay some small amount every month for disposable instances in the cloud can be spun up spun down and you know there's not really much attached to them um it

would seems easier to and more palatable to you know for some people to be involved in a model like that to as Gary kind of touches on spend the multiple years that it can take to plan a cluster in multiple years a couple of years or so it can take to plan a cluster or to get all that budgeting in place um it would seem that you know from what I've seen you know wanting to put together uh for example a cluster you know there were huge financial considerations with um as I think I touched on before um you know even beyond the academic space

like where the equipment was sourced from and stuff like that you know we can't go buy an entire cluster from one place the government would prefer that you piecemeal it out to 20 different companies and then you know because that's how their Contracting works um so it would seem you know just uh you know basically to Echo what Gary said there would seem like you know the actual effort that it takes to stand up a cluster and the permanence of that type of thing you know everybody names their clusters but who actually names their you know Cloud instances or whatever that they're using for something

and so in the end there's a lot more you know permanence attached to um you know perhaps this is an obvious statement but in the end there can easily be a lot more psychological permanence attached to an actual machine that you've named and you've built from the ground up and you've spent so much time working to create um than there is with just you know seemingly disposable Cloud instances that are a lot cheaper and a lot more disposable that's a really funny Point actually um Gary will remember this that uh what when when a scientist actually takes their researching funds to actually go build a

system not only does it get named but it gets photo opportunities so they actually take photos with it I'm standing next to it you're touching it and you know and and posing with it um there is something very psychological about kind of having that physical resource that you've just spent a huge amount of money on uh and then knowing that this is going to be used for our science for solving this problem it almost turns that into a tangible win in a matter of speaking so that I completely agree with that um oh I think we just lost forest yeah last Force that's funny you

mentioned it I actually took one of those photos last week it's a real thing they show the program managers they show this is what we're doing this look at what we have you know then they wrap it all up and you know what they're accomplishing if we're you know with the with the dollars um it's part of it so I've held the opinion for a while that I think that they're you know I I'm I'm not quite as far negative to Cloud as as John is but um I I definitely have had my share of um uh complaints about it and and again I'm very

pragmatic when I look at things and the complaints that I have are just about what makes sense like how do you do it how do you implement it how do you put together a system that really does make sense and drive science and drive research and solve problems and the the main issue that I've had with Cloud for so long is it's coming from it's a mandate from management coming from above coming down saying we have a cloud initiative and we have to solve this Cloud initiative and not a question of there's this Cloud thing how cool is it and will it solve our problems

it's more like we need to be buzzword compatible and I'm not by the way I'm not talking about lbl or doe right now I'm just talking in general right this is this is this is I've seen this over and over and over again and um I think though if we look at it from a pragmatic perspective there's this thing out there with supposedly infinite amount of resources or at least that's what they'd like to tell us and like us to believe there's this infinite amount of resources it's more expensive we have to deal with data we have to deal with movement of jobs and whatnot

but is there a way that this can be a valuable resource to science and how do we maximize that that that resource yeah I I've been saying for a long time that the the value of the cloud like people talk about Greg you're talking about the infinite pool of data and people imagine scaling up infinitely and there are certainly again in our capex versus Opex argument there are ways that that is a good thing where if you're building an application that serves the world and you want it to be able to elastically scale up as the world comes online but if that's not your business

model um to me the the benefit of the cloud is that you have a functionally infinite um uh like quality of product if you can imagine you're just you just kind of uh divest doing it properly to someone else but now you can carve the tiniest slice of it off and get all of the benefit of someone else having done everything the right way and you can scale down to a small prototype level and still get all the benefit of that huge infrastructure so where I think that the benefit of the cloud comes in for research Computing and even high performance Computing is where you

have new workloads and new workflows new new research interests that aren't certain whether they fit into any of the traditional like uh existing on-prem infrastructure we're talking about Cloud versus on-prem but to me it's kind of there's a two-dimensional Matrix where Cloud can mean either someone else's computer or it can mean the new culture of kind of API driven compute infrastructure as a service that can exist on-prem too but maybe there's some new way that someone's going to construct an infrastructure that better fits their research or their workflow that's something that's easiest to do and the you know imagined infinite thing that I can take

a tiny piece of and do whatever I want with it um and you know that's that's what John was calling getting around policy but if you if you do that intentionally where you're like we don't have a solution for you yet go play over there do something that works for you and then once you have it we'll help you scale it up and chances are if you have a large but relatively fixed workload that you're working towards that's going to mean bringing it back on-prem uh but you've got a lot of value of not doing the wrong thing and having to buy a bunch of

infrastructure that wasn't the right the right thing and just having to keep it long term so I'm going to jump in on that on that note um and this is a little bit of a loaded question um because as those of you have been following for a little while know we've been working on this problem um but I am kind of curious like how do in in a more traditional sense how are organizations um giving Cloud access to researchers kind of is like an introductory or like like you can't just give the cloud resources you got to install you gotta install the operating system you

got to install all of the applications the libraries the the connectors for between inter-process communication are you running a schedule I mean like there's a lot of Shadow I.T costs that now come into our shadow HPC costs that come into play when doing that so you can't just hand over Cloud resources at least effectively to researchers generally so how are we how do organizations make that simpler for for people from not good Jonathan that question comes from a state where HPC is a mature concept inside of an organization but if we go back even just 10 years especially like 15 or 20 years even with

what we call HBC today the answer to that was you give them grad students and you have them figure it out and do it and that still happens with what we would call traditional clustering like why not if you're talking about that early stage where you're doing a new thing like you can't scale that you have to just throw people at it the cheapest people are grad students hi and Saint Ducks as we build our own internal Cloud for researchers and that's something that I've been running across more and more in the conversations I've been having in Forest you and I have talked about this

quite a bit when it comes to the cloud and there are a lot of things that places that the hyperscalers can actually purchase some of that more expensive Hardware and you get to go play with it for the first time and see does it really make sense to bring back in I think Jonathan you kind of alluded to that but I know that you and I have talked about that and I think you've seen that before where people are doing that correct me if I'm wrong as far as building out an internal Cloud for people internal clouds and then being able to to try things

in the in the hyperscalers in the cloud before they make the investment on-prem so kind of a which leads into hyper uh hybrid Cloud so we'll talk about that next but um I have not personally seen so much um like kind of internal Cloud type stuff being built out like I haven't been involved in a lot of that personally um but I know that it's becoming more and more of an interest to places as they realize that they can invest in their own infrastructure and provide those type of um capabilities similarly to the cloud um but I would actually say in general I haven't um

like worked a ton with that personally those type of internal clouds I'm sure they're out there especially with um well some of the cloud providers trying to provide their own appliances and stuff like that so people can even play on some of their platforms um I'm sure it's going on uh but I think I would more so equate that with you know the concept of on-prem computing than an internal cloud I suppose it's been interesting because I've seen I've talked to several people recently that have open stack environments so what you think of is from a on-prem cloud in the Enterprise so rose has a

good question I will open this up to everyone please share your favorite names of HPC Greg I I know you have a favorite I got a good one and and my whole goal with putting up my hand and whatnot was just so Gary doesn't get it before me um so when I was when when I was at I'll be all with Gary when we ordered clusters they would typically come fully built and assembled and in racks and you know they'd be you know they'd be integrated or merged coming out of the merge Center or whatever depending on our vendor and they'd come and come in

racks and you basically roll them in worst case scenario sometimes you just get you know you know pallets of nodes and you got to put the nodes in and to the rack and many vendors would do that we ordered one cluster and I'm going to say uh protect you know not mention the names to protect the guilty um but we basically ordered this one cluster from a vendor we didn't order from before A big tier one vendor and it came in they won the procurement they won the deal they came in and it came in on pallets like nothing was integrated like PCI cards memory

chips everything was in separate boxes it was a giga when we were done with this the amount of extra cardboard that we had to deal with and whatnot and recite it was massive so we called it Ikea nice that's awesome well you've still you stole Gary's story but uh Gary do you have a favorite other than Ikea oh yeah you mean about clusters about building clusters the name of a cluster yes oh I I don't I don't know I I'd have to give that some thought we've built a lot of clusters Greg and I in fact I was just uh just like adding it up

the other day because I was trying to like recall them all but you know over the course of uh the last 20 some odd years we've we've put together 65 clusters no way yeah a lot a lot that's a lot we had one I'm not going to mention the person's name because the scientists will be but we had one system this was early on that got broken into um and uh when we rebuilt it we didn't know what to call it so we named it the pi and um yeah so we had a a cluster named a pi the pi so I'm sure that made

very interesting conversations to their internal group meetings it's fantastic anybody else yeah I'll share with them I I I'll say one that didn't make it and then I'm glad it didn't but we we decided we have a cluster that's being built right now in fact Dell was on site installing it in racks yesterday so it was as Greg would say you know we would we would they would do these large procurements and he'd come in and do the installation and the wiring and everything and um and uh the uh we thought we would let the people in the the researchers name it so there was

a poll and uh I won't say who it was but I'll say that the second place name was cluster mcclusty face and so I'm glad that it did not make it as the number one oh that's great um my since uh since I uh worked at chaos Castle was the last time I was involved in a big big deployment of a single machine and so now I don't name clusters anymore I name environments because the environment grows over time and nodes come nodes go and the OS gets upgraded but we never have a single large deployment anymore that works all right so I think excuse

me Todd's sorry did you see Todd's comment that was very funny that was fantastic black and blue thank you and and yeah we've actually had clusters almost fall off the truck um as they're unloading them and sometimes it's really scary but yeah um yep that happens I had a server get delivered one time that had forklift had put a fork through the middle of the server and they still delivered it and put it in a wreck didn't work all right so I think it's time to have the cost conversation I figured this one's going to probably spark some interesting conversation um I'm gonna start with

John because I know this is the reason that you're pulling stuff back into the data center is cost uh talk to me about that yeah I mean there's there's not much to say there it's objectively more expensive to run in the cloud for a lot of workloads and Life Sciences being one of the ones that's uh like a poster child for that so anything that we do assuming you have the slump cost of a data center space then anything you do is going to be more expensive in Cloud than it would be to do the same thing on Prem uh the cost that bothers me

most about using the cloud that I'm absolutely Evangelical about is the opportunity cost you lose when you spend your money in the cloud and that that was driven home to me yet again today as I listened to a presentation of a project at biohub that moved back on-prem from Amazon and they cut their pipeline time from Days down to 11 hours the cost for them effectively went away because they're running on-prem on our cluster now and the last comment they made when they were explaining this was and now we no longer fear reprocessing data when a new algorithm comes out or we want to try

something new and that's the opportunity cost that is lost when people spend their money in the cloud if you buy those machines and put them on Prem you can always use them for something but that doesn't apply when you're paying by the hour and and that's the thing that I I really hate to see scientists that should be using their brain power to do science worrying about how much is it going to cost when I hit the enter key absolutely so Gary I know we talked a little bit earlier about people wanting to allow others to access that data and we look at Cloud even

if you have we'll talk about committed spends in a minute but that that data egress fee is that does that become a problem when people start pushing data back out having other researchers come in and pull that data down yeah so we're part of a large we're Berkeley lab is managed by University of California and so we could take advantage of academic discounts and so without actually saying what the discount is I can say that a lot of the uh the um discounts for uh between uh academic institutions and Cloud providers uh they usually will waive um a percentage of the they'll waive the network

egress cost um so long as that your compute span is you know less than a certain percentage your compute Spin and so the egress cost for us goes away um and so it does so that's not an issue for us interesting but it could be it could be for others but but for us yeah yeah I've seen very large egress data bills from Enterprise where you move stuff in you don't realize you try to pull it back on Prem to John's point and you have a massive ability you aren't anticipating which is kind of what insane Dove just pointed out the egress bills being very

high and they catch people unaware off guard I have uh I have about two petabytes of data right now that I sent the Deep archive because it was too expensive to bring it home that I just realized I'm going to bring home through Gary's cluster there you go problem solved yeah yeah so one of the other things that that has become interesting from my perspective over the last several years is these large hyperscalers go in and make agreements with companies and it is typically you have a committed spend it's usually a very large amount of money that you're going to spend and Gary I'm thinking

maybe this is kind of what you guys are in that you have a number you've agreed on this is what we're going to commit to spend to you over this period of time but how does I've always seen that from the Enterprise side and not from the HPC side and I can see some of these high-end resources of course you and I talked about how expensive some of these gpus are in the cloud you could blow through that amount of money pretty quick so I'm just curious from your opinion of that is that something that would help make it easier or is that something that

could possibly make that next bill even bigger is it good so just to make sure I understood it is it something where you're kind of like cost averaging or or your committed spin so that you're committed to a certain dollar amount is that you're basically buying a discount Gary so you have you commit to whatever 20 million dollars over the next 12 months I'm I get so much resource anything over that I pay for but I get such a massive discount of what that 20 million dollars will buy me so it just kind of becomes a free-for-all but I can see in a large research

environment if it's considered a free-for-all and you can do whatever you want the resources are typically very heavily discounted that you could still go through that pretty quick when you're doing web services maybe not as much but some of these high-end gpus they cost thousands of dollars a day that could go fast yeah yeah well you know we we have something like that that we're trying out with one of our providers where uh We've committed we've looked at what the spend is and we uh said okay let's let's set a number and of you know based on history what what it should be and let's

lock it in that's what it's going to be the next 12 months if you go if you run under is still the same amount if you go over still the same amount and uh and so it works out well is that then um people who do blow through a lot or their cost level runs or their mistakes which do happen where where people have made some very expensive mistakes um you know then you're protected against that and the the only thing that we caution people is that you know we're you know we renew that we're the plan is to renew it at the end of

12 months and for us that'll be coming up in December and then we'll look at what the usage was and it'll reset to whatever that is and so if everybody was just going nuts because it was at a fixed rate you know then you know when we go to re-up then then the bar is gonna you know that dollar figure is going to be a lot higher so there it is that's something that everybody watches over time so everybody can kind of see where you are in that process and this kind of goes back to John's Point earlier does that make people nervous especially that

last month or two of that contract nervous to hit enter to submit their work um yeah you know this is our first time doing it our first year so you know so we're you know or we're looking you know the the spend was X and we set it at X and it looks like you know when we renew it that um you know based on usage it has increased enough that the the base level will be 2X and so that means everybody was paying a roughly about a fixed amount because we proportion it out to all the researchers but they're roughly going to be paying

about double for it for their Cloud usage um so they they may have like started off you know not using a whole lot and then they ramped it up and they probably were thrilled that it was cost them the same amount but now now they're going to be hit with a bill where it's going to be twice as much interesting Boris Jonathan are you running across these types of conversations when you're talking to customers and doing pocs and implementations about Cloud usage and fear so far from a customer perspective what I've encountered are people that are already pretty well down One path or the other

and maybe they're talking about you know either that or they're in one of those camps and they're wanting to go hybrid but it's it's not to the point yet where they've encountered that fear um so not not really yet Forest anything that you've seen in general I find that from the people that I discuss this with they're generally pretty well aware of the costs in associated with both traditional HBC on the cloud they're pretty aware of the fact that you can easily burn through that budget um so for the most part what I see is more so an excitement about moving to the cloud and

kind of seeing what those possibilities are more so than I see apprehension from like the sysadmin side um there are definitely concerns about the cost but in general the theme is more so we're aware that the um the cost of the cloud can come up but we're interested in seeing anyway um how this can end up improving our operation so there's a uh you know a concern for cost but there's definitely you know an excitement about the possibilities of the cloud and what it can end up providing for them that makes it worth it for them to investigate in spite of that apprehension thank you

for us Gary one of the things that I'm thinking about um in terms of how to you know alleviate some of the the fear around the possible future billing thing like that's the story that I always hear is people did a bunch of work didn't realize how much it was going to cost and and you know we we say blow through budget but really it's they get a huge bill that they weren't anticipating um that's all this post pay model where you kind of get whatever compute you ask for and then at the end of the month or whatever a bill appears are you aware

of anyone offering any kind of prepay model like even an on-prem when we were doing uh like resource allocations I was always a proponent of not just trying to you know theoretically build people after the fact but always having a bucket of allocation that they have access to and then if they run out of that their jobs don't run on the cluster for the month or whatever the period is or maybe de-prioritized or something uh are any of the hyperscalers doing that where you can put money in a bucket per account or something like that and then their instances just shut down when uh oh

when they run out um yeah you know what we're doing is um you know we've had as I mentioned we have we've had it an instance where there was a big cost overrun and then so we do have uh set limits on on so that this way they um you know we'll we'll try to set some kind of reasonable limit and you have to do it per project um and uh so that this way it won't run past a certain point or you'll get a notification is that what you're asking so is that an actual hard limit where it will stop and not bill past

that or it just says hey by the way you're exceeding your budget and you might want to take an action um I think we set it up as a hard limit I think it is that's cool but I I I'd have to go back and check I'd have to check I can assume that would upset people if it just hard shut off but it is a great way to make people pay attention a little better and it depends on what you want if if you're approaching this and you're like I would want to do cloud computing stuff but I have heard these stories about these

unexpected bills coming in than having this available to you would give you the the confidence that you can do whatever you want and you're not going to wake up to a huge bill yeah we we you know the the cost over run thing is is real and it's like could be tens of thousands of dollars in a short order and um you know the depending on the institution um you know you're the amount of cyber security you have for people running in a cloud may not be as good as what you have for on-prem because you're on a on a campus or institution where you

have some kind of network perimeter that you can monitor and you can see all kinds of stuff but if you don't necessarily have that visibility into the cloud so you know a cost overrun you know somebody starts hitting something really expensive like bigquery and it goes not so over the weekend or uh the other one as I was starting to say about the cyber security the hackers somebody hacks into this you know then um you know that's a great place for them to get some free uh crypto mining done you know and and that could run up really fast and so you having some kind

of hard shop is kind of it's you know is useful because uh the dollars can rack up in hours absolutely so Gary Gary where do you see the future of scientific research in the cloud are we are we there are we kind of at that point where it's we are where we are do you think this is going to become more of a more prevalent more stuff's going to move to the cloud at some point is it going to take over on-prem where most research goes to the cloud um you know we're seeing we're we're kind of moving into this just past doing computation and

so uh even before for small researchers you know running um just in the cloud or just on on pram um you know then uh you know that's that's not a not a difficult problem to solve but um when you're hosting large collaborations you know people now expect you know this has some kind of web presence they expect it to be up like all the time and so where you might have like not had the reliable availability on your research cluster that you would have on your Enterprise systems you know now now we're starting to see expectations where that people want that reliability and availability and

uh so that means you know accounting for um outages due to you know work on the data center or your your Cal your campus or your site or you know or because of some kind of natural disaster you know multi-site so we're seeing people start thinking a lot more about that when they're talking about large multi-side collaborations about the availability and um and you know sometimes you're just limited by by what your the site that you're you happen to be located at so we see we see interested people do that and then the other thing is is that um people large collaborations now don't just

do Computing in one place there's more of a move to use Computing wherever so they're doing multi-site collaborations and a portfolio of computing and so um having something available and reliable um all the time that can reach out and use different Computing at different places um you know a cloud could be a good place to like say be the locus for that and then make use of the other things so um I I kind of think the you know the tools that are coming out you know I um you know I know ciq was working on some really interesting stuff that makes makes it location

independent um so I uh you know I I think we're we're getting a lot closer than we ever have before and it would be nice if we can get to something that's truly hybrid uh but more suitable for scientific Computing and at a cost that people come to expect for scientific Computing tools as opposed to like say Enterprise tools so you know people do multi-site Computing for Enterprise and but the tools are just you know cost prohibitive for any kind of scientific research Institution foreign thank you Gary Greg where do you see the future of cloud and HPC um I kind of alluded to this

earlier uh from my perspective um it's hybrid I mean we really have to figure out how to do this in a way that is transparent uh to users to researchers to whatever the sources of the jobs are and uh come up with really good ways of doing I don't know if we call it meta orchestration or meta scheduling but some level of higher um higher scheduling to to figure out you know is this going to land you know on-prem this is going to go to a different system on-prem or a different data center or is this going to Cloud how are we going to deal

with availability regions of cloud and how are we going to do all of this in a way that honestly focuses on making science better and research better and not increase the barrier of Entry but lower that barrier of Entry um and how do we do that across the clouds so um again I'm I'm a little bit biased because that's what we've been working on uh but um yeah that's where I see the future of of this really needs to go um I don't believe that um it should be either or on-prem or Cloud I I get it that that's how we're approaching it right now

because I think a lot of the tools still need to be better um but I think in the future I think it'd be better to to hybridize it all and to merge it all into virtual systems Federated systems thank you Greg John where do you see the future Club I still get uh Nigerian prince uh spam and so I assume since I still get that that scam still works so I have no doubt Cloud will be popular in the future because there's a sucker born every minute thank you yeah more more seriously um we're we're using it as a tape archive we basically rather than

buy tape we put stuff in deep archive and so there's you know there's certainly plenty of legitimate use cases about the cloud but the the bursting into the cloud and standing up entire big large-scale infrastructures in the cloud that's very dependent on available finances and it's all fun and games the last five years six years because we've been in a bubble that Bubble Burst that money to spend on cloud is going to go away and when people start tightening their budget that's going to be the first thing that they cut um and you'll either you know they'll either start doing it on Prem cheaper or

they'll not do that work because they just don't have the budget for it so it'll always be there it's got its uses but it you know for these large projects I think that's always going to depend on the financial uh backing behind it and the ability to waste money quick follow-up on that do you need to extend your auto insurance um or auto warranty I get that one more than the more than anything else at this point I've got to be responsible for paying for all of my own car repairs that's what happens when you have your on-prem car the cloud car Gary I think

it's something about that yeah I did um and thanks um you know I I think I I don't know if we had hit it harder enough but I I do want to cover the say something about the cloud versus on-prem costs and uh the reason being that you know I had done and we had done an extensive study recently on that cost at Berkeley laboratory we we did it about 11 years ago uh it was done at nurse the Magellan Project there was a paper on that uh but now we're at the point where we're uh you know we're at the limits of our current

data center and we would have to make some large Capital investments in a new data center a modular Data Center and so it comes you're like okay now we're you're due diligence like okay let's look at the cost of cloud versus on-prem so I you know before I say what we found out I'll I'll say that you know we have 20 years of experience of buying hardware and software so we know what the hardware investment is you know over sustained period of time and then at the same with the data center you know the data center has gone through several upgrades including a recent four

million dollar upgrade to you know replace the Transformers at a chiller you know so that this way we can get some additional capacity out of that so I included all those costs you know with the labor costs and cyber security but you know to cut to the chase um you know if I look to at amount of what we deliver in cycles and storage not not capacity just what we actually deliver to researchers um it's it's about at list price is five times more expensive in a cloud and with our institutional discounts it works out to be up four times as much and then if

you try to whittle that down further and use like reserved instances or committed instances you can get it down to like 3x um you know there's even a paper I think fermilab had a paper out there if you go look it up and you know they they are really trying to they said okay we're only going to do high throughput workloads we're only going to use spot instances where we can get it at 25 percent of the pot price you know they did all made all these kind of concessions to the workload and they could the best they could do is make it one and

a half times as as uh Cloud was still one and a half times more than the on-prem cost um so uh you know after we had done this then we we had talked to some other institutions and so there are some other large institutions who took a similar approach and they came up with similar numbers um so I I feel pretty good about the uh the analysis um people are interested I can walk them through it uh but I did want to say something about the cost because you know I you know about an analysis that we did without we thought was pretty thorough that's

great here because Donovan actually has a question Donovan Campbell asked how do we predict future costs and how to migrate I think that's a maybe a sidebar conversation but that is a great question foreign Gary oh that's it I'm done I'm done okay thank you very good nope and we are right up at the end of time guys I'm sorry to kind of cut it off but I want to be respectful I appreciate Gary John as always really appreciate your time Greg Jonathan Forrest it's always good to see you thanks for stopping by guys if you like this go like it subscribe we enjoy the

the interaction with you and we're looking forward to next week thanks guys thank you foreign [Music]