Research Computing Roundtable - Turnkey HPC: Monitoring

got to the point where i did not find it confusing or a pain and so often you know i know there are better tools out there but i have historically just tended to fall back into nauseous because it's what i know it's what i know how to config and it's what i know how it behaves and so i look at what especially kind of the web scale people use and you know a big tool like prometheus and i've used um what was it i used influx db and kind of their whole suite of tools for a while and i i appreciate that at a an

architectural level but often find it difficult to advocate figuring all that out when i already understand the nauseous part i can see that glenn you've got to have some thoughts on this yeah i uh so ganglia was like my first experience in monitoring i still have scars um not because if it came out of berkeley or anything like that but um no it seemed to be you know i was thinking about this topic and it seemed to be like gangly seemed to be enough to do enough but uh kind of its ancient interface and uh not able to really give you a picture of you

know hundreds of nodes for example kind of is where i thought it was running into limitations and so interfaces that could show you you know um you know compute nodes with a color code scale for example you know a really small amount of pixels so you could see you know 128 nodes or something like that i thought were going to be the future and so things like prometheus another kind of cloud scale type of monitoring tools i think are more effective and also i find when you can put things into a database whether it's influx or or elastic and then play things back like you

can you can play back what your cluster was doing over a certain period of time i find that that's uh really helpful particularly for trying to debug why someone's job was you know affecting every you know having you know affecting all the nearest neighbors things like that so i'd be interested to hear what other folks think about what they would actually find important in a monitoring tool right is it just cpu is it percent memory um being being consumed or or other absolutely of course coming out of a recent academic environment what did you guys do um i've been thinking about this in the run-up

for this meeting to make sure i had the answer to this but it's a little bit murky in my memory uh if i recall correctly we were basically just a nagios ganglia shop i can see the charts that we had in our head or i can see the charts that we had to monitor things in my head i don't think we were on grafana um i'm pretty sure we were using uh just basically the nagios and ganglia stack i don't have too many specific thoughts there um it worked for our purposes we were a pretty small cluster so we were mostly just looking at kind

of the standards of what you would expect um you know cpu that type of thing trying to watch out for uh you know jobs that are utilizing nodes that aren't actually spinning on them at all um some of the stuff that i kind of did on the side as far as this went was basically just creating the custom scripts to parse what slurm would give you about different bits of info so that you know i could see all at once uh what nodes had stuff scheduled on them all that type of thing without having to put in four or five different serum commands so i

had some stuff like that that i had set up just kind of for my own custom thing but for the most part as i recall we were basically uh nagios and ganglia about what you'd expect thank you forest greg i know you've got some thoughts on this area especially after you had a shot taken at you which shot was that thing uh yeah sure no is it the berkeley one yes that one okay just checking bernard's not on the call so i can do it oh bernard is really the one to blame i know but but uh so when i first so so there there's

a bunch of kind of legacy here that i can talk about uh as i usually do uh so when we first started doing werewolf back in 2001-ish there was a big request there was a lot of people interested in not just provisioning but also management and there was a piece of werewolf which actually did monitoring and ww top you do you remember and i actually just did a google search and i just cause i don't have any screenshots of wwe top actually running anymore so i was gonna i was gonna see if i can whoops no not sure that um i was going to share

something just so everybody can kind of see what it looked like all right let me guess look like top um but looking at independent systems it was a real time monitoring of you know your entire cluster and you can sort by each field this one's sorted by node name because you can see it's bold um and you can basically go through and take a look at your entire cluster you can also sort according to you know idle what's um uptime and all these things you can find out what systems are dead what systems are alive uh but this was never hooked into like the resource

management system and whatnot but you know with the appropriate configuration you can very easily uh have it you know link in to the resource manager and just show you nodes that are running a particular job uh as an example so but this actually had a daemon process that sat on every compute node and basically every second would query the local resources of that system come up with you know how many cpus do you have architecture and whatnot but also you know what is your current memory utilization uh and you can see you know everything from this this person has temp in there as well um

how many processes are running etc and so forth but this gave a really good kind of real-time analysis now you know we already talked a little bit about ganglia and at the time i'm going to stop sharing now at the time ganglia was really the the prominent solution that was being used one of the issues that that i i and other people had with ganglia was it was using a multicast channel to do all of the updates so you can so you can imagine if you're not familiar with multicast think of it like broadcasting to a particular uh group or to a particular port and

so you can basically do something like every ganglia daemon that's running on every single node is listening and broadcasting to every other node in your cluster what this means is ev which means is every node you know in some times you know dozens or hundreds of times a second are hearing about every other node in your cluster now for smaller clusters it actually was it was kind of convenient because every node always knew everything and you always had backup data but i'm not sure how relevant that backup data was but uh every node just knew everything it was never the major benefit i guess was

that there was no configuration you just turn it on it jumps into a multicast channel and it starts broadcasting and receiving data and then you can put a web front end on it and whatnot and make it real pretty but when you're dealing with bigger clusters you don't want all nodes listening and hearing and parsing xml from all other nodes on every single status change of that node so it was kind of a heavyweight solution so what what we did differently with with werewolf is it was it was very lightweight and even though it did create a context switch once a second in an interrupt

once a second as it was collecting uh resources and collecting current status and then shooting that over the wire if certain things have reached certain thresholds of changes so it tried to be as optimal as it could on that but there's no getting away from it you're still caught at causing context switches you're causing operating system noise and jiffy's and if you're doing tightly coupled hpc every single thing you do is going to have an effect on the overall job and performance of the system so we tried to be as lightweight as we could we sent a udp packet that had everything that we need

for that nodes directed to a particular headnote so no other nodes heard it but this was incredibly rudimentary still right there's so much more that we could be monitoring effectively the question is how do we do it effectively and how do we do it that's not redundant with the resource manager so the resource manager is typically is already doing some amount of uh monitoring so how do we augment that with the resource manager without adding addition any additional load or resource consumption to that system and so that's my real big kind of perspective on monitoring is how do we get what is the d what

is the information we need how do we get that information without causing you know latencies bottlenecks and whatnot and then how do we portray that and visualize that in an effective way to users i think it's interesting that glenn brought up being able to go back and replay stuff that had happened in the past to troubleshoot but then john maybe you don't have that need or it's not something that's relevant or it just takes too much time so from your perspective john why wouldn't that be something that you would do or is it just too much to deal with it's probably yeah i have a

cynical view of this of course um it you back up to why you're going to monitor if i'm going to monitor because i'm a system man i need to troubleshoot stuff i would just rather do that in real time run the application and and debug stuff in real time see what's going on or look at the cluster in real time with a workload on it to see what's going on i i wouldn't really have much need for playback okay but if i have a boss you know point here boss somewhere who loves to click pretty things in a gui then that's a fantastic feature to

keep them busy so they're not bothering me while i'm trying to do real work so in that case i would like to have playback hey why don't you go look and see what the cluster is doing a month ago and leave me alone that'd be a great feature for me to have excellent makes sense thank you there's a lot of chat going on i there are several people i know allen had posted in there kind of what they're doing uh from the bmc's and they're they're collecting that from a time series database and organizing things so there are a lot of opinions that are going

through it's great thank you guys for chiming in on this love it the bmc is a good point i mean the only way you can collect metrics from a node without impacting the node is to collect them from the bmc anything you do in the os is going to cause a problem and that's actually the last time i used ganglia that's why we stopped using ganglia because it had a severe impact on open phone performance and so we just dropped ganglia from the cluster completely and stopped doing that was when i that was when i first stopped collecting metrics on nodes it's the point that

uh raymond just pointed out as they have it ganglia monitoring 2000 servers and they had to have a dedicated server to do it so it worked he says that's very cool so how could we help people get more value out of a more modern tool or is that something we should even worry about jonathan i'll go to you first have picked on you in a minute you're on mute i've struggled to find kind of a best practice or or best in class and still fully open i'll say because that's a bit of a a preference for me anyway monitoring stack to point people out and

say not only is this good and it's it's representative of what people who know about monitoring would recommend you do today um but it's also not a huge complex thing to get deployed because what i've run into a lot let's say elasticsearch as a sync for data the the last time i've tried to do an elastic search deployment not only has it had a whole bunch of licensing encumbrance issues uh recently um but uh it also very opinionatedly wants you to have a three node cluster to store your data in right off the bat because it's expecting that you're going to be doing web scale

work and why would you ever have a single point of failure in your monitoring system and finding that that kind of middle ground where you want a good solution that you could scale up in the future but gets you up and running really quickly and demonstrates its value really early is is something i've frankly struggled with i i don't know the answer to it yet it's one of the things i was hoping to to get out of this conversation if you think about the stuff that you really want to collect um you know you're you're talking about enough ants maybe maybe enough doubles to fill

a a jumbo frame and ethernet so if you if you sit out at the beginning to just say i want a monitoring system that will update my information every two seconds every five seconds or whatever and just give me enough data to fit in a jumbo frame and set that as a constraint like you can never send more than a single jumbo frame full of information by definition you're going to wind up with a pretty efficient uh collection system right you've limited this down to something that if you drop tcp and just do a udp and accept it to be lossy you've made that thing

super efficient at that point that's what i would like to see i'm if there if such a tool like that existed i would probably be interested in monitoring again but i yeah i have no desire to set up prometheus or ganglia never again um i will do some stuff with graffana but usually just from my slurm controller uh dumping a few you know minor things out of slurm just to see what slurm's up to you're describing what werewolf was the wwe top and werewolf monitoring system but my question that i would push back is because i haven't i haven't developed that in literally a decade

and a half i mean it's it's end of life um and i i haven't used one in my hpc systems in at least a decade maybe even a decade and a half what kind of information do we need to be exposing via monitoring to users and system administrators like what do we what do we what's missing right now i i will take that at least for me for users there is nothing i want to expose to users they can engage slurm profiling and get an hdf5 file with this fine grain metrics on their multi-node job whatever they're running as they could possibly want to do

so for a user that's doing troubleshooting between sshinging to the node and just what the built-in slurm profiling does there's nothing that i want to show those people this would be purely for me as a troubleshooting tool to be able to see cluster-wide what's going on and you're a system administrator but what about those people [Laughter] what about those users uh what what do they need to be more effective and and is it already there like you brought up profiling for example uh profiling is absolutely critical but most hpc users that i'm familiar with don't spend much time profiling it's like what do we need

to be exposing sorry jonathan one of the things that that i've seen be useful in the past and wish that we had more data on it uh is the time spent in file i o um and especially differentiating like uh actual data read and write from metadata um you know the characteristics of different file systems are different whether it's a clustered file system or just a remote file system or local um the the biggest overlap we've had uh where we took some systems monitoring and made the user's life better was pointing out that if they adjusted their file i o and their their i o

pattern that they would get drastically better performance for their overall application um but that's difficult to visualize and difficult to collect in in my experience often with the the tools that i've used it's great alan keeps putting stuff in here i kind of wish we could have alan pop in to kind of tell his story they're doing interesting stuff and they're collecting data and be able to go back and do replays uh multi-node mpi even memory and cpu so memory cpu and power consumption that's thank you for adding all the color on we appreciate it steve's adding some stuff in sorry go ahead for us

i'm just gonna say yeah that's really interesting what he's talking about there pulling the info just based on native slurm apis and the bmc network that's pretty interesting yeah and steve asked a question earlier about how are users managed and or monitored from either allocation utilization or scheduling so being able to monitor an individual user are they consuming your entire cluster like how is that being done or do people even pay attention anymore i think um slurm provides a lot of that type of functionality i know for example the cluster i worked on we had uh you know just to address one specific thing can

the user use the entire cluster we had some set within slurm that would only allow you to request up to eight nodes at once um from a certain queue so i think a lot of sites would have stuff built in natively with slurm that would allow them to keep track of i mean you um sq is the classic command that shows you everything running everything pending um then yeah that's all controllable through kind of slurm natively so policies studying policies of what they can mostly can't do yeah for that kind of accounting information uh what i most often see people to deploy is a tool

called xd mod um that was that came out of the exceed project and i know nsf at least liked it uh when you deployed that and liked the reports that came out of it um it made more sense in a proper exceed environment um but we we were using it as a as a tier three site as well at one point i never got that familiar with it or that uh i don't know that adept at using it but you know it integrated with slurm and pulled in all that accounting data and let you show multi-user a multi-account uh usage graphs over time yeah thank

you jonathan earlier when i said i couldn't quite remember exactly what we used at boise state i had x at the front of my mind but i was wondering if it was just because i've been messing around with a lot of uh x11 debugging yes xd mod was something that we used and produced a lot of graphs and stuff like that that we were able to serve back out so i know that um yeah that's still popular out there great thank you greg any thoughts on this before i change topics a little bit i'm still really interested more on the user side um [Music] john

is looking at it from the system administrator not to put words in your mouth john tell me if i'm wrong kick me you're looking at from the system administration perspective jonathan brought up the uh the storage perspective and being able to identify bottlenecks on the storage via io locks and and whatnot and i think that's incredibly valuable as well but again that's not so much user so i'm really kind of curious so when i when we first did like wwe top the reason was was because users wanted to know exactly what their applications were doing you know are they effectively using the cores and the

memory of that system but what is what are the what are the holes now for users and and is there anything that we need to be any problems that we really still need to solve and or are there tools out there that effectively do this again on the user side our our users can ssh to the node where the job is running and run top h-top a top i haven't ever given them access to io top but probably should figure out a way for them to run i o top any tool they can run they can ssh to a node with a running job and

run that tool i don't know that there i probably do have some users who would be happy to have that in a web interface somewhere but the ones that are actually able to understand what they're doing when they're tuning something in profiling troubleshooting they would prefer just have ssh on the node and go work on the thing directly is that true even for users that have multi-node like mpi workloads because that's where i would expect that kind of paradigm for kind of high-level application profiling through node monitoring would fall over but maybe i i've never been an mpi developer so i don't know i don't

have those users but back when i did have those users which was 10 years ago or more they still ssh to the node i mean people wanted to run things like um there was a debugging uh there's a debugger tool that that i used to use but i think a total view is that something they were thinking of yeah yeah they were that and then other debuggers that would do parallel debugging so they really just wanted access to start that debugger and ssh to the nodes and and let the debugger do the work that's great thanks for that so kind of forest and i have

talked about this quite a bit and it's an interesting topic that i don't we don't have it on the outline so this is going to be kind of new to most people on here and i want to get reactions and thoughts because it's an interesting topic um again forest and i've talked about this back and forth quite a bit what happens we start throwing things like ai into this and having hpc have ai associated with it and being able to help with this in a in a more artificially intelligent way and i'll let flores kind of dive into it of what we talked about and

what he's seen kind of thoughts go for it for us yeah just generally you know we kind of focus a lot on the human side of monitoring and you know providing something that a person can sit there and look at and monitor and you know derive insight from to my knowledge there's never been any effort to do you know ai driven analytics on a supercomputer um you know i'm talking things like intelligence scheduling where instead of things just being governed by an algorithm there's something actually looking and trying to decide in whatever artificially intelligent way that is developed there where things fit in best like

i said as opposed to just having like an algorithm um you know i see the the possibility that as clusters become very very big and we kind of stop viewing them as collections of nodes and more as like perhaps a single industrial machine that is being leveraged for a certain purpose rather than like a collection of nodes it might be useful if we could um you know have an all-cluster way of analyzing these very large streams of info from these very large clusters at once and i think part of the answer there might be ai to be able to derive um those type of analytics

from the data being generated on these clusters and of course you know some of the first things there are you know is there a data set out there that we can look at you know a million scheduled jobs and begin to you know train ai over that type of thing so i think this is something that hasn't really been explored um i think a lot of these solutions are pretty manual um and they kind of you know require a human operator looking at them but i just kind of want to bring up you know the idea what about you know ai driven monitoring what about

something that we have designed to just look at a cluster and see where things could be more efficient where things could be improved that type of stuff is there any efforts going on with that any thoughts on you know connecting ai with super computer management so i i don't know if there's any efforts going on but as as you're kind of describing this where it brings my head into is if we can instrument whatever's spawning the job so we can monitor the job in a way that we can pull in that context and feedback back into a model we can start making intelligent decisions or

rather recommendations as to is this a good hardware for you to run this job on do you you know are you sure you don't want to give more memory to this job um you know things like that are you sure you want to run on this architecture maybe you want to run on a gpu which happens to be available right now i think that there's a lot of intelligence that we can glean with the appropriate monitoring monitoring and modeling of that data i would say 100 yes i think that would be dependent on the type of ai so if you modern ai that we're all

doing on gpus now it's just pattern recognition i can't think of any pattern in my logs or any metrics that i might produce that i couldn't just grip out myself and write a simple rule for so i wouldn't really need to train a model for that stuff but for the stuff greg's talking about that's like the old ai stuff where you would do a tree search to do you know to take a derivative of something to solve a calculus problem and in that case once you once you know the rules to be able to produce a decision tree from that that somebody can say i'm

going to run this job that does this and then that decision tree produces a valid suggestion for them there probably is a ton of use for that to present stuff to user space in a better way so people would would make you know get a more optimal use out of it by being able to follow that decision tree great thanks john jonathan yeah so i have two primary uh experiences with this um both reflecting different parts of what forest brought up so uh one was there was a project at a past site where i was that that ended up being someone's graduate thesis i think

where they so at the time the luster environment for that system was was pretty unstable and kept falling over and there were particular user workloads that would cause the luster to become unstable and crash and so they were looking for ways to you know beyond trying to figure out why it was happening in the first place they were looking for ways to predict that it would happen and so there was an effort to dump all of the high high verbosity luster logs and and file system telemetry uh and then train an ai model on it to see if the ai model could predict when the

luster would fail and my my understanding was that it became better than you know even odds but and you know it was an interesting academic project that wasn't something that would be particularly useful uh at least not from from the data set that we had fed it which was a single luster system with a single set of of user workloads so maybe you could provide or develop something like that that you could train up for a given environment and that it would learn the kinds of things that happen in that environment but in terms of generalizing it for any luster environment or any cluster i

you know it at least wasn't part of the effort um the other thing that i've seen uh are people trying to do ai enhanced scheduling and all of this comes from uh ultimately if users were truthful or fully accurate about what their job was going to do ahead of time then you could schedule the system perfectly and you wouldn't need something like ai um but they they don't they specify you know friendly values for wall time and memory use and cpu requirements and things like that and their their requirement definition isn't precisely tuned to what their job will actually do in the real world and

so what i've seen are ai driven attempts to take job history and see what jobs actually did versus what they said they were going to need and then make different scheduling decisions based on historical patterns and you know over subscribe a node based on what the jobs requested uh and and try and do it based on what the jobs are actually going to require for example maybe they requested more memory but we the ai model has learned over time that this kind of job tends to use half the memory that it requests that kind of thing and and schedule the the resource better that way

and you can get interesting results that way but i i tend to see two problems with it one of which is that the the uh the impact of a failure of the ai model is very high if you say you needed 10 gigabytes of memory and then over provisioned that on a node and your job fails because you actually needed that memory and the ai model thought you wouldn't that's a bad outcome uh and it would be bad for the scheduler to create a job failure that would not have happened otherwise the other is that it's not just an automated system and it's already difficult

enough to get users to trust the scheduler one of the most often uh or one of the most frequent questions that we would have is why is my job not running yet and you know we could go into detail about how the fair share scheduling algorithm works and why their job isn't working yet by plugging all the values into the uh into the formula that determined job priority uh and over time we could all like without fail we could show them the historical records and say look you have been allocated this share of the machine and they have been allocated the share of the machine

and over the past month you've gotten exactly that and it's scheduling to produce that outcome but that's with a very you know relatively speaking well characterized algorithm that you can point to the values that go into it and say this is why it's happening that way the minute it becomes a fuzzy ai model that you just have to trust as making good scheduling decisions i think it would become even more difficult to get the users to trust that the scheduling is fair or correct and the the worst case of that is where you're scheduling all becomes a reservation based and everyone is just reserving time

in the cluster and you lose out on all of the benefits of batch scheduling because oh this is my time on it and then i have it all monday or that kind of thing and that when people stop trusting the algorithm that's what they start doing is trying to get dedicated time on it instead glenn you're kind of our ai guy around here and i wanted your thoughts on this as well and then we'll go to chris thanks for joining chris yeah i think uh that the ai could be used to um better like for suggestions to like uh what someone was saying like oh

we're going to submit this job it's like well are you sure you want to run it here you know historically it runs better on a gpu maybe you should think about scheduling on a gpu or um i think that will be useful but um yeah trying to kind of try to beat the current algorithms at backfilling and things like that i think are probably a bad idea it would just lead to a lot of a lot of chaos chris introduce yourself if you don't mind yeah i don't mind at all uh chris stackpole um i've been doing uh hbc for a long time and uh

currently i work for advanced clustering technologies and we're kind of a systems integrator and builder for people who need hpc systems thank you thanks for joining so we're talking about monitoring and i'm i hope you have some thoughts on monitoring and any of the topics we've covered so far that you want to touch on please do sure um yeah i actually do a decent amount of monitoring i've been working with xavix in some capacity since about 2007 and i really do like uh that for monitoring especially for uh hpc systems uh from an admin perspective um a couple different reasons uh i monitor uh anytime

i have a problem um i make sure to monitor it whether that's as simple as you know an ssl cert going bad or a particular use case where um you know we had a user who um one aspect of their job uh was writing so many small files that they would actually uh tank or storage solution i mean just hundreds and hundreds of thousands of small files really really fast so monitoring those situations and then with xavix i can add in rules to perform actions so if i notice you know um we had this one application that was run that didn't clean up after itself

very well and every once in a while when it got into a weird stuck state it would just chomp memory until it just took over the node so you know look for these conditions and then reset that service and have that do it for me automatically so that way i'm not always having to go in and mess with the different nodes somebody else earlier mentioned um i got in late so i was watching at 2x for a while but they mentioned user performance data and with xavix being able to create dashboards for users that they could see here's what my cpu usage was here's what

my memory usage was um things like that really helped especially for those users that um didn't really know what they were doing they don't have a whole lot of skills a lot of them you know are connecting in and just writing a script or something the entry level ones they really like to see little graphs so not only just from monitoring what's going on the network but um how this system is being used can i address problems before there are problems that tank the system um and have that done automatically um to even providing useful information to a user for what their jobs look like

well that's great thanks for library on that chris appreciate it chris you have any thoughts on ai hpc monitoring um i i'm not sure i've got a ton of those there was a an article i think uh i think it was soylent news but it may have been slashdot recently talking about you know the downsides of using ai for two small projects so your data set isn't um big enough to really have meaningful conclusions um and i think there's a lot of valid points in that like i can understand why somebody who you know is a major company that's got a lot of data they

can churn through to make sure that their data that the ai is acting on good data and has seen all these little use cases but the vast majority of the time i think we see a lot of small problems creep up and as sysadmins were usually pretty good about saying hey this is one problem that keeps coming up so i'm going to write a script or something to fix it for me and then we just move on and so when you see these oddball problems come up i mean that's a good chunk of our job is oh great what did happen this time that i

got to go figure out you know something new um and so i think that's where it gets really hard with a lot of ai unless they've seen a lot of the data so unless you're talking about something like on the size of like exceed if you're pulling in all the clusters from uh exceed um and you could probably get some really good idea as to what use cases for generating problems um out of that yeah just to touch on yeah what i said earlier that's one of the biggest things about this is that i don't think there are any unified hpc data sets that could

be looked at that have attempted to aggregate data of this nature and jonathan touches on you know luster performance you know that'd be a huge effort to get that from a lot of different sites so i think that you know much like uh you know tends to be a problem with new things in machine learning the lack of data is what would um you know really prohibit that development for the moment because there's no unified way to get you know all of exceed to put their logs into one thing for example and whether or not that's even possible is another question entirely sure i see

we've had alan joined thank you alan hey folks short notice hopping in appreciate it you guys are doing a good job so don't take any criticism from my previous comments no no you've been posting a lot of comments and i think it's an interesting segue i mean you have obviously a great story and you're doing something that maybe we weren't thinking of or weren't looking at so if you don't mind kind of elaborate on on what you're doing yeah so um first i'll just go back and reinforce a point made earlier by someone else in the chat that uh reminded me of uh when we're

building the databases for the first distributed computing at cdf in the early days of the grid and we uh we had a database administrator said keep your freaking monitoring off of my machine i will tell you when it's working right so i i fully uh you know endorsed that point of view um if you know what your machines are doing um but this is the modern day and age we're spinning off clusters as cloud jobs we have to have automated systems to to keep track of them so what we've been working on is what i like to call precursors to ai it's a lot of

discussion about ai in the data center and uh you know you can understand the marketing motivation for this language um but um i'm first of all not a big believer in ai at all i'll put my biases right in the table i tell people the machines are not that smart uh what we have is obscure scripting with un undetermined failure outcomes so what we don't want in the monitoring world is any of that stuff you don't want as as uh john said uh you know pretty pictures for management to click on and get bored at and never look at again so we've been working on

these kind of precursors to ai things that could be automated and you ask yourself the question why hasn't it been up to now and one very obvious limitation is lack of sufficient data uh as mentioned earlier you know people who implement in-band monitoring quickly take it out if they're doing anything serious because you don't want it interfering with your hpc so that automatically limits the rate at which you can get information therefore making it useless for computer science purposes we've been kind of examining the assumptions there the first obvious thing was baseboard management controllers are way smarter than they used to be i mean they're

16 core arm systems with you know 16 gigabytes of memory i mean that's those the most advanced ones and the smallest ones i've seen are four core so you know you could get very decent amount of computing done just in the bmc and then you examine why is that not being used well ipmi totally sucks as a protocol so let's not use that and the industry has been busily making that better through a variety of open source projects to program bmc's but also the standard redfish standard from dmtf has a uh has a streaming telemetry feature so let's turn that on and all of a

sudden you're getting multiple hertz of uh multiple measurements per second in a manageable way that scales to thousands and tens of thousands of nodes okay so now that's all in the links that i put in the chat what i'm interested is what you guys been talking about most recently how do we actually instrument the code in a sensible way and when we're talking about hpc you know by and large we're talking about mpi code i mean high throughput computing is very valuable and useful don't get me wrong but how do we instrument mpi code without getting in the way of it and uh for that

i think we have to turn back to the mpi community itself and look at some of the stuff being done by uh amv pitch projects and uh livermore has this caliper tool but that requires instrumenting your code um so i we are working on some stuff this center that i posted linked to is an industry supported center and we have some stuff i can't talk about yet but it is aimed at this this goal of uh non-intrusive comes with the package you don't have to do anything high quality uh you can and and you can shove it in a in a time series database i'll

just uh stop here because i've been talking too long but i don't know you haven't been talking too long that's fantastic and i forgot to ask you to introduce yourselves that are oh yeah so i just i just started talking yeah so i'm the managing director at high performance computing center at texas tech and i i'm one of uh several co-directors of a multi-university uh national science foundation funded industry university cooperative research center called cloud and autonomic computing you always have at least one piece of jargon that your boss doesn't understand that's that's the naming principle for that center um so i just wanted to

put in a plug for time scale db if you haven't been playing with it if you like postgres and you just wish it were better at time series is fantastic uh open source implementations are very powerful yes you can get commercially supported ones but um for what we're doing we didn't need it and so that data that you're collecting alan are you going back and doing replays of that data are you people asking to go back and do that well yeah so that's actually why we put it together the way we did we didn't know what would be interesting we'd know what knows that sometimes

nodes become unreachable why i'd like to know what was happening before that node became unreachable what was it doing we do not yet have a solution for the file io it's a harder problem than it sounds like but i agree with forrest that that's immensely interesting and so we're looking for a smart way there but what we found was we could just look at um with the bmc you can actually ask it how much power are you spending in this cpu package and how much power are you spending on memory um in principle with the redfish we can get actually the error rates of the

hcas for the network cards we haven't got that instrumented yet but you know one thing about redfish that people don't understand is it's more than just a replacement protocol for rpmi it's it's how the components of the motherboard talk to each other they they actually use redfish internally uh to to compose the the the platform you're working on they just don't use the htp parts um so the big difference between redfish and ipmi from a from a system manager's point of view is that ipmi was udp and streaming and no any one message is not necessarily going to get through uh redfish is tcp and

it's deterministic but you know it suffers from the chattiness of tcp uh and we were thinking of trying to do clever things with http 3 but then we found that we talked we were able to talk the standards organization and implementing this telemetry feature you can just go tell the bmc here's a list of things i want you to tell me about and you you check them internally as fast as you can or as fast as i tell you to and you know just do server side push to me uh anytime you got a result so it totally inverts the the bandwidth uh and uh

and and gets rid of the shadiness problem so if if you for no other reason you wanted to learn redfish learn learn it for telemetry um and now i'll be curious yeah i'm sorry i'd be curious in knowing more information about it because the last time i looked at redfish wasn't that too long ago there wasn't a great standard for it so if you're trying to pull metrics back for cpu temperature it's going to be different for gigabyte versus asus versus intel standard but they have their hierarchy is their hierarchy is different though yeah but it's discoverable it's like any api you say you say

the api what have you got and it tells you this is what i've gotten and you say okay give me those uh so it's yeah uh any particular manufacturer can can customize to their so if you want to think of the design structure of redfish think of an outline it's just an outline of what's in your server and so i guess it was that like intel i think had like cpu one temperature um but like asus would start like cpu zero or somebody else would just say you have spaces or they would just have mb temp yeah they were they were not consistent in what

that actually neither is the hardware neither is the hardware so what you do is you go to this neural property and say what have you got listed under thermal properties and it tells you you say okay i want these in any given data center you'll have a relatively modest number of such things but uh right but they discover then you're querying for every level though right because you have to query each level in order to know what the level is below it so then you end up with these request strings that are just really long and they're different for every motherboard type yes think of

cloud services every cloud service out there is probably going to be different it may be different each time you query it and so a well-designed microservices architecture will will use the discoverability features of say open api or uh async api or something to to to navigate that uh i have a nice tool if you want it's in our github.com nsfcac called redfish prompt have you ever played with http prompt that's uh basically turns any api into a interactive session uh and it you can navigate through it like directories so what h what redfish prompt does is it goes and it pulls the open api spec

of the redfish thing you're talking to and it pre-populates that so you can navigate through and it has you know uh auto completion and stuff like that i guess the short answer is yes it's standardized you do have to learn how to use the standard but the the benefit of doing so is any graduate student can write the uh the code that fetches things you don't need snmp traps or any of that crap and i can i can prove that statement because most of our code is written by graduate students if you haven't looked at the visualization and the links it's astonishing yeah i'd like

to link to that uh redfish tool if you wouldn't mind please absolutely chris and we will take the links that allen provided and we will post them with this video so they will be available download jonathan i think you had something you wanted to add i was just going to say that i have all have been meaning to get into redfish for a while and i had you know i would talk to the sales people at the different server vendors and they'd be talking up that support but same thing like i was talking with najis i know ipmi and i'm used to using it so

i've been kind of stuck with the limitations of that and it's it's really encouraging to hear about i think he said redfish prompt as a possible uh kind of on-ramp for getting familiar with the protocol and and learning what it could do and i'd be interested to know you know mate i think this makes sense but to say what kind of integration for red redfish we might be able to bake into future versions of werewolf or something like that so where it's using ipmi today that might be a good way to get into it yeah so uh let me put some realism here um it's

great it solves the problems from my point of view for talking to bmc's and um the direction that we're pushing now is moving away from just the bmc to try to talk into have apis to talk to the mpi layers that's our research work right now that's definitely not ready for prime time um the other direction you can go however that you can do with redfish is into the data center which is uh you know rack pdus and crack units and power systems and cooling systems and that is happening um so redfish is no longer just a protocol for talking standard protocols talking to the

baseboard management controller servers it's a data center api and um in my center we have a lot of work focused on trying to get automation to the point that we can put and operate data centers at renewable energy power sources remotely i mean we're not new to this but we think we have some special tools and uh just to you know try to help save the planet and stuff um we wanna and also we have a lot of wind power out here in west texas right so um uh you know we we want to try to you know yeah and you really need to talk

to everything in the data center on an equal basis you can't just say oh i can talk to my server a lot but i have no idea what's happening to the air conditioning so redfish has got extensive uh tools for for data center automation now i'll put it i put a link in and stop talking oh that's fantastic thanks alan really appreciate it i think we do have some questions coming in i've seen several uh one of them so yep that's all i was going to read so george asks what really needs to be monitored in a cluster other than cpu and io traffic hey

jonathan you had some some thoughts on that some wishes yeah i mean you have within the cluster you have those things you also have kind of memory uh over time and watermarks that kind of thing um some there sometimes we've seen smaller things that you know we're far outside of our ability to monitor things like uh cpu cached hits and misses um uh where we saw applications behave very differently on seemingly similar cpus that were really just differentiated by the member the amount of of level three cash they had things like that could be interesting but it's one of these games where you can't predict

what you need to know until you have the problem and then you don't have the information anymore so there's there's kind of a whole bunch of stuff that you could maybe want to know sorry forest i know anything in particular john that you think other than those those simple things cpu io um i really i think jonathan hit on it earlier when he said monitoring storage would be really good and i there is no solution for that and that's probably the most critical thing in high throughput computing for sure would be to monitor storage and i i i've tried tackling this before in the past

and failed miserably and i really think that's you're getting to a level there where the only thing in the entire mix of tools we're playing with that can answer that question is the resource manager um and with slurm finally supporting uh v2 of c groups i'm really hopeful that having jobs track i o and c groups will finally give us an end to finding out what each individual job is actually doing with regards to io okay of course i think you had something you wanted to say i was just basically going to say um i think we can broaden the category of things that we're

looking for with regards to monitoring um if we you know kind of look at things through an automation standpoint you know just like what we're touching on the storage here um jonathan you touched on someone's thesis that pulled out you know some interesting logs from luster and that type of thing and they were trying to do analytics over that um alan it sounds like the system that you're putting together provides huge amounts of just all kinds of different analytics that can be looked at um yes so you know that's all good yeah i was just gonna say well yeah we may have more conventional needs

like you know of course we need to see the cpus of course we need to see you know what's running that type of thing i think that if we did you know kind of look at automation and we kind of looked at this from the perspective of what small real-time tuning can we generate from these different analytics that a cluster is generating i mean if we had a way to automatically look at those correlate them with each other you know in such a way that a person doesn't have to sit there and lunge through all that data um i think that would really really broaden

what we're looking at with regards to what we actually want to monitor if something can you know once again just like jonathan said analyze something like a luster log versus you know the temperatures and you know look at those different things to find correlations and like i said real-time tuning that may not be obvious to a human operator but may require a computer's view of all the data at once if that makes sense it does yeah so i'll add that slurm of course has this ability to work with luster file systems through their job stats we found that actually remarkably difficult to turn on and

utilize um well because you really have to coordinate across all your mds and oss servers all your metadata and object storage servers and then you have to uh correlate the the slurm job id with activity on each of these separate machines and then put it back together um so we almost had an implementation but uh there is some stuff in the liquid tool uh i love the definition of the liquid tool like i know what i'm doing li-k-w-i-d uh i can't i don't have a link for that now but i'll look for one and then tak when we were talking about this pointed out that

they actually have an i o limiting tool which i'll put in the chat here that might give people some ideas for pursuit here but we don't have a solution there we'd like to um we're going to try filling out the mpi stuff first and then going back to io um as for what to measure the beyond cpu well just once you get started there are dozens of things just to go back to the bmcu there are 187 different quantities on our current model of dell server that we can monitor through redfish at the time scale of hertz so um if we need to so um

you have to ask yourself what's interesting about that so one of the most trivial ones that i think is very instructive is that you can get from the vmc is but i mentioned earlier the memory power as opposed to the cpu power now you know if you look at the operating system it'll tell you how much memory you've allocated but that's not going to tell you anything about how that memory is being accessed memory talent power is directly correlated with what you're doing to access that memory which is i call it memory churn and sometimes you can get very interesting insights into what the limitations

of the code are without doing a single bit of of code instrumentation just by looking at the memory power to cp power ratio so there's dozens of things you can monitor the old thing old saying right if you're not measuring it you can't use it it's very true thank you alan chris i think you had something you wanted to add yeah i'm going to uh borrow henry nieman's uh phrase it depends uh i think there's a lot of users and by users i mean just kind of users of hpc whether they're doing some sort of administration um on a smaller cluster they've got shoved under

their desk um or they're running a big data center there's a lot of people who are using the hpc architecture to do a lot of work and a lot of them they don't care about most of it what they want to know is is the system on and can i run my jobs and so you're monitoring at that point is the very basics of you know is the system up is it responsive is it working um but there's also a lot of other details that you do start to care about as you start to grow in your resource usage and the number of nodes especially

if you're a limited shop which a lot of people i tend to work with they're you know a couple of admins for a big cluster for campus right so time is limited so the more that you can monitor um and i wouldn't go through and just say all right i'm going to set up all the data sets first i typically grow out um so i'd start pretty small with each new cluster and then add things that i know have bitten me um so you know alan was mentioning uh power we actually had system where we didn't have the calculation right for a power and at

certain load it would trip and we lose the nodes right so now i really care about is my power level at a certain spot um things like uh temperature uh very important especially when you're running a whole lot of different types of jobs right for in a condensed space which some of these systems that i've worked with are just shoved into a closet so like that becomes a huge difference and you really care about that monitoring of temperature at that point so there's a lot of things that it's hard to say here's all the things that you should be monitoring because it's going to vary

a little bit more per group and what information you actually care about so you know i said before i take the approach of if it's bitten me i want to monitor it and if possible automate that monitoring um and again i use zebix so you know when you see this use case fix it and only tell me about that if you can't fix it or your attempt to fix it fails so that's the stuff that i would really want to be alerted on are the critical things that involve me actually having to step in uh anything i can automate and monitoring in the meantime is

great to go back through my reports and say hey this one node it always seems to trip this one thing it fixes but it's always tripping it's always this one node that's the value of the historical aspect of it but it really does kind of depend on how much you want to do and your monitoring tool because some monitoring tools aren't going to get you a lot others you can get a whole lot um you can do a lot of stuff with xavix but i'll be the first to admit it's complicated there are so many tunables that you can add that it takes a bit

of investment but at this point you know i can throw in all sorts of values and i never really worry about the impact of performance generally it's either a cluster that's not being hit hard enough to really bind um or we've already segregated the monitoring traffic to its own network um and then it's not that much of an impact and the value of having that data is more important to me that's great thanks chris i do see one question just popped up from tron is is anybody familiar with elk anybody used elk i've seen elk in the enterprise quite a bit kind of people choosing

that over things like splunk or log aggregation that's what we were trying to do at cu and just struggling to get the stack up and running with the staff that we had in the documentation that's available and mostly starting with the the elasticsearch part of it yeah it's a really powerful tool especially if you're ingesting a whole lot of data uh and you're trying to look for patterns in it i've known a lot of people who are adjusting you know logs from hundreds of systems from different types of systems whether the web servers or data servers or hpc or whatever and they're looking for certain

types of patterns because it is very splunk-like in that regard for the work that i've done with hpc um it's more overkill than what i've needed before um but if somebody is familiar with the tool it can do it um that's a different story but um i would be cautious of just saying i'm gonna jump in with this monitoring tool without doing a little bit more understanding of what elk is first yeah i've seen that one be quite complicated once it's up and running and you get it configured to manage and monitor the things you want it's great but it is it is a learning

curve to get it up and running i've seen people take up here a year 18 months to get it set to where it's actually meaningful to them and actually get something back out of it so great tool it is complicated so we are actually up on time we went over a little bit and i really appreciate john allen chris really appreciate you guys jumping on and helping us talk through this i think we all got a lot of really good information i know i have things to go play with and read from alan from your links thank you very much thanks for the hopping on

with short notice uh guys we will see you next week and we appreciate you joining us today goodbye everyone you