Cybersecurity for Scientific and High Performance Computing

the high performance Computing Center at Texas Tech University and because that doesn't keep me busy enough I I co-direct a multi University industry University Cooperative Research Center in uh cloud and autonomic Computing as we figure what up the web is we'll let you know um and uh that keeps the rest of my time occupied excellent well thank you for joining so the topic today is cyber security and I know Fernando we really appreciate you being here you had several thoughts that you shared uh that we're interested to talk through and I have been watching some things I've been reading some things and I'll bring those

up as we go but I know your biggest thing that you wanted to really discuss and dive into was not as much how we do cyber security and HPC but more of the end packs of cyber security and HPC so dive into it a little bit I know we'll have questions as we go but I think from that view and wanting to talk about it from that perspective why is that top of mind for you so I was thinking about the breach what they call climate gate back in 2009 uh where um email server somewhere in the UK was hacked and those emails were exposed

and um there was a lot of anti-climate folks that use that as fodder for trying to prove to people that climate science wasn't real and you know this had a huge impact on the HBC Center because when I joined the lab I needed to get a clearance just to be able to work on NOAA supercomputers and so the impact that that that had for Noah was the difficulty in finding people that were willing to go through this you know the whole background check process Etc so I limited the talent that could work on the supercomputer it limited the location where supercomputers could live and limit

access to to to scientists because they needed to go through a background check before they can even use the super computer so that incident that had huge impact you know um in in climate science in general was born out of an email hack right it had nothing to do with actual resource itself but down the line the decisions that were made post uh hack uh impacted quite a bit I think of the science so I wanted to talk about like what is the impact of science uh when when cyber security events happen no that's that's great and I know going through that process I have

heard it can take up to a year to actually get your clearance and be able to be a part of that so I can imagine we see quite often that research centers are already having trouble finding people that are qualified to work and then whenever you add something like that on top of it if you're waiting a year that that's a lot of just problem in getting resources so I say mute so hey thanks Rose yeah you know I'm here to laugh at you yeah thank you so when I was when I was reading through and looking at some stuff there were three different categories

that were kind of identified that I thought were interesting and it was confidentiality integrity and availability were there three different the CIA of security and HPC and it was kind of brought to the attention that you can't really run typical malware on an HPC it just doesn't make sense that it would consume too much resource it's not really how an HPC is used so it was more of looking for patterns I think Alan you and I have talked about this before looking for patterns in research and how things are are done is kind of how you have to watch an HVAC the usual pattern is

that people look for like high CPU usage network activity that's that's the goal in HPC right so it's hard to distinguish you know good operation from bad absolutely so whenever you start looking at that kind of thing I mean we've talked about impact how does that impact you but when you're looking at how do you look for those types of things you've got to have the resources to do it the right resources and you have to know what you're looking for so that has to make it very difficult to actually try to do this Jonathan would you agree um sure uh so I've always a

little bit afraid to speak too definitively about you know intrusion detection and that kind of thing because you don't know what you didn't find right and I'm always worried oh there was so much it was so much worse than I realized but the intrusions that we did find uh you know when when I had such purview we're always off the back of some kind of network monitoring because nobody's just getting into your like whatever people could try and remember where malware on your system but what they're trying to get out of it is some output or something that it's doing on the network or or

finding something else on your network or or reporting back to a control server or we've had people trying to send spam out of a cluster before for some reason and and so that's where in in my experience having that close relationship with kind of an upstream or an a larger it organization really helped because our internet gateway went through their traditional kind of I.T network monitoring and security systems and they were able to bring that thing to our attention far sooner than you know a university uh cluster operations group would have by itself sorry I just want to say I have seen formal programs um

launched I will just say it that way uh with the goal to differentiate uh Mel you know malware activities compared to regular hbcus I have not seen a whole lot of output from these projects but there was a one that uh Idaho National Lab that uh um that uh seem to be aimed exactly this space I did provide some links that perhaps Rose can put up later uh I guess if I had only one to pass on verbally it would probably be trustedci.org which is an NSF funded cyber security Center of Excellence that is aimed specifically at trying to meet the the uh the Cyber

infrastructure provider Community where where it lives and uh provide advice on how to you know encourage running secure and customer the Cyber infrastructures it's it's largely aimed at academic centers but you know a lot of the advice is good general advice uh and they run workshops and so forth so uh this trustedci.org is another one called regulated research.org that is just a group of Institutions that support um Research into um uh you know compliance and regulation how can we live unfortunately both of these are fairly U.S Centric very U.S Centric efforts um you know uh this seems to be one of those topics that has

to be repeated in each country or or um you know regulation sphere of influence you know if I were to predict what the EU would do next week I would just uh declare that uh you know all CI must be secure and and find you if you don't or something you know there are different approaches and they're the different uh but these two uh interested ci.org and regulated research.org I think are good starting points for U.S based folks excellent thank you for sharing that out so Fernanda yeah what would someone find if they go to their uh are these like playbooks guidebooks for uh the

kinds of things to look at are there software tools that people will use and to play in their cluster what what are what are the takeaways yes this sort of Interest yes each of them when they started uh oh thanks Fernandez putting a link that maybe can be shared for European uh stuff um each of them started fairly modestly and has been around long enough that they have a wide uh Suite of you know downloadable outputs at this point um so uh everything from what do you do to comply with gdpr as a U.S Institution to uh you know Workshop series reports and so forth

um so uh I don't know if we can get the links uh into the notes when they're when they're when this thing is put to the web but that's those would be good uh to include we absolutely will and we've actually posted them in the chat already so excellent people can follow along with those and of course we'd like to hear back from people who might have other resources on their own right so certainly if you have any please share them with us so now back to Fernanda you have seen a lot of different centers you've seen a lot of different scientific Computing over the

last several years how is this changing oh you know um the primary concern when I was in government was that um you know even though we're open science or you know by definition HBC centers are generally open science right so they're generally open to the public and IP address that's generally something that you can dedos or whatever right so it just is an impact of like stopping you know basic science from being used that that's the first the easiest I would say the probably the easiest thing that could happen but for for a center like Oak Ridge we still have some export control um you

know projects right that that weren't there and you think like well what what's the what's the motivation for somebody to hack super Computing Center well number one just a lot of them may be young and just for the fact that they can say they could um and and for the exercise of it and it doesn't really matter ultimately what the reason is but for the super Computing centers that are in the US export control is a really big deal and getting that kind of information especially now as we're tightening export control information to say going to China right it have to do with materials research

or any sort of technology that would support their own development of their own ships Etc um that could really that could be a really bad because once that info is gone it's gone right so it's not necessarily that the impact is going to be to the general public uh in the sort of pii typical sense that we hear on cyber security breach but in the link that I shared you know within the chat here talks about a hack event that happened in the UK and it was was it four or five Super competing centers that were hit and what do they do they took it

all offline to go figure out where it was coming from and he's almost always either related to foreign entities or just again somebody just trying to see if they could um it also reminded me of the there's an article once that that came out and wired um while I was at the lab that was post Snowden um you know data collection Etc and um in that wired article and mentioned data data pipeline going into Oak Ridge uh it didn't mention which lab just if you know that there you know at least at that time used to be three Labs now there's two labs in Oak

Ridge one is high security and an sa the other one is open science and it didn't differentiate in the article which one it was and I had a whole bunch of my friends saying I thought you were doing science what are you doing with my data and we're like whoa we're doing science this is all open science but uh because it didn't differentiate because it is a government entity because many places are many HBC centers are hosted by government entities there is the attractiveness to you know put those super computers down their most powerful supercomputers in the world um and again it goes back to

impact what happens when a bunch of scientists were talking in the thousands like nurse is a center that has 8 000 users what happens when nurse can't you know allow 8 000 users to log in on a daily basis that that's a huge impact to science so are you seeing more people try to put things in air gapped environments which makes it obviously a lot harder for open science I mean I had the pleasure of working in one air gap environment and it was really painful every packet you wanted to install had to go into the special image it was completely isolated from the broad

internet anything that wanted that you wanted to install had to be hand carried into the room um and had to be pre-verified by somebody else and verified that everything that we were in stone was you know okay uh because of data that that existed in there was very sensitive pii data having to do with medical data and medical research that we were doing so I don't know that the answer is air gapping uh APC centers it would totally kill the whole vibe of open science which is what most of us are doing but for some of this extra AI work that we want to do

and and problems that are bigger than us like healthcare um like climate um I I can see where this might need to happen more often um I don't know that that's you know going to be the case for most sort of University centers but certainly for government entities that want to take this problem really seriously it might it might need to be the case yeah I think we should maybe talk about the low-hanging fruit things that people could do if they haven't done already I mean first of all uh the next release will open SSL will drop support for many of your favorite keys and

algorithms you should be looking at that if you have SSH exposure put it behind uh you know a firewall or not not just a fireball a uh a Bastion host or you know other intermediate login VPN those are things you know I used to be on the camp that oh it's SSH just leave it on the network and I'm not in that camp um so these are things you can do yourself uh uh don't make people change their passwords every six months that's stupid and and this test a document that you can give to your boss that says that's stupid so don't do that um

you know there's some low-hanging fruit and people can and it should and I think this is always the case in in issues of uh security there's some things you can do quickly that will get a good fraction of it I don't know if it's all or you know a large portion but you know you can cut your attack surface and um you know everyone wants to get rid of passwords and so uh we are all on that campaign but um uh the level yeah what will VPN do to all of the workflows right so CERN has a workflow that sends jobs to the entire world

so we can't be we can't put HPC Center in behind our VPN well so yes since I helped you know create the grid I'm sensitive to what you see but what we've done in Grid Computing for example is that we've gotten rid of the original x509 based uh uh short-lived certificate uh uh authentication mechanism and gone to more modern oauth 2 based workflows and yes so a lot of the things that you used to need to keep we used to have this approach we still do it called uh science vpns you know science uh uh dmz's I'm sorry that was the phrase science DMZ the

idea was that large-scale Labs would be running their research networks and they could expose their research networks to each other for traffic flow if you go back far enough this is the you know the original uh deck net across the country before arpanet came along and uh so forth it you know in my experience um the one of the first use cases of that was uh advisors spying on their graduate students to make sure they were working at 11 o'clock at night uh so yes uh you can actually put many of the infrastructures that we're used to behind more levels of security and even better

switch to um distributed workflows that use modern uh cryptography methods um I actually fought the x519 reversion uh because if you look at cloud computing one of the most active projects with the biggest uptake in the uh uh the the cloud infrastructure is something called spiffy SPID and its implementation is called spire and this is an entirely x519 based but very short-libs certificate-based workflow for uh doing what we used to do with grids but that two orders of magnitude faster speeds and uh probably three orders of magnitude more control so it's not the technology it's how you use the technology uh when we started the

grid we wanted to issue a an x519 certificate to every researcher so the thing that turns your little icon lock icon green when you visit your bank we had to issue that certificate uh to each researcher and we had to have a renewed you know and the person had to show up or the government issued photo ID to the representative others they hated it but these are actually uh now highly automated and available Technologies uh so Zayn's heard me tell this joke before but you know we we started the grid with the idea of uh protecting the world's biggest supercomputers and we put Security in

the Forefront and Along Comes Jeff Bezos and he says oh you want to be root what's your credit card number and so it was you know and he won right he's very rich and I still work for a living so um you know it's it's sort of a commercial aspect of the convenience versus security approach thank you for that Rose do you have a question yeah so I I actually just saw a uh a post I think yesterday from the resf um the rocky enterprise software Foundation if you are not aware they are amazing and are very happy to help meet them get them at

um where's the best place that matter most right Rockies matter most so maybe Jack you can pop that link up there as well um it's so funny like you know after like you've been doing something for a while you just assume everybody else knows what you know and then you actually start talking to people it's like no people are busy giving their other their own lives right like nobody knows all of these things that we know because we hear it every single day so anyway Rockies matter most Jess will put up the link thank you you are amazing get involved with the community they are

radical and when I say radical what I mean is they kind of have a little bit of like a competition of like who is going to answer the question the fastest and the best so if you need that go there so this is my question what is Rocky doing for security I saw a post the other day where I was talking about Rocky Linux and how they're approaching security so I don't know if any of you guys really have like that in-depth kind of knowledge but I think that would be really cool I mean obviously we're here you know this is ciq's channel so we

love you go to our website you know reach out to us we are happy to talk about it so there's there's three different open source and so we're talking about security Fernanda thanks so much for bringing this up such an interesting kind of like you know full view Twist of it right because like like obviously we all want to be secure we all want to be safe we want our data to be safe we want our systems to be safe we want us to be safe like we want that but then at what price right so you can lock everything down and be super safe

alone in your room or you can be totally free so like open open source project so the three of them that we support at ciq the rocky the werewolf and the obtainer like wow like how do we take those and then and then create some security around them so that's my question here like how does rocky address security how does obtainer address security how does werewolf address security maybe Jonathan you probably have some some knowledge about that yeah so I one one thing uh that my my good colleague uh Dave godlove would like to point out about app painter for example um is enhancements to

let's say Supply Chain management where where your code comes from and whether you're running what you think you're running that you get with obtainer because of its support for mostly certificates or yeah assigned uh containers that you you can sign it when you create it and then you can verify that signature uh cryptographically when you receive it there's also support for encryption though that's that's really a different conversation here um but could be uh you know could be part of your security strategy depending on whether your your code is what you're trying to secure rather than the resource um Rocky licks you know is is

a it's it's a fully featured modern uh Enterprise Linux with what you would expect so uh we have a secure supply chain there too and you can run it securely or in securely as you as your practices allow um and where where we try to contribute there is in that Supply Chain management again making sure that the packages are what you expect them to be and that when you download a package from Rocky that you can trust it or from ciq through one of our add-on packages um with werewolf werewolf is more of a a classic HPC internal service so there's some security measures in

there of course um for you know making sure that you're delivering configuration to nodes that you trust and things like that um but for the most part when you're dealing with werewolf you're dealing with something inside of the security perimeter more than something that would sit at the boundary and that would be part of your overall security uh conversation foreign is always interesting to me having an encrypted container that will only execute if it's got the right key side so I think it's very cool and from the encryption side of it um just in case someone in our audience doesn't know um the the idea

there is your your container on disk or or at rest is only ever encrypted and when you provide your encrypt your decryption key um that decrypts it only into memory so the the decrypted container Never Lands on disk anywhere which is good if you're trying to protect the contents of it yeah so when it leads to another question that hasn't come up yet but I think is relevant here which is that the supply chain um software supply chain uh security and uh uh so one question is you know how do you make sure that the applications themselves that you're running have been uh uh compromised

and uh so uh having you know good CI CD practices I think there were some examples recently where was it pie pie itself got uh got actually compromised yep and uh so having a CI CD chain and you know um the ability to roll back to known good versions and uh you know control of where you get your software is important very thank you Alan so do you have a question yeah Dave de bonus coming in strong hey Dave some facilities employ Dev secops pipelines to try to detect malware prior to system deployment do you feel that this imposes too much restriction on the general

HPC user limitations on certain SWS it's swm good some degree it's the point I was just making that you have to have an organized pipeline uh I don't think you'd have to give it a fancy name um you uh as far as detection um I think what we've seen in practice is that people flag open source software more quickly or at least they flag it quickly let's say um if if they observe bad behaviors um you know it goes back to the database software that they used to tell me keep your monitoring off my machine I'll tell you when it's when it's bad and the

users will definitely tell you when it feels bad so you know uh you have lots of eyes on it um that can help I'm not sure you can do automated malware detection uh these days the injections are pretty sophisticated uh I used to read my dad's flying magazine to crash reports that when I was a kid growing up and if you could read some of these recent intrusion reports you you find that it's not going to be a simple thing to go looking through or some uh code that says malware here you know but do bad stuff you know so um detection I think I

think it goes back to the supply chain topic that uh that having the ability to control your pipeline and say hey this version got compromised I'm rolling back uh may be helpful it's going to follow up to that before we go to this one I know it's your favorite topic Alan but how do you think AI will help in that or do you think it'll be worse quantum oh it's Quantum oh that just makes my head hurt no I will disagree with Forest here I think um uh the MPI has gotten more and more generalized and we even have code that lets you uh run

uh you know tools that were compiled with one MPI on a different MPI so I'm not sure that it it's clearly the case that malware and HBC is uh is MPI specific uh I'd be happy to be proven wrong I haven't you notice how carefully I avoided answering the question about AI I did I have a hot take on this one actually I disagree with Alan uh and and actually wrote down notes so that I can get this right so [Laughter] I think the common the forest made is really interesting uh you know as in HBC malware fails because it's built on different open NPI

but a couple of thoughts one we've been trying for years and getting it right this convergence of HPC and Enterprise level software right we're starting to see things like kubernetes running on HPC sensors and containers earning on hvc centers so there's this convergence happening and those same targets in the Enterprise those same tools are going to work uh if that converter becomes uh you know greater right so that's one two we have the kernel the Linux kernel moving um to more privileges at a user level right so at some point HPC centers you could not install python uh packets uh at you know in your

user in your local right slash local uh but later 3.x kernels allows you to do that and and you say well sure but there's more of a separation with the kernel and it's much harder for you to get Roots privileges but I counter with you do not need root privileges to bring an HPC Center down anyone that has run an application that writes 20 000 files to your luster you know a storage nose you can grind that to a halt so you don't need that kind of sophisticated attack to disable an HPC Center doing something poorly right and you may be able to disable it

for several hours and in the case of that European attack which was specific to like uh HPC centers that were running covid-19 research it doesn't take a lot I mean it could have taken there were people within the lab systems that were refusing to return to the labs and return to work because it refused to take the vaccine so that could be internal players it doesn't even have to be external entities all of this to say you don't need the kinds of sophisticated attacks to Enterprise systems to disable an HPC Center uh you know even temporarily yeah thank you is to to DDOS in HBC

Center I think that's a unfortunately a relatively compared to other things you might be trying to do relatively easy thing to do yeah I'll I'll uh admire the way in which you slid from disagreeing with me to making an uh irrefutable point about uh how the average single new bhpc user can bring the cluster to its needed so it's artfully done uh so yeah I was looking for a link to a nice tool attack uh Texas Advanced Computing Center where they they actually had the ability to put individual iOS models on uh individual users or jobs uh I don't think I can lay my hands

on it in time but we just say there are ways of of mitigating that behavior even short term if you're trying to chase down a problem um but uh yeah so I'll go back to the point that forced me was making about MPI I'll just I'll modify my answer to say it's possible to make an attack work on a different MPI than it was built for yeah we also need to remember that HPC systems are highly coupled right that's the difference between a cluster and an HPC so one even a small part of of an HPC machine can disable the rest of the HPC machine

we've seen that with um you know a network that should now be named shall not be named because they've been having some major issues with that network uh you know hint in Frontier and that and it's has been very painful so you don't again you don't have to be very sophisticated in an attack when it's a highly coupled machine you can you can do some damage oh good hey Greg uh so now that HPC 2.0 is starting to use containers more um AKA fuzzball is that complicating or simplifying Security in this space all right Jonathan you're the fuzzball guy I'm a great Jonathan question I

think it's mostly just changing it one way that I really like the uh um the impact to security here is it it It's Pat to say containerizes but it gives you the system's side where you can run you know Dave was talking about a devsec oct type SEC Ops pipeline you can totally validate your system side and then you know that everything that an end user is going to bring in is coming in these neat little containerized packages and you it's at that point easier to imagine building them with a with a CI CD pipeline inspecting them through a common process and you know very

clearly where that boundary is when uh the user applications are spread out all through a shared file system and users have access to the same executable code and things like that I think that the dependencies make a lot of the auditing of that more complex but it also means there's more moving pieces and there in my mind it's simpler to think of it and have all these pieces separated and audit them individually and apply different policies and different practices to each part of it and there's there's a big line for me between auditing and securing the infrastructure and auditing and securing if you care about

it and use your applications um but it is different and whenever there's change in practice you have a learning curve and you have potential for new problems that come up from unforeseen things or just people who aren't used to doing it the new way but who's doing the audit though um contain you know one thing is for you to download an application that's on GitHub where the source is all available but a container now becomes you know a a little Trojan Horse of of you know possible death of a supercomputer right even who's doing the audit on what applications are inside that container what sorts

of data isn't you know is there and uh I don't know that any HPC Center can manage like how does a nurse do that with 8 000 users and typically you you well I'd say a common workflow is to build a container so that you don't have to bother with dependencies changing so you know you you do it to somewhat isolate yourself from updates and so you build up a collection of containers all of which are pretty much guaranteed to have old versions how does that uh fold into your security posture so these are complex topics I worry that we're getting a little theoretical though

uh just wonder if people can let's go back to the question we asked Fernanda you know can uh can we think of some examples of of compromises and breaches that we each encountered um I have a couple of but uh I'd like to hear from others yeah well I got 60 bucks in the mail from Yahoo that was pretty awesome I didn't even know it was coming but apparently there was some breach years ago and there was a lawsuit and I got 60 bucks so checks in the mail are a good thing um but for me it was good probably not for them oh so

a good story yeah thank you thank you um but yeah that's a that's a great question what are some like uh you know specific examples Fernando you shared one do you have more uh I don't have anything that would be specific to an external cyber security incident coming in I have a lots of examples of you know squirrels getting on power lines and raccoons becoming ground next to some Transformer bringing bringing a center to a halt lots of human specific mistakes but none that I can think of that were specific to at least the centers that I've been privy to or had access to so

there was a no there was no hacking incidences that I remember in my in my career so the most common uh compromises I'm aware of are have all been through stolen SSH keys and and SSH compromises and this is you know good software I'm not bashing on uh on ssh in particular but uh uh the fact that most super Computing centers uh have no mechanism in fact there's no mechanism built in to open SSL to let you um require past phrases on SSH keys this was one of the features of grid security we we like we liked which was that you couldn't use the grid

certificate directly you had to use a Time limited proxy that required you enter your passphrase and there's no way to get one without having a passphrase on your proxy Generation Um so right before I took over um uh the hpcc here that there was a really bad hack um where someone got in through a vulnerable version of open SSH and leftward kits all over the cluster and uh it took weeks to get rid of all the stuff fortunately it had a fairly obvious signature and it was well known it wasn't one of the new obscure variants um but I think that again I'll go back

to what I said at the very beginning if you have SSH exposed to the outside world don't do that put it through a a proxy Gateway or VPN or something and uh this will just give you that much more um protection um you know there are Advanced products in this space for managing SSH keys so we haven't talked about this but you know every cluster needs uh someone who can be root uh and uh assuming you don't do that by the credit card mechanism um you know how do you do it so you have a team of people there are products that are aimed at

uh you know uh better housekeeping for that set of SSH keys or access mechanisms and you know I won't endorse one or the other room we don't actually use any of them we have our own home built thing um and uh So speaking of like RSA tokens I do have one example um when I was a we last at Oakbridge I was answering a ticket somebody was mentioning that they um they had this fob that someone had given to them and I assume that first that someone was their advisor in other words they got a little packet they got a little envelope and then the

advisor distributed these RSA bods and they locked themselves out but the email that was coming from didn't match a username and then you know one of the things that you have to sign in these centers is you're not going to be sharing your RSA fog and this person indeed was sharing it with another graduate student now they were just they were just trying to get their phds you know work done right innocent enough but it was a No-No and once I caught it I had the responsibility to go report it and then I had to walk over to my boss and say there's fob sharing

Happening Here we had a phone call with a person and indeed there was Fox sharing again these are human factors right this is not like external hacking this is just somebody trying to get their research done trying to get their data done and instead of requesting a new user account they share the cloud yeah but that can lead to all kinds of other issues too social engineering obviously is a very large thing it kind of goes back to the Integrity part of this what how are we protecting the data I mean the data sits and sometimes it'll sit for years that nobody touches it looks

at it does anything with or interacts with it so how are we protecting the data on these systems well I just if you let me before we go on to that very interesting topic and I'll bookmark it uh that's not this the opportunity to social engineer your boss and get those uh fobs replaced with uh individual tokens and smart cards and uh you know I've gotten a lot of nice equipment put in to improve the security here by simply filling out the nist evaluation and turning it in and saying you know it's an action item now you have to approve this budget item but I've

requested for two years so go back to protecting the data but yeah don't miss the opportunity to socially engineer better security by scaring your boss out of his mind otherwise for a long time we weren't even allowed the electronic RSA tokens you know the app essentially on the phone which I think in my opinion would be more secure because you know that phone is my phone I'm not going to hand my phone to somebody else uh and but you know they insisted on the the physical fob for a long time did you have a whole keychain of them am I right around I had like

15 16 of them and I never knew which one was which in the label I had the lanyard too so it kind of Jingles on the lanyard as I walk by people kind of you kind of could tell people by the jingle in their lanyards yeah the good old days Oh I thought Jonathan was finding a lanyard never mind he's about to pull out a whole bunch of RSA tokens so protecting the data is there is there I promise to come back to it you you raised that topic no I just I was asking is there something specific we do around protecting the data open

to anybody who wants to take it Jonathan looked interested I I just I don't know what to say Beyond normal linuxy stuff right any scientific data store that I've ever been on is a big shared file system of some kind with Unix permissions on it and that's as far as it went to us and you had to get to it through either you know a login node or a Globus endpoint um so maybe that counts but and then you know Globus does the whole grid security thing so that's to it but I mean so I mentioned the regulated research.org site and one of the things

that uh specifically pursues Fernando brought up the uh personally identifiable information aspect there's this whole category of non-classified but uh but sensitive information and uh so uh that would health information would be an example and uh there there are circumstances in which you you as a as a cluster operator need to set up different file systems different protections to to protect that sort of thing and that's exactly what that Community the regulated research.org focuses on how do you comply with with these requirements in a practical way and you know there are hundreds of of Institutions involved so uh you can really learn from from each

other that way we actually don't do it in my cluster we say if you FBI I talk to the Health Sciences Center across the freeway there and they they can do that for you thank you Alan so we are running close on time let's throw up our last question from Frank I used Rocky 88 now on my server I run firewall firewall D and fail the ban is there something else I can do to harden my server this could be a whole Topic in and of itself I'm gonna answer first and say turn off any other services you don't need it's always the first place

to start in my opinion anyone else thoughts running firewaldi is not the same as having rules right so that's true study study the rules uh I keep pointing out there's no such thing as a secure port number there are simply secure protocols on each end of the communication right so only support the communications you want um turning off a port number doesn't save you anything if you have you know other traffic on other ports yeah this is a long-standing fight between me and the university uh I used to say that um you know you know the old saying that painter is the natural enemies of

electricians you know they're always painting over the sockets and stuff firewalls are the natural enemies of grids I used to say but uh that's because we weren't sophisticated about it uh you know there's sort of No Limit you can go to Advanced uh protocols that you use port knocking it really depends on your setting and how you're exposed the quickest way to get more secure is to make sure that nobody else can log into your server except the people that you want to if you're running in the cloud that's a whole interesting topic we need another webinar on that absolutely Jonathan just when you want

to add yeah I really like fail to ban and stuff like that so many intrusion attempts are you know they require a lot of time and attempts and something like fail to ban uh which if people aren't familiar uh watches logs and other data sources to see failed login attempts and then creates firewall rules or other Access Control limitations that cut off and attack before it has a chance to uh to get in um there's another one that I haven't used yet but that I'm really interested in I think it's called crowdsack that's what I'm looking at right now um that uh that does that

but then also publishes the the results of it so if if your server discovers an attacker at an IP address it publishes that information to a website that other crowd sex servers can can subscribe to and so you can ban attackers before they even get to you I like that I just like I I imagined in my mind Banning attackers yeah so anyway that was my ninja move for the day you guys I think that we could go on this topic for ever and ever and um Fernanda thank you so much for joining us I think this is the first time I've met you so

it's really nice to see you really appreciate you being here and thank you for disagreeing with Alan because that's awesome all right so you know we are crq ciq we are so happy to be here again you know we give the the tier three Enterprise level support uh for Rocky uh Linux for werewolf or aptainer and then we've got some other really cool products we'd love to chat with you about it was mentioned here HPC 2.0 so if that Sparks some interest it's fuzzball go check us out on the website ciq.com make sure that you like you comment we answer all comments and questions we

want to hear from you we are so so grateful that you were here and Zayn thank you very much Jonathan good to see you guys are people who want to continue the conversation do we have a security channel on the slack for hpc.social go to hpc.social click the chat chat button and join the channel you're in awesome we'll grab that link and put it in the comments as well for you guys cool absolutely all right we'll see you here next week same time same place thank you everyone [Music]