New Warewulf API & UI Webinar April 24, 2025

Awesome. Yeah. So for just real quick for people that don't already know and presumably if you signed up for a Warewulf webinar you know this but uh Warewulf is um a cluster management system open source uh developed in the community and then supported and and we also contribute uh development from CIQ um and you know it's as with most of these tools it's very command line driven um you you SSH into a server and use a Warewulf control uh system to access it as we've tried to get it out in front of more and more people and especially tried to to get it kind of available with your cluster when you first get it.

One of the first barriers that we've hit uh when we talk to system integrators is does it have a web interface and um you know there's there's like a a perspective on that that's uh well no web admins don't or sorry system admins HPC admins they don't use a web interface that doesn't need one. Um but for one like a lot of times a web interface like this can just serve as a proxy for um the maturity of the project and and the the capability of the the community and the pro project and the the company behind it. So there's a little bit of that just like fit and finish that it implies for the project and Warewulf is it's getting really good.

So we want to make sure it's putting its best foot forward. But I also have a strong opinion about interfaces like this making a product more approachable and more uh discoverable for new users or even for established users that um that have been using it for a while but might not know about new features as they come in because maybe you don't go dig through all the documentation to see um to see what's new when when new uh new versions come out. So, we want to use the a web interface to highlight the existence of features and and how things fit together and and make it almost an extension of the documentation.

Um, so yeah, how that actually looks in the UI is like that little circle with the question mark next to things. Yeah, there there's some of that. Yeah. Yeah, I like that. We'll get I'm also not an admin, but it's helpful. But I am opinionated about interfaces and and I mean like how different systems interact with each other. So um I wanted to make sure that if we were going to build something like this, it wasn't built in such a way that it was going to be like a pain to administer or a pain to uh to integrate or or for us to keep maintained with the upstream project over time.

So, uh, the first step for that, uh, was to make an some kind of programmatic API that, um, that we could then use and build from in building an interface. So, that's the first thing that we want to show off. And so, Warewulf 4.6.1, which came out just at the beginning of this month, uh, introduces a new REST API, uh, for Warewulf. And this is kind of our our second attempt to create an API for for Warewulf, but I think this one this one's going to take off if we have a downstream for it now. Um, let me pull well. So, uh, yeah. So, it's if you have Warewulf 461 today, uh, then you have access to this, but you might not know how where it is.

Oh, trying to find my share button. Here we go. Here. And Oh, here. And this big green one with the arrow. Yeah, the the big green one. Okay. Now, too many windows layered over each other. Totally. So, um yeah. So, if if you have uh Warewulf 461, you have access to this API, but it's disabled by default. And that's because like ultimately Warewulf runs like the all of the stuff that Warewulf changes on your server. It's like root level operations right now. now. And this something we might want to not not might want to that we want to clean up in the future and might be able to clean up in the future where where maybe the Warewulf server can run more unprivileged.

But for now, the API is mostly a thin wrapper in front of some relatively privileged operations that occur on your server. So we don't just want to have you upgrade your server and suddenly gain this new uh um potential ingress security vulnerability into your server. So the first thing is that it's it's disabled by default. And if you go to your server configuration, you can see this window too, right? My my little terminal window here. Great. Uh so if you go to your Warewulf config file, this is at /etc/warewulf warewulf.conf. Oh, not not cd, but there we go. Um we can see that I have this API section down here at the bottom.

I think the default config file won't have anything in here, but as documented here, if you put enabled true, you will gain a web interface or or sorry, a not a web interface, an API. And the uh the the second thing to be aware of is even if this API is enabled, by default, it's only accessible from the server itself. So, other processes on the server could talk to that API. Um, and that's what we do for the web UI that we'll demonstrate in a moment. Um, but that's again the the default is for it to be as locked down and secure as possible. So, uh, for my purposes and if you want to use the API from outside the server, you need to tell it what networks are allowed to talk to it.

So, these two settings are default. These are different uh expressions of the local server. Um, but if you want to add other servers or other networks, you you add it here. And again, that's documented here. So that's that's the situation that I have here in my development environment. And one more step just before we're able to show things. Thank you for that, Jonathan. Thank you for explaining that and doing this work behind the scenes because that issue is going to come up and one of the things I was thinking about. So, I really appreciate you highlighting this and and making sure that people know not just how to enable it if they want to, but that it is not automatically there.

Thank you. This is one of the best parts of getting this the community because the first uh the first submission of this didn't have it this way. And in fact, we didn't even have filters like this uh in the initial uh the poll request, the hey, here's a new API, please incorporate it upstream. And it was through the community process and review with the technical steering committee that additional security controls were added and it's a huge win. Uh so yeah, I'm really glad that those got in there. The next step and this was always going to be part of the API. Again, it's a largely root capable API so we want to protect it is just having it enabled and just having access granted is not authenticating to it and you must still authenticate to it.

So, uh, in this REST API section in the docs, there's a whole section on this new REST API. Really, we should also have this API configuration section in here. So, apologies for splitting it across two different places, but there's this REST API uh, section that tells you how to set up this new configuration file in the same directory, warewulf.conf. And this is where you can define users that are able to access this REST API. And so we we ship a a default file there that's like doesn't have a password and won't work at all because there's no password. Um but there's information here for how to generate a password hash and how to update this file.

It's not integrated into the command line experience so far. You have to go edit this file directly. But in mine um we can see oh not Warewulf o.com. Ah there we go. So I have a single user and admin user and it has a password hash here. Uh this hash is is the super secure representation the password admin. So again just for the development purposes but you should put a secure password here. All of this to get us to this point. So for automation and and this really like an API here opens up so many uh opportunities for Warewulf. We have hopes and dreams for integrating it into our fuzzball product for example in the future.

Um or you could integrate into it as part of some kind of CI pipeline. Or maybe you have a larger um uh like data center management system that you want to add a node into that system and have it automatically put that node into Warewulf. You can do that now with this API. Um all of that is through the same Damon. There's always been a Warewulf damon running. uh now it just grows a slash API directory or or path on that Damon when the API is enabled. And then we also have this slashdocs uh path enabled um which is an autogenerated little tiny web interface for this API mostly as a a documentation exercise a self-documentation for it.

So we can see for example uh that if we go to slap API/nodes uh and then it gives us a little example here of how we would access that API with curl for example um then we can get uh we can get JSON it it it outputs uh JavaScript object uh data for that. So this this one's already run but we will click this can this try it out again and hit execute. And when we do uh this is data out of the live system, my live text test system. We see I have a a node N1 and it has these three profiles and comments and images.

And if we scroll down far enough, we'll see the next node and the next node. Or we can roll that back up again or we can get a single node. And this interface lets us try that as well. I can say I want to see node N1 and execute and get just the data for that node. So again, it's we don't necessarily expect people would use this interface. This is certainly not the web interface that we're going to show off in a moment. Um, but this is some nice autogenerated active try it out level documentation for the API if you want to build your own automations or use this as part of a larger scripted solution in your environment.

Awesome. I think that that overview answered some of the questions, but did you see in the chat and the Q&A there's a couple of questions? I haven't, but I'm looking at right now. So, okay. Uh I'll kind of go from uh bottom up. So, can other methods of O be used? For example, JWT. So, right now it's just it's just B uh HTTP basic O with a username and password. totally open to other forms of authentication, but for the first like this is the the first commit, the first uh introduction of this API, this this is all that's supported today. Um I I wouldn't imagine it's too difficult to add in other authentication schemes, but for now, this is what we have.

Um yeah, and so next one up, can it o against an LDAP or Microsoft Active Directory? Um, this one I still know right now because it is just literally um authenticating against like the username and password in that file. But one conversation we did have as part of getting the API in in the first place was authenticating with PAM. That's the pluggable authentication modules for Linux. And that I think is a really easy next step of supporting PAM authentication for this. And then from that we could easily authenticate against uh AD or or another LDAP. And uh yeah, so those are the questions so far. So there's actually a Q&A section.

Oh. Oh yeah. I'm not looking at that one. No, I I know it's a it's an an additional one. Um so do you see it from Sumit? I have just found it. What does this REST API for Warewulf do? Oh, okay. So it uh it one of the things that we've been trying to do on the community side of Warewulf is kind of simplify the top level object model for Warewulf. There have been a bunch of like little hangers on and right now most everything inwolf is a node or a profile or an image or an overlay or a file within an overlay. And so uh the basic upfront um capabilities of this are to create, retrieve, update and delete those four things as well as some other um operations on them.

So the a big one is you can build the overlays for a node. That's a thing that uh that is a a relatively common operation. Um you can uh import an image which is it's kind of create but it's a little bit different because it supports like putting uh pointing at a registry and pulling the image into Warewulf. You can do that through the API. You can also build your image uh in Warewulf which means like taking your image and making it into a format that is suitable for provisioning out to compute nodes. So all of those operations can be uh provoked through the API.

There are more that even we for the web interface are going to need and so there will be iteration on this API over time to add additional functionality but what's there we're hoping should remain relatively stable and uh would just have additional endpoints or additional verbs that you could uh attach to the API over time. Awesome. Okay, so this is going to be the last question before we move on so that we um can continue. We have like a little list of things we want to show you guys. This is a good question, Alan. He says, um, do you have an open API description for the API?

So, we are building it with an open API library, but I I don't know if we have that. Oh. Oh, it's probably this. Here it is. So, I think it's just this definition as linked through this uh Swagger documentation that's coming out. I'm not a huge open API expert, so if this is not what you're looking for, let's talk and let's figure out how to get you what uh what you're looking for. But I think this is probably it. Um and again, that was just available as a link um right here uh in the web interface. Ah, he says that's it. Agrees. Thanks, Alan. All right.

So, all of this all of this was in service at CIQ. uh um was in service of building a web interface for the purposes of making Warewulf as a product more approachable um and easier to support uh as something that would be integrated into an overall cluster management uh solution uh both within CIQ and with our integration partners. And so uh I had this or you I I I was making a suggestion that a good way to do that would be um through the cockpit system. So if people are not familiar cockpit is another thing that's already available with Rocky Linux uh at least nine I think I think it's just an eight as well but all of my testing has been on nine so far.

Uh so um it's often installed by default. It's probably not running by default, but uh if you start it and it's just a service, then you can log in to cockpit on your server uh by default at port 9090. And you log in with your your main server credentials. And I can only imagine that this supports like other authentication types depending on how you have your authentication set up for the server. But um right now I'm logging in with my standard uh Unix password on this system. There we go. And it's got all kinds of functionality. So, uh you you know you get like nice graphs and things.

You can see what's going on. You can manage packages on your server. Um I I just enabled this Damon down here that's uh tracking memory and CPU and other utilization metrics over time. Um one of my favorite things to do with cockpit is to just straight up you can use it to manage virtual machines on the system. So, if you're used to something like um VMware or even Proxmox, if you just want to manage virtual machines on one server, um it's very easy to do that all from here and not need a whole other system for it. You can just log in and manage virtual machines on this server through this interface.

It's great. Uh but because all of these things existed, um we we thought, well, perhaps we could just make a plugin for Warewulf for this. And that that has two benefits. one is that we don't have to introduce a completely new Damon, a completely new interface, a completely new paradigm for people. Um, and we get to build off of this existing community platform. Uh, but the other is that it encourages people to adopt and use this platform. I think cockpit's great. Uh, I think that people would be served well to to use it more. Um, so, uh, we wanted to try and integrate this, uh, this Warewulf web experience through this interface and we've done that here in this Warewulf tab here.

It's relatively simple today, but uh, in in certain respects that can be a good thing. Just trying to get the data for your system to you uh, in a as transparent and useful a way as possible. So right now we have those four entities that we described uh kind of at the top level of Warewulf's object model. We have nodes, we have profiles, we have images, and we have overlays. So if you're if you're not familiar, um Warewulf organizes the systems that you're going to boot as nodes. It calls them nodes. And then profiles are a way to uh they they they behave very much like nodes except for they aren't actual nodes.

You can configure them however you could configure a node. and then apply multiple pro a profile to multiple nodes and multiple nodes to a given profile. Uh or it's many to many. Uh but if you don't know that, if you're new to Warewulf, again, one of the things that we really want to emphasize and want to use an interface like this for is discoverability. So, if you log into this and you don't know what a node is, you don't already have five of them defined or or uh um you know, it's a new environment for you, we're trying to drop uh help text in here that tells you what you're looking at, what it's for, what it does, and give pointers into the community documentation, too.

So, I think this one Oh. Oh, no. Maybe if I uh um open it in a new tab. Yeah, there we go. It just didn't want to open it inside of cockpit. I'll have to look into getting it to do that by default. So, um, give you pointers into the upstream community documentation from this interface for how to learn more about this part of the system. We can learn more about how Warewulf boots, this is all community documentation that's out there, but given the number of questions we get about it, um, it's maybe not as discoverable as we would like. Uh this makes it right there front and center how how nodes get booted, what they are, how they work.

Uh similar things for profiles. We can learn what a profile is and how you use it, where you learn more about it in the documentation. And then if you want to go and edit one of your profiles, you know, they have a number of um of configuration parameters in here. There's going to be more. This is one what we haven't said today, maybe not yet. Uh is that this is not done. Um, we're we're putting it out uh for our customers as a tech preview uh for people to get access to and provide feedback on and see how well it integrates into their environment. Um, and one of the things we're still working on is filling out these editors to have all of the available metadata that you can configure on a node in a profile.

But everything that we put in here, we want to uh again have little help text in here that tells you what it is, what it does, why you would make this setting, how it's meant to be used, and and make again Warewulf configuration, Warewulf uh use cases more discoverable and approachable for new users and people that just don't know about this specific feature. So we can I talked about this earlier but we can build overlays for a node uh or we can build overlays for all the nodes which again is a relatively common operation for uh for a Warewulf cluster. These little alerts go away over time but we can close them also.

And uh we can import an image. So I have three here already. I can see what kernel is in that image. If this image had more than one it would list them all here. But each of these only have one right now. Uh, and I can import one. So the the one I usually do for demonstration purposes is just to import an Alpine image in here. And I can import this direct from DockerHub. Now an Alpine image isn't going to boot immediately because it doesn't have an init and it doesn't have a kernel. Um, but just, you know, you can import from any registry.

you can import from the the container registry that we publish Warewulf node images for uh on GitHub as well um just directly into this interface and then uh build that image for use by Warewulf as well from here and then finally uh over here in overlays uh if you're not familiar if you're coming from Warewulf 4.5 let's say you might be used to there just being kind of two really big overlays in Warewulf 4.6 six, we broke that up. So now each function within the standard what we now call distribution overlays uh is represented as a discrete overlay. So they can be enabled and disabled uh individually.

So now we have a network manager overlay and a an FS tab overlay and and other things like that. But one of the things that people often don't know is supported. So overlays can contain templates. I can see that template here. Oh, let's get the uh the managed one. So this is the file that gets distributed on a network manager enabled system to configure um to configure network manager for each network interface that you want to configure. And you know it's a it's a Golang text template uh uh yeah formatted template. Um but people sometimes don't know that you can easily just see what that template would look like if rendered for a given node.

And we're making that capability more discoverable here with this interface by saying, "Hey, just pick a node here and see what that will look like if this overlay were distributed on your node. You can do this on the command line as well. It's a really good way to develop new overlays and and make a change to a template, see how it would change if it uh applied to that node. But uh right now you can't edit overlay templates in this interface. But that is the goal to be able to edit the template quickly see what it would do on the node, go back to the editor, make more changes perhaps side by side.

Um we're still working on that experience. So that's the majority of what we have to show here right now. Um let me look back. Do we have more questions or more uh more things in the chat? Let's see. There's not more questions um from the group just yet. You guys feel free to put that in the chat or the Q&A. Either one, we will definitely get to them. I did have a question and this is probably just part of my ignorance, but also uh sometimes that comes up because CIQ as a company, we do provide the support and the customization and all the things around Rocky Linux and I see Rocky Linux or Rocky on here.

Um, is there any barrier for people who are maybe not using Rocky or what is kind of the relation? Ah, that that's a good question. So, right now, uh, I've only built the the web interface we're talking about, not the API here, right? I've only built the web interface for Rocky 9 today. Um, and mostly I think any issues we would have would be about making sure it's compatible with whatever version of Cockpit is in your system. So if if you're uh I forget what version of Cockpit uh Rocky 9 has right now, but we've aligned this to the version that's with Rocky 9 right now.

Um it may or may not need modification for slightly different versions on either side. Um, we definitely should at least try building it against Rocky Linux 8, testing compatibility there. Um, but it might be more difficult with, let's say, Abuntu or Debian, depending on what cockpit version they have. Um, for uh for other let's say like Rocky Pier distributions um like CentOS or or anything like that, um it should be just fine on the Enterprise Linux 9 side of it. Um again we would need to build packages for eight if you wanted to deploy it there. I see you see a question in the chat. Do we need to take any steps to enable Warewulf in cockpit?

So uh it is installed as an RPM for our tech preview at least. We're creating a a separate product uh that you would have in addition to a Warewulf product from CIQ um that distributes that RPM. you install that RPM and then there is one file that you have to add uh right now to tell um cockpit how to log into the API. So that is stored at uh in your home directory at dot dubdub ai where uh wwapi.json and that um at at minimum needs to include a username and password and then by default it will attach to local host and and the default port.

Um, what we're going to do, but have not done yet, is add a login experience to this interface so that if you try to access this Warewulf tab and you don't have this file present, you'll be prompted to enter whatever connection information you need. Um, and it will create this file for you. Right now, uh, you have to add this file yourself. But yeah, other than that little bit of configuration, it's just a package that you install. Uh, so I see my IT security would flag plain text password with their security tools. Um, you're always going to need some kind of shared secret to allow access to an API.

So, you know, we're calling it password here. Other systems kind of skirt around this by calling it like an API access key or something like that. And perhaps we could do that. Um, but there's always going to be some kind of secret that you have to give to um, uh, to an interface in order to to to a a client of an API to allow it to access that API. Um, does this suggest CIQ will be building other apps for cockpit? Uh, I I would not say so. Uh I think this has been a relatively good experience and a and a positive thing but so far that I'm aware it is the only such um uh initiative within CIQ um to to do something with cockpit.

But uh if it goes really well, if we get a lot of buy in, we get a lot of um of interest in this, I don't see why not. So I'm going to go back up just for a second because there's a couple of questions that I I um didn't want to just kind of get lost in the thread. Um, are the install instructions already available to the public on Git? So, the install instructions for the API are yes. Uh, instructions for uh the web interface are not a uh part. So, first of all, the web interface is not part of the uh upstream community project.

that is a CIQ add-on that uh we you know we consume that public API but the web UI is available to CIQ customers. We do still need to um uh write the documentation for that. Uh it'll be largely what I have just expressed. Um but we're we're working on the documentation for that as part of the the tech preview process and and making that available to to people that are participating in that. Right. Exactly. And you guys can definitely get a hold of us. I mean any any form that you fill out on um our website ciq.com it'll come to us and we'll we'll figure that out.

You can also just email me directly rstein@ciq.com. Yeah. Um cool. Uh John says nice work. Thank you. You you know how much work Jonathan and the team have put behind this. So thank you for that. Um Trevor catching up a few more in the back. So will there be a section to send IPMI commands or view BMC status or uh console? So I right now I'm thinking probably not console because I'm thinking console will be better integrated with other console solutions that I'm hoping you can just do natively in uh in in cockpit. Anyway, one thing that is also good about being so for for uh um features that aren't already in this web interface, um you can just go to a terminal in here and run Warewulf commands uh directly.

So, you can do that. Um but for normal like IPMI power commands, absolutely. I think that should be included here. And I would expect uh there to be like power on power off commands in this menu here. Maybe like a power on all uh button up here, that kind of thing. Um, also, yeah, there there is a message in here about like the comments not going to the right place. So, we are the hosts. Wait, are we the host? Hosts and panelists. Yeah. So, you want to want to make sure that the question um goes to hosts and panelists. If it's just the attendees, then you guys are just seeing it yourself.

And I think that was just because we had skipped over a question earlier. I think we've gotten back to that. Um, okay, good. Um, did you already answer the question about the IPMI? Yes. So, that was like power on, power off, and console. Um, yeah, definitely the power level stuff should be in here. Um, we've just been prioritizing getting the the uh configuration settings in first. Um, but I would expect that. Uh, in fact, I think it's already on the backlog for a GA release of this. Perfect. And there's a question about training. Yes, 100%. We have a wonderful one-day training uh for companies and it's been going really well.

That that's probably one of the top things that are selling right now because people are like, "Okay, let's get in. Let's really understand it and let's move on." So, yes. Yeah. And then we can extend that if if you have like an environment that you want advice on. Sometimes we'll wrap some training with some professional services level stuff to look at your environment and give advice for how or maybe have a second day with Q&A where you take what we went through with the first day and then have questions from your environment. That kind of thing can go really well. Yes. Okay, Ian, I'm glad that you came in talking about Ubuntu.

Okay, so Ubuntu 2404 cockpit appears to be 333. I'm happy to guinea pig as always. Thank you, Ian. Taking the charge over for Ubuntu. Yeah, I realized I could just check here on this terminal. So, it looks like we have built against 33 32.1. So, we would need to check and see uh what if any changes would need to be made trying to build it against another cockpit. But so far that's what we've been building it against is this one. Yes. Um there is a a question to clarify. So yes, the Warewulf module or extension or page for cockpit is uh the currently available only to CIQ customers.

That's right. And if you are or want to be um uh such a customer and would like access to this um definitely reach out. We would love to get you into the tech preview because the goal is to uh get feedback on it and see what you like, what you don't like, what is missing that we may have forgotten or that we just haven't gotten to yet. And that'll help us prioritize things as well. Yes, thank you for your willingness to do that, Omar. We'll definitely uh be in touch. Timeline for GA. Uh um there's not an official one right now. Um but hopefully not too long.

we have uh a a slowly growing list of things that we want to have done in terms of a GA um uh you know not the least of which is just having uh all of the configuration options available but um I I would well I don't want to make any promises here uh for the purposes of the webinar but the goal is to have it be relatively quickly not not years from now or anything like that. Yes. Okay. Good. Yes. Texas Tech is definitely in. Okay. Okay. I know it's early, but commercial open source. Jeff, we gave you the API. Yeah. Yeah. So, and right now the plan is to keep the UI itself as part of a a CIQ offering and not have it be contributed to um to the community project.

Um, and there are a number of reasons for this. Uh, the most of which is just trying to get people I don't know how do I say this? Um like the the reality is uh CIQ is contributing a lot of time and and uh development into Warewulf and we we need to find a way to get people into uh support for it uh because we're getting a lot of community interest but uh we we want to find a way to demonstrate the value that CIQ is providing to that community. So this is one of those differentiators that we're trying to provide that we think is useful and that people will like especially as part of an integrated solution with a cluster um but uh hopefully isn't a detriment to be missing uh in the community side.

So that's the plan right now uh is to have it remain part of a a differentiated CIQ supported offering around Warewulf but to always and forever uh keep contributing to the core Warewulf system and never have any like functionality gated into this or anything like it. And then another benefit to that is uh it it uh hopefully will let us provide more integrations with other um CIQ data sources. So, one of the things we want to do is provide um more and more like pre-built images for different use cases, pre-built overlays for different use cases that are specific to our customers needs and showcase those through this interface and have like library and catalog access to those things.

Awesome. Um Jess, thank you so much for posting the link to do like an official sign up for the tech preview. Um so um yeah whether you're a customer or not a customer or you're interested and you want to be part of it definitely go to that link and um and sign up for that. And then there was also a question about this being is so we are recording it. Yes. Our intention was to record. So I I I heard an alert that we're recording at the beginning. Okay. Perfect. So yes, it is recorded and we will be putting it up um uh you know live and available on our YouTube channel um early next week.

So you will definitely be able to to rewatch that. So no no worries. Okay. Uh wow. Thank you. Thank you. Thank you. Um and some of the other members of the Warewulf community were also popped in and joined. So thank you so much for being here and for all of your guys's hard work as well. Many, many thanks. Thank you so much for being here, for all the good work. Is there any um any last words that you have here for us, Jonathan? No, I like I said, I'm really excited to get this uh out into the open to show it off. Um I I was busily filling out uh this interface yesterday, just put writing copy for each of these things.

I'm really happy with the way it's coming together, with the way things look in it. Um I I do uh it's been kind of a a really great learning experience to get integrated with the cockpit system. I'd love to talk about that uh even on the community side just what that experience was like if anybody is interested in building cockpit extensions. I know there was questions there were questions about that earlier in the chat as well. Um that I think uh we could we could save people some time if they wanted to dig into that. not quite as well documented as I wish it was, but uh ultimately it's it's come together pretty well.

Yeah, this is great. Very helpful. Thank you everybody. All right. Well, so again um in the little chat there or if you're watching it um on the replay in the comments there will be the sign up link. So ciq.comwolfui. So we are very excited to chat with you. We want to get this in your hands. We want your feedback. We want to continue to iterate. And thank you so much Warewulf community and everything that you've done, Jonathan. No, thanks to everyone who's a part of it. And thank you, Rose, for uh being here and uh and shephering us through it.