Inside the DGX Spark benchmarks for RLC Pro AI

People are can connect these over over connect X or NVLink. They can get stacks of them and that will enable them to train, develop, and build capital applications on some of the larger open weight models. So, what does that mean? DGXs are getting closer and closer to the point where you can deploy to production. Right? So, what we wanted to do is could we take our OS that was designed and built for as a server OS for production workloads and help that experience by ensuring that it ran in your development phase of the life cycle on a DGX. Right? And so, what this means is a developer can build in a production like environment and have confidence that it's not going to change.

But what it also does is it enables those smaller shops that are running DGX for production or quasi production, we're giving them a production grade OS. >> So, I think you're you were kind of leading my question here that I wanted to ask, which is um and maybe that's a little bit more nuanced. Why should I care which OS is running underneath it? Oh, besides just hey, I can run it on on my dev box and then maybe go to production. >> All right. So, I that's that that the really compelling part of that I'm going to leave for Andrew to show versus me telling, right?

But I'll also be a little bit of um um why should you care? You should care because no matter what type of workload you're running, and ultimately whether you're running on a DGX or not, this little thing that we call the Linux kernel, which by the way, Linux runs 95% of AI workloads, right? I think that stat since I first read it is probably updated as people deploy on Apple metal or on MLX, right? But generally Linux runs 90% of of AI workloads. It runs your DGX for our DGX owners. And every access to your compute resources, to your memory, to your storage, ultimately at some point has to go through arbitration and orchestration of that kernel.

Um and um to some extent Linux um user space. So, this thin thing that we call the OS that is maybe non-sexy, basically dictates how much you're going to get out of those dollars that you've spent for that box, for those GPU and GPU hours, and even DGXs are not cheap. All right. Um so, that's why you should care about the OS. It is between your application and the compute that powers it. >> All right. So, before we get to kind of Andrew and and so on and so forth, I have one more question.

This is one that that really bugs me cuz I don't know kind of how that works is let's say we have that dev and prod environment and I've got two different distributions running on those things and mainly because of of uh performance differences between what I'm running in production which might be a little bit slower but more stable and then what I'm running in dev which is a little bit more cutting edge but something I probably want don't want to deploy to you know 150 nodes or something like that. >> Right. >> What does that rebuild actually cost a team if you need to go from one distribution on dev to another one in production?

>> Um I'm first going to say honestly it's um probably not quantifiable though we have heard um hours to days to weeks. Uh we actually had a recent person that had deployed uh and this will get to why it's hard to quantify. Um we had a customer recently uh deployed RLC Pro AI for their production workloads across um I think eight H200s. Um he said prior to standing up that stack and they were already delivering to customers. Um they spent weeks trying to debug out um dependency issues, dependency dual loops and um and then the incremental performance drop. So to just get a stack that worked against the hardware they had in the production environment they had, they spent weeks of time.

And oftentimes it meant triaging customer complaints, right? So there's an opportunity cost lost to it. Um but why do I say it's unquantifiable? Because there's so many moving pieces and I think there's impact that people miss, right? If we just look at uh moving from dev to prod and changing environments, surely it's like you're moving from like a GB10 um which is on your spark to a GB300. Your hardware's changed. It then means that at the OS level, the drivers you need to support that hardware change. What it can also mean, particularly if you have different distributions, and I'll just, you know, I'm going a bit of detail is, well, your build time and run time flags um uh for your kernel and compiled user space packages can change, right?

And then so what happens is is your performance profile can shift more than necessary. What you'd like to say is everything's the same, I moved to a bigger GPU, and now I just have more performance, right? A bigger box with more RAM. Um but the thing is is once your uh performance profile starts to change, once your kernel flags start to change, your resource utilization changes, not only can you like impact the customer experience, higher latency Well, first I'd say you may just spend a bunch of time debugging configurations and versions. You get a stack to work, now all of the sudden you find your customer experience and even possibly your model behavior is different than what you saw in dev, right?

And we know these are highly script-based processes, so I think you potentially gain hours, if not weeks, of configuration debugging with deterministic performance if you just do that old principle of software dev in CI/CD, right? Build it in a production-like environment, so you don't pay for it later. >> Good. I know, that's that's good. That's good. So that's some really good context about why we're looking at the OS. Anyways, uh I'm a numbers guy, so I like to get into the numbers as as soon as possible. So >> Accountant in you, huh? >> Yeah, yeah, yeah. No, I did data for a very long time. Uh so with that said, Andrew, can you walk us through these benchmarks and kind of what they were measuring and how we went about doing this?

>> Uh yeah, so so the the benchmarks um are really about uh you you your your your throughput on these inferencing workloads, right? Like like how many how many tokens per second can you push through this system? Uh there are pieces of the benchmarks that are also about like how quickly the data loads in the first place so that you can use it. Um and Do you want to talk about the like the setup for the system or or do you want to just do you want to just stay focused on what the benchmarks are and what they mean? >> I mean just high maybe high-level what the system looked like.

Like did we like handcraft this specific one for benchmarking kind of a thing? >> Right. >> want to say fudging the numbers but we see that >> So we you know, when I when I was asked for benchmark numbers, you know, my my mind goes immediately to I need to make sure that the system that we're benchmarking is something that our customers could could reproduce. Right? So I want to I want to know that this system is clean. Now, you know, at first when I first got my hands on this system it was not it was not clean. It'd been used by a number of different developers and whatnot.

So you know, and I and I did the same thing at first, you know, tinkering with it and whatever. But but before before running these final benchmarks, you know, we I I reinstalled the system from scratch just using the RLC Pro AI 9 ISO image that that you can download. And you know, booted up clean just I I didn't even change much in the in the installer, right? I just let it install what it was installing. Um you know, I'd be happy to share you know, what what that looks like the the kinds of things that we did in setting up the benchmark um to to kind of show that it's that it's just a vanilla system.

>> Yeah, absolutely. Love to see that. >> All right. So I did I did record this um using a cinema so so we wouldn't have any uh big surprises. But um Okay, so here I am on our DGX work system. And this is just a vanilla RLC Pro AI 9. Um I did uh configure Depo and log in and set up the, you know, the the product so that we could do an update before starting the rest of this process. Um, so And for I know some >> And I'm sorry, Andrew, just and I'll state it so for people watching, um, Depo is our secure repository manager package.

>> Right. Right. Okay. So, uh, first I'm I'm just I'm just showing here that like this is this is just our vanilla system. It's just been installed. Uh, it does come with the driver configured, but we've got it configured with one of long-term support drivers for Stos. Um, we've got you'll you'll need some disk space cuz obviously the data sets are quite large. And then, um, we're going to go ahead and ins- we're going to swap the driver from the 580 driver that was installed here to the, uh, what we're currently calling our latest driver, which is 595. Um, so we're going to command install these drivers.

There's a uh, bunch of other setup that we'll have to do along the way. Uh, it's not going to take a long time, uh, and I cut some of the longer parts out of the recording here. Um, but basically this is just swapping the driver and then we're going to reboot the system. Um, after the system's rebooted, want to make sure that it actually loaded this 595 driver. Uh, we've got to grab Docker because the DGX Spark benchmarks use Docker for their for their actual runs. And that's kind of an equalizer, right? If we're running the same software stack for the benchmark on everybody's system, then then we know that the that we're running we're running kind of an apples-to-apples comparison, right?

>> So, that just kind of isolated to say the only thing that's different is the operating system. >> Right, yeah. The the kernel, the drivers underneath, like the those those are the things that are that are different here. Um, so we're going to enable Docker, uh, uh add myself as a user, make sure Docker is running. Uh next thing we want to do is install the NVIDIA container toolkit, which is basically the thing that makes sure that the GPU is shared with the container when you're running inside the container. Uh and we're going to configure that to work with Docker and restart Docker. >> Now, if someone wasn't running benchmarks, they would already have RLC Pro AI ready to use, right?

This is benchmarks >> yeah. You You know everything after the first like 30 seconds of what we were talking about is just what you need to do to be set up for the benchmark that that we're going to be running next, right? Um okay, so if I'm sorry >> May I interrupt, Andrew? I just want to double-check cuz like you also didn't have to compile any drivers, right? When you ran the DNF like they they just came right in without really Yeah, yeah. So >> We We compile the drivers. Um and and by the way, like earlier when we were uh Ooh, let me scroll up here.

You'll notice this this is a recent kernel. This is the kernel 6.18 series. Uh this is our CIQ Linux kernel um 6.18. So this is a more modern Linux kernel. And this is part of what we what we've done that's that's different from just your regular Rocky Linux is is that you get this newer kernel that has support built into it for the Spark hardware, right? Um and and and that's built and signed by CIQ uh and uh we'll we'll get to some of this other stuff, too. But yeah, I didn't have to compile the driver. I didn't download the dot run file from NVIDIA and have that kind of pollute my system with I don't know what that script does, right?

I just installed packages from CIQ's repositories. Um Now, to to get the actual benchmark tool running here, we're we're going to use UV to install it. Uh UV's just a nice Python executable installer. Um okay. So, the next thing I want to show is that this system is uh is secure boot enabled. Um because if you're if you're running an enterprise workload, you you want to make sure that the whole boot chain, the boot loader, the kernel, and all the drivers running on this system are actually what they claim to be, right? So, so every piece of that system here was built and signed at CIQ.

Um and and so, we can see here that the system is secure boot enabled. We can see that the kernel is kernel currently running in integrity mode. And that the module here, this the the NVIDIA module, is signed by Control IQ, right? Uh and now we're just going to make sure we're we we're on that 595 driver. Uh and now it's time to download the the the benchmark. Um the benchmark. Uh what what what do they call these? Uh the YAML file that describes the parameters of the bench benchmark, right? Because the idea here is I need to use the same parameters as everybody else is using, so that when those comparisons are made, we're we're we're looking at something that's meaningful, right?

I'm going to download the the benchmark here. Um the uh Spark Arena benchmarks recommend you clear the caches, and so we're going to do that. Uh and then uh we're going to run the benchmark. Now, in my recording here, of course, the benchmark takes uh 20 something minutes. Um so, we're we're going to we're going to cut some of the boring stuff here. Um the uh permission denied here, don't worry about that. That's just because uh we're running locally on this host instead of remotely, and and so uh it tried to log into itself, which it didn't need to do, and and and uh So, moving on.

Uh so, now it's it's running the benchmark on this on this Spark machine. >> So this is running like a a specific model with a specific set of instructions. >> this this is the the Quen, oh gosh, which one was this? Okay, Quen 3.5 uh, the 0.8 billion parameters version of that model. Um, so that's so this is the benchmark that it's running and it does this at like various various different parameters and uh, you can see we we we cut a little bit here that's, you know, it takes another 20 minutes and stuff to actually run through the rest of that stuff. And then here are our benchmark numbers.

>> So an interesting thing I'll just call out in the numbers is that there's a concur- concurrency column. Um, so back to your question about dev versus production, one of the reasons these benchmarks are important, yes, 10 is a low concurrency number, but where you're going to see really different behavior and in your performance is um, across uh, different concurrent um, um, users or queries. >> Right, inference stops. >> Right, so if you only have one user, you're going to get uh, you know, a much higher throughput than you're going to get if you're running >> 10 users at a time. >> Yeah. >> Right? >> Yeah.

Right. And and so the let's let's talk about like what the numbers are saying and what they're not saying uh, is is I think good here. What's the end result of running these these benchmarks and and what are they really telling us at this point? >> I mean, this is where actually it's more useful to look at the the I think the the slides that that Brian had um, set up set up for this cuz if you when when you take these numbers and you compare them with the benchmark runs that other people have done on their systems usually using Ubuntu or some or some other operating system, these numbers um, on RLC Pro AI 9 often come out a little better than than what's already been posted in in the in the Spark Arena benchmarks.

Uh and they and they mean a a variety of things. Um I'm going to have to admit that I actually don't know what these little abbreviations mean, TG and TTFR and and whatever, but um what I do know is that the numbers are the numbers come out better in in some cases. Now, there is a spread, right? So So, if you're if you're running a benchmark multiple times, you're going to get different results sometimes. It's not always going to be the same result. Um but that that spread puts RLC AI 9 a little bit better than what than what you're seeing on a lot of the benchmarks.

>> Yeah, I and I'll say, while it's funny, I was just looking at it. If I thought about it, I may be able to remember what PP and TG means it does. Um >> I'm going to try to look it up while you're talking. >> You know, at the end of the day, what we're measuring, those particular benchmarks, they have some latency recording, right? Latency being I think it actually said it there, time to first token. So, from the time that I load a model and then I send a prompt to it, uh how long does it take for a token to come back to me?

That time to first token is important because whenever you're say we're in Claude, Anthropic, or a chatbot on your healthcare website, the first time I open that and send a prompt, usually we're cold loading a model, we're booting a system, cold loading a model, and then we have to wait for the GPU to hit up, the cache to fill, and then that's when you get to a token. So, it's your perceived wait time. Um then on the other end of it, the primary thing, but the core thing, and where we really stand out that these benchmarks are measuring, are tokens per second, which is throughput, right?

And it's given a certain amount of time, uh or crudely put is how much inferencing can you get out of your GPU? How many tokens in and tokens out? And so, where we found um with Andrew's work um that we really stood out um across both the Quoin 8B and the Quoin 2B uh in single and concurrent use is we really stood out in tokens per second. In some cases, I think up to 9 10% benchmarked on one system, but with this most recent run of submitted benchmarks, uh we're averaging around 4 to 5% gain over four of the five first place leaders on that spark-arena.com board.

>> And I'm assuming that that uh a lot of those leaders on that uh that board are are probably not uh enterprise grade distributions. Uh that they're probably uh meant to be used for specifically for cutting-edge kind of stuff. >> Right. Well, I I I I imagine most of them are probably just using the the operating system that that you you know, that you get with the DGX Spark by default, right? Which is which is one of the one of the Ubuntu OS's. >> Right. >> Yeah. >> And which actually, interestingly enough, has a newer kernel. It has a 7.0 kernel. So, it has a newer kernel than enterprise Linux than a 5.4.

And this is to where we get to another point. But the catch is, no offense to Debian, Ubuntu, right? Um there are a lot of benefits to it. Many people use it, but enterprises have invested decades of of expertise um into a skill set and pipelines built around enterprise Linux, >> Right. >> right? And that enterprise Linux or or open ELA and vendors that create enterpr- or groups that create enterprise Linux distributions has spent a lot of time ensuring that they're managing that and including packages and default configurations that are attuned to supporting critical workloads in the enterprise. >> Right. >> Right. And you you'll have you'll have your your package scanning systems and and your your management tooling and whatnot that that's all built on the systems that you've already been using.

And if you're starting to adopt AI, and that means that you're forced to change operating systems and try to adapt, you're going to lose some visibility in what's going on there. You're going to have a lot of extra work getting getting the systems that you were already using working on this new system. Uh yeah, there's a there's a lot of advantages to being able to use an enterprise Linux system on cutting-edge hardware. >> Yeah. I think we can probably measure it by the number of times you invoke man commands to do something simple, right? Man apt, man, right? Just trying to figure out the nuance differences, even.

Uh um so, the man the man invocation per minute ratio is one that we can >> Is that one of our benchmarks, friend? >> [laughter] >> You just made up another benchmark. I love it. This is great. >> Uh um man, this was a great amount of information. I just want to make sure that there's one other thing. These benchmarks that you run, is this something that is repeatable if I want to run it myself? Um that's that's easy to do, or is it something I'm going to have to do a heavy uplift? I mean, besides getting my hands on a DGX >> I mean, like I showed, like the my whole point in how I tried to run these benchmarks was to make sure that you could run them, right?

To make sure that that every step that I took was a step that you could take, and it was pretty obvious why you were doing it and where you were getting your your bets from, right? And and it's not even that many steps. And we've got a blog post about about exactly the steps you'd take to reproduce these benchmarks. >> Look at this guy segueing for me. Uh that's a great amount of information. I don't want to thank both of you for And absolutely, for more detailed information on these benchmarks, we encourage you guys to go to ciq.com. Uh look for the ROC Pro AI page itself.

It has a a wealth of information not about just about the operating system itself or that distribution, uh but about the results. It's got some of the benchmark numbers on there and then it has links to a specific uh solution brief around these benchmarks as well as you can search for the blog article that we're just talking about on how to reproduce it yourself uh right there on the site itself. Um I want to thank both of you guys for doing this stuff. It was great. Uh this was a really good kind of informational point. I will also point out this too. If you are more interested watching this on getting any more information more than just going to it, we would love to talk to you individually and set up some time with you to say, "Hey, what is your use case here?

Um what are you trying to do and where are you going from dev into production and how can we kind of help guide you through that process and see where you're going?" So, reach out to us for sure. Um again, both of you I want to thank you for being here. Um And uh thanks for spending time with us. >> Yeah, can I And thank you. I do want to squeeze in a brag about my teammates, Bernard. Um uh when I come here and the reason, you know, do come talk to us. Um we are all figuring out the evolution of AI, um but we have such an amazing group of lifelong HPC and Linux experts.

We have Andrew who has spent decades uh building, delivering enterprise Linux. We have the creator of Apple. We have uh one of the you know, a team of the early builders of Ama- Amazon Linux. Um so, you know, we want to help you along your journey leveraging this amazing group of consolidated HPC, AI, and Linux expertise. >> Couldn't have said it better. Appreciate you guys. Thank you.