Nextflow + Fuzzball demo: Pharma-grade RNASeq workflow execution

Before we can actually run the script, we do need a fuzzball um context activated. For this, we need the Fuzzball command line interface installed. Here's the URL that tells you how to do that for your particular system. And the command would be Fuzzball context login, but I'm not actually executing this because I already have an active context. Next, we run the actual command. Um, in in this particular case, we are going to only do a dry run because I've already submitted this workflow. And we we're going to show what this looks like later on. But as I mentioned before, the structure is the the script followed by some options that tell it how to interact with fus ball.

In this case, what do I want to name the main job that's submitting that is running next flow and submitting subtasks. We also have to tell it that we're running an NF core pipeline because NF core requires some special settings for next flow and in this case we're doing a dry run which will just dump out the fus ball workflow as it would be run if we had submitted it followed by double dashes and then a normal next flow command. In this case we will be running um the NF core NF core pipeline RNA seek. We have to tell it to use a fball profile. The fball profile contains all the information and code required to submit uh tasks as fastball jobs.

And we're also telling it to use the test profile which includes a small test data set, small genome, and so on. And then we're specifying a path on fuzbar storage where to store the output. And again because this is a dry run, we will just look at the output and um you can see that it generates a essentially a fuzzball workflow that runs next flow which then in turn submits jobs to the Fuzzball cluster that it's running on. Okay, now that we've submitted this, we can go and look what this actually looks like in the Fuzzball interface. When we first come over to the bus interface, we can see that the submission script has started the main fus ball job.

Um, in this case, we're still setting up the storage, transferring files, obtaining the next flow container. Um, and we will revisit this in a few minutes when the actual processing has start. Okay, so we're back a few minutes later now and you can see that the actual next flow main controller job has started now and we can look at it and we can see the logs and let's look at all of them and you can see that it started running the NF core RDA pipeline and here it is submitting uh running starting all the tasks and if we go back to the list of workflows we can now see that we're running um one two I'm running about 10 tasks concurrently.

I think that's that was the the number of jobs that I set it to run concurrently. So this will run through the entire pipeline until it's done and then we can reconvene when that's completed and look at the outputs. Okay, we're back after a little bit. The next full pipeline execution for this NF core RNA pipeline finished. For those of you with good eyes, you'll notice that this is actually a different workfl ID because we had to restart the pipeline once due to completely unrelated to reasons completely unrelated to either the next flow or the next flow plug-in. Uh you see the pipeline completed successfully. Again, this is the log from the main controller job with 130 jobs using results from reusing results from a previous run and 79 tasks that were actually run as part of this execution of the pipeline.

And again, each of those runs as a separate fuzz workflow. And because this is just a normal NF core pipeline run, you get all the usual outputs. uh you get the the QC report, you get an Xflow report that shows basic statistics like runtimes, resource usage and so on and you can get a timeline of the execution. So it in essence this is just a regular execution of in next flow pipeline which is the goal of this whole project. Um but it was using resources provisioned by fuzzball. Okay, so that was a basic overview of how you can run Nextflow pipelines on fuzzball and let me go back and talk a little bit more about future plans.

Uh there is still improvements to be made for efficiency and execution. Uh in particular, we're looking at batching multiple tasks into a single spa workflow which should help with with workflow execution. We're also going to make this into a easily installable for the command line submission. We're going to going to make this into an easily installable Python package. And we will also be making this into an entry in the fuzbal workflow catalog which is essentially a a pre-anned um set of applications that you can run directly from um the fuzal graphical user interface. So you should be able to submit your Nexflow pipelines directly from the from the GUI if you prefer that over command line submission.

And we will also be working on improving um some aspects of storage in particular remote storage handling. Okay. Well, that concludes our tutorial and we will see you um when the next version comes out and there's hopefully new and exciting features. Thank you very much for your attention.