Episodes
-
In this episode of Channels, the Nextflow podcast, host Geraldine from Seqera and her colleague Ben Sherman dive into the differences and similarities between Nextflow and the Workflow Description Language (WDL). They discuss the origins and development of WDL at the Broad Institute, the challenges of working with different workflow languages, and how Nextflow's channel-based data flow model compares to WDL's approach. The conversation covers key features, type systems, and runtime specifications, highlighting the strengths and limitations of both languages. Ben hints at upcoming improvements and enhancements for Nextflow, aimed at making the language more robust and user-friendly. The episode offers valuable insights for bioinformaticians and developers navigating the complexities of workflow management.
-
In this episode of Channels, the Nextflow podcast, join us as we explore Nextflow and AI. Our special guest, Sasha from Seqera, shares his remarkable journey from the consumer tech industry to bioinformatics. Learn about the birth of TinyBio, its innovations in bioinformatics AI applications, and its recent acquisition by Seqera. We delve into AI concepts, the challenges faced, and the future of integrating AI with Nextflow. Tune in to discover how advancements in AI are reshaping the bioinformatics landscape.00:00 Podcast 45: Tinybio : Nextflow x AI00:09 Introduction00:34 Sasha's background03:17 How tinybio started05:57 Tinybio co-founder: Vishal Patel06:35 tinybio chat public launch09:20 Failed feature launches10:39 Running the code too12:46 User trust14:11 Accessing files via chat15:08 Experiment idea generator17:16 Pipeline generator20:25 AI concepts: Foundational models22:36 AI concepts: RAG24:53 AI concepts: Langchain26:37 AI concepts: Fine tuning29:14 Challenges in bioinformatics AI31:53 AI concepts: Q&A training33:09 Motivation for joining Seqera34:43 What happens to tinybio now?35:36 Future plans38:20 AI and open-source40:50 Find more updates41:37 Wrap up
-
Episodes manquant?
-
In episode 44 of the Nextflow Channels podcast, Phil Ewels welcomes Alex Pelzer— a key figure in the Nextflow and nf-core community. Alex shares his journey from academia to the pharmaceutical industry, highlighting the critical role of regulatory frameworks in bioinformatics.
🌟 Key Highlights:
The transition from working at QBiC to Boehringer Ingelheim. The importance of maintaining high-quality, reproducible pipelines in bioinformatics. Understanding regulatory requirements and their impact on clinical trials. How regulatory standards and open-source initiatives like @nf-core are shaping the future of bioinformatics.Alex also discusses the formation of the @nf-core Regulatory Special Interest Group, which aims to bring together experts in the field to establish and promote best practices for ensuring pipeline quality and compliance with regulatory standards.
Looking forward to seeing many of you at the upcoming #NextflowSummit in Barcelona this October! We’ll be working with this #regulatory group during the nf-core hackathon.
00:00 Episode 44: Regulatory revolution00:06 Introductions00:33 Upcoming Nextflow Summit Announcement01:34 Alex’s Background and Early Days with Nextflow04:41 Formation and Growth of nf-core06:18 Transition to Pharma and Regulatory Work08:56 Regulatory Challenges and Practices in Bioinformatics13:30 Tool Selection and Quality Assurance15:23 Evaluating Software Trustworthiness17:08 Overhead in Academia vs. Industry18:11 Downstream analysis20:57 Regulatory Special Interest Group25:28 Special Interest Groups in nf-core28:59 Future Directions and Community Involvement30:15 Open source contributions in pharma33:27 Conclusion and Upcoming Events
-
In this episode we explore the new features of Seqera's Data Studios and Data Explorer, with Phil Ewels, Rob Newman and Rob Syme from Seqera.
Discover how to use these tools for troubleshooting Nextflow pipelines, tertiary analysis and Nextflow development. We discuss the pain points that led to the creation of Data Studios and how it's designed to allow scientists to interactively and collaboratively work with data and complex workflows, without having to move large datasets around.
Rob Syme wows us with another fantastic practical demonstration, setting up and using Data Studios to write and test a Nextflow pipeline in VSCode running on the cloud in a Data Studio environment, including running the Nextflow CLI with task submission to AWS Batch.
We cover features like session persistence to save work states, and upcoming custom container support for your own specialized applications.
Learn how these tools can enhance your computational biology projects and make seamless cloud integration a reality.
00:00 Channels Podcast 43: Data Studios00:26 Introductions01:54 Data Studios04:51 Move the compute to the data06:13 Real-time collaboration06:47 Data Explorer09:41 Access to public data10:45 Data Explorer demo13:56 Data Studios setup20:17 Session persistance22:52 Data Studios RStudio demo28:24 Nextflow development in Data Studios36:17 Future development37:01 Custom containers40:01 Boston Summit demo44:01 Lifetime management47:14 Wrap up
-
A small but intrepid team of Seqerans recently attended the ISMB 2024 conference in Montreal, a great mix of computational biology and bioinformatics with a largely academic / research-focused audience. Join Geraldine Van der Auwera, Rob Syme and Florian Wuennemann for a lively discussion about the scientific themes of the conference, their experience running the booth, meeting with the ISMB and BOSC community, and resources for job seekers.
-
Join us in exploring the latest Nextflow release, 24.04. Phil Ewels and Ben Sherman dive deep into the new updates in Nextflow’s most recent version. We discuss advanced retry strategies, job arrays, resourceLimits, Singularity’s OCI mode, and the game-changing Workflow Output Definition.
We’re actively seeking feedback from the community on these new features. We would love for you to experiment with the new syntax and let us know your thoughts.
Please let us know what you think either via Nextflow GitHub issues or via the community forum.
In this episode we refer to a the recent blog post about the release, which you can find here: Nextflow 24.04 - Release highlights.
Intro - tweaks, fixes and new support in 24.04
Performance and Stability Improvements
Ben kicks off the discussion with key performance and stability improvements. The latest stable release, 24.04, includes numerous bug fixes and performance enhancements. There’s a significant focus on closing gaps and adding retry strategies for better stability, especially concerning cloud providers. Notably, if Nextflow’s API calls against cloud providers fail for any server-side reason, it will retry automatically, avoiding pipeline failures caused by temporary server issues.
Publishing Changes
Previously, if a file failed to publish, Nextflow would only issue a warning. It was possible for a pipeline to complete successfully without noticing missing output files. Now, by default, Nextflow will fail the pipeline if the publishing fails, though there’s an option to revert to the old behavior. Additionally, retry strategies for publishing have been introduced to ensure retry attempts if an issue arises.
Job Arrays
One highly requested feature is the introduction of job arrays. This allows users to submit many jobs as a single submission, alleviating strain on schedulers. The submission happens in a batch, and then the scheduler can process and plan effectively. Once a job is submitted as part of a job array, the jobs run independently. If any child job fails, it is resubmitted without affecting the rest of the array.
Singularity OCI mode and GA4GH TES
In the past, Singularity supported Docker images by converting them into SIF files, consuming storage and time. Now, both Singularity and Apptainer can run OCI images directly, saving valuable resources.
Additionally, the TES executor has seen significant improvements, now supporting TES 1.1, which brings broader compatibility and integration with existing workflows.
Major new syntax features
Topic Channels
A new feature called topic channels offers a more straightforward approach to collecting channel outputs across pipelines. Channels can emit data to a named topic, simplifying the collection and use of version information from various processes within Nextflow pipelines.
Eval Outputs
Eval outputs simplify the addition of shell commands to tasks. With eval outputs, necessary post-task commands can be defined neatly, avoiding repetitive code within process definitions.
Workflow Output Definition
The concept of workflow output definitions has been introduced. This new syntax streamlines the publication of files by defining publish targets within workflows. Instead of defining publication behaviors within process definitions, users can now manage them at the workflow level, ensuring better clarity and fewer repetitions in the code.
-
Join us to hear Phil Ewels chat with Maxime Garciaand Franziska Bonath about how nf-core started at theNational Genomics Infrastructure (NGI) at SciLifeLab in Sweden.We chat about how the nf-core/sarek pipeline grew alongside and eventually joined nf-core,as well as the early days of nf-core bytesize talks and outreach organisation.Links:- SciLifeLab- National Genomics Infrastructure- Maxime's Nextflow Summit Boston 2023 presentation about Sarek- nf-core bytesize talks
-
Tune in to hear all about Pixelgen Technologies - creators of a novel technologyfor doing single-cell spatial proteomics.
We chat about how this fascinating technology works, what you can do with the dataand why they chose to develop their analysis suite as an open-source nf-core pipeline.
Links:
Software: nf-core/pixelator pipeline Pixelator Python package Science: Nature Methods publication: “Molecular pixelation: spatial proteomics of single cells by sequencing” Presentations: Nextflow Summit 2022 talk: “Pixelgen Technologies ❤︎ Nextflow” Nextflow Summit 2024 Boston talk: “Nextflow allows Pixelgen innovation engine to reach the market quicker” Nextflow Summit 2024 Barcelona Pixelgen: Pixelgen Technologies homepage @PixelgenTech on twitter / X and LinkedIn BioLizard: BioLizard homepage @BioLizard_nv on twitter / X and LinkedInChapters:
Start (00:00) Welcome (00:09) Introductions (00:50) Intro to Pixelgen Technologies (01:41) Lab methods (03:08) Nature Methods publication (06:15) Data visualisation (09:35) Working with single cell data (12:29) Cell interactions (15:06) Why Pixelgen chose open source (16:05) Pixelator Python package (20:10) Building within nf-core (21:40) Florian - Biolizard (23:01) Pipeline overview (25:19) Using nf-core from a commercial setting (26:50) Coming out of stealth (28:53) Extending Pixelator (32:24) Dealing with customers (34:58) Looking to the future (39:38) Interactive analysis (41:32) Advice for others (43:12) Pixelgen at the Nextflow Summit (45:16) -
Evan Floden (CEO and co-founder of Seqera) joins Phil Ewels (Product Manager for OSS at Seqera) to talk about two of the big announcements at the recent Nextflow Summit Boston 2024 - Seqera Pipelines and Seqera Containers.We dive into what these two new community offerings are, why we built them and how they work. Links:
https://seqera.io/pipelines/ https://seqera.io/containers/ Announcement blog post Nextflow Summit Boston 2024 agenda Summit YouTube playlist Nextflow 24.04 blog postTimeline:
00:00 - Introduction
00:37 - Nextflow Summit Boston 2024
01:14 - Summit talks available online
01:57 - Summit announcements
04:32 - Seqera Pipelines - motivations
05:59 - Seqera Pipelines - first look
06:48 - Testing requirements for community pipelines
10:05 - How we tested pipelines
11:53 - SRA-Explorer tangent
12:34 - Seqera Pipelines - live demo
16:14 - Getting Nextflow launch commands
19:12 - Seqera Containers - intro
24:03 - Seqera Containers - live demo
27:13 - Containers - ARM cpu arch builds
28:33 - Singularity containers
29:09 - Wave CLI
31:19 - Nextflow with Seqera Containers
33:22 - Seqera Containers - future usage
35:09 - Where to read more
36:53 - Thank you to the community
-
Join us as we discuss the brand new @nf-core / EuroFaang pre-print! 🔥 We chat to some of the key authors, covering the background to the paper, the main messages and how EuroFAANG and nf-core have collaborated. 🧬 👩🏻🔬
-
Join us in this episode of the Channels podcast for a closer look at the Nextflow Ambassadors Program. Marcel Ribeiro-Dantas and Geraldine Van der Auwera discuss the origins of the program, recent highlights and next steps, including the open call for new Ambassadors, which will close June 14.
Relevant links:
Call for applications: https://www.nextflow.io/ambassadors.html Original announcement: https://www.nextflow.io/blog/2023/introducing-nextflow-ambassador-program.html Marcel’s favorite Ambassador-contributed Nextflow blog post: https://www.nextflow.io/blog/2024/reflections-on-nextflow-mentorship.html -
This podcast episode is quite different to normal. Rather than an interview or a discussion, you'll get to watch as Ben Sherman guides Phil Ewels through the process of creating a new Nextflow Plugin from scratch. Right from the first line of code to creating a release and publishing the plugin.Nextflow plugins have been getting more attention lately and we've been getting a lot of questions in the community. There is better documentation and resources planned, but in the mean time we hope that this can be a useful guide for anyone curious in how they work, and interested in getting started.After this we will be back to our usual routine in the podcast, but we'd love to hear what you thought of this taster!
00:00:00 Welcome00:03:27 Nextflow docs00:08:34 Starting by forking nf-hello00:10:32 Overview of project files00:17:13 Trying a first compile00:18:12 Different publishing methods00:19:54 nf-boost local publish method00:23:05 Trying the new compile00:24:00 Running locally with the plugin00:27:32 Looking at the nf-hello plugin code00:39:29 Deleting files we don't need00:42:34 Finding event names in the Nextflow source00:47:46 Writing some custom functions00:49:41 Testing our new code00:51:51 Modifying the test pipeline to create files00:54:14 Coding up the JSON output00:59:20 Looking at nf-prov code for BCO files01:05:30 Testing JSON output01:09:39 Automatic work dir cleanup / nf-boost01:12:17 Publishing a release on GitHub01:19:07 Custom plugin repositories01:26:47 Publishing for all Nextflow users01:29:52 Conclusion and end -
In this episode of Channels, we talk to Jakob Zeitler - Head of R&D at Matterhorn Studios.We dive into how they use Nextflow and Seqera Platform for material science research, paving the way for cheaper and more eco-friendly products for the future ⚗️👩🏻🔬🔬🌎It may not be bioinformatics, but there's a lot that's in common!
-
In this episode of the podcast, Geraldine and Ben talk about the Global Alliance for Genomics and Health (GA4GH) and its efforts to develop interoperability standards for pipelining infrastructure.If the words TES, WES, TRS and DRS are just as much alphabet soup to you, tune in to learn all about how organizations around the world are collaborating to streamline the process of sharing and analyzing data at scale.
-
In this episode of Channels, we talk to Josh Chorlton - CTO & cofounder of BugSeq Bioinformatics Inc.We talk about how BugSeq got into using Nextflow and MultiQC and the tips and tricks that they've employed to push scale and performace of their tools to the limit.We cover topics like testing and CI/CD, passing around structured data objects between Groovy and Python using Protobufs, building Docker images at scale and the ins and outs of using MultiQC for custom clinical data reports.
-
A recap of our current training formulas, how we got here and where we’re heading to next.
Join us in this episode of the Channels podcast as Geraldine Van der Auwera and Chris Hakkaart discuss options for learning Nextflow in 2024. A recap of our current training formulas, how we got here and where we’re heading to next.
Upcoming training opportunities
Community Foundational Nextflow Training — Online, 5-6 March 2024 Training at Seqera Sessions — London, 21 March 2024 Training at Nextflow Summit — Boston, 21-22 May 2024Other upcoming events of interest
nf-core Hackathon — Online + Local Hubs, 18-20 March 2024 -
In this episode of Channels, we talk to Paolo Di Tommaso (creator of Nextflow, Seqera CTO & cofounder) and Jordi Deu Pons (software engineer @ Seqera) about Fusion - a file system written specifically for Nextflow.
-
Join us in this episode of the Channels podcast to get a recap of some of the biggest features to be added to Nextflow in 2023 and take a look at some of the things coming in 2024.
We tried to do this in Episode 27 but ended up spending nearly all the time discussing community and nf-core, so this episode is dedicated to just Nextflow features.
We cover Phil's top hits:
2023 Fusion support on Azure Batch, Google Batch, SLURM, LSF Spack integration Markdown docs, developer docs New `nextflow inspect` command Channel "topics" AWS Fargate for compute tasks 2024 Job arrays Garbage collection (aka work directory cleanup) Command line interface v2 Improvements to Nextflow packaging Workflow inputs and outputs schema Module configuration / config v2 -
Rob Lalonde of Seqera interviews Kelsey Florek, Senior Genomics and Data Scientist at the Wisconsin State Laboratory of Hygiene. In this interview, Rob and Kelsey discuss how genomic surveillance has changed since the COVID pandemic, and how Seqera Platform and Nextflow pipelines have substantially lowered the barrier to entry for bioinformaticians running workflows in public health labs and in the field.The viralrecon pipeline Kelsey mentioned in the interview can be found at https://nf-co.re/viralrecon
-
Today in the Channels podcast, we're talking about what happened this year in the Nextflow world, with particular emphasis on the many exciting developments in nf-core.We cover:- Community growth (with stats and graphs!)- The evolution of nf-core governance- The latest crop of nf-core pipelines that saw their first release this year and the expansion into domains beyond genomics (even beyond biology!)- Infrastructure updates including subworkflows and testing with nf-test- Community updates including mentorships and training- Upcoming events in 2024: nf-core online hackathon in March, Boston Summit in May and Barcelona Summit in OctoberAfter today’s episode, we’ll be going on break for the winter holiday and will return in January 2024 with a fun mix of topics. Stay tuned!
- Montre plus
