# Table of Contents - [Nextflow — Nextflow documentation](#nextflow-nextflow-documentation) - [Amazon S3 — Nextflow documentation](#amazon-s3-nextflow-documentation) - [Dataflow — Nextflow documentation](#dataflow-nextflow-documentation) - [Overview — Nextflow documentation](#overview-nextflow-documentation) - [Configuration — Nextflow documentation](#configuration-nextflow-documentation) - [Executors — Nextflow documentation](#executors-nextflow-documentation) - [Scripts — Nextflow documentation](#scripts-nextflow-documentation) - [Sharing pipelines — Nextflow documentation](#sharing-pipelines-nextflow-documentation) - [Processes — Nextflow documentation](#processes-nextflow-documentation) - [Operators — Nextflow documentation](#operators-nextflow-documentation) - [Installation — Nextflow documentation](#installation-nextflow-documentation) - [Unknown](#unknown) - [Overview — Nextflow documentation](#overview-nextflow-documentation) - [Installation — Nextflow documentation](#installation-nextflow-documentation) - [Environment setup — Nextflow documentation](#environment-setup-nextflow-documentation) - [Your first script — Nextflow documentation](#your-first-script-nextflow-documentation) - [Command line — Nextflow documentation](#command-line-nextflow-documentation) - [Caching and resuming — Nextflow documentation](#caching-and-resuming-nextflow-documentation) - [Working with files — Nextflow documentation](#working-with-files-nextflow-documentation) - [Reports — Nextflow documentation](#reports-nextflow-documentation) - [Modules — Nextflow documentation](#modules-nextflow-documentation) - [Workflows — Nextflow documentation](#workflows-nextflow-documentation) - [VS Code integration — Nextflow documentation](#vs-code-integration-nextflow-documentation) - [Secrets — Nextflow documentation](#secrets-nextflow-documentation) - [Notifications — Nextflow documentation](#notifications-nextflow-documentation) - [Git — Nextflow documentation](#git-nextflow-documentation) - [Conda environments — Nextflow documentation](#conda-environments-nextflow-documentation) - [Spack environments — Nextflow documentation](#spack-environments-nextflow-documentation) - [Wave containers — Nextflow documentation](#wave-containers-nextflow-documentation) - [Containers — Nextflow documentation](#containers-nextflow-documentation) - [Amazon Web Services — Nextflow documentation](#amazon-web-services-nextflow-documentation) - [Azure — Nextflow documentation](#azure-nextflow-documentation) - [Fusion file system — Nextflow documentation](#fusion-file-system-nextflow-documentation) - [Kubernetes — Nextflow documentation](#kubernetes-nextflow-documentation) - [Google Cloud — Nextflow documentation](#google-cloud-nextflow-documentation) - [Overview — Nextflow documentation](#overview-nextflow-documentation) - [Using plugins — Nextflow documentation](#using-plugins-nextflow-documentation) - [Nextflow plugin registry — Nextflow documentation](#nextflow-plugin-registry-nextflow-documentation) - [Feature flags — Nextflow documentation](#feature-flags-nextflow-documentation) - [Developing plugins — Nextflow documentation](#developing-plugins-nextflow-documentation) - [Syntax — Nextflow documentation](#syntax-nextflow-documentation) - [Channel factories — Nextflow documentation](#channel-factories-nextflow-documentation) - [Updating Nextflow — Nextflow documentation](#updating-nextflow-nextflow-documentation) - [Preparing for strict syntax — Nextflow documentation](#preparing-for-strict-syntax-nextflow-documentation) - [CLI reference — Nextflow documentation](#cli-reference-nextflow-documentation) - [Workflow Diagram — Nextflow documentation](#workflow-diagram-nextflow-documentation) - [Configuration scopes — Nextflow documentation](#configuration-scopes-nextflow-documentation) - [Getting started with data lineage — Nextflow documentation](#getting-started-with-data-lineage-nextflow-documentation) - [Migrating to workflow outputs — Nextflow documentation](#migrating-to-workflow-outputs-nextflow-documentation) - [Using Nextflow with Flux — Nextflow documentation](#using-nextflow-with-flux-nextflow-documentation) - [Understanding task resource metrics — Nextflow documentation](#understanding-task-resource-metrics-nextflow-documentation) - [Configuration options — Nextflow documentation](#configuration-options-nextflow-documentation) - [AWS Java SDK v2 — Nextflow documentation](#aws-java-sdk-v2-nextflow-documentation) - [Spot Instance failures and retries — Nextflow documentation](#spot-instance-failures-and-retries-nextflow-documentation) - [Using the Nextflow Gradle plugin — Nextflow documentation](#using-the-nextflow-gradle-plugin-nextflow-documentation) - [Environment variables — Nextflow documentation](#environment-variables-nextflow-documentation) - [Migrating to the Nextflow plugin registry — Nextflow documentation](#migrating-to-the-nextflow-plugin-registry-nextflow-documentation) - [Process reference — Nextflow documentation](#process-reference-nextflow-documentation) - [Operators — Nextflow documentation](#operators-nextflow-documentation) - [Overview — Nextflow documentation](#overview-nextflow-documentation) - [Getting started with rnaseq-nf — Nextflow documentation](#getting-started-with-rnaseq-nf-nextflow-documentation) - [Standard library — Nextflow documentation](#standard-library-nextflow-documentation) - [Migration notes — Nextflow documentation](#migration-notes-nextflow-documentation) - [Packages — Nextflow documentation](#packages-nextflow-documentation) - [Groovy and Java classes — Nextflow documentation](#groovy-and-java-classes-nextflow-documentation) - [Namespaces — Nextflow documentation](#namespaces-nextflow-documentation) - [Types — Nextflow documentation](#types-nextflow-documentation) - [Migrating to 25.04 — Nextflow documentation](#migrating-to-25-04-nextflow-documentation) - [Migrating to 25.10 (preview) — Nextflow documentation](#migrating-to-25-10-preview-nextflow-documentation) - [Migrating to 24.04 — Nextflow documentation](#migrating-to-24-04-nextflow-documentation) - [nextflow — Nextflow documentation](#nextflow-nextflow-documentation) - [nextflow.cache — Nextflow documentation](#nextflow-cache-nextflow-documentation) - [nextflow.cloud.aws — Nextflow documentation](#nextflow-cloud-aws-nextflow-documentation) - [nextflow.config — Nextflow documentation](#nextflow-config-nextflow-documentation) - [nextflow.executor — Nextflow documentation](#nextflow-executor-nextflow-documentation) - [nextflow.cli — Nextflow documentation](#nextflow-cli-nextflow-documentation) - [nextflow.cloud.google — Nextflow documentation](#nextflow-cloud-google-nextflow-documentation) - [nextflow.cloud.aws.nio — Nextflow documentation](#nextflow-cloud-aws-nio-nextflow-documentation) - [Migrating from DSL1 — Nextflow documentation](#migrating-from-dsl1-nextflow-documentation) - [nextflow.extension — Nextflow documentation](#nextflow-extension-nextflow-documentation) - [nextflow.scm — Nextflow documentation](#nextflow-scm-nextflow-documentation) - [nextflow.ast — Nextflow documentation](#nextflow-ast-nextflow-documentation) - [nextflow.cloud.azure — Nextflow documentation](#nextflow-cloud-azure-nextflow-documentation) - [nextflow.script — Nextflow documentation](#nextflow-script-nextflow-documentation) - [nextflow.dag — Nextflow documentation](#nextflow-dag-nextflow-documentation) - [nextflow.processor — Nextflow documentation](#nextflow-processor-nextflow-documentation) - [nextflow.container — Nextflow documentation](#nextflow-container-nextflow-documentation) - [nextflow.secret — Nextflow documentation](#nextflow-secret-nextflow-documentation) - [nextflow.k8s — Nextflow documentation](#nextflow-k8s-nextflow-documentation) - [nextflow.plugin — Nextflow documentation](#nextflow-plugin-nextflow-documentation) - [nextflow.trace — Nextflow documentation](#nextflow-trace-nextflow-documentation) - [Migrating to 24.10 — Nextflow documentation](#migrating-to-24-10-nextflow-documentation) --- # Nextflow — Nextflow documentation * [](https://www.nextflow.io/docs/latest/#) * Nextflow * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/index.md) * * * Nextflow[](https://www.nextflow.io/docs/latest/#nextflow "Permalink to this heading") ======================================================================================= _“Dataflow variables are spectacularly expressive in concurrent programming”_ [Henri E. Bal , Jennifer G. Steiner , Andrew S. Tanenbaum](https://dl.acm.org/doi/abs/10.1145/72551.72552) [![Nextflow CI](https://github.com/nextflow-io/nextflow/workflows/Nextflow%20CI/badge.svg)](https://github.com/nextflow-io/nextflow/actions/workflows/build.yml?query=branch%3Amaster+event%3Apush) [![Nextflow version](https://img.shields.io/github/release/nextflow-io/nextflow.svg?colorB=58bd9f&style=popout)](https://github.com/nextflow-io/nextflow/releases/latest) [![Nextflow Twitter](https://img.shields.io/twitter/url/https/nextflowio.svg?colorB=58bd9f&&label=%40nextflow&style=popout)](https://twitter.com/nextflowio) [![Nextflow Publication](https://img.shields.io/badge/Published-Nature%20Biotechnology-26af64.svg?colorB=58bd9f&style=popout)](https://www.nature.com/articles/nbt.3820) [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?colorB=58bd9f&style=popout)](http://bioconda.github.io/recipes/nextflow/README.html) [![Nextflow license](https://img.shields.io/github/license/nextflow-io/nextflow.svg?colorB=58bd9f&style=popout)](https://github.com/nextflow-io/nextflow/blob/master/COPYING) Nextflow is a workflow system for creating scalable, portable, and reproducible workflows. It uses a dataflow programming model that simplifies writing parallel and distributed pipelines by allowing you to focus on data flow and computation. Nextflow can deploy workflows on a variety of execution platforms, including your local machine, HPC schedulers, and cloud. Additionally, Nextflow supports a range of compute environments, software container runtimes, and package managers, allowing workflows to be executed in reproducible and isolated environments. Get started[](https://www.nextflow.io/docs/latest/#get-started "Permalink to this heading") --------------------------------------------------------------------------------------------- To get started with Nextflow: 1. See the Nextflow [overview](https://www.nextflow.io/docs/latest/overview.html#overview-page) to learn key concepts. 2. Download and [install](https://www.nextflow.io/docs/latest/install.html#install-page) Nextflow. 3. Set up an [environment](https://www.nextflow.io/docs/latest/developer-env.html#devenv-page) with the [Nextflow VS Code extension](https://www.nextflow.io/docs/latest/developer-env.html#devenv-nextflow) . 4. Run [your first script](https://www.nextflow.io/docs/latest/your-first-script.html#your-first-script) . To continue learning about Nextflow, visit the [Nextflow community training portal](https://training.nextflow.io/latest/) and find a training course that is right for you. Seqera, the company that develops Nextflow, also runs a variety of training events. See [Seqera Events](https://seqera.io/events/) for more information. Community[](https://www.nextflow.io/docs/latest/#community "Permalink to this heading") ----------------------------------------------------------------------------------------- You can post questions in the [Nextflow community forum](https://community.seqera.io/) or the [Nextflow Slack](https://www.nextflow.io/slack-invite.html) . Bugs and feature requests should be reported as [GitHub issues](https://github.com/nextflow-io/nextflow/issues/new/choose) . The Nextflow community is highly active with regular community meetings, events, a podcast, and more. You can view this material on the [Nextflow](https://www.youtube.com/@Nextflow) YouTube channel. The [nf-core](https://nf-co.re/) project is a community effort aggregating high-quality Nextflow workflows that can be used by everyone. Contributing[](https://www.nextflow.io/docs/latest/#contributing "Permalink to this heading") ----------------------------------------------------------------------------------------------- Contributions to Nextflow are welcome. See [Contributing](https://www.nextflow.io/docs/latest/developer/index.html#contributing-page) for more details. License[](https://www.nextflow.io/docs/latest/#license "Permalink to this heading") ------------------------------------------------------------------------------------- Nextflow is released under the Apache 2.0 license. Nextflow is a [registered trademark](https://github.com/nextflow-io/trademark) . Citations[](https://www.nextflow.io/docs/latest/#citations "Permalink to this heading") ----------------------------------------------------------------------------------------- If you use Nextflow in your work, please cite: P. Di Tommaso, et al. Nextflow enables reproducible computational workflows. Nature Biotechnology 35, 316–319 (2017) doi:[10.1038/nbt.3820](http://www.nature.com/nbt/journal/v35/n4/full/nbt.3820.html) --- # Amazon S3 — Nextflow documentation * [](https://www.nextflow.io/docs/latest/index.html) * Amazon S3 * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/amazons3.md) * * * Amazon S3[](https://www.nextflow.io/docs/latest/amazons3.html#amazon-s3 "Permalink to this heading") ====================================================================================================== Nextflow includes support for AWS S3 storage. Files stored in an S3 bucket can be accessed transparently in your pipeline script like any other file in the local file system. S3 path[](https://www.nextflow.io/docs/latest/amazons3.html#s3-path "Permalink to this heading") -------------------------------------------------------------------------------------------------- In order to access an S3 file, you only need to prefix the file path with the `s3` schema and the `bucket` name where it is stored. For example, if you need to access the file `/data/sequences.fa` stored in a bucket named `my-bucket`, that file can be accessed using the following fully qualified path: s3://my\-bucket/data/sequences.fa The usual file operations can be applied to a path handle with the above notation. For example, the content of an S3 file can be printed as follows: println file('s3://my-bucket/data/sequences.fa').text See [Working with files](https://www.nextflow.io/docs/latest/working-with-files.html#working-with-files) and the [Path](https://www.nextflow.io/docs/latest/reference/stdlib-types.html#stdlib-types-path) reference to learn more about available file operations. Security credentials[](https://www.nextflow.io/docs/latest/amazons3.html#security-credentials "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------------- AWS access credentials can be provided in two ways: 1. Using AWS access and secret keys in your pipeline configuration. 2. Using IAM roles to grant access to S3 storage on AWS EC2 instances. ### AWS access and secret keys[](https://www.nextflow.io/docs/latest/amazons3.html#aws-access-and-secret-keys "Permalink to this heading") The AWS access and secret keys can be specified by using the `aws` section in the `nextflow.config` configuration file as shown below: aws { accessKey \= '' secretKey \= '' region \= '' } If the access credentials are not found in the above file, Nextflow looks for AWS credentials in the following order: 1. The `nextflow.config` file in the pipeline execution directory 2. The environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` 3. The environment variables `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` 4. The profile in the AWS credentials file located at `~/.aws/credentials` * Uses the `default` profile or the environment variable `AWS_PROFILE` if set 5. The profile in the AWS client configuration file located at `~/.aws/config` * Uses the `default` profile or the environment variable `AWS_PROFILE` if set 6. The temporary AWS credentials provided by an IAM instance role. See [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) documentation for details. More information regarding [AWS Security Credentials](http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html) are available in the AWS documentation. ### IAM roles with AWS EC2 instances[](https://www.nextflow.io/docs/latest/amazons3.html#iam-roles-with-aws-ec2-instances "Permalink to this heading") When running your pipeline in an EC2 instance, IAM roles can be used to grant access to AWS resources. In this scenario, you only need to launch the EC2 instance with an IAM role which includes the `AmazonS3FullAccess` policy. Nextflow will detect and automatically acquire the permission to access S3 storage, without any further configuration. Learn more about [Using IAM Roles to Delegate Permissions to Applications that Run on AWS EC2](http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-usingrole-ec2instance.html) in the AWS documentation. China regions[](https://www.nextflow.io/docs/latest/amazons3.html#china-regions "Permalink to this heading") -------------------------------------------------------------------------------------------------------------- To use an AWS China region, make sure to specify the corresponding AWS API S3 endpoint in the Nextflow configuration file as shown below: aws { client { endpoint \= "https://s3.cn-north-1.amazonaws.com.cn" } } Read more about AWS API endpoints in the [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html) S3-compatible storage[](https://www.nextflow.io/docs/latest/amazons3.html#s3-compatible-storage "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------------------ To use S3-compatible object storage such as [Ceph](https://ceph.io/) or [Minio](https://min.io/) specify the endpoint of your storage provider and enable the [S3 path style access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) in your Nextflow configuration as shown below: aws { accessKey \= '' secretKey \= '' client { endpoint \= '' s3PathStyleAccess \= true } } Advanced configuration[](https://www.nextflow.io/docs/latest/amazons3.html#advanced-configuration "Permalink to this heading") -------------------------------------------------------------------------------------------------------------------------------- Read [AWS configuration](https://www.nextflow.io/docs/latest/reference/config.html#config-aws) section to learn more about advanced S3 client configuration options. --- # Dataflow — Nextflow documentation * [](https://www.nextflow.io/docs/latest/index.html) * Dataflow * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/channel.md) * * * Dataflow[](https://www.nextflow.io/docs/latest/channel.html#dataflow "Permalink to this heading") =================================================================================================== Nextflow uses a **dataflow** programming model to define workflows declaratively. In this model, [processes](https://www.nextflow.io/docs/latest/process.html#process-page) in a pipeline are connected to each other through _dataflow channels_ and _dataflow values_. Channels[](https://www.nextflow.io/docs/latest/channel.html#channels "Permalink to this heading") --------------------------------------------------------------------------------------------------- A _dataflow channel_ (or simply _channel_) is an asynchronous sequence of values. The values in a channel cannot be accessed directly, but only through an operator or process. For example: channel.of(1, 2, 3).view { v \-> "channel emits ${v}" } channel emits 1 channel emits 2 channel emits 3 ### Factories[](https://www.nextflow.io/docs/latest/channel.html#factories "Permalink to this heading") A channel can be created by factories in the `channel` namespace. For example, the `channel.fromPath()` factory creates a channel from a file name or glob pattern, similar to the `files()` function: channel.fromPath('input/\*.txt').view() See [Channel factories](https://www.nextflow.io/docs/latest/reference/channel.html#channel-factory) for the full list of channel factories. ### Operators[](https://www.nextflow.io/docs/latest/channel.html#operators "Permalink to this heading") Channel operators, or _operators_ for short, are functions that consume and produce channels. Because channels are asynchronous, operators are necessary to manipulate the values in a channel. Operators are particularly useful for implementing glue logic between processes. Commonly used operators include: * [combine](https://www.nextflow.io/docs/latest/reference/operator.html#operator-combine) : emit the combinations of two channels * [collect](https://www.nextflow.io/docs/latest/reference/operator.html#operator-collect) : collect the values from a channel into a list * [filter](https://www.nextflow.io/docs/latest/reference/operator.html#operator-filter) : select the values in a channel that satisfy a condition * [flatMap](https://www.nextflow.io/docs/latest/reference/operator.html#operator-flatmap) : transform each value from a channel into a list and emit each list element separately * [groupTuple](https://www.nextflow.io/docs/latest/reference/operator.html#operator-grouptuple) : group the values from a channel based on a grouping key * [join](https://www.nextflow.io/docs/latest/reference/operator.html#operator-join) : join the values from two channels based on a matching key * [map](https://www.nextflow.io/docs/latest/reference/operator.html#operator-map) : transform each value from a channel with a mapping function * [mix](https://www.nextflow.io/docs/latest/reference/operator.html#operator-mix) : emit the values from multiple channels * [view](https://www.nextflow.io/docs/latest/reference/operator.html#operator-view) : print each value in a channel to standard output See [Operators](https://www.nextflow.io/docs/latest/reference/operator.html#operator-page) for the full list of operators. Values[](https://www.nextflow.io/docs/latest/channel.html#values "Permalink to this heading") ----------------------------------------------------------------------------------------------- A _dataflow value_ is an asynchronous value. Dataflow values can be created using the [channel.value](https://www.nextflow.io/docs/latest/reference/channel.html#channel-value) factory, and they are created by processes (under [certain conditions](https://www.nextflow.io/docs/latest/process.html#process-out-singleton) ). A dataflow value cannot be accessed directly, but only through an operator or process. For example: channel.value(1).view { v \-> "dataflow value is ${v}" } dataflow value is 1 See [Value](https://www.nextflow.io/docs/latest/reference/stdlib-types.html#stdlib-types-value) for the set of available methods for dataflow values. --- # Overview — Nextflow documentation * [](https://www.nextflow.io/docs/latest/index.html) * Overview * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/overview.md) * * * Overview[](https://www.nextflow.io/docs/latest/basic.html#overview "Permalink to this heading") ================================================================================================= Why Nextflow?[](https://www.nextflow.io/docs/latest/basic.html#why-nextflow "Permalink to this heading") ---------------------------------------------------------------------------------------------------------- The rise of big data has made it increasingly necessary to be able to analyze and perform experiments on large datasets in a portable and reproducible manner. Parallelization and distributed computing are the best ways to tackle this challenge, but the tools commonly available to computational scientists often lack good support for these techniques, or they provide a model that fits poorly with the needs of computational scientists and often require knowledge of complex tools and APIs. Nextflow was created to address these challenges. The Nextflow language is inspired by [the Unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy) , in which many simple command line tools can be chained together into increasingly complex tasks. Similarly, a Nextflow script consists of composing many simple processes into increasingly complex pipelines. Each process executes a given tool or scripting language, and by specifying the process inputs and outputs, Nextflow coordinates the execution of tasks for you. The Nextflow runtime integrates with many popular execution platforms (HPC schedulers, cloud providers) and software tools (Git, Docker, Conda), allowing you to fully describe a computational pipeline with all of its dependencies and run it in nearly any environment – write once, run anywhere. Processes and dataflow[](https://www.nextflow.io/docs/latest/basic.html#processes-and-dataflow "Permalink to this heading") ----------------------------------------------------------------------------------------------------------------------------- In practice a Nextflow pipeline script is made by joining together different processes. Each process can be written in any scripting language that can be executed by the Linux platform (Bash, Perl, Ruby, Python, etc.). A process can define one or more _inputs_ and _outputs_. Data flows from process to process through asynchronous dataflow structures, known as _channels_ and _values_ in Nextflow. The data dependencies between these processes implicitly determines the flow of execution. A Nextflow script looks like this: // Script parameters params.query \= "/some/data/sample.fa" params.db \= "/some/path/pdb" process blast\_search { input: path query path db output: path "top\_hits.txt" script: """ blastp -db $db -query $query -outfmt 6 > blast\_result cat blast\_result | head -n 10 | cut -f 2 > top\_hits.txt """ } process extract\_top\_hits { input: path top\_hits path db output: path "sequences.txt" script: """ blastdbcmd -db $db -entry\_batch $top\_hits > sequences.txt """ } workflow { def query\_ch \= channel.fromPath(params.query) blast\_search(query\_ch, params.db) extract\_top\_hits(blast\_search.out, params.db).view() } The above example defines two processes. Their execution order is not determined by the fact that the `blast_search` process comes before `extract_top_hits` in the script (it could also be written the other way around). Instead, execution order is determined by their _dependencies_ – `extract_top_hits` depends on the output of `blast_search`, so `blast_search` will be executed first, and then `extract_top_hits`. When the workflow is executed, it creates two processes (`blast_search` and `extract_top_hits`) connected by the channel `query_ch`. Each process executes a task and emits a value for each input that it receives. Whenever `blast_search` emits a value, `extract_top_hits` receives it through the `query_ch` channel. See [Processes](https://www.nextflow.io/docs/latest/process.html#process-page) , [Dataflow](https://www.nextflow.io/docs/latest/channel.html#dataflow-page) , and [Workflows](https://www.nextflow.io/docs/latest/workflow.html#workflow-page) to learn more about these features. Execution abstraction[](https://www.nextflow.io/docs/latest/basic.html#execution-abstraction "Permalink to this heading") --------------------------------------------------------------------------------------------------------------------------- While a process defines _what_ command or script has to be executed, the _executor_ determines _how_ that script is actually run on the target system. If not otherwise specified, processes are executed on the local computer. The local executor is very useful for pipeline development and testing purposes, but for real world computational pipelines an HPC or cloud platform is often required. In other words, Nextflow provides an abstraction between the pipeline’s functional logic and the underlying execution system. Thus it is possible to write a pipeline once and to seamlessly run it on your computer, a grid platform, or the cloud, without modifying it, by simply defining the target execution platform in the configuration file. The following batch schedulers are supported: * [Open Grid Engine](http://gridscheduler.sourceforge.net/) * [Univa Grid Engine](http://www.univa.com/) * [Platform LSF](http://www.ibm.com/systems/technicalcomputing/platformcomputing/products/lsf/) * [SLURM](https://computing.llnl.gov/linux/slurm/) * [Flux Framework](https://flux-framework.org/) * [PBS](http://www.pbsworks.com/gridengine/) * [Torque](http://www.adaptivecomputing.com/products/open-source/torque/) * [HTCondor](https://research.cs.wisc.edu/htcondor/) The following cloud platforms are supported: * [Amazon Web Services (AWS)](https://aws.amazon.com/) * [Microsoft Azure](https://azure.microsoft.com/) * [Google Cloud Platform (GCP)](https://cloud.google.com/) * [Kubernetes](https://kubernetes.io/) Read the [Executors](https://www.nextflow.io/docs/latest/executor.html#executor-page) to learn more about the Nextflow executors. Scripting language[](https://www.nextflow.io/docs/latest/basic.html#scripting-language "Permalink to this heading") --------------------------------------------------------------------------------------------------------------------- Nextflow is a workflow language based on [Java](https://en.wikipedia.org/wiki/Java_(programming_language)) and [Groovy](https://groovy-lang.org/) . It is designed to simplify writing scalable and reproducible pipelines. In most cases, users can leverage their existing programming skills to develop Nextflow pipelines without the steep learning curve that usually comes with a new programming language. See [Scripts](https://www.nextflow.io/docs/latest/script.html#script-page) for more information about the Nextflow scripting language. Configuration options[](https://www.nextflow.io/docs/latest/basic.html#configuration-options "Permalink to this heading") --------------------------------------------------------------------------------------------------------------------------- Pipeline configuration properties are defined in a file named `nextflow.config` in the pipeline execution directory. This file can be used to define which executor to use, the process’s environment variables, pipeline parameters etc. A basic configuration file might look like this: process { executor \= 'sge' queue \= 'cn-el6' } Read the [Configuration](https://www.nextflow.io/docs/latest/config.html#config-page) section to learn more about the Nextflow configuration file and settings. --- # Configuration — Nextflow documentation * [](https://www.nextflow.io/docs/latest/index.html) * Configuration * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/config.md) * * * Configuration[](https://www.nextflow.io/docs/latest/config.html#configuration "Permalink to this heading") ============================================================================================================ Configuration file[](https://www.nextflow.io/docs/latest/config.html#configuration-file "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------- When a pipeline script is launched, Nextflow looks for configuration files in multiple locations. Since each configuration file may contain conflicting settings, they are applied in the following order (from lowest to highest priority): 1. The config file `$HOME/.nextflow/config` (or `$NXF_HOME/config` when [NXF\_HOME](https://www.nextflow.io/docs/latest/reference/env-vars.html#nxf-env-vars) is set). 2. The config file `nextflow.config` in the project directory 3. The config file `nextflow.config` in the launch directory 4. Config files specified using the `-c ` option Tip You can alternatively use the `-C ` option to specify a fixed set of configuration files and ignore all other files. Syntax[](https://www.nextflow.io/docs/latest/config.html#syntax "Permalink to this heading") ---------------------------------------------------------------------------------------------- The Nextflow configuration syntax is based on the Nextflow script syntax. It is designed for setting configuration options in a declarative manner while also allowing for dynamic expressions where appropriate. A Nextflow config file may consist of any number of _assignments_, _blocks_, and _includes_. Config files may also contain comments in the same manner as scripts. See [Syntax](https://www.nextflow.io/docs/latest/reference/syntax.html#syntax-page) for more information about the Nextflow script syntax. ### Assignments[](https://www.nextflow.io/docs/latest/config.html#assignments "Permalink to this heading") A config assignment consists of a config option and an expression separated by an equals sign: workDir \= 'work' docker.enabled \= true process.maxErrors \= 10 A config option consists of an _option name_ prefixed by any number of _scopes_ separated by dots. Config scopes are used to group related config options. See [Configuration options](https://www.nextflow.io/docs/latest/reference/config.html#config-options) for the full set of config options. The expression is typically a literal value such as a number, boolean, or string. However, any expression can be used: params.helper\_file \= "${projectDir}/assets/helper.txt" ### Blocks[](https://www.nextflow.io/docs/latest/config.html#blocks "Permalink to this heading") A config scope can also be specified as a block, which may contain multiple configuration options. For example: // dot syntax docker.enabled \= true docker.runOptions \= '-u $(id -u):$(id -g)' // block syntax docker { enabled \= true runOptions \= '-u $(id -u):$(id -g)' } As a result, deeply nested config options can be assigned in various ways. For example, the following three assignments are equivalent: executor.retry.maxAttempt \= 5 executor { retry.maxAttempt \= 5 } executor { retry { maxAttempt \= 5 } } ### Includes[](https://www.nextflow.io/docs/latest/config.html#includes "Permalink to this heading") A configuration file can include any number of other configuration files using the `includeConfig` keyword: process.executor \= 'sge' process.queue \= 'long' process.memory \= '10G' includeConfig 'path/extra.config' Relative paths are resolved against the location of the including file. Note Config includes can also be specified within config blocks. However, config files should only be included at the top level or in a [profile](https://www.nextflow.io/docs/latest/config.html#id1) so that the included config file is valid on its own and in the context in which it is included. Constants[](https://www.nextflow.io/docs/latest/config.html#constants "Permalink to this heading") ---------------------------------------------------------------------------------------------------- The following constants are globally available in a Nextflow configuration file: `baseDir: Path` Deprecated since version 20.04.0. Alias for `projectDir`. `launchDir: Path` The directory where the workflow was launched. `projectDir: Path` The directory where the main script is located. `secrets: Map` Map of pipeline secrets. See [Secrets](https://www.nextflow.io/docs/latest/secrets.html#secrets-page) for more information. Functions[](https://www.nextflow.io/docs/latest/config.html#functions "Permalink to this heading") ---------------------------------------------------------------------------------------------------- The following functions are globally available in a Nextflow configuration file: `env( name: String ) -> String` New in version 24.11.0-edge. Get the value of the environment variable with the specified name in the Nextflow launch environment. Parameters[](https://www.nextflow.io/docs/latest/config.html#parameters "Permalink to this heading") ------------------------------------------------------------------------------------------------------ Pipeline parameters can be defined in the config file using the `params` scope: params.alpha \= 123 params.beta \= 'string value .. ' params { gamma \= true delta \= "params.alpha is ${params.alpha}" } See [Pipeline parameters](https://www.nextflow.io/docs/latest/cli.html#cli-params) for information about how to specify pipeline parameters. Note When including a config file, the included config is evaluated with the parameters that are defined before the include. Parameters defined after the include are not visible to the included config. Process configuration[](https://www.nextflow.io/docs/latest/config.html#process-configuration "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------------- The `process` scope allows you to specify [process directives](https://www.nextflow.io/docs/latest/reference/process.html#process-reference) separately from the pipeline code. For example: process { executor \= 'sge' queue \= 'long' clusterOptions \= '-pe smp 10 -l virtual\_free=64G,h\_rt=30:00:00' } By using this configuration, all processes in your pipeline will be executed through the SGE cluster, with the specified settings. ### Process selectors[](https://www.nextflow.io/docs/latest/config.html#process-selectors "Permalink to this heading") The `withLabel` selectors allow the configuration of all processes annotated with a [label](https://www.nextflow.io/docs/latest/reference/process.html#process-label) directive as shown below: process { withLabel: big\_mem { cpus \= 16 memory \= 64.GB queue \= 'long' } } The above configuration example assigns 16 cpus, 64 Gb of memory and the `long` queue to all processes annotated with the `big_mem` label. In the same manner, the `withName` selector allows the configuration of a specific process in your pipeline by its name. For example: process { withName: hello { cpus \= 4 memory \= 8.GB queue \= 'short' } } The `withName` selector applies both to processes defined with the same name and processes included under the same alias. For example, `withName: hello` will apply to any process originally defined as `hello`, as well as any process included under the alias `hello`. Furthermore, selectors for the alias of an included process take priority over selectors for the original name of the process. For example, given a process defined as `hello` and included as `sayHello`, the selectors `withName: hello` and `withName: sayHello` will both be applied to the process, with the second selector taking priority over the first. Tip Label and process names do not need to be enclosed with quotes, provided the name does not include special characters (`-`, `!`, etc) and is not a keyword or a built-in type identifier. When in doubt, you can enclose the label name or process name with single or double quotes. ### Selector expressions[](https://www.nextflow.io/docs/latest/config.html#selector-expressions "Permalink to this heading") Both label and process name selectors allow the use of a regular expression in order to apply the same configuration to all processes matching the specified pattern condition. For example: process { withLabel: 'hello|bye' { cpus \= 2 memory \= 4.GB } } The above configuration snippet requests 2 cpus and 4 GB of memory for processes labeled as `hello` or `bye`. A process selector can be negated prefixing it with the special character `!`. For example: process { withLabel: 'hello' { cpus \= 2 } withLabel: '!hello' { cpus \= 4 } withName: '!align.\*' { queue \= 'long' } } The above configuration snippet sets 2 cpus for every process labeled as `hello` and 4 cpus to every process _not_ labeled as `hello`. It also specifies the `long` queue for every process whose name does _not_ start with `align`. ### Selector priority[](https://www.nextflow.io/docs/latest/config.html#selector-priority "Permalink to this heading") Process configuration settings are applied to a process in the following order (from lowest to highest priority): 1. Process configuration settings (without a selector) 2. Process directives in the process definition 3. `withLabel` selectors matching any of the process labels 4. `withName` selectors matching the process name 5. `withName` selectors matching the process included alias 6. `withName` selectors matching the process fully qualified name For example: process { cpus \= 4 withLabel: hello { cpus \= 8 } withName: bye { cpus \= 16 } withName: 'aloha:bye' { cpus \= 32 } } With the above configuration: * All processes will use 4 cpus (unless otherwise specified in their process definition). * Processes annotated with the `hello` label will use 8 cpus. * Any process named `bye` (or imported as `bye`) will use 16 cpus. * Any process named `bye` (or imported as `bye`) invoked by a workflow named `aloha` will use 32 cpus. Config profiles[](https://www.nextflow.io/docs/latest/config.html#config-profiles "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------- Configuration files can define one or more _profiles_. A profile is a set of configuration settings that can be selected during pipeline execution using the `-profile` command line option. Configuration profiles are defined in the `profiles` scope. For example: profiles { standard { process.executor \= 'local' } cluster { process.executor \= 'sge' process.queue \= 'long' process.memory \= '10GB' } cloud { process.executor \= 'cirrus' process.container \= 'cbcrg/imagex' docker.enabled \= true } } The above configuration defines three profiles: `standard`, `cluster`, and `cloud`. Each profile provides a different configuration for a given execution environment. The `standard` profile is used by default when no profile is specified. Configuration profiles can be specified at runtime as a comma-separated list: nextflow run \-profile standard,cloud Config profiles are applied in the order in which they were defined in the config file, regardless of the order they are specified on the command line. New in version 25.02.0-edge: When using the [strict config syntax](https://www.nextflow.io/docs/latest/strict-syntax.html#updating-config-syntax) , profiles are applied in the order in which they are specified on the command line. Danger When defining a profile in the config file, avoid using both the dot and block syntax for the same scope. For example: profiles { cluster { process.memory \= '2 GB' process { cpus \= 2 } } } Due to a limitation of the legacy config parser, the first setting will be overwritten by the second: $ nextflow config \-profile cluster process { cpus = 2 } This limitation can be avoided by using the [strict config syntax](https://www.nextflow.io/docs/latest/strict-syntax.html#updating-config-syntax) . Workflow handlers[](https://www.nextflow.io/docs/latest/config.html#workflow-handlers "Permalink to this heading") -------------------------------------------------------------------------------------------------------------------- Deprecated since version 25.10.0: Use a [trace observer](https://www.nextflow.io/docs/latest/plugins/developing-plugins.html#plugins-trace-observers) in a plugin to add custom workflow handlers to a pipeline via configuration. You can define workflow event handlers in the config file: workflow.onComplete \= { // any workflow property can be used here println "Pipeline complete" println "Command line: $workflow.commandLine" } workflow.onError \= { println "Error: something went wrong" } This approach is useful for handling workflow events without modifying the pipeline code. See [Workflow handlers](https://www.nextflow.io/docs/latest/notifications.html#workflow-handlers) for more information. --- # Executors — Nextflow documentation * [](https://www.nextflow.io/docs/latest/index.html) * Executors * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/executor.md) * * * Executors[](https://www.nextflow.io/docs/latest/executor.html#executors "Permalink to this heading") ====================================================================================================== In the Nextflow framework architecture, the _executor_ is the component that determines the system where a pipeline process is run and supervises its execution. The executor provides an abstraction between the pipeline processes and the underlying execution system. This allows you to write the pipeline functional logic independently from the actual processing platform. In other words, you can write your pipeline script once and have it running on your computer, a cluster resource manager, or the cloud — simply change the executor definition in the Nextflow configuration file. AWS Batch[](https://www.nextflow.io/docs/latest/executor.html#aws-batch "Permalink to this heading") ------------------------------------------------------------------------------------------------------ Nextflow supports the [AWS Batch](https://aws.amazon.com/batch/) service that allows job submission in the cloud without having to spin out and manage a cluster of virtual machines. AWS Batch uses Docker containers to run tasks, which greatly simplifies pipeline deployment. The pipeline processes must specify the Docker image to use by defining the `container` directive, either in the pipeline script or the `nextflow.config` file. To enable this executor, set `process.executor = 'awsbatch'` in the `nextflow.config` file. The pipeline can be launched either on a local computer, or an EC2 instance. EC2 is suggested for heavy or long-running workloads. Additionally, an S3 bucket must be used as the pipeline work directory. Resource requests and other job characteristics can be controlled via the following process directives: * [accelerator](https://www.nextflow.io/docs/latest/reference/process.html#process-accelerator) * [arch](https://www.nextflow.io/docs/latest/reference/process.html#process-arch) (only when using Fargate platform type for AWS Batch) * [container](https://www.nextflow.io/docs/latest/reference/process.html#process-container) * [containerOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-containeroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [disk](https://www.nextflow.io/docs/latest/reference/process.html#process-disk) (only when using Fargate platform type for AWS Batch) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [resourceLabels](https://www.nextflow.io/docs/latest/reference/process.html#process-resourcelabels) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) See [AWS Batch](https://www.nextflow.io/docs/latest/aws.html#aws-batch) for more information. Azure Batch[](https://www.nextflow.io/docs/latest/executor.html#azure-batch "Permalink to this heading") ---------------------------------------------------------------------------------------------------------- Nextflow supports the [Azure Batch](https://azure.microsoft.com/en-us/services/batch/) service that allows job submission in the cloud without having to spin out and manage a cluster of virtual machines. Azure Batch uses Docker containers to run tasks, which greatly simplifies pipeline deployment. The pipeline processes must specify the Docker image to use by defining the `container` directive, either in the pipeline script or the `nextflow.config` file. To enable this executor, set `process.executor = 'azurebatch'` in the `nextflow.config` file. The pipeline can be launched either on a local computer, or a cloud virtual machine. The cloud VM is suggested for heavy or long-running workloads. Additionally, an Azure Blob storage container must be used as the pipeline work directory. Resource requests and other job characteristics can be controlled via the following process directives: * [container](https://www.nextflow.io/docs/latest/reference/process.html#process-container) * [containerOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-containeroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [disk](https://www.nextflow.io/docs/latest/reference/process.html#process-disk) * [machineType](https://www.nextflow.io/docs/latest/reference/process.html#process-machinetype) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [resourceLabels](https://www.nextflow.io/docs/latest/reference/process.html#process-resourcelabels) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) See [Azure Batch](https://www.nextflow.io/docs/latest/azure.html#azure-batch) for more information. Bridge[](https://www.nextflow.io/docs/latest/executor.html#bridge "Permalink to this heading") ------------------------------------------------------------------------------------------------ New in version 22.09.1-edge. [Bridge](https://github.com/cea-hpc/bridge) is an abstraction layer to ease batch system and resource manager usage in heterogeneous HPC environments. It is open source software that can be installed on top of existing classical job schedulers such as Slurm, LSF, or other schedulers. Bridge allows you to submit jobs, get information on running jobs, stop jobs, get information on the cluster system, etc. For more details on how to install the Bridge system, see the [documentation](https://github.com/cea-hpc/bridge) . To enable the Bridge executor, set `process.executor = 'bridge'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) Flux Executor[](https://www.nextflow.io/docs/latest/executor.html#flux-executor "Permalink to this heading") -------------------------------------------------------------------------------------------------------------- New in version 22.11.0-edge. The `flux` executor allows you to run your pipeline script using the [Flux Framework](https://flux-framework.org/) . Nextflow submits each process to the cluster as a separate job using the `flux submit` command. To enable the Flux executor, set `process.executor = 'flux'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) Note Flux does not support the `memory` directive. Note By default, Flux will send all output to the `.command.log` file. To send this output to stdout and stderr instead, set `flux.terminalOutput = true` in your config file. Google Cloud Batch[](https://www.nextflow.io/docs/latest/executor.html#google-cloud-batch "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------------ New in version 22.07.1-edge. [Google Cloud Batch](https://cloud.google.com/batch) is a managed computing service that allows the execution of containerized workloads in the Google Cloud Platform infrastructure. Nextflow provides built-in support for the Cloud Batch API, which allows the seamless deployment of Nextflow pipelines in the cloud, offloading the pipeline process executions. The pipeline processes must specify the Docker image to use by defining the `container` directive, either in the pipeline script or the `nextflow.config` file. Additionally, the pipeline work directory must be located in a Google Storage bucket. To enable this executor, set `process.executor = 'google-batch'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [accelerator](https://www.nextflow.io/docs/latest/reference/process.html#process-accelerator) * [container](https://www.nextflow.io/docs/latest/reference/process.html#process-container) * [containerOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-containeroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [disk](https://www.nextflow.io/docs/latest/reference/process.html#process-disk) * [machineType](https://www.nextflow.io/docs/latest/reference/process.html#process-machinetype) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [resourceLabels](https://www.nextflow.io/docs/latest/reference/process.html#process-resourcelabels) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) See the [Google Cloud Batch](https://www.nextflow.io/docs/latest/google.html#google-batch) page for further configuration details. HTCondor[](https://www.nextflow.io/docs/latest/executor.html#htcondor "Permalink to this heading") ---------------------------------------------------------------------------------------------------- Warning _Experimental: may change in a future release._ The `condor` executor allows you to run your pipeline script by using the [HTCondor](https://research.cs.wisc.edu/htcondor/) resource manager. Nextflow manages each process as a separate job that is submitted to the cluster using the `condor_submit` command. The pipeline must be launched from a node where the `condor_submit` command is available, which is typically the cluster login node. Note The HTCondor executor for Nextflow does not currently support HTCondor’s ability to transfer input/output data to the corresponding job’s compute node. Therefore, the data must be made accessible to the compute nodes through a shared file system directory from where the Nextflow workflow is executed (or specified via the `-w` option). To enable the HTCondor executor, set `process.executor = 'condor'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [disk](https://www.nextflow.io/docs/latest/reference/process.html#process-disk) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) HyperQueue[](https://www.nextflow.io/docs/latest/executor.html#hyperqueue "Permalink to this heading") -------------------------------------------------------------------------------------------------------- New in version 22.05.0-edge. Changed in version 24.06.0-edge: HyperQueue 0.17.0 or later is required. Changed in version 25.01.0-edge: HyperQueue 0.20.0 or later is required. The `hyperqueue` executor allows you to run your pipeline script by using the [HyperQueue](https://github.com/It4innovations/hyperqueue) job scheduler. Nextflow manages each process as a separate job that is submitted to the cluster using the `hq` command line tool. The pipeline must be launched from a node where the `hq` command is available, which is typically the cluster login node. To enable the HyperQueue executor, set `process.executor = 'hq'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [accelerator](https://www.nextflow.io/docs/latest/reference/process.html#process-accelerator) * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) Kubernetes[](https://www.nextflow.io/docs/latest/executor.html#kubernetes "Permalink to this heading") -------------------------------------------------------------------------------------------------------- The `k8s` executor allows you to run a pipeline on a [Kubernetes](http://kubernetes.io/) cluster. Resource requests and other job characteristics can be controlled via the following process directives: * [accelerator](https://www.nextflow.io/docs/latest/reference/process.html#process-accelerator) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [disk](https://www.nextflow.io/docs/latest/reference/process.html#process-disk) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [pod](https://www.nextflow.io/docs/latest/reference/process.html#process-pod) * [resourceLabels](https://www.nextflow.io/docs/latest/reference/process.html#process-resourcelabels) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) See the [Kubernetes](https://www.nextflow.io/docs/latest/kubernetes.html#k8s-page) page to learn how to set up a Kubernetes cluster to run Nextflow pipelines. Local[](https://www.nextflow.io/docs/latest/executor.html#local "Permalink to this heading") ---------------------------------------------------------------------------------------------- The `local` executor is used by default. It runs the pipeline processes on the computer where Nextflow is launched. The processes are parallelised by spawning multiple threads, taking advantage of the multi-core architecture of the CPU. The `local` executor is useful for developing and testing a pipeline script on your computer, before switching to a cluster or cloud environment with production data. Resource requests and other job characteristics can be controlled via the following process directives: * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) * [container](https://www.nextflow.io/docs/latest/reference/process.html#process-container) * [containerOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-containeroptions) Note While the `local` executor limits the number of concurrent tasks based on requested vs available resources, it does not enforce task resource requests. In other words, it is possible for a local task to use more CPUs and memory than it requested, in which case it may starve other tasks. An exception to this behavior is when using [Docker](https://www.nextflow.io/docs/latest/container.html#container-docker) or [Podman](https://www.nextflow.io/docs/latest/container.html#container-podman) containers, in which case the resource requests are enforced by the container runtime. The local executor supports two types of tasks: * Script tasks (processes with a `script` or `shell` block) - executed via a Bash wrapper * Native tasks (processes with an `exec` block) - executed directly in the JVM. LSF[](https://www.nextflow.io/docs/latest/executor.html#lsf "Permalink to this heading") ------------------------------------------------------------------------------------------ The `lsf` executor allows you to run your pipeline script using a [Platform LSF](http://en.wikipedia.org/wiki/Platform_LSF) cluster. Nextflow manages each process as a separate job that is submitted to the cluster using the `bsub` command. The pipeline must be launched from a node where the `bsub` command is available, which is typically the cluster login node. To enable the LSF executor, set `process.executor = 'lsf'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) Note LSF supports both _per-core_ and _per-job_ memory limits. Nextflow assumes that LSF works in the _per-core_ mode, thus it divides the requested [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) by the number of requested [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) . When LSF is configured to work in the _per-job_ memory limit mode, you must specify this limit with the `perJobMemLimit` option in the [executor](https://www.nextflow.io/docs/latest/reference/config.html#config-executor) scope of your Nextflow config file. See also the [Platform LSF documentation](https://www.ibm.com/support/knowledgecenter/SSETD4_9.1.3/lsf_config_ref/lsf.conf.lsb_job_memlimit.5.dita) . Moab[](https://www.nextflow.io/docs/latest/executor.html#moab "Permalink to this heading") -------------------------------------------------------------------------------------------- New in version 19.07.0. Warning _Experimental: may change in a future release._ The `moab` executor allows you to run your pipeline script using the [Moab](https://en.wikipedia.org/wiki/Moab_Cluster_Suite) resource manager by [Adaptive Computing](http://www.adaptivecomputing.com/) . Nextflow manages each process as a separate job that is submitted to the cluster using the `msub` command provided by the resource manager. The pipeline must be launched from a node where the `msub` command is available, which is typically the cluster login node. To enable the `Moab` executor, set `process.executor = 'moab'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) NQSII[](https://www.nextflow.io/docs/latest/executor.html#nqsii "Permalink to this heading") ---------------------------------------------------------------------------------------------- The `nqsii` executor allows you to run your pipeline script using the [NQSII](https://www.rz.uni-kiel.de/en/our-portfolio/hiperf/nec-linux-cluster) resource manager. Nextflow manages each process as a separate job that is submitted to the cluster using the `qsub` command provided by the scheduler. The pipeline must be launched from a node where the `qsub` command is available, which is typically the cluster login node. To enable the NQSII executor, set `process.executor = 'nqsii'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) OAR[](https://www.nextflow.io/docs/latest/executor.html#oar "Permalink to this heading") ------------------------------------------------------------------------------------------ New in version 19.11.0-edge. The `oar` executor allows you to run your pipeline script using the [OAR](https://oar.imag.fr/) resource manager. Nextflow manages each process as a separate job that is submitted to the cluster using the `oarsub` command. The pipeline must be launched from a node where the `oarsub` command is available, which is typically the cluster login node. To enable the OAR executor set `process.executor = 'oar'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) When specifying `clusterOptions` as a string, multiple options must be separated by semicolons to ensure that the job script is formatted correctly: clusterOptions \= '-t besteffort;--project myproject' New in version 24.04.0. The same behavior can now be achieved using a string list: clusterOptions \= \[ '-t besteffort', '--project myproject' \] See [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) for details. PBS/Torque[](https://www.nextflow.io/docs/latest/executor.html#pbs-torque "Permalink to this heading") -------------------------------------------------------------------------------------------------------- The `pbs` executor allows you to run your pipeline script using a resource manager from the [PBS/Torque](http://en.wikipedia.org/wiki/Portable_Batch_System) family of batch schedulers. Nextflow manages each process as a separate job that is submitted to the cluster using the `qsub` command provided by the scheduler. The pipeline must be launched from a node where the `qsub` command is available, which is typically the cluster login node. To enable the PBS executor, set `process.executor = 'pbs'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) PBS Pro[](https://www.nextflow.io/docs/latest/executor.html#pbs-pro "Permalink to this heading") -------------------------------------------------------------------------------------------------- The `pbspro` executor allows you to run your pipeline script using the [PBS Pro](https://www.pbspro.org/) resource manager. Nextflow manages each process as a separate job that is submitted to the cluster using the `qsub` command provided by the scheduler. The pipeline must be launched from a node where the `qsub` command is available, which is typically the cluster login node. To enable the PBS Pro executor, set `process.executor = 'pbspro'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) SGE[](https://www.nextflow.io/docs/latest/executor.html#sge "Permalink to this heading") ------------------------------------------------------------------------------------------ The `sge` executor allows you to run your pipeline script using a [Sun Grid Engine](http://en.wikipedia.org/wiki/Oracle_Grid_Engine) cluster or a compatible platform ([Open Grid Engine](http://gridscheduler.sourceforge.net/) , [Univa Grid Engine](http://www.univa.com/products/grid-engine.php) , etc). Nextflow manages each process as a separate grid job that is submitted to the cluster using the `qsub` command. The pipeline must be launched from a node where the `qsub` command is available, which is typically the cluster login node. To enable the SGE executor, set `process.executor = 'sge'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [penv](https://www.nextflow.io/docs/latest/reference/process.html#process-penv) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) SLURM[](https://www.nextflow.io/docs/latest/executor.html#slurm "Permalink to this heading") ---------------------------------------------------------------------------------------------- The `slurm` executor allows you to run your pipeline script using the [SLURM](https://slurm.schedmd.com/documentation.html) resource manager. Nextflow manages each process as a separate job that is submitted to the cluster using the `sbatch` command. The pipeline must be launched from a node where the `sbatch` command is available, which is typically the cluster login node. To enable the SLURM executor, set `process.executor = 'slurm'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [cpus](https://www.nextflow.io/docs/latest/reference/process.html#process-cpus) * [memory](https://www.nextflow.io/docs/latest/reference/process.html#process-memory) * [queue](https://www.nextflow.io/docs/latest/reference/process.html#process-queue) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) Note SLURM partitions can be specified with the `queue` directive. Note Nextflow does not provide direct support for SLURM multi-clusters. If you need to submit workflow executions to a cluster other than the current one, specify it with the `SLURM_CLUSTERS` variable in the launch environment. New in version 23.07.0-edge: Some SLURM clusters require memory allocations to be specified with `--mem-per-cpu` instead of `--mem`. You can specify `executor.perCpuMemAllocation = true` in the Nextflow configuration to enable this behavior. Nextflow will automatically compute the memory per CPU for each task (by default 1 CPU is used). TCS[](https://www.nextflow.io/docs/latest/executor.html#tcs "Permalink to this heading") ------------------------------------------------------------------------------------------ The `tcs` executor allows you to run your pipeline script using a [Fujitsu Technical Computing Suite (TCS)](https://software.fujitsu.com/jp/manual/manualindex/p21000155e.html) . Nextflow manages each process as a separate job that is submitted to the cluster using the `pjsub` command. The pipeline must be launched from a node where the `pjsub` command is available, which is typically the login node. To enable the TCS executor, set `process.executor = 'tcs'` in the `nextflow.config` file. Resource requests and other job characteristics can be controlled via the following process directives: * [clusterOptions](https://www.nextflow.io/docs/latest/reference/process.html#process-clusteroptions) * [time](https://www.nextflow.io/docs/latest/reference/process.html#process-time) Note Use `clusterOptions` to specify system-dependent options such as queue (resource group), CPU, and node. These options vary across target systems and are not standardized. They correspond to `-L` options in the arguments of the `pjsub` command and should be configured according to the requirements of the specific cluster environment. For example: process { executor \= 'tcs' time \= '00:30:00' clusterOptions \= '-L rscgrp=a-batch -L vnode-core=4' } --- # Scripts — Nextflow documentation * [](https://www.nextflow.io/docs/latest/index.html) * Scripts * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/script.md) * * * Scripts[](https://www.nextflow.io/docs/latest/script.html#scripts "Permalink to this heading") ================================================================================================ Nextflow is a workflow language that runs on the Java virtual machine (JVM). Nextflow’s syntax is very similar to [Groovy](https://groovy-lang.org/) , a scripting language for the JVM. However, Nextflow is specialized for writing computational pipelines in a declarative manner. See [Syntax](https://www.nextflow.io/docs/latest/reference/syntax.html#syntax-page) for a full description of the Nextflow language. Nextflow scripts can also make full use of the Java and Groovy standard libraries. See [Standard library](https://www.nextflow.io/docs/latest/reference/stdlib.html#stdlib-page) for more information. Warning Nextflow uses UTF-8 as the default character encoding for source files. Make sure to use UTF-8 encoding when editing Nextflow scripts with your preferred text editor. Warning Nextflow scripts have a maximum size of 64 KiB. To avoid this limit for large pipelines, consider moving pipeline components into separate files and including them as modules. Hello world[](https://www.nextflow.io/docs/latest/script.html#hello-world "Permalink to this heading") -------------------------------------------------------------------------------------------------------- You can use the `println` function to print to the console: println 'Hello, World!' Variables[](https://www.nextflow.io/docs/latest/script.html#variables "Permalink to this heading") ---------------------------------------------------------------------------------------------------- Variables are declared using the `def` keyword: def num \= 1 println num def date \= new java.util.Date() println date def x \= \-3.1499392 println x def flag \= false println flag def str \= "Hi" println str Warning Variables can also be declared without `def` in some cases. However, this practice is discouraged outside of simple code snippets because it can lead to a [race condition](https://www.nextflow.io/docs/latest/cache-and-resume.html#cache-global-var-race-condition) . Lists[](https://www.nextflow.io/docs/latest/script.html#lists "Permalink to this heading") -------------------------------------------------------------------------------------------- Lists are defined using square brackets: def myList \= \[1776, \-1, 33, 99, 0, 928734928763\] You can access a given item in the list with square-bracket notation (indexes start at 0): println myList\[0\] In order to get the length of the list use the `size` method: println myList.size() See [List](https://www.nextflow.io/docs/latest/reference/stdlib-types.html#stdlib-types-list) for the set of available list operations. Maps[](https://www.nextflow.io/docs/latest/script.html#maps "Permalink to this heading") ------------------------------------------------------------------------------------------ Maps are used to store _associative arrays_ (also known as _dictionaries_). They are unordered collections of heterogeneous, named data: def scores \= \["Brett": 100, "Pete": "Did not finish", "Andrew": 86.87934\] Note that each of the values stored in the map can be of a different type. `Brett` is an integer, `Pete` is a string, and `Andrew` is a floating-point number. We can access the values in a map in two main ways: println scores\["Pete"\] println scores.Pete To add data to or modify a map, the syntax is similar to adding values to list: scores\["Pete"\] \= 3 scores\["Cedric"\] \= 120 You can also use the `+` operator to add two maps together: def new\_scores \= scores + \["Pete": 3, "Cedric": 120\] When adding two maps, the first map is copied and then appended with the keys from the second map. Any conflicting keys are overwritten by the second map. Tip Copying a map with the `+` operator is a safer way to modify maps in Nextflow, specifically when passing maps through channels. This way, a new instance of the map will be created, and any references to the original map won’t be affected. See [Map](https://www.nextflow.io/docs/latest/reference/stdlib-types.html#stdlib-types-map) for the set of available map operations. Tuples[](https://www.nextflow.io/docs/latest/script.html#tuples "Permalink to this heading") ---------------------------------------------------------------------------------------------- Tuples are used to store a fixed sequence of heterogeneous values. They are created using the `tuple` function: person \= tuple('Alice', 42, false) Tuple elements are accessed by index: name \= person\[0\] age \= person\[1\] is\_male \= person\[2\] Tuples can be destructured in assignments: (name, age, is\_male) \= person As well as closure parameters: coords \= \[\ tuple(1, 2),\ tuple(2, 4),\ tuple(3, 6),\ tuple(4, 8)\ \] coords.each { x, y \-> println "x=$x, y=$y" } Tuples are immutable – once a tuple is created, its elements cannot be modified. See [Tuple](https://www.nextflow.io/docs/latest/reference/stdlib-types.html#stdlib-types-tuple) for the set of available tuple operations. Operators[](https://www.nextflow.io/docs/latest/script.html#operators "Permalink to this heading") ---------------------------------------------------------------------------------------------------- Operators are symbols that perform specific functions on one or more values, and generally make code easier to read. This section highlights some of the most commonly used operators. Note Operators in this context are different from _channel operators_, which are specialized functions for working with channels. See [Dataflow](https://www.nextflow.io/docs/latest/channel.html#dataflow-page) for more information. The `==` and `!=` operators can be used to test whether any two values are equal (or not equal): assert 2 + 2 \== 4 assert \[2, 2\] != \[4\] assert 'two plus two' != 'four' Tip The `assert` keyword simply tests a condition and raises an error if the condition is false. Every assert that you see on this page will succeed if executed. Comparison operators can be used to compare two values: assert 3 < 3.14 // numbers are compared as numbers assert 3 <= 3 assert 'hello' < 'world' // strings are compared alphabetically Logical operators can be used to perform Boolean logic: assert true && false \== false // logical AND assert true || false \== true // logical OR assert !true \== false // logical NOT The `in` and `!in` operators can be used to test _membership_, i.e. whether a collection contains a value: assert 2 in \[1, 2, 3\] assert 'a' in \[a: 1, b: 2, c: 3\] Arithmetic operators can be used to do math: assert 2 + 2 \== 4 assert 2 \- 2 \== 0 assert 2 \* 2 \== 4 assert 2 / 2 == 1.0 assert 2 \*\* 2 == 4 // exponent assert 2 % 2 == 0 // modulo (division remainder) Some arithmetic operators can be used with other types of values. For example, `+` can be used to concatenate lists, maps, and strings: assert \[1, 2, 3\] + \[4\] \== \[1, 2, 3, 4\] Conditional execution[](https://www.nextflow.io/docs/latest/script.html#conditional-execution "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------------- One of the most important features of any programming language is the ability to execute different code under different conditions. This can be done with an if-else statement: def x \= Math.random() if( x < 0.5 ) { println 'You lost.' } else { println 'You won!' } In some cases, conditional statements can be expressed more concisely as a conditional expression (also known as a _ternary expression_): def message \= Math.random() < 0.5 ? 'You lost.' : 'You won!' println message A shortened version of the conditional expression can be used to return a value if it is “truthy”, or fallback to a second value otherwise: def counts \= \['A': 1, 'B', 2\] assert counts\['C'\] ?: 0 \== 0 // x is "truthy" if !!x == true Tip The `?:` operator is also known as the [elvis operator](https://en.wikipedia.org/wiki/Elvis_operator) . Strings[](https://www.nextflow.io/docs/latest/script.html#strings "Permalink to this heading") ------------------------------------------------------------------------------------------------ Strings can be defined by enclosing text in single or double quotes (`'` or `"` characters): println "he said 'cheese' once" println 'he said "cheese!" again' Strings can be concatenated with `+`: def a \= "world" print "hello " + a + "\\n" ### String interpolation[](https://www.nextflow.io/docs/latest/script.html#string-interpolation "Permalink to this heading") There is an important difference between single-quoted and double-quoted strings: Double-quoted strings support variable interpolations, while single-quoted strings do not. In practice, double-quoted strings can contain the value of an arbitrary variable by prefixing its name with the `$` character, or the value of any expression by using the `${expression}` syntax, similar to Bash/shell scripts: def foxtype \= 'quick' def foxcolor \= \['b', 'r', 'o', 'w', 'n'\] println "The $foxtype ${foxcolor.join()} fox" def x \= 'Hello' println '$x + $y' This code prints: The quick brown fox $x + $y ### Multi-line strings[](https://www.nextflow.io/docs/latest/script.html#multi-line-strings "Permalink to this heading") A block of text that span multiple lines can be defined by delimiting it with triple single or double quotes: def text \= """ hello there James how are you today? """ Note Like before, multi-line strings inside double quotes support variable interpolation, while single-quoted multi-line strings do not. As in Bash/shell scripts, terminating a line in a multi-line string with a `\` character prevents a newline character from separating that line from the one that follows: def myLongCmdline \= """ blastp \\ -in $input\_query \\ -out $output\_file \\ -db $blast\_database \\ -html """ def result \= myLongCmdline.execute().text In the preceding example, `blastp` and its `-in`, `-out`, `-db` and `-html` switches and their arguments are effectively a single line. Warning Do not put any spaces after the backslash when using backslashes to continue a multi-line command. Spaces after the backslash will be interpreted as an escaped space and will make your script incorrect. It will also print this warning: unknown recognition error type: groovyjarjarantlr4.v4.runtime.LexerNoViableAltException Regular expressions[](https://www.nextflow.io/docs/latest/script.html#regular-expressions "Permalink to this heading") ------------------------------------------------------------------------------------------------------------------------ Regular expressions are the Swiss Army knife of text processing. They provide the ability to match and extract patterns from strings. Use `=~` to check whether a given pattern occurs anywhere in a string: assert 'hello' \=~ /hello/ assert 'hello world' \=~ /hello/ Use `==~` to check whether a string matches a given regular expression pattern exactly. assert 'hello' \==~ /hello/ assert !('hello world' \==~ /hello/) ### String replacement[](https://www.nextflow.io/docs/latest/script.html#string-replacement "Permalink to this heading") To replace pattern occurrences in a given string, use the `replaceFirst` and `replaceAll` methods: def x \= "colour".replaceFirst(/ou/, "o") println x // prints: color def y \= "cheesecheese".replaceAll(/cheese/, "nice") println y // prints: nicenice To remove part of a string, simply replace it with a blank string: def z \= 'Hello World!'.replaceFirst(/(?i)\\s+Wo\\w+/, '') println z // prints: Hello! ### Capturing groups[](https://www.nextflow.io/docs/latest/script.html#capturing-groups "Permalink to this heading") You can match a pattern that includes groups. First create a matcher object with the `=~` operator. Then, you can index the matcher object to find the matches: `matcher[0]` returns a list representing the first match of the regular expression in the string. The first list element is the string that matches the entire regular expression, and the remaining elements are the strings that match each group. Here’s how it works: def programVersion \= '2.7.3-beta' def m \= programVersion \=~ /(\\d+)\\.(\\d+)\\.(\\d+)-?(.+)/ assert m\[0\] \== \['2.7.3-beta', '2', '7', '3', 'beta'\] assert m\[0\]\[1\] \== '2' assert m\[0\]\[2\] \== '7' assert m\[0\]\[3\] \== '3' assert m\[0\]\[4\] \== 'beta' Applying some syntactic sugar, you can do the same in just one line of code: def programVersion \= '2.7.3-beta' def (full, major, minor, patch, flavor) \= (programVersion \=~ /(\\d+)\\.(\\d+)\\.(\\d+)-?(.+)/)\[0\] println full // 2.7.3-beta println major // 2 println minor // 7 println patch // 3 println flavor // beta Closures[](https://www.nextflow.io/docs/latest/script.html#closures "Permalink to this heading") -------------------------------------------------------------------------------------------------- A closure is a function that can be used like a regular value. Typically, closures are passed as arguments to _higher-order functions_ to express computations in a declarative manner. For example: def square \= { v \-> v \* v } The above example defines a closure, which takes one parameter named `v` and returns the “square” of `v` (`v * v`). The closure is assigned to the variable `square`. `square` can now be called like a function: println square(9) The above example prints `81`. The main use case for a closure is as an argument to a higher-order function: \[ 1, 2, 3, 4 \].collect(square) The `collect` method of a list applies a mapping function to each value in the list and produces a new list. The above example produces: \[ 1, 4, 9, 16 \] The example can be expressed more concisely as: \[ 1, 2, 3, 4 \].collect { v \-> v \* v } Another example is the `each` method of a map, which takes a closure with two arguments corresponding to the key and value of each map entry: \[ "Yue" : "Wu", "Mark" : "Williams", "Sudha" : "Kumari" \].each { key, value \-> println "$key = $value" } Prints: Yue \= Wu Mark \= Williams Sudha \= Kumari Closures can access variables outside of their scope: def counts \= \["China": 1, "India": 2, "USA": 3\] def result \= 0 counts.keySet().each { v \-> result += counts\[v\] } println result A closure can also declare local variables that exist only for the lifetime of each closure invocation: def result \= 0 myMap.keySet().each { v \-> def count \= myMap\[v\] result += count } While the `each` method is a convenient way to iterate through a collection and build up some result, a more idiomatic way to do this is to use the `inject` method: def result \= counts.values().inject { sum, v \-> sum + v } This way, the closure is fully “self-contained” because it doesn’t access or mutate any variables outside of its scope. Note When a closure takes a single parameter, the parameter can be omitted, in which case the implicit `it` parameter will be used: \[1, 2, 3\].each { println it } Script definitions[](https://www.nextflow.io/docs/latest/script.html#script-definitions "Permalink to this heading") ---------------------------------------------------------------------------------------------------------------------- So far, we have been focusing on the basic building blocks of Nextflow code, like variables, lists, strings, and closures. In practice, however, Nextflow scripts are composed of _workflows_, _processes_, and _functions_ (collectively known as _definitions_), and can _include_ definitions from other scripts. To transition a code snippet into a proper workflow script, simply wrap it in a `workflow` block: workflow { println 'Hello!' } This block is called the _entry workflow_. It serves as the entrypoint when the script is executed. A script can only have one entry workflow. Whenever a script contains only simple statements like `println 'Hello!'`, Nextflow simply treats it as an entry workflow. You can also break up code into functions, for example: def sayHello() { println 'Hello!' } def add(a, b) { a + b } workflow { sayHello() println "2 + 2 = ${add(2, 2)}!" } See [Workflows](https://www.nextflow.io/docs/latest/workflow.html#workflow-page) , [Processes](https://www.nextflow.io/docs/latest/process.html#process-page) , and [Modules](https://www.nextflow.io/docs/latest/module.html#module-page) for more information about how to use these features in your Nextflow scripts. --- # Sharing pipelines — Nextflow documentation * [](https://www.nextflow.io/docs/latest/index.html) * Sharing pipelines * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/sharing.md) * * * Sharing pipelines[](https://www.nextflow.io/docs/latest/sharing.html#sharing-pipelines "Permalink to this heading") ===================================================================================================================== Nextflow seamlessly integrates with popular Git providers, including [BitBucket](http://bitbucket.org/) , [GitHub](http://github.com/) , and [GitLab](http://gitlab.com/) for managing Nextflow pipelines as version-controlled Git repositories. This feature allows you to easily use other people’s Nextflow pipelines and publish your own pipelines. Note Nextflow is not meant to completely replace the [Git](https://git-scm.com/) tool. You may still need `git` to create new repositories or commit changes, etc. Git configuration[](https://www.nextflow.io/docs/latest/sharing.html#git-configuration "Permalink to this heading") --------------------------------------------------------------------------------------------------------------------- You can configure your credentials for various Git providers in the Git configuration file, located at `$HOME/.nextflow/scm`. See [Git](https://www.nextflow.io/docs/latest/git.html#git-page) for more information. Using a local repository[](https://www.nextflow.io/docs/latest/sharing.html#using-a-local-repository "Permalink to this heading") ----------------------------------------------------------------------------------------------------------------------------------- Nextflow can work with repositories stored in a local or shared file system. The repository must be created as a [bare repository](https://craftquest.io/articles/what-is-a-bare-git-repository) . For example, given a bare repository at `/shared/projects/hello.git`, Nextflow is able to run it using the following syntax: nextflow run file:/shared/projects/hello.git See [Git documentation](https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server) for more details about how create and manage bare repositories. Publishing your pipeline[](https://www.nextflow.io/docs/latest/sharing.html#publishing-your-pipeline "Permalink to this heading") ----------------------------------------------------------------------------------------------------------------------------------- In order to publish your Nextflow pipeline to GitHub (or any other supported platform) and allow other people to use it, you only need to create a GitHub repository containing all your project script and data files. If you don’t know how to do it, follow this simple tutorial that explains how [create a GitHub repository](https://help.github.com/articles/create-a-repo) . Nextflow only requires that the main script in your pipeline project is called `main.nf`. A different name can be used by specifying the `manifest.mainScript` attribute in the `nextflow.config` file that must be included in your project. For example: manifest.mainScript \= 'my\_very\_long\_script\_name.nf' To learn more about this and other project metadata information, that can be defined in the Nextflow configuration file, read the [Manifest](https://www.nextflow.io/docs/latest/reference/config.html#config-manifest) section on the Nextflow configuration page. Once you have uploaded your pipeline project to GitHub other people can execute it simply using the project name or the repository URL. For if your GitHub account name is `acme` and you have uploaded a project into a repository named `hello` the repository URL will be `http://github.com/acme/hello` and people will able to download and run it by using either the command: nextflow run acme/hello or nextflow run http://github.com/acme/hello See the [CLI](https://www.nextflow.io/docs/latest/cli.html#cli-page) page to learn how to use the Nextflow command line to run pipelines and manage pipeline projects. Managing dependencies[](https://www.nextflow.io/docs/latest/sharing.html#managing-dependencies "Permalink to this heading") ----------------------------------------------------------------------------------------------------------------------------- Computational pipelines are rarely composed by a single script. In real world applications they depend on many other components, including other scripts and tools, databases, and specialized environments which provide compute and storage. These external dependencies are the primary challenge when sharing software, because the users need to recreate the environment around a tool in order to use it. This setup process is often painful and error prone, which severely hinders the ability to reproduce computational results on a system other than the one on which it was originally developed. Nextflow tackles this problem by integrating with existing tools for reproducible software, namely Git for source code and Docker for containers. These tools allow you to keep all the dependencies of your pipeline project in one place and track changes over time with version control. By making your pipeline project is self-contained, meaning all of its dependencies are fully defined in the project itself, you gain two major advantages: * **Portability**: the pipeline can be run in virtually any environment with a Java VM and a container runtime * **Reproducibility**: any results produced by the pipelined can be easily reproduced, even across different environments One way to account for dependencies is to break them down into three categories: code, data, and environment. Here we will describe how to include each of these dependencies in your Nextflow pipeline: ### Code[](https://www.nextflow.io/docs/latest/sharing.html#code "Permalink to this heading") Aside from pipeline scripts, you may have additional scripts and tools used by individual tasks. #### Standard dependencies[](https://www.nextflow.io/docs/latest/sharing.html#standard-dependencies "Permalink to this heading") Many standard tools can be accessed as Docker containers or Conda/Spack packages. In this case, you only need to specify the container image URL (e.g. from [DockerHub](https://hub.docker.com/) ) or package name with the `container` or `conda` directive, respectively. Make sure to enable the desired method in your Nextflow configuration: // containers docker.enabled \= true // conda packages conda.enabled \= true This way, when you launch your pipeline, Nextflow will automatically download the necessary dependencies to run your tasks based on this configuration. Read the [Containers](https://www.nextflow.io/docs/latest/container.html#container-page) page to learn more about how to use containers with Nextflow, and the [Conda environments](https://www.nextflow.io/docs/latest/conda.html#conda-page) page for Conda packages. Tip For maximal reproducibility, make sure to define a specific version for each tool. Otherwise, your pipeline might use different versions across subsequent runs, which can introduce subtle differences to your results. #### The `bin` directory[](https://www.nextflow.io/docs/latest/sharing.html#the-bin-directory "Permalink to this heading") As for custom scripts, you can include executable scripts in the `bin` directory of your pipeline repository. When configured correctly, these scripts can be executed like a regular command from any process script (i.e. without modifying the `PATH` environment variable or using an absolute path), and changing the script will cause the task to be re-executed on a resumed run (i.e. just like changing the process script itself). To configure a custom script: 1. Save the script in the `bin` directory (relative to the pipeline repository root). 2. Specify a portable shebang (see note below for details). 3. Make the script executable. For example: `chmod a+x bin/my_script.py` Tip To maximize the portability of your bundled script, use `env` to dynamically resolve the location of the interpreter instead of hard-coding it in the shebang line. For example, shebang definitions `#!/usr/bin/python` and `#!/usr/local/bin/python` both hard-code specific paths to the Python interpreter. Instead, the following approach is more portable: #!/usr/bin/env python #### The `lib` directory[](https://www.nextflow.io/docs/latest/sharing.html#the-lib-directory "Permalink to this heading") The `lib` directory can be used to add utility code or external libraries without cluttering the pipeline scripts. The `lib` directory in the Nextflow project root is added to the classpath by default. Classes defined in the `lib` directory will be available in pipeline scripts. Functions defined outside of classes will not be available in pipeline scripts. For example, `lib/DNASequence.groovy` defines the `DNASequence` class: // lib/DNASequence.groovy class DNASequence { String sequence // Constructor DNASequence(String sequence) { this.sequence \= sequence.toUpperCase() // Ensure sequence is in uppercase for consistency } // Method to calculate melting temperature using the Wallace rule double getMeltingTemperature() { int g\_count \= sequence.count('G') int c\_count \= sequence.count('C') int a\_count \= sequence.count('A') int t\_count \= sequence.count('T') // Wallace rule calculation double tm \= 4 \* (g\_count + c\_count) + 2 \* (a\_count + t\_count) return tm } String toString() { return "DNA\[$sequence\]" } } The `DNASequence` class is available in the execution context: // main.nf workflow { channel.of('ACGTTGCAATGCCGTA', 'GCGTACGGTACGTTAC') .map { seq \-> new DNASequence(seq) } .view { dna \-> "Found sequence '$dna' with melting temperaure ${dna.getMeltingTemperature()}°C" } } It prints: Found sequence 'DNA\[ACGTTGCAATGCCGTA\]' with melting temperaure 48.0°C Found sequence 'DNA\[GCGTACGGTACGTTAC\]' with melting temperaure 50.0°C Note Package declarations in the `lib` directory are ignored. The package of a class is determined by the directory structure within the `lib` directory. For example, if the above example were defined in `lib/utils/DNASequence.groovy`, the class would need to be referenced in pipeline scripts as `utils.DNASequence`. ### Data[](https://www.nextflow.io/docs/latest/sharing.html#data "Permalink to this heading") In general, input data should be provided by external sources using parameters which can be controlled by the user. This way, a pipeline can be easily reused to process different datasets which are appropriate for the pipeline. Parameters can be declared with default values in the main script or in the configuration file: params.my\_input \= 'default input file' params.my\_output \= 'default output path' params.my\_flag \= false // ... When launching a pipeline, parameter values can be provided on the command line or in a params file (using the `-params-file` option). Options prefixed with a double dash (`--`) are interpreted as parameters: nextflow run \--my\_input /path/to/input/file \--my\_output /other/path \--my\_flag true When a pipeline requires some small data that rarely changes, it may be easier to include the data in the pipeline repository. You can reference this data from the pipeline script in a portable manner (i.e. without relying on an absolute path) by using the `projectDir` implicit variable, which refers to the local copy of the pipeline repository. The following example references the file `dataset/sequences.fa` in the pipeline repository: sequences \= file("$projectDir/dataset/sequences.fa") sequences.splitFasta { println it } ### Environment[](https://www.nextflow.io/docs/latest/sharing.html#environment "Permalink to this heading") The “environment” refers to any other aspects of the environment in which your pipeline is executed, such as environment variables and resource managers like SLURM. Any environment variable that may be required by the tools in your pipeline can be defined under the `env` scope in your Nextflow configuration. For example: env { DELTA \= 'hello' GAMMA \= 'world' } Similarly, if you use an HPC scheduler like SLURM or a cloud batch service like AWS Batch to execute tasks in a distributed manner, you can use a configuration profile to define the settings for a given environment. See [Configuration](https://www.nextflow.io/docs/latest/config.html#config-page) for more information about Nextflow configuration and [Executors](https://www.nextflow.io/docs/latest/executor.html#executor-page) for more information about executors. --- # Processes — Nextflow documentation * [](https://www.nextflow.io/docs/latest/index.html) * Processes * [Edit on GitHub](https://github.com/nextflow-io/nextflow/blob/master/docs/process.md) * * * Processes[](https://www.nextflow.io/docs/latest/process.html#processes "Permalink to this heading") ===================================================================================================== In Nextflow, a **process** is a specialized function for executing scripts in a scalable and portable manner. Here is an example process definition: process hello { output: path 'hello.txt' script: """ echo 'Hello world!' > hello.txt """ } See [Process](https://www.nextflow.io/docs/latest/reference/syntax.html#syntax-process) for a full description of the process syntax. Script[](https://www.nextflow.io/docs/latest/process.html#script "Permalink to this heading") ----------------------------------------------------------------------------------------------- The `script` section defines, as a string expression, the script that is executed by the process. A process may contain only one script, and if the `script` guard is not explicitly declared, the script must be the final statement in the process definition. The script string is executed as a [Bash](http://en.wikipedia.org/wiki/Bash_(Unix_shell)) script in the host environment. It can be any command or script that you would normally execute on the command line or in a Bash script. Naturally, the script may only use commands that are available in the host environment. The script section can be a simple string or a multi-line string. The latter approach makes it easier to write scripts with multiple commands spanning multiple lines. For example: process blast { """ blastp -db $db -query query.fa -outfmt 6 > blast\_result cat blast\_result | head -n 10 | cut -f 2 > top\_hits blastdbcmd -db $db -entry\_batch top\_hits > sequences """ } As explained in the script tutorial section, strings can be defined using single-quotes or double-quotes, and multi-line strings are defined by three single-quote or three double-quote characters. There is a subtle but important difference between them. Like in Bash, strings delimited by a `"` character support variable substitutions, while strings delimited by `'` do not. In the above code fragment, the `$db` variable is replaced by the actual value defined elsewhere in the pipeline script. Warning Since Nextflow uses the same Bash syntax for variable substitutions in strings, you must manage them carefully depending on whether you want to evaluate a _Nextflow_ variable or a _Bash_ variable. When you need to access a system environment variable in your script, you have two options. If you don’t need to access any Nextflow variables, you can define your script section with single-quotes: process echo\_path { ''' echo "The path is: $PATH" ''' } Otherwise, you can define your script with double-quotes and escape the system environment variables by prefixing them with a back-slash `\` character, as shown in the following example: process blast { """ blastp -db \\$DB -query query.fa -outfmt 6 > blast\_result cat blast\_result | head -n $MAX | cut -f 2 > top\_hits blastdbcmd -db \\$DB -entry\_batch top\_hits > sequences """ } In this example, `$MAX` is a Nextflow variable that must be defined elsewhere in the pipeline script. Nextflow replaces it with the actual value before executing the script. Meanwhile, `$DB` is a Bash variable that must exist in the execution environment, and Bash will replace it with the actual value during execution. ### Scripts _à la carte_[](https://www.nextflow.io/docs/latest/process.html#scripts-a-la-carte "Permalink to this heading") The process script is interpreted by Nextflow as a Bash script by default, but you are not limited to Bash. You can use your favourite scripting language (Perl, Python, R, etc), or even mix them in the same pipeline. A pipeline may be composed of processes that execute very different tasks. With Nextflow, you can choose the scripting language that best fits the task performed by a given process. For example, for some processes R might be more useful than Perl, whereas for others you may need to use Python because it provides better access to a library or an API, etc. To use a language other than Bash, simply start your process script with the corresponding [shebang](http://en.wikipedia.org/wiki/Shebang_(Unix)) . For example: process perl\_task { """ #!/usr/bin/perl print 'Hi there!' . '\\n'; """ } process python\_task { """ #!/usr/bin/python x = 'Hello' y = 'world!' print "%s - %s" % (x,y) """ } workflow { perl\_task() python\_task() } Tip Since the actual location of the interpreter binary file can differ across platforms, it is wise to use the `env` command followed by the interpreter name, e.g. `#!/usr/bin/env perl`, instead of the absolute path, in order to make your script more portable. ### Conditional scripts[](https://www.nextflow.io/docs/latest/process.html#conditional-scripts "Permalink to this heading") The `script` section is like a function that returns a string. This means that you can write arbitrary code to determine the script, as long as the final statement is a string. If-else statements based on task inputs can be used to produce a different script. For example: mode \= 'tcoffee' process align { input: path sequences script: if( mode \== 'tcoffee' ) """ t\_coffee -in $sequences > out\_file """ else if( mode \== 'mafft' ) """ mafft --anysymbol --parttree --quiet $sequences > out\_file """ else if( mode \== 'clustalo' ) """ clustalo -i $sequences -o out\_file """ else error "Invalid alignment mode: ${mode}" } In the above example, the process will execute one of several scripts depending on the value of the `mode` parameter. By default it will execute the `tcoffee` command. ### Template[](https://www.nextflow.io/docs/latest/process.html#template "Permalink to this heading") Process scripts can be externalized to **template** files, which allows them to be reused across different processes and tested independently from the pipeline execution. A template can be used in place of an embedded script using the `template` function in the script section: process hello { input: val STR script: template 'hello.sh' } workflow { channel.of('this', 'that') | hello } By default, Nextflow looks for the template script in the `templates` directory located alongside the Nextflow script in which the process is defined. An absolute path can be used to specify a different location. However, this practice is discouraged because it hinders pipeline portability. An example template script is provided below: #!/bin/bash echo "process started at \`date\`" echo $STR echo "process completed" Variables prefixed with the dollar character (`$`) are interpreted as Nextflow variables when the template script is executed by Nextflow and Bash variables when executed directly. For example, the above script can be executed from the command line by providing each input as an environment variable: STR\='Hello!' bash templates/my\_script.sh The following caveats should be considered: * Template scripts are recommended only for Bash scripts. Languages that do not prefix variables with `$` (e.g. Python and R) can’t be executed directly as a template script. * Variables escaped with `\$` will be interpreted as Bash variables when executed by Nextflow, but will not be interpreted as variables when executed from the command line. This practice should be avoided to ensure that the template script behaves consistently. * Template variables are evaluated even if they are commented out in the template script. If a template variable is missing, it will cause the pipeline to fail regardless of where it occurs in the template. Tip Template scripts are generally discouraged due to the caveats described above. The best practice for using a custom script is to embed it in the process definition at first and move it to a separate file with its own command line interface once the code matures. ### Shell[](https://www.nextflow.io/docs/latest/process.html#shell "Permalink to this heading") Deprecated since version 24.11.0-edge: Use the `script` section instead. Consider using the [strict syntax](https://www.nextflow.io/docs/latest/strict-syntax.html#strict-syntax-page) , which provides error checking to help distinguish between Nextflow variables and Bash variables in the process script. The `shell` section is a string expression that defines the script that is executed by the process. It is an alternative to the [Script](https://www.nextflow.io/docs/latest/process.html#process-script) definition with one important difference: it uses the exclamation mark `!` character, instead of the usual dollar `$` character, to denote Nextflow variables. This way, it is possible to use both Nextflow and Bash variables in the same script without having to escape the latter, which makes process scripts easier to read and maintain. For example: process hello { input: val str shell: ''' echo "User $USER says !{str}" ''' } workflow { channel.of('Hello', 'Hola', 'Bonjour') | hello } In the above example, `$USER` is treated as a Bash variable, while `!{str}` is treated as a Nextflow variable. Note * Shell script definitions require the use of single-quote `'` delimited strings. When using double-quote `"` delimited strings, dollar variables are interpreted as Nextflow variables as usual. See [String interpolation](https://www.nextflow.io/docs/latest/script.html#string-interpolation) . * Variables prefixed with `!` must always be enclosed in curly brackets, i.e. `!{str}` is a valid variable whereas `!str` is ignored. * Shell scripts support the use of the [Template](https://www.nextflow.io/docs/latest/process.html#process-template) mechanism. The same rules are applied to the variables defined in the template script. ### Native execution[](https://www.nextflow.io/docs/latest/process.html#native-execution "Permalink to this heading") The `exec` section executes the given code without launching a job. For example: process hello { input: val name exec: println "Hello Mr. $name" } workflow { channel.of('a', 'b', 'c') | hello } will display: Hello Mr. b Hello Mr. a Hello Mr. c A native process is very similar to a [function](https://www.nextflow.io/docs/latest/reference/syntax.html#syntax-function) . However, it provides additional capabilities such as parallelism, caching, and progress logging. Stub[](https://www.nextflow.io/docs/latest/process.html#stub "Permalink to this heading") ------------------------------------------------------------------------------------------- New in version 20.11.0-edge. You can define a command _stub_, which replaces the actual process command when the `-stub-run` or `-stub` command-line option is enabled: process salmon\_index { input: path transcriptome output: path 'index' script: """ salmon index --threads $task.cpus -t $transcriptome -i index """ stub: """ mkdir index touch index/seq.bin touch index/info.json touch index/refseq.bin """ } The `stub` section can be defined before or after the `script` section. When the pipeline is executed with the `-stub-run` option and a process’s `stub` is not defined, the `script` section is executed. This feature makes it easier to quickly prototype the workflow logic without using the real commands. The developer can use it to provide a dummy script that mimics the execution of the real one in a quicker manner. In other words, it is a way to perform a dry-run. Inputs[](https://www.nextflow.io/docs/latest/process.html#inputs "Permalink to this heading") ----------------------------------------------------------------------------------------------- The `input` section defines the input of a process, similar to function arguments. A process may have at most one input section, which must contain at least one input declaration. The input section follows the syntax shown below: input: An input declaration consists of a _qualifier_ and a _name_. The input qualifier defines the type of data to be received. This information is used by Nextflow to apply the semantic rules associated with each qualifier, and handle it properly depending on the target execution platform (grid, cloud, etc). When a process is invoked in a workflow, it must be provided a channel or dataflow value for each input in the process input section, similar to calling a function with specific arguments. The examples provided in the following sections demonstrate how a process is invoked. The following input qualifiers are available: * `val`: Access the input value by name in the process script. * `path`: Handle the input value as a path, staging the file properly in the execution context. * `env`: Use the input value to set an environment variable in the process script. * `stdin`: Forward the input value to the process `stdin` special file. * `tuple`: Handle a group of input values having any of the above qualifiers. * `each`: Execute the process for each element in the input collection. See [process reference](https://www.nextflow.io/docs/latest/reference/process.html#process-reference-inputs) for the full list of input methods and options. ### Input variables (`val`)[](https://www.nextflow.io/docs/latest/process.html#input-variables-val "Permalink to this heading") The `val` qualifier accepts any data type. It can be accessed in the process script by using the specified input name, as shown in the following example: process echo { input: val x script: """ echo "process job $x" """ } workflow { def num \= channel.of(1,2,3) echo(num) } In the above example, the process is executed three times: once for each value emitted by the `num` channel. The resulting output is similar to the one shown below: process job 3 process job 1 process job 2 Note Processes do not necessarily process items in the order that they are received. In the above example, the value `3` was processed before the others. Note When the process declares exactly one input, the pipe `|` operator can be used to provide inputs to the process, instead of passing it as a parameter. Both methods have identical semantics: process echo { input: val x script: """ echo "process job $x" """ } workflow { channel.of(1,2,3) | echo } ### Input files (`path`)[](https://www.nextflow.io/docs/latest/process.html#input-files-path "Permalink to this heading") The `path` qualifier allows you to provide input files to the process execution context. Nextflow will stage the files into the process execution directory, and they can be accessed in the script by using the specified input name. For example: process blast { input: path query\_file script: """ blastp -query ${query\_file} -db nr """ } workflow { def proteins \= channel.fromPath( '/some/path/\*.fa' ) blast(proteins) } In the above example, all the files ending with the suffix `.fa` are sent over the channel `proteins`. These files are received by the process, which executes a BLAST query on each of them. It’s worth noting that in the above example, the name of the file in the file-system is not used. You can access the file without even knowing its name, because you can reference it in the process script by the input name. There may be cases where your task needs to use a file whose name is fixed, i.e. it does not have to change along with the actual provided file. In this case, you can specify a fixed name with the `name` attribute in the input file parameter definition, as shown in the following example: input: path query\_file, name: 'query.fa' or, using a shorter syntax: input: path 'query.fa' The previous example can be re-written as shown below: process blast { input: path 'query.fa' script: """ blastp -query query.fa -db nr """ } workflow { def proteins \= channel.fromPath( '/some/path/\*.fa' ) blast(proteins) } In this example, each file received by the process is staged with the name `query.fa` in a different execution context (i.e. the folder where a task is executed). Tip This feature allows you to execute the process command multiple times without worrying about the file names changing. In other words, Nextflow helps you write pipeline tasks that are self-contained and decoupled from the execution environment. As a best practice, you should avoid referencing files in your process script other than those defined in your input section. Channel factories like `channel.fromPath` produce file objects, but a `path` input can also accept a string literal path. The string value should be an absolute path, i.e. it must be prefixed with a `/` character or a supported URI protocol (`file://`, `http://`, `s3://`, etc), and it cannot contain special characters (`\n`, etc). process cat { input: path x script: """ cat $x """ } workflow { cat('/some/data/file.txt') } Note Process `path` inputs have nearly the same interface as described in [Path](https://www.nextflow.io/docs/latest/reference/stdlib-types.html#stdlib-types-path) , with one difference which is relevant when files are staged into a subdirectory. Given the following input: path x, name: 'my-dir/file.txt' In this case, `x.name` returns the file name with the parent directory (e.g. `my-dir/file.txt`), whereas normally it would return the file name (e.g. `file.txt`). You can use `x.fileName.name` to get the file name. ### Multiple input files[](https://www.nextflow.io/docs/latest/process.html#multiple-input-files "Permalink to this heading") A `path` input can also accept a collection of files instead of a single value. In this case, the input variable will be a list. When the input has a fixed file name and a collection of files is received by the process, the file name will be appended with a numerical suffix representing its ordinal position in the list. For example: process blast { input: path 'seq' script: """ echo seq\* """ } workflow { def fasta \= channel.fromPath( "/some/path/\*.fa" ).buffer(size: 3) blast(fasta) } will output: seq1 seq2 seq3 seq1 seq2 seq3 ... The target input file name may contain the `*` and `?` wildcards, which can be used to control the name of staged files. The following table shows how the wildcards are replaced depending on the cardinality of the received input collection. | Arity | Name pattern | Staged file names | | --- | --- | --- | | any | `*` | named as the source file | | one | `file*.ext` | `file.ext` | | one | `file?.ext` | `file1.ext` | | one | `file??.ext` | `file01.ext` | | many | `file*.ext` | `file1.ext`, `file2.ext`, `file3.ext`, .. | | many | `file?.ext` | `file1.ext`, `file2.ext`, `file3.ext`, .. | | many | `file??.ext` | `file01.ext`, `file02.ext`, `file03.ext`, .. | | many | `dir/*` | named as the source file, created in `dir` subdirectory | | many | `dir??/*` | named as the source file, created in a progressively indexed subdirectory e.g. `dir01/`, `dir02/`, etc. | | many | `dir*/*` | (as above) | The following example shows how a wildcard can be used in the input file definition: process blast { input: path 'seq?.fa' script: """ cat seq1.fa seq2.fa seq3.fa """ } workflow { def fasta \= channel.fromPath( "/some/path/\*.fa" ).buffer(size: 3) blast(fasta) } Note Rewriting input file names according to a named pattern is an extra feature and not at all required. The normal file input syntax introduced in the [Input files (path)](https://www.nextflow.io/docs/latest/process.html#process-input-path) section is valid for collections of multiple files as well. To handle multiple input files while preserving the original file names, use a variable identifier or the `*` wildcard. New in version 23.09.0-edge. The `arity` option can be used to enforce the expected number of files, either as a number or a range. For example: input: path('one.txt', arity: '1') // exactly one file is expected path('pair\_\*.txt', arity: '2') // exactly two files are expected path('many\_\*.txt', arity: '1..\*') // one or more files are expected When a task is executed, Nextflow will check whether the received files for each path input match the declared arity, and fail if they do not. When the arity is `'1'`, the corresponding input variable will be a single file; otherwise, it will be a list of files. ### Dynamic input file names[](https://www.nextflow.io/docs/latest/process.html#dynamic-input-file-names "Permalink to this heading") When the input file name is specified by using the `name` option or a string literal, you can also use other input values as variables in the file name string. For example: process grep { input: val x path "${x}.fa" script: """ cat ${x}.fa | grep '>' """ } In the above example, the input file name is determined by the current value of the `x` input value. This approach allows input files to be staged in the task directory with a name that is coherent with the current execution context. Tip In most cases, you won’t need to use dynamic file names, because each task is executed in its own directory, and input files are automatically staged into this directory by Nextflow. This behavior guarantees that input files with the same name won’t overwrite each other. The above example is useful specifically when there are potential file name conflicts within a single task. ### Input environment variables (`env`)[](https://www.nextflow.io/docs/latest/process.html#input-environment-variables-env "Permalink to this heading") The `env` qualifier allows you to define an environment variable in the process execution context based on the input value. For example: process echo\_env { input: env 'HELLO' script: ''' echo "$HELLO world!" ''' } workflow { channel.of('hello', 'hola', 'bonjour', 'ciao') | echo\_env } hello world! ciao world! bonjour world! hola world! ### Standard input (`stdin`)[](https://www.nextflow.io/docs/latest/process.html#standard-input-stdin "Permalink to this heading") The `stdin` qualifier allows you to forward the input value to the [standard input](http://en.wikipedia.org/wiki/Standard_streams#Standard_input_.28stdin.29) of the process script. For example: process cat { input: stdin script: """ cat - """ } workflow { channel.of('hello', 'hola', 'bonjour', 'ciao') | map { v \-> v + '\\n' } | cat } will output: hola bonjour ciao hello ### Input tuples (`tuple`)[](https://www.nextflow.io/docs/latest/process.html#input-tuples-tuple "Permalink to this heading") The `tuple` qualifier groups multiple values into a single input definition. Each element in the tuple is associated with a corresponding element in the `tuple` definition. For example: process cat { input: tuple val(id), path('input.txt') script: """ echo "Processing $id" cat input.txt > copy """ } workflow { channel.of( \[1, 'alpha.txt'\], \[2, 'beta.txt'\], \[3, 'delta.txt'\] ) | cat } In the above example, the `tuple` input consists of the value `x` and the file `input.txt`. A `tuple` definition may contain any of the following qualifiers, as previously described: `val`, `env`, `path` and `stdin`. Files specified with the `path` qualifier are treated exactly the same as standalone `path` inputs. ### Input repeaters (`each`)[](https://www.nextflow.io/docs/latest/process.html#input-repeaters-each "Permalink to this heading") The `each` qualifier allows you to repeat the execution of a process for each item in a collection, each time a new value is received. For example: process align { input: path seq each mode script: """ t\_coffee -in $seq -mode $mode > result """ } workflow { sequences \= channel.fromPath('\*.fa') methods \= \['regular', 'espresso', 'psicoffee'\] align(sequences, methods) } In the above example, each time a file of sequences is emitted from the `sequences` channel, the process executes _three_ tasks, each running a T-coffee alignment with a different value for the `mode` parameter. This behavior is useful when you need to repeat the same task over a given set of parameters. Input repeaters can be applied to files as well. For example: process align { input: path seq each mode each path(lib) script: """ t\_coffee -in $seq -mode $mode -lib $lib > result """ } workflow { sequences \= channel.fromPath('\*.fa') methods \= \['regular', 'espresso'\] libraries \= \[ file('PQ001.lib'), file('PQ002.lib'), file('PQ003.lib') \] align(sequences, methods, libraries) } In the above example, each sequence input file emitted by the `sequences` channel triggers six alignment tasks, three with the `regular` method against each library file, and three with the `espresso` method. Note When multiple repeaters are defined, the process is executed for each _combination_ of them. Note Input repeaters do not support tuples. Use the [combine](https://www.nextflow.io/docs/latest/reference/operator.html#operator-combine) or [cross](https://www.nextflow.io/docs/latest/reference/operator.html#operator-cross) operator to combine the repeated input with the other inputs to produce all of the desired input combinations. ### Multiple inputs[](https://www.nextflow.io/docs/latest/process.html#multiple-inputs "Permalink to this heading") A process can declare multiple inputs, which allows it to accept inputs from multiple dataflow sources. Warning Do not supply more than one channel when calling a process with multiple inputs. Invoking a process with multiple channels can lead to [non-deterministic behavior](https://www.nextflow.io/docs/latest/cache-and-resume.html#cache-nondeterministic-inputs) . All additional inputs should be dataflow values. When a process is defined with multiple inputs, it waits for a value from each input and launches a new task with the combined values. When one of the inputs is a channel, the process repeats until all values in the channel are consumed. If the channel is empty, the process will not launch any tasks. For example: process echo { input: val x val y script: """ echo $x and $y """ } workflow { x \= channel.value(1) y \= channel.of('a', 'b', 'c') echo(x, y) } The above example executes the `echo` process three times. The dataflow value `x` is reused for each value in `y`. It outputs: 1 and a 1 and b 1 and c See also: [Singleton outputs](https://www.nextflow.io/docs/latest/process.html#process-out-singleton) . Outputs[](https://www.nextflow.io/docs/latest/process.html#outputs "Permalink to this heading") ------------------------------------------------------------------------------------------------- The `output` section defines the outputs of a process, similar to a function return. A process may have at most one output section, which must contain at least one output declaration. The output section follows the syntax shown below: output: \[, :