Test definitions

Adding tests to your subworkflow is a critical step to ensure it works correctly and continues to do so as the codebase evolves. The nf-core command automatically generates the test infrastructure when creating a subworkflow. Inside the tests directory of your subworkflow, you’ll find main.nf.test, which contains inputs, test cases, and assertion instructions for nf-test, the Nextflow test framework. Open it and follow through the rest of this section.

Note

Comparatively to modules tests, subworkflows are meant to be tested functionnally. All subworkflows tests are run as stubs, meaning no processes are executed and output data is faked. This provides quick feedback, while completely covering the dataflows implemented by subworkflows.

Native subworkflows

Native nextflow subworkflows don’t include any modules in their definition. Those will need to be unit tested in addition to stub testing.

Enable stubbing

Each module comes with it’s stub. To activate stubbing for the subworkflows, add the following tag and options to the nextflow_workflow block :

nextflow_workflow {
    name "Test Workflow <SUBWORKFLOW_NAME>"
    script "../main.nf"
    ...

    tag "stub"
    options "-stub-run"
    ...
}

Note

If some tests need to be carried without stubbing, move the tag and options definitions inside each of the test cases that require stubbing.

Setup before tests

Local setup for a test case

The setup section can either be used globally, before the test cases definitions, or inside a specific test case, where they will override the global setup.

Setup sections are used to prepare data before test cases are run, to download data, prepare algorithms and generate awaited return values. In nf-neuro, test datasets are provided via the LOAD_TEST_DATA subworkflow (refer to this section to learn how to find packages). Once you’ve selected the archives and files needed, add a setup section before the test section. Replace the <archive> with the name of the one you need and you’ll be able to access its content within your test suite :

setup {
    run("LOAD_TEST_DATA", alias: "LOAD_DATA") {
        script "../../../../subworkflows/nf-neuro/load_test_data/main.nf"
        workflow {
            """
            input[0] = Channel.from( [ "<archive>" ] )
            input[1] = "test.load-test-data"
            """
        }
    }
}

What is happening here?

1. The run instruction is used to call the LOAD_TEST_DATA subworkflow, under the alias LOAD_DATA. The alias could be useful if multiple calls to the same subworkflow are needed. The script parameter inside the run block specifies the path to the LOAD_TEST_DATA subworkflow file.
2. The workflow block contains definitions for the input channels of the subworkflow. The input[0] channel corresponds to the archives to download for the test, while input[1] is a test dataset identifier for the current run.

Note

When using the LOAD_TEST_DATA subworkflow, or any other component in the setup section, make sure to tag it in the nextflow_workflow block accordingly:

tag "subworkflows"
tag "subworkflows/load_test_data"

Define test cases

Test cases for subworkflows are defined each in their own test block:

test("example - test") {
    when{
        workflow {
            ...
        }
    }
    then{
        ...
    }
}

A block minimally contains a when (what to test) and a then (what to assert). Inside of them, no need to define the name of the subworkflow being tested, or import it. nf-test will take care of that for you when the test will be run, through the workflow placeholder inside the when block.

Define test inputs

The only job of the when block is to define the inputs to supply to your subworkflow. You do so by adding a list of inputs inside the workflow block, as follows:

test("example - test") {
    when {
        workflow {
            """
            input[0] = LOAD_DATA.out.test_data_directory.map{
                test_data_directory -> [
                    [ id:'test1', single_end:false ], // meta map   // -> meta
                    file("\${test_data_directory}/image.nii.gz"),   // -> image file
                    []                                              // -> an optional input, not provided
                ],
                [
                    [ id:'test2', single_end:false ], // meta map   // -> meta
                    file("\${test_data_directory}/image.nii.gz"),   // -> image file
                    file("\${test_data_directory}/mask.nii.gz")     // -> an optional input, provided this time
                ]
            }
            """
        }
    }
    then {
        ...
    }
}

What is happening here?

1. The first input channel of the subworkflow is created from the output of the LOAD_TEST_DATA subworkflow, called in the setup section, and accessed through its alias LOAD_DATA.
2. A first dataset to input into the subworkflow is created from the content downloaded by the subworkflow. The subworkflow has an optional path in input, which is not provided using the empty list [].
3. A second dataset is created, this time with a mask as input.

In this example, for this single test case, the subworkflow would run separately on two inputs, 2 and 3.

Define assertions

The assertions for a subworkflow have to finely inspect the structure and shape of the output data of the subworkflow, without accessing the data itself. The only file required to be snapshotted is the versions.yml file that keeps track of dependencies :

then {
    assertAll(
        { assert workflow.success },
        { assert snapshot(
            workflow.out
                .findAll{ !it.key.isInteger() && it.value  }
                .collect{ item -> ["versions"].contains(item.key)
                    ? item.value
                    : file(item.value.get(0).get(1)).name } ) },
        { assert workflow.out
            .findAll{ !it.key.isInteger() }
            .every{ channel ->
                channel.value.every{ item ->
                    item instanceof ArrayList ? item.get(1) : item } } }
    )
}

What is happening here?

1. The assertAll function is used to group multiple assertions together, ensuring that all of them are checked.
2. The first assertion checks that the workflow ran successfully, using workflow.success.
3. The second assertion snapshots the versions.yml file content and checks file naming, ensuring their consistency with maintenance practices. Other files are usually empty in stub mode, so only their names are checked.
4. The third assertion checks that all output channels area populated with results, ensuring that the subworkflow produced the expected outputs.

If the content of the channels outputed by the subworkflow can vary with configuration, then output channels have to be further verified to validate either the absence or presence of data in them. Given a list of channels names [ch_A, ch_B, ...] that should not produce outputs, use the following assertion :

then {
    assertAll(
        ...,
        { assert workflow.out
            .findAll{ !it.key.isInteger() }
            .every{ channel -> ["ch_A",
                                "ch_A",
                                ...].contains(channel.key)
                ? channel.value.size() == 0
                : channel.value.every{ item ->
                    item instanceof ArrayList ? item.get(1) : item } } }
    )
}

What is happening here?

1. Channels names in the list should be empty and their size be 0.
2. Channels not in the list should be populated with data. They either contain single items or lists. The every operator iterates on all subjects in the channel and either emits the item itself, or the second element of the list (the item placed after the meta map).

Configure the subworkflow

To configure parameters of the subworkflow, use the config parameter, either globally or at the scope of the test case, to import a nextflow.config file.

Layered configuration

nf-test supports layering configuration files. It applies configuration in hierarchy, from parent config statements defined in the nextflow_process block down to overriding config in children test blocks.

Generate tests snapshots

Caution

Verify you are located at the root of nf-neuro (not inside subworkflows) before running commands!

Once you have correctly setup your test cases and made sure the data is available, the test subworkflow has to be pre-tested so output files that gets generated are snapshotted correctly before being pushed to nf-neuro.

To do so, run:

nf-core subworkflows test -u <subworkflow_name>

All the test cases you defined will be run, watch out for errors! Once everything runs smoothly, look at the snapshot file produced at tests/main.nf.test.snap in your subworkflow’s directory and validate that ALL outputs produced by test cases are caught. Their md5sum is critical to ensure future executions of your test produce valid outputs.

A complete example for a subworkflow

Here’s an example of a complete test for a hypothetical subworkflow that preprocesses imaging data:

nextflow_workflow {

    name "Test Workflow PREPROCESSING"
    script "../main.nf"
    workflow "PREPROCESSING"
    config "./nextflow.config"

    tag "subworkflows"
    tag "subworkflows/preprocessing"
    tag "subworkflows/load_test_data"

    tag "stub"
    options "-stub-run"

    setup {
        run("LOAD_TEST_DATA", alias: "LOAD_DATA") {
            script "../../../../subworkflows/nf-neuro/load_test_data/main.nf"
            workflow {
                """
                input[0] = Channel.from( [ "raw_b0.zip", "raw_segmentation.zip" ] )
                input[1] = "test.load-test-data"
                """
            }
        }
    }

    test("preprocessing - standard") {
        when {
            workflow {
                """
                ch_split_test_data = LOAD_DATA.out.test_data_directory
                    .branch{
                        b0: it.simpleName == "raw_b0"
                        segmentation: it.simpleName == "raw_segmentation"
                    }
                ch_images = ch_split_test_data.b0.map{
                    test_data_directory -> [
                        [ id:'test', subject:'01' ],
                        file("\${test_data_directory}/b0.nii.gz")
                    ]
                }
                ch_masks = ch_split_test_data.segmentation.map{
                    test_data_directory -> [
                        [ id:'test', subject:'01' ],
                        file("\${test_data_directory}/brainmask/slices/axial.nii.gz")
                    ]
                }
                input[0] = ch_images
                input[1] = ch_masks
                """
            }
        }
        then {
            assertAll(
                { assert workflow.success },
                { assert snapshot(
                    workflow.out
                        .findAll{ !it.key.isInteger() && it.value }
                        .collect{ item -> ["versions"].contains(item.key)
                            ? item.value
                            : file(item.value.get(0).get(1)).name } ) },
                { assert workflow.out
                    .findAll{ !it.key.isInteger() }
                    .every{ channel ->
                        channel.value.every{ item ->
                            item instanceof ArrayList ? item.get(1) : item } } }
            )
        }
    }
}