nf-core/configs: AWS Batch with Tower Configuration

To be used when submitting jobs to AWS Batch by using Tower Forge. If you are not using Tower Forge, consider using the profile awsbatch where you can directly specify the Batch queue, AWS region and AWS cli path.

This profile defines awsbatch as executor, and allows overwrite of trace, timeline, report and dag to allow resuming pipelines.

Config file

See config file on GitHub

conf/aws_tower
//Nextflow config file for running on AWS batch
params {
    config_profile_description = 'AWS Batch with Tower Profile'
    config_profile_contact     = 'Gisela Gabernet (@ggabernet)'
    config_profile_url         = 'https://aws.amazon.com/batch/'
}
 
timeline {
    overwrite = true
}
report {
    overwrite = true
}
trace {
    overwrite = true
}
dag {
    overwrite = true
}
 
process.executor = 'awsbatch'

Pipeline configs

See config file on GitHub

conf/pipeline/demultiplex/aws_tower
// Profile config names for nf-core/configs
 
params {
    // Specific nf-core/configs params
    config_profile_contact     = 'Edmund Miller(@edmundmiller)'
    config_profile_description = 'nf-core/demultiplex AWS Tower profile provided by nf-core/configs'
}
 
aws {
    batch {
        maxParallelTransfers = 24
        maxTransferAttempts  = 3
    }
    client {
        maxConnections   = 24
        uploadMaxThreads = 24
        maxErrorRetry    = 3
        socketTimeout    = 3600000
        uploadRetrySleep = 1000
        uploadChunkSize  = 32.MB
    }
}
 
process {
    withName: BASES2FASTQ {
        cpus   = 16
        memory = 48.GB
    }
}
On this page