Configuration#

Coiled uses Dask’s built-in configuration system. You can set a number of configuration settings, including:

  1. Coiled account settings

  2. Cluster hardware

  3. Customization of Dask scheduler or workers

  4. Dask configuration

Coiled account settings#

The coiled login command line tool (see coiled login) automatically creates and populates the ~/.config/dask/coiled.yaml configuration file with the following:

coiled:
  account: "<your account>"
  server: https://cloud.coiled.io
  token: "<your token>"
  user: "<your username>"

(Other existing Dask config, if you have any, will not be modified by running coiled login.) You can change any of these values directly by editing your local ~/.config/dask/coiled.yaml yaml file.

Coiled configuration default yaml file
coiled:
  user: ""
  token: null
  server: https://cloud.coiled.io
  account: null
  backend-options: null
  no-minimum-version-check: false
  protocol: tls
  scheduler-options: {}
  worker-options: {}
  wait-for-workers: 0.3
  software: null
  worker:
    cpu: null
    gpu: null
    memory: null
    class: null
    vm-types: null
    gpu-types: null
  scheduler:
    cpu: null
    memory: null
    gpu: null
    class: null
    vm-types: null
  name: null
  shutdown-on-close: true
  private-to-creator: false
  use_aws_creds_endpoint: true
  cluster-state-check-interval: 1s

  analytics:
    disabled: false
    computation:
      interval: 15s
      code:
        transmit: true
    profile:
      transmit: false
      interval: 60s
    events:
      interval: 10s
      allow:
        - invalid-worker-transition
        - invalid-task-states
        - worker-fail-hard
    idle:
      timeout: null

distributed:
  diagnostics:
    computations:
      nframes: 2

For most login options, you can use the CLI tool to set these values, depending on the value you would like to change. To change your default account, for example, you can use coiled login --account.

Cluster infrastructure/hardware#

There are a number of cluster infrastructure and hardware settings you can configure, including:

Customization of Dask scheduler and workers#

There are a number of keyword arguments you can pass to the Dask Scheduler class and Worker class. A number of these arguments are handled by Coiled, however, a few common use cases for Coiled users include:

  • Setting the idle timeout for the scheduler (see Idle Shutdown)

  • Setting the number of threads per worker (see Single-Threaded Workers)

  • Specify Dask resources to constrain how your tasks run on your workers

You can set these when making a Coiled cluster using scheduler_options and worker_options keyword arguments on coiled.Cluster (see Worker and Scheduler Options).

Dask configuration#

Dask configuration controls a variety of options for customizing Dask’s behavior. For example, you can use this to control memory thresholds where Dask will spill to disk or to adjust task queuing. Other packages in the Dask ecosystem also make use of the Dask configuration system for exposing various options (e.g. Dask-jobqueue, Dask Cloud Provider).

Since Coiled uses Dask’s configuration system, you can set these values in any of the ways you usually would when using Dask locally:

  • Configuration yaml files in ~/.config/dask/

  • Environment variables

  • Directly in Python code using dask.config

When you start a Coiled cluster by calling coiled.Cluster, we get all of the configuration values set in your local environment and ship these to your cluster.

Note

Changes made to local YAML files or environment variables will not affect already running clusters. To change a configuration value after your cluster is already running, the best way is to use dask.config.set.

In addition to Dask-specific configuration, you can also use any of the above methods to set Coiled-specific configuration values.

Equivalent ways of setting commonly used configuration values#

YAML Key

Environment variable

dask.config.set

Description

account

DASK_COILED__ACCOUNT

dask.config.set({"coiled.account": <your-account-name>})

The Coiled account you want to use.

token

DASK_COILED__TOKEN

dask.config.set({"coiled.token": <your-token>})

The Coiled token for your personal account.

software

DASK_COILED__SOFTWARE

dask.config.set({"coiled.software": <your-senv-name>})

Name of the software environment to use.