Configuration

Coiled uses Dask’s built-in configuration system. You can set a number of configuration settings, including:

1. Coiled account settings

2. Cluster hardware

3. Customization of Dask scheduler or workers

4. Dask configuration

Coiled account settings

The coiled login command line tool (see coiled login) automatically creates and populates the ~/.config/dask/coiled.yaml configuration file with the following:

coiled:
  account: "<your account>"
  server: https://cloud.coiled.io
  token: "<your token>"
  user: "<your username>"

(Other existing Dask config, if you have any, will not be modified by running coiled login.) You can change any of these values directly by editing your local ~/.config/dask/coiled.yaml yaml file.

Coiled configuration default yaml file

coiled:
  user: ""
  token: null
  server: https://cloud.coiled.io
  account: null
  backend-options: null
  no-minimum-version-check: false
  protocol: tls
  scheduler-options: {}
  worker-options: {}
  wait-for-workers: 0.3
  software: null
  worker:
    cpu: null
    gpu: null
    memory: null
    class: null
    vm-types: null
    gpu-types: null
  scheduler:
    cpu: null
    memory: null
    gpu: null
    class: null
    vm-types: null
  name: null
  shutdown-on-close: true
  private-to-creator: false
  use_aws_creds_endpoint: true
  cluster-state-check-interval: 1s

  analytics:
    disabled: false
    computation:
      interval: 15s
      code:
        transmit: true
    profile:
      transmit: false
      interval: 60s
    events:
      interval: 10s
      allow:
        - invalid-worker-transition
        - invalid-task-states
        - worker-fail-hard
    idle:
      timeout: null

distributed:
  diagnostics:
    computations:
      nframes: 2

For most login options, you can use the CLI tool to set these values, depending on the value you would like to change. To change your default account, for example, you can use coiled login --workspace.

Cluster infrastructure/hardware

There are a number of cluster infrastructure and hardware settings you can configure, including:

• Instance types (see Allowable Instance Types)

• Custom network security (see Bring Your Own Network and Configure Firewalls)

• Cloud provider-specific configuration, e.g. whether to use Spot instances or selecting a region (see Backend Options for AWS or Backend options for Google Cloud)

Customization of Dask scheduler and workers

There are a number of keyword arguments you can pass to the Dask Scheduler class and Worker class. A number of these arguments are handled by Coiled, however, a few common use cases for Coiled users include:

• Setting the idle timeout for the scheduler (see Idle Shutdown)

• Setting the number of threads per worker (see Single-Threaded Workers)

• Specify Dask resources to constrain how your tasks run on your workers

You can set these when making a Coiled cluster using scheduler_options and worker_options keyword arguments on coiled.Cluster (see Worker and Scheduler Options).

Dask configuration

Dask configuration controls a variety of options for customizing Dask’s behavior. For example, you can use this to control memory thresholds where Dask will spill to disk or to adjust task queuing. Other packages in the Dask ecosystem also make use of the Dask configuration system for exposing various options (e.g. Dask-jobqueue, Dask Cloud Provider).

Since Coiled uses Dask’s configuration system, you can set these values in any of the ways you usually would when using Dask locally:

• Configuration yaml files in ~/.config/dask/

• Environment variables

• Directly in Python code using dask.config

When you start a Coiled cluster by calling coiled.Cluster, we get all of the configuration values set in your local environment and ship these to your cluster.

Note

Changes made to local YAML files or environment variables will not affect already running clusters. To change a configuration value after your cluster is already running, the best way is to use dask.config.set.

In addition to Dask-specific configuration, you can also use any of the above methods to set Coiled-specific configuration values.

Equivalent ways of setting commonly used configuration values

YAML Key	Environment variable	dask.config.set	Description
account	DASK_COILED__ACCOUNT	dask.config.set({"coiled.account": <your-account-name>})	The Coiled account you want to use.
token	DASK_COILED__TOKEN	dask.config.set({"coiled.token": <your-token>})	The Coiled token for your personal account.
software	DASK_COILED__SOFTWARE	dask.config.set({"coiled.software": <your-senv-name>})	Name of the software environment to use.

previous

Coiled Client Release Notes

next

Data Privacy