Creating clusters

Spinning up Dask clusters with Coiled is done by creating a coiled.Cluster instance. coiled.Cluster objects manage a Dask cluster much like other cluster object you may have seen before like distributed.LocalCluster or dask_kubernetes.KubeCluster.

Simple example

In a simple case, you can create a cluster with five Dask workers with:

import coiled

cluster = coiled.Cluster(n_workers=5)


Creating a cluster involves provisioning various resources on cloud-based infrastructure. This process takes about a minute in most cases.

Once a cluster has been created, you can connect Dask to the cluster by creating a distributed.Client instance:

from dask.distributed import Client

client = Client(cluster)

To view the Dask diagnostic dashboard for your cluster, navigate to the cluster’s dashboard_link:


which should output an address along the lines of "".


Any Coiled cluster you create will automatically shut down after 20 minutes of inactivity. No more large bills from leaving a cluster running over the weekend 🎉! You can also customize this idle timeout if needed – see the Custom workers and scheduler section for an example that does this.

The coiled.Cluster class has several keyword arguments you can use to further specify the details of your cluster. These parameters are discussed in the following sections.

Hardware resources

The hardware resources your cluster is launched on (e.g. number of CPUs, amount of RAM, etc.) can be configured with the following coiled.Cluster keyword arguments:





Number of CPUs allocated for each worker



Number of GPUs allocated for each worker. Note that GPU access is disabled by default. If you would like access to GPUs, please contact



Amount of memory to allocate for each worker

"16 GiB"


Number of CPUs allocated for the scheduler



Amount of memory to allocate for the scheduler

"8 GiB"

For example, the following creates a cluster with five workers, each with 2 CPUs and 8 GiB of memory available, and a scheduler with 2 CPUs (default value) and 16 GiB of memory available:

import coiled

cluster = coiled.Cluster(
    worker_memory="8 GiB",
    scheduler_memory="16 GiB",

Note that while specifying worker_gpu will give your cluster workers access to GPUs, there are some additional best practices to ensure GPU-accelerated hardware is fully utilized. See the GPU best practices documentation for more information.

Software environment

The scheduler and each worker in a Coiled cluster are all launched with the same software environment. By default, they will use a software environment with Python, Dask, Distributed, NumPy, Pandas, and a few more commonly used libraries. This default environment is great for basic tasks, but you’ll also want to create your own custom software environments with the packages you need on your cluster.

Coiled supports building and managing custom software environments using pip and conda environment files. For more details on custom software environments, see the Software Environments documentation page. Once you have a custom software environment you can use the software keyword argument for coiled.Cluster to use that software environment on your cluster.


Software environments used in Coiled clusters must have distributed >= 2.23.0 installed as Distributed is required to launch Dask scheduler and worker processes.

For example, the following uses a custom software environment with XGBoost installed:

import coiled

cluster = coiled.Cluster(software="examples/scaling-xgboost")

Custom workers and scheduler

Dask supports using custom worker and scheduler classes in a cluster which allows for increased flexibility and functionality in some use cases (e.g. Dask-CUDA’s CUDAWorker class for running Dask workers on NVIDIA GPUs). Additionally, worker and scheduler classes also have keyword arguments that can be specified to control their behavior (for an example, see Dask’s worker class API documentation).

The worker and scheduler class used in a Coiled cluster, as well as the keyword arguments, passed to those classes can be specified with the following coiled.Cluster keyword arguments:





Class to use for cluster workers



Mapping with keyword arguments to pass to worker_class



Class to use for the cluster scheduler



Mapping with keyword arguments to pass to scheduler_class


For example, the following creates a cluster which uses Distributed’s Worker class for workers (instead of the default Nanny class) and specifies idle_timeout="2 hours" when creating the cluster’s scheduler:

import coiled

cluster = coiled.Cluster(
    scheduler_options={"idle_timeout": "2 hours"},

Backend options

Depending on where you’re running Coiled, there may be backend-specific options (e.g. which AWS region to use) that you can specify to customize Coiled’s behavior. For more information on what options are available, see the Backends documentation.

Cluster configurations

As seen in the previous sections, there are a variety of ways in which you can customize your clusters. In some cases it’s useful to save the various parameters that define a cluster (e.g. worker_memory, scheduler_options, etc.) so they can be easily reused or shared with colleagues.

Coiled supports this through the concept of a cluster configuration. A cluster configuration is a group of cluster parameters (e.g. worker_memory, scheduler_cpu, software, etc.) that can be saved and then reused by yourself or other Coiled users. When creating a cluster, you can use the configuration keyword argument for coiled.Cluster to specify a cluster configuration to use.





Name of cluster configuration to create cluster from


Cluster configurations are discussed in detail on the Cluster configurations documentation page.