Dask Clusters

Coiled manages Dask clusters. It manages cloud resources, networking, software environments, and everything you need to scale Python in the cloud robustly and easily.

Simple Example

The main entry point to launch a Coiled cluster is the coiled Python API. In the simplest case you can run the following from anywhere that you can run Python.

import coiled

# Spin up a Dask cluster using Coiled
cluster = coiled.Cluster()

And then you can connect to that cluster with Dask

from dask.distributed import Client

client = Client(cluster)

Configuration

Though in real-world use cases there are many parameters that we may want to control:

  • The amount of RAM and number of CPUs in each worker

  • The number of workers to use

  • Whether or not to use GPUs

  • The software used in each worker

  • …

These can also be specified when you create a Coiled cluster. For example:

import coiled

# Spin up a Dask cluster using Coiled
# where each worker has 4 CPUs and 16 GiB of RAM
cluster = coiled.Cluster(
    worker_cpu=4,
    worker_memory="16 GiB",
)

More details about creating Dask clusters with Coiled are discussed on the Creating clusters documentation page.

Learn more

In the next sections we’ll learn more about how to configure and launch Dask clusters.

Getting Started Creating clusters