Dask Clusters#
Coiled manages Dask clusters. It manages cloud resources, networking, software environments, and everything you need to scale Python in the cloud robustly and easily.
Simple Example#
The main entry point to launch a Coiled cluster is the coiled Python API. In
the simplest case you can run the following from anywhere that you can run
Python.
import coiled
# Spin up a Dask cluster using Coiled
cluster = coiled.Cluster()
And then you can connect to that cluster with Dask
from dask.distributed import Client
client = Client(cluster)
Configuration#
Though in real-world use cases there are many parameters that we may want to control:
The amount of RAM and number of CPUs in each worker
The number of workers to use
Whether or not to use GPUs
The software used in each worker
…
These can also be specified when you create a Coiled cluster. For example:
import coiled
# Spin up a Dask cluster using Coiled
# where each worker has 4 CPUs and 16 GiB of RAM
cluster = coiled.Cluster(
worker_cpu=4,
worker_memory="16 GiB",
)
More details about creating Dask clusters with Coiled are discussed on the Creating clusters documentation page.
Learn more#
In the next sections we’ll learn more about how to configure and launch Dask clusters.