Dask Clusters¶
Coiled manages Dask clusters. It manages cloud resources, networking, software environments, and everything you need to scale Python in the cloud robustly and easily.
Simple Example¶
The main entry point to launch a Coiled cluster is the coiled Python API. In
the simplest case you can run the following from anywhere that you can run
Python.
import coiled
# Spin up a Dask cluster using Coiled
cluster = coiled.Cluster()
And then you can connect to that cluster with Dask
from dask.distributed import Client
client = Client(cluster)
Configuration¶
Though in real-world use cases there are many parameters that we may want to control:
The amount of RAM and number of CPUs in each worker
The number of workers to use
Whether or not to use GPUs
The software used in each worker
…
These can also be specified when you create a Coiled cluster. For example:
import coiled
# Spin up a Dask cluster using Coiled
# where each worker has 4 CPUs and 16 GiB of RAM
cluster = coiled.Cluster(
worker_cpu=4,
worker_memory="16 GiB",
)
More details about creating Dask clusters with Coiled are discussed on the Creating clusters documentation page.
Learn more¶
In the next sections we’ll learn more about how to configure and launch Dask clusters.