Coiled manages Dask clusters. It manages cloud resources, networking, software environments, and everything you need to scale Python in the cloud robustly and easily.
The main entry point to launch a Coiled cluster is the
coiled Python API. In
the simplest case you can run the following from anywhere that you can run
import coiled # Spin up a Dask cluster using Coiled cluster = coiled.Cluster()
And then you can connect to that cluster with Dask
from dask.distributed import Client client = Client(cluster)
Though in real-world use cases there are many parameters that we may want to control:
The amount of RAM and number of CPUs in each worker
The number of workers to use
Whether or not to use GPUs
The software used in each worker
These can also be specified when you create a Coiled cluster. For example:
import coiled # Spin up a Dask cluster using Coiled # where each worker has 4 CPUs and 16 GiB of RAM cluster = coiled.Cluster( worker_cpu=4, worker_memory="16 GiB", )
More details about creating Dask clusters with Coiled are discussed on the Creating clusters documentation page.
In the next sections we’ll learn more about how to configure and launch Dask clusters.