After you’ve created a cluster with Coiled, you can scale it up or down using
coiled.Cluster.scale(). Since there is a delay between requesting to scale a cluster and when workers are added or removed, you can pair this with
In the following example, you can start a cluster with 10 workers, scale it up to 15 workers, and then submit the blocking call to wait to continue until 15 workers have been allocated.
import coiled cluster = coiled.Cluster(n_workers=10) client = cluster.get_client() cluster.scale(15) client.wait_for_workers(15)
You can also monitor your cluster as workers are added or removed on the cluster infrastructure page.
You can use
coiled.Cluster.adapt() to let the scheduler decide how many workers it should create (or destroy) depending on the workload. With adaptive scaling, you can specify a range of requested workers and the Dask scheduler will handle scaling the cluster up or down for you (see Adaptive Deployments in the Dask docs).
In the following example, you can create a cluster and set the minimum number of workers.
import coiled cluster = coiled.Cluster() client = cluster.get_client() cluster.adapt(minimum=1, maximum=100)
Coiled turns off clusters after 20 minutes of inactivity (configurable Idle Shutdown) but you can manually shut down a cluster with the following command: