Scale

Manual scaling

After you’ve created a cluster with Coiled, you can scale it up or down using coiled.Cluster.scale(). Since there is a delay between requesting to scale a cluster and when workers are added or removed, you can pair this with Client.wait_for_workers.

In the following example, you can start a cluster with 10 workers, scale it up to 15 workers, and then submit the blocking call to wait to continue until 15 workers have been allocated.

import coiled

cluster = coiled.Cluster(n_workers=10)
client = cluster.get_client()

cluster.scale(15)
client.wait_for_workers(15)

You can also monitor your cluster as workers are added or removed on the cluster infrastructure page.

Adaptive scaling

You can use coiled.Cluster.adapt() to let the scheduler decide how many workers it should create (or destroy) depending on the workload. With adaptive scaling, you can specify a range of requested workers and the Dask scheduler will handle scaling the cluster up or down for you (see Adaptive Deployments in the Dask docs).

In the following example, you can create a cluster and set the minimum number of workers.

import coiled

cluster = coiled.Cluster()
client = cluster.get_client()

cluster.adapt(minimum=1, maximum=100)

Shutting down

Coiled turns off clusters after 20 minutes of inactivity (configurable Idle Shutdown) but you can manually shut down a cluster with the following command:

cluster.shutdown()