v2#

Coiled v2 is our rewritten backend. As of version 0.2.0, the Coiled client package (coiled) uses v2.

It was written with the following objectives:

  • Reliability: clusters come up with greater certainty, especially at larger scale.

  • Cost: we provide a richer and more responsive access to sets of instance types and spot instances.

  • Visibility: when situations occur, issues are more plainly visible and easier to debug.

v2 provides both a richer and more robust interaction with the underlying cloud, and delivers this information to you so that you can understand what is going on.

Terminal dashboard displaying the Coiled cluster status overview, configuration, and Dask worker states.

Here’s how to try v2 now:

Reasons to Stay on v1#

For now, v2 is missing a few features:

  • GPU support is not implemented.

  • Spot instance support is not implemented for GCP.

Also, for operating in your own AWS/GCP accounts, there are some new IAM permissions we need.

If those features essential to you or you haven’t yet updated the IAM permissions, you may want to stay on v1.

How to Stay on v1#

We will continue to support the original Coiled system for some months to allow you to adapt your workflows and make any required changes. We want to ensure a smooth experience and don’t expect many breaking changes.

If you want to stay with the original system, pin your coiled library to <0.2 in your Python environment:

coiled < 0.2

This will keep you on a 0.0.x client version, which will not switch to v2. We are not planning any new 0.0.x client releases except for critical bug fixes.

How to Upgrade, Step 1: Updated IAM Permissions#

If you’re not running Coiled in your own AWS/GCP account, then there’s nothing you need to do for the upgrade to v2.

If you are running in your own account, the IAM permissions we need in v2 are slightly different, so refer to the updated policy documents can here: GCP / AWS.

How to Upgrade, Step 2: Code Changes#

Most users shouldn’t need any code changes, but if you’re using options like worker_memory, worker_cpu, scheduler_memory, scheduler_cpu then there’s a chance you may need small changes to accommodate v2’s stricter (more correct) logic for selecting instance types.

You might have this in v1:

from coiled import Cluster

cluster = Cluster(worker_memory="32GiB", worker_cpu=1)

In v2, this gives an error because we couldn’t find an instance type with that much memory and only one core.

Instead, you can leave out the worker_cpu argument, or give us a range of acceptable core counts:

from coiled.v2 import Cluster

cluster = Cluster(worker_memory="32GiB", worker_cpu=[1, 8])

Deprecations#

Cluster configurations have been deprecated so the configuration argument is no longer allowed. Instead, configuration is now directly passed to the Cluster class at creation time.

The protocol parameter (which was used for proxying through Coiled to the scheduler) is not planned for v2.