Quick Reference#

Creating Dask clusters#

Create a Dask cluster with 6 workers (Learn more)

 
import coiled
cluster = coiled.Cluster(n_workers=6)

Create a Dask cluster with a custom software environment (Learn more)

 
import coiled
cluster = coiled.Cluster(software="my-pip-env")

Create a Dask cluster with 1 GPU per worker (Learn more)

 
import coiled
cluster = coiled.Cluster(worker_gpu=1)

Create a Dask cluster in a Team account (Learn more)

 
import coiled
cluster = coiled.Cluster(account="my-team-account-name")

Working with Dask clusters#

Connect to a cluster (Learn more)

 
from dask.distributed import Client
client = Client(cluster)
print('Dashboard:', client.dashboard_link)

Once connected, run a Dask computation as usual (Learn more)

 
import dask.dataframe as dd
df = dd.read_csv(...).persist()
df.groupby(...).tip_amount.mean().compute()

Scale the number of workers (Learn more)

 
cluster.scale(15)

Reuse an existing cluster (Learn more)

 
cluster = coiled.Cluster(name="existing-cluster-name")

Generate a performance report (Learn more)

 
from coiled import performance_report

with performance_report(filename="dask-report.html"):
    df.groupby(...).value.mean().compute()  ## Your dask computation(s)

Terminate a cluster (Learn more)

 
cluster.close()  # if shutdown_on_close=True

Packages and environments#

Create a software environment from a list of conda packages

 
coiled.create_software_environment(
    name="my-conda-env",
    conda={
        "channels": ["conda-forge", "defaults"],
        "dependencies": ["dask", "xarray=0.15.1", "numba"],
    },
)

Create a software environment from an environment.yml file

 
coiled.create_software_environment(
    name="my-conda-env",
    conda="environment.yml",
)

Create a software environment from a list of pip packages

 
coiled.create_software_environment(
    name="my-pip-env",
    pip=["dask[complete]", "xarray==0.15.1", "numba"],
)

Create a software environment from a requirements.txt file

 
coiled.create_software_environment(
    name="my-pip-env",
    pip="requirements.txt",
 )

Create a software environment from an existing Docker image

 
coiled.create_software_environment(
    name="my-docker-env",
    container="rapidsai/rapidsai-core:23.02-cuda11.5-runtime-ubuntu20.04-py3.8",
)

previous

TLS expects a ssl_context error

next

API Reference