Quick Reference

Creating Dask clusters

Create a Dask cluster with 10 workers (Learn more)

import coiled
cluster = coiled.Cluster(n_workers=10)

Create a Dask cluster with a custom software environment (Learn more)

import coiled
cluster = coiled.Cluster(software="my-pip-env")

Create a Dask cluster with 1 GPU per worker (Learn more)

import coiled
cluster = coiled.Cluster(worker_gpu=1)

Create a Dask cluster in a Team account (Learn more)

import coiled
cluster = coiled.Cluster(account="my-team-account-name")

Working with Dask clusters

Connect to a cluster (Learn more)

from dask.distributed import Client
client = Client(cluster)
print('Dashboard:', client.dashboard_link)

Once connected, run a Dask computation as usual (Learn more)

import dask.dataframe as dd
df = dd.read_csv(...).persist()
df.groupby(...).tip_amount.mean().compute()

Scale the number of workers (Learn more)

cluster.scale(15)

Reuse an existing cluster (Learn more)

cluster = coiled.Cluster(name="existing-cluster-name")

Generate a performance report (Learn more)

from coiled import performance_report

with performance_report(filename="dask-report.html"):
    df.groupby(...).value.mean().compute()  ## Your dask computation(s)

Terminate a cluster (Learn more)

client.close()  # if shutdown_on_close=True

or

coiled.delete_cluster(name="my-cluster")

Packages and environments

Create a software environment from a list of conda packages (Learn more)

coiled.create_software_environment(
    name="my-conda-env",
    conda={
        "channels": ["conda-forge", "defaults"],
        "dependencies": ["dask", "xarray=0.15.1", "numba"],
    },
)

Create a software environment from an environment.yml file (Learn more)

coiled.create_software_environment(
    name="my-conda-env",
    conda="environment.yml",
)

Create a software environment from a list of pip packages (Learn more)

coiled.create_software_environment(
    name="my-pip-env",
    pip=["dask[complete]", "xarray==0.15.1", "numba"],
)

Create a software environment from a requirements.txt file (Learn more)

coiled.create_software_environment(
    name="my-pip-env",
    pip="requirements.txt",
 )

Create a software environment from an existing Docker image (Learn more)

coiled.create_software_environment(
    name="my-docker-env",
    container="rapidsai/rapidsai:latest",
)