Dask Clusters

Dask is a general purpose library for parallel computing. Dask can be used on its own to parallelize Python code, or with integrations to other popular libraries to scale out common workflows.

Dask has its own docs and examples, but we’ll include a few typical use cases below.

Local Execution

Simpler examples work well with a local Dask cluster if you just want to experiment.

from dask.distributed import LocalCluster

cluster = LocalCluster(processes=False)
client = cluster.get_client()

Cloud Execution

In many cases you’ll likely want a Coiled cluster set up.

from coiled import Cluster

cluster = Cluster(n_workers=15)
client = cluster.get_client()

Once you have a Dask cluster you can then run Python code on that cluster. Here is the simplest code you could run:

def inc(x):
    return x + 1

future = client.submit(inc, 10)
future.result() # returns 11

Examples

Here are examples using Dask Cluster:

• General Python parallelism

• Processing tabular data on S3

• Using Xarray for processing large datasets

• Easy Scalable Production ETL

• Cloud Parquet ETL with Dask DataFrame

• Visualize 1,000,000,000 points interactively

• XGBoost for Gradient Boosted Trees

• Hyperparameter optimization with XGBoost