Using Coiled with JupyterLab#

Coiled integrates with the tools you already use. In this guide, you’ll learn about a few useful open source JupterLab extensions we recommend:

You can download this jupyter notebook to follow along in your own JupyterLab session.

Before you start#

It’s important to have the same libraries installed both in your local Python environment (e.g. on your laptop), as well as on the remote Dask workers in your cluster. If you are unfamiliar with creating software environments, you can first follow the tutorial on setting up a custom software environment.

Coiled maintains a default remote environment with the coiled-runtime metapackage. Run the snippet below from the command line to create this same environment locally:

conda create -n jupyter-example -c conda-forge python=3.9 coiled-runtime

Conda kernels extension#

The nb_conda_kernels extension enables you to access other conda environments on your machine from Jupyter Notebook or JupyterLab. This allows you to smoothly switch between different software environments you’ve installed locally.

Note

Any other environments you want to access must have a kernel package, e.g. ipykernel, installed in them. By default, any software environment you’ve installed with coiled-runtime will have ipykernel automatically installed to enable use with nb_conda_kernels.

conda kernels extension

To select the conda environment to use with a notebook in JupyterLab, click the text indicating the current kernel in the upper righthand corner of the notebook (screenshot above). This will bring up a “Select Kernel” dropdown menu where you can select which kernel you would like to use for the notebook. You can select the jupyter-example environment you created in the previous example.

Coiled cluster widget#

When you create a cluster from JupyterLab or using Jupyter Notebooks, you’ll notice a widget detailing the cluster state overview and progress bars as resources are provisioned. This widget relies on the ipywidgets extension.

import coiled

cluster = coiled.Cluster(
    name="jupyter-example", software="coiled/default-py39", n_workers=5
)

widget

Dask JupyterLab extension#

The Dask community maintains a JupyterLab extension which allows Dask dashboard plots to be embedded directly into a JupyterLab session. Viewing diagnostic plots in JupyterLab instead of in a separate browser tab or window is often a better user experience.

Note

The Dask JupyterLab extension is automatically installed with coiled-runtime. If you are using dask-labextension versions 2.x, see the installation section of the Dask JupyterLab Extension README file.

Open the Dask JupyterLab extension by clicking the Dask logo in the JupyterLab left sidebar. Then select the magnifying glass icon in the upper right-hand corner and copy and paste the address from the Dashboard Address in the cluster widget to connect the extension to your cluster.

jupyterlab extension

Each orange button corresponds to a different diagnostic plot. Try clicking one of the buttons and then arranging the plots. See this screencast for a live demo.

Once you’ve arranged the plots, you can see the Dashboard at work while running a Dask computation:

import dask

# generate random timeseries of data
df = dask.datasets.timeseries("2000", "2005", partition_freq="2w").persist()

# perform a groupby with an aggregation
df.groupby("name").aggregate({"x": "sum", "y": "max"}).compute()

See the Dask documentation for more details on intrepreting the Dask dashboard. Once you’ve completed your work, you can shutdown your cluster:

# close the cluster
cluster.close()
# close the connection to the client
client.close()