Software Environments#

A crucial part of doing your work is making sure you have the software packages you need. Coiled helps you manage software environments by building Docker images from pip and conda environment files for you. You can then use these environments locally, remotely in a Dask cluster, and can share them with your friends and colleagues.

The video below will walk you through the process of creating and managing software environments.

Note

Coiled has a feature called package sync which scans your local Python environment and replicates it to the cluster, so you no longer need to go through the extra step of creating a Coiled software environment. See Package Synchronization for more details and how to try it out.

Supported software specifications#

Coiled supports publicly accessible conda packages, pip packages, and/or Docker images for creating software environments. You can also compose these steps by, for example, conda installing packages into a custom Docker image.

Design#

Coiled uses packaging conventions you’re already familiar with. You can point Coiled to a list of conda or pip packages:

import coiled

coiled.create_software_environment(
    name="my-software-env",
    conda=["dask", "xarray=0.15.1", "numba"],
)

or to a local conda environment.yml or pip requirements.txt file:

coiled.create_software_environment(
    name="my-software-env",
    conda="environment.yml",
)

to have custom Docker images built and stored for later use. Note that you do not need to have Docker installed for Coiled to build Docker images for you!

Usage#

Coiled software environments can be used both locally by installing the software environment on your machine and on remote Dask clusters (e.g. running on AWS):

import coiled

# Create a cluster that uses the custom "my-software-env" software environment
cluster = coiled.Cluster(software="my-software-env")

You can also collaborate with your friends and colleagues by easily sharing software environments.

Coiled-runtime#

Coiled maintains a number of software environments, found on the Coiled software environments page. For default coiled environments (e.g. coiled/default) you’ll notice coiled-runtime as one of the dependencies:

channels:
  - conda-forge
dependencies:
  - python=<x.x>
  - coiled-runtime=<x.x.x>

coiled-runtime is a conda metapackage that contains the recommended version of Dask and associated packages to make it easy to get started with Dask and Coiled (see the recipe file for the complete list of included packages and versions).

To learn more, follow the tutorial on using coiled-runtime to set up a custom software environment or watch the video below: