Installing software environments locally¶
When performing distributed computation with Dask, you’ll create a
distributed.Client object to connect your local Python process (e.g.
your laptop) to your remote Dask cluster (e.g. running on AWS). Dask
s are the user-facing entry point for submitting tasks to a Dask cluster. When
Client to submit tasks to your cluster, Dask will package up and
send data, functions, and other Python objects needed for your computations
from your local Python process where your
Client is running to your
remote Dask cluster in order for them to be executed.
This means that if you want to run a function on your Dask cluster, for example
numpy.mean() function, then you must have NumPy installed in your
local Python process so Dask can send the
numpy.mean function from your
Client to the workers in your Dask cluster. For this and other
reasons, it’s recommended to have the same libraries installed on both your
local machine and on the remote workers in your cluster.
As such, Coiled software environments can be installed locally to have a consistent set of libraries between your local environment and the environment your cluster is running in.
Install Coiled software environments locally¶
You can install a Coiled software environment locally using the
coiled install command line tool.
coiled install installs an existing
Coiled software environment on your machine as a local conda environment. For
example, to install the
coiled/default software environment locally:
# Create local version of the coiled/default software environment $ coiled install coiled/default $ conda activate coiled-coiled-default
coiled/default name after
coiled install specifies which Coiled
software environment to install locally. Generally Coiled software environments
are specified in the form of
“<coiled-account-name>/<software-environment-name>”. So in the above example
coiled install to create the Coiled software environment named
“default” in the “coiled” account locally.
Note that currently
coiled install requires conda to be installed locally
and does not support software environments with custom Docker images.
For windows users
If you try something like
coiled install coiled/default
from a Windows command prompt it will simply return a list of
dependencies for the selected environment. It is
straightforward to use that dependency list to create a
environment.yml file, and with that use
conda env create -f
environment.yml to create the desired software environment.
See our tutorial for how to do this.