Automatic Package Synchronization#

By default Coiled inspects your local Python environment for Python packages and emulates that environment on the remote cluster.

import coiled

cluster = coiled.Cluster(
   n_workers=20,  # Coiled automatically syncs packages by default
)
client = cluster.get_client()

Automatic package synchronization works with many kinds of locally installed packages:

  • Conda packages

  • Pip installed packages from PyPI

  • Locally installed editable packages

  • .py files importable in your Python environment

Debugging Issues#

Automatic package synchronization does not always work, especially with messy environments. In these cases we recommend one of the following approaches:

  1. Create a fresh conda or virtualenv environment and run from there:

    conda create -n myenv -c conda-forge coiled dask ipykernel

    Automatic package synchronization works reliably well in these cases.

  2. Use a Manual Software Environments

  3. Use a Docker image

  4. Exclude errant packages using the package_sync_ignore= keyword (likely you will be prompted in an error message if this applies to you.

previous

Manage Software

next

Manual Software Environments