Automatic Package Synchronization#
By default Coiled inspects your local Python environment for Python packages and emulates that environment on the remote cluster.
import coiled
cluster = coiled.Cluster(
n_workers=20, # Coiled automatically syncs packages by default
)
client = cluster.get_client()
Automatic package synchronization works with many kinds of locally installed packages:
Conda packages
Pip installed packages from PyPI
Locally installed editable packages
.py
files importable in your Python environment
Debugging Issues#
Automatic package synchronization does not always work, especially with messy environments. In these cases we recommend one of the following approaches:
Create a fresh conda or virtualenv environment and run from there:
conda create -n myenv -c conda-forge coiled dask ipykernel
Automatic package synchronization works reliably well in these cases.
Use a Docker image
Exclude errant packages using the
package_sync_ignore=
keyword (likely you will be prompted in an error message if this applies to you.