Automatic Package Synchronization#
By default Coiled inspects your local Python environment for Python packages and emulates that environment on the remote cluster.
import coiled cluster = coiled.Cluster( n_workers=20, # Coiled automatically syncs packages by default ) client = cluster.get_client()
Automatic package synchronization works with many kinds of locally installed packages:
Pip installed packages from PyPI
Locally installed editable packages
.pyfiles importable in your Python environment
Automatic package synchronization does not always work, especially with messy environments. In these cases we recommend one of the following approaches:
Create a fresh conda or virtualenv environment and run from there:
conda create -n myenv -c conda-forge coiled dask ipykernel
Automatic package synchronization works reliably well in these cases.
Use a Docker image
Exclude errant packages using the
package_sync_ignore=keyword (likely you will be prompted in an error message if this applies to you.