Package Sync Best Practices#

Coiled runs your code on remote cloud VMs from anywhere. An important part of sending code between machines (for example, from your laptop to a VM on AWS) is making sure the software environments match on both machines.

By default, Coiled will automatically replicate your local software environment on cloud VMs using Coiled’s automatic package synchronization feature (no Docker!). This works by scanning all the packages installed in your current Python environment

  • Pip installed packages from PyPI

  • Conda packages

  • Locally installed editable packages

  • .py files in your working directory

and installing them on your cloud machines.

 

For most users, this works well automatically and they don’t really need to think about software management. However, there are some situations were package sync will run into issues. This page details some suggested best practices when using package sync.

Use a fresh environment#

Many software environments evolve organically and can become inconsistent over time. This makes it difficult to replicate some long-lived environments.

In these cases, creating a new, fresh environment, then installing any packages you need, almost always resolves package consistency issues.

You can use venv, conda, pipenv, uv, or any other tool you prefer for managing your local software environment. Package sync will pick up whatever libraries you have installed independent of what tool you’re using to manage local environments.

Use conda for non-Python libraries#

For pure Python packages installed with pip or conda, package sync usually works great. However, for more complex libraries that involve installing additional system packages like gdal, graphviz, etc., automatic package synchronization can fail if those additional system packages weren’t installed with pip or conda (e.g. apt or brew were used instead).

In these cases, we recommend using conda to install more complex libraries.

Using gdal as an example, replace this:

brew install gdal  # Install system gdal
pip install gdal   # Install Python gdal

with this:

conda install -c conda-forge gdal  # Install system gdal and Python bindings

Don’t mix conda channels#

When using conda to install software, it’s possible to install packages from multiple conda channels (for example, conda-forge and defaults). While this is sometimes okay, it can lead to hard to diagnose errors like

ImportError: dlopen(.../site-packages/rpy2/rinterface/_rinterface.so, 2): Library not loaded: @rpath/libicuuc.54.dylib
Referenced from: .../site-packages/rpy2/rinterface/_rinterface.so
Reason: image not found

In these cases, we recommend using a single conda channel when possible to avoid these types of errors. We recommend using the conda-forge channel if all your dependencies are available.

Private software#

Package sync handles both publicly available and private software. Public software works without any additional setup. See this section for how to use private (or other custom) PyPI indexes and private conda channels.

On this page