Package Sync Limitations#

System dependencies aren’t synced#

Package sync doesn’t handle non-Python, system-level packages you’ve installed with a system package manager like apt or homebrew. It’s rare these days, but some Python packages cannot simply be pip-installed and need extra system dependencies to function. In these cases, you can:

(preferred) install system dependencies with conda instead. It’s likely you can find everything you need on conda-forge. Package sync will then mirror all these dependencies automatically, including the non-Python ones.
(advanced) not use package sync, and use Docker instead. You’ll have to build and publish your own Docker image, register it as a software environment, and specify that when creating the cluster.

Dependencies don’t update on running clusters#

If you install a new package, or edit a local dependency, after launching a cluster, the cluster doesn’t update automatically. You have to re-create the cluster to pick up any changes. If you have a use case where this is strictly necessary, you can use Dask’s PipInstall Worker Plugin.

Unsolvable environments#

Your environment needs to be consistent with what your package manager would accept as a valid environment. For example, this will fail:

dask==2022.1.10
distributed==2022.05.1

Pip will error out trying to install this environment, as will conda, because distributed==2022.05.1 requires exactly dask==2022.05.1. If you manage to get your local environment into a state where both those incompatible packages are installed, package sync will not be able to synchronize it.

Packages that do not list their Python build time requirements#

This is a slight variant of compiled packages missing build time system dependencies. If a package uses the deprecated setup.py and imports something that is not listed as a PEP 517 build requirement or setup_requires, the build will also fail.

An example of this is crick 0.0.3, which imports numpy and Cython in setup.py but does not list them as build dependencies.