Package Sync Limitations¶
System dependencies aren’t synced¶
Package sync doesn’t handle non-Python, system-level packages you’ve installed with a system package manager like apt or homebrew. It’s rare these days, but some Python packages cannot simply be pip-installed and need extra system dependencies to function. In these cases, you can:
(preferred) install system dependencies with conda instead. It’s likely you can find everything you need on conda-forge. Package sync will then mirror all these dependencies automatically, including the non-Python ones.
(advanced) not use package sync, and use Docker instead. You’ll have to build and publish your own Docker image, register it as a software environment, and specify that when creating the cluster.
Dependencies don’t update on running clusters¶
If you install a new package, or edit a local dependency, after launching a cluster, the cluster doesn’t update automatically. You have to re-create the cluster to pick up any changes. If you have a use case where this is strictly necessary, you can use Dask’s PipInstall Worker Plugin.
Unsolvable environments¶
Your environment needs to be consistent with what your package manager would accept as a valid environment. For example, this will fail:
dask==2022.1.10
distributed==2022.05.1
Pip will error out trying to install this environment, as will conda, because distributed==2022.05.1 requires exactly dask==2022.05.1. If you manage to get your local environment into a state where both those incompatible packages are installed, package sync will not be able to synchronize it.
Local packages must be buildable as wheels¶
Your local package must either not have compiled dependencies, or you must be running on the same platform as the cluster (64bit linux)—we do not try to cross compile your package!
Packages that have special build requirements¶
If you have packages installed with pip that don’t have pre-built wheels available, and have requirements beyond what is included
in the standard build-essentials and python-dev ubuntu packages, you’ll find package sync fails. If your package
is available in a conda repo, we suggest using that instead.
Packages that do not list their Python build time requirements¶
This is a slight variant of compiled packages missing build time system dependencies. If a package uses the deprecated
setup.py and imports something that is not listed as a PEP 517 build requirement
or setup_requires, the build will also fail.
An example of this is crick 0.0.3, which imports numpy and Cython in setup.py
but does not list them as build dependencies.