How to get local code on your cluster#
The easiest way to upload your local Python code to your Coiled cluster is to package it and let Package Synchronization handle building the wheel and installing it on your cluster.
All importable Python packages installed in your environment are synced (e.g. pip install -e
, python setup.py install
, python setup.py develop
or similar). However, importable Python modules in your local directory are not synced (e.g. folders with __init__.py
or various Python files).
This guide will go through:
How to turn your local Python code into an installable package
Confirming your package can be installed in your cluster
Turning Python code into a package#
In this section, you’ll learn how to package a simple Python project. This guide was adapted from this PyPA tutorial and omits steps required to upload your package to PyPI, which is not required for using package sync.
Example project#
This guide uses an example project named my_package
. Create the following file structure locally:
my_project/
└── src/
└── my_package/
├── __init__.py
└── my_module.py
Open my_module.py
and enter the following:
def inc(x):
return x + 1
The src
directory should contain all modules and packages meant for distribution (see the setuptools documentation on src-layout).
__init__.py
is required to import the directory as a package, and should be empty.
my_module.py
is an example of a module in the package that could contain classes, functions, etc.
For more background on Python modules and importing packages (ie, why you need __init__.py
), see the Python documentation on packages and modules.
Creating pyproject.toml#
Note
Projects utilizing the deprecated setup.py
style for packaging are still supported by package sync
Create the pyproject.toml
file in the top level of your my_project
directory. Your file structure should now look like this:
my_project/
├── pyproject.toml
└── src/
└── my_package/
├── __init__.py
└── my_module.py
Open pyproject.toml
and enter the following [build-system]
and [project]
tables:
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "my_package"
version = "0.0.1"
The [build-system]
table tells build tools, like pip, which backend tool to use when creating your project. You can choose from a number of backends; this guide uses setuptools, but you can also also use others like Hatchling, Flit, or PDM.
requires
is a list of packages needed to build your package. You don’t need to install these, the build frontend you’re using, like pip, will automatically do so in a temporary, isolated environment when the package is built.build-backend
is the name of the Python object that the build frontend will use to perform the build.
The [project]
table includes metadata about your project. name
and version
are required, though there are a number of keywords you can include (see the Python documentation on declaring project metadata).
name
is the distribution name of your package. This can be any name as long as it only contains letters, numbers, ., _ , and -. It’s convention for the directory containing your Python files to match the project name.version
is the package version. See PyPA version specifiers.
Tip
Why weren’t my Python files included in my package?
A common problem is that you’ve created a package, but your Python files are not included, often manifesting as an ImportError
.
To fix this problem, you can use setuptools
-specific configuration options like py-modules
and packages
(see the setuptools documentation). Alternatively, you can use the find
directive, which is particularly helpful for projects that don’t follow a src- or flat-layout:
[tool.setuptools.packages.find]
where = ["src"] # list of folders that contain the packages (["."] by default)
include = ["my_package*"] # package names should match these glob patterns (["*"] by default)
exclude = ["my_package.tests*"] # exclude packages matching these glob patterns (empty by default)
namespaces = false # to disable scanning PEP 420 namespaces (true by default)
Installing your package#
Now that you’ve packaged your Python code, you can install my_package
into your Python environment. You can do this in a new virtual environment, installing your package from a local src
(see the PyPA documentation ).
Note
In this guide, you’re installing my_package
locally from src
. It is also common to install packages from a version control system, like Git, which is also supported by package sync (see the pip documentation on VCS support).
You can do this either in Development Mode
python3 -m pip install -e my_project
py -m pip install -e my_project
Or normally from src
python3 -m pip install my_project
py -m pip install my_project
You’ll also want to have Coiled installed:
pip install coiled 'dask[complete]'
Validation#
Import your package locally#
Now that you’ve installed my_package
, you can start a Python session and confirm you are able to import your package locally:
from my_package.my_module import inc
If there are modules missing from your package, you may get an import error. A good first step is to use pip wheel my_package
, extract the resulting .whl
file, and inspect the contents (using unzip my_package.whl
, for example). You may need to be more explicit about which files to include in your package in your pyproject.toml
file, see our tip above.
Import your package in the cloud#
Before you start your cluster, you can use the coiled package-sync scan
CLI tool to check if it will be included in your environment:
coiled package-sync scan
You should see an output similar to the following. You can see my_package
listed at the top. If it were not built correctly, then in the last column you would see “Can build wheel” set to false or the package would not be listed at all.

When you create a Coiled cluster, you can see the output from package sync as my_package
is built:
cluster = coiled.Cluster(n_workers=10)
client = cluster.get_client()

Now, you can import and call inc
in your cluster:
import dask
from my_package.my_module import inc
dask.delayed(inc(10)).compute()
Next steps#
For more details on packaging a Python project, we recommend you follow the full PyPA tutorial.