Prefect on Coiled#
This is experimental and only works on Linux and/or an environment where a user is running from a path entirely contained in
/home/ for now.
You can use the
coiled prefect worker start CLI tool to start Prefect workers for use in running your deployed workflows
coiled prefect worker start --help
Have the following installed in your environment:
Sign up for an acount at prefect.io.
Authenticate with Prefect locally in your terminal
prefect cloud login -w
Let’s start by creating a simple Prefect flow in a file we’ll call
# file: myflow.py from prefect import flow, task from prefect.deployments import Deployment from coiled.extensions.prefect import CoiledTaskRunner @task def inc(x): return x + 1 @task def double(x): return x * 2 @task def add(x, y): return x + y @task def sum_all(z): return sum(z) @flow(log_prints=True, task_runner=CoiledTaskRunner()) def my_flow(): a = inc.map(range(10)) b = double.map(range(10)) c = add.map(x=a, y=b) return sum_all(c) # Make deployment from this flow if __name__ == "__main__": deployment = Deployment.build_from_flow( flow=my_flow, name="prefect-on-coiled", work_queue_name="default", work_pool_name="default" ) deployment.apply()
Start a Prefect Worker on Coiled#
With that in place, let’s go start a Prefect worker using the
coiled prefect worker start CLI:
coiled prefect worker start --prefect-workspace MY_WORKSPACE_NAME
It’s assumed the above is ran in the current directory where
myflow.py is located, and from the same environment
the flow’s dependencies are installed. This will leverage the syncing of this flow’s code along with
automatic package synchronization feature of Coiled.
Additionally, this will upload the entire working directory.
This functionality will not be enabled if you are using the
In Coiled we can see our worker is running:
And the work-queue we created is available in Prefect:
Make and Run a Prefect Deployment#
Now we make a
Deployment to Prefect Cloud by running our
which we can see in Prefect:
Things are looking good, let’s run the deployment using the
prefect deployment run CLI:
In Prefect we see the successful flow run:
coiled prefect worker start#
Deploy Prefect Worker(s) on a Coiled cluster.
This should be ran from the same code (and environment) used when deploying your Prefect flow/deployment. It will sync that directory’s code to the VM(s) where the worker(s) will be running. Running the command from the desired Python environment also ensures all packages are synced to the Worker(s)/VM(s).
coiled prefect worker start [OPTIONS]
- --prefect-workspace <prefect_workspace>#
Required Name of the Prefect workspace.
- --prefect-key <prefect_key>#
API key for Prefect workspace, used to login Prefect workers.
- --prefect-pool <prefect_pool>#
Name of pool for Prefect worker.
- --prefect-work-queue <prefect_work_queue>#
Name of Prefect work-queue for worker.
- --cluster-name <cluster_name>#
Name of Coiled cluster
- --n-workers <n_workers>#
Number of Prefect workers to launch
Opt out of defaulting to an adaptive cluster
- --adaptive-min <adaptive_min>#
Minimum workers in adaptive cluster configuration
- --adaptive-max <adaptive_max>#
Maximum workers in adaptive cluster configuration
- --container <container>#
Docker image uri to use, ex: coiled/default:sha-2021.06.0