Coiled Docs

👋 Welcome to Coiled’s documentation!

What is Coiled?

Coiled provides cluster-as-a-service functionality to provision hosted Dask clusters on demand. It takes the DevOps out of data science and enables data engineers and data scientists to spend more time on their real job and less time setting up networking, managing fleets of Docker images, creating IAM roles, etc.

Hosted Dask Clusters

Securely deploy Dask clusters from anywhere you run Python.

Software Environments

Build, manage, and share conda, pip, and Docker environments. Use them locally or in the cloud.

Manage Teams & Costs

Manage teams, collaborate, set resource limits, and track costs.

Where does Coiled run?

Coiled runs in our managed cloud environment so you can get started without any of your own infrastructure. You can also run Coiled in your preferred cloud provider or in your own AWS, GCP, or Azure account.

Use Coiled with AWS

Use Coiled with Amazon Web Services (AWS)

Use Coiled with GCP

Use Coiled with Google Cloud Platform (GCP)

Use Coiled with Azure

Use Coiled with Microsoft Azure

How Coiled works

Coiled handles the creation and management of Dask clusters on cloud computing environments. This process involves your user environment, Coiled Cloud, and a cloud computing environment. Coiled can used with our managed cloud environment or your own AWS, GCP, or Azure cloud account.

Coiled Architecture

Coiled Architecture (click image to enlarge)

  • User Environment

    This is where you use the Coiled Python package along with your preferred tools to create Dask clusters and submit Dask computations. This could be a Jupyter Notebook on your laptop, a Python script on a cloud-hosted VM, or Python code within a task in a workflow management system.

  • Coiled Cloud

    Coiled Cloud provides a cluster dashboard that you can use to manage clusters, users, teams, software environments, etc. Coiled Cloud also handles provisioning the necessary cloud infrastructure for your Dask clusters so you don’t have to!

  • Cloud Computing Environment

    This is the cloud environment where Dask clusters will be created and where your Dask computations will run. By default, your Dask cluster will be created on Coiled’s managed cloud environment. You can also configure Coiled to run on different backends and cloud providers, including AWS, GCP, or Azure.

Coiled at a glance

Coiled manages Dask clusters and everything you need to scale Python in the cloud robustly and easily. Learn more in the Dask clusters docs.

# Launch a cluster with Coiled
import coiled

cluster = coiled.Cluster(
    n_workers=5,
    worker_cpu=4,
    worker_memory="16 GiB",
)

# Connect Dask to your cluster
from dask.distributed import Client

client = Client(cluster)

Coiled helps you manage software environments by building Docker images from pip and conda environment files for you. Learn more in the software environment docs.

# Create a custom "ml-env" software environment
# with the packages you want
import coiled

coiled.create_software_environment(
    name="ml-env",
    conda=["dask", "scikit-learn", "xgboost"],
)

# Create a Dask cluster which uses your "ml-env"
# software environment
cluster = coiled.Cluster(software="ml-env")

Try Coiled

Ready to get started with Coiled? Sign in and create a free account!

Register / sign-in to Coiled

Thank you for trying out Coiled!