👋 Welcome to Coiled’s documentation!
What is Coiled?¶
Coiled provides cluster-as-a-service functionality to provision hosted Dask clusters on demand. It takes the DevOps out of data science and enables data engineers and data scientists to spend more time on their real job and less time setting up networking, managing fleets of Docker images, creating AWS IAM roles, etc.
Hosted Dask Clusters
Securely deploy Dask clusters from anywhere you run Python.
Build, manage, and share conda, pip, and Docker environments. Use them locally or in the cloud.
Manage Teams & Costs
Manage teams, collaborate, set resource limits, and track costs.
How Coiled works¶
Coiled handles the creation and management of Dask clusters on cloud computing environments. This process involves your user environment, Coiled Cloud, and a cloud computing environment.
This is where you use the Coiled Python package along with your preferred tools to create Dask clusters and submit Dask computations. This could be a Jupyter Notebook on your laptop, a Python script on a cloud-hosted VM, or Python code within a task in a workflow management system.
Coiled Cloud provides a cluster dashboard that you can use to manage clusters, users, teams, software environments, etc. Coiled Cloud also handles provisioning the necessary cloud infrastructure for your Dask clusters so you don’t have to!
Cloud Computing Environment
This is the cloud environment where Dask clusters will be created and where your Dask computations will run. By default, your Dask cluster will be created on Coiled’s AWS account. However, you can run Coiled on different backends, including AWS, Azure, or GCP.
Coiled at a glance¶
Coiled manages Dask clusters and everything you need to scale Python in the cloud robustly and easily. Learn more in the Dask clusters docs.
# Launch a cluster with Coiled import coiled cluster = coiled.Cluster( n_workers=5, worker_cpu=4, worker_memory="16 GiB", ) # Connect Dask to your cluster from dask.distributed import Client client = Client(cluster)
# Create a custom "ml-env" software environment # with the packages you want import coiled coiled.create_software_environment( name="ml-env", conda=["dask", "scikit-learn", "xgboost"], ) # Create a Dask cluster which uses your "ml-env" # software environment cluster = coiled.Cluster(software="ml-env")