AWS Backend

Using Coiled’s AWS account

By default your computations run inside Coiled’s AWS account. This makes it easy for you to get started quickly, without needing to set up any additional infrastructure.

../_images/backend-coiled-aws-vm.png

Using your own AWS account

You can have Coiled run computations in your own AWS account. This allows you to take advantage of any account or security configurations (such as startup credits or custom data access controls) that you already have in place. When configuring AWS as a backend in the Account page of your Coiled account, you can choose to run Coiled using an ECS-based backend or a VM-based backend.

../_images/backend-external-aws-vm.png

To do this, create an access key ID and secret access key and add them to the “Cloud Backend Options” section of the Account page of your Coiled account.

From now on, clusters you create with Coiled will be launched in your AWS account.

Note

The AWS credentials you supply must be long-lived (not temporary) tokens, and have sufficient permissions to allow Coiled to set up management infrastructure and create & launch compute resources from within your AWS account.

Also, note that if you have not used AWS Elastic Container Service in this account before, you may need to create the necessary service-linked IAM role – we cannot yet create it automatically.

The below JSON template lists all the above permissions and can be used to create the required IAM policy directly.

See also

You might be interested in reading the knowledge base tutorial on How to limit Coiled’s access to your AWS resources.

Backend options

There are several AWS-specific options you can specify (listed below) to customize Coiled’s behavior. Additionally, the next section contains an example of how to configure these options in practice.

Name

Description

Default

region

AWS region to create resources in

us-east-2

fargate_spot

Whether or not to use spot instances for cluster workers

False

The currently supported AWS regions are:

  • us-east-1

  • us-east-2

  • us-west-1

  • eu-central-1

  • eu-west-2

Example

You can specify backend options directly in Python:

import coiled

cluster = coiled.Cluster(backend_options={"region": "us-west-1"})

Or save them to your Coiled configuration file:

# ~/.config/dask/coiled.yaml

coiled:
  backend-options:
    region: us-west-1

to have them used as the default value for the backend_options= keyword:

import coiled

cluster = coiled.Cluster()

GPU support

This backend allows you to run computations with GPU-enabled machines if your account has access to GPUs. See the GPU best practices documentation for more information on using GPUs with this backend.

Workers currently have access to a single GPU, if you try to create a cluster with more than one GPU, the cluster will not start, and an error will be returned to you.