Reference#

Granting Coiled Permissions#

Coiled needs limited permissions to operate in your AWS account.

Our automatic, recommended, and default method for this is delegating access via an IAM role. We have two automated methods to set up this IAM role with delegated access:

  1. coiled setup aws, which you can run if you have AWS CLI credentials set up locally.

  2. A CloudFormation template, which you run from the AWS console in your browser. This is linked from the setup instructions in the Coiled UI under “Cloud Provider”.

Both of these methods will set up an IAM role with attached “setup” and “ongoing” policies (described below) that Coiled can use to operate in your AWS account.

It is also possible for you to manually create an IAM user along with the required policies:

IAM Policies#

Coiled requires a limited set of IAM permissions to provision infrastructure and compute resources in your AWS account (see the guide for configuring AWS).

IAM Setup policy document (JSON)
{
    "Statement": [
        {
            "Sid": "Setup",
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "ec2:AssociateRouteTable",
                "ec2:AttachInternetGateway",
                "ec2:CreateInternetGateway",
                "ec2:CreateRoute",
                "ec2:CreateRouteTable",
                "ec2:CreateSubnet",
                "ec2:CreateVpc",
                "ec2:DeleteInternetGateway",
                "ec2:DeleteRoute",
                "ec2:DeleteRouteTable",
                "ec2:DeleteSubnet",
                "ec2:DeleteVpc",
                "ec2:DetachInternetGateway",
                "ec2:DisassociateRouteTable",
                "ec2:ModifySubnetAttribute",
                "ec2:ModifyVpcAttribute",
                "iam:AddRoleToInstanceProfile",
                "iam:AttachRolePolicy",
                "iam:CreateInstanceProfile",
                "iam:CreatePolicy",
                "iam:CreateRole",
                "iam:CreateServiceLinkedRole",
                "iam:DeleteRole",
                "iam:GetPolicy",
                "iam:ListAttachedRolePolicies",
                "iam:ListInstanceProfiles",
                "iam:TagInstanceProfile",
                "iam:TagPolicy",
                "iam:TagRole"
            ]
        }
    ],
    "Version": "2012-10-17"
}
IAM Ongoing policy document (JSON)
{
    "Statement": [
        {
            "Sid": "Ongoing",
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:CreateFleet",
                "ec2:CreateLaunchTemplate",
                "ec2:CreateLaunchTemplateVersion",
                "ec2:CreateSecurityGroup",
                "ec2:CreateTags",
                "ec2:DescribeAvailabilityZones",
                "ec2:DescribeImages",
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceTypeOfferings",
                "ec2:DescribeInstanceTypes",
                "ec2:DescribeInternetGateways",
                "ec2:DescribeLaunchTemplates",
                "ec2:DescribeRegions",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVpcs",
                "ec2:RunInstances",
                "iam:ListPolicies",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents",
                "logs:PutRetentionPolicy",
                "logs:TagLogGroup",
                "logs:TagResource"
            ]
        },
        {
            "Sid": "OngoingInstanceProfile",
            "Effect": "Allow",
            "Resource": "arn:aws:iam::*:instance-profile/coiled-*",
            "Action": [
                "iam:GetInstanceProfile"
            ]
        },
        {
            "Sid": "OngoingAttachInstancePolicy",
            "Effect": "Allow",
            "Resource": "arn:aws:iam::*:role/coiled-*",
            "Action": [
                "iam:GetRole",
                "iam:PassRole"
            ]
        },
        {
            "Sid": "OngoingDestructive",
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "ec2:DeleteFleets",
                "ec2:DeleteLaunchTemplate",
                "ec2:DeleteLaunchTemplateVersions",
                "ec2:DeleteSecurityGroup",
                "ec2:TerminateInstances"
            ],
            "Condition": {
                "StringEquals": {
                    "ec2:ResourceTag/owner": "coiled"
                }
            }
        },
        {
            "Sid": "OptionalLogPull",
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "logs:GetLogEvents",
                "logs:FilterLogEvents"
            ]
        },
        {
            "Sid": "OptionalECR",
            "Effect": "Allow",
            "Resource": "*",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:CompleteLayerUpload",
                "ecr:CreateRepository",
                "ecr:DescribeImages",
                "ecr:DescribeRepositories",
                "ecr:GetAuthorizationToken",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:InitiateLayerUpload",
                "ecr:ListImages",
                "ecr:PutImage",
                "ecr:UploadLayerPart",
                "ecr:TagResource"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Once you have completed setting up Coiled with your AWS account you can detach the Setup policy. For more advanced IAM configuration options see Limit Coiled’s Access to AWS Resources.

Resources#

Coiled provisions the following resources on your AWS account:

../../../_images/backend-coiled-aws-architecture.png

AWS resources for a Dask cluster with 5 workers#

When you create additional Dask clusters with Coiled, then another scheduler VM and additional worker VMs will be provisioned within the same public and private subnets, respectively. As you create additional Dask clusters, Coiled will reuse and share the existing VPC and other existing network resources that were initially created. See Tracking AWS resources.

If you encounter any issues when setting up resources, you can use the method coiled.get_notifications() to have more visibility into this process.

Warning

If you get a permissions error when reading from an S3 bucket, you may need to attach S3 policies to the role that Coiled creates to attach to EC2 instances. The role name is the same as your Coiled workspace slug.

Quotas#

Each AWS service has pre-defined quotas, or limits, which are the maximum values for the resources in your AWS account (see the AWS Service Quotas guide). You may want to request an increase in these quotas, especially if you have received LimitExceeded error messages while using Coiled.

You can run coiled setup aws --quotas to view or adjust your quotas.

Tip

No local AWS credentials?

You can use AWS CloudShell, which already has the AWS CLI installed and configured with your AWS credentials.

pip3 install coiled && \
coiled setup aws --quotas

Or, you can view them from the Service Quotas console and follow these instructions from AWS Support to request an increase.

Backend Options#

There are several AWS-specific options that you can specify (listed below) to customize Coiled’s behavior. Additionally, the next section contains an example of how to configure these options in practice.

Name

Description

Default

region_name

AWS region to create resources in

us-east-1

zone_name

AWS Availability Zone to create cluster

depends on region (see Availability Zone)

firewall

Ports and CIDR block for the security groups that Coiled creates

{"ports": [22, 443], "cidr": "0.0.0.0/0"}

The currently supported AWS regions are:

  • us-east-1

  • us-east-2

  • us-west-1

  • us-west-2

  • ap-southeast-1

  • ca-central-1

  • ap-east-1

  • ap-northeast-1

  • ap-northeast-2

  • ap-south-1

  • ap-southeast-1

  • ap-southeast-2

  • eu-central-1

  • eu-north-1

  • eu-west-1

  • eu-west-2

  • eu-west-3

  • sa-east-1

Note

Coiled will choose the us-east-1 region by default. If you don’t wish to use this region, you should provide a different region.

You can specify backend options when creating a cluster:

import coiled

cluster = coiled.Cluster(backend_options={"region_name": "us-west-1"})

Or at the account level for yourself or your team members using coiled.set_backend_options():

import coiled

coiled.set_backend_options(aws_region="us-west-1")

Or save them to your Coiled configuration file ~/.config/dask/coiled.yaml (see Configuration):

coiled:
  backend-options:
    region: us-west-1

GPU support#

This backend allows you to run computations with GPU-enabled machines if your account has access to GPUs. See the GPU best practices documentation for more information on using GPUs with this backend.

Workers currently have access to a single GPU, if you try to create a cluster with more than one GPU, the cluster will not start, and an error will be returned.

Logs#

If you are running Coiled on your own AWS account, cluster logs will be saved within your AWS account. Coiled will use CloudWatch to store logs.

Coiled will create a log group with your workspace name and add a log stream for each instances that Coiled creates. These logs will be stored for 30 days.

Log Storage

Storage time

Cloudwatch

30 days

Availability Zone#

The availability of different VM instance types varies across AZs, so choosing a different AZ may make it easier to create a cluster with the desired number and type of instances.

This option allows you to pick the Availability Zone (AZ) to use for a cluster. Each AZ is one or more distinct data centers located within a region. For example, the us-east-1 region contains the us-east-1a zone, (as well as b, c, d, and f zones).

You can specify the zone to use when creating an individual cluster like so:

cluster = coiled.Cluster(backend_options={"zone_name": "us-east-1b"})

In order to create a Dask cluster in a given AZ, we need a subnet for that specific zone.

When you configure Coiled to use your AWS account (as described above), Coiled attempts to create a subnet for every zone in the selected region instead of just the default zone (note that there are no additional AWS or Coiled costs associated with each subnet).

When creating a Dask cluster, you can specify the zone to use for that cluster. Ideally the specified zone already has the required subnet (created when you configured Coiled to use your AWS account) but if not, we’ll attempt to create a subnet at cluster-creation time. This may fail if Coiled no longer has “setup” IAM permissions; you’ll get an error message if we are unable to find or create a subnet in the specified zone.

Assuming we are able to find or create the required subnet, then we’ll then create your Coiled cluster in the specified availability zone.

If no zone is specified when creating an individual cluster, we’ll use the zone set at the workspace-level (currently this can only be set if you configure your account backend using the the Python API), and if that isn’t set, we’ll use the default zone for the region your account is configured to use.

Refer to the AWS documentation on Regions and Availability Zones for additional information.

Networking#

When Coiled is configured to run in your own AWS account, you can customize the security group ingress rules for resources that Coiled creates in your AWS account.

By default, Dask schedulers created by Coiled will be reachable via port 443 from any source network and port 22 from the IP address of the client machine starting the cluster. This is consistent with the default ingress rules that Coiled configures for its AWS security groups:

Protocol

Port

Source

tcp

443

0.0.0.0/0

tcp

22

client IP address

Note

Port 443 is used by the Dask dashboard and Dask client protocol. Port 22 optionally supports incoming SSH connections to the virtual machine.

Configuring firewall rules#

While allowing incoming connections on the default Dask ports from any source network is convenient, you might want to configure additional security measures by restricting incoming connections. This can be done by using coiled.set_backend_options() or by using the backend_options.

On this page