GCP Backend

Using Coiled’s GCP account

You can have Coiled launch computations on Google Cloud Platform (GCP). Your computations will run inside Coiled’s Google Cloud account by default, this makes it easy for you to get started quickly, without needing to set up any additional infrastructure.

../_images/backend-coiled-gcp-vm.png

Note

GCP support is currently experimental with new features under active development.

Tip

In addition to the usual cluster logs, our current GCP backend support also includes system-level logs. This provides rich insight into any potential issues while GCP support is still experimental.

Switching Coiled backend to GCP

To use Coiled on GCP, log in to your Coiled account and access your dashboard. Click on Account on the left navigation bar, then click the Edit button to configure your Cloud Backend Options:

../_images/cloud-backend-options.png

On the Select Your Cloud Provider step, select the GCP option, then click the Next button:

../_images/cloud-backend-provider-gcp.png

Proceed by selecting “Launch in Coiled’s GCP Account”. Select Next, then the registry you wish to use and Submit.

Using your own GCP Account

Alternatively, you can configure Coiled to create Dask clusters and run computations entirely within your own GCP account (within a project of your choosing). This allows you to make use of security/data access controls, compliance standards, and promotional credits that you already have in place within your GCP account.

../_images/backend-external-gcp-vm.png

Note that when running Coiled on your GCP account, Coiled Cloud is only responsible for provisioning cloud resources for Dask clusters that you create. Once a Dask cluster is created, all computations, data transfer, and Dask client-to-scheduler communication occurs entirely within your GCP account.

Note

GCP support is currently experimental with new features under active development. GCP customer-hosted operation is currenlty in alpha deployment and available only to early-adopter users. Interested users should contact coiled support.

Step 1: Obtain GCP credentials

Coiled provisions resources on your GCP account by using a key tied to a Service Account, Role, and Project within your organization’s GCP account. You can use an existing service account (with appropriate permissions, see below) or create a service account and generate a key. To do that, please consult the corresponding GCP docs:

The service account credentials will be saved with a file name like, gcp-project-name-d9e9114d534e.json, with content like this:

{
  "type": "service_account",
  "project_id": "gcp-ch-01",
  "private_key_id": "##################################",
  "private_key": "-----BEGIN PRIVATE KEY-----\################################################",
  "client_email": "service-account-name@project-name.iam.gserviceaccount.com",
  "client_id": "##############################",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-name%40project-name.iam.gserviceaccount.com"
}

Step 2: Create a custom IAM Role

Coiled requires a limited set of IAM permissions to be able to provision infrastructure and compute resources in your GCP account.

You’ll need to create a new IAM role and assign the appropriate set of permissions to it. Specify an IAM role name such as Coiled that will make it easy to locate in the next step. For information on creating an IAM role, see the GCP custom role creation documentation.

Due to the amount of permissions that you will have to add, we recommend that you use the gcloud command to create your role from the following yaml file.

Step 3: Connect the Service Account and Role

Finally, you you will need to grant that Service Account access to that Role. For information on connecting service accounts to roles, see the corresponding GCP role access granting documentation.

Step 4: Configure Coiled cloud backend

Now you’re ready to configure the cloud backend in your Coiled Cloud account to use your GCP account and GCP credentials. To configure Coiled to use your GCP account, log in to your Coiled account and access your dashboard. Click on Account on the left navigation bar, then click the Edit button to configure your Cloud Backend Options:

../_images/cloud-backend-options.png

On the Select Your Cloud Provider step, select the GCP option, then click the Next button:

../_images/cloud-backend-provider-gcp.png

On the Configure GCP step, select the GCP region that you want to use by default (i.e., when a region is not specified), choose the Launch in my GCP account option, and add your service account credentials file, then click the Next button.

On the Container Registry step, select where you would like to store your software environments, then click the Next button.

Coiled is now configured to use your GCP Account!

Region

GCP support is currently only available in the us-east1 region. If you have data in a different region on Google Cloud, you may be charged transfer fees.

Backend options

Similar to the AWS backend, the GCP backend uses preemptible instances for the workers by default. Note that GCP automatically terminates these after 24 hours.

Name

Description

Default

region

GCP region to create resources in

us-east1

zone

GCP zone to create resources in

us-east1-c

spot

Whether or not to use preemptible instances for cluster workers

True

Example

You can specify backend options directly in Python:

import coiled

cluster = coiled.Cluster(backend_options={"region": "us-west1", "spot": False})

Or save them to your Coiled configuration file:

# ~/.config/dask/coiled.yaml

coiled:
  backend-options:
    region: us-west1

to have them used as the default value for the backend_options= keyword:

import coiled

cluster = coiled.Cluster()

GPU support

This backend allows you to run computations with GPU-enabled machines if your account has access to GPUs. See the GPU best practices documentation for more information on using GPUs with this backend.

Workers currently have access to a single GPU, if you try to create a cluster with more than one GPU, the cluster will not start, and an error will be returned to you.