Configuring GCP#

This guide assumes you have already created your Coiled account and created a Google Cloud Platform (GCP) account. If you don’t have a GCP account, you can sign up for a Free Tier account. You should also make sure you already have the gcloud CLI installed.

In this guide you will configure Coiled to run Dask computations entirely within your own GCP account. You will first create a service account from your Google Cloud account then configure your Coiled cloud account to use your Google Cloud account.

Create a service account#

First, you will create a service account in your Google Cloud account. Then you can configure your Coiled cloud account.

1. Sign in to the console#

Sign in to the Cloud Console. See this Google Cloud console reference for the different ways to access the Cloud console.

2. Create an IAM custom role#

Coiled requires a limited set of IAM permissions to provision infrastructure and compute resources in your GCP account. We recommend creating a IAM custom role to align with the principle of least privilege (see the Google Cloud documentation on understanding IAM custom roles).

Important

Ask your administrator to grant you the Role Administrator IAM role (roles/iam.roleAdmin) for a project or Organization Role Administrator IAM role (roles/iam.organizationRoleAdmin) for an organization (see the Google Cloud documentation on required roles).

Follow the steps in the Google Cloud documentation on creating a custom role. Select the gcloud tab and follow the instructions using the gcloud CLI to create a custom role using a YAML file using the YAML file in dropdown below:

Then, use the gcloud command to create your custom IAM role in a PROJECT-ID of your choosing:

gcloud iam roles create coiled --project=<PROJECT-ID> --file=coiled.yaml

3. Create a service account#

A service account provides the necessary identity and authentication for running Coiled (see the Google Cloud documentation on service accounts best practices). In this step you will create a service account. During this process, you will also grant the IAM role you created in the previous step to this service account. Follow the steps in the Google Cloud documentation on creating a service account, selecting the Console tab.

Important

Ask your administrator to grant you the Service Account Admin (roles/iam.serviceAccountAdmin) IAM role on the project (see the Google Cloud documentation on permissions).

When you get to step 6 “Optional: Choose one or more IAM roles to grant to the service account on the project”, choose the coiled IAM custom role you just created.

Note

You can also bind the service account to the IAM custom role at any time using the Console or gcloud CLI:

gcloud projects add-iam-policy-binding <PROJECT-ID> \
    --member=serviceAccount:<CLIENT-EMAIL> \
    --role=projects/<PROJECT-ID>/roles/coiled

4. Create a service account key#

Once you have a service account for working with Coiled, you will need to create a JSON service account key. Follow the steps in the Google Cloud documentation to create and manage a service account key.

After you create a JSON service account key, the key will be saved to your local machine with a file name such as gcp-project-name-d9e9114d534e.json with contents similar to:

{
  "type": "service_account",
  "project_id": "project-id",
  "private_key_id": "25a2715d43525970fe7c05529f03e44a9e6488b3",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhki...asSSS5J4526eqmrkb1OA=\n-----END PRIVATE KEY-----\n",
  "client_email": "service-account-name@project-name.iam.gserviceaccount.com",
  "client_id": "102238688522576776582",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-name%40project-name.iam.gserviceaccount.com"
}

Keep your JSON service account key handy since you will use it to configure the Coiled Cloud backend in the last step.

5. Optional: Create a second service account for instances#

If the resources you need to access while running your computation are publicly available, then you can skip this step. If, however, you require access to private resources (e.g. BigQuery or Cloud Storage buckets), then read on.

By default, Coiled uses the service account that you created in the previous step and attaches it to each instance created while launching a Dask cluster. This primary service account requires a number of permissions that you configured in 2. Create an IAM custom role, including network-related resources, firewall-related resources, and access to Cloud Storage. Therefore, it is recommended you create a second service account (referred to as the instance service account) with permissions to only access the resources that you need while running your computation, such as access to BigQuery, Cloud Storage buckets and so on.

Note

If you decide to create a specific service account to be used as the instance service account, you should grant it the logging.logEntries.create permission so logs can be exported from the instance to GCP Logging.

Then in the step where you configure your Coiled Cloud backend, you can provide the email of this instance service account, and Coiled will use this service account and attach it to each instance created.

We recommend not using the same service account as the one you provide us to create clusters, since it’s best practice to grant your cluster the “least privilege” it needs and the primary service account you provide us has much stronger permissions than is needed by the code running on your cluster.

6. Configure Google Artifact Registry#

If you want to store the Docker containers for your software environments in your own GCP account, Coiled stores them in the Google Artifact Registry (GAR). If you want to store your software environments in Docker Hub or another external Docker registry, you can skip this step and configure the registry settings when you configure your Coiled Cloud backend.

In this step, you’ll enable the Google Artifact Registry (GAR) API, create a GAR repository for Coiled, and create an IAM policy binding that grants limited access to the service account for Coiled. Using this configuration, Coiled will not have access to any other repositories in your GCP account, and Coiled does not require admin-level permissions to enable APIs or create repositories.

To enable the Google Artifact Registry API, run the following gcloud command in a terminal:

gcloud services enable --project=<PROJECT_ID> artifactregistry.googleapis.com

Create a GAR repository for Coiled to use by running the following command in a terminal. Note that the repository must be named coiled exactly as shown, and that the location should be one that we currently support: us-east1 or us-central1. If you’d like to use a different region, please get in touch with Coiled Support.

gcloud artifacts repositories create coiled \
  --project=<PROJECT_ID> \
  --repository-format=docker \
  --location=<REGION>

Finally, grant access to the repository we just created:

gcloud artifacts repositories add-iam-policy-binding coiled \
   --project=<PROJECT_ID> \
   --location=<REGION> \
   --member=serviceAccount:<CLIENT-EMAIL> \
   --role=roles/artifactregistry.repoAdmin

Note

Ensure that the region specified in the location option is the same region you use when you configure your Coiled Cloud backend. If you want to store software environments in multiple regions, then you can repeat these commands with the desired REGION.

It can take a few minutes for the policy binding to propagate. Keep this in mind if you quickly complete the next step and get an error related to Google Artifact Registry.

Configure your Coiled cloud account#

Now you’re ready to configure the cloud backend in your Coiled cloud account to use your GCP account and GCP service account credentials.

1. Log in to your Coiled account#

First, log in to your Coiled account. In the navigation bar on the left, click on Setup. Select Cloud Provider Configuration, then click the Edit button:

../_images/cloud-backend-start.png

Note

You can configure a different cloud backend for each Coiled account (i.e., your personal/default account or your Team account). Be sure that you’re configuring the correct account by switching accounts at the top of the left navigation bar in your Coiled dashboard if needed.

2. Select your cloud provider#

On the Select Your Cloud Provider step, select the GCP option, then click the Next button:

../_images/cloud-backend-provider-gcp.png

3. Network configuration#

On the Network Configuration step, select whether you would like Coiled to automatically create new or manually use existing VPC and network resources (see Bring your own network):

../_images/cloud-backend-network.png

4. Configure GCP#

On the Configure GCP step, select the zone you want to use by default (i.e., when a zone is not specified in the Coiled Python client). You will need to add your JSON service account key file. Optionally, if you created an instance service account, enter the service account email now. Then click the Next button:

../_images/cloud-backend-keys-gcp.png

5. Container registry#

On the Container Registry step, select where you want to store Coiled software environments, then click the Next button:

../_images/cloud-backend-registry-gcp.png

6. Review#

Review the cloud backend provider options that you’ve configured, then click on the Submit button:

../_images/cloud-backend-review-gcp.png

Next Steps#

Congratulations, Coiled is now configured to use your GCP account!

Follow the Getting Started tutorial to create a Coiled cluster and run a computation. See Google Cloud Platform (GCP) for a more detailed description and additional configuration options.