Bring Your Own Network¶

Usually Coiled creates all the cloud networking resources required for running a cluster. For customers who are hosting Coiled in their own AWS or Google Cloud account, we also provide the option to have Coiled use an existing network which you have created.

While this means you’re responsible managing more aspects of hosting Coiled, it also enables you to run Coiled while meeting specific needs for network security or configuration, such as:

you need to peer the VPC used for Coiled clusters with other networks
you need to configure additional network security, for example, routing traffic through a customer-managed firewall or limiting inbound connections to a VPN
you need to configure network access to your data sources, for example, using AWS PrivateLink
you need to limit the IAM permissions that you grant to Coiled

If you provide a network for Coiled to use, you’ll be responsible for:

VPC
subnet(s)
routing and internet access (including NAT for VMs without public IP address)
security groups (AWS) or firewall rules (GCP)

If you provide a network, Coiled will still be responsible for creating VMs (and associated storage, network interface, and public IPs).

You can configure Coiled to use your network on the Infrastructure tab under Cloud Provider for your Coiled workspace.

Network requirements¶

See Network Architecture for details about the networking needs of a Coiled cluster.

The network you provide for Coiled to use needn’t match the networks we create by default, but they do need to meet some minimal requirements.

Our default network allows public ingress to the scheduler on ports 8786 and 8787. This isn’t a requirement, so long as the machine running the Python client is able to connect to the scheduler. For instance, you could be running the client on a machine inside a paired VPC or go through a VPC which allows you to connect to private IP of the scheduler. Ports 8786, 8787 need to be open for ingress so that the client can connect to scheduler.

It’s necessary that the scheduler and workers be able to download software (as well of course as any data used in your computations). This can be achieved by using a NAT Gateway which is set as next hop for outbound connections, but it can also be achieved by allowing us to assign public IP addresses for workers as well as the scheduler.

Example for a single, public subnet¶

One way to structure your network on GCP is to have a single public subnet that’s used for both schedulers and workers. Scheduler and workers would all use public IP addresses, and you could use firewall rules to block ingress to the workers from outside the cluster.

The main components involved are:

VPC Network with a subnet to use for Coiled clusters
Routes which route traffic to the CIDR block of your VPC to the VPC and route traffic to 0.0.0.0/0 to the default internet gateway
three firewall rules:
- Allow egress to 0.0.0.0/0 for all ports. This will have two target tags, one which will be used as scheduler tag, one that will be used as cluster tag
- Allow ingress on ports 8786 and 8787 from the your Python client, which could be achieved by opening these ports to traffic from anywhere (0.0.0.0/0), or a more limited IP range such as a VPN you’re using to connect to scheduler from Python client. This will have target tag which will be used as scheduler tag.
- Allow ingress for all ports with the source as the two target tags you’re using, the scheduler tag and the cluster tag. This rule should target the cluster tag.

When configuring Coiled to use your network, you’d specify the same subnet as both scheduler subnet and worker subnet. Since you aren’t using NAT Gateway, you’ll need to configure Coiled to give the workers public IP addresses.

Manual Setup

Configure Firewalls