Bring Your Own Network¶
Usually Coiled creates all the cloud networking resources required for running a cluster. For customers who are hosting Coiled in their own AWS or Google Cloud account, we also provide the option to have Coiled use an existing network which you have created.
While this means you’re responsible managing more aspects of hosting Coiled, it also enables you to run Coiled while meeting specific needs for network security or configuration, such as:
you need to peer the VPC used for Coiled clusters with other networks
you need to configure additional network security, for example, routing traffic through a customer-managed firewall or limiting inbound connections to a VPN
you need to configure network access to your data sources, for example, using AWS PrivateLink
you need to limit the IAM permissions that you grant to Coiled
If you provide a network for Coiled to use, you’ll be responsible for:
VPC
subnet(s)
routing and internet access (including NAT for VMs without public IP address)
security groups (AWS) or firewall rules (GCP)
If you provide a network, Coiled will still be responsible for creating VMs (and associated storage, network interface, and public IPs).
You can configure Coiled to use your network on the Infrastructure tab under Cloud Provider for your Coiled workspace.
Network requirements¶
See Network Architecture for details about the networking needs of a Coiled cluster.
The network you provide for Coiled to use needn’t match the networks we create by default, but they do need to meet some minimal requirements.
Our default network allows public ingress to the scheduler on ports 8786 and 8787. This isn’t a requirement, so long as the machine running the Python client is able to connect to the scheduler. For instance, you could be running the client on a machine inside a paired VPC or go through a VPC which allows you to connect to private IP of the scheduler. Ports 8786, 8787 need to be open for ingress so that the client can connect to scheduler.
It’s necessary that the scheduler and workers be able to download software (as well of course as any data used in your computations). This can be achieved by using a NAT Gateway which is set as next hop for outbound connections, but it can also be achieved by allowing us to assign public IP addresses for workers as well as the scheduler.