FAQ¶
Usage¶
Can I use Coiled to do machine learning, data science, etc.?
Yes! Coiled builds on the popular PyData ecosystem of tools, and Dask in particular. Refer to the following resources to learn more about what you can do with Dask and Python:
You may also want to check out our YouTube channel for interviews with community members that are using Python at scale.
Does Coiled support Jupyter notebooks?
Yes, you can use Coiled from a Jupyter Notebook on your laptop or in a cloud-hosted environment.
Coiled also supports running hosted, sharable notebooks. Click on the
Notebooks tab in the Coiled app for example notebooks and steps on how
to get started. You may also want to try some of our notebooks at
cloud.coiled.io/examples/notebooks.
Note that today Coiled does not yet persist user state across notebook sessions. Refer to the documentation on Notebooks for more information.
Can I use Coiled from Sagemaker, VS Code, PyCharm, etc.?
Yes, you can use Coiled from anywhere that you can run Python. Coiled is agnostic to any specific user environment, editor, or IDE.
Does Coiled support GPUs?
Yes, however, access to GPU resources is not currently available when using the free tier in Coiled Cloud.
For more information on using GPUs with Coiled Pro and Coiled Enterprise, refer to coiled.io/pricing or reach out to us at sales@coiled.io.
Can I use a container registry other than DockerHub or ECR?
Yes, you can use any Docker registry that supports the Docker Registry API V2. Get in touch with our support team at support@coiled.io if you encounter any issues.
Data¶
How do I access my data from Coiled?
When you run computations on Dask clusters managed by Coiled, you can access many different file formats using the typical approaches used by Dask, Python, and related libraries.
Coiled can provision Dask clusters on different cloud providers. Therefore, large datasets should be stored on the cloud using services such as Amazon S3 to avoid large data transfer costs. Be sure to also consider which region you are running Coiled in compared to which region the data resides in.
Do I need to migrate my data to Coiled?
No, Coiled does not store any of your data. Rather, your data can remain in its current location. Coiled manages computation and helps you load data from your existing data sources, process it, and write results to those same (or other) data sources.
Does Coiled collect logs from my cluster?
If you have configured Coiled to run within your own cloud provider account on AWS, GCP, or Azure, then Coiled doesn’t collect or store cluster or server logs. In these cases, Coiled uses a token to access the logs in your account to display them from within the cluster dashboard.
Why does Coiled need permissions for my container registry?
Coiled stores built software environments as Docker images in the container registry in your cloud provider account based on your pip/conda dependencies and uses these images when you create a cluster. Even if you don’t plan to install any dependencies, Coiled still needs the permissions to access your container registries when creating container-only software environments.
Libraries¶
How do I install libraries on my Coiled clusters?
Coiled helps you manage software environments both on your local machine and
on cloud providers. You can specify custom environments using pip or conda
environment files with the coiled.create_software_environment function
and Coiled will manage building Docker images that can then be used as
software environments in Dask clusters or other jobs on the cloud.
Refer to the documentation on Software Environments for more information.
Why do I need a local software environment?
When performing distributed computations with Dask, you’ll create a
distributed.Client object which connects your local Python process
(e.g., your laptop) to your remote Dask cluster (e.g., running on the
cloud). Dask Client s are the user-facing entry point for submitting
tasks to a Dask cluster. When using a Client to submit tasks to your
cluster, Dask will package up and send data, functions, and other Python
objects needed for your computations from your local Python process where
your Client is running to your remote Dask cluster in order for them
to be executed.
This means that if you want to run a function on your Dask cluster, for
example NumPy’s numpy.mean() function, then you must have NumPy
installed in your local Python process so Dask can send the numpy.mean
function from your local Dask Client to the workers in your Dask
cluster. For this reason, it’s recommended to have the same
libraries/versions installed on both your local machine and on the remote
workers in your cluster.
Refer to the documentation on Software Environments for more details on how to easily synchronize your local and remote software environments using Coiled.
Why do I get Version Mismatch warnings?
When running cloud computations from your local machine, we need to ensure some level of consistency between your local and remote environments. For example, your Python versions should match, and if you want to use a library like PyTorch or Pandas remotely, then you should probably also install it locally. When Coiled detects a mismatch, it will inform you with a warning.
Matching versions can be challenging if handled manually. Fortunately, Coiled provides functionality to help build and maintain software environments that match across local and remote environments. Refer to the documentation on Installing software environments locally for more information.
Deployment¶
Which cloud providers does Coiled support?
Coiled currently supports running within our managed AWS, GCP, and Azure environments or within your own AWS, GCP, or Azure account. Refer to the documentation on Backends for more information on backend configuration, supported regions, and GPUs.
Can Coiled provision resources in my own cloud account?
Yes! By default, Coiled creates Dask clusters in our managed AWS environment. However, for security or compliance reasons, you may prefer to have Coiled provision compute resources within your own AWS, GCP, or Azure account. Refer to the documentation on Backends for more information.
Can I run Coiled on-premises?
If you want to use Coiled in your own private cloud account, Coiled can be configured to run within your own AWS, GCP, or Azure account. Refer to the question “Can Coiled provision resources in my own cloud account?” above.
If you want to run Coiled on your own machines in your own data center, we would love to hear from you. Please contact sales@coiled.io to start a conversation with us.
Backends¶
Which cloud services and resources does Coiled use on the backend?
Coiled currently uses VM-based backends for Dask clusters on AWS, Azure, and GCP.
After working with users on different backends (ECS, Fargate, EC2, Kubernetes) and use cases over the past year, we’ve observed that using a VM-based backend for Dask clusters results in the right balance of performance, consistency across cloud providers, explicit control over CPU/RAM/GPU resources, ease of use for working with larger instances and GPUs, and scalability. Therefore, starting in August 2021, we will be deprecating the ECS/Fargate backend and migrating all Coiled users to use the VM-based backend by default.
Which backend am I currently using in Coiled?
You can identify which backend you are currently using by viewing the
Account page in your Coiled dashboard. If you registered for a Coiled
account on or before August 12, 2021 and have not changed your backend
settings, then you will be using the ECS/Fargate backend. If you registered
for a Coiled account after August 12, 2021 and have not changed your backend
settings, then you will be using the VM backend.
What are the differences between the ECS and VM backends in AWS?
In our experience working with users, the VM backend for Dask clusters provides improved performance, consistency across cloud providers, explicit control over CPU/RAM/GPU resources, ease of use for working with larger instances and GPUs, and scalability compared to other backend choices.
Regarding cost differences, costs in ECS are a function of the amount of requested CPU/RAM/GPUs, whereas costs for VMs are a function of specific instance types and the amount of CPU/RAM/GPUs provided by those instances. Similar clusters that are using ECS vs. VMs will cost approximately the same, but there will be some slight differences in instance types and costs due to the layer of abstraction provided by ECS. In practice, an on-demand cluster in ECS would cost about 10% to 20% less per hour than an equivalently sized cluster in EC2/VM, but the limitations do not justify the cost savings for typical Dask needs. Additionally, the use of spot/preemptible instances for VMs (which Coiled uses by default) would further reduce costs of VM-backed Dask clusters by 60% to 80%.
Do I need to take any action if I’m using the ECS backend?
If you are using the ECS backend in Coiled, then you’ll be automatically migrated to the VM backend in the near future. You’ll receive more information about this change before it occurs. If you’d like, you can also manually switch to the VM backend by following the documented steps and use your preferred backend and cloud provider.
Availability¶
How much does Coiled cost?
When using the free tier in Coiled Cloud, you will not be charged for any of the compute resources you use, however there is a limit of 100 concurrently running cores per user and a monthly quota/limit based on usage.
For more information on pricing, Coiled Pro, and Coiled Enterprise, refer to coiled.io/pricing.
How do I invite colleagues, students, etc.?
We’re glad that you’re enjoying Coiled and want to invite colleagues or students. Coiled is currently open access, so your colleagues can join on their own without any additional steps.
If you want to work within a team account with a group of users from your organization, then you can send an e-mail to sales@coiled.io with a team name and we’ll set you up as an administrator for your new team. Refer to the documentation on Teams for more information.
Security¶
Can I use Coiled to read private data on AWS?
Yes. If you create a Coiled cluster from an environment that has AWS credentials defined, then Coiled will generate a secure token from those credentials and forward it to your Dask workers. The Dask workers will have the same rights and permissions that you have by default.
For additional control, Coiled can be deployed within your own AWS, GCP, or Azure account where you can specify and manage IAM roles directly. Refer to the documentation on Security & Privacy for more information.
Are my computations and data secure?
Coiled provides end-to-end network security by the use of both cloud networking policies and with SSL/TLS encryption. Coiled does not persist or store any of your data, data only resides in memory as long as you are performing computations.
For additional control, Coiled can be deployed within your own AWS account where you can specify and manage data access controls directly. Refer to the documentation on Security & Privacy for more information.
Connect with us¶
How can I submit a bug report, feature request, or other question?
First, thank you! Your feedback is highly valuable and will help influence the future of Coiled.
For bug reports, please open a support ticket and our support team will be happy to help you.
For feature requests or other usability feedback, we’d love to hear from you on our feedback portal!
For other questions, please join our Coiled Community Slack where you can ask questions and interact with our engineers as well as the Coiled community.
How can I keep up with the latest news about Coiled?
To stay up to date with Coiled, you can subscribe to our newsletter and follow us on Twitter, YouTube, and LinkedIn.