FAQ#
Usage#
Can I use Coiled to do machine learning, data science, etc.?
Yes! Coiled builds on the popular PyData ecosystem of tools, and Dask in particular. Refer to the following resources to learn more about what you can do with Dask and Python:
You may also want to check out our YouTube channel for interviews with community members that are using Python at scale.
Does Coiled support Jupyter notebooks?
Yes, you can use Coiled from anywhere that you can run Python, including a Jupyter Notebook on your laptop, a cloud-hosted Jupyter Notebook, or a multi-user JupyterHub instance.
And if you’re using JupyterLab, you can learn more about some extensions that we recommend trying to make JupyterLab even better when working with Coiled and Dask.
Can I use Coiled from Sagemaker, VS Code, PyCharm, etc.?
Yes, in addition to using Coiled from Jupyter Notebooks, you can use Coiled from anywhere that you can run Python. Coiled is agnostic to any specific user environment, editor, or IDE.
Coiled was built for the purpose of making it easy to work with remote Dask clusters from anywhere, and most of our users work with Jupyter Notebooks or their favorite Python IDE to work with Coiled and Dask.
Does Coiled support GPUs?
Yes! See GPUs for more details.
Can I use a container registry other than DockerHub or ECR?
Yes, you can use any Docker registry that supports the Docker Registry API V2. Get in touch with our support team at support@coiled.io if you encounter any issues.
Data#
How do I access my data from Coiled?
When you run computations on Dask clusters managed by Coiled, you can access many different file formats using the typical approaches used by Dask, Python, and related libraries.
Coiled can provision Dask clusters on different cloud providers. Therefore, large datasets should be stored on the cloud using services such as Amazon S3 to avoid large data transfer costs (see Remote Data Access). Be sure to also consider which region you are running Coiled in compared to which region the data resides in.
Do I need to migrate my data to Coiled?
No, Coiled does not store any of your data. Rather, your data can remain in its current location. Coiled manages computation and helps you load data from your existing data sources, process it, and write results to those same (or other) data sources.
Does Coiled collect logs from my cluster?
If you have configured Coiled to run within your own cloud provider account on AWS or GCP, then Coiled doesn’t collect or store cluster or server logs. In these cases, Coiled uses a token to access the logs in your account to display them from within the cluster dashboard.
Libraries#
How do I install libraries on my Coiled clusters?
Coiled helps you manage software environments both on your local machine and
on cloud providers. You can specify custom environments using pip or conda
environment files with the coiled.create_software_environment() function
and Coiled will manage building Docker images that can then be used as
software environments in Dask clusters on the cloud.
Refer to the documentation on Software Environments for more information.
Why do I need a local software environment?
When performing distributed computations with Dask, you’ll create a
distributed.Client object which connects your local Python process
(e.g., your laptop) to your remote Dask cluster (e.g., running on the
cloud). The Dask Client is the user-facing entry point for submitting
tasks to a Dask cluster. When using a Client to submit tasks to your
cluster, Dask will package up and send data, functions, and other Python
objects needed for your computations from your local Python process where
your Client is running to your remote Dask cluster in order for them
to be executed.
This means that if you want to run a function on your Dask cluster, for
example NumPy’s numpy.mean() function, then you must have NumPy
installed in your local Python process so Dask can send the numpy.mean
function from your local Dask Client to the workers in your Dask
cluster. For this reason, it’s recommended to have the same
libraries/versions installed on both your local machine and on the remote
workers in your cluster.
Refer to the documentation on Package Synchronization for more details on how to easily synchronize your local and remote software environments using Coiled.
Why do I get Version Mismatch warnings?
When running cloud computations from your local machine, we need to ensure some level of consistency between your local and remote environments. For example, your Python versions should match, and if you want to use a library like PyTorch or Pandas remotely, then you should probably also install it locally. When Coiled detects a mismatch, it will inform you with a warning.
Matching versions can be challenging if handled manually. Fortunately, Coiled provides functionality to help build and maintain software environments that match across local and remote environments. Refer to the documentation on Package Synchronization for more information.
Deployment#
Which cloud providers does Coiled support?
Coiled currently supports running within your AWS or GCP account. See How Coiled works in your cloud account for more information on configuration, supported regions, and GPUs.
Can I run Coiled on-premises?
If you want to run Coiled on your own machines in your own data center, we would love to hear from you. Please contact sales@coiled.io to start a conversation with us.
Availability#
How do I invite colleagues, students, etc.?
We’re glad that you’re enjoying Coiled and want to invite colleagues or students. Coiled is currently open access, so your colleagues can join on their own without any additional steps.
If you want to work within a team account with a group of users from your organization, then you can send an e-mail to sales@coiled.io with a team name and we’ll set you up as an administrator for your new team. Refer to the documentation on Teams for more information.
Security#
Can I use Coiled to read private data on AWS?
Yes. If you create a Coiled cluster from an environment that has AWS credentials defined, then Coiled will generate a secure token from those credentials and forward it to your Dask workers. The Dask workers will have the same rights and permissions that you have by default.
Are my computations and data secure?
Coiled provides end-to-end network security by the use of both cloud networking policies and with SSL/TLS encryption. Coiled does not persist or store any of your data, data only resides in memory as long as you are performing computations (see our Security page for more details).
Connect with us#
How can I submit a bug report, feature request, or other question?
First, thank you! Your feedback is highly valuable and will help influence the future of Coiled.
For bug reports, feature requests, or other usability feedback, we’d love to hear from you! Please submit an issue.
For other questions, please join our Coiled Community Slack where you can ask questions and interact with our engineers as well as the Coiled community.
How can I keep up with the latest news about Coiled?
To stay up to date with Coiled, you can subscribe to our newsletter and follow us on Twitter, YouTube, and LinkedIn.