You can access many different file formats using Dask and related libraries.
Large datasets should live on the cloud in services like S3 to avoid data
Additionally, if you have small data locally you can use other Dask APIs, like
Dask-ML or Dask Futures to do heavy computation on small datasets comfortably.
Coiled helps you manage software environments both on your local machine
and on cloud resources. You can specify custom environments with conda or pip
environments files with the coiled.create_software_environment function and
Coiled will manage building Docker images for you that can then be used in Dask
clusters or other jobs on the cloud.
See Software Environments for more information.
Yes. Coiled accounts run by default in our hosted AWS account, but for
pricing or security reasons you may prefer to run things on your own
See Backends for more information.
No. Your data stays in its native location(s).
Coiled manages computation. Coiled helps you get data from your existing data stores,
process it, and write out to those same data stores.
Yes. If you create a Coiled cluster from a location that has AWS credentials
then Coiled will generate a secure token from those credentials and forward it
to your Dask workers. Your Dask workers will have the same rights and
permissions that you have by default.
Alternatively, for additional control Coiled can be deployed within your own AWS
account and allow you to specify and manage IAM roles directly.
See Security & Privacy for more information.
Coiled provides end-to-end network security with both cloud networking policies
and with TLS encryption. Coiled does not persist or store any of your data,
except in memory as you are doing computations.
You can also use Coiled to manage computation in your own account or on
your own hardware (see the next question below)
Coiled is currently in beta. During this time Coiled is free for all beta users.
You will not be charged for any of the compute resources you use, however there is a limit of
100 concurrently running cores per user. This policy will change in the future when Coiled is opened
up to a broader audience, but until then we are happy to provide beta users cloud computing
resources at no cost. Thank you for trying out Coiled!
For more information on pricing, see coiled.io/pricing
Today Coiled provides full support for AWS with GCP and Azure in the works
(we’re aiming for Q2 2021).
In the meantime, you can set up your own Kubernetes cluster on GKE or AKS
and have Coiled manage that.
Please open an issue on the
Coiled issue tracker. Feel free to report bugs, submit
feature requests, ask questions, or provide other input. Your feedback is valued and will help influence
the future of Coiled.
For generic questions, please join our Coiled Community Slack where you can ask questions and interact with our engineers and with the Coiled community.
We’re glad that you’re having a good time and want to invite colleagues or
students. Coiled is currently open access, so your colleagues can join
without any special setup.
Additionally, if you want to organize a team account send a quick e-mail to
firstname.lastname@example.org with a team name and we’ll set you up as an administrator
over your new team.
See Teams for more information.
When running cloud computations from your local machine we need to ensure some
level of consistency between your local and remote environments. For example
your Python versions should match, and if you want to use a library like
PyTorch or Pandas remotely we should probably also install it locally. When
Dask notices a mismatch, it will tell you with a warning.
Matching versions can be challenging if handled manually. Fortunately Coiled
provides services to help build and maintain software environments that match
across local and remote environments.
See the Software Environments documentation for more information.
Yes, but GPU support is only available today to paying users.
Yes, you can use Coiled from anywhere that you can use Python.
Coiled is agnostic to user environment.
Yes, see the Notebooks tab in the application for example notebooks for
notes on how to get started. You may also want to try some of our
notebooks at cloud.coiled.io/examples/notebooks.
Coiled can run and manage arbitrary Python processes in the cloud,
including Jupyter notebooks. Note that today Coiled does not yet persist
user state across notebook sessions.
There are two types of on-prem that we support. For those looking to run
Coiled in your own cloud account, we support that today. See the question
“Can I run computation in my own account?” above.
For those looking to run Coiled on your own machines in your own data
center, we would love to hear from you. Please contact email@example.com to
start a conversation with us.
If by on-prem you mean “in yo.
When performing distributed computation with Dask, you’ll create a distributed.Client
object which connects your local Python process (e.g. your laptop) to your remote Dask cluster
(e.g. running on AWS). Dask Client s are the user-facing entry point for submitting tasks to
a Dask cluster. When using a Client to submit tasks to your cluster, Dask will package up and send data,
functions, and other Python objects needed for your computations from your local Python process
where your Client is running to your remote Dask cluster in order for them to be executed.
This means that if you want to run a function on your Dask cluster, for example NumPy’s numpy.mean()
function, then you must have NumPy installed in your local Python process so Dask can send the numpy.mean
function from your local Dask Client to the workers in your Dask cluster. For this reason,
it’s recommended to have the same libraries installed on both your local machine and on the remote
workers in your cluster.
See the Software Environments section for more details on how to easily
synchronize your local and remote software environments using Coiled.
You might want to sign up for our newsletter, and follow us on Twitter or LinkedIn.
Yes! Coiled builds on the popular PyData ecosystem of tools, and Dask in
particular. To learn more about what you can do with Dask and Python see the
You may also want to check out our Youtube channel for interviews with community members using
Python at scale.