Why Coiled?

Coiled is a lightweight cloud platform designed for ease of use and efficiency.

Promise and Challenges of Cloud

Cloud is like a bulldozer: immensely powerful but hard to use and expensive when misused.

Cloud Promise: Cloud computing provides large scale computing close to your data on any hardware with by-the-minute pricing. This is potentially game-changing.

Cloud Challenges: However, cloud utilization remains low in most organizations largely because usability remains poor. Similarly costs are high due to misuse.

 

Coiled provides an easy-to-use platform designed for data professionals that retains the full power and flexibility of cloud computing, while drastically lowering the barrier to entry.

Challenges to overcome

Users often report the following challenges with cloud:

Hard to use: The cloud wasn’t designed for ease of use by data people.

  • Baroque APIs: Cloud APIs are verbose and obtuse, often requiring deep understanding to do even simple things.

  • Docker: Data professionals rely on highly custom and frequently changing software environments. Managing packages on one’s personal laptop is hard, but distributing these to cloud instances is often infeasible, especially as many of these people don’t understand Docker well.

  • Opaque: When things break, it’s hard to tell what happened. Failures aren’t as obvious, and while logs exist, but may be hidden away. This makes it difficult and time consuming to move quickly.

Expensive: While the cloud can be very cheap, it rarely is.

  • Idle resources: Costly resources often are left on unused, either due to forgetfulness, or because of users leave resources on all-day or all-week in order to avoid the time-consuming process of recreating them.

  • Personnel: People are expensive, including the people who manage the cloud for your team. Often even small data teams sacrifice half of an FTE to the task of tending the in-house Kubernetes cluster.

  • Surprise Bills: Often due to a lack of understanding, people receive surprisingly large bills with costs that they did not intend to incur. A few examples follow:

    • Cross-region data access: They may pull data from us-west-1 into machines in us-east-2 without realizing. They may do this a lot before they notice.

    • Cross-availability-zone data transfer: Even within one region one can spin up multiple machines in different AZs / data centers and suffer costs from moving between them.

    • NAT Gateways: Traffic in and out of the cloud may be costly, depending on the network infrastructure used to mediate that access.

    • Stopped but persistent infrastructure: Resources like IP addresses, storage, and so on that are attached to deleted computational resources may persist indefinitely, persistently racking up costs.

Insecure: Mismatch between data users and IT departments.

Most users don’t complain about this directly. Instead they complain that their IT Department won’t let them use the cloud due to security concerns.

This concern is valid, most data users are bad at security.

Coiled’s Approach

With Coiled we wanted a lightweight cloud platform that was …

  • Easy to use: as easy as common Python libraries

  • Flexible: let us use any cloud hardware for any purpose

  • Cost efficient: a pass-through in terms of cloud costs, with no costs at-rest

To achieve this we follow a few design principles and architecture decisions. This page helps explain some of those broader principles behind the Coiled platform.

What we didn’t want

This vision is more clear with counter-example. Many data platforms we had seen and built prior to Coiled looked like the following:

  1. Deploy Kubernetes on one node-type in one region

  2. Install 3-5 common frameworks on top (Spark, JupyterHub, Airflow, Prometheus, and the model management system du jour)

  3. Develop lightweight user management and billing

  4. Market as “all-in-one data platform”

This was great initially. It checked all of the boxes and these initial choices were good. Invariably however users broke from our initial opinions and asked for different choices:

  1. Can I use bigger instance types? How about GPUs?

  2. Can I use this new framework?

  3. Can I read data from this other region?

  4. Can I use Spot instances?

  5. Can I drive this from my CI service?

  6. Can I script this platform from Python?

  7. …

Our common data platform model was inflexible and we were always working to catch up to the creativity of users, a losing battle.

To address this we set out to build something different. Coiled operates on raw cloud hardware with no at-rest infrastructure. It combines the serverless nature of AWS Lambda with the cost-efficiency and low-tech nature of the venerable EC2 instance.

This document says more about our choices, and what they help you achieve.

Control from our cloud, Deploy into yours

Coiled launches resources in your cloud, not ours.

../_images/coiled-architecture.png

Coiled Architecture (click to enlarge)

The Coiled control service lives in our account on AWS, shared among all users. Coiled deploys resources within your cloud account on your behalf.

This configuration accomplishes two goals:

  1. Easy for you to deploy (minimal cloud setup)

  2. Safe to use on secure private data (your data never leaves your cloud)

For more information, see Security.

Raw VMs

Coiled launches raw VMs in your cloud account.

This makes it easy to deploy resources of

  • Any hardware / VM type

  • In any region

  • In about a minute

  • At any scale provided by your cloud quotas

  • With no fixed costs at rest

  • Without anyone else seeing your data (even us)

This is as opposed to other options, like Kubernetes, ECS, AWS Lambda, or hosted solutions which would have been easier to start with, but came with challenging constraints which eventually led to frequent user pain.

This lets you ask, for example, for “500 4-core ARM machines with SSDs in London” and have them arrive 90 seconds later fully secured and running exactly your software. You don’t need to mansge container registries, kubernetes node auto-scaling pools, and so on.

From anywhere

Coiled runs anywhere you run Python.

Other solutions often require you to run from their cloud development environment, like a hosted notebook or specific job runner service. We found this to constrain both user experience, and flexibility of integration.

Instead, Coiled focuses on connecting from anywhere with public internet access. You can drive Coiled from…

  • The computer sitting on your desk

  • A cloud notebook run by your favorite service

  • A CI service like GitHub Actions or a Jenkins server in your basement

  • An AWS Lambda function

  • … anywhere else

Coiled inspects your environment for software, credentials, files, and more, and constructs remote machine that look similar, connecting you in 1-2 minutes.

Coiled makes it easy to continue working as you have always done, just bigger.

Local environment synchronization

Configure your laptop how you like it, Coiled configures your cloud to match.

Coiled’s approach is to scrape your local environment and replicate it as faithfully as is possible in your cloud resources. For example…

  • If you want to install a new library on your cloud machines, then just pip install that library on your laptop

  • If you want to get access to an AWS resource, then just add a credentials file to your ~/.aws/ directory on your computer

and so on. This allows Coiled users to rely on existing approaches and not have to learn new workflows.

For more information, see Manage Software.

Free for modest use

Coiled is free to use for your first 10,000 CPU-hours per month (although you still have to pay your cloud provider). Beyond 10,000 CPU-hours Coiled nominally costs $0.05 per CPU-hour, although this tends to get negotiated down as users commit to more usage.

10,000 CPU hours each month is a lot, and most users don’t pay any money. Even users with larger workloads tend not to pay money for their first few months while they ramp up. We have this policy for three reasons:

  1. We want Coiled to be easy to try out

  2. We’re really only looking to have commercial conversations in the $1000 or greater range. We have no desire to charge individuals $50.

  3. This is a simple way for Coiled to support academic, individual, and hobbyist developers

For more information, see Manage Costs.

previous

Coiled Documentation

next

Best Practices