Jupyter Notebooks#

Sometimes you just want a big machine in the cloud.

Quickstart#

Replace your local Jupyter command with Coiled:

# jupyter lab            # replace this
coiled notebook start    # with this

This will take 1-2 minutes and open a browser tab connecting to an authenticated Jupyter server running on a VM in your cloud.

If you have Mutagen installed and use the --sync flag, then Coiled will synchronize your local notebook files between your hard drive and the remote machine, giving you a seamless experience.

# synchronize changes between cloud and local
coiled notebook start --sync

Features#

Coiled notebooks benefit from the standard Coiled features:

Package versions synchronize from your environment
Cloud permissions synchronize from your environment
Files synchronize bidirectionally with the --sync keyword (unique to notebooks)
Runs on AWS, GCP, or Azure on any region on any hardware
Robust logging, cost controls, and metrics collection
Without additional cloud infrastructure (like Kubernetes)

Tip

You can use pip install "coiled[notebook]" to install coiled alongside additional notebooks dependencies like jupyterlab.

Configuration#

Manage Hardware#

You can select any VM type available on your cloud (see VM Size and Type). For example:

# 128 Cores and 512 GB of RAM
coiled notebook start --vm-type m6i.32xlarge

# A single T4 NVIDIA GPU
coiled notebook start --vm-type g4dn.xlarge

Manage Software#

If you do not select a software environment with the --software flag then Coiled will automatically use Package Sync to reproduce your local versions of software packages on the Jupyter server.

File Synchronization#

The --sync flag will optionally synchronize the files you have locally to the cloud machine, and copy any changes made on the cloud machine back to your local hard drive.

This gives you the experience of just working on your local file system.

To do this, you will need to install mutagen. On macOS, it is available on Homebrew:

brew install mutagen-io/mutagen/mutagen

And use the --sync flag when starting your notebook:

coiled notebook start --sync

Then you will get live synchronization between the remote server (/scratch/synced) and your local machine. This allows you to edit files on either machine and have the edits quickly show up on the other.

Limitations#

Files larger than 1 GiB are not synced. If you would like to work in an entire directory that isn’t synced, you can create one in another directory, for example at /scratch/not_synced/.

API#

coiled notebook start#

Launch or re-open a notebook session, with optional file syncing.

If a notebook session with the same name already exists, it’s not re-created. If file sync was initially not enabled, running coiled notebook start --sync will begin file sync without re-launching the notebook.

coiled notebook start [OPTIONS]

Options

--name <name>#: Cluster name. If not given, defaults to a hash based on current working directory.

--account, --workspace <account>#: Coiled workspace (uses default workspace if not specified). Note: –account is deprecated, please use –workspace instead.

--sync#: Sync the working directory with the filesystem on the notebook. Requires mutagen.

--allow-sync-from <allow_sync_from>#: IP address or CIDR from which connections to port 22 (SSH) are open; can also be specified as ‘everyone’ (0.0.0.0/0) or ‘me’ (automatically determines public IP detected for your local client). Note that –sync relies on SSH connection.

--software <software>#: Software environment name to use. If neither software nor container is specified, all the currently-installed Python packages are replicated on the VM using package sync.

--container <container>#: Container image to use. If neither software nor container is specified, all the currently-installed Python packages are replicated on the VM using package sync.

--vm-type <vm_type>#: VM type to use. Specify multiple times to provide multiple options.

--cpu <cpu>#: Number of CPUs requested for this notebook.

--memory <memory>#: Amount of memory for this notebook, Coiled will use a +/-10% buffer from the memory you specify.

--gpu#: Use GPU notebook server.

--disk-size <disk_size>#: Use larger-than-default disk on VM, specified in GiB.

--region <region>#: The cloud provider region in which to run the notebook.

--open#: Whether to open the notebook in the default browser once it’s launched.

--block, --no-block#: Whether to block while the notebook is running.

--include-vcs#: Include version control files when syncing (e.g., .git).

--idle-timeout <idle_timeout>#: Period of idleness after which to automatically shut down the notebook, e.g. ‘20 minutes’ or ‘1 hour’ or ‘10 days’ (default is ‘4 hours’). The notebook is considered active (not idle) if a browser is connected, even if no code is running.

--allow-ingress-from <allow_ingress_from>#: Control the CIDR from which cluster firewall allows ingress to scheduler; by default this is open to any source address (0.0.0.0/0). You can specify CIDR, or ‘me’ for just your IP address.

-e, --env <env>#: Environment variables securely transmitted to run command environment. Format is KEY=val, multiple vars can be set with separate --env for each.

-t, --tag <tag>#: Tags. Format is KEY=val, multiple vars can be set with separate --tag for each.

--sync-ignore <sync_ignore>#: Paths to not sync when using --sync.

--mount-bucket <mount_bucket>#: S3 or GCS bucket(s) to mount as volumes.

--host-setup-script <host_setup_script>#: Path to local script which will be run on each VM during setup process.

coiled notebook stop#

Shut down a notebook session

coiled notebook stop [OPTIONS] NAME

Options

--account, --workspace <account>#: Coiled workspace (uses default workspace if not specified). Note: –account is deprecated, please use –workspace instead.

Arguments

NAME#: Required argument