CLI Jobs#

Run simple commands and scripts in the cloud

Sometimes you just want to run a simple script on a VM in the cloud. This is common for different reasons:

  1. You want to run close to your data

  2. You want a GPU

  3. You want a bigger machine

  4. You want to run the script many times while scaling out

This is easy with the coiled run command. Coiled will run your command on the infrastructure of your choosing.

coiled run python my_script.py

coiled run bash my_bash_script.sh

coiled run --gpu python pytorch_model_train.py

coiled run --vm-type m6i.32xlarge python parallel_script.py

for i in {1..10};
    do coiled run python my_script.py --param $i &
done

Use Cases#

Features#

Coiled Run benefits from the standard Coiled features:

  • Easy to use API

  • Creates and manages VMs for you in the cloud

  • Automatically synchronizes your local software and credentials

  • Gives access to any cloud hardware (like GPUs) in any region

Hardware#

It’s easy to customize the hardware and software resources a run needs. You can select any VM type available on your cloud (see VM Size and Type). For example:

# 128 Cores and 512 GB of RAM
coiled run --vm-type m6i.32xlarge python myscript.py

# A single T4 NVIDIA GPU
coiled run --vm-type g4dn.xlarge python train.py

Software for Coiled Run will work the same as with Coiled clusters. By default, Coiled will automatically synchronize your local software environment. This works well in most cases, but you can also specify an manual software environment or Docker image:

# Manual software environment
coiled run --software my-env python my_script.py

# Docker image
coiled run --container nvcr.io/nvidia/rapidsai/base:23.08-cuda11.8-py3.10 python rapids_script.py

Files#

Any files listed in your command will automatically be uploaded to the cloud for your run to complete. If additional files are needed (for example, your Python script imports code from another local Python module) you can use the --file option to ensure additional files are also included with your run.

coiled run --file utils.py python my_script.py

These files will be available in the working directory, unless you are using both --file and --sync together (in which case these the files will be in the parent of the working directory and --sync’s files are in the working directory).

Feedback#

Coiled Run is under active development. We highly value feedback from users and encourage you to play with coiled run and then let us know about your experience on this issue tracker or by reaching out to support@coiled.io.

API#

coiled run#

Run a command on the cloud.

coiled run [OPTIONS] [COMMAND]...

Options

--name <name>#

Run name. This identifier controls whether coiled run invocations are dispatched to the same cloud VM or not. Use the same name to run multiple commands on the same VM. Defaults to a unique name with no VM reuse.

--account, --workspace <account>#

Coiled workspace (uses default workspace if not specified). Note: --account is deprecated, please use --workspace instead.

--software <software>#

Software environment name to use. If neither software nor container is specified, all the currently-installed Python packages are replicated on the VM using package sync.

--container <container>#

Container image to use. If neither software nor container is specified, all the currently-installed Python packages are replicated on the VM using package sync.

--vm-type <vm_type>#

VM type to use. Specify multiple times to provide multiple options.

--gpu#

Have a GPU available.

--region <region>#

The cloud provider region in which to run the notebook.

--disk-size <disk_size>#

Use larger-than-default disk on VM, specified in GiB.

--keepalive <keepalive>#

Keep your VM running for the specified time, even after your command completes. In seconds (--keepalive 60) unless you specify units (--keepalive 3m for 3 minutes).Default to shutdown immediately after the command finishes.

-f, --file <file>#

Local files required to run command. Can be either individual file or entire directory. Multiple values can be specified, such as --file foo.txt --file my-subdir/.

-e, --env <env>#

Environment variables securely transmitted to run command environment. Format is KEY=val, multiple vars can be set with separate --env for each.

--subdomain <subdomain>#

Custom subdomain for the VM hostname.

--allow-ssh-from <allow_ssh_from>#

IP address or CIDR from which connections to port 22 (SSH) are open; can also be specified as ‘everyone’ (0.0.0.0/0) or ‘me’ (automatically determines public IP detected for your local client). Note that coiled run relies on SSH connection for executing commands on VM.

--port <port>#

Open extra ports in network firewall for inbound connections (multiple ports can be set with separate --port for each).

-it, --interactive#

Open an interactive session, e.g., coiled run --interactive bash or coiled run --interactive python.

--detach#

Start the run in the background, don’t wait for the results.

--sync#

Sync files between local working directory and /scratch/synced.

--root#

Act as root in Docker container.

--no-credential-forwarding#

Disable automatic forwarding of local cloud credentials; use if you want your code to use credentials attached to the VM (e.g., AWS Instance Profile).

--forward-gcp-adc#

Forward long-lived Google Cloud Application Default Credentials to VM for data access.

-t, --tag <tag>#

Tags. Format is KEY=val, multiple vars can be set with separate --tag for each.

--sync-ignore <sync_ignore>#

Paths to not sync when using --sync.

--mount-bucket <mount_bucket>#

S3 or GCS bucket(s) to mount as volumes.

--package-sync-strict#

Require exact package version matches when using package sync.

--package-sync-conda-extras <package_sync_conda_extras>#

A list of conda package names (available on conda-forge) to include in the environment that are not in your local environment.

Arguments

COMMAND#

Optional argument(s)

On this page