API Reference

Python API Reference

coiled.create_software_environment([name, …])

Create a software environment

coiled.create_cluster_configuration(name, …)

Create a cluster configuration

coiled.list_software_environments([account])

List software environments

coiled.list_cluster_configurations([account])

List cluster configurations

coiled.list_clusters([account])

List clusters

coiled.delete_software_environment(name)

Delete a software environment

coiled.delete_cluster_configuration(name)

Delete a cluster configuration

coiled.delete_cluster(name[, account])

Delete a cluster

coiled.Cluster([n_workers, configuration, …])

Create a Dask cluster with Coiled

coiled.install(coiled_uri)

Create a Coiled software environment locally

Software Environments

coiled.create_software_environment(name=None, conda=None, pip=None, container=None, region=None, log_output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, post_build=None, conda_env_name=None)[source]

Create a software environment

Parameters
  • name (Optional[str]) – Name of software environment.

  • conda (Union[list, dict, str, None]) – Specification for packages to install into the software environment using conda. Can be a list of packages, a dictionary, or a path to a conda environment YAML file.

  • pip (Union[list, str, None]) – Packages to install into the software environment using pip. Can be a list of packages or a path to a pip requirements file.

  • container (Optional[str]) – Docker image to use for the software environment. Must be the name of a docker image on Docker hub. Defaults to coiled/default.

  • post_build (Union[list, str, None]) – List of commands or path to a local executable script to run after pip and conda packages have been installed.

  • region (Optional[str]) – Name of the AWS region to create the software environment in (e.g. “us-east-2”). If not specified, will check the coiled.aws.region configuration value.

  • log_output – Stream to output logs to. Defaults to sys.stdout.

  • conda_env_name (Optional[str]) – Name of conda environment to install packages into. Note that this should only be used when specifying a non-default value for container and when the non-default Docker image used expects commands to run in a conda environment not named “base”. Defaults to “base”.

Return type

dict

coiled.list_software_environments(account=None)[source]

List software environments

Parameters

account – Name of the Coiled account to list software environments. If not provided, will use the coiled.account configuration value.

Returns

Dictionary with information about each software environment in the specified account. Keys in the dictionary are names of software environments, while the values contain information about the corresponding software environment.

coiled.delete_software_environment(name)[source]

Delete a software environment

Parameters

name – Name of software environment to delete.

coiled.install(coiled_uri)[source]

Create a Coiled software environment locally

Parameters

coiled_uri (str) –

Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

The “name” portion of (<account>/)<name> can only contain ASCII letters, hyphens and underscores.

Examples

>>> import coiled
>>> coiled.install("coiled/default")
coiled.inspect(name)[source]

View the details of a Coiled software environment

Parameters

name (str) –

Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

The “name” portion of (<account>/)<name> can only contain ASCII letters, hyphens and underscores.

Examples

>>> import coiled
>>> coiled.inspect("coiled/default")

Cluster Configurations

coiled.create_cluster_configuration(name, software, worker_cpu=1, worker_gpu=0, worker_memory='4 GiB', worker_class='dask.distributed.Nanny', worker_options=None, scheduler_cpu=1, scheduler_memory='4 GiB', scheduler_class='dask.distributed.Scheduler', scheduler_options=None)[source]

Create a cluster configuration

Parameters
  • name (str) – Name of cluster configuration. Optionally prefixed with an account name like “myaccount/myname”

  • software (str) –

    Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

    For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

    The “name” portion of (<account>/)<name> can only contain ASCII letters, hyphens and underscores.

  • worker_cpu (int) – Number of CPUs allocated for each worker. Defaults to 1.

  • worker_gpu (int) – Number of GPUs allocated for each worker. Defaults to 0 (no GPU support). Note that this will _always_ allocate GPU-enabled workers, so is expensive.

  • worker_memory (str) – Amount of memory to allocate for each worker. Defaults to 4 GiB.

  • worker_class (str) – Worker class to use. Defaults to “dask.distributed.Nanny”.

  • worker_options (Optional[dict]) – Mapping with keyword arguments to pass to worker_class. Defaults to {}.

  • scheduler_cpu (int) – Number of CPUs allocated for the scheduler. Defaults to 1.

  • scheduler_memory (str) – Amount of memory to allocate for the scheduler. Defaults to 4 GiB.

  • scheduler_class (str) – Scheduler class to use. Defaults to “dask.distributed.Scheduler”.

  • scheduler_options (Optional[dict]) – Mapping with keyword arguments to pass to scheduler_class. Defaults to {}.

See also

dask.utils.parse_bytes()

Return type

None

coiled.list_cluster_configurations(account=None)[source]

List cluster configurations

Parameters

account – Name of the Coiled account to list cluster configurations. If not provided, will use the coiled.account configuration value.

Returns

Dictionary with information about each cluster configuration in the specified account. Keys in the dictionary are names of cluster configurations, while the values contain information about the corresponding cluster configuration.

coiled.delete_cluster_configuration(name)[source]

Delete a cluster configuration

Parameters

name – Name of cluster configuration to delete.

Clusters

class coiled.Cluster(n_workers=4, configuration=None, software=None, worker_cpu=None, worker_gpu=None, worker_memory=None, worker_class=None, worker_options=None, scheduler_cpu=None, scheduler_memory=None, scheduler_class=None, scheduler_options=None, name=None, asynchronous=False, cloud=None, account=None, shutdown_on_close=None, region=None)[source]

Create a Dask cluster with Coiled

Parameters
  • n_workers (int) – Number of workers in this cluster. Defaults to 4.

  • configuration (Optional[str]) – Name of cluster configuration to create cluster from. If not specified, defaults to coiled/default for the current Python version.

  • name (Optional[str]) – Name to use for identifying this cluster. Defaults to None.

  • software (Optional[str]) –

    Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

    For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

    The “name” portion of (<account>/)<name> can only contain ASCII letters, hyphens and underscores.

  • worker_cpu (Optional[int]) – Number of CPUs allocated for each worker. Defaults to 2.

  • worker_gpu (Optional[int]) – Number of GPUs allocated for each worker. Defaults to 0 (no GPU support). Note that this will _always_ allocate GPU-enabled workers, so is expensive.

  • worker_memory (Optional[str]) – Amount of memory to allocate for each worker. Defaults to 8 GiB.

  • worker_class (Optional[str]) – Worker class to use. Defaults to “dask.distributed.Nanny”.

  • worker_options (Optional[dict]) – Mapping with keyword arguments to pass to worker_class. Defaults to {}.

  • scheduler_cpu (Optional[int]) – Number of CPUs allocated for the scheduler. Defaults to 1.

  • scheduler_memory (Optional[str]) – Amount of memory to allocate for the scheduler. Defaults to 4 GiB.

  • scheduler_class (Optional[str]) – Scheduler class to use. Defaults to “dask.distributed.Scheduler”.

  • scheduler_options (Optional[dict]) – Mapping with keyword arguments to pass to scheduler_class. Defaults to {}.

  • asynchronous (bool) – Set to True if using this Cloud within async/await functions or within Tornado gen.coroutines. Otherwise this should remain False for normal use. Default is False.

  • cloud (Optional[Cloud]) – Cloud object to use for interacting with Coiled.

  • account (Optional[str]) – Name of Coiled account to use. If not provided, will default to the user account for the cloud object being used.

  • shutdown_on_close – Whether or not to shut down the cluster when it finishes. Defaults to True, unless name points to an existing cluster.

  • region (Optional[str]) – Name of the AWS region to create the cluster in (e.g. “us-east-2”). If not specified, will check the coiled.aws.region configuration value.

get_logs(scheduler=True, workers=True)[source]

Return logs for the scheduler and workers :type scheduler: bool :param scheduler: Whether or not to collect logs for the scheduler :type scheduler: boolean :type workers: bool :param workers: Whether or not to collect logs for the workers :type workers: boolean

Returns

logs – A dictionary of logs, with one item for the scheduler and one for the workers

Return type

Dict[str]

property loop
scale(n)[source]

Scale cluster to n workers

Parameters

n (int) – Number of workers to scale cluster size to.

Return type

Tuple[List[Dict], List[Dict]]

coiled.list_clusters(account=None)[source]

List clusters

Parameters

account – Name of the Coiled account to list clusters. If not provided, will use the coiled.account configuration value.

Returns

Dictionary with information about each cluster in the specified account. Keys in the dictionary are names of clusters, while the values contain information about the corresponding cluster.

coiled.delete_cluster(name, account=None)[source]

Delete a cluster

Parameters

name (str) – Name of cluster to delete.

Command Line API Reference

coiled login

Configure your Coiled account credentials

coiled login [OPTIONS]

Options

-s, --server <server>

Coiled server to use

-t, --token <token>

Coiled user token

coiled install

Create Coiled conda software environment locally

coiled install [OPTIONS] COILED_URI

Arguments

COILED_URI

Required argument

coiled env create

Create a Coiled software environment

coiled env create [OPTIONS]

Options

-n, --name <name>

Name of software environment.

--container <container>

Base docker image to use.

--conda <conda>

Conda environment file.

--pip <pip>

Pip requirements file.

--post-build <post_build>

Post-build script.

coiled env delete

Delete a Coiled software environment

coiled env delete [OPTIONS] NAME

Arguments

NAME

Required argument

coiled env list

List the Coiled software environments in an account

coiled env list [OPTIONS] [ACCOUNT]

Arguments

ACCOUNT

Optional argument

coiled env inspect

View the details of a Coiled software environment

coiled env inspect [OPTIONS] NAME

Arguments

NAME

Required argument