API Reference

Python API Reference

coiled.create_software_environment([name, …])

Create a software environment

coiled.create_cluster_configuration(name, …)

Create a cluster configuration

coiled.create_notebook(name[, container, …])

Create a notebook

coiled.create_job_configuration(name, …[, …])

Create a job configuration

coiled.list_software_environments([account])

List software environments

coiled.list_cluster_configurations([account])

List cluster configurations

coiled.list_clusters([account])

List clusters

coiled.list_job_configurations([account])

List job configurations

coiled.list_core_usage([json, account])

Get a list of used cores.

coiled.list_local_versions([json])

Get information about local versions.

coiled.delete_software_environment(name)

Delete a software environment

coiled.delete_cluster_configuration(name)

Delete a cluster configuration

coiled.delete_cluster(name[, account])

Delete a cluster

coiled.delete_cluster_configuration(name)

Delete a cluster configuration

coiled.Cluster([n_workers, configuration, …])

Create a Dask cluster with Coiled

coiled.cluster_cost_estimate([n_workers, …])

Estimate the cluster hourly cost

coiled.install(name)

Create a Coiled software environment locally

coiled.start_job(configuration[, account, …])

Start a job

coiled.stop_job([name])

Stop a running job

coiled.list_jobs([account])

List jobs

coiled.info([json, account])

Get information about Coiled.

Software Environments

coiled.create_software_environment(name=None, conda=None, pip=None, container=None, log_output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, post_build=None, conda_env_name=None, backend_options=None, private=False)[source]

Create a software environment

Parameters

  • name (Optional[str]) – Name of software environment. Name can’t contain uppercase letters.

  • conda (Union[list, dict, str, None]) – Specification for packages to install into the software environment using conda. Can be a list of packages, a dictionary, or a path to a conda environment YAML file.

  • pip (Union[list, str, None]) – Packages to install into the software environment using pip. Can be a list of packages or a path to a pip requirements file.

  • container (Optional[str]) – Docker image to use for the software environment. Must be the name of a docker image on Docker hub. Defaults to coiled/default.

  • post_build (Union[list, str, None]) – List of commands or path to a local executable script to run after pip and conda packages have been installed.

  • log_output – Stream to output logs to. Defaults to sys.stdout.

  • conda_env_name (Optional[str]) – Name of conda environment to install packages into. Note that this should only be used when specifying a non-default value for container and when the non-default Docker image used expects commands to run in a conda environment not named “coiled”. Defaults to “coiled”.

  • backend_options (Optional[Dict]) – Dictionary of backend specific options (e.g. {'region': 'us-east-2'}). Any options specified with this keyword argument will take precedence over those stored in the coiled.backend-options cofiguration value.

  • private (bool) – Whether this software environment is private or public. Defaults to False

Return type

dict

coiled.list_software_environments(account=None)[source]

List software environments

Parameters

account – Name of the Coiled account to list software environments. If not provided, will use the coiled.account configuration value.

Returns

Dictionary with information about each software environment in the specified account. Keys in the dictionary are names of software environments, while the values contain information about the corresponding software environment.

coiled.delete_software_environment(name)[source]

Delete a software environment

Parameters

name – Name of software environment to delete.

coiled.install(name)[source]

Create a Coiled software environment locally

Parameters

name –

Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

The “name” portion of (<account>/)<name> can only contain ASCII letters, hyphens and underscores.

Examples

>>> import coiled
>>> coiled.install("coiled/default")
coiled.inspect(name)[source]

View the details of a Coiled software environment

Parameters

name (str) –

Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

The “name” portion of (<account>/)<name> can only contain ASCII letters, hyphens and underscores.

Examples

>>> import coiled
>>> coiled.inspect("coiled/default")

Cluster Configurations

coiled.create_cluster_configuration(name, software, worker_cpu=1, worker_gpu=0, worker_memory='4 GiB', worker_class='dask.distributed.Nanny', worker_options=None, scheduler_cpu=1, scheduler_memory='4 GiB', scheduler_class='dask.distributed.Scheduler', scheduler_options=None, private=False)[source]

Create a cluster configuration

Parameters

  • name (str) – Name of cluster configuration. Optionally prefixed with an account name like “myaccount/myname”

  • software (str) –

    Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

    For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

    The “name” portion of (<account>/)<name> can only contain ASCII letters, hyphens and underscores.

  • worker_cpu (int) – Number of CPUs allocated for each worker. Defaults to 1.

  • worker_gpu (int) – Number of GPUs allocated for each worker. Defaults to 0 (no GPU support). Note that this will _always_ allocate GPU-enabled workers, so is expensive.

  • worker_memory (str) – Amount of memory to allocate for each worker. Defaults to 4 GiB.

  • worker_class (str) – Worker class to use. Defaults to “dask.distributed.Nanny”.

  • worker_options (Optional[dict]) – Mapping with keyword arguments to pass to worker_class. Defaults to {}.

  • scheduler_cpu (int) – Number of CPUs allocated for the scheduler. Defaults to 1.

  • scheduler_memory (str) – Amount of memory to allocate for the scheduler. Defaults to 4 GiB.

  • scheduler_class (str) – Scheduler class to use. Defaults to “dask.distributed.Scheduler”.

  • scheduler_options (Optional[dict]) – Mapping with keyword arguments to pass to scheduler_class. Defaults to {}.

  • private (bool) – Whether this cluster configuration is private or public. Defaults to False

See also

dask.utils.parse_bytes

Return type

None

coiled.list_cluster_configurations(account=None)[source]

List cluster configurations

Parameters

account – Name of the Coiled account to list cluster configurations. If not provided, will use the coiled.account configuration value.

Returns

Dictionary with information about each cluster configuration in the specified account. Keys in the dictionary are names of cluster configurations, while the values contain information about the corresponding cluster configuration.

coiled.delete_cluster_configuration(name)[source]

Delete a cluster configuration

Parameters

name – Name of cluster configuration to delete.

Clusters

class coiled.Cluster(n_workers=4, configuration=None, software=None, worker_cpu=None, worker_gpu=None, worker_memory=None, worker_class=None, worker_options=None, scheduler_cpu=None, scheduler_memory=None, scheduler_class=None, scheduler_options=None, name=None, asynchronous=False, cloud=None, account=None, shutdown_on_close=None, backend_options=None, credentials='account', timeout=None)[source]

Create a Dask cluster with Coiled

Parameters

  • n_workers (int) – Number of workers in this cluster. Defaults to 4.

  • configuration (Optional[str]) – Name of cluster configuration to create cluster from. If not specified, defaults to coiled/default for the current Python version.

  • name (Optional[str]) – Name to use for identifying this cluster. Defaults to None.

  • software (Optional[str]) –

    Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

    For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

    The “name” portion of (<account>/)<name> can only contain ASCII letters, hyphens and underscores.

  • worker_cpu (Optional[int]) – Number of CPUs allocated for each worker. Defaults to 2.

  • worker_gpu (Optional[int]) – Number of GPUs allocated for each worker. Defaults to 0 (no GPU support). Note that this will _always_ allocate GPU-enabled workers, so is expensive.

  • worker_memory (Optional[str]) – Amount of memory to allocate for each worker. Defaults to 8 GiB.

  • worker_class (Optional[str]) – Worker class to use. Defaults to “dask.distributed.Nanny”.

  • worker_options (Optional[dict]) – Mapping with keyword arguments to pass to worker_class. Defaults to {}.

  • scheduler_cpu (Optional[int]) – Number of CPUs allocated for the scheduler. Defaults to 1.

  • scheduler_memory (Optional[str]) – Amount of memory to allocate for the scheduler. Defaults to 4 GiB.

  • scheduler_class (Optional[str]) – Scheduler class to use. Defaults to “dask.distributed.Scheduler”.

  • scheduler_options (Optional[dict]) – Mapping with keyword arguments to pass to scheduler_class. Defaults to {}.

  • asynchronous (bool) – Set to True if using this Cloud within async/await functions or within Tornado gen.coroutines. Otherwise this should remain False for normal use. Default is False.

  • cloud (Optional[Cloud]) – Cloud object to use for interacting with Coiled.

  • account (Optional[str]) – Name of Coiled account to use. If not provided, will default to the user account for the cloud object being used.

  • shutdown_on_close – Whether or not to shut down the cluster when it finishes. Defaults to True, unless name points to an existing cluster.

  • backend_options (Optional[Dict]) – Dictionary of backend specific options (e.g. {'region': 'us-east-2'}). Any options specified with this keyword argument will take precedence over those stored in the coiled.backend-options cofiguration value.

  • credentials (Optional[str]) – Which credentials to use for Dask operations and forward to Dask clusters – options are “account”, “local”, or “none”. The default behavior is to prefer credentials associated with the Coiled Account, if available, then try to use local credentials, if available. NOTE: credential handling currently only works with AWS credentials.

  • timeout (Optional[int]) – Timeout in seconds to wait for a cluster to start, will use default_cluster_timeout set on parent Cloud by default.

get_logs(scheduler=True, workers=True)[source]

Return logs for the scheduler and workers :type scheduler: bool :param scheduler: Whether or not to collect logs for the scheduler :type scheduler: boolean :type workers: bool :param workers: Whether or not to collect logs for the workers :type workers: boolean

Returns

logs – A dictionary of logs, with one item for the scheduler and one for the workers

Return type

Dict[str]

property loop
scale(n)[source]

Scale cluster to n workers

Parameters

n (int) – Number of workers to scale cluster size to.

Return type

Tuple[List[Dict], List[Dict]]

coiled.list_clusters(account=None)[source]

List clusters

Parameters

account – Name of the Coiled account to list clusters. If not provided, will use the coiled.account configuration value.

Returns

Dictionary with information about each cluster in the specified account. Keys in the dictionary are names of clusters, while the values contain information about the corresponding cluster.

coiled.delete_cluster(name, account=None)[source]

Delete a cluster

Parameters

name (str) – Name of cluster to delete.

coiled.cluster_cost_estimate(n_workers=4, configuration=None, worker_cpu=None, worker_gpu=None, worker_memory=None, scheduler_cpu=None, scheduler_memory=None, backend_options=None)[source]

Estimate the cluster hourly cost

Parameters

  • n_workers (int) – Number of workers in this cluster. Defaults to 4.

  • configuration (Optional[str]) – Name of cluster configuration to create cluster from. If not specified, defaults to coiled/default for the current Python version.

  • worker_cpu (Optional[int]) – Number of CPUs allocated for each worker. Defaults to 1.

  • worker_gpu (Optional[int]) – Number of GPUs allocated for each worker. Defaults to 0 (no GPU support). Note that this will _always_ allocate GPU-enabled workers, so is expensive.

  • worker_memory (Union[str, int, None]) – Amount of memory to allocate for each worker. Defaults to 4 GiB.

  • scheduler_cpu (Optional[int]) – Number of CPUs allocated for the scheduler. Defaults to 1.

  • scheduler_memory (Union[str, int, None]) – Amount of memory to allocate for the scheduler. Defaults to 4 GiB.

  • backend_options (Optional[Dict]) – Dictionary of backend specific options (e.g. {'region': 'us-east-2'}). Any options specified with this keyword argument will take precedence over those stored in the coiled.backend-options cofiguration value.

See also

dask.utils.parse_bytes

Return type

str

coiled.list_core_usage(json=False, account=None)[source]

Get a list of used cores.

Returns a table that shows the limit of cores that the user can use and a breakdown of the core usage split up between account, user, clusters and jobs.

Parameters

  • account (Optional[str]) – Name of the Coiled account to list core usage. If not provided, will use the coiled.account configuration value.

  • json (bool) – If set to True, it will return this list in json format instead of a table.

Notebooks

coiled.create_notebook(name, container='coiled/notebook:latest', conda=None, pip=None, post_build=None, cpu=1, gpu=0, memory='4 GiB', files=None, command=('/bin/bash', 'start.sh', 'jupyter', 'lab'), ports=(8888), description=None)[source]

Create a notebook

This allows you to create a hosted notebook which is runnable on https://cloud.coiled.io.

Parameters

  • name – Name of the notebook.

  • container – Docker image to use for the notebook’s software environment. Defaults to coiled/notebook:latest. Note that when using a non-default container, you should check whether the default command is compatible with the container being used and, if not, update the command such that it launches a Jupyter session within the container.

  • conda – Specification for packages to install into the notebook’s software environment using conda. Can be a list of packages, a dictionary, or a path to a conda environment YAML file.

  • pip – Packages to install into the notebook’s software environment using pip. Can be a list of packages or a path to a pip requirements file.

  • post_build – List of commands or path to a local executable script to run after pip and conda packages have been installed.

  • cpu (int) – Number of CPUs allocated for the notebook session. Defaults to 1.

  • gpu (int) – Number of GPUs allocated for the notebook session. Defaults to 0 (no GPU support).

  • memory (str) – Amount of memory to allocated for the notebook session. Defaults to 4 GiB.

  • files – List of local files to upload.

  • command – Command to run. If not specified, defaults to running JupyterLab.

  • ports – List of ports to expose. If not specified, defaults to (8888,)

  • description – Description for notebook.

Jobs

coiled.create_job_configuration(name, command, software, cpu=1, gpu=0, memory='4 GiB', ports=None, files=None, description=None)[source]

Create a job configuration

Parameters

  • name (str) – Name of the job configuration Optionally prefixed with an account name like “myaccount/myname”

  • command (List[str]) – Command to run this job. Defaults to None.

  • software (str) –

    Identifier of the software environment to use, in the format (<account>/)<name>. If the software environment is owned by the same account as that passed into “account”, the (<account>/) prefix is optional.

    For example, suppose your account is “wondercorp”, but your friends at “friendlycorp” have an environment named “xgboost” that you want to use; you can specify this with “friendlycorp/xgboost”. If you simply entered “xgboost”, this is shorthand for “wondercorp/xgboost”.

  • cpu (int) – Number of CPUs allocated for this job. Defaults to 1.

  • gpu (int) – Number of GPUs allocated for this job. Defaults to 0.

  • memory (str) – Ammount of memory allocated for this job. Defaults to 4 GiB.

  • ports (Optional[List]) – List of ports to expose. Defaults to None.

  • files (Union[str, List[str], None]) – List of local files to upload.

  • description (Optional[str]) – Description for the job.

Return type

dict

coiled.delete_job_configuration(name)[source]

Delete a job configuration

Parameters

name – Name of job configuration to delete.

coiled.list_job_configurations(account=None)[source]

List job configurations

Parameters

account – Name of the Coiled account to list job configurations. If not provided, will use the coiled.account configuration value.

Returns

Dictionary with information about each job configuration in the specified account. Keys in the dictionary are names of job configurations, while the values contain information about the corresponding job configuration.

coiled.start_job(configuration, account=None, backend_options=None, log_output=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)[source]

Start a job

Parameters

  • configuration (str) – Name of the job configuration to create the job from.

  • account (Optional[str]) – Name of Coiled account to use. If not provided, will default to the user account for the cloud object being used.

  • backend_options (Optional[Dict]) – Dictionary of backend specific options (e.g. {'region': 'us-east-2'}). Any options specified with this keyword argument will take precedence over those stored in the coiled.backend-options cofiguration value

  • log_output – Stream to output logs to. Defaults to sys.stdout.

coiled.stop_job(name=None)[source]

Stop a running job

Parameters

name (Optional[str]) –

Name of job to stop.

You can get a list of running jobs and their names by using the command coiled.list_jobs()

coiled.list_jobs(account=None)[source]

List jobs

Parameters

account – Name of the Coiled account to list jobs. If not provided, will use the coiled.account configuration value.

Returns

Dictionary with information about each job in the specified account. Keys in the dictionary are names of jobs, while the values contain information about the corresponding job.

Information

coiled.list_local_versions(json=False)[source]

Get information about local versions.

Returns the versions of Python, Coiled, Dask and Distributed that are installed locally. This information could be useful when troubleshooting issues.

Parameters

json (bool) – If set to True, it will return this list in json format instead of a table.

coiled.info(json=False, account=None)[source]

Get information about Coiled.

This command will call others to dump information that could help you in troubleshooting issues.

Parameters

  • account (Optional[str]) – Name of the Coiled account to list core usage. If not provided, will use the coiled.account configuration value.

  • json (bool) – If set to True, it will return this list in json format instead of a table.

Command Line API Reference

coiled login

Configure your Coiled account credentials

coiled login [OPTIONS]

Options

-s, --server <server>

Coiled server to use

-t, --token <token>

Coiled user token

--retry, --no-retry

Whether or not to automatically ask for a new token if an invalid token is entered

coiled install

Create Coiled conda software environment locally

coiled install [OPTIONS] NAME

Arguments

NAME

Required argument

coiled env create

Create a Coiled software environment

coiled env create [OPTIONS]

Options

-n, --name <name>

Name of software environment, it must be lowercase.

--container <container>

Base docker image to use.

--conda <conda>

Conda environment file.

--pip <pip>

Pip requirements file.

--post-build <post_build>

Post-build script.

--private

Flag to set software environment private.

coiled env delete

Delete a Coiled software environment

coiled env delete [OPTIONS] NAME

Arguments

NAME

Required argument

coiled env list

List the Coiled software environments in an account

coiled env list [OPTIONS] [ACCOUNT]

Arguments

ACCOUNT

Optional argument

coiled env inspect

View the details of a Coiled software environment

coiled env inspect [OPTIONS] NAME

Arguments

NAME

Required argument

coiled create-kubeconfig

Create a kubeconfig file for connecting Coiled to a Kubernetes cluster.

coiled create-kubeconfig [OPTIONS]

Options

-n, --namespace <namespace>

Namespace to grant Coiled access to.

-o, --output <output>

Path to output kubeconfig file.

Notebooks Examples