Analytics#
Measurement is the foundation of performance.
Coiled Analytics lets you track Dask usage wherever Dask is run. See the installation instructions to get started.
Motivation#
When running computations we often ask ourselves questions like the following:
Did my computation finish?
Did any exceptions occur?
How much did that cost me?
What is taking most of the time?
Why is that cluster still running?
Experienced users know that Dask presents answers to these questions visually through the Dask dashboard. However, the Dask dashboard only tracks the real-time performance of a single Dask cluster. Coiled extends Dask by tracking many Dask clusters across many users and storing those results over time for later analysis. Coiled analytics provides a team-wide view of all clusters over time.
Analytics Overview#
You can view analytics from https://cloud.coiled.io/<your-account-name>/analytics,
or by selecting Analytics from the left navigation pane after you login.
You’ll see three different views of usage statistics for your Coiled account:
Overall account statistics: Shows total resources used since Coiled account creation. Includes cluster time, compute time, number of tasks and workers, and data processed. Cluster and compute time both measure the number of core hours, where cluster time includes idleness and compute time measure utilization.
Cluster compute usage by account members: A heat-map of compute time for account members over time.
Clusters: Detailed cluster statistics for each cluster created. You can filter clusters by date by selecting a date in the above view.
You can click on a row to see more detailed performance tracking for a specific cluster including computations, exceptions, and cost per operation.
What information does Coiled Track?#
Coiled tracks aggregate information about cluster activity including the following (see Data Privacy).
Basic level statistics
Number of active workers and worker threads
Amount of used and total memory
Software versions of common libraries
Performance statistics
Task information, including names, numbers, and compute and transfer durations
Profiling, including which functions and lines of code take the most time
Code snippets surrounding the Dask calls
How long has it been since any work was completed
Error tracking
Every user-level exception
Every dask-level exception
User-level tracking
Which user within an account created the cluster
Costs (estimated when run on non-Coiled architecture)
Idleness