Machine Learning#

Dask is tightly integrated with common Python machine learning libraries like Scikit-Learn, XGBoost, LightGBM, Optuna, and PyTorch. This makes it easy to train models at scale, and run predictions on large scale data.

import xgboost as xgb
import dask.array as da
import coiled

cluster = coiled.Cluster(n_workers=20)
client = cluster.get_client()

# X and y must be Dask DataFrames or Arrays
# example with 1e5 observations and 20 features
X = da.random.random(size=(1e5, 20), chunks=(1000, 20))
y = da.random.random(size=(1e5, 1), chunks=(1000, 1))

dtrain = xgb.dask.DaskDMatrix(client, X, y)

output = xgb.dask.train(
    {"tree_method": "hist", "objective": "reg:squarederror"},
    evals=[(dtrain, "train")],

For more in-depth machine learning examples consider the following: