https://github.com/holoviz/datashader/raw/main/doc/_static/logo_horizontal.svg

Visualize 1,000,000,000 Points#

In this notebook we process and interactively visualize over one billion points. You can download this jupyter notebook to follow along.

Before you start#

You’ll first need to install the necessary packages. For the purposes of this example, we’ll do this in a new virtual environment, but you could also install them in whatever environment you’re already using for your project.

conda create -n coiled-datashader -c conda-forge python=3.10 coiled dask s3fs pyarrow datashader hvplot jupyter_bokeh
conda activate coiled-datashader

You also could use pip for everything, or any other package manager you prefer; conda isn’t required.

When you later create a Coiled cluster, your local coiled-datashader environment will be automatically replicated on your cluster.

Create Cluster#

import coiled

cluster = coiled.Cluster(
    n_workers=20,
    region="us-east-2",  # Start workers in same region as data to minimize costs
) 

client = cluster.get_client()













Load and prep data#


To start, let’s use Dask DataFrame’s read_parquet functionality to load our dataset from S3.






import dask.dataframe as dd

df = dd.read_parquet("s3://coiled-datasets/dask-book/nyc-tlc/2009-2013/")
df.head()













  
    

      
      vendor_id
      pickup_datetime
      dropoff_datetime
      passenger_count
      trip_distance
      pickup_longitude
      pickup_latitude
      dropoff_longitude
      dropoff_latitude
      payment_type
      fare_amount
      surcharge
      tip_amount
      tolls_amount
      total_amount
    

  
  
    

      0
      VTS
      2009-01-04 02:52:00
      2009-01-04 03:02:00
      1
      2.63
      -73.991957
      40.721567
      -73.993803
      40.695922
      CASH
      8.9
      0.5
      0.00
      0.0
      9.40
    

    

      1
      VTS
      2009-01-04 03:31:00
      2009-01-04 03:38:00
      3
      4.55
      -73.982102
      40.736290
      -73.955850
      40.768030
      Credit
      12.1
      0.5
      2.00
      0.0
      14.60
    

    

      2
      VTS
      2009-01-03 15:43:00
      2009-01-03 15:57:00
      5
      10.35
      -74.002587
      40.739748
      -73.869983
      40.770225
      Credit
      23.7
      0.0
      4.74
      0.0
      28.44
    

    

      3
      DDS
      2009-01-01 20:52:58
      2009-01-01 21:14:00
      1
      5.00
      -73.974267
      40.790955
      -73.996558
      40.731849
      CREDIT
      14.9
      0.5
      3.05
      0.0
      18.45
    

    

      4
      DDS
      2009-01-24 16:18:23
      2009-01-24 16:24:56
      1
      0.40
      -74.001580
      40.719382
      -74.008378
      40.720350
      CASH
      3.7
      0.0
      0.00
      0.0
      3.70
    

  







Next, we’ll select ride data that’s near NYC.






df = df.loc[
    (df.dropoff_longitude > -74.1) & (df.dropoff_longitude < -73.7) & 
    (df.dropoff_latitude > 40.6) & (df.dropoff_latitude < 40.9) &
    (df.pickup_longitude > -74.1) & (df.pickup_longitude < -73.7) &
    (df.pickup_latitude > 40.6) & (df.pickup_latitude < 40.9)
]









We now have all the information we need, but still need to format our data so it can be easily visualized. We do this by splitting each ride into two rows, one with the pickup location and one with the dropoff location.






# Dropoff locations
df_drop = df[["dropoff_longitude", "dropoff_latitude"]]
df_drop["journey_type"] = "dropoff"
df_drop = df_drop.rename(columns={'dropoff_longitude': 'lon', 'dropoff_latitude': 'lat'})

# Pickup locations
df_pick = df[["pickup_longitude", "pickup_latitude"]]
df_pick["journey_type"] = "pickup"
df_pick = df_pick.rename(columns={'pickup_longitude': 'lon', 'pickup_latitude': 'lat'})

# Combine into single DataFrame
df = dd.concat([df_drop, df_pick])

# Convert `journey_type` column to categorical (needed for plotting below)
df = df.astype({"journey_type": "category"})
df["journey_type"] = df["journey_type"].cat.set_categories(["dropoff", "pickup"])









Finally, we repartition for efficient follow-up computations and load the dataset onto our cluster with persist().






%%time

df = df.repartition(partition_size="256 MiB").persist()
print(f"Number of records: {len(df)}")









Number of records: 1693136554
CPU times: user 3.63 s, sys: 580 ms, total: 4.21 s
Wall time: 1min 26s









We’re now ready to visualize this data.






Visualize#






import datashader
import holoviews as hv
import hvplot.dask
hv.extension('bokeh')

df.hvplot.scatter(
    x="lon", 
    y="lat", 
    aggregator=datashader.by("journey_type"), 
    datashade=True, 
    cnorm="eq_hist",
    frame_width=700, 
    aspect=1.33, 
    color_key={"pickup": "#EF1561", "dropoff": "#1F5AFF"},
)










  










  
  








  












Conclusion#


Dask DataFrame and Datashader together make it easy to visualize very large scale data. Coiled makes it easy to scale up computing resources in the cloud.


If you have a cloud account it’s easy to try this out yourself.  The data is public and this computation costs less than $0.20 in AWS cloud costs.  You can sign up under the Coiled free tier here (setup takes a couple minutes).








   



                

              
              
              
              
              
                
              
            

            
            
              
                



  



     On this page