Building a scientific Python stack with Coiled#

At the heart of most scientific computing stacks are a handful of core libraries that provide general-purpose numerical methods, domain-specific analyses, and the machinery for machine learning.

NumPy and SciPy provide fundamental array operations, linear algebra, and scientific routines. Pandas and Xarray handle structured/tabular and labeled multi-dimensional data, respectively. Matplotlib and Plotly enable visualization. Scikit-learn, PyTorch, and TensorFlow power machine learning and deep learning workflows. Dask scales computations from a single laptop to a distributed cluster.

In many cases, custom libraries or domain-specific frameworks are also part of the mix, especially for specialized fields like geospatial analysis, genomics, or computational physics.

Traditionally, these tools are used on local machines, possibly leveraging multi-core CPUs or single-GPU setups. As dataset sizes grow or as computational needs (such as parameter sweeps, simulations, or parallel data processing) increase, teams quickly reach the limits of what a single piece of hardware can deliver.

The challenge: Scaling beyond a workstation#

Scaling up scientific Python workflows introduces technical barriers that often distract from the actual scientific or analytical task. Installing and synchronizing software environments across different machines creates consistency challenges. Managing cluster orchestration—starting, stopping, and monitoring resources—requires specialized expertise. Teams must troubleshoot network, storage, and reliability issues while dealing with cost controls and resource optimization to avoid unexpected bills.

For a data science team in biomedical research, e.g., these challenges might manifest when analyzing terabytes of RNA sequencing data. A local workstation might handle preliminary analyses, but running full pipelines, bootstrapped permutations, or deep learning model training on such massive datasets is impractical without access to scalable compute resources.

The transition to distributed computing is often daunting. Traditional solutions require extensive setup through frameworks like MPI or Kubernetes and frequent code rewrites to adapt to new APIs or cluster environments. As a result, many teams either over-invest in local infrastructure with high up-front costs or underutilize the cloud due to operational friction.

Coiled: A platform for scalable Python computing on the cloud#

Coiled addresses these obstacles by providing a platform that lets teams run their Python code—unchanged—on the cloud, at any scale. The platform enables frictionless development where Python developers can continue using their existing tooling and libraries, with no need to rewrite code for new environments.

Coiled provides efficient, on-demand cloud computing by provisioning the right resources in the right cloud regions, running jobs, and tearing down resources automatically. Environment synchronization ensures whatever is installed on your development machine is mirrored in the cloud cluster, maintaining consistency and minimizing dependency errors.

Integrated dashboards and auto-shutdown prevent runaway costs and provide clarity on performance through built-in observability and cost controls.

Consider a research group in climate science building an ensemble model to forecast temperature patterns at high spatial resolution. Their workflow starts with basic array processing and data cleaning but quickly escalates to bootstrapped uncertainty estimation and model validation across decades of global data. With Coiled, they simply update their workflow to request cloud resources, and everything else remains familiar. The cluster scales up, the code runs, and results are returned—without the team needing to assemble custom Docker images or coordinate with IT to set up cloud networking.

Integration with the scientific Python stack#

Coiled fits into the existing ecosystem for scientific Python by leveraging key libraries. Through Dask integration, Coiled enables parallel and distributed computing with minimal code changes. Complex dataframes, arrays, and machine learning computations can be chunked and performed in parallel, using the same high-level APIs as local development.

Beyond distributed Dask clusters, Coiled Batch enables parallel processing of any code—whether that’s running hundreds of Python simulations with different parameters or leveraging Julia for high-performance scientific computing. This flexibility means teams can parallelize Monte Carlo simulations, parameter sweeps, or any embarrassingly parallel workload across cloud resources without being limited to specific frameworks.

Custom environments allow researchers to add dependencies or switch between projects without delay. For example, a genomics team needing a new release of Biopython only needs to install it locally; Coiled syncs the environment automatically when the job is launched on the cloud.

Coiled works with existing Python packaging, workflow orchestration tools like Prefect or Dagster, and any Python-compatible editor or notebook environment. Teams can mix and match open-source tools without vendor lock-in.

Consider the Lawrence Berkeley National Lab using Xarray to analyze satellite imagery data. As the area of analysis grows from local watersheds to entire continents, memory limitations become severe. Rather than rewriting their codebase, the team used Coiled to provision a distributed cluster, leveraging cloud resources to handle continent-scale analyses.

Example: Mineral exploration at scale#

KoBold Metals, a company dedicated to discovering new battery metals, depends on evaluating vast regions for signs of subsurface mineral deposits using remote sensing, geophysics, and AI models.

Early in a project, they analyze limited geochemical samples and low-resolution magnetic maps. But as interest grows in a region, the data volume expands: airborne geophysics, electromagnetic surveys, drilling logs, and hyperspectral imagery all need to be synthesized at fine spatial resolution.

On a single machine, these tasks are not feasible. Workflows such as computing spatial derivatives via Fourier transforms on gigapixel magnetic map rasters cannot complete in a reasonable timeframe or within available memory.

Using Coiled, KoBold shifts its analysis to a distributed Dask cluster, using their existing Python codebase. The computation, which previously failed or took days, now completes in hours across the cloud, allowing the exploration team to rapidly test hundreds of geological hypotheses. This approach increases scientific rigor and shortens time-to-decision, letting the business allocate exploration budgets where data suggests the best prospects.

As KoBold scales its operations, the technical team does not need to build or maintain cloud infrastructure. Scientists and engineers can install the latest analysis libraries as needed, confident that their development environment will be matched in each distributed run.

Observability and optimization#

Scientific computing demands not only raw power but also transparency. When data loads or model training jobs fail, researchers need immediate, actionable feedback. Coiled provides detailed dashboards showing cluster health, progress, and cost, with direct links to Python-level logs and errors.

A data science team training deep learning models on environmental sensor data, using Optuna and Dask to parallelize hyperparameter tuning, might notice that memory usage spikes in certain epochs or that certain data partitions take significantly longer to process. Using Coiled’s observability features, they identify bottlenecks and optimize both their models and cluster resource allocation.

Cost control and cloud efficiency#

Cloud resources can be cost-effective when used conscientiously, or can become unexpectedly expensive through misuse or oversight. Common pitfalls include leaving machines running after computations are complete, unnecessary data transfer between cloud regions, or selecting oversized machine types for small jobs.

Coiled is designed to encourage responsible usage. Clusters auto-shutdown when idle, leaving no orphaned resources. Dashboards make cloud spend visible and actionable. Jobs and resources are configured to match the scale of the task, avoiding both underutilization and overprovisioning.

For a biomedical research lab processing hundreds of thousands of medical images nightly, this means they can scale up to meet demand but also automatically scale down—maintaining only minimal persistent resources and controlling costs in line with real activity.

Focus on science, not infrastructure#

The primary advantage of building a scientific Python stack with Coiled is the shift of focus back to scientific or analytical value. Many teams report significant reductions in the time engineers spend managing infrastructure, debugging environments, or adapting workflows to unfamiliar distributed systems.

Domain experts can iterate on analysis code, experiments, or modeling approaches without waiting for IT or DevOps support. New projects or collaborations can begin immediately, without lengthy setup or onboarding. Teams can experiment more freely, attempting analyses or rescaling projects that would previously have been impractical due to technical hurdles.

Example: Mapping tropical forests at planetary scale#

Space Intelligence, a 60-person company, tackles one of the most computationally demanding geospatial challenges today: creating detailed maps of the world’s tropical forests to support their protection and restoration. To map a country like Brazil, they process hundreds of terabytes of multi-spectral satellite data at 10-meter resolution across multiple time periods.

Their workflow combines data acquisition, dimension reduction, machine learning classification, and post-processing—all built on the Python geospatial stack including Xarray, Zarr, Cloud-Optimized GeoTIFFs, and Scikit-learn, orchestrated through Dagster.

Initially, the team tried local processing on laptops, then moved to AWS Lambda functions, and eventually built their own Kubernetes-based Dask infrastructure. However, their DIY approach suffered from significant inefficiencies. Up to half their compute usage was wasted due to cluster fragmentation, with memorable incidents like keeping 1,000 CPUs running for three days due to a single long-running job.

After transitioning to Coiled, Space Intelligence found the setup “phenomenally easy” and can now process continental-scale data using up to 300 workers. The platform’s simple VM-based architecture eliminated the fragmentation issues they experienced with Kubernetes, while integrated monitoring provided the high-quality metrics their team needed.

Most importantly, Coiled’s expert support extended beyond just the platform itself, helping optimize their broader Python geospatial stack and debug performance issues across the entire ecosystem. This shift allowed the team to focus on their core mission: building detailed forest maps that power conservation efforts and carbon credit programs in the fight against climate change.

Business and scientific impact#

The capabilities provided by Coiled have downstream implications beyond individual project acceleration. Companies can prototype and move to production faster, testing new hypotheses or products in days instead of weeks or months. Engineering and research time is freed up to focus on core innovation. Access to scalable computing means organizations are no longer constrained by prior infrastructure investments or organizational silos.

In highly competitive domains—whether mineral exploration, pharmaceuticals, or climate analytics—this capability translates to actionable insights, smarter resource allocation, and speed to impact in the market.

Conclusion#

Coiled makes large-scale, distributed scientific computing feel as natural as local development. By abstracting complexity, ensuring environment consistency, and embedding intelligent cost controls, teams can step directly from ideas to execution at any scale.

As the world’s most pressing challenges demand larger and more complex analyses, the ability to build scalable scientific Python stacks without friction becomes essential. Coiled provides this foundation, ensuring computational power is available where it’s needed most.

FAQs#

What is scientific Python?#

Scientific Python refers to a collection of Python libraries and tools specifically designed for scientific computing, data analysis, and visualization. At its core are libraries like NumPy and SciPy for numerical computations, Pandas and Xarray for data manipulation, Matplotlib and Plotly for visualization, and libraries like Scikit-learn, PyTorch, and TensorFlow for machine learning. These tools collectively enable researchers, data scientists, and analysts to perform complex scientific computations efficiently.

How is Python used in science?#

Python is used extensively across various scientific disciplines. Scientists use it for data analysis, simulation, modeling, and visualization. For example, climate scientists use Python to build multi-model ensembles for temperature forecasting, biomedical researchers analyze RNA sequencing data, and geospatial analysts process satellite imagery. Python’s flexibility allows it to scale from simple analyses on local workstations to complex distributed computations on cloud platforms using tools like Dask and Coiled, making it suitable for both small experiments and large-scale scientific projects.

Does NASA use Python?#

Yes, there are a number of different teams at NASA, like the National Snow and Ice Data Center, that extensively use Python. Python plays a crucial role in processing and analyzing the vast amounts of data collected from space missions, telescopes, and satellites. NASA scientists use Python libraries for tasks ranging from orbital mechanics calculations to image processing of planetary and deep space observations. Python’s accessibility and robust scientific computing capabilities make it valuable for NASA’s research teams working on complex problems in astrophysics, planetary science, and space mission planning.