How to run a Python script on EC2#

2025-06-25

5 min read

There are a lot of ways to run Python on AWS. You might use Lambda for event-driven tasks, Glue for managing data pipelines, or SageMaker for machine learning workloads. Each of these tools offers helpful abstractions, but also comes with constraints. For example:

  • Lambda is fast and simple, but capped at 15-minute execution time.

  • EMR can scale data processing across a cluster, but has limited support for custom libraries.

  • SageMaker is powerful for ML, but can be heavyweight for general-purpose Python scripts.

That’s why a lot of engineers will instead use EC2, which is highly configurable. You get you raw virtual machines anywhere in the world in about 30 seconds, fully under your control, for just pennies.

While AWS EC2 offers remarkable flexibility, it’s pretty low-level and not the most intuitive. Many data scientists and engineers avoid using it directly because it can feel tedious or overly complex. Worse, it’s often inefficient (like spinning up a machine and accidentally leaving it on all week). Coiled aims to address these issues, without sacrificing that flexibility.

Coiled makes it easy to run Python scripts on EC2 and scale out to run your jobs in parallel. It provisions machines on your behalf, replicates your local Python environment, and shuts everything down when the job finishes. You get the versatility of EC2, without the constraints of other AWS tools.

This post walks through two ways to run Python on EC2:

  1. The manual route (for full control).

  2. A Python-native method using Coiled.

Option 1: Run Python on EC2 manually#

This process gives you full control over how and where your Python script runs in AWS, but it’s a bit involved. Here’s how to do it, correctly and safely.

Step 1: Launch an EC2 instance#

  1. Go to the EC2 Console: https://console.aws.amazon.com/ec2/

  2. Launch Instance: Click “Launch Instance”.

  3. Choose an AMI: Pick something like Ubuntu or Amazon Linux.

  4. Select Instance Type: t3.medium is a good starting point. Use larger instance types for bigger workloads (Vantage has a nice EC2 comparison table).

  5. Create a Key Pair (RSA): Download the .pem file (keep it private, you’ll need this for SSH access).

  6. Configure Network Settings:

    • Open SSH (port 22) in the security group only to your IP (not 0.0.0.0/0).

    • Optionally, enable HTTP (port 80) or HTTPS (443) if your script serves a web app.

  7. Launch the Instance

Step 2: Connect to your EC2 instance#

Once your EC2 instance is running, click “Connect” to see the instructions for how to connect to your instance.

Ensure the .pem file has the right permissions:

chmod 400 your-key.pem

ssh -i your-key.pem ubuntu@<your-ec2-public-ip>

ssh -i your-key.pem ec2-user@<your-ec2-public-ip>

Step 3: Set up your Python environment#

Once you’re connected to the instance:

The Ubuntu AMI should already have Python 3 installed, but if not, you’ll need to install it first.

# Update the system
sudo apt update && sudo apt upgrade -y

# Install Python
sudo apt install python3 python3-venv -y

# Set up a virtual environment
python3 -m venv env
source env/bin/activate

The Amazon Linux AMI should already have Python 3 installed, but if not, you’ll need to install it first.

# Update the system
sudo yum update -y

# Install Python
sudo yum install python3 -y

# Set up a virtual environment
python3 -m venv env
source env/bin/activate

Step 4: Transfer your Python script#

From your local machine, use scp to upload your Python files to EC2:

scp -i your-key.pem script.py ubuntu@<your-ec2-ip>:~

scp -i your-key.pem script.py ec2-user@<your-ec2-ip>:~

You can also copy folders or zipped environments if needed.

Step 5: Run your Python script#

You should now be able to see the Python script you uploaded on your EC2 instance. You can list the files in your working directory to check with ls -l. Then, you can run the script:

python3 script.py

Step 6: Turn off your instance#

Once you’re done, if you don’t turn off your EC2 instance, AWS will keep charging you. You can do this from the AWS console. It’s worth noting that if you have an attached EBS volume, AWS will continue to charge you for storage unless you terminate the instance.

Screenshot of the AWS console showing how to terminate an EC2 instance to avoid runaway costs.

AWS EC2 best practices#

  • Use security groups to restrict SSH to your IP to control who can access your instances

  • Terminate EC2 instances when you’re done to avoid a surprise bill

  • Use S3 for data storage to decouple your data from ephemeral VMs

  • Since you’ll need to SSH into your remote VM, it’s helpful to at least know some basic shell commands

This manual approach offers full control but can be cumbersome, especially for scaling or handling complex dependencies.

Option 2: Use Coiled to run Python on EC2#

Coiled is a Python-native platform designed to simplify running Python workloads on the cloud. It allows you to easily manage virtual machines with minimal code. Coiled automates the provisioning of EC2 instances in your AWS account, synchronizes your environment, executes your code, and ensures resources are efficiently managed. Coiled:

  • Eliminates the need complex cloud console navigation.

  • Replicates your local Python environment and packages.

  • Supports any hardware available on EC2 (GPUs, ARM, Spot, etc.)

  • Automatically shuts down instances after execution to save costs.

  • Offers support for custom VPCs, subnets, firewalls, and private IPs.

Step 1: Install Coiled#

pip install coiled

Step 2: Configure AWS access#

Coiled requires limited IAM permissions to provision resources in your AWS account. You can set this up using the Coiled CLI:

coiled setup aws

Step 3: Run a Python script on AWS with Coiled#

my_script.py#
#COILED n-tasks     10
#COILED memory      8 GiB
#COILED region      us-east-2

import os

print(f"Hello from {os.environ['COILED_BATCH_TASK_ID']}")

And then launch your script with coiled batch run:

$ coiled batch run my_script.py

Coiled makes it easy to scale your Python scripts to run on as many VMs as you need. In this example, we ran my_script.py in parallel on 10 EC2 instances. Under the hood Coiled will:

  • Inspect your script

  • Spin up the appropriate machines as defined by the #COILED comments

  • Download your software onto them or use the container you specified

  • Run your script, automatically scaling out as needed with #COILED n-tasks

  • Shut down the machines

Using batch jobs is one easy way to run a simple Python script, but there are a number of other options:

Like AWS Lambda, but better for long-running jobs or if you need GPUs. Decorate a function to run it in the cloud. Scale out with the .map method.

import coiled
import pandas as pd

@coiled.function(region="us-west-2")  # Run close to the data
def process(filename):
    output_filename = filename[:-4] + ".parquet"
    df = pd.read_csv(filename)
    df.to_parquet(output_filename)
    return output_filename

# result = process("s3://my-bucket/data.parquet")  # one file

results = process.map(filenames)   # many files in parallel
for filename in results:
    print("Finished", filename)

Like AWS EMR, but easier to use (plus, Dask is often faster than Spark).

import coiled
cluster = coiled.Cluster(
    n_workers=100,
)
client = cluster.get_client()

# Use Dask + Pandas together
import dask.dataframe as dd

df = dd.read_parquet("s3://bucket/lots-of-data.parquet")
df.groupby("name").amount.sum().compute()

Run Jupyter on large cloud VMs. Synchronize files back to your local hard drive.

coiled notebook start --sync --vm-type m6i.16xlarge

When would you want to use EC2 manually vs. Coiled to run Python?#

Coiled supports a lot, including custom VPCs and firewall rules, but there are some cases where manually running Python on EC2 might be better:

  • Custom OS or low-level system configuration

  • Custom AMI creation

  • Highly regulated cloud environments with air-gapped requirements

  • Advanced networking (beyond custom VPCs), like hybrid clouds

Next steps#

It’s easy to get started with Coiled:

$ pip install coiled
$ coiled quickstart

Learn more: