ryanf.dev ⚡️
AWS Lambda Python Deployments: Zipped, Layers, Docker Containers

AWS Lambda Python Deployments: Zipped, Layers, Docker Containers


aws lambda python terraform

On this page

Zipped Lambda Deployments

Zipped deployments are the original and easiest way to deploy Lambda functions so this will be a brief review. It’s a straight forward process where you zip the function code plus any of its dependencies, then upload the zip directly to Lambda or put it in an S3 bucket. The problem is the limitation of 50MB for the compressed zip file and 250MB uncompressed. This is usually okay since Lambdas are intended to short running functions with a maximum execution time of 15 minutes.

You can do a zipped deployment like this:

Create a Python virtual environment and activate it

python3 -m venv venv
source ./venv/bin/activate

Install dependencies

pip install -r requirements.txt

Zip the site-packages folder that was created in the virtual environment folder

zip lambda.zip -r ./venv/lib/python3.10/site-packages/

Add the lambda function python file to the zip

zip lambda.zip lambda_function.py

Upload the zip to Lambda or S3 Bucket

And that’s it for zipped deployments.

Issues with zipped deployments:

The issue that comes up most often is when your development environment is different than the Lambda environment where your code runs in AWS. Python is an interpreed language but some of the libraries you install have modules written in the language C, which is a compiled language. Your Lambda functions will be running on Amazon Linux 2 or the newer Amazon Linux 2023 so if your code is compiled on a system with a different architecture, it’s not going to work. When you run into this problem you will see an error message something like this:

{
 "errorMessage": "Unable to import module 'lambda_function': Unable to import required dependencies:\nnumpy: No module named 'numpy'\npytz: No module named 'pytz'",
}

There are several ways around this. AWS SAM can make Lambda deployments pretty simple and will use the AWS Linux images when it’s creating your Lambda package. If you don’t want to use SAM, you could still use those AWS Linux images yourself. You could also try adding in these flags during the pip install:

# this example is for x86_64 architecture. 
#  for arm64 you need to change --platform to _aarch64

pip install \
    --platform manylinux2014_x86_64 \
    --target=package \
    --implementation cp \
    --python-version 3.10 \
    --only-binary=:all: --upgrade \
    pandas

You can find more information in the AWS docs where they provide more examples and best practices.

Lambda Layers

Layers don’t contain the function code itself but instead you zip the dependencies, custom runtimes and configuration files. Let’s say you use Pandas in 3 different functions, instead of deploying each function with Pandas in the zip, you could create a layer and share the layer among your functions. This drastically reduces deployment file size for functions and can help with the compiled code issue at the same time.

When Lambda pulls in a layer, it extracts the libraries to the /opt directory of the function’s execution environment so even though layers externalize the packages, you can import and use them just like a regular zipped deployment.

Making a Lambda layer for Python will usually go like this

Make a virtual environment and activate it

python3 -m venv venv

source ./venv/bin/activate

Install dependencies

pip install -r requirements.txt

Make a new directory so files can be copied into it

mkdir -p python

Copy the venv’s installed dependencies into the new directory

cp -r venv/lib/ python/

Zip and upload to Lambda or S3

zip -r lambda_layer.zip ./python

Lambda Docker Container Images

Lambda has supported container images since 2020 and has gotten some interesting performance upgrades since its initial release. This 2023 UseNix talk from Marc Brooker goes into detail on how they’ve used lazy loading, deduplication and other techniques to achieve up to 15x faster cold start times while having 10gb max image size.

As Mark says in the UseNix speech, they are leaning into the fact many people are re-using the same set of base images like Ubuntu, Alpine or Nginx and a majority of what’s uploaded are bit-for-bit identical. Since they’re so similar they can break these up into encrypted chunks and store them in S3 as an address store to be used by Lambda workers later.

Lambda with containers also gives the benefit of being able to do testing earlier in the deployment process that is more in line with what most CI/CD pipelines are using. Local testing was another challenge Lambda developers faced but now a Lambda runtime API can be included in the image and get access to the Lambda Runtime Interface Emulator(RIE) to use during the CI/CD build process. AWS base images come with RIE included.

Sample Dockerfile for AWS Lambda container image

FROM public.ecr.aws/lambda/python:3.10

ENV  AWS_LAMBDA_FUNCTION_TIMEOUT=60

COPY requirements.txt ${LAMBDA_TASK_ROOT}

RUN pip install -r requirements.txt

COPY lambda_function.py ${LAMBDA_TASK_ROOT}

CMD [ "lambda_function.handler" ]

Lambda Docker Python deployment process

This process is using an AWS base image and may be different for custom images.

Find an AWS base image for Python or make your own

AWS Base Images Link

Python3.10 on ECR:

public.ecr.aws/lambda/python:3.10-x86_64

Make a Dockerfile in root directory of Lambda function

Use the sample Dockerfile from above and paste it into a file named Dockerfile

Build and run the Dockerfile locally to make sure it works

docker build --platform linux/amd64 -t docker-image:test .  

docker run --platform linux/amd64 -p 9000:8080 docker-image:test

Curl the running container

curl "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

Kill container

docker ps
docker kill [container id]

Login to Elastic Container Registry

aws ecr get-login-password \
--region us-east-1 | docker login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com

Create a repository in ECR

aws ecr create-repository \
--repository-name mylambdarepo \
--region us-east-1 --image-scanning-configuration scanOnPush=true --image-tag-mutability MUTABLE

Copy repositoryURI of the output from the command above

 # REPLACE ACCOUNT ID AND REGION

"repositoryUri": "111122223333.dkr.ecr.us-east-1.amazonaws.com/mylambdarepo"

Tag the Docker image with the repositoryUri

docker tag docker-image:test 111122223333.dkr.ecr.us-east-1.amazonaws.com/mylambdarepo:latest

Push the image to ECR

docker push 111122223333.dkr.ecr.us-east-1.amazonaws.com/mylambdarepo:latest

Create an Execution Role for the Lambda AWS docs

aws iam create-role \
 --role-name docker-lambda \
 --assume-role-policy-document '{
    "Version": "2012-10-17","Statement": 
    [{ "Effect": "Allow", "Principal": 
    {"Service": "lambda.amazonaws.com"}, 
    "Action": "sts:AssumeRole"}]}'

Attach the managed AWS Execution Policy to the role

aws iam attach-role-policy \ 
    --role-name docker-lambda \
    --policy-arn \
    arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Zip the lambda_function.py file

zip function.zip lambda_function.py

Create Lambda Function use the ARN from the Execution Role

aws lambda create-function \
--function-name docker-lambda \
--package-type Image \
--code ImageUri=111122223333.dkr.ecr.us-east-1.amazonaws.com/mylambdarepo:latest \
--role arn:aws:iam::111122223333:role/lambda-ex

Finally, time to invoke the function

aws lambda invoke --function-name hello-world response.json

If everything went according to plan the response you see should look like this:

{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

If you don’t get a 200 StatusCode then double check the values you entered above match your AWS account ID, function name, role name and region.

Exploring Lambda Execution Environment

This is a look at installing Python packages after a Lambda function starts. You will probably never want to do this, but maybe you’ll find it interesting anyway. Keep in mind Lambda costs are determined by a combination of: number of executions, the function’s duration and memory usage, and data transfer. In the example below I used yfinance and it actually takes so long to install you would have to increase Lambda’s default timeout to higher than 3 seconds or you get an error. This would be a terrible idea on a function that’s regularly used due to Lambda’s duration cost but there might be some rare use cases for doing this. Of course if the next Lambda request reuses that environment it won’t have the long cold start, but warm starts are not reliable. I’ll add more about the Lambda execution lifecycle at the end.


import os
import sys
import subprocess

subprocess.call('pip install yfinance -t /tmp/ --no-cache-dir'.split(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

sys.path.insert(1, '/tmp/')

import yfinance as yf

def lambda_handler(event, context):

    msft = yf.Ticker('MSFT')
    print(msft)

Execution Environment Lifecycle

When a request is made to run a Lambda function, AWS creates the execution environment, in short this means Lambda downloads your code then allocates some memory and a runtime for the function. Creating this environment takes time and is what’s known as the cold start. Once a Lambda function has executed and is no longer running, AWS keeps this environment alive for a short amount of time and if another request comes in then it can reuse the environment and avoid a cold start. What happens if multiple requests arrive at the same time? Well, Lambda has to scale up by making multiple execution environments and each one will include a cold start.