This all started at work, with the need to write an AWS lambda function in python, with a few additional dependencies, then discovering at least on of the dependencies required pip to compile something from C.
So the function would run just fine locally (on Mac) but fail on lambda execution due to incorrect binary format "Invalid ELF header" on Lambda execution.
For Lambda, all dependencies need to be packaged in the same zipfile as your lambda function code.
So we start with these 2 files in a directory
pip3 install --target=lib -r requirements.txt, to give this structure
main.py requirements.txt ## either manually created, or `pip freeze > requirements.txt` lib/ ## all the dependencies are installed here
Add the following at the top of your python file, before any other imports, to load packages from that local lib folder
import sys import os cwd = os.path.dirname(os.path.abspath(__file__)) sys.path.append("%s/lib" % cwd)
Finally, to publish as a lambda, create a zip of the code directory (including lib) and use the AWS console, cloudformation, terraform etc to upload the zipfile as function code.
Here is where I ran into “Invalid ELF header” errors on Lambda, so there's a bit more work to do. Basically, pip3 needs to run in the same runtime environment (Amazon Linux 2) as the python code will use when run as a lambda function. Luckily for us, Amazon publishes their Linux 2 as a docker container.
So in this step, we'll boot docker locally & have that handle pip3 and the code packaging.
src/main.py src/requirements.txt docker-compose.yml Dockerfile
# docker-compose.yml # run docker to do pip3 install version: "2" services: pip: build: ./ volumes: - ./src:/src
`docker-compose.yml` is very simple, mount our source code directory into the docker image, so docker can access it & so all files created by pip3 and zip will be in the same local source directory on your mac.
# Dockerfile FROM amazonlinux:2 # important, same underlying system as used by the AWS Lambda runtime RUN yum install -y python3 python3-pip zip CMD cd /src \ && rm lambda.zip \ ## cleanup && rm -rf lib \ ## cleanup && pip3 install --target=lib -r requirements.txt \ && zip -r lambda.zip main.py lib
Run this with `docker-compose up` from the directory containing `docker-compose.yml`, and you'll end up with a new zipfile containing your python lambda function & all its dependencies, with binaries in the correct format to run as an AWS Lambda function.
src/main.py src/requirements.txt src/lib/ src/lambda.zip ## function bundled with deps, upload this file to lambda docker-compose.yml Dockerfile