Writing & Building AWS Lambdas, In Python, On a Mac

This all started at work, with the need to write an AWS lambda function in python, with a few additional dependencies, then discovering at least on of the dependencies required pip to compile something from C.

So the function would run just fine locally (on Mac) but fail on lambda execution due to incorrect binary format "Invalid ELF header" on Lambda execution.

First steps, basic dependency handling

For Lambda, all dependencies need to be packaged in the same zipfile as your lambda function code.

So we start with these 2 files in a directory

            main.py
            requirements.txt
            

Then run pip3 install --target=lib -r requirements.txt, to give this structure

            main.py
            requirements.txt  ## either manually created, or `pip freeze > requirements.txt`
            lib/    ## all the dependencies are installed here
            

Add the following at the top of your python file, before any other imports, to load packages from that local lib folder

            import sys
            import os
            cwd = os.path.dirname(os.path.abspath(__file__))
            sys.path.append("%s/lib" % cwd)
            

Finally, to publish as a lambda, create a zip of the code directory (including lib) and use the AWS console, cloudformation, terraform etc to upload the zipfile as function code.

The Interesting Stuff

Here is where I ran into “Invalid ELF header” errors on Lambda, so there's a bit more work to do. Basically, pip3 needs to run in the same runtime environment (Amazon Linux 2) as the python code will use when run as a lambda function. Luckily for us, Amazon publishes their Linux 2 as a docker container.

So in this step, we'll boot docker locally & have that handle pip3 and the code packaging.

File layout

            src/main.py
            src/requirements.txt
            docker-compose.yml
            Dockerfile
            

            # docker-compose.yml
            # run docker to do pip3 install
            version: "2"

            services:
               pip:
                    build: ./
                    volumes:
                        - ./src:/src
            

`docker-compose.yml` is very simple, mount our source code directory into the docker image, so docker can access it & so all files created by pip3 and zip will be in the same local source directory on your mac.

            # Dockerfile
            FROM amazonlinux:2  # important, same underlying system as used by the AWS Lambda runtime

            RUN yum install -y python3 python3-pip zip

            CMD cd /src \
                && rm lambda.zip \      ## cleanup
                && rm -rf lib \         ## cleanup
                && pip3 install --target=lib -r requirements.txt \
                && zip -r lambda.zip main.py lib
            

Run this with `docker-compose up` from the directory containing `docker-compose.yml`, and you'll end up with a new zipfile containing your python lambda function & all its dependencies, with binaries in the correct format to run as an AWS Lambda function.

            src/main.py
            src/requirements.txt
            src/lib/
            src/lambda.zip      ## function bundled with deps, upload this file to lambda
            docker-compose.yml
            Dockerfile