Deploying python lambda functions with dependencies on native libraries like pandas and numpy
by: Peter Shoukry
The good thing about lambda is the bad thing about it: It saves you from having to manage an underlying infrastructure *but* you also have to use what is there. Publishing a lambda function depending on native libraries, or with dependencies depending on them can be tricky, since you have to build them for the architecture they will ultimately run on, Lambda! ## First solution The steps to do that are straightforward, after figuring them out, namely: 1- Use a docker file to build an image with all the dependencies in a virtual environment. ```docker from amazonlinux:latest RUN yum update -y RUN yum install -y python3 python3-devel python3-virtualenv python3-setuptools gcc gcc-c++ findutils zip RUN mkdir /dependencies WORKDIR /dependencies # Create a virtual env using copies instead of symlinks RUN python3 -m venv --copies lambda_build # activate the virtual env and install the dependencies RUN source lambda_build/bin/activate && pip install --upgrade pip wheel && pip install numpy && pip install pandas # Zip all your dependencies RUN cd /dependencies/lambda_build/lib64/python3.7/site-packages/ && zip -r9 /dependencies/function.zip . ``` 2- Create a docker-compose file that will add the lambda function to the generated zip file and create a copy in an output folder ```docker version: '3' services: release: build: context: . command: sh -c "zip -g function.zip app.py && cp function.zip /dependencies/output/" volumes: - ./app.py:/dependencies/app.py - ./output:/dependencies/output ``` 3- The python3 package installed by yum now references version 3.7 by default so when you create your lambda function select python 3.7 ## A second solution With the introduction of [layers](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html) in lambda you can add a new layer with the dependencies.