Ramping AWS Lambda Using Docker Images
I have always loved leveraging serverless technologies, and no service has been used more by me than Lambda for a whole bunch of projects. Being able to run a script that can be triggered by a plethora of services and methods without having to worry about underlying infrastructure is amazing.
As someone who majorly works with Python, using lambda functions for scripts is something that always felt right, but it too had its limitations. Leaving the time and memory constraints aside (which also have increased a lot since its inception), the biggest problem we always faced with Lambda back in the day was dependencies. If you needed any third-party library, you had to bundle it with your code. You also had to create a zip with all the files of the library included. This would become even more difficult if the library you were using depended on C binaries. In these cases, one considered dropping Lambda for the solution altogether.
Frameworks like serverless frameworks made packaging libraries easier. They came with plugins that can be used to manage dependencies. You can specify your code and the framework was responsible for bundling all the Python libraries you mentioned in your requirement with it. This worked for most cases, but in cases where you may need additional C binaries, you were left scratching your head.
The solution is to create a docker image that was based on Amazon Linux. Inside the container, install the needed libraries and dependencies, copy over the library files and the needed C binary to your host machine, bundle them together and upload them to your lambda…… and your package is too big to be used by lambda, remove some c binaries which you think might not be needed and try again… and the library no longer works….
Yes, I went through this. For one project, I had to use a cryptography library to verify a SAML message on Lambda, and it took me HOURS and sleepless nights (s) to figure out which binaries needed to be bundled and which we could do without. For this purpose, my rule of thumb was not to run any script that had complex dependencies on lambda.
All that changed when in 2020, AWS allowed us to deploy lambda functions using docker images. All you have to do now is, create a Docker image that is based on Amazon’s Lambda base image, install all the needed libraries and dependencies, push the image to ECR and deploy a lambda function using the image.
The Docker image itself is pretty straightforward.
FROM public.ecr.aws/lambda/python:3.9
We are using the base image for python lambda as provided by AWS
COPY requirements.txt /requirements.txt
We then copy over our requirements.txt
RUN python3 -m pip install -r /requirements.txt — target ${LAMBDA_TASK_ROOT}
We then install our dependencies
COPY ./code ${LAMBDA_TASK_ROOT}
We copy over our source code
CMD [“handler.lambda_handler”]
And finally, we tell which command needs to be run, when the lambda is executed. This points to the handler function where we have written code.
And using that simple Docker file, we can create an image that can then be used to run our Python code, along with its dependencies on Lambda.
We can add all the libraries we need in the requirements.txt. We can also install any third party C libraries or binaries we need to run our code in the Lambda environment. For example if you want to install psycopg2-binary, you can add the following command to install the libraries it needs.
RUN yum install -y postgresql-devel gcc*
I have created a simple sample Git repo that contains the most basic Dockerfile along with the code structure you need to create a docker image that can be used to deploy lambda functions.
Details on how to use this example code to deploy the imgae to lambda are given in the README.md
https://github.com/hassanj-576/python-docker-example
If you want to create an even more complicated docker image, which is based on let us say ubuntu and not Amazon’s own Lambda image, you can even do that and use a library like awslambdaric (https://pypi.org/project/awslambdaric/). This lets you install Lambda’s runtime environment to any custom docker image.
Now, of course, this comes with its own drawbacks. The cold boot of a lambda running using a Docker image is slower than if you had uploaded a simple zip file. The docker image itself will be much bigger. So if you are writing a simple script with not many dependencies, simply create a zip file and upload it for your lambda code. On the other hand, if you are using something more complex, a script that depends on a number of Python libraries and C binaries, the best route is to use docker images. It is faster than having to manage all your dependencies manually and allows you to spend more time actually solving the problem at hand instead of pulling your hair out trying to make a lambda function work.
More content at PlainEnglish.io.
Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.
