Easily download any Python package for AWS Lambda Layers

https://ift.tt/3Eyn14I Ready-to-upload AWS Lambda Layer packages in under 2 minutes. Using a Lambda to create layers for our Lambdas Imag...

https://ift.tt/3Eyn14I

Ready-to-upload AWS Lambda Layer packages in under 2 minutes. Using a Lambda to create layers for our Lambdas

Image by Pixabay

AWS Cloud services, like AWS Lambda, are a flexible way to quickly deploy data solutions. But, handling packages and related dependencies can be frustrating when deploying complex Lambda infrastructure, even if you are using Lambda Layers.

AWS uses their own flavor of the Linux operating system. Which means we can’t just pip install Python packages locally and then deploy them to AWS Lambda directly. Usually, we’d need to use Docker or an EC2 to prep our Python package layers prior to upload to Lambda.

Turns out we can use AWS Lambda to automate AWS Lambda Layer preparation..

How to:

1. Create a landing Bucket in S3

• Create a Bucket in S3. We’ll have all of our Python packages land in this new Bucket. Note: The Lambda we’ll create will automatically organize the Python packages we download by its version.

2. Create a fresh Lambda

• Create a new Lambda, be sure to add an inline policy to allow this Lambda to Put and Delete objects into the target landing Bucket in S3.
• Be sure to adjust the Lambda’s maximum memory and timeout to accommodate for larger packages/dependencies. (E.g. Pandas used 280MB and took 15.153 seconds)
• Paste the code below into your Lambda:

• Be sure to swap out parameters with the Python package you want to download, its version, and your target S3 Bucket.

4. Enjoy

Paying a penny (or often less than that) to compile your Lambda Layers is definitely worth the time saved.

After the Lambda runs, you can check your S3 Bucket for the Python packages prepared by the Lambda.

Example of S3 Bucket organization by package. Image by author.

Each package you request will be organized by its version as well. So you can download different versions of the same package (if needed), and it will automatically be organized.

Pandas package versions that are ready for upload as a Lambda Layer. Image by author.

Now you can easily download the zip file from the S3 Bucket, and upload the zip as a Layer to AWS Lambda.

Thanks for reading ~.

---

Easily download any Python package for AWS Lambda Layers was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

from Towards Data Science - Medium https://ift.tt/3bpr3Qq
via RiYo Analytics