Running around the block: a beginner meets AWS Lambda
Lambda is at the forefront of the serverless revolution, allowing developers to run code without managing servers. As an eager newbie getting started with Lambda, I was quickly enamored by its simplicity. However, my excitement soon turned into frustration when I hit the deployment package size limit.
In this ~2600 word story, I elaborate on my journey as a beginner running around the block trying to get a grip on Lambda and its little challenges. Buckle up for an honest account with all the learning, unlearning and relearning along the way!
Why I opted for Serverless Lambda
As a full stack developer working on a photo editing application, I had to make a choice on the infrastructure – should I go the traditional route of procuring and managing EC2 instances OR take advantage of the new serverless paradigm?
Pros of Lambda:
No server management: With Lambda, I don‘t have to worry about provisioning and managing servers at all. Lambda handles this itself allowing me to fully focus on just the code.
Continuous scaling: My application needs to handle large workloads during peak times. Lambda can scale up instantly to support any traffic spike without capacity planning.
Granular billing: Pay per request pricing of Lambda ensures I pay only for what I use rather than overpaying for idle resources. At $.20 per million requests, costs are already negligible.
Fully managed: Lambda takes care of everything like capacity planning, patches, high availability so I can build without devops overhead.
Cons of Lambda:
Cold starts: When a function is invoked after being idle, cold starts can cause some latency. For a latency-sensitive application, this needs consideration.
Vendor lock-in: Lambda code cannot be easily ported. I‘ll have to rewrite functions to migrate from AWS unlike portable containers.
Function composition: Coordinating communication between Lambda functions brings complexity compared to a monolithic application.
On weighing the pros and cons, Lambda‘s auto-scaling capabilities and hands-off serverless experience clearly made sense for my use case. The cost savings were just an added bonus!
Lambda vs Containers vs VMs
It‘s useful to distinguish serverless offerings like Lambda from traditional infrastructure like VMs or containers.
Virtual Machines: Services like EC2 provide access to a dedicated virtual server that you can configure and run applications on. But you still have to continually manage, scale and patch these VMs.
Containers: Container services like ECS make it easier to deploy applications without worrying about the underlying server infrastructure. But you still have to handle container orchestration and clustering at scale.
Serverless platforms: Lambda completely abstracts away servers, container management and even execution runtimes. The provider handles this so developers just have to supply code.
So while VMs offer ultimate flexibility, they involve huge ops overhead. Containers simplify by standardizing environments while still requiring orchestration logic.
Lambda takes abstraction to another level altogether by making server infrastructure disappear! This perfectly matched my needs.
Why size limitations on Lambda packages
With growing adoption, Lambda usage has exploded from 1.3 billion requests/day in early 2017 to over 13 billion requests per day in 2022!
To sustain this massive scale, AWS has to optimize Lambda runtime performance. That‘s why strict limits are enforced on deployment package sizes based on the correlation between package size and:
Cold start times: Large packages increase cold start durations as more code has to be loaded.
Memory usage: Bigger packages consume more initialization memory before executing code.
By capping package sizes at 50MB uncompressed (and 250MB compressed), AWS ensures optimal cold start times and low memory utilization for most functions.
However, for beginners like myself, running into this package size limit can be perplexing if you‘re unaware. So let‘s see what happened when I deployed my first Lambda function.
Slamming into the size wall
When I first bundled my photo editing Lambda code written in Node.js along with modules like Sharp for image processing, it exceeded the 50MB limit:
Error: Deployment package too large: 7423466 bytes (must be smaller than 5242880 bytes)Turns out libraries like Sharp contain platform-specific native binaries and bindings for image processing that bloat up package size.
My journey to trim down the function size began…😓
Techniques to slim down functions
Through trial-and-error across dozens of code uploads, I found a combination of optimizations worked well to slim down functions:
Minify Code
Removing whitespace, comments and unused code reduces package size. For JavaScript, we can use Terser to minify code before deploying:
// before minification
function resizeImage(img) {
// Set resize options
const options = {
width: 1920,
height: 1080
}
// Resize image
return sharp(img).resize(options);
}
// after
function resizeImage(a){const b={width:1920,height:1080};return sharp(a).resize(b)} For my function, Terser based minification reduced package size by 34%.
Tree-shake Dependencies
If you import entire libraries but use only parts, tree shaking removes unused code from bundles.
For example, my function imported the sharp library but used only image resizing capabilities:
import * as sharp from ‘sharp‘;
export function handler(event) {
// only sharp‘s resize used
let resized = sharp(image).resize({...});
return resizedImage;
}After tree shaking, all unused sharp functions were stripped from the bundle reducing package size by over 40%.
Use Docker Base Images
If native dependencies like Sharp still bloat packages, you can switch to using Docker images instead up to 10GB in size.
So I could build a custom Docker image with Sharp installed globally. My function then used this scaled-up environment eliminating native binaries in the bundle itself.
This helped cut package size by 65%.
Lazy load dependencies
Lazy loading refers to importing libraries only when needed instead of initializing all on cold starts.
For my function, I lazy loaded parts of the Sharp module that weren‘t used in every invocation:
// Only imports the S3 sdk on first call
const AWS = require(‘aws-sdk‘);
const s3 = new AWS.S3({apiVersion: ‘2006-03-01‘});
exports.handler = async () => {
if(!sharp) {
const sharp = require(‘sharp‘); //Lazy loaded
}
//Rest of handler
}Lazy loading helped shave off another ~15% from bundle size.
By combining all these techniques, my package size reduced from 7.4MB to just under 50MB meeting AWS limits.
Principles for Optimization
Based on this experience, here are 3 key principles for optimizing Lambda package sizes:
- Eliminate overhead: Remove all non-essential code and unused dependencies
- Distribute load: Break down functions and distribute load across modules
- Offload responsibilities: Offload tasks like dependencies and runtimes to separate containers
Adopting these principles helps developers minimize what needs to be packaged.
Alternative solutions to size issues
If despite your best efforts, the Lambda deployment package still exceeds 50MB, another option is to decompose the function into smaller functions.
For example, my photo editing function could split into:
- resizePhotosHandler – Handles resizing
- sharpenPhotosHandler – Handles sharpening
- uploadPhotosHandler – Uploads images to S3
By breaking down the function into smaller parts, each function‘s business logic is simpler allowing much smaller code bundles.
These functions can then be orchestrated in a sequence using AWS Step Functions to reproduce the workflow of the larger function.
This brings some added complexity of orchestrating Lambda functions but may be an acceptable price for avoiding size limitations.
Pain points developing with Lambda
While Lambda takes away enormous devops overhead, developers still face some headaches like:
Observability issues: Logging and monitoring functions distributed across regions with varying data formats can make debugging tricky.
Startup latencies: Cold starts due to functions initializing can impact latency-sensitive apps. Keeping functions warm helps but incurs costs.
Vendor lock-in: Lambda functions built with native AWS services causes lock-in. Porting to another cloud means rewriting function logic.
No optimization visibility: Unlike containers where CPU and memory are configurable, developers have little visibility into how AWS optimizes Lambda environments internally.
However, these pains are outweighed by Lambda‘s immense benefits. Using tools like Dashbird and architectural patterns like async processing help overcome drawbacks.
And AWS continues to invest in Lambda – expanding runtime support, storage limits and improving observability to ease developer experience.
Key Takeaways from getting started with Lambda
Know your deployment options: Understand tradeoffs between deploying Lambda functions versus provisioning infrastructure like VMs or containers based on your workload patterns.
Package optimally: Follow best practices around code minification, tree shaking, lazy loading etc. to reduce bundle sizes close to the 50MB limit so you can use Lambda without any restrictions.
Monitor and debug: Use cloud native observability tools for tracing, logging and monitoring to ease identifying issues in production across your serverless environment.
While Lambda simplified immense devops complexity for me, it was still no cakewalk. I faced my share of challenges that forced me to run around the block debugging issues from cold starts to package sizes.
But the skills I picked up through this baptism by fire makes me feel battle-hardened as a serverless developer now. Instead of complex infrastructure management, I can dedicate 95% of my time to just building applications… and that excites me!
So I encourage all newcomers like me – don‘t hesitate, dive headfirst into exploring Lambda. The lessons you learn by grappling with real world complications will make you a sharper developer in the end!
