Using AWS Lambda can save a lot of time in getting services up and running but has some major drawbacks when used extensively. It is important to understand these pitfalls as they don’t usually matter until you run into them in production when it is too late to avoid.
1. Private API Gateway is Incompatible with Custom DNS
It is common in large scale software development to use private networking to increase security of internal services. This helps prevent unauthorized access to services that should never be available for public consumption. AWS Lambda supports this by using AWS VPC when configured appropriately.
It is also standard in most software development to use DNS to decouple services. DNS lets you deploy a new version of a service and cutover to it without modifying the existing service, this works because DNS records are just a pointer for where the service is hosted.
One of the common configurations of AWS Lambda uses AWS API Gateway to make the lambda accessible as a HTTP endpoint. However if you use API Gateway in a private network configuration you cannot leverage custom hostnames. This means you can only use the AWS API Gateway DNS record that is tightly coupled to the infrastructure and therefore need to modify the infrastructure to release a new version of the service.
Modifying the existing service infrastructure to release a new version increases the risk of the change and makes it harder to roll back.
2. Application Load Balancer to Lambda is Expensive
Another frontend for AWS Lambda is AWS Application Load Balancer which fixes the prior mentioned DNS and networking issue. However, compared to the extremely low costs AWS API Gateway, this is an expensive alternative.
In my region AWS ALB costs approximately $18 per month as a base (before the cost of compute power). This might not seem like much but with the prominence of microservices it is common for organisations to provision hundreds of different AWS Lambdas, which can lead to huge additional costs.
3. AWS Lambda Environment Variables Exposed
AWS Lambda can be configured with environment variables for things like database URLs, usernames, and passwords. This configuration shows up in the console when viewed with read only privileges. This means that giving people read only access in production can lead to exposed credentials.
Depending on your control of AWS IAM and your business environment this may or may not impact you but it is worth thinking about when configuring your services.
An alternative solution to this is embed your configuration into your application zip, I will cover this in a future article so follow me here to get updates about that.
4. API Gateway & Lambda Timeout Limits
AWS Lambda has a hard timeout of 15 minutes and when used in conjunction with AWS API Gateway a maximum timeout of 29 seconds. This means that any request that goes longer than one of these, depending on the context, will throw an error and fail.
It is important to handle these exceptions gracefully and return a response to the user but in order to do so we need to timeout before the timeout limit. This requires additional code to track the current duration of the execution, and cancelling it when getting too close to the timeout limit.
AWS Lambda timeouts also cause extra work to manage long running processes. To run a permanent background process you will need to handle starting and stopping constantly to fit within timeouts. This is usually far more difficult than an EC2 based solution that keeps a single process open.
5. SQS & Lambda Throughput Limit Errors
When using the AWS SQS as a direct trigger for AWS Lambda and limiting the throughput of the AWS Lambda Function any additional requests will throw errors. These errors can wreak havoc on alerting systems because each of these errors will show in your metrics.
It can also be a problem if you leverage dead letter queues as these failures contribute to the maximum retries per task without even invoking the lambda. Which can cause good messages to be pushed into the dead letter queue due to throughput limitations.
In Conclusion
AWS Lambda is a great service that enables software development with minimal operational overhead, but like all tools it has limits and it is helpful to understand them. I often use AWS Lambda to get services running but find that moving to AWS EC2 based services is often more cost effective and simple in the long term.
Follow me here for more content or contact me on:
- Twitter: @BenTorvo
- Email: ben@torvo.com.au
- Website: torvo.com.au