Packaging and publishing NodeJS Lambda functions in Terraform
I’ve been working about 70/30 in CDK and Terraform over the last year, and there are many aspects of each that I wish I could bring to the other. CDK has “constructs” which are akin to Terraform modules. One of my favourite constructs is NodejsFunction - it packages up Lambda code that’s collocated in the same repository and uploads the resulting zip to AWS.
Here’s what it looks like in use:
new nodejs.NodejsFunction(this, "MyCoolFunction", {
entry: "handler/index.ts",
handler: "index.handler",
bundling: {
minify: true,
sourceMap: true,
sourceMapMode: nodejs.SourceMapMode.INLINE,
target: "es2021",
},
});
The construct uses a Docker container by default, but we can easily install esbuild and have it bundle the code together. esbuild takes care of bundling all the code into one file, tree shaking and TypeScript transpilation.
I love this developer experience, especially the ability to collocate Lamba functions that provide the “glue” we often write as platform teams in the same codebase as our Infrastructure as Code (IaC). How can we achieve the same thing in Terraform?
First attempt
The first solution I built was a script that would package up the Lambda handlers and populate a tfvars file with their source code hash and file name:
function hash_package() { /* ... */ }
echo "Cleaning build directory..."
rm -rf build
echo "Ensuring code compiles..."
yarn compile
echo "Cleaning bundle code..."
rm -rf packaging
echo "Bundling code..."
yarn esbuild \
--bundle \
--external:@aws-sdk'*' \
--external:@aws-lambda-powertools'*' \
--minify \
--outdir=packaging \
--platform=node \
--sourcemap \
--target=es2021 \
--sourcemap=inline \
src/handlers/*/index.ts
LAMBDAS=""
echo "Packaging Lambda function..."
cd packaging
for BUNDLE in *; do
cd "$BUNDLE"
zip package.zip index.js
PACKAGE_HASH=$(hash_package package.zip)
FINAL_PACKAGE_PATH="package-${PACKAGE_HASH}.zip"
mv package.zip "$FINAL_PACKAGE_PATH"
echo "Created $FINAL_PACKAGE_PATH for $BUNDLE"
echo "Uploading..."
aws s3 cp "$FINAL_PACKAGE_PATH" "s3://automate-aws-access-removal-${AWS_ACCOUNT_ID}-${AWS_REGION}/${FINAL_PACKAGE_PATH}"
echo "Uploaded $FINAL_PACKAGE_PATH"
cd ..
LAMBDA=$(cat <<EOF
"${BUNDLE}" = {
s3_key = "${FINAL_PACKAGE_PATH}"
source_code_hash = "$(echo -n "$PACKAGE_HASH" | xxd -revert -plain | base64)"
}
EOF
)
if [[ -z "$LAMBDAS" ]]; then
LAMBDAS="$LAMBDA"
else
LAMBDAS="$LAMBDAS
$LAMBDA"
fi
done
cd ..
cat << EOF > ../infrastructure/main.auto.tfvars
# Generated by lambda/scripts/package.sh - do not manually edit
lambdas = {
${LAMBDAS}
}
EOF
terraform fmt ../infrastructure/main.auto.tfvars
The generated tfvars file lets us feed the values we need directly into the aws_lambda_function resource:
lambdas = {
"excluded-users-listener" = {
s3_key = "package-5b965c5f4f92abd08970a6e7a608f932269d58f2be2f3ccd8278284013d64121.zip"
source_code_hash = "W5ZcX0+Sq9CJcKbnpgj5MiadWPK+LzzNgngoQBPWQSE="
}
"user-deleted-listener" = {
s3_key = "package-64b6cf60cd25373c20ee08686f0b5076fb6822dc0aae7d63b7eaeec9b7ab8db2.zip"
source_code_hash = "ZLbPYM0lNzwg7ghobwtQdvtoItwKrn1jt+ruyberjbI="
}
// ...
}
The script worked, but got a bit frustrating when the source code hash changed every time I repackaged the code. No-one likes a Terraform plan that’s never clean! Let’s fix that by setting the modified time of the bundle file and using it in the zip:
# Before:
zip package.zip index.js
# After:
function set_timestamp() {
local touch_cmd
touch_cmd="touch"
if [[ "$OSTYPE" == "darwin"* ]]; then
touch_cmd="gtouch"
fi
# Use a fixed date to make reproducible bundles
$touch_cmd --no-dereference --date='2020-08-31 00:00:00' "$1"
}
set_timestamp index.js
zip --latest-time package.zip index.js
Much better! This script approach was reasonably simple, but it’s fiddly to use when you’ve got to remember to run it before your terraform plan. What could we do to improve the dev experience?
Attempt #2 - a custom provider
Terraform provides a framework to help us write the providers which power the resources and data sources we rely on, and we’re lucky that both esbuild and Terraform are written in Go. Unfortunately, the first repository I found was the hashicups example project, which uses an older approach to building providers. The starter project I really wanted was terraform-provider-scaffolding-framework. Let’s have a look at how we make our own provider!
NB: wanting to skip ahead and start using it? Check out the finished solution to package NodeJS Lambda functions in Terraform in the Terraform provider registry.
First, we define a provider with our data source:
// ...
func (p *NodeLambdaPackagerProvider) DataSources(ctx context.Context) []func() datasource.DataSource {
return []func() datasource.DataSource{
NewLambdaPackageDataSource,
}
}
func New(version string) func() provider.Provider {
return func() provider.Provider {
return &NodeLambdaPackagerProvider{
version: version,
}
}
}
Then we set some parameters that can be passed into our data source:
func (d *LambdaPackageDataSource) Schema(ctx context.Context, req datasource.SchemaRequest, resp *datasource.SchemaResponse) {
resp.Schema = schema.Schema{
Description: description,
MarkdownDescription: strings.ReplaceAll(markdownDescription, "BACKTICK", "`"),
Blocks: map[string]schema.Block{},
DeprecationMessage: "",
Attributes: map[string]schema.Attribute{
"args": schema.ListAttribute{
ElementType: types.StringType,
Required: true,
Description: "Arguments to pass to esbuild.",
},
"entrypoint": schema.StringAttribute{
Required: true,
Description: "Path to lambda function entrypoint.",
},
"working_directory": schema.StringAttribute{
Required: true,
Description: "Typically the folder containing the package.json at the root of your Lambda project.",
},
},
}
}
Finally, we call esbuild with the parameters provided by our user:
import (
// ...
"github.com/evanw/esbuild/pkg/api"
"github.com/evanw/esbuild/pkg/cli"
// ...
)
func (d *LambdaPackageDataSource) Read(ctx context.Context, req datasource.ReadRequest, resp *datasource.ReadResponse) {
// ...
buildArgs, err := cli.ParseBuildOptions(args)
result := api.Build(buildArgs)
if len(result.Errors) >= 1 {
// ...
}
// ...
}
The rest of the code to form the zip file isn’t very exciting, but here’s what we get as the developer experience when we use this provider:
data "node-lambda-packager_package" "this" {
args = [
"--bundle",
"--external:@aws-sdk*",
"--external:@aws-lambda-powertools*",
"--minify",
"--platform=node",
"--sourcemap",
"--target=es2021",
"--sourcemap=inline",
]
entrypoint = var.entrypoint
working_directory = var.working_directory
}
resource "aws_lambda_function" "this" {
function_name = "my-cool-function"
handler = "index.handler"
memory_size = 256
runtime = "nodejs18.x"
filename = data.node-lambda-packager_package.this.filename
source_code_hash = data.node-lambda-packager_package.this.source_code_hash
}
When no changes are made to the Lambda code, the source_code_hash remains the same and we see no changes in our plan. Lambda code collocated with our IaC? Check. Automatic building, tree-shaking and compilation? Check. A very new Terraform provider which needs your input and feedback? Triple check!
Conclusion
With the addition of a custom provider, we can bring some of the great developer experience of CDK to Terraform. I’m a big believer in keeping relevant code close together, and this keeps us away from multiple projects or multiple pipelines that are all too easy when we’re deploying an application and IaC.
Find the Terraform provider and the full source code on GitHub to get started bundling and publishing your Lambdas from the same codebase as your Terraform code!
Learning AWS' CDK in TypeScript? Check out my course on Udemy.