jSherz.com

Using AI to put the bins out

Thu, 09 Jan 2025 19:56:00 +0000

Aside from using GitHub’s Copilot when coding, I’ve not dabbled much in the tools that have appeared from the recent AI craze. Generating funny images is one thing, but I haven’t seen many use cases that are a meaningful improvement over what I could search for or do manually. Over the Christmas period, I found that I could request e-mail reminders from the local council for when I was due to put a wheelie bin out the following morning. I already run my whole life on Todoist - it helps me with everything from keeping track of my tasks in the working day to remembering birthdays - so I was keen to see if I could parse the relevant information out from those e-mails and feed it into a Todoist API call. Normally this would be a painful exercise in parsing information and hoping that nothing about the structure of the e-mail changes over time, which is never realistic. I started to wonder, could I use AI to process this text?

Laying the foundations

As someone outside of most of the AI hype, I wanted to be able to try different models and compare how well they worked for my use case. I didn’t want to have to sign up to multiple providers and load in some credits or pay for a subscription, so I was very keen to try out AI models on AWS’ Bedrock. I started by finding a region with reasonable latency that had the models available. At the time of writing, there are restrictions on access in Ireland, but Frankfurt had everything I needed. I chose Meta’s Llama 3.2 3B Instruct v1 and Claude 3.5 Sonnet by Anthropic.

It only took a few minutes for my requests for these models to be approved, and so it’s on to the architecture design!

An architecture for processing e-mails with AI

We’ll use SES incoming to receive e-mails going to an address we chose, for example put-the@bin-out.example.com. It will place the e-mail into an S3 bucket in its raw form, and then trigger a Lambda function. We’ll use the Lambda to call AWS Bedrock, process the message and send a REST request to the Todoist API.

When the machines can talk

Calling text-based models in Bedrock is simple as we can use a chat-style API to give our AI instructions on how to behave. We can then feed it some user input, in this case the e-mail, and have it respond back to us. But how do we generate machine-readable output?

Bedrock lets us set a ‘system’ prompt that teaches the AI how to behave, and my first iteration of this looked as follows:

You are a bot that receives e-mails that might be about putting a bin out the
following day. Respond with a JSON object with the following fields:

- 'isReminder': checks if this is an e-mail about an upcoming collection.
- 'day': the day the bin should be put out in ISO 8601 format.
- 'bin': the type of bin that should be put out.

For chat messages, I sent something like this:

Here's the e-mail:

**email text here**

I stored this prompt as a string and then appended the text of the e-mail at the bottom, but this didn’t have good results. Often the AI would return extra characters or other information outside the JSON object, and that made parsing the response very challenging. I then found a brilliant tip: ask the AI to improve the prompt you’ve given it!

A quick tip for faster feedback

To iterate faster, I would recommend using the “Chat / Text” playground in Bedrock. I did this in code the first time around, and that made things fiddly when I had to switch from one model to the next.

Machines telling machines what do to

The prompt for improving my prompt was very simple: “How can I improve this prompt: <existing prompt>”. I’ll summarize the tips that the AI provided:

Use HTML-style tags. I surrounded each part of the prompt in tags that look like HTML. For example:
```
<instructions>
...
</instructions>
  
<email>
...
</email>
```
Clarify how the output should be delivered. One thing I didn’t think to do was to tell the AI not to include any extra characters other than the JSON. That made a surprisingly big difference.
Return a confidence score. A really helpful diagnostic tool was asking the AI to add an additional field to the response body with a confidence score. If that was low, I could have my application return an error or refuse to create a task.
Add error handling. I always specified the isReminder field in the prompt, but initially I didn’t include clear instructions about how to handle error cases.
Clarify the problem space. Sometimes the AI would be too smart and think of edge cases that can’t be possible in the real world. For example, the council will never collect more than one bin in a day. I added extra instructions that defined these “rules” that may not be obvious to a computer.

Did it work and would I do it again?

With some trial-and-error in prompt engineering, Bedrock returned trustworthy JSON that I have successfully been parsing to power reminders. I’ve spent a grand total of two cents on AI models so far, which feels like good value for having a play and learning some more about how I can augment tricky text processing jobs.

As you can see above, it worked great for me - even if my use case is rubbish!

My experiences creating a Udemy course

Sat, 04 Jan 2025 16:07:00 +0000

I decided to try self-employment in early 2023, and I had a contract cancelled at the very last minute. While I was on holiday, in fact! I wanted to make use of the free time I had, so I decided to write and publish a Udemy course. I’ve always enjoyed learning about programming languages or the cloud, and most of that has been through Udemy courses. So, why not!

My experience with tutorial videos

It’s worth noting that prior to creating this course, I’d already filmed a number of tutorial videos for programming. I’ve always enjoyed making them, and so I’ve been through a fair few failed attempts while I tried different ways of filming videos. My personal approach is to create a rough outline of each video and then to try and film it all in one take. I often make mistakes, so I leave enough time to easily edit out that chunk and then start again repeating the last ~15 seconds.

If you haven’t ever tried creating tutorial videos, I’d suggest filming a few and uploading them to YouTube. If you’re anything like me, it’ll take some time to get used to hearing your own voice and overcoming the desire to perfect what you film before you upload it.

Choosing a topic

Part of my motivation for creating the course was my experiences working with people learning AWS’ CDK. I wasn’t happy with the documentation that was available, and I felt it left out a lot of important tips, tricks and workarounds. At that point in time, CDK had a lot of rough edges, and I wanted to share the methods I’d found to make it easier to use.

If I could give you only one piece of advice, it would be to create a course that’s related to something you’re living and breathing every day. I’ve tried to make tutorial videos in the past for topics I was less excited about, and they were nowhere near as good.

How long did it take to make the course?

End-to-end it took me around three full weeks to film and edit the course. I was working on it full time, and spent a number of evenings getting the last few videos over the line. I didn’t have to spend very long researching any topics as I’d been working with CDK daily before starting. Expect to spend a lot longer if you need to brush up your own knowledge first.

I ended up with around eight and a half hours of content to upload to Udemy.

How much money did I make?

Udemy is very clear about how the revenue split works, and at the time of writing they pay 37% of the transaction value if they bring the user to the course. If you refer a user with your own link, you get 97%. The real learning here for me is that external marketing or using your own network is vastly more effective than relying on Udemy to bring in sales that don’t earn a huge commission. If I were to create another course, I would explore doing my own marketing or using agencies. If you go that route - let me know how it goes.

All in all, I made just over $1,000 dollars in roughly a year of sales with the course. At just over £6 an hour, it’s certainly not the best paid job I’ve ever had, but I really enjoyed making it and got a lot out of the process. Let me be an example of the fact that selling the course on Udemy requires work.

What advice would I give you?

There are a number of things that I would do differently if I created another course:

Editing software and video storage

I used iMovie to edit the course, and in reality would have been better served by using something with more functionality. I would normalize audio volumes across videos and found iMovie’s concept of projects quite clunky.

I also underestimated how much storage space would be required. I reached just shy of 900GB of video files, which isn’t super easy to store on a laptop with everything soldered in place.

Removing noise from the audio

I manually edited out most of the “mouth noises” I made while filming the video, but would’ve been better served by finding a way to automate that and investing in an audio setup that makes it less likely to happen. Take it from me: it’s a time-consuming and boring job.

Marketing

It’s mentioned above, but I’d explore ways to market the course myself. Any customer acquired for around seven pounds or less would’ve been a win.

Conclusion - would I do it again?

As I’ve mentioned, creating video tutorials was an existing passion of mine and so the lacklustre sales figures wouldn’t put me off from trying something again in the future. If I were to create another course, I would most likely release it for free on YouTube, and then build a community first. I’ve worked with some extremely talented people who’ve successfully built a following by consistently putting out really high quality articles around AWS. I think they’re very well positioned to do something like this with more commercial success.

For me, it’s much more about doing something that keeps my passion for tech going than creating a high-performing side hustle. Udemy users get a lot of value for the small sum they pay for courses, myself included, so you have to make your own judgement about if it’s worthwhile.

Keeping base and CI/CD Docker images up-to-date in AWS

Sat, 13 Apr 2024 13:47:00 +0000

If you’re building containerised services, or using a CI/CD system, you’ll likely end up building base images that contain the customisations that fit your organisation’s needs. For example, you might update OS packages, install a newer version of a package manager, or install the CLI tool(s) of your chosen cloud provider. Keeping these images up-to-date can become a maintenance burden, especially if you want to keep several versions available for different programming languages or runtimes.

Let’s explore how we can automate this process.

In the shersoft-ltd/evergreen-ci-and-base-images repository, we’ll start by creating some Amazon Elastic Container Registry (ECR) repositories for our images (modules/ecr-repository/main.tf):

resource "aws_ecr_repository" "main" {
  name = var.name
}

We’re going to be making lots of images each day, so let’s also add in a lifecycle policy that will clean up any draft images, and those that are untagged:

resource "aws_ecr_lifecycle_policy" "main" {
  repository = aws_ecr_repository.main.id

  policy = jsonencode({
    "rules" : [
      {
        "rulePriority" : 1,
        "description" : "Retire draft images.",
        "selection" : {
          "tagStatus" : "tagged",
          "tagPrefixList" : [
            "draft"
          ],
          "countType" : "sinceImagePushed",
          "countUnit" : "days",
          "countNumber" : 1
        },
        "action" : {
          "type" : "expire"
        }
      },
      {
        "rulePriority" : 2,
        "description" : "Retire untagged images.",
        "selection" : {
          "tagStatus" : "untagged",
          "countType" : "sinceImagePushed",
          "countUnit" : "days",
          "countNumber" : 1
        },
        "action" : {
          "type" : "expire"
        }
      }
    ]
  })
}

With that in place, we’ll setup the following file structure:

runtimes/node/Dockerfile
runtimes/node/self-test.js
runtimes/node/self-test.sh
runtimes/node-ci-cd/Dockerfile
runtimes/node-ci-cd/self-test.js
runtimes/node-ci-cd/self-test.sh
runtimes/python/Dockerfile
runtimes/python/self-test.py
runtimes/python/self-test.sh

Each runtime we’re building images for will have a Dockerfile that defines how to build the image, and a self-test.sh file that will be used to verify that the built image works as we expect. Let’s have a look at the Dockerfile for the node runtime:

ARG RUNTIME
ARG VERSION

FROM node:${VERSION}

# Try and minimise active vulnerabilities by updating all OS packages
RUN apt-get update && \
    apt-get dist-upgrade --yes && \
    rm -rf /var/lib/apt/lists/*

COPY self-test.js /usr/local/bin/self-test.js
COPY self-test.sh /usr/local/bin/self-test

We’ll support teams developing in multiple runtime versions by varying the base image with the VERSION environment variable. For example, this could be 18 or 20 for Node LTS versions. If we had to vary the setup greatly by version, we could use if functions in the Dockerfile, or call different scripts or different Dockerfiles based on versions.

The repo linked above (shersoft-ltd/evergreen-ci-and-base-images) contains a full GitHub Workflow and CodePipeline / CodeBuild example. Let’s have a look at the GitHub version first. We’ll define a workflow that is run for pull requests, merges to the default branch and that also refreshes the images each day:

name: 'Build, verify and publish'

on:
  push:
    branches:
      - main
  pull_request:
  schedule:
    - cron: '25 6 * * *'

The first job will build the images and publish them with a draft-* tag. This lets us try them out in later stages, and also pull them down locally if required.

jobs:
  build-images:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    # We're going to authenticate with AWS, so we'll need an OIDC token
    permissions:
      contents: read
      id-token: write

    # Here's where we can define which runtime(s) and version(s) we're using
    strategy:
      matrix:
        image:
          - runtime: node
            version: 18
          - runtime: node
            version: 20
          - runtime: node-ci-cd
            version: 18
          - runtime: node-ci-cd
            version: 20
          - runtime: python
            version: 3.11
          - runtime: python
            version: 3.12
      fail-fast: false

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Authenticate with AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::$:role/github-actions
          aws-region: eu-west-1

      - name: Login to Amazon ECR
        id: login_to_ecr
        uses: aws-actions/amazon-ecr-login@v2

      # We build images for multiple platforms, and use QEMU for platforms
      # that we're not running on
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build, tag, and push draft image to Amazon ECR
        env:
          REGISTRY: $
          RUNTIME: $
          VERSION: $
        working-directory: runtimes/$
        run: |
          docker buildx build \
            --platform linux/amd64,linux/arm64 \
            --tag ${REGISTRY}/${RUNTIME}:draft-${VERSION}-$ \
            --build-arg RUNTIME=${RUNTIME} \
            --build-arg VERSION=${VERSION} \
            --push \
            .

The next job will pull down the image we just built, and will run the self test script that’s defined in the Dockerfile. The self test script could do things like:

Ensure certain language or runtime features are available;
Install a package to verify the package manager is working;
Check the runtime version that’s installed, and where it’s installed;
Check the package manager version that’s installed, and where it’s installed.

  verify-images:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    needs:
      - build-images

    permissions:
      contents: read
      id-token: write

    strategy:
      matrix:
        image:
          - runtime: node
            version: 18
          - runtime: node
            version: 20
          - runtime: node-ci-cd
            version: 18
          - runtime: node-ci-cd
            version: 20
          - runtime: python
            version: 3.11
          - runtime: python
            version: 3.12
      fail-fast: false

    steps:
      - name: Authenticate with AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::$:role/github-actions
          aws-region: eu-west-1

      - name: Login to Amazon ECR
        id: login_to_ecr
        uses: aws-actions/amazon-ecr-login@v2

      # The 'setup-test' script is setup as a command in the `Dockerfile`
      - name: Run verification script
        env:
          REGISTRY: $
          RUNTIME: $
          VERSION: $
        run: docker run --entrypoint self-test ${REGISTRY}/${RUNTIME}:draft-${VERSION}-$

The final job then tags the image as the final version, and cleans up the draft-* tag.

  push-images:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    needs:
      - verify-images

    permissions:
      contents: read
      id-token: write

    strategy:
      matrix:
        image:
          - runtime: node
            version: 18
          - runtime: node
            version: 20
          - runtime: node-ci-cd
            version: 18
          - runtime: node-ci-cd
            version: 20
          - runtime: python
            version: 3.11
          - runtime: python
            version: 3.12
      fail-fast: false

    if: (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'schedule'

    steps:
      - name: Authenticate with AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::$:role/github-actions
          aws-region: eu-west-1

      - name: Login to Amazon ECR
        id: login_to_ecr
        uses: aws-actions/amazon-ecr-login@v2

      # We can use this command to avoid pulling, tagging, and pushing the image
      - name: Publish Docker image without draft prefix
        env:
          REGISTRY: $
          RUNTIME: $
          VERSION: $
        run: docker buildx imagetools create --tag ${REGISTRY}/${RUNTIME}:${VERSION} ${REGISTRY}/${RUNTIME}:draft-$-$

      # Now we use ECR's API via the AWS CLI to clean up the tag
      - name: Remove draft tag
        env:
          REGISTRY: $
          RUNTIME: $
          VERSION: $
        run: |
          aws ecr \
            batch-delete-image \
            --repository-name ${RUNTIME} \
            --image-ids imageTag=draft-${VERSION}-$

The CodeBuild version is considerably more involved, as we’ll build each platform (e.g. ARM / X86) as a separate job. This technique is adapted from the AWS blog post Creating multi-architecture Docker images to support Graviton2 using AWS CodeBuild and AWS CodePipeline.

Let’s start with the build job:

version: 0.2

batch:
  fast-fail: false
  build-graph:
    # This is very similar to the matrix we provided to GitHub Actions, even if
    # it's a bit more verbose. We define each of the runtime(s) and version(s)
    # to build.
    - identifier: node_18_arm
      env:
        type: ARM_CONTAINER
        image: aws/codebuild/amazonlinux2-aarch64-standard:3.0
        variables:
          RUNTIME: node
          VERSION: "18"
          ARCHITECTURE: arm
      ignore-failure: false

    - identifier: node_18_x86
      env:
        type: LINUX_CONTAINER
        variables:
          RUNTIME: node
          VERSION: "18"
          ARCHITECTURE: x86
      ignore-failure: false

    - identifier: node_20_arm
      env:
        type: ARM_CONTAINER
        image: aws/codebuild/amazonlinux2-aarch64-standard:3.0
        variables:
          RUNTIME: node
          VERSION: "20"
          ARCHITECTURE: arm
      ignore-failure: false

    - identifier: node_20_x86
      env:
        type: LINUX_CONTAINER
        variables:
          RUNTIME: node
          VERSION: "20"
          ARCHITECTURE: x86
      ignore-failure: false

    - identifier: node_ci_cd_18_arm
      env:
        type: ARM_CONTAINER
        image: aws/codebuild/amazonlinux2-aarch64-standard:3.0
        variables:
          RUNTIME: node-ci-cd
          VERSION: "18"
          ARCHITECTURE: arm
      ignore-failure: false

    - identifier: node_ci_cd_18_x86
      env:
        type: LINUX_CONTAINER
        variables:
          RUNTIME: node-ci-cd
          VERSION: "18"
          ARCHITECTURE: x86
      ignore-failure: false

    - identifier: node_ci_cd_20_arm
      env:
        type: ARM_CONTAINER
        image: aws/codebuild/amazonlinux2-aarch64-standard:3.0
        variables:
          RUNTIME: node-ci-cd
          VERSION: "20"
          ARCHITECTURE: arm
      ignore-failure: false

    - identifier: node_ci_cd_20_x86
      env:
        type: LINUX_CONTAINER
        variables:
          RUNTIME: node-ci-cd
          VERSION: "20"
          ARCHITECTURE: x86
      ignore-failure: false

    - identifier: python_3_11_arm
      env:
        type: ARM_CONTAINER
        image: aws/codebuild/amazonlinux2-aarch64-standard:3.0
        variables:
          RUNTIME: python
          VERSION: "3.11"
          ARCHITECTURE: arm
      ignore-failure: false

    - identifier: python_3_11_x86
      env:
        type: LINUX_CONTAINER
        variables:
          RUNTIME: python
          VERSION: "3.11"
          ARCHITECTURE: x86
      ignore-failure: false

    - identifier: python_3_12_arm
      env:
        type: ARM_CONTAINER
        image: aws/codebuild/amazonlinux2-aarch64-standard:3.0
        variables:
          RUNTIME: python
          VERSION: "3.12"
          ARCHITECTURE: arm
      ignore-failure: false

    - identifier: python_3_12_x86
      env:
        type: LINUX_CONTAINER
        variables:
          RUNTIME: python
          VERSION: "3.12"
          ARCHITECTURE: x86
      ignore-failure: false

phases:
  pre_build:
    commands:
      - echo Login in to Amazon ECR
      - aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $REGISTRY

  build:
    commands:
      - echo Check Docker version
      - docker version

      - echo Move into correct directory
      - cd runtimes/$RUNTIME

      - echo Build, tag, and push draft image to Amazon ECR
      - docker buildx build --tag ${REGISTRY}/${RUNTIME}:draft-${VERSION}-${ARCHITECTURE}-${CODEBUILD_RESOLVED_SOURCE_VERSION} --build-arg RUNTIME=${RUNTIME} --build-arg VERSION=${VERSION} --push .

Once built, we can verify each image. I’ve removed the batch configuration to keep this example shorter:

version: 0.2

batch:
  fast-fail: false
  # ... snipped - see GitHub repo ...

phases:
  pre_build:
    commands:
      - echo Login in to Amazon ECR
      - aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin $REGISTRY

  build:
    commands:
      - echo Test built image
      - docker run --entrypoint self-test ${REGISTRY}/${RUNTIME}:draft-${VERSION}-${ARCHITECTURE}-${CODEBUILD_RESOLVED_SOURCE_VERSION}

Finally, we can push the published versions and clean up the draft tags. We don’t have to repeat this for each architecture, so the batch configuration is slightly less involved.

version: 0.2

batch:
  fast-fail: false
  build-graph:
    - identifier: node_18
      env:
        variables:
          RUNTIME: node
          VERSION: "18"
      ignore-failure: false

    - identifier: node_20
      env:
        variables:
          RUNTIME: node
          VERSION: "20"
      ignore-failure: false

    - identifier: node_ci_cd_18
      env:
        variables:
          RUNTIME: node-ci-cd
          VERSION: "18"
      ignore-failure: false

    - identifier: node_ci_cd_20
      env:
        variables:
          RUNTIME: node-ci-cd
          VERSION: "20"
      ignore-failure: false

    - identifier: python_3_11
      env:
        variables:
          RUNTIME: python
          VERSION: "3.11"
      ignore-failure: false

    - identifier: python_3_12
      env:
        variables:
          RUNTIME: python
          VERSION: "3.12"
      ignore-failure: false

phases:
  pre_build:
    commands:
      - echo Login in to Amazon ECR
      - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $REGISTRY

  build:
    commands:
      - echo Publish ARM image
      - docker pull ${REGISTRY}/${RUNTIME}:draft-${VERSION}-arm-${CODEBUILD_RESOLVED_SOURCE_VERSION}
      - docker tag ${REGISTRY}/${RUNTIME}:draft-${VERSION}-arm-${CODEBUILD_RESOLVED_SOURCE_VERSION} ${REGISTRY}/${RUNTIME}:${VERSION}-arm
      - docker push ${REGISTRY}/${RUNTIME}:${VERSION}-arm

      - echo Publish X86 image
      - docker pull ${REGISTRY}/${RUNTIME}:draft-${VERSION}-x86-${CODEBUILD_RESOLVED_SOURCE_VERSION}
      - docker tag ${REGISTRY}/${RUNTIME}:draft-${VERSION}-x86-${CODEBUILD_RESOLVED_SOURCE_VERSION} ${REGISTRY}/${RUNTIME}:${VERSION}-x86
      - docker push ${REGISTRY}/${RUNTIME}:${VERSION}-x86

      - echo Create multi-arch image
      - docker manifest create ${REGISTRY}/${RUNTIME}:${VERSION} ${REGISTRY}/${RUNTIME}:${VERSION}-arm ${REGISTRY}/${RUNTIME}:${VERSION}-x86

      - echo Publish image
      - docker manifest push ${REGISTRY}/${RUNTIME}:${VERSION}

      - echo Delete draft tags
      - aws ecr batch-delete-image --repository-name ${RUNTIME} --image-ids imageTag=draft-${VERSION}-arm-${CODEBUILD_RESOLVED_SOURCE_VERSION} imageTag=draft-${VERSION}-x86-${CODEBUILD_RESOLVED_SOURCE_VERSION}

Conclusion

With a little help from a CI/CD system, we can keep a set of Docker images that fit our team’s needs up-to-date and minimise the number of active vulnerabilities in them. Our service image Dockerfiles can avoid some repetition, as they’ll know that the OS packages are already up-to-date and ready to go.

Check out the GitHub repo (shersoft-ltd/evergreen-ci-and-base-images) to see the full examples in GitHub Actions and CodePipeline/CodeBuild.

Tracking technical initiatives with Backstage.io

Thu, 23 Nov 2023 21:09:00 +0000

In this article, we’re going to cover using the developer portal or service catalogue Backstage.io to track technical initiatives as they’re rolled out across a business.

Why use a tool like Backstage.io for this?

Change in technical ways of working is inevitable, but tracking progress manually with a spreadsheet or document requires a lot of error-prone and tedious effort by a human. As the driver or responsible person for these initiatives, we might miss a service/project that needs to change, and could end up as the sole person who’s updating the tracker. A service catalogue like Backstage.io lets teams self-serve this information, including using APIs to consume it programmatically and to trigger automation if they wish. Anchoring change around Backstage helps bake the tool into the day-to-day work that teams perform, including encouraging them to clearly describe and document the services they produce.

What are our aims of building this automation?

Here are some example goals we might have when deploying this solution:

Developers and tech leads can self-serve information about their own services to track their progress in rolling out technical initiatives or change.
- They can update their solutions with the new ways of working and see their projects automatically marked as compliant / updated, without any interaction with a person or team.
Central governance functions like an architecture team, engineering managers and platform teams can view the progress over all services, including finding the responsible party for any given service/project.
Encouraging more regular use of Backstage improves the discoverability and documentation of all services. Teams begin to use it as a place to find information and service owners in response to alerts/notifications.

What kind of technical initiatives could we track this way?

Deprecations of application runtimes, e.g. NodeJS 16 becoming end-of-life.
The adoption of a new CI/CD tool.
The adoption of CI/CD components, i.e.: custom GitHub Actions, templates or standardised Docker base images.
A change in Infrastructure as Code (IaC) tool, for example moving from AWS’ CDK to Terraform.

What does Backstage give us out of the box?

I’d categorise Backstage as an open-source project that you fork and customise, as opposed to a turn-key solution that’s ready to work for your use case. We’ll be using the GitHub integration to discover users, groups and repos in our organization, and then we’ll enrich and add to this data.

The customisability is a double-edged sword: it may act as a ‘hook’ where a platform team or interested individuals can build interesting functionality, but there’s a decent learning curve and plenty to build to harden the default offering.

Let’s build!

The final solution discussed here is provided as proof-of-concept quality code in this GitHub repo. I’d suggest that you use that as a baseline rather than try and recreate it step-by-step with the following instructions.

We’ll start off by creating a new Backstage project:

npx @backstage/create-app@latest

With that out the way, it’s easy to get started and to view your service catalogue:

cd my-backstage-app
yarn dev # will open a browser

Let’s start by giving our service catalogue a nicer name:

diff --git a/app-config.yaml b/app-config.yaml
index 08d2abf..bb7ee71 100644
--- a/app-config.yaml
+++ b/app-config.yaml
@@ -1,9 +1,9 @@
 app:
-  title: Scaffolded Backstage App
+  title: Shersoft Service Catalogue
   baseUrl: http://localhost:3000

 organization:
-  name: My Company
+  name: Shersoft Ltd

We’ll use a Postgres database to store our Backstage data. Create a Docker compose config file for testing in development (docker-compose.yml):

version: '3.8'

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: service_catalogue
      POSTGRES_USER: service_catalogue
      POSTGRES_PASSWORD: service_catalogue
    ports:
      - 5432:5432

After a quick docker compose up, we can tell Backstage to use our database rather than the default one that’s run in-memory:

diff --git a/app-config.yaml b/app-config.yaml
index 08d2abf..bb7ee71 100644
--- a/app-config.yaml
+++ b/app-config.yaml
@@ -18,38 +18,46 @@ backend:
     # Uncomment the following host directive to bind to specific interfaces
     # host: 127.0.0.1
   csp:
-    connect-src: ["'self'", 'http:', 'https:']
+    connect-src: [ "'self'", 'http:', 'https:' ]
     # Content-Security-Policy directives follow the Helmet format: https://helmetjs.github.io/#reference
     # Default Helmet Content-Security-Policy values can be removed by setting the key to false
   cors:
     origin: http://localhost:3000
-    methods: [GET, HEAD, PATCH, POST, PUT, DELETE]
+    methods: [ GET, HEAD, PATCH, POST, PUT, DELETE ]
     credentials: true
   # This is for local development only, it is not recommended to use this in production
   # The production database configuration is stored in app-config.production.yaml
   database:
-    client: better-sqlite3
-    connection: ':memory:'
+    client: pg
+    connection:
+      host: localhost
+      user: service_catalogue
+      password: service_catalogue
+      port: 5432
+    plugin:
+      auth:
+        connection:
+          database: service_catalogue
   # workingDirectory: /tmp # Use this to configure a working directory for the scaffolder, defaults to the OS temp-dir

Next, create a GitHub Application in your GitHub organization that has at least the following permissions:

Read access to commit statuses, members, and metadata
Read access to actions, code, and pull requests

We’ll use environment variables to pass in the required data. I’m using an IntelliJ-based editor which makes this easy to do in a run configuration, but you may wish to adopt a library like dotenv if your editor doesn’t make this easy.

If you need help setting up the GitHub Application, check out GitHub Apps in their docs.

Update the integration in your app-config.yaml:

backend:
  integrations:
    github:
      apps:
        - appId: ${AUTH_GITHUB_APP_ID}
          webhookUrl: ${AUTH_GITHUB_WEBHOOK_URL}
          clientId: ${AUTH_GITHUB_CLIENT_ID}
          clientSecret: ${AUTH_GITHUB_CLIENT_SECRET}
          webhookSecret: ${AUTH_GITHUB_WEBHOOK_SECRET}
          privateKey: ${AUTH_GITHUB_PRIVATE_KEY}

While we’re at it, we can use the same GitHub App to authenticate users with Backstage:

backend:
  auth:
    providers:
      github:
        development:
          clientId: ${AUTH_GITHUB_CLIENT_ID}
          clientSecret: ${AUTH_GITHUB_CLIENT_SECRET}

    # Please, please, change this to a randomly generated secret in an env var!
    session:
      secret: test123

NB: This is where we run into a small but unfriendly footgun - as what Backstage would call the “integrators”, it’s our job to ensure that auth(n|z) actually happens. Even though we’ve configured authentication above, all API endpoints are still open to whoever can reach them. Do not host your Backstage instance on the internet!

Let’s add some form of protection to the endpoints by verifying the user:

diff --git a/packages/backend/src/index.ts b/packages/backend/src/index.ts
index 04c4ff9..06534ac 100644
--- a/packages/backend/src/index.ts
+++ b/packages/backend/src/index.ts
@@ -30,7 +30,12 @@ import techdocs from './plugins/techdocs';
 import search from './plugins/search';
 import { PluginEnvironment } from './types';
 import { ServerPermissionClient } from '@backstage/plugin-permission-node';
-import { DefaultIdentityClient } from '@backstage/plugin-auth-node';
+import {
+  DefaultIdentityClient,
+  IdentityApi,
+} from '@backstage/plugin-auth-node';
+import { Request, Response, NextFunction } from 'express';
+import { Logger } from 'winston';

 function makeCreateEnv(config: Config) {
   const root = getRootLogger();
@@ -71,6 +76,51 @@ function makeCreateEnv(config: Config) {
   };
 }

+function buildAuthMiddleware(identity: IdentityApi, logger: Logger) {
+  return function authMiddleware(
+    req: Request,
+    res: Response,
+    next: NextFunction,
+  ) {
+    if (req.originalUrl.startsWith('/api/auth')) {
+      return next();
+    }
+
+    if (req.ip === '127.0.0.1' || req.ip === '::1') {
+      if (logger.isLevelEnabled('debug')) {
+        logger.debug('loopback request - allowing');
+      }
+
+      return next();
+    }
+
+    identity
+      .getIdentity({ request: req })
+      .then(result => {
+        if (result?.identity) {
+          if (logger.isLevelEnabled('info')) {
+            logger.info('user is authenticated', {
+              identity: result.identity,
+            });
+          }
+
+          next();
+        } else {
+          if (logger.isLevelEnabled('info')) {
+            logger.info('user is not authenticated');
+          }
+
+          res.status(401);
+          res.json({ error: 'Unauthorized.' });
+        }
+      })
+      .catch(err => {
+        logger.warn('failed to authenticate user', { err });
+        next();
+      });
+  };
+}
+
 async function main() {
   const config = await loadBackendConfig({
     argv: process.argv,
@@ -86,7 +136,10 @@ async function main() {
   const searchEnv = useHotMemoize(module, () => createEnv('search'));
   const appEnv = useHotMemoize(module, () => createEnv('app'));

+  const identity = createEnv('auth').identity;
+
   const apiRouter = Router();
+  apiRouter.use(buildAuthMiddleware(identity, appEnv.logger));
   apiRouter.use('/catalog', await catalog(catalogEnv));
   apiRouter.use('/scaffolder', await scaffolder(scaffolderEnv));
   apiRouter.use('/auth', await auth(authEnv));

OK - back to the GitHub setup. In app-config.yaml, we’re going to add the GitHub provider:

catalog:
  providers:
    github:
      # the provider ID can be any camelCase string
      shersoftLtdGitHub:
        organization: 'shersoft-ltd' # string
        catalogPath: '/catalog-info.yaml' # string
        filters:
          branch: 'main' # string
          repository: '.*' # Regex

Change the provider ID and organization name to fit your context.

We’ll also want to use the Postgres database to hold our search-engine data:

diff --git a/packages/backend/src/plugins/search.ts b/packages/backend/src/plugins/search.ts
index 467ac60..c980f52 100644
--- a/packages/backend/src/plugins/search.ts
+++ b/packages/backend/src/plugins/search.ts
@@ -8,14 +8,17 @@ import { PluginEnvironment } from '../types';
 import { DefaultCatalogCollatorFactory } from '@backstage/plugin-search-backend-module-catalog';
 import { DefaultTechDocsCollatorFactory } from '@backstage/plugin-search-backend-module-techdocs';
 import { Router } from 'express';
+import { PgSearchEngine } from '@backstage/plugin-search-backend-module-pg';

 export default async function createPlugin(
   env: PluginEnvironment,
 ): Promise<Router> {
   // Initialize a connection to a search engine.
-  const searchEngine = new LunrSearchEngine({
+  const searchEngine = await PgSearchEngine.fromConfig(env.config, {
+    database: env.database,
     logger: env.logger,
   });
+
   const indexBuilder = new IndexBuilder({
     logger: env.logger,
     searchEngine,

The last step will be adding a GitHub sign-in button to the frontend:

diff --git a/packages/app/src/App.tsx b/packages/app/src/App.tsx
index 8d62f29..880b4e4 100644
--- a/packages/app/src/App.tsx
+++ b/packages/app/src/App.tsx
@@ -27,12 +27,17 @@ import { entityPage } from './components/catalog/EntityPage';
 import { searchPage } from './components/search/SearchPage';
 import { Root } from './components/Root';

-import { AlertDisplay, OAuthRequestDialog } from '@backstage/core-components';
+import {
+  AlertDisplay,
+  OAuthRequestDialog,
+  SignInPage,
+} from '@backstage/core-components';
 import { createApp } from '@backstage/app-defaults';
 import { AppRouter, FlatRoutes } from '@backstage/core-app-api';
 import { CatalogGraphPage } from '@backstage/plugin-catalog-graph';
 import { RequirePermission } from '@backstage/plugin-permission-react';
 import { catalogEntityCreatePermission } from '@backstage/plugin-catalog-common/alpha';
+import { githubAuthApiRef } from '@backstage/core-plugin-api';

 const app = createApp({
   apis,
@@ -53,6 +58,20 @@ const app = createApp({
       catalogIndex: catalogPlugin.routes.catalogIndex,
     });
   },
+  components: {
+    SignInPage: props => (
+      <SignInPage
+        {...props}
+        auto
+        provider={{
+          id: 'github-auth-provider',
+          title: 'GitHub',
+          message: 'Sign in using GitHub',
+          apiRef: githubAuthApiRef,
+        }}
+      />
+    ),
+  },
 });

 const routes = (

OK, that’s the basics done! What next?

Discovering entities

Backstage tracks entities of various types. They could be things like our services, the infrastructure components that form them, or even the team of people that own them.

We’re going to demonstrate a use-case in which a technical leader wants to track which NodeJS runtimes are use in Lambda functions that have been created by CloudFormation projects. This could be used when a runtime version is nearing end-of-life, and we want to give teams an early nudge to upgrade.

Here’s what this will look like in the Backstage user-interface when it’s done:

In AWS, we’ll have Backstage assume a role to describe information about our organization, and then we’ll have it assume a role in each account to perform its discovery:

Inside Backstage, our discovery process will happen in two steps:

We’ll create an “Entity Provider” which will scan for CloudFormation Stacks in each account and produce a Location.

A Location is a resource that identifies something like a GitHub repo or a file containing some Backstage entities.
We’ll create an “Entity Processor” which will read each of our Locations to find the resources there and create further Backstage entities for them.

Building the entity provider

A quick reminder: all this code is available in this GitHub repo.

Our entity provider starts with a lot of glue code that’s not very interesting, but it’s initialised as follows:

new TechMaturityEntityProvider(
  // A Winston-based logger
  options.logger,

  // The CRON schedule for how often we scrape AWS
  options.schedule,

  // The AWS region(s) we're interested in
  regions,

  // AWS SDK v3 for NodeJS - client used to list accounts in the org
  new OrganizationsClient({
    credentials: topLevelCredentials,
  }),

  // The role name we'll form into an ARN for each account
  destinationRoleName,
)

The topLevelCredentials are derived from a role that Backstage will assume to read organization data. We do this to allow Backstage to live outside the org management account. Here’s a quick tip, use fromTemporaryCredentials to avoid having to manually assume a role and feed the credentials back into the AWS SDK:

import { fromTemporaryCredentials } from '@aws-sdk/credential-providers';

const topLevelCredentials = fromTemporaryCredentials({
  params: {
    RoleArn: sourceRoleArn,
  },
})

OK - enough of the setup! Let’s head to the refresh method that Backstage will call on our chosen schedule. We begin by listing all AWS accounts using the built-in pagination utilities:

const listAccounts = paginateListAccounts(
  { client: this.organizationsClient },
  {},
);
const accounts: Account[] = [];

for await (const result of listAccounts) {
  result.Accounts?.forEach(account => accounts.push(account));
}

this.logger.info('loaded AWS accounts from the organization', {
  numAccounts: accounts.length,
});

With that done, we scrape each account for all of its CloudFormation stacks:

const stacks: DeferredEntity[] = [];

await pMap(
  accounts,
  async account => {
    try {
      for (const region of this.regions) {
        const roleArn = `arn:aws:iam::${account.Id}:role/${this.destinationRoleName}`;

        const cloudFormationClient = new CloudFormationClient({
          credentials: fromTemporaryCredentials({
            masterCredentials: this.organizationsClient.config.credentials,
            params: {
              RoleArn: roleArn,
            },
          }),
          region,
        });

        const describeStacks = paginateDescribeStacks(
          { client: cloudFormationClient },
          {},
        );

        for await (const result of describeStacks) {
          for (const stack of result.Stacks || []) {
            if (!IN_SCOPE_STACK_STATUSES.includes(stack.StackStatus!)) {
              this.logger.debug(
                "not considering stack because it's not in a status we accept",
                { account, stack },
              );

              return;
            }

            stacks.push({
              locationKey: this.getProviderName(),
              entity: locationSpecToLocationEntity({
                location: {
                  type: TechMaturityEntityProvider.CloudFormationStackLocationType,
                  target: JSON.stringify({
                    accountId: account.Id,
                    roleArn: roleArn,
                    stackName: encodeURIComponent(stack.StackName!),
                    region,
                  }),
                },
              }),
            });

            this.logger.debug('discovered CloudFormation Stack', {
              account,
              stack,
            });
          }
        }
      }

      this.logger.info('processed AWS account', { account });
    } catch (err) {
      this.logger.warn('failed to detect resources in AWS account', {
        account,
        err,
      });
    }
  },
  { concurrency: 3 },
);

We’ll break the above code down into sections as this is a rather lengthy method. If you’re using this code to form a production-grade service catalogue solution, I think it could benefit from some refactoring! The same fromTemporaryCredentials method is used again to chain together the role we’ve assumed in the org management account and the role in each destination account:

const roleArn = `arn:aws:iam::${account.Id}:role/${this.destinationRoleName}`;

const cloudFormationClient = new CloudFormationClient({
  credentials: fromTemporaryCredentials({
    masterCredentials: this.organizationsClient.config.credentials,
    params: {
      RoleArn: roleArn,
    },
  }),
  region,
});

The SDK client setup is followed by some boilerplate code to list all the CloudFormation stacks in the account and to ensure they’re in a suitable state:

const describeStacks = paginateDescribeStacks(
  { client: cloudFormationClient },
  {},
);

for await (const result of describeStacks) {
  for (const stack of result.Stacks || []) {
    if (!IN_SCOPE_STACK_STATUSES.includes(stack.StackStatus!)) {
      this.logger.debug(
        "not considering stack because it's not in a status we accept",
        { account, stack },
      );

      return;
    }

    // Create a location here

    this.logger.debug('discovered CloudFormation Stack', {
      account,
      stack,
    });
  }
}

For each stack, we build a Location entity ready to give to Backstage:

// This replaces the "Create a location here" comment above
stacks.push({
  locationKey: this.getProviderName(),
  entity: locationSpecToLocationEntity({
    location: {
      type: TechMaturityEntityProvider.CloudFormationStackLocationType,
      target: JSON.stringify({
        accountId: account.Id,
        roleArn: roleArn,
        stackName: encodeURIComponent(stack.StackName!),
        region,
      }),
    },
  }),
});

We’ll use the target parameter in our processor to identify the stack and the account it’s located in.

When each account has been scraped, we’re left with a set of entities we can return to Backstage:

await this.connection?.applyMutation({
  type: 'full',
  entities: stacks,
});

The use of type: 'full' tells Backstage that it can discard the entities we are no longer returning. And that’s it for the entity provider! Let’s move on to the processor to see how those Location entities are turned into further entities.

Building the entity processor

We can then build a class that implements CatalogProcessor. The method we’re interested in is called readLocation, and will be called for every Location entity registered to Backstage, even ones that we’re not interested in. We start the readLocation method by filtering entities down to just ones that we’ve made in our EntityProvider.

async function readLocation(
  location: LocationSpec,
  _optional: boolean,
  emit: CatalogProcessorEmit,
  _parser: CatalogProcessorParser,
  _cache: CatalogProcessorCache,
) {
  if (
    location.type !==
    TechMaturityEntityProvider.CloudFormationStackLocationType
  ) {
    return false;
  }

  // Do some stuff
}

We’ll start by reading the configuration we saved in the target field of the Location, and we’ll use it to build an AWS SDK client:

const { accountId, roleArn, stackName, region } = JSON.parse(
  location.target,
);

const logger = this.logger.child({
  roleArn,
  stackName,
  accountId,
  region,
});

logger.info('reading location', {
  location,
});

const cloudFormationClient = new CloudFormationClient({
  credentials: fromTemporaryCredentials({
    masterCredentials: this.topLevelCredentials,
    params: {
      RoleArn: roleArn,
    },
  }),
});

With a client authenticated in our destination account, we can read the resources and template of the stack:

const stack = await cloudFormationClient.send(
  new DescribeStacksCommand({ StackName: stackName }),
);

if (!stack.Stacks?.[0]?.StackStatus?.includes('COMPLETE')) {
  logger.info("not reading stack as it's not in a supported state", {
    status: stack.Stacks?.[0]?.StackStatus,
  });

  return true;
}

const template = await cloudFormationClient.send(
  new GetTemplateCommand({
    StackName: stackName,
  }),
);

if (!template.TemplateBody) {
  throw new Error(
    `CloudFormation stack ${stackName} (account = ${accountId}, region = ${region}) missing template body.`,
  );
}

Backstage entities must have unique names, and we’re quite constrained by the characters we can use and how many we’re allowed. We’ll hash together parts of the CloudFormation stack’s identifiers to form a string that’s no more than 63 characters long:

const stackResourceName =
  'aws-cfn-' +
  crypto
    .createHash('shake256', { outputLength: 27 })
    .update(Buffer.from(stack.Stacks![0]!.StackId!, 'utf-8'))
    .digest()
    .toString('hex');

With that done, we’ve got a fairly lengthy bit of code to build a Backstage entity that describes our CloudFormation stack, including linking it to the Location:

emit(
  processingResult.entity(location, {
    apiVersion: 'backstage.io/v1alpha1',
    kind: 'Resource',
    metadata: {
      // Use the name we generated based on a hash above
      name: stackResourceName,

      description: `Auto-detected AWS CloudFormation Stack: ${stackName}`,

      // Annotations can be longer, so store information about the stack here
      annotations: {
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}region`]:
          region,
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}lookedUpWith`]:
          roleArn,
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}accountId`]:
          accountId,
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}cloudFormationStackName`]:
          stackName,
      },

      // Give users in the UI an easy way to get to the stack
      links: [
        {
          url: `https://${region}.console.aws.amazon.com/cloudformation/home?region=${region}#/stacks/stackinfo?filteringText=&filteringStatus=active&viewNested=true&stackId=${encodeURIComponent(
            stack.Stacks![0].StackId!,
          )}`,
          title: 'CloudFormation stack in the AWS console',
        },
      ],
    },
    spec: {
      type: 'aws-cloudformation-stack',

      // We need to provide these fields, so base them from stack tags
      lifecycle: tags['shersoft-ltd:backstage:lifecycle'] || 'production',
      owner: tags['shersoft-ltd:backstage:owner'] || 'platform',
      dependencyOf: tags['shersoft-ltd:backstage:project']
        ? tags['shersoft-ltd:backstage:project']
        : [],
    },
  }),
);

The final step is to create an entity for each Lambda function that’s part of the CloudFormation stack, and to ensure it has a relationship with the stack:

emit(
  processingResult.entity(location, {
    apiVersion: 'backstage.io/v1alpha1',
    kind: 'Resource',
    metadata: {
      name: lambdaResourceName,
      description: `Auto-detected AWS Lambda function: ${resource.PhysicalResourceId}`,
      annotations: {
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}region`]:
          region,
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}lookedUpWith`]:
          roleArn,
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}accountId`]:
          accountId,
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}functionName`]:
          resource.PhysicalResourceId!,
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}cloudFormationStackName`]:
          stackName,
        [`${TechMaturityCatalogProcessor.AnnotationPrefix}cloudFormationLogicalId`]:
          resource.LogicalResourceId!,
      },
      links: [
        {
          url: `https://${region}.console.aws.amazon.com/cloudformation/home?region=${region}#/stacks/stackinfo?filteringText=&filteringStatus=active&viewNested=true&stackId=${encodeURIComponent(
            stack.Stacks![0].StackId!,
          )}`,
          title: 'CloudFormation stack in the AWS console',
        },
      ],
    },
    spec: {
      type: 'aws-lambda-function',
      lifecycle:
        tags['shersoft-ltd:backstage:lifecycle'] || 'production',
      owner: tags['shersoft-ltd:backstage:owner'] || 'platform',
      dependencyOf: [
        ...(tags['shersoft-ltd:backstage:project']
          ? tags['shersoft-ltd:backstage:project']
          : []),
        // Create a relationship between this function and the stack
        `resource:${stackResourceName}`,
      ],

      // Here's where we link it to the Lambda runtime
      dependsOn: [
        `resource:aws-lambda-runtime-${resourceDetail.Properties.Runtime}`,
      ],
    },
  }),
);

There’s plenty more code to arrive at that entity, but it’s not very exciting, and you can always read it in this GitHub repo.

Conclusion

We can create custom EntityProviders and use them to scrape infrastructure or services that exist in our environment. We can process entities by making our own CatalogProcessor, including doing further processing of entities that we didn’t initially discover. For example, we could check if a GitHub project has a CI template we’re rolling out, and link it up to a Backstage Resource that represents it.

Creating relationships between resources in Backstage lets us understand how components are consumed and, importantly, lets us track how they evolve over time.

Running a GitHub Actions pipeline for every AWS account in an organization

Sat, 18 Nov 2023 20:33:00 +0000

If you’re already deploying software with GitHub Actions, you might be wondering if you could use the same mechanism to deploy infrastructure that’s standard across all accounts. AWS provides CloudFormation StackSets for this use-case, but you may be using an alternative Infrastructure as Code (IaC) tool, and want to keep consistency across all your projects.

Start by creating an action in the .github/workflows folder in your project. We’ll call ours github-actions-deploy.yml. We’ll give our action a name, and have it run for merge requests and merges to our default branch:

name: 'Terraform'

on:
  push:
    branches:
      - main
  pull_request:

With that out the way, we’ll add an environment variable for the organization management account ID:

env:
  ORGANIZATION_MANAGEMENT_ACCOUNT_ID: 123123123123

We’ll be using Terraform as our IaC tool, and so we’ll ensure only one instance of the job can run at once:

concurrency: terraform

The string terraform could be anything, but it fits for our use-case. Let’s start the main body of the GitHub Action with a setup job that will look up all the account IDs in the AWS organization:

jobs:
  setup:
    name: 'setup'
    runs-on: ubuntu-22.04

    #
    # Use bash for more advanced shell features and to ensure consistency if we
    # change the base image at a later date.
    #
    defaults:
      run:
        shell: bash

    #
    # We'll be using OIDC-based authentication for AWS, so we need an ID token
    # to be made available in our job.
    #
    # See: https://github.com/aws-actions/configure-aws-credentials
    #
    permissions:
      contents: read
      id-token: write

    #
    # Publish the "account_ids" output once it's been calculated below.
    #
    outputs:
      account_ids: ${{steps.list_accounts.outputs.account_ids}}

    steps:
      #
      # Download the latest code.
      #
      - name: Checkout
        uses: actions/checkout@v4

      #
      # We're using Terraform, so we need to install it.
      #
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.6.3

      #
      # Perform OIDC-based authentication in our AWS management account, using
      # a role called "github-actions".
      #
      - name: Authenticate with AWS
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{env.ORGANIZATION_MANAGEMENT_ACCOUNT_ID}}:role/github-actions
          aws-region: eu-west-1
          mask-aws-account-id: false

      #
      # Find active accounts and share them as an environment variable.
      #
      - name: List accounts
        id: list_accounts
        run: |
          echo "account_ids="$(aws organizations list-accounts | jq '.Accounts | map(select(.Status == "ACTIVE")) | map(select(.Id != "${{env.ORGANIZATION_MANAGEMENT_ACCOUNT_ID}}")) | map(.Id)') >> "$GITHUB_OUTPUT"

To break the final step down further, we start by listing all AWS accounts:

aws organizations list-accounts

With that out the way, we use jq to find only active accounts, filter out the organization management account, and then form a list of IDs.

# Look in the Accounts key of the response from AWS
.Accounts |

# Filter to only active accounts
map(select(.Status == "ACTIVE")) |

# Filter out the management account
map(select(.Id != "${{env.ORGANIZATION_MANAGEMENT_ACCOUNT_ID}}")) |

# Turn it into a list of IDs
map(.Id)'

The output is a JSON array in this form:

[
  "111111111111",
  "222222222222",
  "333333333333",
  "444444444444"
]

To create an environment variable for future steps or jobs to use, we create an entry in the following form and then append it to a file that GitHub defines in the $GITHUB_OUTPUT environment variable.

account_ids=...

The last step is the outputs that we’ve already run into above:

outputs:
  account_ids: ${{steps.list_accounts.outputs.account_ids}}

With the account IDs available, let’s see how we can run a deployment job for each account:

jobs:
  deploy:
    #
    # Create a relationship between these deployment jobs and the setup job to
    # ensure the account_ids variable is made available here.
    #
    needs: setup

    strategy:
      #
      # Don't quit all jobs if one account fails.
      #
      fail-fast: false

      #
      # Create a job for each account ID, and each AWS region that we're
      # interested in.
      #
      matrix:
        account_id: ${{fromJson(needs.setup.outputs.account_ids)}}
        region:
          - eu-west-1

    #
    # Define the rest of your deployment as usual.
    #
    steps:
      - name: Do the AWS stuff

You’ll likely find that you need the account ID and region in your deployment job. Here’s an example of dynamically configuring a Terraform backend and workspace based on the account and region we’re deploying to:

jobs:
  deploy:
    steps:
      - name: Terraform init
        run: |
          terraform init \
            -backend-config="bucket=my-biz-landing-zone-${{env.ORGANIZATION_MANAGEMENT_ACCOUNT_ID}}-${{matrix.region}}-tf-state" \
            -backend-config="region=${{matrix.region}}"
          terraform workspace select -or-create ${{matrix.account_id}}

We could also use the Terraform convention for setting variables based on environment variables:

- name: Terraform plan
  run: terraform plan -out plan
  env:
    TF_VAR_management_account_id: ${{env.ORGANIZATION_MANAGEMENT_ACCOUNT_ID}}
    TF_VAR_account_id: ${{matrix.account_id}}
    TF_VAR_region: ${{matrix.region}}
    TF_VAR_ref: ${{github.ref_name}}

Conclusion

With the AWS CLI, and a bit of JSON wrangling, we can easily run an IaC tool like Terraform against each AWS account in an organization. For the full GitHub Action, check out this gist.

Happy shipping!

Tracking unauthorised AWS API calls to drive platform improvements

Sat, 28 Oct 2023 16:56:00 +0000

In the DevOps space, it’s really tempting to hit everything with the automation stick. However, I’d argue there are plenty of valid reasons for wanting a human in the loop. If you’re part of a platform team, you’ll be used to balancing the needs and desires of multiple teams - especially when it comes to Identity and Access Management (IAM). In this post, we’re going to explore how we can use data you’re (hopefully) already collecting to understand what’s happening in our AWS organization, and to attempt to preempt user needs and requests.

Wanting to skip straight to the code? Check out the solution on GitHub.

CloudTrail

If you’ve missed it, CloudTrail is an AWS Service that records an audit log of API calls. It’s really easy to setup, can aggregate API calls from all accounts in an AWS organization, and is cost-effective to boot.

This article assumes that you’ve configured CloudTrail in the organization management account to aggregate all API calls into a single S3 bucket.

Here’s an example CloudTrail log:

{
  "eventVersion": "1.09",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAT2EYEKAEIO6FE237W:james@jsherz.com",
    "arn": "arn:aws:sts::123123123123:assumed-role/AWSReservedSSO_administrator-access_0c88b0f85221b941/james@jsherz.com",
    "accountId": "123123123123",
    "accessKeyId": "ASIAT2EYEKAEBTRIY4VI",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROAT2EYEKAEIO6FE237W",
        "arn": "arn:aws:iam::123123123123:role/aws-reserved/sso.amazonaws.com/eu-west-1/AWSReservedSSO_administrator-access_0c88b0f85221b941",
        "accountId": "123123123123",
        "userName": "AWSReservedSSO_administrator-access_0c88b0f85221b941"
      },
      "attributes": {
        "creationDate": "2023-10-24T20:20:48Z",
        "mfaAuthenticated": "false"
      }
    }
  },
  "eventTime": "2023-10-24T20:20:50Z",
  "eventSource": "support.amazonaws.com",
  "eventName": "DescribeTrustedAdvisorChecks",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "123.123.123.123",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
  "errorCode": "AccessDenied",
  "errorMessage": "User: arn:aws:sts::123123123123:assumed-role/AWSReservedSSO_administrator-access_0c88b0f85221b941/james@jsherz.com is not authorized to perform: support:DescribeTrustedAdvisorChecks with an explicit deny in a service control policy",
  "requestParameters": null,
  "responseElements": null,
  "requestID": "54d4f88c-3097-449a-8fb7-aa59b37646d2",
  "eventID": "675a51b3-9706-45ec-88ce-1de54a23649f",
  "readOnly": true,
  "eventType": "AwsApiCall",
  "managementEvent": true,
  "recipientAccountId": "123123123123",
  "eventCategory": "Management",
  "tlsDetails": {
    "tlsVersion": "TLSv1.2",
    "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "clientProvidedHostHeader": "support.us-east-1.amazonaws.com"
  },
  "sessionCredentialFromConsole": "true"
}

If we zoom in to the errorMessage, we can see that this API call was blocked as the result of a Service Control Policy (SCP):

User: …james is not authorized to perform: support:DescribeTrustedAdvisorChecks with an explicit deny in a service control policy

What can we learn from unauthorised API calls?

We’re not trying to detect security breaches or users doing things they’re not supposed to in this solution. Instead, we’re looking for the following:

API activity in regions we don’t expect;

This could be user error, or could indicate a region should be enabled / allowed for a particular workload.
Users trying to use new services;

It’s very likely that users will be faster than a platform team to try new services - probably before everything’s in place to allow them access.

If it makes sense for the new service to be enabled in an SCP, we can add this to our todo list and get it enabled before the user demand really kicks in. One example of this is the User Notifications service that AWS launched to let users centralise and customise notifications.
AWS API calls that have changed IAM actions;

AWS occasionally switches to new IAM actions for the same operations, or to achieve the same objective. An example of this would be the AWS portal changes that introduced the migration from aws-portal to billing, accounts, purchase-orders, and tax. Users might hit these permissions as they’re navigating around the AWS console, but still be able to achieve their goals. If this happens, they might not immediately report a problem, but we could still do with fixing the broken access.
New API calls or IAM actions that have been added to services;

Services are often expanding to include new IAM prefixes, for example the addition of pipes:*, scheduler:* or schemas:* to EventBridge. As above, a user clicking around in the console might hit these, and we can add them to relevant SCPs nice and quickly.

Solution architecture

When we talk about searching through CloudTrail data at scale, your mind might immediately jump to CloudTrail Lake. CloudTrail Lake lets you run SQL-like queries against CloudTrail data, and can store it for an impressive seven years! If you’re not doing a lot of searching of your CloudTrail data, the $2.50 / GB storage pricing (scales down above 5 TB) might feel a little prohibitive. We’re going to use Athena to search through the CloudTrail data you’ll already have stored in S3. We’ll only be charged per query, so we can try this solution and decide if it provides value before committing to the likes of CloudTrail Lake. The mapping of S3 data to fields in Athena will be stored in Glue, and the whole thing will be orchestrated with EventBridge and Lambda.

Our Lambda function will be triggered on a schedule, and will read the AWS account data from AWS Organizations. We need these account IDs as the S3 data is stored in the following format:

s3://<bucket name>/AWSLogs/<org ID>/<account ID>/CloudTrail/<region>/<YYYY>/<MM>/<DD>/

You can view the solution on GitHub, but we’re setting Athena up to scan only the data that is required to answer our query. If we provide the account IDs and regions, it can automatically calculate which “folders” in S3 it has to look through. It’s even smart enough to turn the timestamp as a date into every required value. For example, this query:

SELECT * FROM my_table
WHERE timestamp >= '2023/10/01'

Would look in all the folders from 2023/10/01, 2023/10/02, 2023/10/03, all the way up to the current date.

Here’s what our full Athena query will look like:

SELECT eventsource, eventname, COUNT(*) as num_occurrences
FROM cloudtrail
WHERE 1 = 1
-- Only look at the last period
AND timestamp >= '${startDate}'
-- Cause Athena to look through each auto-generated partition
AND account_id IN (
    '111111111111',
    '222222222222',
    -- ...
)
AND region IN ('eu-west-1', 'us-east-1')
-- Exclude Config as it's very noisy
AND sourceipaddress != 'config.amazonaws.com'
-- Only denied requests
AND errorcode IS NOT NULL
AND (
    errorcode LIKE '%AccessDenied%'
 OR errorcode LIKE '%Forbidden%'
 OR errorcode LIKE '%Unauthorized%'
)
GROUP BY eventsource, eventname
ORDER BY num_occurrences DESC;

When it’s run successfully, we can send an e-mail report to a mailing list or distribution list of our choice, and review these by hand once a week (or a cadence that makes sense for your context):

That’s a wrap! If you think this could provide value in your team, check out the solution on GitHub.

Making lots of the same thing in IaC (CDK & Terraform)

Wed, 18 Oct 2023 19:09:00 +0000

It’s a common requirement in Infrastructure as Code (IaC) tools like CDK and Terraform to make many of the same thing, with slightly differing properties. Let’s use a fictional example in which we’re writing an audit service that receives notifications from other services when a user performs an action. Here’s our architecture:

The designer of the audit service specified that each incoming service should get its own SQS queue. Let’s model that in IaC.

In CDK, that might look like this:

import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import { Queue } from "aws-cdk-lib/aws-sqs";

const SERVICES = [
    "service-a",
    "service-b",
    "service-d",
    "service-e",
    "service-f",
];

export class AuditServiceStack extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        this.createIncomingQueues();
    }

    private createIncomingQueues() {
        for (let i = 0; i < SERVICES.length; i++) {
            new Queue(this, "incoming-queue-" + i, {
                queueName: `audit-service-incoming-${SERVICES[i]}`,
            });
        }
    }
}

In Terraform, that might look like this:

locals {
  services = [
    "service-a",
    "service-b",
    "service-c",
    "service-d",
    "service-e",
    "service-f",
  ]
}

resource "aws_sqs_queue" "incoming" {
  count = length(local.services)

  name = "audit-service-incoming-${local.services[count.index]}"
}

Both of these code snippets create the same resources, and both have the same caveat. What happens when we decide to retire service C, and thus take it out of our list of services?

CDK

We make the change, run a cdk diff and get:

Stack AuditService
Resources
[-] AWS::SQS::Queue incoming-queue-5 incomingqueue58150E916 destroy
[~] AWS::SQS::Queue incoming-queue-2 incomingqueue2A2310290 replace
 └─ [~] QueueName (requires replacement)
     ├─ [-] audit-service-incoming-service-c
     └─ [+] audit-service-incoming-service-d
[~] AWS::SQS::Queue incoming-queue-3 incomingqueue366F6CA78 replace
 └─ [~] QueueName (requires replacement)
     ├─ [-] audit-service-incoming-service-d
     └─ [+] audit-service-incoming-service-e
[~] AWS::SQS::Queue incoming-queue-4 incomingqueue4F0571194 replace
 └─ [~] QueueName (requires replacement)
     ├─ [-] audit-service-incoming-service-e
     └─ [+] audit-service-incoming-service-f


✨  Number of stacks with differences: 1

Terraform

We make the change, run a terraform plan and get:

aws_sqs_queue.incoming[0]: Refreshing state... [id=https://sqs.eu-west-1.amazonaws.com/123123123123/audit-service-incoming-service-a]
aws_sqs_queue.incoming[4]: Refreshing state... [id=https://sqs.eu-west-1.amazonaws.com/123123123123/audit-service-incoming-service-e]
aws_sqs_queue.incoming[3]: Refreshing state... [id=https://sqs.eu-west-1.amazonaws.com/123123123123/audit-service-incoming-service-d]
aws_sqs_queue.incoming[1]: Refreshing state... [id=https://sqs.eu-west-1.amazonaws.com/123123123123/audit-service-incoming-service-b]
aws_sqs_queue.incoming[5]: Refreshing state... [id=https://sqs.eu-west-1.amazonaws.com/123123123123/audit-service-incoming-service-f]
aws_sqs_queue.incoming[2]: Refreshing state... [id=https://sqs.eu-west-1.amazonaws.com/123123123123/audit-service-incoming-service-c]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_sqs_queue.incoming[2] must be replaced
-/+ resource "aws_sqs_queue" "incoming" {
      ~ arn                               = "arn:aws:sqs:eu-west-1:123123123123:audit-service-incoming-service-c" -> (known after apply)
      + deduplication_scope               = (known after apply)
      + fifo_throughput_limit             = (known after apply)
      ~ id                                = "https://sqs.eu-west-1.amazonaws.com/123123123123/audit-service-incoming-service-c" -> (known after apply)
      ~ kms_data_key_reuse_period_seconds = 300 -> (known after apply)
      ~ name                              = "audit-service-incoming-service-c" -> "audit-service-incoming-service-d" # forces replacement
      + name_prefix                       = (known after apply)
      + policy                            = (known after apply)
      + redrive_allow_policy              = (known after apply)
      + redrive_policy                    = (known after apply)
      ~ sqs_managed_sse_enabled           = true -> (known after apply)
      - tags                              = {} -> null
      ~ tags_all                          = {} -> (known after apply)
      ~ url                               = "https://sqs.eu-west-1.amazonaws.com/123123123123/audit-service-incoming-service-c" -> (known after apply)
        # (7 unchanged attributes hidden)
    }

  # aws_sqs_queue.incoming[3] must be replaced

~~ snipped for brevity ~~

  # aws_sqs_queue.incoming[4] must be replaced

~~ snipped for brevity ~~

  # aws_sqs_queue.incoming[5] will be destroyed

~~ snipped for brevity ~~

  # (because index [5] is out of range for count)

Plan: 3 to add, 0 to change, 4 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Saved the plan to: plan

To perform exactly these actions, run the following command to apply:
    terraform apply "plan"

In both cases, we’re deleting and recreating lots of resources unintentionally. The example we’ve got here would lead to a temporary loss of audit events as we delete and then recreate the queues. We’ll also lose any messages that are currently on the queues and waiting to be processed. Not good!

The CDK-heads reading may be screaming out “Hey James, that’s an unrealistic example!” - and they’d be right to. Bare with me!

This kind of change can easily be missed in non-production environments if they don’t have a lot of traffic going through them, so we have to make sure we consider the impact of our infrastructure changes - seeing no impact in a lower environment is not a guarantee they’ll ‘just work’ in the future.

How can we model these resources differently to avoid this problem?

Using properties in resource identifiers

One option is to use a property of the resource that’s unique for each instance, for example the service name:

CDK

diff --git a/lib/audit-service-stack.ts b/lib/audit-service-stack.ts
index 7ca53e0..c7bb904 100644
--- a/lib/audit-service-stack.ts
+++ b/lib/audit-service-stack.ts
@@ -19,9 +19,9 @@ export class AuditServiceStack extends cdk.Stack {
   }
 
   private createIncomingQueues() {
-    for (let i = 0; i < SERVICES.length; i++) {
-      new Queue(this, "incoming-queue-" + i, {
-        queueName: `audit-service-incoming-${SERVICES[i]}`,
+    for (const service of SERVICES) {
+      new Queue(this, `incoming-queue-${service}`, {
+        queueName: `audit-service-incoming-${service}`,
         removalPolicy: RemovalPolicy.DESTROY,
       });
     }

In the above snippet, we replace the array index in CDK’s logical ID with the name of the service. Doing that in CDK feels a lot more natural to me anyway, but there are cases where you can’t use user input in the logical ID. If a user forms the name with tokens (e.g. Aws.ACCOUNT_ID or a reference to another resource), CDK will complain with the following error:

ID components may not include unresolved tokens: …

If you’re the author of a downstream construct, you may not necessarily control user input (properties), and so I’d recommend avoiding using it in logical IDs.

Terraform

So what does this look like in Terraform? Luckily, we have the for_each operator:

diff --git a/main.tf b/main.tf
index 8fd59bf..ec8121c 100644
--- a/main.tf
+++ b/main.tf
@@ -10,7 +10,7 @@ locals {
 }
 
 resource "aws_sqs_queue" "incoming" {
-  count = length(local.services)
+  for_each = local.services
 
-  name = "audit-service-incoming-${local.services[count.index]}"
+  name = "audit-service-incoming-${each.key}"
 }

Unfortunately, running this gives us an error:

╷
│ Error: Invalid for_each argument
│
│   on main.tf line 13, in resource "aws_sqs_queue" "incoming":
│   13:   for_each = local.services
│     ├────────────────
│     │ local.services is tuple with 6 elements
│
│ The given "for_each" argument value is unsuitable: the "for_each" argument must be a map, or set of strings, and you have provided a value of type tuple.
╵

We can do some kind of value conversion ourselves, but my preference is to have the user input a map of friendly names to resource names:

diff --git a/main.tf b/main.tf
index 8fd59bf..6d40064 100644
--- a/main.tf
+++ b/main.tf
@@ -1,16 +1,16 @@
 locals {
-  services = [
-    "service-a",
-    "service-b",
-    "service-c",
-    "service-d",
-    "service-e",
-    "service-f",
-  ]
+  services = {
+    service_a = "service-a",
+    service_b = "service-b",
+    service_c = "service-c",
+    service_d = "service-d",
+    service_e = "service-e",
+    service_f = "service-f",
+  }
 }
 
 resource "aws_sqs_queue" "incoming" {
-  count = length(local.services)
+  for_each = local.services
 
-  name = "audit-service-incoming-${local.services[count.index]}"
+  name = "audit-service-incoming-${each.value}"
 }

This might feel a little silly in our contrived example, but makes more sense when the input data is not just a simple string value. For example:

locals {
  services = {
    service_a = {
      name            = "service-a",
      allowed_regions = ["eu-west-1", "eu-west-2"],
      account_id      = ["1111111111111"]
    }
    service_b = {
      name            = "service-b",
      allowed_regions = ["eu-west-1"],
      account_ids     = ["222222222222"]
    }
    service_c = {
      name            = "service-c",
      allowed_regions = ["eu-west-1"],
      account_ids     = ["333333333333", "444444444444"]
    }
    service_d = {
      name            = "service-d",
      allowed_regions = ["us-east-1"],
      account_ids     = ["555555555555"]
    }
    service_e = {
      name            = "service-e",
      allowed_regions = ["eu-west-1", "us-east-1"],
      account_ids     = ["666666666666"]
    }
    service_f = {
      name            = "service-f",
      allowed_regions = ["eu-west-1", "eu-west-2"],
      account_ids     = ["777777777777"]
    }
  }
}

Conclusion

Using sets, lists or arrays of items when making many of the same resource requires care when chooing the logical identifier. Aim for a plan or diff that changes the absolute bare minimum, and consider that a lack of complaints or alarms in lower environments does not guarantee success in later ones.

As the designer of Terraform modules or CDK constructs, we have to understand how user input will affect the logical IDs and resource names that we choose.

Right-sizing Lambda functions that process files

Sun, 18 Jun 2023 19:23:00 +0000

You’ve probably seen an architecture that looks like this:

How much memory should we allocate to the Lambda function processing files? One approach is to take some sample data and run it through Lambda Power Tuning, but what if we’re processing files of different sizes? The easy way out is to keep increasing the memory until you can safely handle all uploads, but this comes at a cost. Let’s look at how we can choose the right amount of memory for each of our files!

Is it really worth it?

It’s tempting to add functionality like this during the initial development process, but let’s do the maths to understand if we’ll actually get a return on our time investment.

Example 1 - low traffic upload

At one Request Per Second (RPS), we’ll handle somewhere in the region of two and a half million requests per month. If it takes 256 MB of RAM to process most files, and 1024 MB of RAM to process 10% of the files which happen to be larger, our costs are as follows:

All invocations at 1024 MB, all taking 3s: $98.67 / month
90% invocations at 1024 MB, 10% at 256 MB, all taking 3s: $23.29 / month

That seventy dollar saving does feel nice, but we have to evaluate the developer cost to implement this functionality, and the operational cost of more Lambdas (e.g. using more complex dashboards that show more data).

Example 2 - high traffic upload

At one hundred RPS, the numbers get more interesting:

All invocations at 1024 MB, all taking 3s: $3,410.39 / month
90% invocations at 1024 MB, 10% at 256 MB, all taking 3s: $10,414.36 / month

How about if the big files are only 1% of the uploads?

All invocations at 1024 MB, all taking 3s: $3,410.39 / month
99% invocations at 1024 MB, 1% at 256 MB, all taking 3s: $2,710.55 / month

You’ll have to perform your own calculations to decide what makes sense for your context. Let’s see how we can pick the right amount of Lambda memory for our input file size.

Right-sizing Lambda functions

Before we continue: it’s assumed that you understand the size(s) of data that your function will process, and have performed Lambda Power Tuning to find the most cost-effective memory setting for each category. That process might give you a table that looks like this:

Minimum file size	Maximum file size	Ideal Lambda RAM
0	100 KB	256 MB
100 KB	25 MB	1024 MB
25 MB	Any	4096 MB

The original method of triggering a Lambda function or other event when a file is uploaded to an S3 bucket is to use object notifications, but these don’t let us perform any advanced filtering. In late 2021, AWS released an integration with EventBridge. With that in place, we can create an event rule that routes our object created event to the right place. Here’s what the event looks like:

{
  "version": "0",
  "id": "2ffe83a5-48ff-fccd-2dda-a57a03c22f4b",
  "detail-type": "Object Created",
  "source": "aws.s3",
  "account": "123456789012",
  "time": "2023-06-18T19:17:31Z",
  "region": "eu-west-1",
  "resources": [
    "arn:aws:s3:::right-size-lambdas-files"
  ],
  "detail": {
    "version": "0",
    "bucket": {
      "name": "right-size-lambdas-files"
    },
    "object": {
      "key": "large",
      "size": 31457280,
      "etag": "f6877c498a97505d3a45f485cefc2a40-2",
      "sequencer": "00648F584721ECA86B"
    },
    "request-id": "2JR2MJWXPFG2JME4",
    "requester": "123456789012",
    "source-ip-address": "123.123.123.123",
    "reason": "CompleteMultipartUpload"
  }
}

You’ll see that the event contains a size field in bytes. You may have made an EventBridge rule that looks like this before:

{
  "detail": {
    "bucket": {
      "name": [
        "right-size-lambdas-files"
      ]
    }
  },
  "detail-type": [
    "Object Created"
  ],
  "source": [
    "aws.s3"
  ]
}

But we can’t match an exact number of bytes - it’ll be different every time. Luckily, we can use the numeric operator to match a range. Let’s see what that looks like for the first entry in our table:

{
  "detail": {
    "object": {
      "size": [
        {
          "numeric": [
            ">",
            0,
            "<=",
            100000
          ]
        }
      ]
    }
  }
}

The above event rule captures the smallest files with two conditions. The medium-sized Lambda’s event rule looks nearly identical, just with different size values, and the large files event rule has only a starting range:

{
  "detail": {
    "object": {
      "size": [
        {
          "numeric": [
            ">",
            25000000
          ]
        }
      ]
    }
  }
}

It’ll capture all files over 25 MB.

We can update our architecture diagram to indicate that we’re choosing the right Lambda for the job:

That’s it! Small files go to the Lambda with the least memory, and the biggest files get a full four gigabytes of RAM.

Full working example

You can find a fully working example of this technique on GitHub. It uses exactly the same codebase to create three Lambda functions of different sizes. Give it a try!

Conclusion

At high volumes, it can make sense to route files to an appropriately sized Lambda function. This doesn’t replace Lambda Power Tuning however - we still want to find the most cost-effective memory setting for each group of files we want to process.

With EventBridge, we can filter on ranges of numeric values. This lets us send the right S3 object created event to the appropriate Lambda, with no changes to our Lambda code!

Enforcing AWS tags the right way - without Tag Policies

Thu, 15 Jun 2023 18:12:00 +0000

Tags are essential in AWS. They let us allocate costs to different teams, projects, services or business areas, and can be critical in operational response. Tags get us from a page or security related notification to the right team much faster, especially for resources that don’t support names or those that are named automatically by Infrastructure as Code (IaC) tools like CDK. In Shift security left with AWS Config, I discussed providing near-instant feedback to builders when their resources don’t meet our compliance needs. That whole approach revolves around tags!

With a set of tagging standards agreed in your organisation, your next thought is likely “so how do we enforce these?”, and the obvious looking solution is AWS’ Tag Policies. Let’s examine challenges using Tag Policies and how we can achieve a better result.

What’s so bad about Tag Policies?

Resource support and wildcards

If you read the documentation of supported resources, you’ll immediately see that many AWS resources just aren’t supported. Additionally, there’s a real mix of services that do and don’t support a wildcard syntax, e.g. athena:* to target all Athena resources. This matters as we’re trying to fit as much as we can into the 10,000-character limit applied to each individual policy. How many characters does it take to include all the supported resources, taking advantage of wildcards where possible? 4,681. Per tag! With the rest of the JSON document and the tag name included, we can fit about two tags per policy. We can attach a maximum of ten policies to one organisational unit, so that gives us a maximum of twenty tags with the resources supported today. If you need to exceed that number, you can add layers of tag policies in nested organisational units to overcome the limit. I don’t understand why we can’t have wildcards for every service, or even a *:* to target every resource, but you didn’t come to AWS for great user experience now, did you?!

Bad documentation

An additional challenge with the documentation page linked above is that some of the values are wrong. If we take the complete list of resources and place them into the editor in the UI, we get errors that look like this:

backup:*, elasticmapreduce:* and wisdom:knowledge are specified in the docs, but aren’t accepted by AWS. My preferred approach is to spelunk my way through the source code of the UI to find the real allowed values.

Taking the above list gives us 334 allowed resource types, a far cry from the 290 in the documentation. Check out the full diff at the end of the article if you’re curious about what’s missing.

No tags, no problem!

If we create a resource in AWS with no tags at all, or with a tag in the policy missing, the policies aren’t applied, and thus we get off scot-free. You might be tempted to enforce the use of tags with Service Control Policies (SCPs), but many services and tools apply the tags with a second operation once the resource has been created or updated. The net result? We can’t enforce that a user provides our mandatory tags.

What’s the solution?

We’ve spent a fair amount of time diving into why Tag Policies aren’t a great way to ensure tag compliance, so what can we do instead? Luckily, AWS Config allows us to inspect the tags on a resource and write custom rules to check they’re as we expect. If you haven’t heard of Config, it’s a service that evaluates your resources to decide if they meet certain compliance criteria, for example checking an S3 bucket isn’t open to the public. In Shift security left with AWS Config, I use a feature of Config to write an SQL query that fetches tags for any supported resource. Super handy!

Let’s write a custom Config rule that checks a resource has the tags we require. We’re going to use the CloudFormation Guard Domain-Specific-Language (DSL), but you can also make custom Config rules with Lambda functions.

We’ll start by requiring a tag that we want to enforce for all resource types:

tags["shersoft-ltd:workload:project"] !empty

If you want to follow along in the console while you’re developing your rule, see these steps:

With the rule in place, we can immediately see which resource are and aren’t compliant. We can expand our Config rule to cover some other tags:

tags["shersoft-ltd:workload:project"] !empty

tags["shersoft-ltd:workload:ref"] !empty

tags["shersoft-ltd:devops:environment"] !empty

We could additionally ensure that the tag value matches one of a set of allowed values. Let’s do that with the environment tag:

tags["shersoft-ltd:devops:environment"] == /staging|production/

At this point, you might be wondering how you can debug rules written in the Guard language, especially without incurring the cost of repeated Config evaluations. Luckily there’s a CLI that we can use for this purpose. Install it following the instructions on GitHub. I’m on a Mac, so will just do:

brew install cloudformation-guard

We’ll take the configuration item JSON, in this case a CodeArtifact repo, and place it into resource.json:

{
  "version": "1.3",
  "accountId": "123456789012",
  "configurationItemCaptureTime": "2023-05-24T19:50:28.418Z",
  "configurationItemStatus": "ResourceDiscovered",
  "configurationStateId": "1684957828418",
  "configurationItemMD5Hash": "",
  "arn": "arn:aws:codeartifact:eu-west-1:123456789012:repository/test/repo",
  "resourceType": "AWS::CodeArtifact::Repository",
  "resourceId": "arn:aws:codeartifact:eu-west-1:123456789012:repository/test/repo",
  "resourceName": "repo",
  "awsRegion": "eu-west-1",
  "availabilityZone": "Regional",
  "tags": {},
  "relatedEvents": [],
  "relationships": [],
  "configuration": {
    "RepositoryName": "my-little-repo",
    "Name": "repo",
    "DomainName": "test",
    "DomainOwner": "123456789012",
    "Arn": "arn:aws:codeartifact:eu-west-1:123456789012:repository/test/repo",
    "ExternalConnections": [],
    "Upstreams": [
      "npm-store"
    ],
    "Tags": []
  },
  "supplementaryConfiguration": {},
  "resourceTransitionStatus": "None"
}

You’ll see that it’s got no tags. With that in place, we’ll put the tag rule we’re developing into tag-rule.guard, and run the CLI:

cfn-guard validate --rules tag-rule.guard --data resource.json

We’ll be given detailed debugging information about the rule evaluation - scroll to the right in the code pane to view the full message:

resource.json Status = FAIL
FAILED rules
tag-rule.guard/default    FAIL
---
Evaluation of rules tag-rule.guard against data resource.json
--
Property traversed until [/tags] in data [resource.json] is not compliant with [tag-rule.guard/default] due to retrieval error. Error Message [Could not find key shersoft-ltd:workload:project inside struct at path /tags[L:13,C:10]]
Property traversed until [/tags] in data [resource.json] is not compliant with [tag-rule.guard/default] due to retrieval error. Error Message [Could not find key shersoft-ltd:workload:ref inside struct at path /tags[L:13,C:10]]
Property traversed until [/tags] in data [resource.json] is not compliant with [tag-rule.guard/default] due to retrieval error. Error Message [Could not find key shersoft-ltd:devops:environment inside struct at path /tags[L:13,C:10]]
--

How about with all tags present, and an invalid tag value in environment? Try updating the tags property in resource.json:

{
  // ...
  "tags": {
    "shersoft-ltd:workload:project": "test",
    "shersoft-ltd:workload:ref": "test",
    "shersoft-ltd:devops:environment": "lemons"
  },
  // ...
}

Then re-run the validate command:

cfn-guard validate --rules tag-rule.guard --data resource.json

We get a new error!

... provided value ["lemons"] did not match expected value ["/staging|production/"]...

This allows us to rapidly iterate on our tag Config rule for free. We could even store the configuration items as a set of JSON test fixtures, call the CLI and then verify the resulting output in integration tests.

Tag Policies let us select resource types that we want to enforce the tags for, and we can achieve the same thing in our Config rule:

rule check_retention_tag when resourceType == "AWS::S3::Bucket" {
  tags["shersoft-ltd:compliance:retention"] !empty
}

cfn-guard will now indicate that it didn’t check for that tag:

Rule [tag-rule.guard/check_retention_tag] is not applicable for template [resource.json]

Conclusion

Tag Policies have a number of practical limitations, including their lack of resource support, and requiring SCPs that are incompatible with some IaC tools. We can use a custom Config rule to mandate a set of tags based on the resource type, and achieve tagging harmony across our AWS organisation.

We’re not blocking resources from being created or updated with this solution, but we do get reporting of their compliance. We can combine this solution with the one in Shift security left with AWS Config to give builders near-instant feedback on the resource’s tagging - before their Pull Request lands.

Shift security left with AWS Config

Sat, 27 May 2023 18:17:00 +0000

No-one likes a telling off from the security team, and we can’t rely on good will and experience to keep our infrastructure secure: we have to make comprehensive guardrails. One of the services commonly used to achieve that is AWS Config.

What is AWS Config?

NB: if you’re playing around with AWS Config in a lab environment, please be aware of the pricing. It’s not free and I spent around ~£20 deploying a realistic set of rules to my lab environment (~114 rules, ~7 AWS accounts) and leaving them running for a fortnight.

AWS Config lets you define rules that check your resources are built against best practices - both AWS’ and those of your own organization. Example rules include:

Checking an S3 bucket has a public access block enabled.
Checking a DynamoDB table has Point-in-Time-Recovery (PITR) enabled.
Checking an IAM user has MFA enabled.

The full list is in the docs: List of AWS Config Managed Rules

How are AWS Config rules defined?

There are a number of sources for Config rules. You might define a “conformance pack”, a YAML-based template for a set of rules and their configuration. You could also define an organization conformance pack in your management account and deploy it to all member accounts. Some AWS security services also manage the creation of rules, e.g. the “security standards” that are part of Security Hub. These rule sources can be combined with rules deployed to individual accounts, for example if a workload has stricter compliance needs than the rest of your organization.

Where do AWS Config reports go?

One of the more powerful features of AWS Config is being able to aggregate all findings into a central account in your AWS organization. We can centrally deploy rules, and then have each account report back with non-compliant infrastructure.

So far, so good!

Flow and fast feedback

We’ve had a whistle-stop tour of what Config rules are, how they’re defined, and how reporting is done, but how does that help us as a human building infrastructure? We’re interested in keeping flow with our current piece of work, not on fancy dashboards in a far away account. How can we ensure our infrastructure meets the required standards without an e-mail arriving from another team two weeks after we made it?

Developer-centric tooling

One option we have is to embed a tool like terraform-compliance or cdk-nag in our build pipeline. This is a big help, but there’s some effort required to keep the rules of those tools in-sync with the Config rules to our AWS account.

Feedback in the Pull Request

You can love or hate Pull Requests, but many teams use them as a way to prepare work before it’s checked in to their main branch. What if we could get feedback on our infrastructure before we’ve even hit the big green merge button?

Let’s make it happen!

PS: if you want to skip straight to the code, checkout jSherz/shift-security-left-aws-config on GitHub.

A quick segue into tags

This solution will rely on tags that indicate which project the infrastructure resources are for, and which git branch they’re deployed from. I really hope you’ve already got a tagging standard defined in your organization, but if not there really is no time like the present to start.

No strategy? “Defining and publishing a tagging schema” in the AWS docs is a great place to start, and you’ll notice that the tags below follow a similar standard.

We’re going to use two tags for our resources:

Tag name	Example value	Purpose
jsherz-com:workload:project	git@github.com:jSherz/shift-security-left-aws-config.git	Identifies which GitHub project contains this infrastructure.
jsherz-com:workload:ref	feature/my-cool-thing	Helps us find the right Pull Request for this infrastructure.

The names aren’t important - you can customise the values in the code for this solution - but we want to be able to quickly identify the right project that contains the Infrastructure of Code (IaC) for the resources we build.

Responding when resources fail compliance checks

We want to give a user feedback ASAP when their resources fail a compliance check. We’ll do this by listening for notifications from AWS Config and then triggering a Lambda function which can add a GitHub Pull Request comment. It’s not as fast as the in-editor feedback you’d get with a compiler/linter, or the in-pipeline feedback you’d get with a tool like terraform-compliance, but it gives feedback that’s accurate and up-to-date with our organization-wide standards.

Here’s what that feedback will look like:

If the user can’t identify what the rule relates to by name, they can click the link to see its description in the AWS console. Additionally, they can view the resources itself in the Config portal to find out any other rules that have failed.

Here’s what the architecture that powers that looks like:

Config sends an event to the default EventBridge event bus with the following format:

{
  "version": "0",
  "id": "33a1416f-68e6-6fe1-6313-7074141badb9",
  "detail-type": "Config Rules Compliance Change",
  "source": "aws.config",
  "account": "111111111111",
  "time": "2023-05-11T17:48:07Z",
  "region": "eu-west-1",
  "resources": [],
  "detail": {
    "resourceId": "jsj-bucket-777",
    "awsRegion": "eu-west-1",
    "awsAccountId": "111111111111",
    "configRuleName": "s3-bucket-logging-enabled-conformance-pack-mogxp88qe",
    "recordVersion": "1.0",
    "configRuleARN": "arn:aws:config:eu-west-1:111111111111:config-rule/aws-service-rule/config-conforms.amazonaws.com/config-rule-latfki",
    "messageType": "ComplianceChangeNotification",
    "newEvaluationResult": {
      "evaluationResultIdentifier": {
        "evaluationResultQualifier": {
          "configRuleName": "s3-bucket-logging-enabled-conformance-pack-mogxp88qe",
          "resourceType": "AWS::S3::Bucket",
          "resourceId": "jsj-bucket-777",
          "evaluationMode": "DETECTIVE"
        },
        "orderingTimestamp": "2023-05-11T17:47:56.716Z"
      },
      "complianceType": "NON_COMPLIANT",
      "resultRecordedTime": "2023-05-11T17:48:06.945Z",
      "configRuleInvokedTime": "2023-05-11T17:48:06.801Z"
    },
    "notificationCreationTime": "2023-05-11T17:48:07.470Z",
    "resourceType": "AWS::S3::Bucket"
  }
}

My first instinct when building this project was to use the Resource Groups Tagging API to lookup tags, but this isn’t supported by all AWS services and the event above doesn’t contain the resource ARN. Instead, we can use the query feature of Config to find the tags regardless of the service it originated from:

export class ConfigService implements IConfigService {
    constructor(private readonly configServiceClient: ConfigServiceClient) {
    }

    async lookupResourceTags(
        resourceType: string,
        resourceId: string,
    ): Promise<Record<string, string>> {
        const response = await this.configServiceClient.send(
            new SelectResourceConfigCommand({
                Expression: `
                    SELECT
                      tags
                    WHERE
                      resourceType = '${resourceType}'
                      AND resourceId = '${resourceId}'
                `,
                Limit: 100,
            }),
        );

        /*
            If we got a query result, turn the list of tags into an object
            of tag keys -> tag values.
         */
        if (response.Results && response.Results.length === 1) {
            return (
                JSON.parse(response.Results[0]) as ILookupResourceTagsQueryResult
            ).tags.reduce((out, curr) => {
                out[curr.key] = curr.value;
                return out;
            }, {} as Record<string, string>);
        } else {
            return {};
        }
    }
}

Once we’ve got the tags, we can use the Octokit library for the GitHub API to find a Pull Request for that project/branch combination:

const project = tags[`${companyIdentifier}:workload:project`];
const ref = tags[`${companyIdentifier}:workload:ref`];

if (ref === "main") {
    logger.info("not for a pull request - skipping", {ref});
    return;
}

const [owner, repo] = project
    .replace("git@github.com:", "")
    .replace(".git", "")
    .split("/");

const relevantPullsResponse = await octokit.request(
    "GET /repos/{owner}/{repo}/pulls?state={state}",
    {
        owner,
        repo,
        state: "open",
        head: ref,
        headers: {
            "X-GitHub-Api-Version": "2022-11-28",
        },
    },
);

If one is found, we can add a comment to the PR:

 if (relevantPulls.length === 1) {
    const issueNumber = relevantPulls[0].number;

    await octokit.request(
        "POST /repos/{owner}/{repo}/issues/{issue_number}/comments",
        {
            owner,
            repo,
            issue_number: issueNumber,
            body: comment,
            headers: {
                "X-GitHub-Api-Version": "2022-11-28",
            },
        },
    );
}

Putting it all together: the GitHub App

If you want to get stuck in to the code behind this solution, you can view the jSherz/shift-security-left-aws-config project on GitHub.

Let’s setup a GitHub App in our GitHub organization. It’ll be private and thus not available for any user/organization on GitHub to install.

Create the app in your GitHub organization:
Use a service that can capture requests, e.g. webhook.site to listen for the app being installed in step 4:
Allow read/write access of issues and pull requests:
Install the application:
Note down the installation ID - we’ll need it later.

Putting it all together - AWS infrastructure

With the app setup, we can complete the final step of deploying the jSherz/shift-security-left-aws-config project. I’ll leave you to the instructions in that README to get the Lambda function deployed and to configure GitHub API access. It’s (almost) as simple as a terraform apply.

What did we achieve?

With that solution in place, we can tag the infrastructure deployed for our Pull Requests and have near-instant feedback that helps us understand if we’re meeting our organization’s compliance needs. Compliance isn’t sexy, but staying in a flow state and getting feedback on the spot sure is.

At least to me anyway.

Security is everyone’s job, and now we’ve put the right tools into the hands of the people building the infrastructure. They get to fix any problems before the IaC code has even landed on the main/trunk branch.

jSherz.com

Using AI to put the bins out

Laying the foundations

An architecture for processing e-mails with AI

When the machines can talk

A quick tip for faster feedback

Machines telling machines what do to

Did it work and would I do it again?

My experiences creating a Udemy course

My experience with tutorial videos

Choosing a topic

How long did it take to make the course?

How much money did I make?

What advice would I give you?

Editing software and video storage

Removing noise from the audio

Marketing

Conclusion - would I do it again?

Keeping base and CI/CD Docker images up-to-date in AWS

Conclusion

Tracking technical initiatives with Backstage.io

Why use a tool like Backstage.io for this?

What are our aims of building this automation?

What kind of technical initiatives could we track this way?

What does Backstage give us out of the box?

Let’s build!

Discovering entities

Building the entity provider

Building the entity processor

Conclusion

Running a GitHub Actions pipeline for every AWS account in an organization

Conclusion

Tracking unauthorised AWS API calls to drive platform improvements

CloudTrail

What can we learn from unauthorised API calls?

Solution architecture

Making lots of the same thing in IaC (CDK & Terraform)

Using properties in resource identifiers

Conclusion

Right-sizing Lambda functions that process files

Is it really worth it?

Example 1 - low traffic upload

Example 2 - high traffic upload

Right-sizing Lambda functions

Full working example

Conclusion

See also

Enforcing AWS tags the right way - without Tag Policies

What’s so bad about Tag Policies?

Resource support and wildcards

Bad documentation

No tags, no problem!

What’s the solution?

Conclusion

See also

The full documentation vs. source code diff for Tag Policy resources

Shift security left with AWS Config

What is AWS Config?

How are AWS Config rules defined?

Where do AWS Config reports go?

Flow and fast feedback

Developer-centric tooling

Feedback in the Pull Request

A quick segue into tags

Responding when resources fail compliance checks

Putting it all together: the GitHub App

Putting it all together - AWS infrastructure

What did we achieve?

Further reading