Archive
-
Keeping base and CI/CD Docker images up-to-date in AWS
AWS DockerIf you’re building containerised services, or using a CI/CD system, you’ll likely end up building base images that contain the customisations that fit your organisation’s needs. For example, you might update OS packages, install a newer version of a package manager, or install the CLI tool(s) of your chosen cloud provider. Keeping these images up-to-date can become a maintenance burden,...
-
Tracking technical initiatives with Backstage.io
Backstage AWS Service CatalogueIn this article, we’re going to cover using the developer portal or service catalogue Backstage.io to track technical initiatives as they’re rolled out across a business. Why use a tool like Backstage.io for this? Change in technical ways of working is inevitable, but tracking progress manually with a spreadsheet or document requires a lot of error-prone and tedious effort by...
-
Running a GitHub Actions pipeline for every AWS account in an organization
AWS GitHub Actions CIIf you’re already deploying software with GitHub Actions, you might be wondering if you could use the same mechanism to deploy infrastructure that’s standard across all accounts. AWS provides CloudFormation StackSets for this use-case, but you may be using an alternative Infrastructure as Code (IaC) tool, and want to keep consistency across all your projects. Start by creating an action...
-
Tracking unauthorised AWS API calls to drive platform improvements
AWS Athena Glue CloudTrailIn the DevOps space, it’s really tempting to hit everything with the automation stick. However, I’d argue there are plenty of valid reasons for wanting a human in the loop. If you’re part of a platform team, you’ll be used to balancing the needs and desires of multiple teams - especially when it comes to Identity and Access Management (IAM)....
-
Making lots of the same thing in IaC (CDK & Terraform)
AWS CDK TerraformIt’s a common requirement in Infrastructure as Code (IaC) tools like CDK and Terraform to make many of the same thing, with slightly differing properties. Let’s use a fictional example in which we’re writing an audit service that receives notifications from other services when a user performs an action. Here’s our architecture: The designer of the audit service specified that...
-
Right-sizing Lambda functions that process files
AWS LambdaYou’ve probably seen an architecture that looks like this: How much memory should we allocate to the Lambda function processing files? One approach is to take some sample data and run it through Lambda Power Tuning, but what if we’re processing files of different sizes? The easy way out is to keep increasing the memory until you can safely handle...
-
Enforcing AWS tags the right way - without Tag Policies
AWS security Terraform CloudFormationTags are essential in AWS. They let us allocate costs to different teams, projects, services or business areas, and can be critical in operational response. Tags get us from a page or security related notification to the right team much faster, especially for resources that don’t support names or those that are named automatically by Infrastructure as Code (IaC) tools...
-
Shift security left with AWS Config
AWS security Terraform LambdaNo-one likes a telling off from the security team, and we can’t rely on good will and experience to keep our infrastructure secure: we have to make comprehensive guardrails. One of the services commonly used to achieve that is AWS Config. What is AWS Config? NB: if you’re playing around with AWS Config in a lab environment, please be aware...
-
Terraform vs. CDK
AWS CDK TerraformAt the time of writing, there are at least five mature options for deploying Infrastructure as Code (IaC) to AWS. After working professionally with CDK for around nine months, I’m ready to offer my comparison of the only two I know confidently: Terraform and CDK. I’ll start by saying that I believe both are excellent tools but have their own...
-
Packaging and publishing NodeJS Lambda functions in Terraform
Terraform LambdaI’ve been working about 70/30 in CDK and Terraform over the last year, and there are many aspects of each that I wish I could bring to the other. CDK has “constructs” which are akin to Terraform modules. One of my favourite constructs is NodejsFunction - it packages up Lambda code that’s collocated in the same repository and uploads the...
-
Automatically blocking users from AWS after they leave your organization
AWS Service Control Policy Identity Center Identity & Access ManagementIf your responsibilities include controlling access to systems, you’ll never be far away from the realities of people joining and leaving your workplace. In AWS, we’re advised to use the Identity Center (née SSO) to manage access to accounts with varying permissions. Let’s cover a bit of a footgun with people who are removed from Identity Center, and how we...
-
Safer AWS Service Control Policy (SCP) rollouts
AWS Service Control Policy SCP Control TowerWe use Service Control Policies (SCPs) in AWS to restrict dangerous actions at the account or organization unit level. In an ideal world, you’d design an SCP upfront when a new use case arrives, and then adapt it as that use case evolves over time. Here in the real world, we often have to roll out changes to policies, including...
-
Live debugging EventBridge Event Buses
AWS EventBridge Event BusWhen you hear the term “Event Bus” you might start getting flashbacks to days spent reading a book on Java design patterns. Fear not - these days they’re back in fashion, especially with the AWS service CloudWatch Events that morphed and evolved into its own offering under the EventBridge moniker. We could debug EventBridge Event Buses through conventional means like...
-
Debugging TypeScript/JavaScript CDK projects - a few tips and tricks
AWS CDK Node.JS TypeScriptWhen I moved to the Node.js ecosystem from working in Java, I was struck by how hard it was to get a working debugger. In Java or C#, I’d hit a button in the IDE and get perfect registration of breakpoints without any configuration. In Node, the story was not the same! Things have improved dramatically over the last ~5...
-
Pre-signed URLs in AWS S3 - gotchas that got me
AWS S3 Simple Storage ServicePre-signed URLs are a convenient way of us having our users directly download or upload a file from or to S3. They’re especially helpful when we want to avoid the overhead of processing the file that’s being (up|down)loaded on our services. Your choice of compute option generates a link and returns it to the client’s device. Additional code on the...
-
Trust is a dirty word in access control
trust authorization access Identity & Access Management AWSThere are plenty of words you wouldn’t use in a conversation at work. Maybe they feel inappropriate; maybe they’re considered unacceptable in the culture of your organization, country or region. I want to add another word to the naughty list: trust. Specifically, I want to challenge you and state that we shouldn’t be using “trust” when we talk about access...
-
Break glass access in AWS with Step Functions
Step Functions SRE Lambda API Gateway serverless break glass accessNo-one wants unfettered, widespread access to production all the time, but the pager does have an awful habit of going off and - if your tooling fails you - you might have to pop into a production account to have a look around. I’ve recently been playing around with the use of AWS Step Functions to orchestrate this access. Our...
-
Serverless analytics for CloudFront Distributions
Serverless CloudFront S3 CloudTrail GitHub ActionsCloudFront and S3 make a great combination for hosting static websites, just like this blog that’s generated with Jekyll. There are a number of popular analytics tools, but many compromise the privacy of your users, require JavaScript, or have a monthly fee that’s prohibitive for a hobbyist website. GoAccess is a versatile tool that can be used to generate real-time...
-
Performance testing your services in CI/CD
CI automation Gatling CD NodeJSYou’d likely never dream of having a CI/CD pipeline that doesn’t run your unit or integration tests, but performance testing is easily left to the realms of “oh we should test that before we go live” or perhaps “important customer ABC says they’re getting lots of 502 errors”. We can avoid nasty surprises and have conversations backed with data by...
-
Starting a new project with a Pull Request (or Merge Request)
git GitHub GitLab Pull Request Merge RequestWhen you start a new project in GitHub or GitLab as part of a team, it’s easy to just push the first version of the code to your trunk/main branch and then be left with no easy way to create a Pull Request or Merge Request that is an accurate view of your changes. This makes it harder for colleagues...
-
Using AWS CodeArtifact to host just your private NPM packages
AWS NPM NodeJS CodeArtifactIt’s always frustrated me that NPM’s organisation / teams support doesn’t include the facility to have an automation token that’s not tied to a user. As the person whose token always seems to end up shared across a plethora of CI pipelines, I was really excited to see AWS CodeArtifact launch with support for publishing NPM packages. We can have...
-
Guerrilla upgrades for OpenNebula 5.10 to 5.12
OpenNebula virtual machine vm KVMI’ve used the OpenNebula project for several years to run virtual machines to experiment with different Linux distros and software. Unfortunately, they made the decision to continue releasing the main product as Open Source Software but restrict the database migration scripts to non-commercial use[1] [2] (or require you to pay). This would be fine for my use case but requires...
-
My preferences for TypeScript projects
TypeScript Node.JS Node JavaScriptI have a number of preferences for TypeScript (TS) projects, built up from seeing the language adopted by teams and drawing comparisons with other ecosystems and strongly typed programing languages. These are mainly to do with configuring the compiler on new projects and additionally general rules for how you go about interacting with external data stores. I think TypeScript is...
-
Analytics for CloudFront distributions
CloudFront AWS Scala Akka ElasticsearchIt’s nice to be able to analyse the traffic that your CloudFront distribution is receiving, for example if it powers a static website (like this blog). The built-in metrics aren’t very granular and don’t provide information like what browser the user visited the site with or where in the world they’re located. We can get this information from the distribution’s...
-
Building an automatic backup verification pipeline: Elasticsearch edition
Elasticsearch snapshots backup CodeBuild DockerIf you don’t restore and verify your backups, you don’t know that they’ll actually work when the time comes. Doing this manually is time consuming, easily forgotten and a great candidate for being automated. I’m sure you can imagine that we’d normally be talking about backups for a database or perhaps some archived Elasticsearch indices, but even your local GitLab...
-
Debugging incorrect timestamps in the unified CloudWatch Agent
AWS CloudWatch logsLogs are unhelpful at best and thoroughly misleading at worst if not stored with the correct timestamp. A few seconds off is most likely good-enough, but if your log shipping fails for a period of time or you’re trying to make sense of the order of a number of events that happen in quick succession, any inaccuracy is incredibly frustrating....
-
Visualising ALB logs with Elasticsearch and Kibana
Elasticsearch AWS ALB Application Load Balancer Kibana GeoIPHaving the raw logs from our AWS Application Load Balancer can be really helpful for debugging, especially when the service that you’re load balancing may not have logged a message if it times out or otherwise errors. If you’ve spent any time writing NodeJS services with Express and Promises, I’m sure you’ll have run into the controller or middleware that...
-
Providing secure, authenticated access to an internal service running in ECS
ECS Cloudflare Argo Access sidecar Docker containerGone are the days where every employee sits in an office cubicle from 9AM to 5:30PM, Monday to Friday. Having a physical location with the blinking lights of a VPN appliance or whining server is no longer a given and thus the ‘traditional’ approach of whitelisting company IPs and having your colleagues VPN in to the corporate network just isn’t...
-
Load balancing GRPC services in the Elastic Container Service with Traefik
GRPC ECS AWS Docker containers load-balancer ELB NLB ALBDespite the popularity of Kubernetes, AWS’s Elastic Container Service (ECS) offering is fantastic for many containerised workloads and avoids a lot of the complexity that comes from using and operating a full-blown container orchestration platform. Add in Fargate with its attractive pricing and low management overhead and you have a great way to easily host containers. One challenge of using...
-
Securing private docs with CloudFront & Lambda@Edge
s3 CloudFront Lambda@Edge Jekyll docs ReadTheDocsIn a previous article, we looked at a method of restricting access to a CloudFront distribution with the use of a CloudFront private key that could sign cookies granting access to private files or even a static website. With AWS Lambda@Edge, we can remove a few of the steps in that article and replace them with a Lambda function that...
-
Badges for AWS CodeBuild & CodePipeline with CloudWatch and Lambda
codebuild aws lambda cloudwatch nodejs readmeNB: this approach is no longer required as CodeBuild now supports badges natively. See Build badges sample with CodeBuild in the AWS docs. Most build systems have a built-in method of producing status badges or a plugin that will create them. With CodeBuild, we can use CloudWatch Events and a Lambda function to generate a badge when our build succeeds...
-
Mutual auth with GRPC & Node: start to finish
grpc node nodejs mutual authentication sslSetting up mutual authentication can be a little daunting, especially when the docs for a library you’re using don’t always have a good example. Top it off with having to make your own certificates, and the whole process can be a real PITA! To make it easier, we’re going to be using a tool from the great people at Square,...
-
Providing password-based access to a private S3 bucket with Lambda and CloudFront
infrastructure aws s3 bucket password lambda cloudfrontAmazon’s Simple Storage Service doesn’t natively support password-protected access, however we can use a CloudFront distribution and private ACL to control access to the bucket and then use Lambda to issue signed cookies after validating a password. via GIPHY How it works A user visits the CloudFront distribution. This could either be directly to the abcde.cloudfront.net hostname or a CNAME....
-
Want faster Ansible runs? Try enabling pipelining!
ansible infrastructure platform engineeringI’ve always wondered if there’s a way to speed up Ansible executions as creating a new SSH connection for each command seemed excessive. However, I’d never looked for a solution until today when I discovered a very neat feature that shares SSH connections called pipelining. Enabling it as simple as adding the following to an ansible.cfg file located somewhere it...
-
Finding a connection leak, the easy way
leak memory connection database graphing plotly python linuxFinding a connection or memory leak can be difficult at the best of times and near-impossible without good visibility of what your application is doing. To make things worse, it’s easy to end up in production with something leaking that you may not have existing monitoring or metrics for. Adding this to the code might not be an option and...
-
Note to self: if you want PostgreSQL to accept your non-lowercase database name, enclosure its name with quotes
docker configserver firewall iptables csf debian systemdThe following creates the database frustratingerror: CREATE DATABASE FrustratingError; The following creates the database FrustratingError: CREATE DATABASE "FrustratingError";
-
Configuring ConfigServer Firewall (CSF) for Docker (Debian 8)
docker configserver firewall iptables csf debian systemdBy default, the Docker daemon will automatically configure iptables rules that allow communication with containers and, additionally, the outside world through the use of exposed ports. If you’re like me and using ConfigServer Firewall, this may not be desirable as these rules will bypass the firewall configuration and let anyone access the exposed container ports. To get around this, it’s...
-
Installing RancherOS on OpenNebula
docker rancher rancheros opennebulaAt the time of writing, Rancher does not publish VM images that are not aimed at a particular cloud provider and I therefore set about installing RancherOS with just the provided ISO. Step 1: Download and import images To begin, download the RancherOS ISO and then upload it into OpenNebula’s image system. sudo -u oneadmin -i cd /tmp wget https://releases.rancher.com/os/latest/rancheros.iso...
-
Adding a grub menu option to reboot to the BIOS / UEFI settings on CentOS
centos grub grub2 bios uefi bootI recently played around with a few Linux distros and ended up keeping CentOS as my daily driver. One thing I missed, however, was having the “System settings” option on the grub menu that would reboot the computer into the BIOS / UEFI options (present in Ubuntu & Debian). To add this option on CentOS, create the following file and...
-
Compiling a custom FreeBSD kernel to enable VNET support for jails
freebsd jails kernel vnetOne issue I had while replacing FreeNAS with FreeBSD was that, by defualt, vnet support isn’t enabled in the kernel. To resolve this, I compiled a custom kernel as follows: Step 1 - download the kernel sources Change the FreeBSD version, as appropriate. pkg install subversion svn checkout http://svn.freebsd.org/base/release/10.2.0/ /usr/src Step 2 - create a custom config cd /usr/src/sys/amd64/conf #...
-
Setting up the discovery of a network share / server with Avahi
avahi service mdns cifs smb afp media-serverI recently replaced FreeNAS with FreeBSD after installing even basic tools in the main OS (without first using a jail) became tedious and I wanted to explore how each of the components (e.g sharing & plugins) were implemented. Below is a service description file that Avahi will use to broadcast the server’s share capabilities. It allows an easy one-click to...
-
Installing f.lux on CentOS
centos oss f.luxWhen working late at night, I enjoy using the excellent f.lux app. Unfortunately, setting it up on CentOS wasn’t as easy as I’d hoped. Below are the steps I took to get it working. Step 1 - install dependencies sudo pip install pexpect sudo yum install gnome-python2-gconf pyxdg python-appindicator Step 2 - install f.lux git clone https://github.com/Kilian/f.lux-indicator-applet.git cd f.lux-indicator-applet sudo...
Subscribe via RSS.