Landlord: a tenancy controller and experiment in AI driven product building

Fri, 06 Feb 2026 00:00:00 +0000

Note: The TL;DR of this post is I built a thing with AI, you can check it out here

Back in 2012 I joined a growing startup that sold SaaS software to its customers. For those of you reading who didn’t understand how the infrastructure world looked back then, AWS, Google Cloud and Azure were all relatively niche offerings at the time, Docker hadn’t been invented yet and most of the technologies you wanted to use to manage infrastructure were written in Ruby.

I had never built any SaaS software, but I was still surprised at how things worked behind the scenes when I joined. Provisioning a new customer involved my “TechOps” team running some shell scripts, data was stored in an NFS mount shared between 2 or 3 racks of whitebox servers, and if one customer’s usage got out of control, we would have to rsync the data to another server to avoid any noisy neighbour problems.

Obviously none of this was tenable in the long term, but it worked for the time we were in. There was an acknowledgement amongst our team of 5 or 6 system administrators (that’s DevOps or Platform engineers to you young ‘uns) that we couldn’t scale the business this way, and one of my more senior colleagues rolled up his sleeves and started automating some of the tasks we had in front of us.

What emerged over the course of several weeks was a project we dubbed “selfserve”. Its functionality started simple: anything we could automate with a shell script was now executed by a “runner” which was dispatched from a centralized control plane. It would SSH into a box, execute the script and return the results to the control plane, which would then be rendered in PHP to whoever ran it. Our ability to scale our processes accelerated massively. In parallel, I worked with another colleague to automate the art of bringing servers online (with Puppet and Cobbler, two technologies that seem to be on the path to extinction) and another colleague worked on streamlining our ability to get datacenters off the ground. Within the space of 2 years, we grew from 2 racks in RDU to 7 datacenters around the world with thousands of customers and millions of dollars in revenue.

Another issue we had to solve was tenancy - how should we segment our customer’s data so we meet compliance goals and scale effectively? Remember, this is well before Docker is considered the de facto standard, and this wasn’t a problem any of us had expertise in solving.

Tenancy as a model is relatively common nowadays, and there are many ways to solve it. At the time, the right way seemed to be to have distinct, segregated compute per tenant - and it scaled remarkably well. Back in 2012 to 2015, we felt like it was the right approach, but as the company grew, we all held some regrets we hadn’t been more ambitious about solving this problem..

My career has changed since then, and I work in a more consultative role as a solutions engineer with companies trying to solve technology problems with Tailscale. What has consistently surprised me over this time is how many companies and orgs are solving the tenancy model in the same way! When I think about it now, wearing the scars of being paged at 2am for most of my career, the “compute-per-tenant” model actually makes a remarkable amount of sense. Sure, it’s inefficient but it’s safe, easy and reduces the blast radius of outages. As with all technology decisions, the tradeoffs have to be considered, but I speak to a remarkable amount of customers who have solved the problem the same way.

Why am I telling you this story in a post with AI in the title? Well, I’ve had in the back of my mind that this problem space could be commoditized because of how common and ubiquitous it is. The problem was always how hard it seemed to be to try and solve it on my own as a spare time project. If you haven’t solved this problem before, you’d be forgiven for thinking it’s quite easy to solve now: Just have a provisoning system hit the Kubernetes API! It’ll handle all this stuff for you! I don’t blame anyone for thinking that’s the right way to go about it, but it fails to understand that managing the automation itself then becomes the battle. If you’re a SaaS service, you probably want to have an automated onboarding flow that provisions the tenant for you, but you now need to build a durable process that involves lots of distinct steps and assumes the entire process is reliable. What happens if you run out of compute in your AWS account? What if your Kubernetes API responds slowly? What if the Docker image you’re pulling pulls too slowly and the job times out? What about if the Docker image is wrong in the request, and now you’re in ImagePullBackOff and you have to nurse it through automatically.

You’re essentially in a reliability maths problem, and what seemed like a simple problem becomes a complex web of inter-dependencies that all have to have 99.99999% uptime to work and be successful.

Landlord

As I mentioned earlier, I’ve seen this problem be common enough I wanted to build something to solve it for a modern tech stack, but it seemed to incredibly daunting. If I was lucky enough to not have to work and could spend my time writing code all day, I figured I could probably get this done to a reasonable standard. I’m not what you’d consider the most talented software developer (after all, I started writing shell scripts and PHP!) but I had enough information to be dangerous, and most importantly I feel like I know relatively well how to design a good system.

Then the world was hit by the meteor now called agentic coding, and I had a little crack at building it. It was an incredibly frustrating experience because the LLM couldn’t hold enough context to build such a complicated system, I felt like all I did was fight with the model.

Then a few weeks ago, I started experimenting with spec-driven development. And things changed overnight.

What has emerged is landlord, an experimental, pluggable compute manager that provisions tenants via a series of different compute providers. It leverages pluggable workflow providers to handle the durable executions, and looks very similar in design to how the first versions of selfserve looked back in 2012/2013. Written in Go, it consists of a single API managed control plane that can handle most of the work you need to do to provision tenants in a way that’s reliable and effective.

Walkthrough

Landlord currently supports one workflow provider, Restate which handles the durable execution of the compute. It supports two databases, SQLite and Postgres to store the state of the execution. It currently supports one compute provider, Docker.

We start by provisioning a tenant. Landlord accepts config for the tenant in the format of the compute provider you’re provisioning.

go run ./cmd/cli create --tenant-name lbr \
  --config '{
    "image": "nginx:1.25",
    "env": {
      "FOO": "bar"
    },
    "ports": [
      {
        "container_port": 80,
        "host_port": 8888,
        "protocol": "tcp"
      }
    ]
  }'
Tenant created
ID: 0b79eb9b-2c4c-43fb-85f0-aeb0ed63de78
Name: lbr
Status: requested
Config: {"env":{"FOO":"bar"},"image":"nginx:1.25","ports":[{"container_port":80,"host_port":8888,"protocol":"tcp"}]}
Compute: {"env":{"FOO":"bar"},"image":"nginx:1.25","ports":[{"container_port":80,"host_port":8888,"protocol":"tcp"}]}
Created At: 2026-02-06T16:00:09Z
Updated At: 2026-02-06T16:00:09Z
Version: 1

This request then kicks off a workflow to the pluggable workflow provider, which includes a worker that performs the execution. By dispatching this off to a workflow provider, we mitigate against any errors in the request flow, meaning we can provision tenants in parallel without worrying about building durable code.

The tenant has been requested, meaning the workflow job is dispatching

go run ./cmd/cli list
ID                                    Name  Status     Workflow  Retries
0b79eb9b-2c4c-43fb-85f0-aeb0ed63de78  lbr   requested

The workflow job runs and finally, I can see my tenant is up and running!

ID: 0b79eb9b-2c4c-43fb-85f0-aeb0ed63de78
Name: lbr
Status: ready
Status Message: Workflow execution completed: inv_1gD1CwSdyy410oGeothz81216ifhbZkCAx
Workflow Sub-State: succeeded
Config: {"env":{"FOO":"bar"},"image":"nginx:1.25","ports":[{"container_port":80,"host_port":8888,"protocol":"tcp"}]}
Compute: {"env":{"FOO":"bar"},"image":"nginx:1.25","ports":[{"container_port":80,"host_port":8888,"protocol":"tcp"}]}
Created At: 2026-02-06T16:00:09Z
Updated At: 2026-02-06T16:00:22Z
Version: 3

Something I took from modern reconciliation approaches like Kubernetes was the ability to set desired state. If I want to modify the configuration, I can simply issue a set command with a new image or config. Let’s add a new environment variable

go run ./cmd/cli set --tenant-name lbr \
  --config '{
    "image": "nginx:1.25",
    "env": {
      "FOO": "bar",
      "LAND": "lord"
    },
    "ports": [
      {
        "container_port": 80,
        "host_port": 8888,
        "protocol": "tcp"
      }
    ]
  }'
Tenant updated
ID: 0b79eb9b-2c4c-43fb-85f0-aeb0ed63de78
Name: lbr
Status: updating
Status Message: Update requested
Config: {"env":{"FOO":"bar","LAND":"lord"},"image":"nginx:1.25","ports":[{"container_port":80,"host_port":8888,"protocol":"tcp"}]}
Compute: {"env":{"FOO":"bar","LAND":"lord"},"image":"nginx:1.25","ports":[{"container_port":80,"host_port":8888,"protocol":"tcp"}]}
Created At: 2026-02-06T16:00:09Z
Updated At: 2026-02-06T16:03:36Z
Version: 4

This kicks off a new workflow:

go run ./cmd/cli get --tenant-name lbr
Tenant details
ID: 0b79eb9b-2c4c-43fb-85f0-aeb0ed63de78
Name: lbr
Status: ready
Status Message: Workflow execution completed: inv_17jEoSZg1AhW2mztVGxctly24kh6hua6el
Workflow Sub-State: succeeded
Config: {"env":{"FOO":"bar","LAND":"lord"},"image":"nginx:1.25","ports":[{"container_port":80,"host_port":8888,"protocol":"tcp"}]}
Compute: {"env":{"FOO":"bar","LAND":"lord"},"image":"nginx:1.25","ports":[{"container_port":80,"host_port":8888,"protocol":"tcp"}]}
Created At: 2026-02-06T16:00:09Z
Updated At: 2026-02-06T16:03:52Z
Version: 6

and uses the underlying compute primitives to replace the existing tenant with new config:

docker ps --filter label=landlord.owner=landlord
CONTAINER ID   IMAGE        COMMAND                  CREATED              STATUS              PORTS                  NAMES
0962efd7c26b   nginx:1.25   "/docker-entrypoint.…"   About a minute ago   Up About a minute   0.0.0.0:8888->80/tcp   landlord-tenant-0b79eb9b-2c4c-43fb-85f0-aeb0ed63de78

docker inspect 0962efd7c26b | jq '.[0].Config.Env'
[
  "FOO=bar",
  "LAND=lord",
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "NGINX_VERSION=1.25.5",
  "NJS_VERSION=0.8.4",
  "NJS_RELEASE=3~bookworm",
  "PKG_RELEASE=1~bookworm"
]

I personally believe this is tenancy model that is very common around our industry, and I hope this experimental approach goes some way towards commoditizing this particular part of the stack.

The AI of it all

I couldn’t in good faith write this post without talking about AI. As I mentioned before, I’ve had aspirations to build something like this quite a while, but the level of effort involved seemed so incredibly daunting, I never really got started. Spec driven development has completely changed what I could achieve in this space, and if you take a look at the openspec directory in the landlord repo, you’ll be able to see just how long it took me to build something relatively complex.

Now, how do I feel about the quality of this? Well, as I was reviewing the code generated by Codex 5.2, I did have to corral it a little bit. I’m not particularly enthused about the quality of the code here. There are several areas in here where the model made trade-offs I would have likely not wanted to make, and a lot of times I had to catch it during the process of building a spec and say “actually no, please don’t do that”. The real thing that’s worth talking about here of course if productivity - the fact I could build a relatively complex system with domain knowledge or a problem I’ve already solved is remarkable to me.

What’s next?

I’d like to continue adding pluggable interfaces to landlord, support for Amazon Step Functions and Temporal as workflow engines is high on the agenda, as well as expanding the compute support for ECS, Kubernetes and EC2. After that? Who knows, it was fun remiscing about career years gone by.

FAQs

Remind me again why you wouldn’t just use Kubernetes or Nomad or some other cluster scheduler?

Nomad and Kubernetes solve the scheduling problem. They’re very good at placing workloads once you’ve decided what should exist. Landlord is focused on the control plane problem that sits above that:

orchestrating multi-step provisioning
handling retries and partial failure
managing tenant lifecycle as a first-class concept
integrating workflow durability with compute execution

You can absolutely use Kubernetes or Nomad underneath Landlord. The point is to avoid baking all of this logic directly into cluster automation where it becomes fragile and hard to reason about.

Is this production-ready?

No - and that’s intentional.

This is an experiment in system design, workflow durability, and spec-driven development. The abstractions matter more than the current implementations. Some parts are deliberately simple so the seams are visible.

If this ever becomes “production-ready”, it will be because the design holds up as more providers and workflows are added.

Why spec-driven development instead of just writing code?

Because the hard part here isn’t syntax—it’s decision-making. Specs force clarity:

what guarantees exist
what can fail
what is allowed to change
what must remain stable

Once that’s written down, AI becomes genuinely useful as an accelerator instead of a liability. Without specs, you’re just arguing with the model.

What problem are you actually trying to commoditise?

The boring but critical middle:

tenant provisioning
lifecycle management
durable automation
“what happens when this goes wrong?”

Most teams end up re-building this ad-hoc. Landlord is an attempt to make that layer explicit, inspectable, and reusable.

What models and tools did you use to build this?

I used a combination of GPT-Codex 5.2 with the Codex VSCode extension, as well as GitHub Copilot with Claude Sonnet 4.5 for some light fixes and documentation.

The enshittification of enshittification

Sun, 18 Jan 2026 00:00:00 +0000

Over the last six months, I’ve spent a lot of time in the Tailscale community - helping users debug issues, answering questions about how we sell the product and explaining, repeatedly, how we think about the business behind it.

It’s rewarding work. I genuinely enjoy it.

But threaded through almost every conversation is the same quiet fear:

“Eventually you’re going to take this away from me, aren’t you?”

Enshittification

That fear usually traces back to Corey Doctorow’s concept of enshittification: the idea that any service which genuinely solves a problem will, over time, mutate into a platform that extracts value from the people who depend on it.

It’s frequently paired with another well-worn belief:
“If you’re not paying for anything, you’re the product.”

These ideas have become common shorthand for how people reason about SaaS businesses. They’re not wrong, exactly - but when they’re treated as universal laws rather than patterns, they flatten a lot of important context. Before applying them indiscriminately to everything we like or rely on, it’s worth understanding what actually drives this behavior in the first place.

The VC-backed engine

Most companies that take venture capital do so for the same basic reason: to grow. Growing a business requires capital, and capital is hard to come by when you’re small. You need to invest in marketing, hire engineers to improve the product, and hire salespeople to bring it to market.

The tradeoff is straightforward. Once you take venture capital, you are no longer optimizing solely for users. You are also obligated to return capital to investors, and that obligation shapes every decision that follows. Growth in revenue, market share, and valuation isn’t just encouraged. It’s required.

When it works, this becomes a flywheel. You spend money to grow. Growth improves your market position. That success unlocks more opportunity. Growing companies are fun places to work. I’ve spent most of my career in them because that energy is infectious and motivating.

What’s discussed far less is how hard sustainable revenue growth actually is. Adding users to a free tier is often easy. Converting those users into paying customers is not, unless your go-to-market motion is exceptional. For a lot of companies, that pressure is where shortcuts start to look tempting.

Product-led growth, actually

Before going further, it’s worth being clear about my vantage point. I’m not the person who ultimately sets pricing or product strategy. But as Director of Solutions Engineering, I sit in the middle of customers, sales, and the community. When decisions land poorly, I feel it immediately.

From where I sit, Tailscale’s go-to-market motion is product-led growth in a very literal, very unromantic sense. People show up because they have a real connectivity problem in their own lives and want it to stop being annoying. If the product works, they keep using it. If it doesn’t, they leave.

What makes those users valuable isn’t that we extract revenue from them directly. It’s that those same problems inevitably show up at work. And once someone has used a tool that just works, their tolerance for brittle, frustrating alternatives drops fast.

If a company like Tailscale were to start degrading the personal tier in pursuit of short-term revenue, that incentive chain would collapse. It wouldn’t unlock some hidden pool of money. It would remove the very top of the funnel that drives our revenue growth in the first place. Individual users aren’t a loss leader - they’re the mechanism by which trust, familiarity, and adoption propagate into larger rollouts inside organizations.

This is also why comparisons to consumer platforms tend to fall apart. Those businesses operate at enormous scale, where marginal users can be monetized through ads, fees, or data. The market Talscale operates in is different. Secure business connectivity is a multi-tens-of-billions-of-dollars global market, but that revenue comes from companies, not individuals. There simply aren’t enough people with homelabs, side projects, or personal VPN needs to generate meaningful revenue on their own. And trying to squeeze value out of them would actively harm the thing that makes the business work.

I think that’s why we’re careful about what we do and don’t gate behind a paid plan. In practice, removing useful features from the personal tier doesn’t create value - it just makes the product worse. And making the product worse at the individual level directly undermines the business trying to grow.

From where I sit, we don’t want people paying us to make their personal networking tolerable. We want companies paying us because their employees already like using the product, and want that same experience at work. What keeps this model working isn’t clever pricing or gradual value extraction. It’s people who like the product enough to say, “Hey, this would make our lives easier,” even when that’s not their job.

I’m literally incentivized, as a sales engineer at Tailscale, to sell the product. And I have very little interest in trying to generate revenue from personal users, because unhappy users don’t advocate for anything. Making them happy isn’t a feature - it’s the whole point.

Inevitability is lazy thinking

One of the things I struggle with in the enshittification narrative is how quickly it turns into inevitability, like some self-fulfilling prophecy. There’s this pervasive idea that every company will eventually betray its users, so we may as well stop expecting anything better and that’s just capitalism baby.

I think once we start to bandy around the idea of enshittifiation at literally everything, we’ve enshittified the concept of enshittification. We’ve removed all accountability from the equation.

Companies don’t enshittify because time passes. They enshittify because incentives change, leadership priorities shift, or short-term outcomes are allowed to override long-term trust. Cynicism feels realistic, but more often than not it’s just resignation dressed up as wisdom.

Trust is a business constraint

Trust isn’t a marketing asset - it’s a constraint. Once users believe you will eventually make their experience worse in pursuit of growth, every decision you make is filtered through that assumption and at that point, even good changes are met with suspicion. I’ve seen Reddit comments recently that have framed “new Tailscale features” as the beginning of Tailscale’s enshittification cycle because any change is now considered enshittification.

For a company like Tailscale, trust compounds slowly and breaks quickly. Our product sits directly in the critical path of how people access their networks and their work. If users stop believing we’re acting in good faith, no amount of pricing optimization or feature bundling will fix that and the most important thing about this is that everyone who makes decisions at Tailscale knows it.

What would actually force this to change

None of this is to say the model is immortal. There are conditions under which it would break. If the personal tier stopped being a meaningful driver of business adoption. If the problems we solve no longer overlapped between individuals and organizations. Or if the economics of running the platform changed in a way that made the current structure unsustainable.

If that day ever comes, the honest response wouldn’t be to degrade the experience and hope people don’t notice or don’t care. It would be to explain the tradeoffs openly, change the model explicitly, and accept the consequences of that decision. I suspect in that set of circumstances, Tailscale has enough competitors (including our open source control plane!) that it would lead to a significant drop off in users.

Until then, it’s worth being honest about the consequences. Calling enshittification prematurely doesn’t protect users, it can accelerate the conditions that make it more likely. If you scare people away from trying the product, you erode the very trust and adoption that keeps it working. It might make you feel better, but it’s also self-fulfilling cynicism - and it leaves everyone worse off.

An easy, realistic model for MCP connectivity

Sun, 08 Jun 2025 00:00:00 +0000

You can’t escape it. Everywhere you turn in the tech ecosystem, AI is there.

Whether you’re an AI skeptic or an AI convert, you almost certainly understand how explosive the change in the tech ecosystem has been, and how fast everything is moving right now.

I’ll keep my personal opinions about AI generally out of this blog most (mostly) but I’ve been staying very familiar with the Model Context Protocol (MCP) for a few reasons, the primary one being that it seems absolutely terrifying in ways I can’t really comprehend.

Many of the security lessons we’ve learned over the years seem to have been overlooked in MCP’s rapid development. “Protect your data, it is what’s unique to you!” was the battle-cry of every tech oriented person for most of my career. I’m old enough to remember when Cambridge Analytica (mainly because it wasn’t that long ago..) was raked over the coals because it weaponised our social media data and now a few years later we seem quite content with the idea letting large VC-funded organisations to slurp up mountains of our private data to train their word guessers.

I’m being as glib as I always am on this blog with the last paragraph, but in all seriousness, MCP has a problem - you want to get your data into an LLM, but you don’t want everyone else to be able to see it.

A quick history of the MCP evolution

At its core, MCP is a funnel. How do I get local information - or - information that isn’t crawlable on the public internet - into an LLM so it can analyse it. Anthropic defined a spec that will help you do just that, and in its infancy, the trick was simple - run a local server that speaks JSON so that a local client can call and pipe it into the LLM.

The initial version of the spec seemed destined for local connectivity. The servers were designed to be accessed over a stdio, which you’d run locally and then connect to with an MCP client, like Claude Desktop. Stdio is just “standard input and output streams” and isn’t at all designed to be called remotely, so generally your data is pretty safe. It’s really not easy to intentionally expose your data to the big scary world.

Running a stdio MCP server was pretty straightforward, most of them are written in Typescript or Python, and you’d just add commands that you ran into your MCP client and it’d execute it on startup - easy.

In Claude Desktop for example, here’s how you’d run a filesystem MCP so Claude can analyse your local filesystem:

{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": [
        "-y",
        "@modelcontextprotocol/server-filesystem",
        "/Users/username/Desktop",
        "/Users/username/Downloads"
      ]
    }
  }
}

Pretty straightforward if you can, you know - easily write JSON and understand what the hell npx is. Obviously for non-technical users, this looks like a magical incantation, but that’s okay, MCP is early.

As things have progressed (very very quickly, I might add) it’s become obvious to people that eventually, you want to be able to run these MCP servers somewhere else other than your local machine. Various companies have sprung up around this, with varying degrees of success and questionable security tactics.

So the spec evolved to meet these needs, and the next iteration introduced a mechanism to access MCP servers remotely, which became Server Side Events.

You can see on this page there’s suggestions for how make sure this doesn’t go badly for you:

Security Warning: DNS Rebinding Attacks SSE transports can be vulnerable to DNS rebinding attacks if not properly secured. To prevent this:

Always validate Origin headers on incoming SSE connections to ensure they come from expected sources

Avoid binding servers to all network interfaces (0.0.0.0) when running locally - bind only to localhost (127.0.0.1) instead

Implement proper authentication for all SSE connections

Without these protections, attackers could use DNS rebinding to interact with local MCP servers from remote websites.

SSE started to take off despite these warnings on the server side, but due to the pace of innovation, clients were surprisingly slow to introduce support for this. At the time of writing, I still can’t find a client that is broadly used that will allow you to connect easily to an SSE server.

So we started to see a new type of tool appear - the proxy. It would proxy requests from stdio clients into SSE events, allowing people to run those SSE servers remotely.

Finally, the most recent innovation has been to completely rework SSE (after only 5 months, by my count!) and switch to Streamable HTTP as an alternative, which might just set the land speed record for a protocol deprecation, but still, an evolution nonetheless.

And yet, there’s still a problem

As all this change has happened, I’ve been watching it and thinking to myself “well, okay, but this is still a security nightmare”. There are obviously things happening in the space that are changing here, and there’s intent to fix it, but take another look at the streamable HTTP warnings and spec - there is absolutely no written document at the time of writing this blog post to introduce any sort of authentication to the protocol. It has been drafted and is following a fairly typical pattern of late - “let’s slap some oauth on top of it and call it good”.

Personally, that doesn’t make me particularly happy, because I think oauth is really confusing and easy to screw up and secondly you still have a communication problem. I don’t care how much authentication you put on top of something, if there’s particularly sensitive data behind something, I still don’t want it hanging around on the internet. If you do feel okay with this, I presume all your databases are on the internet as well, but it’s okay because they have passwords on them. Right?

So, this got me thinking. I work for Tailscale, Tailscale’s really quite good at protecting your data and connecting things together. How can we improve this situation?

My first MCP server

As things were evolving, I wrote a little MCP server for the Tailscale API that allows you to query a few things there, primarily to try and understand the protocol and see what useful things we could do. I stuck with stdio and then introduced SSE, but was pretty unhappy with how things looked at that point, so left it at that.

However, when streaming HTTP was published a few weeks ago, I realised there was an opportunity here to think differently about the security model to make it a little more robust and considerably more private.

You can get some of the benefits of Tailscale with streaming HTTP by just having a Tailscale on both ends of the equation. Install Tailscale on your local machine, spin up another one somewhere and run a streamable HTTP server, then configure your MCP client to run an MCP proxy (of which there many, such as sparfenyuk/mcp-proxy or if you prefer Typescript, punkpye/mcp-proxy). When you configure the proxy, set your endpoint to the remote Tailscale address and the port/url your MCP server is listening on and you’re golden.

This is a massive improvement over the “run it on the internet model with oauth” because now you don’t have to run the damn thing on the public internet, which should be fairly obvious. I was going to go with this approach, but then I had an idea - what if we could use Tailscale’s application awareness to improve the security model even more?

So I did two things:

I wrote a little MCP proxy in Go that forwards Tailscale’s headers like X-Tailscale-User to the remote HTTP MCP server
Then I updated my Tailscale MCP server to support reading Tailscale’s grants mechanism to determine what that Tailscale user is allowed to do.

A proper security model in action

So how does this look? Well, I can run my Tailscale MCP server on a remote machine. I spun up a VM in digital ocean and fired it up:

curl -L https://github.com/jaxxstorm/tailscale-mcp/releases/download/v0.0.3/tailscale-mcp-v0.0.3-li
nux-amd64.tar.gz | tar -xzf -
TS_AUTHKEY=<your-auth-key> ./tailscale-mcp --tailnet=<your-tailnet> --api-key=<tailscale-api-key>

And some output logs

4:48:37	INFO	tailscale-mcp/main.go:265	Starting ts-mcp	{"version": "0.0.3", "tailnet": "lbrlabs.com", "hostname": "ts-mcp", "port": 8080, "debug": false, "stdio": false}
2025/06/09 14:48:37 tsnet running state path /root/.config/tsnet-tailscale-mcp/tailscaled.state
2025/06/09 14:48:37 tsnet starting with hostname "ts-mcp", varRoot "/root/.config/tsnet-tailscale-mcp"
2025/06/09 14:48:37 Authkey is set; but state is NoState. Ignoring authkey. Re-run with TSNET_FORCE_LOGIN=1 to force use of authkey.
14:48:37	INFO	tailscale-mcp/main.go:558	Serving MCP via Tailscale	{"address": ":8080"}
14:48:37	INFO	tailscale-mcp/main.go:586	Serving MCP locally	{"address": "127.0.0.1:8080"}
2025/06/09 14:48:42 AuthLoop: state is Running; done

Now, I just need to configure my MCP client (in my case, Claude Desktop) to run my MCP proxy locally.

{
  "mcpServers": {
    "tailscale": {
      "command": "/usr/local/bin/tailscale-mcp-proxy",
      "args": ["--server", "http://ts-mcp:8080/mcp"]
    }
  }
}

I need Tailscale running on my machine so they can communicate with each other and capture the information who I am.

tailscale status
84.243.110  lbr-macbook-pro      mail@        macOS   -
72.57.77    lbr-iphone           mail@        iOS     offline
81.81.4     lon-derp1            tagged-devices linux   -
105.3.74    sea-derp1            tagged-devices linux   -
66.15.114   ts-mcp               mail@        linux   idle; offline, tx 52624 rx 72604

Now, I can fire up Claude Desktop and ask it questions about my Tailnet:

But wait, it’s telling me I don’t have permission? If we look at the MCP server’s logs, we can see I don’t have the right access to run the query:

15:17:08	INFO	tailscale-mcp/main.go:159	No MCP capabilities found
15:17:08	WARN	tailscale-mcp/main.go:172	No MCP capabilities found	{"user": "mail@lbrlabs.com"}

The reason for this is that my Tailscale MCP server is going to use Tailscale’s grants to determine which tools I’m allowed to call. So lets add a grant to my Tailscale ACL to indicate I can call all tools:

{
    "src": ["autogroup:admin"],
    "dst": ["100.66.15.114"], // the address of my Tailscale MCP, I could also use tags
    "ip":  ["tcp:8080"],
    "app": {
        "jaxxstorm.com/cap/mcp": [{
            "tools":     ["*"], // I can call all tools
            "resources": ["*"], // I can call all resources
        }],
    },
},

Now, let’s try that query again!

Success!

The caveats

This is all well and good, but what are some of the considerations to this approach?

As it stands, tsnet is only really usable for Go servers, so you’d need to write your MCP server in Go, which is not officially supported right now. Most MCP servers are written in Typescript or Python. To each their own.

Note: Tailscale does have a proof of concept C based library which could be used for Python and Typescript based MCP servers, and if you’re interested in using it, you should contact us at Tailscale by posting on Reddit

The other side of this of course is that only local clients can implement these proxies. Anthropic supports calling remote MCP servers on its enterprise plans, but it expects them to be on the public internet. I would personally love the idea of being able to connect from your Claude team to your Tailnet (imagine just giving Claude your Tailscale oauth credentials and it provisions a private network for you for your MCP connections!) but I’ll need someone from Anthropic to implement that..

I’m also personally a huge fan of this Tailscale application model of permissions and capabilities, but I suspect the first detraction will be “we want these standards to be open, not have Tailscale in the middle of them!” which I totally get.

The code

Finally, and to close this post down, if you want to try any of this yourself, you can find all the code for the MCP server here and the MCP proxyhere - I hope this inspires you to get something important off the public internet!

Hypocrisy

You’ll recall at the beginning of this post, I was lamenting the idea that we’re going to let LLMs hoover up all our data, and yet here I am enthusiastically writing MCP servers to make it easier.

I suppose that’s the thing about technology - you can either participate in shaping how it develops, or you can stand on the sidelines complaining about how everyone else is doing it wrong. I’ve chosen to participate, even if it means being part of a system I have mixed feelings about.

The best I can do is try to build the secure version of what’s inevitably going to happen anyway. Maybe that makes me a hypocrite, but at least I’m a hypocrite that’s thinking about protecting your data from the scary internet, and only the LLM can access it. That makes it better…right?

The Death of Developer Relations

Tue, 10 Dec 2024 00:00:00 +0000

Every year, I gear up for “conference season,” which includes KubeCon NA (typically held between mid-October and mid-November) and AWS re:Invent, always the week after Thanksgiving in the US. As a sales engineer, this time of year is exhilarating. It’s a chance to speak with customers, prospects, and technology leaders in the ever-evolving cloud-native and cloud computing spaces.

While I usually leave these events motivated and energized, something this year was different — marked by a noticeable absence. I could count on one hand the number of conversations I had with anyone in Developer Relations (DevRel) - whether they were community builders, developer advocates, or part of any similar role.

Admittedly, some of this could be circumstantial. For instance, KubeCon NA was held in Salt Lake City this year, a location that understandably kept some attendees away. But for me, it felt like a reflection of broader, more fundamental shifts happening in the tech industry.

What is DevRel?

DevRel has always been a role that defies simple definition. It’s not marketing, but it overlaps with marketing. It’s not sales, but it supports sales. It’s not customer success, but it builds relationships with users. And it’s certainly not engineering, though it requires technical skills.

It demands a rare combination of traits: technical aptitude, charisma, and customer empathy. Having worked in a DevRel role for six months, I can attest - it’s one of the hardest jobs I’ve ever had.

When done well, DevRel can be transformative. Companies like HashiCorp, Stripe, and Twilio have built exceptional brands in part due to their effective DevRel strategies. But the very ambiguity that has been DevRel’s strength is now starting to feel like a liability.

If You Can’t Measure It, You Can’t Manage It

During my brief stint in DevRel, I often asked my manager, “What does good performance look like in this role?” The answer, more often than not, was fuzzy.

For example, I spent hours answering questions in community Slack channels. It felt impactful, but we couldn’t quantify how it moved the needle on metrics like sales pipeline or customer retention. I gave virtual conference talks, but we struggled to tie those efforts to lead generation or brand impact.

I don’t think this is unique to my experience. Many DevRel professionals resist being measured by traditional KPIs, arguing their work is about “building community,” “fostering trust,” or “engaging developers.” While those outcomes are valuable, they’re hard to quantify in ways a CFO or board member cares about.

Then, external factors came along and forced everyone to take a harder look at roles like DevRel.

The VC Money Printer

For much of the 2010s, startups existed in a world of near-limitless capital. Low interest rates made borrowing cheap, and VCs funded companies generously, hoping to catch the next unicorn. In this environment, initiatives without immediate ROI—like DevRel—had room to thrive.

DevRel was a long-term investment: build trust, foster adoption, grow communities, and the payoff will come eventually. It worked as long as companies didn’t have to justify those investments on a quarterly basis.

But the money printer stopped. Rising interest rates and tightening VC funding forced startups to prioritize sustainability over growth at all costs. Every team came under scrutiny, and functions that didn’t directly contribute to revenue found themselves in jeopardy.

DevRel, for all its intangible benefits, became a glaring target.

DevRel as a Cost Center

In tough times, businesses ask hard questions: “Which teams are directly contributing to the bottom line?” DevRel often struggled to answer.

Sponsoring conferences? Expensive, with unclear ROI.
Running Slack communities? Valuable, but hard to tie directly to revenue.
Conference talks? Great for brand awareness, but hard to tie to lead generation
Social media posts? Effective at generating leads if done from company accounts, but very hard to measure from personal accounts

By contrast, roles like sales, marketing, and engineering have clear outputs tied to outcomes. Sales closes deals. Marketing generates leads. Engineering ships features. DevRel, sitting somewhere between these disciplines, risks being seen as neither fish nor fowl—doing a little of everything but owning none of it.

Product Led Growth

Compounding DevRel’s challenges is the rise of Product-Led Growth (PLG). In a PLG model, the product sells itself through intuitive design, freemium tiers, and frictionless onboarding. Developers don’t need a community advocate or a conference talk—they sign up, try the product, and decide for themselves.

Now, I have strong opinions on PLG—freemium tiers are becoming harder to sustain as the cost of compute rises. But the theory remains sound. Companies like my current employer, Tailscale, have nailed this approach. You can grow your user base organically without the overhead of a dedicated DevRel team.

This isn’t to say DevRel is useless in a PLG world, but it does mean the traditional role—focused on advocacy and education—may no longer be as critical.

The Future of DevRel

Despite the title of this post, I don’t think DevRel is truly dead. The need for technical advocacy and community building hasn’t disappeared - it’s just evolving. Many of the developer relations engineers that have survived this change have embraced the shift into marketing, leaning on their expertise as content generators to create documentation, tutorials, write blog posts and generate video content for YouTube. At Tailscale, we often seen incredible engagement with our content team and wonderful feedback from our YouTube channel. What’s been so obvious about the success here is that they’re measurable outcomes, and the people in these roles have embraced the challenges around having measurable goals.

Content generations is only one part of the DevRel flywheel, and I think I can see a world where we have specific personas tied to different parts of the business that have focus on the parts of a DevRel job that each line of business care about, such as:

Community Solutions Engineer

A blend of solutions engineering and DevRel, focused on driving adoption at the top of the funnel while contributing to measurable ARR metrics.

Community Customer Success

Ensuring existing customers are successful, reducing churn, and driving expansion revenue.

Community Data Engineer

Using analytics to tie activities to outcomes like product adoption, retention, or revenue growth.

The common thread? Measurability. The days of vague metrics like “community engagement” are over.

Is This Really the End?

What we’re witnessing isn’t the death of DevRel—it’s a wake-up call. The traditional model, reliant on soft metrics and long-term value propositions, can’t survive unchanged.

The good news is that there’s still a place for DevRel. Professionals who adapt—aligning their work with sales, customer success, or product teams—will continue to thrive. Those who resist change may not.

The industry has shifted. The question is: will DevRel shift with it?

Why the hell is your Kubernetes API public?

Sat, 23 Mar 2024 00:00:00 +0000

Do you ever really think about how you get access to your Kubernetes control plane? Whatever mechanism you use to provision your cluster, you get a KUBECONFIG and usually just go on your merry way to overcomplicating your infrastructure.

However, if you’ve ever looked at your KUBECONFIG you’ll see you have a server address.

You can check the health of your cluster by doing the following:

curl -k $(kubectl config view --output jsonpath='{.clusters[*].cluster.server}')/healthz

Assuming everything is working as expected (and if it isn’t, you should probably stop reading and go figure out what), it should return ok.

Has it ever come to your attention that this just works from a networking perspective? More than likely you didn’t have to a connect to a VPN, or SSH into a bastion/jump host host?

How did I know that? Well, because the vast majority of Kubernetes clusters are just hanging out on the public internet, without a care in the world.

If you search shodan.io for Kubernetes clusters you’ll see there’s almost 1.4 million clusters, readily accessible and open to the scary, scary world.

For reasons I can’t quite understand, we’ve sort of collectively decided that it’s okay to put our Kubernetes control planes on the public internet. At the very least, we’ve sort of decided it’s okay to give them a public IP address - sure you might add some security groups of firewall rules from specific IP addresses, but the control plane is still accessible from the internet.

To put this into some sort of perspective, how many of the people reading this get a cold shudder when they think about putting their database on the public internet? Or a windows server with RDP? Or a Linux server with SSH?

Established practices say this is generally not a good idea, and yet in order to make our lives easier, we’ve decided that it’s okay to let every person and their dog free access to try and make our clusters theirs.

Private Networks

As soon as you put something in a private subnet in the cloud, you add a layer of complexity to the act of actually using it. You have to explain to everyone who needs access to it, including that developer who’s just joined the team, where it is and how to use it. You might use an SSH tunnel, or a bastion instance, or god forbid that VPN server someone set up years ago that nobody dares touch.

We sort of accept these for things like databases because we very rarely need to get into them except in case of emergency, and we think it’s okay to have to route through something else because the data in them is important enough to protect.

In addition to this, when cloud providers started offering managed Kubernetes servers, most of them didn’t even have private control planes. It was only in the last few years that they started offering this as a feature, and even then, it’s not the default.

So the practice of putting a very important API on the internet has proliferated because it’s just easier to do it that way. The real concern here is that we’re one severe vulnerability from having a bitcoin miner on every Kubernetes cluster on the internet.

An alternative

I previously wrote about the Tailscale Kubernetes Operator and its ability to expose services running inside your Kubernetes cluster to your Tailnet, but it has another amazing feature:

It can act as a Kubernetes proxy for your cluster.

Say what now?

Well, let’s say you provision a Kubernetes cluster in AWS. You decide that you that in order to give yourself another layer of protection, you’re going to make sure the control plane is only accessible within the VPC.

If you install the Tailscale Kubernetes operator and set the apiServerProxyConfig flag, it’ll create a device in your Tailnet that makes it accessible to anyone on the tailnet. This means before you’re able to use the cluster, you need to be connected to the Tailnet. All of that pain I mentioned previously with Bastion hosts and networking just vanishes into thin air.

Let’s take it for a spin!

Installing and using the Tailscale Operator

Prerequisites

You’ll need to have your own Tailnet, and be connected to it. You can sign up for Tailscale and it’s free for personal use.

Once that’s done, you’ll need to make a slight change to your ACL. The Kubernetes operator uses Tailscale’s tagging mechanism so let’s create a tag for the operator to use, and then a tag for it to give to client devices it registers:

"tagOwners": {
		"tag:k8s-operator":           ["autogroup:admin"], // allow anyone in admin group to own the k8s-operator tag
		"tag:k8s":                    ["tag:k8s-operator"],
	},

Then, register an oauth client with Devices write scope and the tag:k8s-operator tag.

Step 1: Get a Kubernetes Cluster

There’s a lot of ways to do this, choose the way you prefer. If you’re a fan of eksctl, this page shows you how to create a fully private cluster.

You can of course use the defaults and try this out with a public cluster if you like, but I’m going to assume you’re doing this because you want to make your cluster private.

You may have to do this from inside your actual VPC, because remember, any post install steps that interact with the Kubenrnetes API server won’t work. I leverage a Tailscale Subnet Router to make this easier, more on this later.

Step 2: Install the Tailscale Kubernetes Operator

The easiest way to do this is with Helm. You’ll need to add the Tailscale Helm repository:

helm repo add tailscale https://pkgs.tailscale.com/helmcharts
helm repo update

Then install the operator with your oauth keys from the prerequites step, and enable the proxy:

helm upgrade \
  --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  --set-string oauth.clientId=${OAUTH_CLIENT_ID}> \
  --set-string oauth.clientSecret=${OAUTH_CLIENT_SECRET} \
  --set-string apiServerProxyConfig.mode="true" \
  --wait

Now, if you look at your Tailscale dashboard, or use tailscale status you should see a couple of new devices - the operator and a service just for the API server proxy.

You can now access your Kubernetes cluster from anywhere that has a Tailscale client installed, no faffing required.

Step 3: Use it

You’ll need to configure your KUBECONFIG using the following command:

tailscale configure kubeconfig <hostname>

This will set up your KUBECONFIG to use the Tailscale API server proxy as the server for your cluster. If you examine your KUBECONFIG you should see something like this:

apiVersion: v1
clusters:
- cluster:
    server: https://eks-operator-eu.tail5626a.ts.net
  name: eks-operator-eu.tail5626a.ts.net
contexts:
- context:
    cluster: eks-operator-eu.tail5626a.ts.net
    user: tailscale-auth
  name: eks-operator-eu.tail5626a.ts.net
current-context: eks-operator-eu.tail5626a.ts.net
kind: Config
users:
- name: tailscale-auth
  user:
    token: unused

Hang on a minute..

You probably have some questions. Firstly, what’s that unused token all about? What does noauauthmodeth mean in our operator installation? How does this work?

Well, if you run a basic kubectl command now (and assuming you’re connected to your Tailnet) you’ll get something back, but it won’t help much:

kubectl get nodes
error: You must be logged in to the server (Unauthorized)

What’s happened here? Well, the good news is, we’ve been able to route to our private Kubernetes control plane, but we’re not sending any information back about who we are. So let’s make a small change to our Tailscale ACL in the Tailscale console. Add the following:

	"grants": [
		{
			"src": ["autogroup:admin"], // allow any tailscale admin
			"dst": ["tag:k8s-operator"], // to contact any device tagged with k8s-operator
			"app": {
				"tailscale.com/cap/kubernetes": [{
					"impersonate": {
						"groups": ["system:masters"], // use the `system:masters` group in the cluster
					},
				}],
			},
		},
	],

By adding this grant, I’m assuming you’re an admin of your Tailnet.

Now, give that kubectl command another try.

kubectl get nodes
NAME                                              STATUS   ROLES    AGE     VERSION
ip-172-18-183-129.eu-central-1.compute.internal   Ready    <none>   13h     v1.29.0-eks-5e0fdde
ip-172-18-187-244.eu-central-1.compute.internal   Ready    <none>   3d18h   v1.29.0-eks-5e0fdde
ip-172-18-42-205.eu-central-1.compute.internal    Ready    <none>   3d18h   v1.29.0-eks-5e0fdde

That’s more like it.

As you can see here, I’ve solved two distinct problems:

I’ve made my public Kubernetes control plane accessible over a VPN, without needing to worry about routing and networking - Tailscale has handled it for me.
I’ve also been able to leveraging Tailscale’s ACL mechanism to provide authentication to Kubernetes groups in a clusterrolebinding.

noauth Mode

Now, if you’re already happy with your current authorization mode, you can still use Tailscale’s access mechanism to solve the routing problem. In this particular case, you’d install the Operator in noauth mode and then use your cloud providers existing mechanism to retrieve a token.

Modify your Tailscale Operator installation like so:

helm upgrade \
  --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  --set-string oauth.clientId=${OAUTH_CLIENT_ID}> \
  --set-string oauth.clientSecret=${OAUTH_CLIENT_SECRET} \
  --set-string apiServerProxyConfig.mode="noauth" \ # using noauth mode
  --wait

Once that’s installed, if you run your kubectl command again, you’ll see a different error message:

kubectl get nodes
Error from server (Forbidden): nodes is forbidden: User "jaxxstorm@github" cannot list resource "nodes" in API group "" at the cluster scope

The reason for this is a little obvious, the EKS cluster I’m using has absolutely no idea who jaxxstorm@github is - because it uses IAM to authenticate me to the cluster.

So let’s modify our KUBECONFIG to retrieve a token as EKS expects. We’ll modify the user section to leverage an exec directive - it should look a little bit like this:

apiVersion: v1
clusters:
- cluster:
    server: https://eks-operator-eu.tail5626a.ts.net
  name: eks-operator-eu.tail5626a.ts.net
contexts:
- context:
    cluster: eks-operator-eu.tail5626a.ts.net
    user: tailscale-auth
  name: eks-operator-eu.tail5626a.ts.net
current-context: eks-operator-eu.tail5626a.ts.net
kind: Config
users:
- name: tailscale-auth
  user:
    exec:
      apiVersion: "client.authentication.k8s.io/v1beta1"
      command: "aws"
      args: [
        "eks",
        "get-token",
        "--cluster-name",
        "kc-eu-central-682884c" # replace with your cluster name
      ]
      env:
      - name: KUBERNETES_EXEC_INFO
        value: '{\"apiVersion\": \"client.authentication.k8s.io/v1beta1\"}'

Now we’re able to route to the Kubernetes control plane, and authenticate with it using the cloud providers authorization mechanism.

Some FAQs

Why wouldn’t I just use a Subnet Router?

A common question I get asked is why wouldn’t I just use a subnet router in the VPC to route anything to the private address of the control plane? I leveraged this mechanism when I installed the operator, because my control plane was initially unrouteable from the internet anyway.

This is a legimate solution to the problem, and if you don’t want to use the operator in auth mode, keep living your life. However, one benefit you get by installing the operator and talking directly to it via your KUBECONFIG is being able to use Tailscale’s ACLs to dictate who can actually communicate with the operator.

If you recall our grant from earlier we were able to dictate who was able to impersonate users inside the cluster in auth mode, but with Tailscale’s ACL system we can also be prescriptive about connectivity. Consider if you removed the default, permissive ACL

//{"action": "accept", "src": ["*"], "dst": ["*:*"]}, # commented out because hujson

Then defining a group of users who can access the cluster, and adding a more explicit ACL:

"groups": {
		"group:engineers": ["jaxxstorm@github"],
	},
  // some other stuff here
"acls": [
		// Allow all connections.
		//{"action": "accept", "src": ["*"], "dst": ["*:*"]},
		{
			"action": "accept",
			"src":    ["group:engineers"],
			"dst":    ["tag:k8s-operator:443"],
		},
	],

I can be much more granular about my access to my cluster, and have a Zero Trust model for my Kubernetes control plane at the network level as well as the authorization level. Your information security team will love you for this.

When you provision the operator in the cluster, you can modify the tags it uses to even further allow you to segment your Tailnet, see the operator config in the Helm chart’s values.yaml

Just remember to ensure your oauth clients has the correct permissions to manage those tags!

Can I scope the access on the cluster side?

If you have the operator installed in auth mode, you can scope the access both at the network level (using the aforementioned tags) and the Kubernetes RBAC system.

Let’s say we want to give our aforementioned engineers group access to only a single namespace called demo.

First, we’d create the ClusterRole (or Role) and then cluster role binding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: engineers-clusterrole
rules:
- apiGroups: ["", "apps", "batch", "extensions"] 
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: engineers-rolebinding
  namespace: demo
subjects:
- kind: Group
  name: engineers # note the group name here
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: engineers-clusterrole
  apiGroup: rbac.authorization.k8s.io

Then update our Tailscale ACL to modify the grants:

"grants": [
		{
			"src": ["group:engineers"],
			"dst": ["tag:k8s-operator"],
			"app": {
				"tailscale.com/cap/kubernetes": [{
					"impersonate": {
						"groups": ["engineers"],
					},
				}],
			},
		},
	],

Now, if I try to access the demo namespace, I can do the stuff I need to do:

kubectl get pods -n demo
NAME                                     READY   STATUS    RESTARTS   AGE
demo-streamer-e3a170e4-85f4f7b88-gppcn   1/1     Running   0          16h
demo-streamer-e3a170e4-85f4f7b88-xc7gz   1/1     Running   0          16h

But not in the kube-system namespace:

 kubectl get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "jaxxstorm@github" cannot list resource "pods" in API group "" in the namespace "kube-system"

Wrap Up

As always with the posts I write on here, I’m writing in my personal capacity, but obviously I’m a Tailscale employee with a vested interest in the success of the company. Do I want you to sign up for Tailscale and pay us money? You bet I do. Do I want you to get your Kubernetes clusters off the public internet even if you don’t want to sign up for Tailscale and pay us money? Yes.

I’ve always had an uneasy feeling about these public clusters, and I can’t help but feel we’re one RCE away from a disaster.

So now you know how easy it is to get your Kubernetes control plane off the internet, what are you waiting for?

Free Kubernetes Load Balancers with Tailscale

Mon, 26 Feb 2024 00:00:00 +0000

Load Balancers are expensive.

If you’re using Kubernetes, they are also a necessity. Figuring out how to expose a Kubernetes workload to the world without a Load Balancer is a bit like trying to make a sandwich without bread. You can do it, but nobody is going to want to deal with it.

If you’re a startup trying to get your project off the ground, firstly, why are you using Kubernetes? Stop. However, if that’s the way you’re going - you’re looking at spending at least $10 a month for a load balancer in the cloud before you take data transfer costs into account. If you’re a hobbyist running a small Kubernetes cluster in your home lab, you might read the Metal LB documentation and think “I didn’t realise I needed to have a CCIE certification to make this work.”

Well, guess what! You don’t need to anymore. Thanks to Tailscale and it’s relatively new Kubernetes Operator you can now get access to Kubernetes workloads using native Service and Ingress objects without paying those greedy cloud providers a single cent.

Note: I recently joined Tailscale as a Solutions Engineer. Make of that what you will.

A cloud agnostic load balancer

Kubernetes services work on every Kubernetes distribution, but if you want to expose that service to the world, you’ll likely need a service of type=LoadBalancer. Without rehashing Kubernetes networking concepts, this mechanism will ultimately create something with an address that is external to the Kubernetes cluster. Let’s take a look at a very simple example on a DigitalOcean Kubernetes cluster:

Note: I chose Digital Ocean for this example because it offers a free control plane, but this scenario works for any cloud provider’s Kubernetes offering.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Here, we provision a standard Kubernetes deployment and service, and give the service type=LoadBalancer. Inside the Kubernetes cluster, there is a cloud controller manager that watches for services of type LoadBalancer and then provisions a load balancer in the cloud provider’s infrastructure.

You can see the result of this when the Load Balancer has provisioned:

k get svc
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.245.0.1      <none>          443/TCP        9m5s
nginx        LoadBalancer   10.245.97.151   <redacted>   80:31554/TCP   2m54s

If you visit the EXTERNAL-IP you’ll see your application, in this case, nginx.

This is all well and good, but as mentioned earlier, this has provisioned an actual resource in Digital Ocean, a Load Balancer. This is going to cost me at least $12 a month per node, which effectively double the prices of running compute.

As a proud Yorkshireman the idea of paying for something I don’t need to is anathema to me. Until recently, I didn’t have a choice. I had to pay for a load balancer.

Tailscale to the rescue

So let’s see what this could look like with Tailscale. We’ll install the Tailscale operator into our cluster, but first we need to knock out a few small steps.

Note: I am showcasing this with DigitalOcean Kubernetes, but this will work on ANY Kubernetes cluster, even your homelab. Try it!

Create your Tailnet

The first step along this journey is to sign up for Tailscale and create a Tailnet. This is a network that your Kubernetes cluster will join, and allows anyone on the Tailnet to access the Kubernetes cluster.

You should also install the Tailscale client on your device of choice. If you already have a Tailnet, skip to the next section.

Modify your ACL file

Before we connect Tailscale to our cluster, we need to make a few changes to our Tailscale ACL to allow the Tailscale operator to correctly authorize the new devices it’ll create.

Inside the tagOwners section of your Tailnet, you should add the following stanzas:

"tagOwners": {
    "tag:k8s-operator": [],
	"tag:k8s":          ["tag:k8s-operator"],
},

This will allow the Tailscale operator to create new devices and assign them to the correct ACLs.

Create an Oauth client

Next, we need to create some credentials that the Tailscale operator will use to create devices. If you’re wondering why we need to do this, it’ll all make sense shortly. In the Settings tab in the Tailscale console, navigate to Oauth clients and create a new one, with the following settings:

Make a note of the credentials it returns, as you need them for the next step.

Install the Tailscale Operator

Now we have our credentials, we can install the operator. Tailscale handily provides us with a Helm Chart, so let’s go ahead and install it:

helm repo add tailscale https://pkgs.tailscale.com/helmcharts # add the helm chart repo
helm repo update # update the repo
helm upgrade \
  --install \
  tailscale-operator \
  tailscale/tailscale-operator \
  --namespace=tailscale \
  --create-namespace \
  --set-string oauth.clientId=${OAUTH_CLIENT_ID} \
  --set-string oauth.clientSecret=${OAUTH_CLIENT_SECRET} \

Now we can make some magic happen - let’s see how we can connect to our nginx deployment without a LoadBalancer.

Create a LoadBalancer

Let’s deploy an nginx service, but add a single line to the manifest - the loadBalancerClass:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  loadBalancerClass: tailscale # add this!
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Wait a few minutes for the operator reconciliation to happen, then check the service:

NAME         TYPE           CLUSTER-IP      EXTERNAL-IP                                    PORT(S)        AGE
kubernetes   ClusterIP      10.245.0.1      <none>                                         443/TCP        28m
nginx        LoadBalancer   10.245.97.151   100.102.166.33,default-nginx.tail7fe3.ts.net   80:31404/TCP   22m

Something truly remarkable has happened here, with absolutely zero input on my part. If you’re on the same Tailnet as the Kubernetes cluster, you can visit the EXTERNAL-IP and see the nginx deployment.

curl default-nginx.tail7fe3.ts.net
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

What exactly has happened here? Let’s examine it a little.

The operator at work

If you have the ability to use the Tailscale CLI, you can run tailscale status and you’ll notice a few devices have appeared in your Tailnet.

tailscale status
109.46.90   macbook-pro-lbr      lee@         macOS   -
102.166.33  default-nginx        tagged-devices linux   idle, tx 900 rx 1452
95.163.103  doks-py-funnel-tailscale-operator tagged-devices linux   -

There’s one for the Tailscale operator I installed, but there’s also a distinct device for my nginx service, that has the same IP as the EXTERNAL-IP of the service. This is the magic of the Tailscale operator - it’s created a device that is accessible from the Tailnet, and it’s done so without needing to provision a cloud resource.

If I take a look in the Tailscale namespace, I can also see what’s actually happened - the operator installed a pod that is responsible for creating these devices:

k get po -n tailscale
NAME                        READY   STATUS    RESTARTS   AGE
operator-6cc69495c6-nck9x   1/1     Running   0          27m
ts-nginx-5jt4h-0            1/1     Running   0          4m1s

Describing that Tailscale pod will give me another interesting nugget of information - check out the Environment section

k get po -n tailscale ts-nginx-5jt4h-0 -o json | jq '.spec.containers[0].env'
[
  {
    "name": "TS_USERSPACE",
    "value": "false"
  },
  {
    "name": "TS_AUTH_ONCE",
    "value": "true"
  },
  {
    "name": "POD_IP",
    "valueFrom": {
      "fieldRef": {
        "apiVersion": "v1",
        "fieldPath": "status.podIP"
      }
    }
  },
  {
    "name": "TS_KUBE_SECRET",
    "value": "ts-nginx-5jt4h-0"
  },
  {
    "name": "TS_HOSTNAME",
    "value": "default-nginx"
  },
  {
    "name": "TS_DEBUG_FIREWALL_MODE",
    "value": "auto"
  },
  {
    "name": "TS_DEST_IP",
    "value": "10.245.97.151"
  }
]

Notice that TS_DEST_IP variable? It’s the IP of the Kubernetes ClusterIP for our service.

Going further

This is all well and good for a simple application, but what if we need HTTPS? How complex can this get? Kubernetes services will do okay for exposing basic TCP passthrough services, but what else can I do.

Well, this let’s try something a bit more interesting.

Create an Ingress

The Tailscale operator will also reconcile Ingress resources, and it’ll handle TLS termination for you as well. You need to allow your Tailnet to create HTTPs certificates first but once you have, you can deploy an ingress just as easily as a service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: ClusterIP # previously was a load balancer
  # also ensure you remove the loadBalancerClass line
  ports:
  - port: 80
  selector:
    app: nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
spec:
  defaultBackend:
    service:
      name: nginx
      port:
        number: 80
  ingressClassName: tailscale
  tls:
  - hosts:
    - nginx

Now, wait for the operator to reconcile, and then examine your ingress:

NAME    CLASS       HOSTS   ADDRESS                 PORTS     AGE
nginx   tailscale   *       nginx.tail7fe3.ts.net   80, 443   22s

Note: If you request a TLS certificate, it will take a little longer for connectivity to be established while a certificate is provisioned.

Look at that!

curl https://nginx.tail7fe3.ts.net
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

No need to install an ingress controller, no need to provision a cloud resource. Just a simple ingress resource and the Tailscale operator does the rest.

Public Services

So far, we’ve provisioned workloads that are only accessible on the same Tailnet. What if you want to expose a service to the wider internet?

Tailscale already has an amazing feature for this called Funnel and you can leverage this feature from the operator to make your applications accessible from anywhere in the world.

Add ACL permissions

You need to make a slight modification your Tailscale ACL file to use this feature. Add a block to your ACL like so:

"nodeAttrs": [
	{
		"target": ["tag:k8s"], // tag that Tailscale Operator uses to tag proxies; defaults to 'tag:k8s'
		"attr":   ["funnel"],
	},
],

Once you’ve updated your ACL, you need to make a single line change to any service or ingress.

Modify your service or ingress

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  type: ClusterIP
  ports:
  - port: 80
  selector:
    app: nginx
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    tailscale.com/funnel: "true" # add this annotation
spec:
  defaultBackend:
    service:
      name: nginx
      port:
        number: 80
  ingressClassName: tailscale
  tls:
  - hosts:
    - nginx

This annotation works on both services and ingress objects. Once applied, you can logout of your Tailnet, and still get access to the service:

# log out of my tailnet
tailscale logout
# get the ingress url
 k get ing
NAME    CLASS       HOSTS   ADDRESS                 PORTS     AGE
nginx   tailscale   *       nginx.tail7fe3.ts.net   80, 443   7m53s

# check access ingress
curl https://nginx.tail7fe3.ts.net

Magic!

Caveats

This is all incredibly powerful, and will save you money. However, there are 2 caveats worth mentioning

Supported URLs

Currently, the URLs generated for both services and ingresses are only tailnet addresses, so support for your own domain is currently not possible.

Ingress HTTPs

There is currently no mechanism to redirect HTTP requests to HTTPS. If you need that sort of functionality, you’ll need to provision an ingress controller. You can expose the ingress controller with a Tailnet service though, more to come on that in another post.

Wrapping Up

Tailscale really has made it incredibly easy to expose your Kubernetes workloads to the world without needing to provision cloud resources. This is only the beginning for what the operator can do, so watch this space for more exciting forays into the world of Tailscale and Kubernetes.

The 300% Production Problem

Thu, 28 Sep 2023 00:00:00 +0000

Earlier this year, I attended CfgMgmtCamp in Ghent and listened to Adam Jacob’s “What if Infrastructure as Code never existed” keynote.

Note: I’d like to extend a huge thanks to Adam for taking the time to review this post, and for an unnamed person for consistently reviewing these posts and for inspiring the thoughts here.

Not only is Adam an excellent speaker, but he can capture thoughts that most people can’t articulate and explain them in a way that revolutionises people’s thinking.

This talk was no exception. If you have yet to see it, I’d like to introduce you to something I’ve always known but have yet to appropriately identify: the 200% knowledge problem.

You can listen to Adam’s excellent explanation of the 200% knowledge problem here. My own explanation of this problem is:

To successfully use an abstraction, you need to understand the problem the abstraction is trying to solve and also understand how the abstraction has solved the problem.

Terraform Modules

My best example of this is when examining the Terraform module ecosystem. Terraform modules, in theory, are designed to solve specific problems in the cloud provider ecosystem. Taking an example like the AWS VPC module removes the need to understand all of the glue that AWS needs to successfully create a VPC, like route tables, subnets and NAT Gateways.

That’s the theory. The reality of these modules is that you need to understand the magical incantation of random functions and use dynamic to succeed.

In addition, the desire to create Terraform modules that meet every single user’s possible use case means that often, the module will expose the entire surface area of the APIs the module is managing to the user. Usually, this leaves you in a position of having to painstakingly read the whole module’s code before using it, and if something breaks, you’re shit out of luck.

Hearing Adam describe this problem has had my brain slowly creaking for a while. We’ve seen lots of literature in the past few years about the explosion of knowledge required to be a successful “DevOps Engineer”, “Site Reliability Engineer” or “Platform Engineer” or whatever that role’s title is this week. As I’ve noodled on this for the past few months, I’ve started to leverage the 200% problem and give it a moniker of my own: “the 300% production problem”.

The 300% Production Problem

The definition of the 300% problem is:

To successfully get an application into production, you need to be an expert in the application itself, the deployment target and the deployment methodology.

Each of these expertise layers is a full-time job or undertaking. If you’re lucky, you have more than one person who needs to be an expert on all these 3 layers. If you’re unlucky, you might read this and think, “holy shit, no wonder I’m burned out”.

Let’s examine these layers and see if we can develop some ideas to reduce the burden.

The Application

Now, if the application you’re deploying is an in-house application written by a team of super-smart developers, you’re in a good spot. If you’re deploying a third-party application, things get a little trickier.

One of the benefits of open-source third-party software is that it allows you to become an expert in the application itself, either by reading the code or the documentation. What’s interesting about the application stack is that there are layers of expertise all the way down. The database tier of the application itself, the software framework it’s written in, and how it’s packaged and maintained or performs under load or scale are just a few parts of how this breaks down.

The responsibility of expertise here essentially belongs to the “Dev” side of the “DevOps Divide”, especially for business logic applications.

The Deployment Target

The deployment target is where things start to explode in complexity, and often, the decisions you make as an engineer can directly affect the level of expertise required to succeed.

When discussing the deployment target, we’re pointing at the cloud provider and the varying layers under that. Cloud providers are hard, and they get even more challenging if you start to sprinkle Kubernetes on stuff to abstract away cloud provider APIs. That’s just compute, as well. Once you start factoring in the networking, data, queuing system requirements and all that other crucial production-grade stuff, you begin to realise just how hard this is.

The Deployment Methodology

The deployment methodology refers to the way you get your application and your infrastructure into a production level. If you’ve read other posts on this blog, you’ll know that my primary focus is Infrastructure as Code and being an expert in that IaC tool is a requirement to successfully get things into production. Now, I’m not going to turn this into another rant about why DSLs are stupid, but when you consider here that the language you use to author your infrastructure is part of this expertise, I’d ask myself this question: do you really only want 4 or 5 people in your company to be the experts in the deployment methodology, or do you want lots of experts?

Multiplying the problem

Once you start to think about problems through the lens of the 300% Production Problem, you begin to realise why there’s a burgeoning backlash against some popular ideas in our industry.

Microservices

Microservices took off as an idea because of the enormous scale needed for some organisations, but what ends up happening is that you multiply the 300% problem across a shitload of applications. Give those application teams the full power and responsibility of “owning their applications in production”. You can offset some of this by letting them choose their infrastructure and deployment methodology, but it’s a lot to ask everyone to be an expert in all 3 of these domains.

Kubernetes

It’s an established meme now that adding Kubernetes increases complexity, but it multiplies the complexity in two areas of the 300% problem. Not only is Kubernetes layered on top of your existing infrastructure, but it also requires you to be an expert in all of the facets of the deployment methodology, whether it be the abject misery of Go templated Helm Charts or building an operator for everything which is the proposed solution of some people who really just want to watch the world burn.

Cloud

Yes. There’s a cloud backlash slowly starting to form. I’m not going to share my thoughts in this post, but if you consider the surface area of most cloud providers, the sheer amount of services and the idiosyncrasies of those services, it’s unsurprising that people are starting to feel cognitive overload trying to meet their third of the 300% problem on the infrastructure side.

What do we do?

The ultimate solution to the 300% problem will be the same across the 3 pillars. Simplicity. Making decisions that reduce the amount of knowledge required to become an expert in one of those 3 pillars will dramatically affect your overall success rate at getting your applications into production.

This isn’t new ground. People have been writing about keeping “Keep it simple stupid” for a long time. The problem is that the “simplicity” is often defined by the people building the system, and those people are experts already in the system they’re building.

When I shared this post with Adam, he gave me an excellent analogy which helped me round this post out:

You want simplicity where it benefits the user, which often requires increased complexity for the developer. My analogy for this is how complex modern cars are, but you push a button to “start” them. Vs a very simple model t, that breaks your arm if you start it wrong.

As usual, Adam has done a much better job of encapsulating the core ideas in this post than I have, but to expand on it, the missing piece to me is the “producification” of the systems we’re building.

What is often overlooked is how important it is that decisions and responsibilities that can be shared are made to appease the opinions of a small number of potential experts rather than the broader organisation. Intelligent individuals with political power will happily introduce this framework, that deployment model, or the other tool because they like it and are experts and they believe it’s simple, which ultimately it is - to them.

Hopefully, if you’re reading this and you’re trying to build something that simplifies a process in your organisation, you’ll consider the 300% problem, and make it simple for everyone, not just yourself.

DSLs are a waste of time

Mon, 04 Sep 2023 00:00:00 +0000

If you’ve read this blog before, or are unfortunate enough to have an actual personal relationship with me, you’ll know that I have strong opinions and can be, shall we say, passionate about them. For posts on this blog, I try to share those opinions and avoid directly addressing the people who hold the opposing view.

However there’s a schism happening in the infrastructure as code world at the moment with the announcement that HashiCorp is changing the software license that it uses for its products. As a result, and for reasons that have been covered in great depth by people far more qualified than I am, this has resulted in a set of vocal competitors to Terraform Cloud and Terraform Enterprise to “fork” (and that’s a loose interpretation of the word because because all we have at the time of writing is a markdown file and some gifs) called OpenTF.

As we watch these two competing factions live out Friedrich Glasl’s model of conflict escalation in public, I have found myself asking a question that has always lingered in the back of mind while working in my day job at Pulumi..

Why the fuck does everyone love this domain specific language so much?

Hold your horses, Terraform isn’t just a DSL!

Your first reaction will undoubtedly be “well, it’s not the DSL we love, it’s Terraform! It’s made me more productive and solves real world problems!”

I understand the impact Terraform has had on Infrastructure as Code and DevOps in general. I’m lucky enough to consider two of the top 10 contributors of all time to Terraform close personal friends and we generally hold similar perspectives on things (although, one of them has appaling taste in beer, I know you’re reading this).

I used Terraform from the very first version, back when every time you ran terraform apply you sort of closed your eyes and hoped you didn’t break everything. I watched in wonder as Terraform introduced modules, started adding hundreds of providers to support basically every cloud provider you could think of. As our industry started to migrate to “cloud first” or as the marketing behemoth likes to call it, “cloud native” I migrated by world view from one of “configuration management” to “infrastructure as code” and became a subject matter expert.

All of this is to say, I get it. I understand why people want to use Terraform, and I can sort of understand why people are so upset about the license change and want to support the forking of the product so it stays “open source”.

The problem here is that while you can say “Terraform is more than a DSL” - if you ever back the layers or try have a conversation with someone who’s a true dyed in the wool Terraform zealot and start to try and understand where they’re coming from, you begin to realise they don’t love Terraform, they love this weird half language because they’re an expert in it, and they’ve built their entire career on being an expert on something you can only use for one specific use case.

A path to nowhere

In college I spent hundreds of hours playing Guitar Hero. I would play songs over and over again until I could get through them on expert mode. My ultimate goal was to be able to at least finish the toughest song on the game, Through The Fire and Flames on the toughest possible difficulty. A friend of mine was an accomplished guitar player and as a passing comment, pointed out “imagine if you’d spent the same amount of time playing an actual guitar”.

It never really registered for me at the time what he was saying, but as the years have gone by and I’ve put thousands of hours into being an “expert” in infrastructure, I’ve started to realise I made the same mistakes in my career as I did with guitar hero.

DSLs are everywhere

I’ve used a DSL before, you see. Even before Terraform. I spent a whole 7 or 8 years writing thousands of lines of DSL.

Puppet has its own Ruby based DSL it now calls the Puppet language. When all of the problems in the infrastructure space existed at the operating system instead of the API layer, Puppet was everywhere and I loved the language because it meant I didn’t have to be a “software developer”. I didn’t want to write PHP or Python, I wanted to manage infrastructure, and Puppet’s DSL let me do that.

The problem with this mentality was that when the industry inevitably changed around me, I was left with an expertise and knowledge of a language that was now, effectively useless. I haven’t written a single line of Puppet language in a long time, and I don’t miss it at all. I don’t miss the syntax, parsing the docs and trying to figure out how these built in functions actually worked and mapping those ideas to problem sets.

Configuration complexity clock

The fact Puppet and Terraform (and other tools, of course) tend to converge on a DSL to solve infrastructure problems doesn’t appear to be a coincidence. Infrastructure (whether it be at the operating system layer or the API layer) is at its core, a sea of configuration. When dealing with configuration, the configuration complexity clock dictates that eventually, you’re going to try and configure a DSL to manage it.

If you’re familiar with the configuration complexity clock, you’ll notice that the next step in the evolution of infrastructure configuration is to “hard code” the values again, which isn’t really a possibility because, well, have you seen how much infrastructure we’re dealing with? So what’s really happening here is that the DSLs (and particularly, the HCL DSL) is becoming it’s own progamming language, adding new constructs and methods which increase the amount of complexity in the language itself.

This is all well and good, because the features solve problems people have, the real issue here is that it’s still a DSL, and you’re still learning to play guitar hero instead of actually learning a useful skill you can learn elsewhere - playing the guitar.

Complexity is in the eye of the beholder

One of the arguments I always hear when talking to people about this problem is the argument that Terraform’s DSL reduces the amount of complexity in the code because of its limited feature set. Putting aside the fact these people are arguing that the lack of features is itself a feature it doesn’t really hold up to any scrutiny.

If you do a quick search of a sufficiently complex Terraform module, you’ll see all kinds of craziness. I just picked a random module from the Terraform registry and looked at the main.tf and found this:

resource "aws_route_table_association" "redshift" {
  count = local.create_redshift_subnets && !var.enable_public_redshift ? local.len_redshift_subnets : 0

  subnet_id = element(aws_subnet.redshift[*].id, count.index)
  route_table_id = element(
    coalescelist(aws_route_table.redshift[*].id, aws_route_table.private[*].id),
    var.single_nat_gateway || var.create_redshift_subnet_route_table ? 0 : count.index,
  )
}

resource "aws_route_table_association" "redshift_public" {
  count = local.create_redshift_subnets && var.enable_public_redshift ? local.len_redshift_subnets : 0

  subnet_id = element(aws_subnet.redshift[*].id, count.index)
  route_table_id = element(
    coalescelist(aws_route_table.redshift[*].id, aws_route_table.public[*].id),
    var.single_nat_gateway || var.create_redshift_subnet_route_table ? 0 : count.index,
  )
}

Now, don’t get me wrong, I know what this is doing, and I also understand why it has to be implemented this way, but if you’re not an expert in this space, you probably have a whole bunch of reasonable questions:

Why are we using count here?
What is coalescelist?
What is element?
What is local?

And that’s just the first few lines. I’m not trying to pick on this module, I’m just trying to demonstrate that even in a language that is supposed to be “simple” and “easy to understand” you can still end up with code that is complex and difficult to understand.

Terraform’s DSL allows you to use abstractions (modules), nest resources inside those abstractions and then create varying levels of indirection when trying to instantiate resources. These things are all necessary when trying to truly create an ecosystem that allows reusability and sharing, but of course that brings with it complexity.

My personal opinion here is that when people say “it’s less complex” - what they actually mean is “I understand it”. Which is totally fine, and a reasonable position to be in, but it’s not the same thing. If you’re a “devops engineer” or “platform engineer” (or whatever the job title is called this week) and you look at the application code you’re supporting - which is all object orientated and scary and you can’t figure out where this values comes from - put yourself in the position of an application developer trying to look at YOUR Terraform code and think about their perspective. It’s just as difficult, and it was your job to learn Terraform’s DSL, the application developer has a full time job AND now you’re expecting them to learn this weird DSL too?

There’s more to Terraform than a DSL

It’s true, that Terraform is more than a DSL. There’s an entire ecosystem of modules, which seem to be just a grouping of Terraform resources (which expose every single possible API element to the user, but that’s a post for another day) and tutorials and even entire companies set up around the Terraform ecosystem. That’s all very compelling when choosing and considering how you’re going to manage your infrastructure, but what’s pretty clear after the last few weeks is that this ecosystem is becoming increasingly fragile. Once the Terraform fork happens (and again, we don’t have anything yet other than a README and some gifs) - will that ecosystem continue to grow and thrive, or will it split and become two separate ecosystems?

If you look at the history of software that has been forked, you’ll see that rarely do the two forks grow along the same path. When the maintainers of OpenTF inevitably decide that they want to add enhancements that require language changes, you as the end user end up in a situation where you’re effectively back to square 1, learning (potentially) a new DSL.

What’s the alternative?

As you can probably ascertain, I think there’s a better path forward here. I’ve made the case before that programming languages are the better authoring model for cloud infrastructure because they allow you to express the complexity of your infrastructure in the most flexible, intuitive way. My argument here is that if you want to continue to cling on to DSLs, you’re going to end up in a situation where you’re going to have to learn a new DSL every time you want to do something new.

I’m not going to make the explicit argument you should use Pulumi as your IaC tool because that would make me even more of a shill than I am now. What I will say is this: in this pivotal moment for the industry where Terraform gets forked, where the ecosystem is going to fracture and diverge, maybe you as a user can ask yourself the question - should I really go and learn another DSL? My sincere hope is that either Terraform or the OpenTF folks decide to invest in the CDK model. It’s an inferior programming language authoring model to Pulumi’s, but at least the concepts you learn in your chosen programming language are applicable elsewhere in your career. Understanding how to handle complex data structures in say, Python is something which might help you fix bug’s in application code, you can’t throw a DSL at that.

Structuring your Infrastructure as Code

Thu, 17 Aug 2023 00:00:00 +0000

If you’re thinking of migrating to another infrastructure as code tool (and why would you, everything is great in the IaC world now, right?!) you might find yourself asking yourself a fundamental question when you get started: how do I structure things in a way that scales well and stands the test of time?

There’s no canonical answer. Everyone does things slightly different, and different tools have different ideas on the best way.

In my day to day role as a Solutions Engineer at Pulumi I get to answer this question a lot. Customers are migrating from other IaC tools and they want to take this opportunity to think about the way they’d like to structure things.

This blog post is designed to detail my high(ish) level thoughts on the concepts and principles I like to use, and why. As we explore these concepts, I’ll talk about some of the lessons I learned from my time in configuration management and the myriad IaC tools I’ve used before today.

A lot of the concepts in this post are focused on Pulumi, but lots are broadly applicable to other tools.

Layers

I’m sure my system administrator background is showing, but I like to think about infrastructure through the concept of layers similar to the OSI Model. Most of the layers I’ll outline here closely mirror the OSI model, but what you’ll likely want to do before you create your Git repo or write a single line of code, is group your cloud infrastructure into layers. The reason why will become apparent later.

Layer 0: Billing

The billing layer is where you sign up or input your credit card. Each cloud provider does this differently

AWS: Organization
Azure: Account
Google Cloud: Account

In my experience, while there’s API for this stuff you likely don’t want to manage this layer with IaC, so do yourself a favour and do it manually.

Layer 1: Privilege

The privilege layer is how you fundementally separate access in the cloud provider. Again, each provider does this a little differently.

Example Resources

AWS: Account
Azure: Subscription
Google Cloud: Project

You might want to manage this layer with IaC, but you need to decide how that’d work. Personally, I find that the API level support for this layer and the rarity of needing to perform this operation means it’s often easier to manage this layer manually.

Layer 2: Network

Now we’re getting to the layers that’ll should definitely be managed by IaC. The network layer is foundational to how everything will work in your infrastructure, and includes things like a VPC, subnets, NAT Gateways, VPNs, and anything else that facilitates network communication.

Example Resources

AWS: VPCs, Subnets, Route Tables, Internet Gateways, NAT Gateways, VPNs
Azure: Virtual Networks, Subnets, Route Tables, Internet Gateways, NAT Gateways, VPNs
Google Cloud: VPCs, Subnets, Route Tables, Internet Gateways, Cloud Nat, VPNs

Layer 3: Permissions

Now we’ve laid down a network layer, we need to allow other people or applications to talk to the cloud provider API. IAM roles, or service principals live in this layer.

Example Resources

AWS: IAM Roles, IAM Users, IAM Groups
Azure: Service Principals, Managed Identities
Google Cloud: Service Accounts

Layer 4: Data

The data layer is where the resources you’re managing really start to open up. This is where you’ll find things like databases, object stores, message queues, and anything else that’s used to store or transfer data.

Example Resources

AWS: RDS, DynamoDB, S3, SQS, SNS, Kinesis, Redshift, DocumentDB, ElastiCache, DynamoDB
Azure: SQL, CosmosDB, Blob Storage, Queue Storage, Event Grid, Event Hubs, Service Bus, Redis Cache
Google Cloud: Cloud SQL, Cloud Spanner, Cloud Storage, Cloud Pub/Sub, Cloud Datastore, Cloud Bigtable, Cloud Memorystore

Layer 5: Compute

The compute layer is where your applications actually run - this is where you’ll find things like virtual machines, containers, and serverless functions.

Example Resources

AWS: EC2, ECS, EKS, Fargate
Azure: Virtual Machines, Container Instances, AKS
Google Cloud: Compute Engine, GKE

Layer 6: Ingress

Layer 6 is where you’ll find the resources that allow your applications to be accessed by the outside world.

Example Resources

AWS: Application Load Balancers, Network Load Balancers, Classic Load Balancers, API Gateways
Azure: Application Gateways, Load Balancers, API Management
Google Cloud: Load Balancers, API Gateways

Layer 7: Application

Once we’ve provisioned all the supporting infrastructure, we now need to actually deploy the application itself. This is where things really get a little tricky and depend entirely on your application’s deployment model, technology and architecture.

You might choose not to use IaC for application at all, but if you do..

Example Resources

AWS: Lambda, ECS Tasks, Kubernetes Manifests, EC2 User Data
Azure: Azure Functions, Kubernetes Manifests
Google Cloud: Cloud Functions, Kubernetes Manifests

Visualization

If you’re a visual learner like me, you might find this visualization helpful:

Layer	Name	Example Resources
0	Billing	AWS Organization/Azure Account/Google Cloud Account
1	Privilege	AWS Account/Azure Subscription/Google Cloud Project
2	Network	AWS VPC/Google Cloud VPC/Azure Virtual Network
3	Permissions	AWS IAM/Azure Managed Identity/Google Cloud Service Account
4	Data	AWS RDS/Azure Cosmos DB/Google Cloud SQL
5	Compute	AWS EC2/Azure Container Instances/GKE
6	Ingress	AWS ELB/Azure Load Balancer/Google Cloud Load Balancer
7	Application	Kubernetes Manifests/Azure Functions/ECS Taks/Google Cloud Functions

Principle 1: The Rate of Change

One of the difficult concepts to quantify when thinking of these layers is the rate of change that these resources undergo when you’re managing them. The lower layers generally will change less frequently than your higher layers, and in addition to this these layers are generally most fraught with risk when changing them.

You might be wondering why this matter - the answer to this is because when structuring your IaC, you’ll want to consider how you group resources together in your chosen IaC’s encapsulation mechanism. For example, Pulumi uses the concept of projects to group resources together.

When creating and defining resources in a Pulumi project, the fundamental consideration you need think of when adding a resource is “which layer does this resource live in?”. You generally shouldn’t have resources from different layers in the same project, because the rate of change of those resources will be different and the risk of changing them is different.

Principle 2: Resource Lifecycle

Note: This principle comes to you thanks to my wonderful colleague Ringo De Smet, who reminded me of the importance of breaking the rules when reviewing this post

As with all Principle in life, there are situations where principle 1 doesn’t broadly apply.

There are resources within the above layers where you might think “ah! this is a network resources so I’ll put it in my network project” but the lifecycle of the resource doesn’t necessarily fit as a shared resource. A great example of this is an AWS security group.

Security groups are generally specific to another resource - perhaps an application you’re deploying, a loadbalancer that’s shared or maybe a database in the data layer. With these resources, it’s generally best to consider the overall lifecycle of the dependent resources when deciding where to put it.

My rule of thumb here is this - if I wanted to provision this resource in a different environment, or better yet, destroy it - what other resources do I want to destroy at the same time?

Another great considering for this is the permissions layer. I already mentioned when discussing permissions that you’ll need to think about that layer as shared permissions, application specific permissions are entirely different - they really want to go directly with your application deployment code.

The summary here is: don’t be afraid to break the first principle, but make sure when you’re doing it you’re thinking about the resource lifecycle.

Principle 3: Repositories

The mono-repo vs multi-repo debate is one that’s will rage long after we’re all done with cloud computing and have migrated back to physical infrastructure, and I’m not going to try and solve it here. What I will say is that I’ve seen both work well, and both work poorly.

When it comes to IaC repositories, I again come back to our layering system and make differing decisions for where the code for deploying the repo should live is based on the layer.

The Control Repo

For the foundational, shared aspects of the infrastructure, I generally like to include those projects in a single mono-repo which back in my days of using Puppet we called a control repo. I still like to use this nomenclature.

If we refer back to our layers, I generally like to ensure layers 1 and 2 in this shared repo. Things get a little trickier once we get to layer 3, the permissions layer. At this stage, we need to decide if the resource itself is shared or not. A good example of this is an IAM role that might be used for human users instead of application user. This is generally going to be shared across multiple humans and teams, so it’s a good candidate for the control repo.

Layer 4 really depends on your application architecture. If you have a message bus spanning multiple applications, putting it in your control repo probably makes sense, but if you have a database that is only used by a single application, you likely don’t want it in the control repo for a variety of reasons.

Layer 5 again depends on your organisation’s permission model and cloud architecture. It’s not uncommon to share shared compute like an ECS cluster or Kubernetes cluster which spans many applications, so including it in an application repo probably isn’t going to make much sense. However if you’re isolating compute on a per-application basis, you’re almost certainly going to want to make this application specific.

Layer 6: As it likely becoming an obvious trend, you’ll need to take your application architecture and permission model into account. If you’re using a shared load balancer and routing traffic that way, you’ll likely want to include it in the control repo, but if you’re using a per-application load balancer you’ll want to include it in the application repo.

Application Repositories

At the very least, if you’re using IaC to deploy your application, having a deploy/ directory in your application repo is a great starting point. If you use Pulumi and want to use the same language as your application to do your deployments, you might consider having all of your dependencies in a single package.json or requirements.txt depending on your chosen language.

You’ll need to think about the rate of change here when you’re defining projects to group resources together. Do you perhaps need to separate your database layer and your application layer resources? I’d argue that you do, because the rate of change of your application layer is likely going to be much higher than your database layer, but you’ll need to make a decision that makes sense for your organisation and project.

Why do this?

The primary reason for making the decision to use both mono-repos and keeping deployment code with applications is built from a perspective of ownership and orchestration.

Foundational infrastructure at layers 1, 2 and possibly up to layer 5 is an order of operations problem and a workflow orchestration problem. In most circumstances, you’ll be creating resources that depend on other resources while building the IaC graph.

By deciding to break the resources into different projects, you can create a workflow that allows you to deploy the resources in the correct order. You’ll be able to utilize Pulumi stack references to share resources between stacks and projects, but you’ll need to ensure that a resource in a project in layer 2 that depends on a project in layer 1 has been created and resolved first.

In a mono-repo, this is as simple as ensuring that the workflow or CI/CD tool runs the projects in the correct order, but in a multi-repo implementation, it becomes a complex orchestration problem that likely involves multi repo webhooks and a lot of duct tape.

Application repos are far enough down the layering system that all of the infrastructure required to run your application will be in place. Placing application deployment infrastructure code in the application repo allows you to give the application developers full ownership of their code from writing and features to getting them into production.

Principle 3: Encapsulation

Once you’ve made the foundational decisions above, you’ll be well on the way to structuring a well defined set of infrastructure as code patterns, but the final thing you’ll need to consider is how you’ll share resource patterns across your control repo and application repos.

Every IaC tool has a different way of managing this. In Pulumi you can create a Component Resource for a single language or if you want to support multiple language, you might want to create a Pulumi Package, but the reason for doing this is the same: you want to encapsulate a set of best practices that you can share across multiple projects.

A good consideration for for when to start encapsulating resources is to think about your organisational structure and application architecture. If you’re only one team deploying a single application, you might not need to go down the path of encapsulating anything, but if you’re a platform team that’s likely to support dozens of teams to deploy to a shared layer 5 compute resource, creating a Pulumi package that encapsulates the best practices for deploying your application or creating a package for a best practice object storage bucket which has the required permissions is going to save the teams you’re supporting a lot of time.

These encapsulations should be in their own, distinct repository. You’ll want to version these encapsulations in the same way you version and release your applications - follow semver and make sure you create an API that your downstream users can use.

As your downstream users start to depend on these encapsulations, you can introduce concepts like unit testing to make sure you don’t break userspace with your infrastructure.

Pitfalls

A common mistake I see at the encapsulation layer when adopting Pulumi is trying to avoid object orientated principles and using a what I like to call the “function based approach”.

As an example of this, you might try and encapsulate some resources into a function. In TypeScript it’d look like this:

export function createBucket(name: string) {
    return new aws.s3.Bucket(name);
}

and in Python like so:

def create_bucket(name: str) -> aws.s3.Bucket:
    return aws.s3.Bucket(name)

The problem with this implementation of an abstraction is that it creates a nested mechanism that is difficult to manage successfully.

If you use a component, you get you get an abstraction mechanism that is much more native to the way the language works. In TypeScript, it looks like this:

export class Bucket extends pulumi.ComponentResource {
    public readonly bucket: aws.s3.Bucket;

    constructor(name: string, args: BucketArgs, opts?: pulumi.ComponentResourceOptions) {
        super("lbrlabs:index:Bucket", name, {}, opts);

        this.bucket = new aws.s3.Bucket(name, args, { parent: this });
    }
}

and in Python:

class Bucket(pulumi.ComponentResource):
    def __init__(self, name: str, args: BucketArgs, opts: Optional[pulumi.ResourceOptions] = None):
        super().__init__("lbrlabs:index:Bucket", name, {}, opts)

        self.bucket = aws.s3.Bucket(name, args, parent=self)

While the instantiation of the resource is more complex, the manageability of this over time is exponentially easier. Trust me, I’ve untangled this mess before.

Putting it together

An example is worth a thousand words, so let’s take a look at a hypothetical control repo and application repo.

Control Repo

Let’s say we’re going to be super original and call our repo infrastructure. Here’s how that might look:

├── certs
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── cluster
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── README.md
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── shared_database
│   ├── Pulumi.development.yaml
|   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── components
│   ├── requirements.txt
│   └── venv
├── domains
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── cache
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── shared_example_app
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── README.md
│   ├── __main__.py
│   ├── productionapp.py
│   ├── requirements.txt
│   └── venv
├── shared_bucket
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
├── vpc
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   ├── __main__.py
│   ├── requirements.txt
│   └── venv
└── vpn
    ├── Pulumi.development.yaml
    ├── Pulumi.production.yaml
    ├── Pulumi.yaml
    ├── __main__.py
    ├── requirements.txt
    └── venv

You can see here that we’re using Pulumi stacks to target differing environments (in this case, development and production), and creating a new project for different layers and resources.

You’ll likely also notice that I’ve been quite liberal with my use of directories for each set of services. I’m not grouping all of the network/layer 2 resources into a single project, however I’m following the layering principle by not grouping any resources from different layers into the same project.

You can definitely reduce the number of projects here (for example, you might choose to groups the VPC and VPN projects together in a network project) but I generally find that projects/directories are “free” and reducing the blast radius of changes makes people feel comfortable about contributing to these shared elements.

Application Repo

Once we get to our application repo, it’s a lot harder to be prescriptive, but let’s say we have a simple Go application called example-app. Here’s how that might look:

.
├── Dockerfile
├── Makefile
├── docker-compose.yml
├── deploy
│   ├── Pulumi.development.yaml
│   ├── Pulumi.production.yaml
│   ├── Pulumi.yaml
│   └── main.go
├── go.mod
├── go.sum
├── main.go
├── readme.md
└── README.md

Hopefully this is fairly self explanatory, you’ve got your application and mechanisms for local development with a Dockerfile and Makefile, and we can put our Pulumi code in a deploy/ directory.

Encapsulation Repo

Finally, let’s take a look at an example encapsulation repo. These repos can be quite complex, so as an example, take a look at this Pulumi package which encapsulates some level compute here.

Conclusion

We’ve covered a lot of ground here, but hopefully this has given you some ideas on how you might want to structure your infrastructure as code. If you’re using Pulumi, as with all my content, always open to hearing better ideas!

Authenticating to AWS the right way for (almost) every use-case

Mon, 05 Sep 2022 00:00:00 +0000

One of my favourite things about AWS is their ability to make the wrong decision easy and the right decision hard. Our world is moving towards managed services and “off the shelf” experiences for provisioning infrastructure and deploying applications. AWS itself has gotten the memo with its introduction of services like AWS Proton, however its real value proposition is in its ability to provider building blocks for the infrastructure needs of today.

It largely does a good job of providing these building blocks, but where it really falls down is in the “reasonable defaults” category with most services. Nowhere is this more obvious than in its authentication strategy.

Authentication vs Authorization

Before I go off into a rant about why AWS authentication is broken by default, I’d like to talk about where AWS Identity and Access Manager’s responsibilities lie. The same service, IAM, is largely responsible for both authentication and authorization within AWS. Let’s define those before we explain IAM’s role.

Authentication is the mechanism by which you tell a service who you are. The most common form of authentication is a username and password, but there are many other ways of providing authentication like AWS access keys and secret keys. You send your credentials over to a service and it verifies those credentials are correct. As an aside here, one of the biggest problems with the internet is that credentials are designed to verify who you are but don’t actually verify the person owning the credentials is the person who’s supposed to. Multi-factor authentication is designed to help with this, but there’s still lots of room for improvement.

Authorization is the way the service knows what you’re allowed to do. An AWS IAM policy is a document that give you the ability to dictate that once someone or something has authenticated to AWS, this is the actions they’re allowed to perform.

I have a lot of time for AWS authorization via IAM policies because they’re extremely verbose and operate on a whitelisting basis, meaning by default you can’t really do anything at all.

However, the “default” mechanism for providing access to AWS is absolutely shit. Here’s why.

The AWS Credential Problem

If you’re a user of AWS, you probably have an AWS Access Key or AWS Secret Key stored on your machine right now. Go on, take a look in ${HOME}/.aws/credentials and have a look if you have them. Do you? That’s bad.

If you don’t have AWS credentials on your personal laptop, bust open your companies flagship application and do a quick search for AWS_ACCESS_KEY. Do you see anything? That’s really fucking bad.

If you answered yes to either of these, it’s best to keep reading. If you answered no, keep reading anyway.

Credential Rotation

The reason it’s bad is simple: static credentials are very easy to steal or extract for an attacker. There are lots of ways this can happen: you could leave your laptop on a train (if you live in a civilized country with access to public transit). You could accidentally check them into version control on the public internet. You might even accidentally blurt them out to someone while in a bar instead of your phone number if you’ve memorized them for some reason. However you choose to tell the world about your authentication keys for AWS, the impact is the same: those keys then need to be manually revoked by you because they have an unlimited lifespan.

The general solution to this problem is to regularly rotate your IAM credentials. AWS tries to do its best to make you do this by shaming you on the AWS Users page:

The problem with rotating these credentials is that it’s such a bloody faff. You have to update the keys wherever they’re used and stuff might break when you’re doing it and I really just can’t be arsed, what’s the worst that could happen?

Well the answer is that it could cost you a lot of your money or your companies money as this lovely Reddit search of the AWS subreddit shows:

If you head into Reddit’s cesspit for a second and read one of those threads, you’ll see the advice to “make sure you have MFA turned on!” which is very good advice. The problem is, AWS access keys and secret keys don’t need MFA to be effective. If someone has your keys, they can do a whole bunch of stuff, including change your password to lock you out. That’s not good, is it?

Temporary Credentials to the rescue

The obvious solution to this is to just not use AWS Access Keys or Secret Keys, but you still need to authenticate to the AWS API to tell it who you are! Luckily, AWS has a solution for this in the form of Temporary Credentials

Session Tokens

AWS can provide you with an AWS Access Key and an AWS secret key via AWS STS, its security token service. Credentials generated via STS are very similar to the tokens you see in the AWS IAM User interface, but they also come with an AWS Session Token which requires you to specify duration. Once that duration has expired, the issued credentials become inactive and can’t be used, so you now need to generate some more.

Hang on…

Yes, that’s right. You’ll have AWS credentials you have to constantly renew in order for them to be effective. You also need to have already authenticated with AWS to retrieve temporary credentials! If this sounds like a lot of work with a very limited amount of benefit, read on.

Back to Authentication

I started off making the point that AWS makes the right thing hard to do, and in the case of temporary credentials, the difficulty comes in knowing what the hell is the right thing to do. AWS makes it really fucking easy to generate long standing AWS credentials, in a lot of ways, the default for AWS is to use AWS IAM Users. It doesn’t tell you about all the alternatives to IAM users, when to use them and how to implement them at all.

While it may or may not be clear to you that temporary credentials provide a huge security benefit, what may not be clear at this stage is how the hell you actually generate them.

The answer to that largely depends on your current use case. Generating temporary credentials is different depending on if you’re a human, machine or application. So now I’ve spent the better part of this post explaining things you probably don’t care about like an online recipe that makes you scroll for 10 minutes, lets actually talk about the different strategies for authenticating to AWS.

I want to..

From here, we’re going to detail the scenarios in the form of user stories, because everyone loves those in their JIRA tickets don’t they?

Authenticate to AWS as a Human User

As a human user, the temptation is there to generate AWS keys via the IAM user interface. That’s wrong. You should be using AWS IAM Identity Center aka the artist formerly known as AWS SSO.

AWS SSO allows you to either use AWS as an identity provider, or hook in your own identity provider like Okta, Auth0 or even Google. Any service that provides services as an Identity Provider (or IdP) can be hooked into AWS IAM Identity Center.

Once you’ve hooked in your IdP or enabled AWS’s IdP, you then use the AWS CLI to authenticate via aws sso login.

You’ll need to configure your AWS configuration file a little bit, mainly to tell it what AWS account and role you want when you login:

[profile personal-management]
sso_start_url = https://lbrlabs.awsapps.com/start
sso_region = us-west-2
sso_account_id = <account-id>
sso_role_name = AWSAdministratorAccess
region = us-west-2
output = json

Once you run aws sso login for that profile, the AWS CLI will walk you through the authentication flow, verify who you are and then issue temporary credentials for you!

These temporary credentials get stored inside ~/.aws/sso/cache and will expire within a duration you specify at the AWS IAM Identity Center level. Generally you will have to reauthenticate once a day via aws sso login, but that’s good because it means if anyone steals your temporary credentials, they will automatically expire after a certain time period.

Once you’re authenticated via SSO, you can also generate new temporary credentials using this handly little tool I wrote once upon a time.

Authenticate to AWS as an EC2 Instance

Hopefully this one is straightforward, but if you’re running workloads inside anything that uses an EC2 Instance, whether that be an unmanaged EC2 instance like EC2 itself, or via a managed service like Fargate, you should be assigning an IAM role and possible an Instance Profile.

Assigning a role to your AWS workload will automatically mean credentials get refreshed for you on a time period. You no longer have hard coded AWS credentials stored in plaintext that can be extracted and used elsewhere. Easy!

Authenticate to AWS as an application that only managed content in an S3 bucket

This one’s a little unique. Let’s say you have an application that uses object storage, and you want to keep those files private from the internet, but you want people to be able to manage files. A good example? Something that allows you to upload a profile photo might fit the bill.

You probably don’t want to generate AWS credentials for each user session, and you definitely don’t want to allow anyone to mess with your S3 bucket (although, lots of people do that. They usually make the news when they do..).

So AWS obviously came up with a solution to this! Presigned URLs!

Presigned URLs generate a unique link to an object within an S3 bucket that expires. There are no temporary credentials, but the URL itself has credentials embedded in it, like this:

https://thisisntarealbucket.s3.eu-west-2.amazonaws.com/image.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<AWS_ACCESS_KEY>%2F20180210%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=202000905T171315Z&X-Amz-Expires=1800&X-Amz-Signature=12b74b0794aa147ad7d3d03b3f20c61f1f91cc9ad8873e3314255dc479a25351&X-Amz-SignedHeaders=host

This has a limited number of use cases, but is incredibly important for the use cases where it comes up. If you think you might need to let users upload or manage objects in a bucket, go with this option where you can.

Authenticate to AWS as a CI/CD Pipeline

As with human users mentioned earlier, it can be incredibly tempting to just generate an IAM user and an Access Key and Secret Key and stick them in the secrets mechanism your CI/CD tool has.

There is another way!

OIDC Providers give you the capability to authenticate to AWS via the OpenIdentity Connect Protocol.

I wrote about this extensively if you’re using GitHub Actions and AWS (as well as other cloud providers), but the reality is that lots of CI/CD tools like GitLab and CircleCI.

If your CI/CD tool doesn’t support OIDC, you should change CI/CD tools. There’s shitloads of them, choose one that does things properly, will you?

Authenticate to AWS as compute I manage that isn’t running inside AWS

The final authentication strategy is one I never thought I’d see materalize. In this scenario, perhaps you’re running compute in another cloud provider or in an on-premises datacenter. If you have the ability to run an operating system daemon you can use the imaginatively named IAM Role Anywhere.

IAM roles anywhere expects you to provision a valid TLS (X.509) certificate and create a trust (via a trust anchor) in AWS.

Once you’ve trusted AWS with your infrastructure, you can then use a credential process to issue some temporary credentials. Once upon a time, none of this was possible and you’d have to manually mess around rotating credentials, but now you get to mess around with Certificate Authorities instead. Is this better? Well, for the purposes of this blog post, lets say it is.

Wrap Up

I think I’ve covered a most of the needs here and given you at the very least, something to Google when you’re in a certain situation and need some AWS credentials. If you think I missed an option, let me know on Twitter.

If you need help implementing any of these strategies, why not Get in Touch!