Latest Posts (20 found)
Aran Wilkinson 1 weeks ago

Using age to keep Headcode's deployment secrets in Git

In my practical guide to , I covered the commands for encrypting and decrypting files. This post is about what happens when those commands become part of a real deployment. Headcode has a small collection of secrets that the application needs to run. They include API keys, a database password, a Cloudflare Tunnel token, and two Google Cloud service-account keys. Rather than keep those files out of Git and add a separate secrets manager, I encrypt them with and commit the ciphertext alongside the rest of the project. That sounds like a small distinction, but it changes the deployment workflow. Git contains the files, their history, and their changes. It just cannot show the contents without one of the private keys. The encrypted files live in . There are four of them: There is also a file. This contains the public keys that are allowed to decrypt the secrets. It is safe to commit because a public key does not give someone the ability to decrypt anything. The private keys are kept by the operators who deploy Headcode. Mine stays on my machine and does not go into Git. That private key is the important part. Anyone who gets a copy of it may be able to decrypt the files encrypted for its corresponding public key, so it needs to be protected like any other credential. This is the basic model behind : encrypt to one or more recipients , which are public keys, and decrypt with the matching identity , which is the private key. Each operator generates their own key pair with , then adds their public key to . There are two common ways to handle deployment secrets. The first is to keep them somewhere outside the repository and retrieve them during deployment. That might mean a hosted secrets manager, a password manager, or a secret store provided by the hosting platform. Those systems can be the right choice, particularly once a team has more people, environments, and access policies to manage. They also introduce another system to operate, back up, secure, and keep available when deploying. For Headcode, I wanted to avoid that operational overhead. This is a small project with one or two operators, and I wanted the secret-handling setup to be straightforward and low-maintenance. I did not want to spend time maintaining a secrets service or password manager when the actual requirement was to encrypt a handful of files and deploy them to one machine. The second approach is what I use for Headcode: encrypt the files before committing them. The encrypted files can then be reviewed, versioned, and deployed like any other part of the project. The deployment process only needs , SSH, and the private identity already held by the operator. The benefit is not that Git has somehow become a secrets manager. It has not. The benefit is that the deployment configuration has one source of history. If a secret changes, that change appears alongside the code or infrastructure change that required it. A new operator can see which encrypted files exist and how they fit into the deployment without needing access to a separate system first. There is a trade-off. The repository now contains encrypted copies of the secrets, and old Git history may contain previous versions. Anyone with an authorised private key can decrypt those historical versions too. If a secret is compromised, rotating the current file is not enough on its own; the old value needs to be treated as compromised as well. This approach suits a project with one or two operators. It is not a universal replacement for a secrets manager. During deployment, the identity file on the operator's machine is used to decrypt each secret with : The important part is what happens to that output. The plaintext is piped directly into an SSH connection to the target VM. It is not written to a temporary file on the operator's machine. The overall shape of the process is: The deployment process handles the remote destination, ownership, permissions, and SELinux labels. The key detail is that the decrypted bytes go from to SSH and then to the destination on the server. There is no intermediate plaintext file on the laptop. On the VM, the resulting files are written to their final locations. For example, the environment file ends up at: The file is then locked down with permissions such as , owned by , and its SELinux labels are restored. The service can read it through its group membership, but it is not an ordinary world-readable configuration file. This protects one part of the workflow: plaintext does not need to sit on the operator's disk. It does not mean the secret never exists in plaintext. The server needs the plaintext to run the application, so the file exists there after deployment. The environment file is not the application's final configuration. The Headcode service uses a YAML configuration file with placeholders for values that come from the environment. For example, the configuration can contain values like these: The path from the encrypted file to the typed Go configuration is: The Go application does not read each secret with a collection of direct calls. Instead, the systemd unit contains: When systemd starts the service, it reads the pairs and adds them to the process environment. The package then expands placeholders such as before the YAML is parsed into the application's typed configuration. The form is useful for values that can sensibly have a local default: Secret-bearing values do not have defaults. If is missing, the configuration should fail rather than quietly start with an empty or fake credential. This gives the application one configuration mechanism without putting secret values directly in . The YAML describes which values it needs; the deployment environment supplies them. The two Google Cloud service-account keys take a slightly different route. They are structured JSON documents, so I deploy them as whole files: The YAML configuration contains the path to each file rather than an environment-variable placeholder. This avoids squeezing a multi-line JSON document into an environment variable and makes the boundary clearer: simple scalar secrets use , while credential documents remain files. The files still go through the same encrypted-at-rest deployment process. The difference is only what happens after they reach the server. Each operator has their own key pair. To add someone, I add their public key to and re-encrypt all four secret files for the expanded recipient list. That means the new operator can decrypt the files without sharing an existing private key. It also means private keys do not need to move between people, which is the important part. Rotating a secret is a separate operation. I replace the value in the local plaintext source, encrypt the file again for the current recipient list, commit the new ciphertext, and deploy it. The running service only sees the new value after it has been restarted and systemd has loaded the updated environment file. If the private identity for an operator is lost, that operator can no longer decrypt the files. Other authorised operators can still deploy, and a replacement key can be added to the recipients file. This is one reason not to make a single private key the only route into the system. I did consider SOPS . SOPS can use as its encryption backend and adds features that are useful for secret files, including encrypting individual values within a structured document and making some multi-key rotation workflows easier. For Headcode, plain is enough for now. The secrets are already separated into a small number of files, and the team is small. Encrypting the whole file keeps the process easy to inspect: the input is a normal file, the output is an ciphertext, and the deployment script decrypts it immediately before sending it to the server. That may change as the project grows. If more people need access, or if several services start sharing structured configuration, SOPS may become the better fit. I do not need to add that layer before the problem exists. This arrangement protects the repository from containing readable secret values and avoids leaving deployment plaintext on the operator's machine. It also gives me a straightforward way to version and review changes to the encrypted files. It does not protect a running server from someone who can read . It does not prevent a privileged user from inspecting the process environment. It does not securely erase the original plaintext files used to create the ciphertext. And it does not tell me whether a public key belongs to the person I think it belongs to; that still needs to be handled when keys are exchanged. There is no magic involved. protects the files between those points in the workflow. The server still needs access to the decrypted values, and the operator still needs to protect their identity file. For Headcode, that is a reasonable boundary. The project has a small number of operators, a single deployment target, and a handful of files that need protecting. Encrypted files in Git give us a simple audit trail without introducing a separate service before we need one. , containing ordinary pairs; , containing the Cloudflare Tunnel token; two encrypted Google Cloud service-account JSON files.

0 views
Aran Wilkinson 1 months ago

Owning the Harness

Over the past year I've started noticing something in the conversations I have with others when talking about AI usage at work. Ask someone how their company is handling AI tooling and you'll almost always get one of two answers. Either they've gone all in. Every engineer gets a Copilot or Cursor or Claude licence, token budgets don't exist, and leadership has basically said "spend what it takes, we'll figure out the ROI later." Or they're in the other camp: tight budgets, a handful of engineers with access, everything scoped to a pilot project with measurable outcomes before anyone else gets a look in. There isn't much middle ground. Companies pick a lane and commit to it, and that choice says more about their culture than any AI strategy document ever could. But the longer I watch this play out, the more I think both camps are arguing about the wrong number. One side doesn't track cost at all and the other tracks it obsessively. Yet the thing that actually decides the bill, and whether any of this is sustainable, is one most of them never touch: the harness their models run through, and who controls it. That's what I want to get into here. The all-in companies aren't subtle about it. Every engineer gets a Copilot, Cursor, Claude or ChatGPT Enterprise seat as part of onboarding. There's no token budget to track, no approval chain to navigate. The philosophy is straightforward: AI is the biggest productivity shift since the internet, and the companies that embed it deepest and fastest will pull ahead. Worrying about per-seat costs right now is like worrying about electricity bills in 1890. In practice this looks like engineers using AI for everything. PR reviews get run through a model before a human sees them. Architecture discussions start with an AI-generated proposal that the team then critiques. Onboarding documentation gets drafted by pasting the codebase into a chat window and asking for a summary. Some teams are running AI agents that pick up tickets, open pull requests, and write their own tests, with varying degrees of success. The cultural shift is the part that interests me most. When everyone has unlimited access, the conversation changes from "should we use AI for this?" to "why wouldn't we?" That's a genuine acceleration. Junior engineers who might have spent a week figuring out a Kafka consumer are shipping in a day because they've got a model walking them through it step by step. Senior engineers are spending less time on boilerplate and more time on the hard problems they actually enjoy. The downside, of course, is that unlimited access doesn't come with unlimited judgement. I've written before about what happens when you run AI-generated SQL without understanding it , and that problem scales with the size of the organisation. When everyone's moving fast and the AI is confidently wrong about something subtle, the blast radius gets bigger. The other camp is just as deliberate, but their starting assumption is different. They see AI as a tool that needs to prove its value before it gets rolled out broadly. So they scope a pilot: maybe five engineers on a single project, maybe a specific workflow like test generation or documentation. Someone owns the budget spreadsheet. Someone else is tracking which prompts produced useful output and which ones burned tokens on nothing. The philosophy here is that cost optimisation is a first-class concern, not something you figure out after the invoices land. These companies want to know what they're getting for their money before they commit to a hundred seats. They're not anti-AI (most of them are genuinely interested), but they're treating it like any other tooling investment rather than a cultural transformation. What this looks like in practice is more restrained. A small team experiments, reports back, and leadership decides whether to expand. Token budgets are real and sometimes tight enough that engineers think twice before asking the model to rephrase a function they wrote six months ago. Project selection is careful: you pick something where the AI's contribution can actually be measured, so you have data to justify the next round of spending. I understand the logic. If you're running a team where every pound matters, you don't hand out AI subscriptions like stickers at a conference. But there's a cost to caution too, and I think it shows up in ways that don't appear on the budget spreadsheet. Here's the part both camps tend to underestimate: a lot of the cost was never theirs to control in the first place. The two biggest levers (which model you run and which harness you run it through) sit largely with the providers, not with you. The model is the obvious one. Running everything through the frontier model versus a cheaper, smaller one can be an order of magnitude difference on the invoice, for output that's often good enough either way. But the harness matters just as much and gets talked about far less. The same task, on the same model, can burn wildly different amounts of tokens depending on the tool wrapping it. A harness that re-reads the entire codebase on every turn, or pads each request with a bloated system prompt, will quietly cost you several times what a leaner setup would for an identical result. You can watch this happen in the tools you already use. Something like Claude Code doesn't send a fixed system prompt. It assembles one on the fly for every request, pulling in your project instructions, the definition of every tool it can call, the skills you've enabled, the MCP servers you've connected, and a pile of environment context on top. Add more skills, connect more servers, write more project config, and that prompt grows, and it gets sent again on every turn of the conversation. Under per-token pricing that's a standing cost most people never see, because the harness assembles it for you and never shows you the system prompt it built. Caching softens the repeated static part, but it doesn't make it free, and the parts that change from request to request aren't cached at all. That has an awkward implication for the cautious camp. You can count seats and cap budgets all you like, but if the real spend is being driven by model choice and harness efficiency, you're optimising the wrong variable. And it's just as awkward for the all-in camp: you're building on pricing and tooling decisions the provider can change under you at any time. A model gets more expensive, a harness gets chattier in an update, and your costs move without you having touched a thing. This is getting harder to ignore as the billing model shifts. The industry is quietly moving from flat-fee subscriptions to consumption-based pricing, and the early numbers are startling. In a June 2026 report , Gartner found that nearly a quarter of technology leaders are already spending between $200 and $500 per developer each month on tokens, with around 6% over $2,000. It also projects that by 2028 AI coding costs will overtake the average developer's salary. That headline deserves a caveat: Gartner's "average" is a global one, pegged to roughly $2,000 a month, not a senior Western salary. But the direction of travel is the point. When a harness can quietly burn 50,000 tokens on a single test-suite run, your spend is set by tooling decisions the provider can change under you, not by how many seats you bought. I've watched this play out up close. When the bill lands on the company card rather than your own, spending thousands of dollars a month on tokens stops feeling like spending at all. I've seen engineers burn through more in a month than their whole tooling budget used to be for a year and not think twice about it, because it simply isn't their money on the line. That's fine while the models hold their price, but they don't. Each new frontier model tends to arrive more capable and more expensive than the last, often close to double, and usage only ever climbs. A way of working that already looks careless at today's prices doesn't become sustainable when the underlying cost doubles. It becomes less. Neither approach is cost-free. The all-in camp gets speed and cultural momentum, but they're burning through budget on something whose ROI is genuinely hard to measure, and they're building a dependency on tools that might change pricing or disappear tomorrow. The cautious camp has control and cost visibility, but they risk falling behind competitors who are iterating faster, and they're potentially demoralising engineers who see peers elsewhere shipping with better tools. The risk I think about most with the cautious approach isn't the money they're saving, it's the talent cost. Good engineers know what tooling is available elsewhere. If they're stuck on a team that's still "evaluating" AI while their friends at other companies are shipping with it daily, that's a retention problem with a price tag that doesn't show up on the AI budget line. I don't think this has to be binary. The answer that makes the most sense to me is structured experimentation: give engineers access, but with guardrails that create visibility without creating friction. Let people use the tools, but measure what's actually working and redirect effort towards the patterns that produce results. That's closer to how I work personally. I use AI heavily, but never as a black box. I write a PRD first, break the work into small, reviewable tasks, and treat the AI's output as a draft that I'm responsible for, not a solution I'm rubber-stamping. The model generates, I review. That separation, with generation and judgement living in different hands, is the part that keeps me from repeating the mistakes I made when I lost that database. It's also why I've spent time building my own tooling and harnesses rather than living entirely inside off-the-shelf ones. Earlier I said the harness is one of the levers the provider controls. Building your own is how you take some of it back. When you own the loop, you decide how much context gets sent on each turn, which tools and MCP servers the model can actually reach, and where a task should stop and hand back to you. That control shows up directly on the invoice: the same work, on the same model, costs a fraction of what it does through a harness that reloads the world on every request. It's more effort up front, but it turns token spend from something that happens to you into something you decide. Owning the loop also means you stop paying frontier prices for work that doesn't need them. Not every step of a task wants the same model. The expensive, heavy-thinking models earn their keep on high-level work such as planning an approach, breaking a problem into reviewable steps, and weighing up an architecture. But once the plan exists, most of the implementation is narrow, well-specified work that a cheaper, faster model handles perfectly well. Route the thinking to the expensive model and the grunt work to the cheap one, and the bill drops again with no real hit to the output. That kind of routing is hard to pull off inside an off-the-shelf tool that runs everything through whichever single model it defaulted to. You can see the payoff in the wild. Mitchell Hashimoto recently described running exactly this kind of split, using one model as a planner and architect, a different one as the coder, and then the first model again as a judge to check the work. The numbers are the striking part. At API pricing he put the planning and judging steps in the region of a few dollars, against the $50 or more that a single full round trip through one frontier model would typically cost. It's an early experiment, and he's the first to say the longevity isn't proven, but the shape of the saving is hard to argue with. Same work, broken across the right models at the right price points, for a fraction of the bill. I'm not the only one going down this road. There's a small but growing group of engineers doing the same, a lot of them building on Pi , a deliberately minimal coding-agent harness you're meant to reshape around your own workflow rather than bend yourself to fit. That framing is the whole point. People are shaping harnesses to match the way they actually work, and tuning them to get the best out of the specific models they're driving. That's exactly the control the off-the-shelf tools don't hand you. There's enough here for its own article, which I'll write separately. For now the point is simply that owning the harness isn't hypothetical. People are already doing it, and it changes both what the tools cost and how well they fit. The approach I landed on isn't complicated: write a plan, break it into tasks, review every piece of AI output before it touches anything real. The structure matters more than the specific tool or model you're using. Without it, you're just hoping the AI doesn't lead you off a cliff. I don't see why that pattern couldn't scale to a team or a company. Give people access. Expect them to use it. But also expect them to own what they ship, to understand the code they're committing, and to stay curious enough to catch the model when it sounds certain and isn't. The guardrail isn't a token budget. It's a culture of reviewing output before trusting it. I don't think either extreme gets it right. The all-in approach risks the kind of blind trust that cost me a database. The cautious approach risks paralysis dressed up as prudence, and in an industry that moves as fast as ours, that's its own kind of expensive. The answer is almost certainly somewhere in between. Give people access to AI tools. They're genuinely useful and they're not going away. But teach them to use those tools with judgement. Create a culture where questioning the AI's output isn't seen as a lack of skill but as a basic professional reflex, the same way you'd review a colleague's pull request even if they were the best engineer on the team. What matters isn't how much you spend or how many tokens you burn. It's whether that spend is something you control or something that just happens to you. The teams that come out of this well won't be the ones with the biggest budgets, or the strictest ones. They'll be the ones who own the harness their models run through, route the right model at the right cost to the right work, and keep a human in the loop whose judgement can tell a right answer from one that only sounds right. Spend you can't see or steer is the real risk. Spend you own is just a tool doing its job. That ownership, not the size of the invoice, is what decides whether AI makes your team better or just makes them faster at being wrong.

0 views
Aran Wilkinson 2 months ago

Headcode started as a project to learn UK rail data

It now has eight live endpoints , a tiered pricing page and an alpha banner warning people the schemas might still move under them. None of that was the plan. The plan was to understand how UK rail data actually fits together, and to work out how to use an LLM properly on something real instead of a toy. The product is just what happened while I was doing that. I'm still not sure it becomes a business. I'm completely sure the way I learned to work on it was worth the time, and that part I now use every day on everything else. One station has more names than you'd believe, and that's the whole problem. King's Cross is KGX to the fares system, KNGX to the timetable and 54311 to the movement feeds, with a handful more codes besides (NLC, ATCO, UIC), each from a different corner of the railway. You don't need to hold those in your head. Nobody ever agreed on one name, so every feed brought its own. It gets worse with size: a big terminus like London Bridge doesn't have one timetable code, it has a cluster of them, roughly one per platform group, so even "which code is the station" isn't a clean question. Then there's the live side. Darwin (the real-time running feed) pushes updates at you as a stream, while the reference data turns up separately, each source on its own schedule and in its own shape. Before you can render a single departure board you've written a reconciliation layer and a code-mapping table, and now you own both of them. That mess is exactly why it was good to learn on. Bounded enough that you can actually finish it, awkward enough that you can't bluff your way through. You either understand how the feeds relate or your departure board quietly shows the wrong train. The workflow I use now didn't arrive fully formed. It evolved, and the project is where each step earned its place. I started where most people start: a prompt and a plan. Ask for a thing, get a plan back, let it build. That's fine for small, self-contained work. It fell apart the moment a task touched code the model hadn't really looked at. It would produce something plausible and confidently wrong, and then I'd spend longer unpicking it than the thing was worth, either fixing it by hand or trying to prompt my way back out, which sometimes just dug the hole deeper. The failure wasn't the model being bad. It was me asking it to act on intent it didn't have. So the requirement moved to the front. Before any code, I'd work up a proper PRD with the model: I'd set the direction and push back, it would draft and fill in. These weren't a paragraph of good intentions. They grew into real documents, with the goals stated plainly and the out-of-scope list stated just as plainly, functional requirements (what it does) sitting next to non-functional ones (how fast, how reliable, what the limits are), a sketch of the technical architecture, the phases it would be built in, how it would be tested and what might go wrong along the way. Then I'd break that down into tasks small enough to review one at a time. The output got noticeably better, because the model was working to a brief instead of filling in the gaps itself. It stayed focused on the thing in front of it, and it had helped write the thing that kept it there. The step that changed the most came later, and it wasn't obvious to me at the start. A PRD is only as good as your understanding of the code it lands in. So before writing the PRD, I'd have the model research the existing codebase and write up how the relevant part actually works, what's already there, which patterns to follow. The thing that made the research useful was a hard rule: document the codebase as it exists today, and nothing else. No suggested improvements, no root cause analysis, no critique, no refactoring proposals, no architecture it wished were there. Only what exists, where it lives, how it works and how the parts connect. Left to its own instincts a model will reach for the fix, because pointing out problems reads as helpful. Forbidding all of that kept the output objective, a technical map of the system as built with the opinion stripped out. Adding that step changed the relationship. The model stopped being the author and became something that I directed, sent to find things out and report back rather than left to decide what to build. The research feeds the PRD, the PRD feeds the tasks, and I'm steering at every handoff. Review runs through all of it. I read the research, the PRD, the tasks, the plans, not just the final diff. That order matters more than it sounds. If the research is wrong, the PRD inherits the error and every task underneath it inherits it again, and by the time you're reviewing code you're three layers downstream of the actual mistake. The review at the top is worth far more than the review at the bottom. That's the honest line between this and vibecoding. It was never about whether I used the AI. It was about whether I ever let go of understanding what it was doing, and I made a point of not letting go. It wasn't all done this way from the start. The early parts of Headcode were built the way most things get built with an LLM, by prompting, getting something working and moving on. The discipline came later. As the research-into-PRD-into-tasks process settled, designing before implementing became the default, and the project quietly split into a scrappy first phase and a deliberate second one. You can see the join. The later work, the bulk of the API surface, the endpoint groups, the schema as it stands now, was designed before any of it was written, the spec settled while the code was still hypothetical. The early prompted bits I've mostly gone back and rebuilt to the same standard, because once you've felt the difference the scrappy version nags at you. What fell out of that patience is an API where the schema is the contract. It's OpenAPI-first, with a downloadable spec you can point a client generator or contract tests at. Identifiers resolve cleanly: hand it any code system and it gives you back all of them, so the reconciliation table that would normally be a thing you maintain becomes a field you read. Vibecoding the same idea would have got me a convincing departure board demo and a wall the moment the identifier resolution got hard. Rail data punishes building before thinking, which is precisely what made it a good teacher. Headcode is in alpha. There's no self-serve signup. Access is by request, you email me with what you're building and I send a token. That's deliberate while the data and the schemas are still settling. I genuinely don't know whether there's external demand for it. It might just stay a personal project, something I experiment with and build other things on top of, now that I've got the rail data in a clean format to start from. That alone was worth doing. I want to build visualisations on top of it, possibly a small app, and a clean API I control is reason enough to have built the thing. If people turn up actually wanting the data, it might grow into a small SaaS. What I'm not going to do is manufacture a roadmap I don't believe in, or pretend there's urgency around a project I started in order to learn. Whatever Headcode turns into, the workflow has already paid for itself. I went in wanting to learn how to use an LLM well on a real codebase, and the research into PRD into tasks pipeline, with review at every layer, is now simply how I work with one. The product is a maybe. The method, I kept. If you want to see what came out of it, Headcode lives at headcode.dev , and the API docs — endpoints, schemas and the OpenAPI spec — are open to browse at docs.headcode.dev .

0 views
Aran Wilkinson 3 months ago

Introducing Headcode: A Unified API for UK Rail Data

Headcode is a unified, developer-friendly JSON API that takes the fragmented, legacy feeds of the UK rail network and turns them into clean, enriched real-time data.

0 views
Aran Wilkinson 3 months ago

Introducing jjw: a workspace manager for jj

I released jjw, a Go CLI for managing jj workspaces with bookmarks and lifecycle hooks. This post explains why I built it, how it works, and how to get started.

0 views
Aran Wilkinson 3 months ago

How I lost a database and learned to actually use AI

I ran AI-generated SQL without reading it properly and lost a database. The experience changed how I work with AI tools, replacing freeform chat sessions with a structured process built around PRDs, small tasks, and frequent commits.

0 views
Aran Wilkinson 11 months ago

building a mqtt client in go

Build a robust MQTT client in Go with real Home Assistant examples. Covers concurrent processing, wildcard matching, and device integrations including Aqara sensors and EV charging data.

0 views
Aran Wilkinson 1 years ago

go 1.24's new tool directive

Discover how to use the new tool directive in Go 1.24 to specify the tools you need to build your project.

0 views
Aran Wilkinson 1 years ago

processing uk rail data in real-time

A Go service that consumes UK rail data from Kafka and persists it to PostgreSQL using table partitioning. Built with franz-go, it handles millions of daily messages while maintaining data consistency.

0 views
Aran Wilkinson 1 years ago

engineering arrogance

Innovation and rapid execution drive success in technology companies, but arrogance within engineering teams can silently erode this foundation. When technical expertise becomes a barrier rather than a bridge, it threatens the very innovation it claims to protect.

0 views
Aran Wilkinson 1 years ago

building with google sheets api in go

Master Google Sheets API integration in Go with practical examples covering A1 notation, batch operations, and production-ready patterns for robust data operations.

0 views
Aran Wilkinson 1 years ago

building stria to streamline payment data integrations

Learn how Stria simplifies payment data integration by syncing Stripe directly to Google Sheets - no code required. See how this new service solves common integration headaches

0 views
Aran Wilkinson 1 years ago

postgresql table paritioning

Discover practical PostgreSQL table partitioning strategies with Go code examples. Learn how to automate partition creation and optimise query performance.

0 views