Latest Posts (20 found)
Aran Wilkinson 4 weeks ago

Software factories

When I first came across the term software factory in discussions about coding agents, I assumed it meant writing a task and handing it to an agent. The agent would gather the context, write the code, run the tests, open a pull request, and perhaps even deploy it. It sounded like a new, agentic way of building software. Then I saw Dex Horthy’s diagram , and it changed how I understood the term. I realised I had been a part of software factories for years. The task tracker, repository, pull request, CI system, deployment process, monitoring, incident response, and users were already connected. A factory was not a new thing that used agents. It was the system I was already using to turn a problem into a change in a running product. An agent changes one part of that system. It may take on more of the implementation, testing, investigation, or operational work, but it does not create the queues, hand-offs, or feedback loops around that work. Those were already there. I just had not been calling them a software factory. I have also been thinking about this in practical terms. What would it take to build a small software factory for my own projects, with agents helping me prototype ideas, build them, maintain them, and investigate problems once they are running? Looking at the whole system seems more useful than starting with a particular agent or tool. Reading Donella Meadows’s Thinking in Systems helped me make sense of this. One lesson I took from it is that a system’s behaviour comes from how its parts interact, not from the parts themselves, and that pushing harder on one part rarely fixes the system. CI is just one machine in the factory. The factory is the whole system: people and components connected by flows of work and information, operating towards some purpose. Here is the diagram that changed my view of it. It shows the software factory around implementation: users, product managers, engineers, pull requests, tests, deployments, monitoring, incidents, and feature requests. Dex Horthy, “Why Software Factories Fail” . What I find useful about the diagram is that it does not stop at deployment. It shows how information from users and production feeds back into what gets built next. Before seeing Dex’s diagram, I would probably have described software delivery as a series of separate pipelines. A product decision became a task, the task became code, the code passed its checks, and the change was deployed. That work would lead to the next task or another change, but I thought of those as separate pieces of work. The diagram gave me a different way to look at something I already understood. It made the connection explicit. What happens after deployment feeds back into the next decision. I had always been part of that process. I just had not thought of it as a loop, or seen all of its parts as one software factory. The first loop is the delivery loop: a decision becomes a change, the change is built, reviewed, tested, and deployed. The second starts with the product rather than the code. Users, monitoring, incidents, and feature requests produce information that feeds back into the next decision. I sketched the two connected loops together as a single flow below. Production is not the end of the process. It is one of the places where we find out what the change actually did. These loops meet when somebody decides what to build next. The delivery loop asks, “Can we change the software safely?” The product loop asks, “Is this the change we should make?” Those are separate questions. A team can get very good at shipping the wrong thing. It can also understand the product problem clearly and still be unable to deliver a safe change without a long queue of manual work. Seeing the work as two loops changed what I thought the factory was producing. The obvious output is a deployable artefact: a change to the software that can be released to users. But the factory also produces information. That information comes from different parts of the factory. Tests tell us about behaviours we thought to check. Static analysis finds certain classes of mistake. A review can expose a confusing design or a missing operational detail. Monitoring tells us what happened after deployment. Users tell us whether the change was useful in the first place. None of these signals answers the whole question. A green build does not mean that users are happy. A successful deployment does not mean that the system is healthy. A passing unit test does not mean that the change belongs in the product. That was the part I had missed when I thought of a software factory as an agent processing a task. I was focused on the change it produced, rather than the information the wider system produced about that change. The factory is useful when those signals make it back to the people deciding what to do next. That was my original mistake when I thought software factories were mainly about agents. I was thinking about how an agent could produce a change, but not about how the wider system would tell it whether the change was useful, safe, or worth keeping. The question is not simply how much code an agent can write. It is what happens to the rest of the factory when an agent takes on part of the work. Dex’s agentic diagram helped me see that the change may be local. An agent can replace or expand the implementation step, but the rest of the factory still has to review, test, deploy, and observe the change. Dex Horthy, “Why Software Factories Fail” . That seems obvious when written down, but it is easy to forget when an agent produces a working pull request quickly. The pull request is only where the change enters the rest of the factory. It is not the end of the work. A diagram published by The Pragmatic Engineer illustrates a more elaborate version, based on interviews about OpenAI’s internal system: The OpenAI example pushes further. It suggests that agents can remain involved after the merge, gathering production context, watching for regressions, and helping with incidents. I find that interesting, but I do not see it as a reference architecture to copy. It raises a more practical question for me: which parts of that loop would actually help a small project, and which would only add another layer of machinery? This is reporting about one company’s internal setup, not a reference architecture for every team. Here is a generic version of that flow: In the generic flow, an agent gathers context from the code, documentation, and internal tools, writes a change, and keeps fixing issues until the build and the checks pass. Specialist review agents inspect the result, and a risk classification decides whether it follows an automated path or needs human review. In the system described by The Pragmatic Engineer, the loop continues after the merge. An agent helps roll the change out to production, watches the relevant signals, and can create monitoring dashboards for the change. A performance system looks for regressions and proposes fixes. An incident-response bot gathers context and suggests mitigations when something goes wrong. Dex’s response is to turn the lights back on , not to abandon automation. If agents make implementation much faster, review becomes the next bottleneck. Removing human review can increase throughput for a while, but it also removes one of the few places where somebody might notice that a change is making the system harder to understand or modify. In the second part of his series, he argues for moving human judgement earlier in the process: agree the product requirements, system architecture, and shape of the program before asking an agent to implement it, then work in small vertical slices that can be checked as they are built. That is a much more ambitious factory than “ask an agent to write some code”. It gives agents access to the context around the code, lets them act on feedback from the build and review systems, and keeps them involved after deployment. If an agent can reach the wider factory only through a human at a terminal, the human becomes the integration layer. They have to notice a failed CI job, collect the logs or metrics, decide what is relevant, and bring that context back to the agent. This makes each loop slower and means the agent sees only the part of the problem the human thought to provide. An agent connected to the factory can query those systems itself: inspecting CI results, reading production logs, comparing metrics, and gathering the context it needs before proposing what to do next. That connection does not require unrestricted access or permission to act autonomously. Access can be read-only and limited to the systems relevant to the task, with human approval still required for consequential actions. An agentic software factory does not need to be fully automated. A person might start an agent manually, mark a feature request as ready for an agent to pick up, or route a failed build, production regression, or incident directly to one. How an agent gets kicked off, manually or automatically, is a design choice, not part of the definition. What matters is that the agent takes part in the factory’s flows of work and information: it can gather the relevant context, act within its permissions, and respond to the results of its work rather than merely suggest the next line of code. For businesses already experimenting with coding agents, I think the next step is to optimise the software factory around them. It is not simply a matter of giving every developer an agent. Businesses need to look at their existing software factory and decide how it should work when agents become participants: which queues they can take work from, what context and tools they can access, how their output is checked, and where people need to approve an action. Most software delivery systems have been built around human teams, so adding agents means reconsidering those interfaces and hand-offs rather than dropping a coding tool into an otherwise unchanged process. Agency does not mean removing people from the system either. A human still needs to be in the loop at the points where judgement or accountability matters. Depending on the risk, that might mean deciding when to start the agent, reviewing its proposed change, approving an action it wants to take, or approving its deployment. The boundary will differ between teams and between changes; the important part is that it is deliberate rather than whatever the tooling happens to allow. Deliberate boundaries also make the human responsibilities easier to see. People still need to define useful outcomes, make the relevant context available, decide what counts as low risk, ensure the tests describe acceptable behaviour, choose the signals that matter, and decide where an agent must stop and ask for help. Automation changes the boundary of the work; it does not remove judgement from the system. A change marked as low risk can still be wrong, so the classification and the automated path need their own checks. The most important detail in the diagram is the feedback path from production back to the coding agent. An agent that can write code but cannot observe its consequences is just a faster way to create a queue of changes. An agent that can see failures, understand their context, and propose a fix starts to look more like a participant in the factory. The agent does not need to remain active or control every stage. The factory could route the results of a failed build, deployment, or production alert to an agent with the relevant context. What matters is whether that path exists. Without it, the wider feedback loop is available to the human but closed to the agent. Even when those feedback paths exist, there is another limit: an agent can only respond to signals the factory can produce. This is part of Dex Horthy’s wider argument. Tests and production metrics can quickly reveal certain failures, but maintainability has a much slower feedback loop. A design that makes the next six changes harder may not reveal itself until months later. This is not unique to agents, but faster implementation allows more of those design decisions to accumulate before their consequences become visible. A factory can have plenty of automation and still be slow. If every change waits several days for a review, producing the initial change faster will not fix the queue. If deployments are frequent but nobody looks at production behaviour, the factory is producing changes without learning from them. If feature requests arrive faster than the team can understand and prioritise them, adding more implementation capacity will not fix that decision queue. The slowest part of the system is often not the compiler or the test suite. It is a hand-off: This is why the arrows in the diagram are as important as the boxes. A box describes an activity. An arrow describes how work and information move between activities. There is a risk in taking the factory analogy too literally. Software is not a pile of identical widgets. The work includes discovery, judgement, and learning. The thing being built often changes shape while we are building it. Optimising only for throughput can make a team worse at its actual job. We might measure the number of pull requests merged, deployments made, or tickets closed because those numbers are easy to count. That does not tell us whether the product is becoming more useful or whether the system is becoming easier to change. The useful part of the analogy is not “make engineers work like machines”. It is “look at the complete system”. Where does work enter? Where does it wait? Which checks happen automatically? Which decisions need a person? How quickly does information from production get back to the team? The factory should make good changes easier to make and bad changes easier to notice. It should not turn every decision into a race for a higher throughput number. I have been thinking about what a small software factory might look like for my own projects. I want to experiment with agents not only as a way to implement tasks, but as part of a system that helps me with the same four jobs I mentioned at the start: prototyping ideas, building them, keeping them maintained, and investigating failures. Headcode is an obvious place to start. In Splitting Headcode into a polyrepo workspace , I described separating it into smaller repositories with clearer boundaries, repository-local checks, and a separate place for agent work plans. I made those changes partly to give agents bounded contexts. That workspace is not yet a software factory, but it gives me a practical system in which to explore how work moves between agents, repositories, checks, deployments, and production feedback. I do not mean a lights-off factory that can be left to ship changes on its own. The interesting part is deciding which loops I can automate safely, where I still need to make decisions, and what context and signals an agent would need to do useful work. To start with, I will keep most of the agent use manual: running an agent when I choose to, rather than wiring it to a trigger. Starting that way keeps costs predictable while I work out which loops are worth automating, and it keeps me close to what the agents are actually doing before I let them run on their own. That raises more useful questions than simply choosing an agent: Adding another tool is not the same as improving the factory. A new tool only helps if it removes a real delay, improves a useful signal, or makes a risky decision safer. I have not built that system yet. In follow-up posts, I want to explore what an agentic software factory looks like in practice, starting with personal projects where I can experiment freely and document what I learn. I am also interested in seeing how these ideas could apply to the systems I work with professionally, where I have permission to experiment. Thinking of it as a factory gives me a better place to start: not “which agent should I use?”, but “which loop am I trying to improve?” Did the change solve the problem? Did it create a new failure mode? Did users understand it? Did it make the system harder to operate? What should we change next? an engineer waits for somebody who understands the product decision; a pull request waits for a reviewer who is also busy building something else; a deployment waits for somebody to perform a manual check; an incident waits for information that is spread across several systems; a user report waits for somebody to turn it into a reproducible problem. Where does work enter the factory: a Linear ticket, a failed build, a production alert, or a note to myself? How do I describe the outcome rather than only the implementation task? What code, documentation, and operational context does the agent need? Which checks provide meaningful feedback rather than merely producing a green build? Which changes are safe to automate, and where should the agent stop? How does information from maintenance, failures, and actual use make it back into the next change?

devops DevOps ci-cd Ci Cd
0 views
Aran Wilkinson 1 months ago

Using age to keep Headcode's deployment secrets in Git

In my practical guide to , I covered the commands for encrypting and decrypting files. This post is about what happens when those commands become part of a real deployment. Headcode has a small collection of secrets that the application needs to run. They include API keys, a database password, a Cloudflare Tunnel token, and two Google Cloud service-account keys. Rather than keep those files out of Git and add a separate secrets manager, I encrypt them with and commit the ciphertext alongside the rest of the project. That sounds like a small distinction, but it changes the deployment workflow. Git contains the files, their history, and their changes. It just cannot show the contents without one of the private keys. The encrypted files live in . There are four of them: There is also a file. This contains the public keys that are allowed to decrypt the secrets. It is safe to commit because a public key does not give someone the ability to decrypt anything. The private keys are kept by the operators who deploy Headcode. Mine stays on my machine and does not go into Git. That private key is the important part. Anyone who gets a copy of it may be able to decrypt the files encrypted for its corresponding public key, so it needs to be protected like any other credential. This is the basic model behind : encrypt to one or more recipients , which are public keys, and decrypt with the matching identity , which is the private key. Each operator generates their own key pair with , then adds their public key to . There are two common ways to handle deployment secrets. The first is to keep them somewhere outside the repository and retrieve them during deployment. That might mean a hosted secrets manager, a password manager, or a secret store provided by the hosting platform. Those systems can be the right choice, particularly once a team has more people, environments, and access policies to manage. They also introduce another system to operate, back up, secure, and keep available when deploying. For Headcode, I wanted to avoid that operational overhead. This is a small project with one or two operators, and I wanted the secret-handling setup to be straightforward and low-maintenance. I did not want to spend time maintaining a secrets service or password manager when the actual requirement was to encrypt a handful of files and deploy them to one machine. The second approach is what I use for Headcode: encrypt the files before committing them. The encrypted files can then be reviewed, versioned, and deployed like any other part of the project. The deployment process only needs , SSH, and the private identity already held by the operator. The benefit is not that Git has somehow become a secrets manager. It has not. The benefit is that the deployment configuration has one source of history. If a secret changes, that change appears alongside the code or infrastructure change that required it. A new operator can see which encrypted files exist and how they fit into the deployment without needing access to a separate system first. There is a trade-off. The repository now contains encrypted copies of the secrets, and old Git history may contain previous versions. Anyone with an authorised private key can decrypt those historical versions too. If a secret is compromised, rotating the current file is not enough on its own; the old value needs to be treated as compromised as well. This approach suits a project with one or two operators. It is not a universal replacement for a secrets manager. During deployment, the identity file on the operator's machine is used to decrypt each secret with : The important part is what happens to that output. The plaintext is piped directly into an SSH connection to the target VM. It is not written to a temporary file on the operator's machine. The overall shape of the process is: The deployment process handles the remote destination, ownership, permissions, and SELinux labels. The key detail is that the decrypted bytes go from to SSH and then to the destination on the server. There is no intermediate plaintext file on the laptop. On the VM, the resulting files are written to their final locations. For example, the environment file ends up at: The file is then locked down with permissions such as , owned by , and its SELinux labels are restored. The service can read it through its group membership, but it is not an ordinary world-readable configuration file. This protects one part of the workflow: plaintext does not need to sit on the operator's disk. It does not mean the secret never exists in plaintext. The server needs the plaintext to run the application, so the file exists there after deployment. The environment file is not the application's final configuration. The Headcode service uses a YAML configuration file with placeholders for values that come from the environment. For example, the configuration can contain values like these: The path from the encrypted file to the typed Go configuration is: The Go application does not read each secret with a collection of direct calls. Instead, the systemd unit contains: When systemd starts the service, it reads the pairs and adds them to the process environment. The package then expands placeholders such as before the YAML is parsed into the application's typed configuration. The form is useful for values that can sensibly have a local default: Secret-bearing values do not have defaults. If is missing, the configuration should fail rather than quietly start with an empty or fake credential. This gives the application one configuration mechanism without putting secret values directly in . The YAML describes which values it needs; the deployment environment supplies them. The two Google Cloud service-account keys take a slightly different route. They are structured JSON documents, so I deploy them as whole files: The YAML configuration contains the path to each file rather than an environment-variable placeholder. This avoids squeezing a multi-line JSON document into an environment variable and makes the boundary clearer: simple scalar secrets use , while credential documents remain files. The files still go through the same encrypted-at-rest deployment process. The difference is only what happens after they reach the server. Each operator has their own key pair. To add someone, I add their public key to and re-encrypt all four secret files for the expanded recipient list. That means the new operator can decrypt the files without sharing an existing private key. It also means private keys do not need to move between people, which is the important part. Rotating a secret is a separate operation. I replace the value in the local plaintext source, encrypt the file again for the current recipient list, commit the new ciphertext, and deploy it. The running service only sees the new value after it has been restarted and systemd has loaded the updated environment file. If the private identity for an operator is lost, that operator can no longer decrypt the files. Other authorised operators can still deploy, and a replacement key can be added to the recipients file. This is one reason not to make a single private key the only route into the system. I did consider SOPS . SOPS can use as its encryption backend and adds features that are useful for secret files, including encrypting individual values within a structured document and making some multi-key rotation workflows easier. For Headcode, plain is enough for now. The secrets are already separated into a small number of files, and the team is small. Encrypting the whole file keeps the process easy to inspect: the input is a normal file, the output is an ciphertext, and the deployment script decrypts it immediately before sending it to the server. That may change as the project grows. If more people need access, or if several services start sharing structured configuration, SOPS may become the better fit. I do not need to add that layer before the problem exists. This arrangement protects the repository from containing readable secret values and avoids leaving deployment plaintext on the operator's machine. It also gives me a straightforward way to version and review changes to the encrypted files. It does not protect a running server from someone who can read . It does not prevent a privileged user from inspecting the process environment. It does not securely erase the original plaintext files used to create the ciphertext. And it does not tell me whether a public key belongs to the person I think it belongs to; that still needs to be handled when keys are exchanged. There is no magic involved. protects the files between those points in the workflow. The server still needs access to the decrypted values, and the operator still needs to protect their identity file. For Headcode, that is a reasonable boundary. The project has a small number of operators, a single deployment target, and a handful of files that need protecting. Encrypted files in Git give us a simple audit trail without introducing a separate service before we need one. , containing ordinary pairs; , containing the Cloudflare Tunnel token; two encrypted Google Cloud service-account JSON files.

json JSON programming Programming Yaml ai AI
0 views
Aran Wilkinson 3 months ago

Owning the Harness

Over the past year I've started noticing something in the conversations I have with others when talking about AI usage at work. Ask someone how their company is handling AI tooling and you'll almost always get one of two answers. Either they've gone all in. Every engineer gets a Copilot or Cursor or Claude licence, token budgets don't exist, and leadership has basically said "spend what it takes, we'll figure out the ROI later." Or they're in the other camp: tight budgets, a handful of engineers with access, everything scoped to a pilot project with measurable outcomes before anyone else gets a look in. There isn't much middle ground. Companies pick a lane and commit to it, and that choice says more about their culture than any AI strategy document ever could. But the longer I watch this play out, the more I think both camps are arguing about the wrong number. One side doesn't track cost at all and the other tracks it obsessively. Yet the thing that actually decides the bill, and whether any of this is sustainable, is one most of them never touch: the harness their models run through, and who controls it. That's what I want to get into here. The all-in companies aren't subtle about it. Every engineer gets a Copilot, Cursor, Claude or ChatGPT Enterprise seat as part of onboarding. There's no token budget to track, no approval chain to navigate. The philosophy is straightforward: AI is the biggest productivity shift since the internet, and the companies that embed it deepest and fastest will pull ahead. Worrying about per-seat costs right now is like worrying about electricity bills in 1890. In practice this looks like engineers using AI for everything. PR reviews get run through a model before a human sees them. Architecture discussions start with an AI-generated proposal that the team then critiques. Onboarding documentation gets drafted by pasting the codebase into a chat window and asking for a summary. Some teams are running AI agents that pick up tickets, open pull requests, and write their own tests, with varying degrees of success. The cultural shift is the part that interests me most. When everyone has unlimited access, the conversation changes from "should we use AI for this?" to "why wouldn't we?" That's a genuine acceleration. Junior engineers who might have spent a week figuring out a Kafka consumer are shipping in a day because they've got a model walking them through it step by step. Senior engineers are spending less time on boilerplate and more time on the hard problems they actually enjoy. The downside, of course, is that unlimited access doesn't come with unlimited judgement. I've written before about what happens when you run AI-generated SQL without understanding it , and that problem scales with the size of the organisation. When everyone's moving fast and the AI is confidently wrong about something subtle, the blast radius gets bigger. The other camp is just as deliberate, but their starting assumption is different. They see AI as a tool that needs to prove its value before it gets rolled out broadly. So they scope a pilot: maybe five engineers on a single project, maybe a specific workflow like test generation or documentation. Someone owns the budget spreadsheet. Someone else is tracking which prompts produced useful output and which ones burned tokens on nothing. The philosophy here is that cost optimisation is a first-class concern, not something you figure out after the invoices land. These companies want to know what they're getting for their money before they commit to a hundred seats. They're not anti-AI (most of them are genuinely interested), but they're treating it like any other tooling investment rather than a cultural transformation. What this looks like in practice is more restrained. A small team experiments, reports back, and leadership decides whether to expand. Token budgets are real and sometimes tight enough that engineers think twice before asking the model to rephrase a function they wrote six months ago. Project selection is careful: you pick something where the AI's contribution can actually be measured, so you have data to justify the next round of spending. I understand the logic. If you're running a team where every pound matters, you don't hand out AI subscriptions like stickers at a conference. But there's a cost to caution too, and I think it shows up in ways that don't appear on the budget spreadsheet. Here's the part both camps tend to underestimate: a lot of the cost was never theirs to control in the first place. The two biggest levers (which model you run and which harness you run it through) sit largely with the providers, not with you. The model is the obvious one. Running everything through the frontier model versus a cheaper, smaller one can be an order of magnitude difference on the invoice, for output that's often good enough either way. But the harness matters just as much and gets talked about far less. The same task, on the same model, can burn wildly different amounts of tokens depending on the tool wrapping it. A harness that re-reads the entire codebase on every turn, or pads each request with a bloated system prompt, will quietly cost you several times what a leaner setup would for an identical result. You can watch this happen in the tools you already use. Something like Claude Code doesn't send a fixed system prompt. It assembles one on the fly for every request, pulling in your project instructions, the definition of every tool it can call, the skills you've enabled, the MCP servers you've connected, and a pile of environment context on top. Add more skills, connect more servers, write more project config, and that prompt grows, and it gets sent again on every turn of the conversation. Under per-token pricing that's a standing cost most people never see, because the harness assembles it for you and never shows you the system prompt it built. Caching softens the repeated static part, but it doesn't make it free, and the parts that change from request to request aren't cached at all. That has an awkward implication for the cautious camp. You can count seats and cap budgets all you like, but if the real spend is being driven by model choice and harness efficiency, you're optimising the wrong variable. And it's just as awkward for the all-in camp: you're building on pricing and tooling decisions the provider can change under you at any time. A model gets more expensive, a harness gets chattier in an update, and your costs move without you having touched a thing. This is getting harder to ignore as the billing model shifts. The industry is quietly moving from flat-fee subscriptions to consumption-based pricing, and the early numbers are startling. In a June 2026 report , Gartner found that nearly a quarter of technology leaders are already spending between $200 and $500 per developer each month on tokens, with around 6% over $2,000. It also projects that by 2028 AI coding costs will overtake the average developer's salary. That headline deserves a caveat: Gartner's "average" is a global one, pegged to roughly $2,000 a month, not a senior Western salary. But the direction of travel is the point. When a harness can quietly burn 50,000 tokens on a single test-suite run, your spend is set by tooling decisions the provider can change under you, not by how many seats you bought. I've watched this play out up close. When the bill lands on the company card rather than your own, spending thousands of dollars a month on tokens stops feeling like spending at all. I've seen engineers burn through more in a month than their whole tooling budget used to be for a year and not think twice about it, because it simply isn't their money on the line. That's fine while the models hold their price, but they don't. Each new frontier model tends to arrive more capable and more expensive than the last, often close to double, and usage only ever climbs. A way of working that already looks careless at today's prices doesn't become sustainable when the underlying cost doubles. It becomes less. Neither approach is cost-free. The all-in camp gets speed and cultural momentum, but they're burning through budget on something whose ROI is genuinely hard to measure, and they're building a dependency on tools that might change pricing or disappear tomorrow. The cautious camp has control and cost visibility, but they risk falling behind competitors who are iterating faster, and they're potentially demoralising engineers who see peers elsewhere shipping with better tools. The risk I think about most with the cautious approach isn't the money they're saving, it's the talent cost. Good engineers know what tooling is available elsewhere. If they're stuck on a team that's still "evaluating" AI while their friends at other companies are shipping with it daily, that's a retention problem with a price tag that doesn't show up on the AI budget line. I don't think this has to be binary. The answer that makes the most sense to me is structured experimentation: give engineers access, but with guardrails that create visibility without creating friction. Let people use the tools, but measure what's actually working and redirect effort towards the patterns that produce results. That's closer to how I work personally. I use AI heavily, but never as a black box. I write a PRD first, break the work into small, reviewable tasks, and treat the AI's output as a draft that I'm responsible for, not a solution I'm rubber-stamping. The model generates, I review. That separation, with generation and judgement living in different hands, is the part that keeps me from repeating the mistakes I made when I lost that database. It's also why I've spent time building my own tooling and harnesses rather than living entirely inside off-the-shelf ones. Earlier I said the harness is one of the levers the provider controls. Building your own is how you take some of it back. When you own the loop, you decide how much context gets sent on each turn, which tools and MCP servers the model can actually reach, and where a task should stop and hand back to you. That control shows up directly on the invoice: the same work, on the same model, costs a fraction of what it does through a harness that reloads the world on every request. It's more effort up front, but it turns token spend from something that happens to you into something you decide. Owning the loop also means you stop paying frontier prices for work that doesn't need them. Not every step of a task wants the same model. The expensive, heavy-thinking models earn their keep on high-level work such as planning an approach, breaking a problem into reviewable steps, and weighing up an architecture. But once the plan exists, most of the implementation is narrow, well-specified work that a cheaper, faster model handles perfectly well. Route the thinking to the expensive model and the grunt work to the cheap one, and the bill drops again with no real hit to the output. That kind of routing is hard to pull off inside an off-the-shelf tool that runs everything through whichever single model it defaulted to. You can see the payoff in the wild. Mitchell Hashimoto recently described running exactly this kind of split, using one model as a planner and architect, a different one as the coder, and then the first model again as a judge to check the work. The numbers are the striking part. At API pricing he put the planning and judging steps in the region of a few dollars, against the $50 or more that a single full round trip through one frontier model would typically cost. It's an early experiment, and he's the first to say the longevity isn't proven, but the shape of the saving is hard to argue with. Same work, broken across the right models at the right price points, for a fraction of the bill. I'm not the only one going down this road. There's a small but growing group of engineers doing the same, a lot of them building on Pi , a deliberately minimal coding-agent harness you're meant to reshape around your own workflow rather than bend yourself to fit. That framing is the whole point. People are shaping harnesses to match the way they actually work, and tuning them to get the best out of the specific models they're driving. That's exactly the control the off-the-shelf tools don't hand you. There's enough here for its own article, which I'll write separately. For now the point is simply that owning the harness isn't hypothetical. People are already doing it, and it changes both what the tools cost and how well they fit. The approach I landed on isn't complicated: write a plan, break it into tasks, review every piece of AI output before it touches anything real. The structure matters more than the specific tool or model you're using. Without it, you're just hoping the AI doesn't lead you off a cliff. I don't see why that pattern couldn't scale to a team or a company. Give people access. Expect them to use it. But also expect them to own what they ship, to understand the code they're committing, and to stay curious enough to catch the model when it sounds certain and isn't. The guardrail isn't a token budget. It's a culture of reviewing output before trusting it. I don't think either extreme gets it right. The all-in approach risks the kind of blind trust that cost me a database. The cautious approach risks paralysis dressed up as prudence, and in an industry that moves as fast as ours, that's its own kind of expensive. The answer is almost certainly somewhere in between. Give people access to AI tools. They're genuinely useful and they're not going away. But teach them to use those tools with judgement. Create a culture where questioning the AI's output isn't seen as a lack of skill but as a basic professional reflex, the same way you'd review a colleague's pull request even if they were the best engineer on the team. What matters isn't how much you spend or how many tokens you burn. It's whether that spend is something you control or something that just happens to you. The teams that come out of this well won't be the ones with the biggest budgets, or the strictest ones. They'll be the ones who own the harness their models run through, route the right model at the right cost to the right work, and keep a human in the loop whose judgement can tell a right answer from one that only sounds right. Spend you can't see or steer is the real risk. Spend you own is just a tool doing its job. That ownership, not the size of the invoice, is what decides whether AI makes your team better or just makes them faster at being wrong.

sql Sql business Business ai AI
0 views
Aran Wilkinson 3 months ago

Headcode started as a project to learn UK rail data

It now has eight live endpoints , a tiered pricing page and an alpha banner warning people the schemas might still move under them. None of that was the plan. The plan was to understand how UK rail data actually fits together, and to work out how to use an LLM properly on something real instead of a toy. The product is just what happened while I was doing that. I'm still not sure it becomes a business. I'm completely sure the way I learned to work on it was worth the time, and that part I now use every day on everything else. One station has more names than you'd believe, and that's the whole problem. King's Cross is KGX to the fares system, KNGX to the timetable and 54311 to the movement feeds, with a handful more codes besides (NLC, ATCO, UIC), each from a different corner of the railway. You don't need to hold those in your head. Nobody ever agreed on one name, so every feed brought its own. It gets worse with size: a big terminus like London Bridge doesn't have one timetable code, it has a cluster of them, roughly one per platform group, so even "which code is the station" isn't a clean question. Then there's the live side. Darwin (the real-time running feed) pushes updates at you as a stream, while the reference data turns up separately, each source on its own schedule and in its own shape. Before you can render a single departure board you've written a reconciliation layer and a code-mapping table, and now you own both of them. That mess is exactly why it was good to learn on. Bounded enough that you can actually finish it, awkward enough that you can't bluff your way through. You either understand how the feeds relate or your departure board quietly shows the wrong train. The workflow I use now didn't arrive fully formed. It evolved, and the project is where each step earned its place. I started where most people start: a prompt and a plan. Ask for a thing, get a plan back, let it build. That's fine for small, self-contained work. It fell apart the moment a task touched code the model hadn't really looked at. It would produce something plausible and confidently wrong, and then I'd spend longer unpicking it than the thing was worth, either fixing it by hand or trying to prompt my way back out, which sometimes just dug the hole deeper. The failure wasn't the model being bad. It was me asking it to act on intent it didn't have. So the requirement moved to the front. Before any code, I'd work up a proper PRD with the model: I'd set the direction and push back, it would draft and fill in. These weren't a paragraph of good intentions. They grew into real documents, with the goals stated plainly and the out-of-scope list stated just as plainly, functional requirements (what it does) sitting next to non-functional ones (how fast, how reliable, what the limits are), a sketch of the technical architecture, the phases it would be built in, how it would be tested and what might go wrong along the way. Then I'd break that down into tasks small enough to review one at a time. The output got noticeably better, because the model was working to a brief instead of filling in the gaps itself. It stayed focused on the thing in front of it, and it had helped write the thing that kept it there. The step that changed the most came later, and it wasn't obvious to me at the start. A PRD is only as good as your understanding of the code it lands in. So before writing the PRD, I'd have the model research the existing codebase and write up how the relevant part actually works, what's already there, which patterns to follow. The thing that made the research useful was a hard rule: document the codebase as it exists today, and nothing else. No suggested improvements, no root cause analysis, no critique, no refactoring proposals, no architecture it wished were there. Only what exists, where it lives, how it works and how the parts connect. Left to its own instincts a model will reach for the fix, because pointing out problems reads as helpful. Forbidding all of that kept the output objective, a technical map of the system as built with the opinion stripped out. Adding that step changed the relationship. The model stopped being the author and became something that I directed, sent to find things out and report back rather than left to decide what to build. The research feeds the PRD, the PRD feeds the tasks, and I'm steering at every handoff. Review runs through all of it. I read the research, the PRD, the tasks, the plans, not just the final diff. That order matters more than it sounds. If the research is wrong, the PRD inherits the error and every task underneath it inherits it again, and by the time you're reviewing code you're three layers downstream of the actual mistake. The review at the top is worth far more than the review at the bottom. That's the honest line between this and vibecoding. It was never about whether I used the AI. It was about whether I ever let go of understanding what it was doing, and I made a point of not letting go. It wasn't all done this way from the start. The early parts of Headcode were built the way most things get built with an LLM, by prompting, getting something working and moving on. The discipline came later. As the research-into-PRD-into-tasks process settled, designing before implementing became the default, and the project quietly split into a scrappy first phase and a deliberate second one. You can see the join. The later work, the bulk of the API surface, the endpoint groups, the schema as it stands now, was designed before any of it was written, the spec settled while the code was still hypothetical. The early prompted bits I've mostly gone back and rebuilt to the same standard, because once you've felt the difference the scrappy version nags at you. What fell out of that patience is an API where the schema is the contract. It's OpenAPI-first, with a downloadable spec you can point a client generator or contract tests at. Identifiers resolve cleanly: hand it any code system and it gives you back all of them, so the reconciliation table that would normally be a thing you maintain becomes a field you read. Vibecoding the same idea would have got me a convincing departure board demo and a wall the moment the identifier resolution got hard. Rail data punishes building before thinking, which is precisely what made it a good teacher. Headcode is in alpha. There's no self-serve signup. Access is by request, you email me with what you're building and I send a token. That's deliberate while the data and the schemas are still settling. I genuinely don't know whether there's external demand for it. It might just stay a personal project, something I experiment with and build other things on top of, now that I've got the rail data in a clean format to start from. That alone was worth doing. I want to build visualisations on top of it, possibly a small app, and a clean API I control is reason enough to have built the thing. If people turn up actually wanting the data, it might grow into a small SaaS. What I'm not going to do is manufacture a roadmap I don't believe in, or pretend there's urgency around a project I started in order to learn. Whatever Headcode turns into, the workflow has already paid for itself. I went in wanting to learn how to use an LLM well on a real codebase, and the research into PRD into tasks pipeline, with review at every layer, is now simply how I work with one. The product is a maybe. The method, I kept. If you want to see what came out of it, Headcode lives at headcode.dev , and the API docs — endpoints, schemas and the OpenAPI spec — are open to browse at docs.headcode.dev .

ai AI data Data programming Programming
0 views
Aran Wilkinson 4 months ago

Introducing Headcode: A Unified API for UK Rail Data

Headcode is a unified, developer-friendly JSON API that takes the fragmented, legacy feeds of the UK rail network and turns them into clean, enriched real-time data.

data Data api API json JSON
0 views
Aran Wilkinson 5 months ago

Introducing jjw: a workspace manager for jj

I released jjw, a Go CLI for managing jj workspaces with bookmarks and lifecycle hooks. This post explains why I built it, how it works, and how to get started.

go Go open-source Open Source
0 views
Aran Wilkinson 5 months ago

How I lost a database and learned to actually use AI

I ran AI-generated SQL without reading it properly and lost a database. The experience changed how I work with AI tools, replacing freeform chat sessions with a structured process built around PRDs, small tasks, and frequent commits.

sql Sql ai AI database Database
0 views
Aran Wilkinson 1 years ago

building a mqtt client in go

Build a robust MQTT client in Go with real Home Assistant examples. Covers concurrent processing, wildcard matching, and device integrations including Aqara sensors and EV charging data.

go Go tutorial Tutorial programming Programming
0 views
Aran Wilkinson 1 years ago

go 1.24's new tool directive

Discover how to use the new tool directive in Go 1.24 to specify the tools you need to build your project.

programming Programming go Go
0 views
Aran Wilkinson 1 years ago

processing uk rail data in real-time

A Go service that consumes UK rail data from Kafka and persists it to PostgreSQL using table partitioning. Built with franz-go, it handles millions of daily messages while maintaining data consistency.

database Database
0 views
Aran Wilkinson 1 years ago

engineering arrogance

Innovation and rapid execution drive success in technology companies, but arrogance within engineering teams can silently erode this foundation. When technical expertise becomes a barrier rather than a bridge, it threatens the very innovation it claims to protect.

career Career business Business
0 views
Aran Wilkinson 1 years ago

building with google sheets api in go

Master Google Sheets API integration in Go with practical examples covering A1 notation, batch operations, and production-ready patterns for robust data operations.

api API
0 views
Aran Wilkinson 1 years ago

building stria to streamline payment data integrations

Learn how Stria simplifies payment data integration by syncing Stripe directly to Google Sheets - no code required. See how this new service solves common integration headaches

api API data Data
0 views
Aran Wilkinson 1 years ago

postgresql table paritioning

Discover practical PostgreSQL table partitioning strategies with Go code examples. Learn how to automate partition creation and optimise query performance.

database Database go Go
0 views