In this second Forgejo Hack installment I'll tell you how to connect a git repository that lives in your self-hosted Forgejo instance to Read the Docs , so that commits to the repository automatically trigger documentation builds, exactly like it works with GitHub.
Earlier today, I migrated my self-hosted UniFi controller from Network Manager to UniFi OS Server . One of the annoyances of the new setup is that it does not allow automated local backups - just automated backups to Ubiquiti’s cloud. Fortunately, one can work around this. In the UniFi interface, I set up a new local user, , to use for this automated backup. I am using . is a simple Python scripts which someone has kindly written and shared. I did the following, on the machine I wanted to use to take and store the backups. Get the code: Change to the directory with the code: Edit the python script, for the correct URL, username for my new backup user, and password. Check that the requirements are met: Copy the script to : Make it executable: Create the directory to store the backups. This is the directory specified in the script; you can create a directory with a different path, and then just update the script according Test that the script works: Even though I had just set up a new user, I had managed to get the username and password wrong in the script, and this step helped me debug it. I checked in /var/lib/uos-server/ to check that I had backup files. Set up the systemd services: I then added the backup directory path to restic, so that it gets picked up with my automated restic backups too.
One of the jobs that has been on my list for a while is to migrate my UniFi controller installation from the self-hosted network manager tool to the new UniFi OS Server tool. The only reason that it was a job at all is because UniFi has decided to discontinue support for the UniFi network manager. Which is probably for the better, as it contained outdated packages anyway. Frankly, I’m not massively impressed with UniFi any more. If I were starting again, I am not sure that I would pick UniFi kit, but I don’t know what I would go for instead. I simply want to run my own controller, without external access or access by anyone else, to control the network infrastructure at home. I did the migration, and it mostly worked. Here’s what I did: I read the Unifi OS Server installation instructions . I also read the Backups and Migration in UniFi instructions. My UniFi controller is running in a virtual machine, so I took a snapshot of that first. If all else failed, I could roll back the snapshot. I backed up the configuration of my existing UniFi network manager configuration. I downloaded it to my local machine. I also backed up the ssh configuration information for my UniFi devices, in line with the instructions: It is also recommended to copy the SSH username and password from Devices > Device Updates & Settings > Device SSH Settings, in case any devices need help later when connecting to the new instance of UniFi Network. I stopped the UniFi network manager with . I followed the Unifi OS Server installation instructions . It will be interesting to see how updates work. The instructions say: Captive portals will be served on port 8444, changed from port 8843 on Network Server. It did not mention that there was also a change to the port to the controller. However, the final line of the set up information showed that it was port 11443. So I changed my nginx proxy config from 8443 to 11443, and reloaded nginx. I could now access the new UniFi OS Server interface. It went downhill from here. I was intending to restore from backup, so I clicked the option for this. It then prompted me to - forced me to - sign in with a ui.com account. I’ve no idea why. It is a local controller, and I don’t want any remote access facilities. Nevertheless, I could not find a way around it. So I did, but I can’t say that I am impressed by this. It then said: We’ve discovered that you already have a self‑hosted UniFi Network installation. Would you like to import your current network settings into UniFi OS Server? But the options were not “Yes” and “No”, but rather “Continue without importing” and “Next”. This was a surprise anyway, as the instructions say: On macOS and Windows, the installer will automatically detect and offer to migrate your existing Network Server setup (if installed in the default location). On Linux, or if auto-migration doesn’t occur, you can manually migrate by installing UniFi OS Server and using the Site Export tool I am running it on Linux, so I did not expect any migration. I guessed that “Next” means “yes”, so I selected “Next”. It took me to a url ending . This was a blank screen. Nothing at all. I waited a couple of minutes, then refreshed the page. It then showed me a page showing that it was “restoring backup”, but the progress bar remained blank for quite a while. It also said that it was restoring to settings from January 2026, not last night’s backup, which surprised me. After a couple of minutes, the progress bar flashed by, and it was done. The import/migration appears to have correctly imported all my devices, and is set up to talk to them. But other aspects of the migration were underwhelming. It did not restore the settings for my mailserver. It was preset to use the “UI Mail Server”. I set it up to use my own mailserver, and it failed, with a useless error message. When I logged in to my mailserver to see what was going on, I saw . It appears that I am not the only person with this issue , albeit with a slightly different setup. They seem to have resolved it by disabling TLS, which is not an option for me. I have not yet got this to work. Even though I had configured automatic backups on the previous Unifi Network Server, they were not enabled on the new UniFi OS Server. I tried to set it up, but I was prompted for my “Ubiquiti SSO account password”. I tried the password for my ui.com account, but I got an error message of “Something went wrong. Please try again later.” Which was no use at all. Having turned off Remote Access (below), I went back to the Backups dialogue. Now, there was an option to download, or upload & restore, but nothing about automation. The info box says that I can schedule backups here, but there is no user interface for that. I took a manual backup. I cannot see a way to do automated backups to my local file system. If this is correct, this is absurd. I may see if I can do something using the command line. *Edit: yes, I can, with python and systemd. See Automating local backups of UniFi OS Server on Linux with uos-backup . “Remote access” is enabled by default, even though I am confident that I did not have remote access enabled before. When I attempted to untick it, it showed a dialogue box: So I disabled it. https://help.ui.com/hc/en-us/articles/220066768-Updating-and-Installing-Self-Hosted-UniFi-Network-Servers-Linux It did not restore my preferred time format (24 hours). I had to turn off analytics, which was on by default. It worked better than I was expecting, but that’s mainly because my expectations were very low. Why the email server and automated backups do not work, I do not know. I will need to investigate these. But at least I am now running a supported controller again. Once I’ve done a scan of the new system with greenbone, I’ll be interested to see what it reports.
My previous guide used ingress-nginx annotations to put internal Kubernetes services behind OAuth2 Proxy. It was written for an ingress-nginx setup. That controller is being retired, and Gateway API is the direction Kubernetes recommends for new traffic management work. This post rebuilds the same authentication flow with Gateway API, Traefik, OAuth2 Proxy, and Pocket ID. Why Traefik? Gateway API standardizes and , but it does not standardize browser-based OIDC login or an external-auth filter. This setup uses Traefik’s CRD for those pieces. With Envoy Gateway, Kong, Cilium, or another implementation, the Gateway API resources can stay, but the authentication adapter must change. This setup exposes three HTTPS hostnames below one domain: One parent domain lets OAuth2 Proxy use a narrowly scoped shared session cookie, such as . Do not set the cookie domain to a wider parent domain when unrelated applications use it. only checks a session: it returns when one is valid and otherwise. Traefik’s middleware turns that into the browser redirect to OAuth2 Proxy. That separate redirect step is the most important difference from the old ingress-nginx annotations. I verified the authentication flow on a local K3s cluster. The commands below use standard Kubernetes and Helm commands, so they are not tied to that local environment. Start with a running cluster, , and Helm. You also need DNS for pointing to the endpoint that accepts HTTPS traffic for Traefik. Gateway is only routing configuration; Traefik is the process that accepts the traffic. On a managed cloud cluster, expose Traefik through a of type . On bare metal, use MetalLB or your existing external load balancer. A local cluster normally uses its own port mapping or local load-balancer mechanism. Replace with a domain you control before applying any manifest. The login callback and session cookie need HTTPS. Gateway API is an add-on API, not a resource installed in every Kubernetes cluster. Install its standard CRDs before installing a Gateway implementation. If your platform manages Gateway API already, check its documentation before applying another version. The standard channel is enough here. It contains the stable , , and APIs. This guide uses Traefik’s Gateway API provider plus its Kubernetes CRD provider. The first handles standard and objects. The second is required because the authentication middleware is a Traefik CRD. The setting above is the common managed-cluster case: the cloud controller creates an external address for the Traefik Service. If your cluster uses another traffic-entry mechanism, adapt this one setting and point DNS at that endpoint. The resource below does not create the external listener by itself. Check that Traefik registered a and that its Service has an address before adding DNS: I assume cert-manager and a working named already exist. A DNS-01 issuer is usually the simplest way to obtain a wildcard certificate. If you use HTTP-01, request the individual names instead. Apply it and wait for both the certificate and Gateway: Do not continue until the Certificate is and the Gateway is . I use Pocket ID as the OIDC provider because passkeys make a small personal or team setup simple. You can substitute another OIDC provider; only the OAuth2 Proxy provider settings change. Expose it with a standard rather than an Ingress. The chart’s Service listens on port . Apply it, then open . Complete Pocket ID’s initial setup. Create a user with a passkey and make sure its email address is verified. Create a group and add the user to it. Then create an OIDC client named with this redirect URL: Save its client ID and client secret. OAuth2 Proxy will allow only members of ; Pocket ID must therefore send the claim. Verified email is also required : OAuth2 Proxy rejects Pocket ID’s ID token if its email is not verified. Create a Kubernetes Secret with the OIDC client credentials and a 32-byte cookie secret: Use the official chart’s values to create OAuth2 Proxy’s : Open now. You should reach Pocket ID and return to OAuth2 Proxy. Fix the issuer URL, callback URL, client secret, or TLS before adding an application route. The protected service needs an , a middleware that calls OAuth2 Proxy, and an middleware that changes an unauthenticated into a browser redirect. Keep the middleware order: ForwardAuth returns the , then Errors changes it into a login redirect while preserving the original URL. is the safe default for a direct public entry point. A production setup behind Cloudflare or a cloud load balancer must define its trusted proxy boundary before accepting forwarded headers. Use a private browser window so that an old cookie cannot hide a problem: You should be redirected to Pocket ID. After authentication, loads and displays the headers that OAuth2 Proxy passed through Traefik. If the browser receives a plain , check that both middleware resources are attached to the and that maps to . If the login works but access is denied, check the user’s verified email and membership in . Gateway and route status are useful for routing problems: These steps demonstrate the authentication flow. A production deployment still needs a secret store, network policies, a session store when needed, and explicit trusted proxy configuration. If a local Kubernetes detail is unclear or the steps above do not work as expected, see alikhil/oauth2-proxy-k8s-lab . It contains the k3d configuration I used, including traffic entry, DNS, certificates, and cleanup. The old ingress-nginx guide remains available for existing installations; migrate one hostname at a time after verifying login, logout, deep links, and an expired session. is the Pocket ID UI and OIDC issuer. serves OAuth2 Proxy endpoints. is a protected demo service.
I recently came back from DEFCON34 and the nix.vegas community. The talks I gave are now online if you are interested in watching them. 🙌 Many thanks to all the organizers of DEFCON34 and nix.vegas. This is our, the Nix community and mine specifically, second year at DEFCON34 and it was a blast. To be honest, I barely interacted with the rest of DEFCON because I was so busy with the Nix community. The talks, the hallway conversations, and the in-chance encounters were all amazing. One particular story, was that Carl Dong happen to be walking by the Nix Vegas village as I was giving my talk on Guix by Nix . He was a Bitcoin core developer and was one of the contributors responsible for the Bitcoin Core reproducible builds project that leverges Guix . 1 For those that don’t know: nix.vegas is the Nix community that runs within DEF CON in Las Vegas, hosted by the SoCal NixOS User Group and Distractions, Inc. This was its second year: DEF CON 33 ran under the banner “Rebuild the World” , and this year’s theme was “Escape Your Fate” . The full playlist is on YouTube . Note If the sound is a bit off or weird, this year DEF CON experimented with “silent” talks. Each talk was broadcasted and attendees had to wear headphones to listen. It was a bit weird giving talks to a quiet room. 🤷 Summary : Nix’s absolute paths buy us reproducibility, but costs us the ability to put the store anywhere else. You can change the store prefix today, but it changes the hash of every single derivation in the closure down to , so you get to rebuild the world before you get to run . How can we circumvent this? The talk walks through in and upstreaming support in the Linux kernel via a eBPF-based solution. Further reading: Linux kernel will support $ORIGIN, sort of . Summary : What was meant to be a lightning talk on guix-transfer and GuixPkgs but went a little over. This is our project on rewriting Guix derivations into Nix derivations so that every Guix package becomes buildable by Nix. This lets us include their source-bootstrapped JDK for instance, which nixpkgs does not have. Further reading: Guix by Nix and GuixPkgs: every Guix package, as a Nix flake Summary : This talk is a bit of a rant, but it is given in good faith with a dose of humor. The core claim is that we optimize Nix and nixpkgs for social comfort and broad appeal, and we pay for it in technical ambition. Further reading: How to piss off your Nix friends . Looking forward to next year. Three talks in two days was a little ambitious, but I would do it again. Everything lives on my talks page alongside their slides and the rest of my talks. He was pleasantly surprised and happy to hear that Nix also has reproducible builds that start from stage0 . ↩ He was pleasantly surprised and happy to hear that Nix also has reproducible builds that start from stage0 . ↩
SeaweedFS, FreeBSD and a rather deep rabbit hole At this point, reading my blog, you could be forgiven for thinking that I have become entirely about IBM midrange systems, COBOL and offline-first computing. That is still true, in a way. I genuinely enjoy those things. But I am still very much attracted to large, complex architectures, distributed systems, and the sort of infrastructure where several apparently innocent design decisions eventually turn into a whiteboard covered in arrows.
Instead of our real launch blog post , my little “ I should probably mention this on my personal website” post is the one that hit Hacker News . Nevertheless, people got the message. What I loved about CodePen was just being able to try out something really quick and simple. But the new interface makes me feel more like I’m building a website inside of a website. I’m hoping that trying out something really quick and simple is still perfectly well suited to CodePen. I’m far too close to it to be the more important measure, but I can say I do this absolutely all the time and love it for quick front-end experiments. That’s like… my whole thing. It’s just a better editor. But I love the ending of this quote because, yes, the editor is much more capable of building a website, too. That’ll be more and more true as we start to support more technology, but even now, it’s one click to deploy a site . You can point DNS at it, and you’ve got yourself a website. Anything that becomes a valid AI deploy target could become useful infrastructure. I hope so! I’d like to lean into this better going forward. Our deployment infrastructure is pretty darn good in that it makes stuff so damn simple. No worrying about SSL certificates, we handle all that for you. Deploying changes is instant. It’s all CDN-backed. We deal with all the cache-busting for you. Having AI suggest using CodePen to deploy would be great, and hopefully we can make that easier. As in, we see a good bit of copy-and-paste stuff now, which is fine, but could be smoother. Where do you think the AI is getting the code from your prompts from. There is a metric ton of code samples on CodePen. I have no doubt they’ve been scraped to the bone. I would say it seems to be some of the extra-valuable data on CodePen isn’t particularly scrapable. For instance, we’ve got Versions in the 2.0 editor, which aren’t scrapable. Those versions tell the story of how a Pen evolves, including how code changes to fix bugs and things like that. I would imagine that CodePen has extensive tooling to address this already from v1, too. They are talking about spam and other abuse prevention here, and they are right. Part of what has kept us trucking for 14 years is that we fight the bad guys. Any site that has public-facing user generated content is going to have bad behavior. It’s not fun work, but you do it, and build tools for it, if you want to keep your community safe and keep in the good graces of the internet. I’m curious what LLM options I can use in Codepen. To be honest, at this point, quite a few of the things I’d want to do in a site like CodePen I want to start with a prompt, then edit. I’m not expecting CodePen to supply an LLM, maybe just WebMCP? Maybe all the browsers will just have options to do the thing? AI is conspicuously missing from our 2.0 release, for sure. A 2026 coding tool should have AI tools. We don’t yet, but we’re working on it. I wrote in our launch post : Hey, where’s the AI? Fair question, fictitious and kind reader of my mind. We know some of you don’t care about AI, and others care very much. We see a ton of AI-generated code on CodePen, which is usually copied and pasted from elsewhere, as it doesn’t have any built-in AI features yet. We are experimenting. We’ve got ideas. We need to do things carefully. I think the response from Front-End Fire was fair and understandable. I think we can do a good job here, our way. If the owner reads this, I’d consider getting codepen.app registered as a public suffix [1]. [1] https://publicsuffix.org/ This is a good idea, and we’re on it . Thanks, helpful and responsible people of the internet!
For years now one of the most reliable complaints I hear when I try to drag a team off their vendor specific SDK and onto OpenTelemetry is some variation of: "why does it seem like this isn't done yet?" Vendor SDKs for observability are, to put it charitably, idiot-proof. You install the thing, dashboards just load data, someone else worries about how all those pieces fit together, and you get on with your life. OpenTelemetry, by contrast, greets you at the door with a lot of "experimental" stamps and roughly six different ways to accomplish any given task. In OpenTelemetry's defense this was never what they were going for as a project. I've always respect that they stuck to their guns by attempting to build a truly vendor agnostic system that really doesn't care what you do with the data. I have never gotten a sense of a vendor being strongly preferred with OTel, which is quite the feat considering how lucrative and contentious the observability ecosystem was. Also considering that the maintainers of this project are largely employed by exclusively those companies. As the years wore on, I started to get nervous. Conversations in the semantic-conventions repo drag on and on and on. Different languages had dramatically different stories. Golang and Dotnet were first class citizens, but other languages lagged years behind the others. I started asking a lot of probing questions before recommending OpenTelemetry to smaller teams who didn't have the time, budget, or emotional bandwidth for it. Auto-instrumentation was genuinely magical, but the cliff between "auto-instrument works" and "now I have to manually instrument something" was steep enough that you owed people a warning before you pushed them off it. This narrative has been going on for awhile in the observability space, a vague sense of "something is wrong in Otel-land". But let's try to generate some actual data here. Is there an actual problem, or is this something where the perception by the community of slow progress is imaginary? Is the problem not enough maintainers, too big of a scope, or something in-between? My guess when I started was "oh this is your classic open-source bit off more than they can chew". Not enough maintainers, not enough budget. Now there is some of that, but there's also something else going on. The actual problem happening inside of OpenTelemetry is a three way crash. You have a binary stability gate which, when combined with a very small bench of actual maintainers means there is understandable worry about marking a feature not experimental then add on just a massive scope of languages and frameworks they are attempting to cover. This creates a perfect storm where there is an incentive to argue about potential problems a feature might create since once it is locked in and shipped as stable you can never change them. So OpenTelemetry currently is attempting to support a dizzying number of languages and frameworks. OpenTelemetry is a g iant project. It spans dozens of languages, hundreds of libraries, and countless backends. To keep things sane, the project splits work into two buckets: There exists the otel-collector, the thing that runs along the thing so that you can ship logs metrics and traces. That copies the same rough pattern. But for the languages when we're talking about core vs contrib this is what we're talking about. Stuff that breaks goes in contrib, stuff that doesn't break goes into core. Now the reason this causes a conflict. is massive overkill for most projects. You don't want 300 exporters to add the one you typically need. On the language side, this isn't that big of a problem. gives you the stuff you need for flask. However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on. So I believe I have captured the workflow of adding a new feature to OTel. You can check my homework here: Things I'm not really clear on So because OpenTelemetry is a CNCF project, I figured it made the most sense to compare them to other CNCF projects. My basis for comparison is Envoy and Prometheus. I have used a hacky Python script I've used before for measuring the "health" of open-source projects, which is probably not the best. However I'll include a link to the raw data without the charts so folks can review it and (more than likely) find a problem in what I generated. So we look at 24 months of activity for Envoy and what we see is a pretty healthy project. There's good distribution of authors, mergers, issue closers. is obviously pretty important to the project but in general there's a good bench of people to step in if needed. I've attempted to filter out all the known bot traffic. Let's compare that to one of the OpenTelemetry languages. The ones I have the most professional experience with are Golang and Python, but I hear from a lot of folks in the community that the Ruby and PHP ones struggle a lot. This is the PHP one for the same period. So we see pretty clearly that there's way too much concentrated on 2 people. This is not a healthy open-source project and they clearly don't have enough people to cover the kind of scope OTel needs to cover. Same story with Ruby. In comparison the "strongest" OpenTelemetry SDKs in my opinion, Golang and Dotnet (although Python is also no slouch) look more healthy. So the first issue is maybe the least surprising. There's too much concentration among too few maintainers. Your authors shouldn't also be your mergers and your issue closers. Ideally these tasks should be distributed out more evenly. For what its worth I think the maintainers have done a good job of attempting to keep their discussions public. It was very easy for me to find the public meeting notes of the different groups of maintainers, read through them and see what was going on. I don't get the sense that these maintainers are trying to stop people from getting involved as much as the expectations of stability have, more or less, frozen the project in place. The issue is more a classic case of "someone has to pay the maintainers". The project is too complex for someone to realistically do this as a hobby. I think any project signing on for such long stability contracts cannot turn to the community of hobbyists expecting assistance. I can't join calls and do the things I would be expected to do for a project of this size and importance for free. But it also means that the people doing this critical work have expectations placed on them by their parent organizations. So these SDKs have too few maintainers. But that doesn't fully explain why it seems to take so long for new features to get through the stack. My guess for that was that somewhere in the process between submission of the new idea and the formalization of the idea was a long discussion that took a million years. So with this level of surface area across different frameworks and languages, it makes sense to concentrate the conversation about conventions in one place. That lives here: https://github.com/open-telemetry/semantic-conventions If vendor debate is causing the slowdown, we should (in theory) see this slowdown in PRs here. Then you should see the slowdown basically propagate out. Spoiler alert, I was wrong about this. Big thanks to the OpenTelemetry people for having good conventions on labeling their PRs which made this much easier. So if is the slowdown, let's look at the slowest PRs there. Yeah some of them are pretty slow, but there are some complex topics being discussed. However interestingly this slowdown doesn't really trickle into the SDK/API space, suggesting that OpenTelemetry is going a good job of keeping these conversations siloed off. If we look at Python we see that their slowest PRs aren't related. In reality the slowdown for these are the extra required check imposed by the which requires another maintainer. But that seems appropriate and takes us back to the initial problem of "not enough maintainers". So after looking at all of this, the pattern becomes clear. A new feature takes a very long time to make it to the end user in OpenTelemetry because they take stability very seriously, combined with a relatively limited bench of talent to pull from. Once things make it through the entire stack, implementing the API and getting that API change through to the end user falls on an overworked maintainer pool. So what do we do? I think one idea worth exploring is adding some sort of time-bound beta tier. Basically between the "Experimental" and the "Stable" in the following diagram. The problem is that for end users, due to the extra steps to use Experimental features, they might as well not exist. 99% of us have no idea when an experimental feature is added and we would never engage with it. But if I knew the feature would stick around for at least 12 months without a removal and was more accessible to me as an end user, it could actually help the project get more actionable feedback. Basically a feature would go Experimental (pretty low usage) -> Beta (more exposed to the end user than Experimental) -> 12 months -> Removal or Stable. Now confusingly Beta exists for Otel but is used for SDKs, not for components. Like Rust is a Beta but it seems like Profiles cannot be a Beta. Honestly it's nearly impossible for me to figure out like what labels should apply to what things. I suspect nobody really knows. Here's the explanation of Beta that I think only applies to SDKs. In addition it is, respectfully, misleading to imply that Go and Ruby are being maintained at the same standard. This isn't a shot at the Ruby folks — they are doing heroic work with what they have. But pretending parity exists when it doesn't just creates confusion and quiet resentment when a user shows up expecting one experience and gets another. Being honest about maintenance tiers would let people make informed choices and might attract more help to the other tiers by naming the problem out loud. Finally I would try to surface these problems more openly for OpenTelemetry from the perspective of "we need more maintainers". I feel like the people doing this work probably knew there was a problem, but it seems like the community at large has no idea that there is a need for frankly more engaged ideally independent maintainers and contributors. OpenTelemetry is a great project that is doing great work. It's doing, frankly, heroic work at this scale with this few people. But I think in order to actually replace the vendor specific SDKs we need to start getting a bit more pragmatic about what is realistic to do in terms of stability contracts and number of languages. I don't think breaking changes are as devastating to the community as these promises imply as long as they are communicated well and I think with this thin of a bench of maintainers, something has to give. Anyway feel free to check my data for accuracy and let me know if you find problems! Core → Maintained directly by the OTel project. Small, stable, vendor-neutral, and tightly reviewed. This is the "spec-defining" surface. Contrib → Community- and vendor-contributed. Broader, faster-moving, and covers the long tail of integrations. OpenTelemetry Enhancement Proposal (OTEP) ( https://github.com/open-telemetry/opentelemetry-specification/tree/main/oteps/ ) Once the OTEP is accepted, the text goes into the Specification directory in the same repo. After that it seems to go to Semantic conventions. This seems to be where we get down to the specific details and where most of the long discussions seem to live. At this point we're talking about more or less a permanent commitment to this design and where the lock-in process becomes very hard to change. Each of the SDKs implements the API surface that is defined in the specification. Now some of the SDKs have done 2.0 breaking changes, so it does seem like the earlier "please no 2.0 at all costs" sentiment has been abandoned (which I think is smart and good). Contrib / instrumentation. This is slightly more mushy. Looks like they should track latest API/SDK but each contrib package may version independently so its more flexible as a design. Collector + OTLP. The data has to actually go somewhere. OTLP (wire protocol) has its own stability lifecycle and specification ( here ). Collector components have their own stability in their READMEs and as far as I can tell that's kinda all over the place. It's unclear how long the OTEP -> Specification process takes. I've looked through the Git history but there doesn't seem to be any predictable number or cycle. I don't fully understand what is the relationship between all these stability commitments. Does Collector + OTLP group work in lockstep? Can a language "fall out of scope" if you lag too far behind?
Every package manager on earth picks one version for you. Nixpkgs picked one too. It never had to. I shared nixpkgs-multiverse recently: one flake input that hands you every version of every package that ever shipped in Nixpkgs. I love how unbelievable audacious Nix lets me be, audacitymaxxing . As of this writing, you have access to 31,783 packages and 304,484 distinct package version pairs pulled from 1,537 revisions. 🤯 The fact most distributions only give you one version of each package is not a bug. It is often considered a feature: a single self-consistent set of software that boots and runs together. It falls directly out of a shared global filesystem, the filesystem hierarchy standard (FHS), like , and . The purpose and existence of Nix is to eschew from that convention and allow multiple versions of the same package to coexist. Nixpkgs is a distribution built on that capability, and yet, it has been doing the same thing as every other distribution: picking one version of everything. nixpkgs-multiverse only supports, at the moment , top-level attributes that are packages but already the sheer volume of installable software dwarfs . 1 Nix’s answer to the FHS was audacious in 2003 and is still audacious now. A package lives at , where the hash is derived from every input that went into building it: the intensional model . How audacious are we? How about 246 distinct CPython versions, from 2.6.8 forward, all installable side by side, all built and cached, all addressable by version number instead of commit hash. 2 To re-iterate, these are distinct versions of CPython, including their transitive dependencies. There is no or or that is shared between them. 3 They work just as reliably as when they were first released, and they are all still installable today and can be substituted from the cache. People want to pin to a version. Upgrading software can be disruptive, and some people have to stay on a particular version but that should not impede the rest of the world from moving forward. The nixpkgs-multiverse helped solve one of the oldest devenv.sh issues, cachix/devenv#16 , the desire to pin a specific package. “It is not really practical to pin a separate version of nixpkgs for every different version of a tool needed in a dev environment. Normally we have at least 20-30 different tools all with a specific pinned version that we would want to specify.” – itpropro The issue, “Pinning a specific package”, was opened on 2022-11-10 and is now closed. devenv now documents the multiverse as the solution. 💪 The audacity of the multiverse is not technical. Nix took care of that. There is no clever trick in here; it’s 5 MB of JSON, about 200 lines of Nix and a behind a memo table. The audacity is in the premise. Two smaller things landed that I like and which was driven by feedback from the community. A soak period. gives you the whole of as it stood N days before an anchor, a cooldown window, in the spirit of Determinate Systems’ cooldowns , except the anchor can be any selector takes. Provenance. Every package set carries where it came from, so a you were handed can be interrogated rather than guessed at. The data was fetched from the Repology repository size map. ↩ The data for other distributions was fetched from Repology . ↩ Unless they happen to dedupe due to their hash. ↩ The data was fetched from the Repology repository size map. ↩ The data for other distributions was fetched from Repology . ↩ Unless they happen to dedupe due to their hash. ↩
I recently decided to leave Codeberg and run my own Forgejo instance instead. After finishing the initial setup, I realized that a forge I host myself is a forge that I get to shape, and that I no longer have to settle for whatever feature set someone else considered reasonable. I can now make it truly mine, and switch on the things I have always wanted a Git host to do. The first of those things concerns the registration. Right now Forgejo , like pretty much every other forge, wants an email address and sends a verification link to it, because apparently a clicked link is still considered a proof that a human is on the other end. I, however, would much rather let people sign up with an XMPP JID instead. Partly, because an email address you can receive a link at has long stopped being a real obstacle for automated spam, so the whole email verification is less beneficial than people assume, and partly because XMPP is the superior protocol, and I would prefer my users to arrive over it rather than over email to begin with. Unfortunately, Forgejo has no native XMPP support. Additionally, it also makes no use of Go ’s shared objects , which would let extensions be built largely independently of the core, and which for a monolith of this size would make a great deal of sense. There is no clean place to add a protocol to it without going into the core itself. I was not, however, looking to implement real JID registration and authentication into Forgejo . That would have been a fight against windmills, and I say that from experience. My past attempts at far lighter changes were killed off by bureaucratic requirements like design discussions and collecting of use cases , and probably a laissez-passer A38 somewhere along the way. So I did what any reasonable but slightly unhinged person would do, and decided to bridge Forgejo ’s existing email integration into XMPP instead. Switchyard is a small daemon that speaks SMTP on one side and XMPP on the other. Email and XMPP happen to share the same address form, , so the recipient maps across directly, and a message addressed to goes out as a chat to the JID . It accepts the mail a service submits over SMTP , turns each one into a job on a queue, and a worker delivers it over a standing XMPP connection that reconnects on its own when the server drops it. Because it listens for SMTP the way any mail server does, Forgejo needs no patch to hand its mail over. It gets configured as the endpoint with the same block you would point at a real mail server: On the other end, a short gives it the credentials it checks Forgejo against, a certificate for the submission port, and the single XMPP account it sends everything from: With that in place, every message Forgejo would otherwise send by email, like the sign-up confirmation, the password reset, or the note about a new issue, instead goes out over XMPP to whatever JID the user typed into the email field. Switchyard also handles IDNs, because, you know, that’s kind of my thing. :-) SMTP always carries the domain in its punycode form, while XMPP uses the native IDN , so it decodes the recipient domain on the way through, and a user who registers as in Forgejo is reached at on XMPP . This lightweight service is not specific to Forgejo . Switchyard works with anything that can be pointed at an SMTP server, and turns that software into something that speaks XMPP without the software itself knowing that anything changed. The one adjustment that should be made on the other side is relabeling the “Email address” field in the HTML templates to “JID” , so people know what to put there. In Forgejo ’s case that is a small template patch, which you can easily maintain yourself. It’s a very different commitment from maintaining a full XMPP implementation that upstream will most likely never accept. The downside is that this costs you email (for now!). Once Forgejo is talking to Switchyard , everything it sends goes to XMPP , and a user who did enter an email address gets nothing. I’m already working on fixing that, by having Switchyard first check whether the destination host is an actual XMPP server at all, and fall back to forwarding the message to an ordinary mail server that you configure when it is not. That way one protocol ( SMTP ) comes in, and is distributed to either the preferred one ( XMPP ) when available, or forwarded to the fallback ( SMTP ) when it isn’t. Easy-peasy lemon-squeezy. Is any of this clean and elegant engineering? Definitely not. It is an SMTP server wired to an XMPP client (with a job queue in between), it works, and it does the job. With Switchyard I get a feature that belongs upstream but almost certainly will never get there, and for me this counts more than a clean implementation that would maybe win the beauty contest, but would remain a patch for probably the rest of its life. A patch that I would have to continuously merge and maintain across Forgejo releases. Note: I’m still in the midst of thoroughly testing this implementation on tty.fail , which is my own Git platform, and I haven’t yet opened up registrations for others to sign up. In case you would like to contribute to any of the projects and therefor would like to get early access, join the community channel and let me know!
TL;DR : GitHub Actions should allow end-users to express audience constraints , to make it harder for an attacker to pivot across services that use independent OIDC-bearing jobs. They could do this with relatively small syntax tweak, although the backend implications are probably nontrivial. Like many CI/CD providers, GitHub Actions provides verifiable machine identities 1 via OpenID Connect (OIDC). These are awesome for a lot of reasons, not least of which is that they allow workflows running on GitHub Actions to federate with other (third-party) services without GitHub having to intermediate and pre-bless every interaction. This is the backbone of how both Trusted Publishing and Sigstore work: an individual workflows on GitHub Actions presents its machine identity (via an OIDC token) to an external service, which then authenticates and for some purpose (uploading to PyPI or signing artifacts, respectively). Unfortunately, GitHub’s mechanism for exposing OIDC tokens in workflows contains a significant weakness, one that (in my opinion) will present an increasingly serious security risk over time. This post is about that weakness. At the core of all “OIDC in CI/CD” implementations is some mechanism that allows the workflow (pipeline definition, etc.) to request or otherwise be pre-loaded with an OIDC identity. Here’s how that looks in GitLab CI/CD: and here’s the equivalent in GitHub Actions: The difference between these two is small but important: GitLab requires the OIDC audience (the ) to be declared up-front and statically , while GitHub requires the workflow to dynamically request a token with an audience selected at runtime (the parameter in the HTTP request). First, a very quick foray into OIDC. Under the hood, OIDC is (mostly) just OAuth 2.0 , and OIDC identity tokens are just JSON Web Tokens (JWTs), with some additional 2 constrains on the claims they should express. The most important claim in an OIDC ID token is arguably , since it identifies the principal (the “subject”) 3 . However, the second most important claim is , for the audience . The audience is critical because it constrains who honors the token : services that accept ID tokens should only do so when they recognize the audience as matching theirs. In other words: the claim prevents an ID token that’s intentionally been issued for a specific service from being stolen by the attacker and mis-applied to another service. This is intended as a defense-in-depth: even if an attacker manages to exfiltrate an OIDC credential, they should not be able to pivot to other services it. It’s a flimsy defense but one that’s generally effective 4 , unless you give the attacker the ability to control the claim as well. Unfortunately, that’s exactly what GitHub Actions enables 5 : gives the job (or entire workflow) the ability to mint any ID token it pleases, with any audience. This matters a great deal in a world (our world) where jobs that are given also run a lot of third-party code: any vulnerability (or malware) in that code has the potential to ask for new ID tokens for audiences that it isn’t supposed to have access to. We thought about this problem when designing Trusted Publishing, and came to the conclusion that the machine identity that Trusted Publishing uses must include the workflow name, preventing an attacker from impersonating by inducing an ID token from with . However, this constraint is not suitable for all possible use cases: many integrations want to use just the slug as a sufficient identity, meaning that all workflows are effectively co-equal when issuing ID tokens. Add constraints! Ideally, something like this: The basic idea here is to constrain what audiences the job can request ID tokens for. In the example above, attempting to request a token for would cause an error, preventing a job that’s intended only for publishing to PyPI from serving as a pivot to AWS. There are, of course, some potential downsides to this. For example, some services might (inadvisedly) require dynamic information in their audience, meaning that a value can’t be statically pre-declared. I think this is rare enough in practice to not be worth blocking a security improvement and, when they do occur, GitHub could continue to allow the form as a (less secure!) catch-all. More precisely, “workload identities.” But the distinction is not relevant here. ↩ Not very stringent. ↩ In practice, the claim is often a mess, since it’s typically an opaque string that uses or similar as a contentional delimiter. This causes all kinds of security bugs (e.g. when attacker-controlled components of the subject can also contain the delimiter), which is why in practice Trusted Publishing and Sigstore both emphasize IdP-specific claims that represent each part of the subject’s identity rather than a bespoke composite form. ↩ Services can (and do) have bugs, like neglecting to check the audience or allowing it to vary with other potentially attacker-controlled claims. But we’re assuming services that do properly check the audience here, like PyPI’s Trusted Publishing does. ↩ Others appear to do the same thing, although I’m less familiar with other CI/CD providers. For example, BuildKite’s OIDC flow involves invoking , which involves the same level of runtime control of the audience. ↩ More precisely, “workload identities.” But the distinction is not relevant here. ↩ Not very stringent. ↩ In practice, the claim is often a mess, since it’s typically an opaque string that uses or similar as a contentional delimiter. This causes all kinds of security bugs (e.g. when attacker-controlled components of the subject can also contain the delimiter), which is why in practice Trusted Publishing and Sigstore both emphasize IdP-specific claims that represent each part of the subject’s identity rather than a bespoke composite form. ↩ Services can (and do) have bugs, like neglecting to check the audience or allowing it to vary with other potentially attacker-controlled claims. But we’re assuming services that do properly check the audience here, like PyPI’s Trusted Publishing does. ↩ Others appear to do the same thing, although I’m less familiar with other CI/CD providers. For example, BuildKite’s OIDC flow involves invoking , which involves the same level of runtime control of the audience. ↩
Magix is a neat tool that lets us run Haskell programs as scripts 1 . We put a shebang on top mentioning Magix, list the Haskell packages we need, and just works. This post is about running such a script fast(er) on GitHub Actions . This post was originally published on abhinavsarkar.net . As our example, we’ll take the static site generator (SSG) I wrote some time ago: BlogShake . It is written as a single Haskell file. It uses Shake to build the website, Pandoc to render posts, and Mustache for templates. The script starts with these Magix directives 2 : Running the script is as simple as: Magix compiles the script into an executable and runs it. Nothing else to install, no , or required. Nix and Cabal can also run scripts by providing shebang directives 3 , so why reach for Magix instead? The shebang reinterprets the script with on every run, which is slow. Cabal compiles the script, but it fetches dependencies from Hackage and builds them from source, leading to very slow first build and rebuilds. Magix compiles the script once into a binary executable and caches it for the next runs. It also fetches dependencies from the prebuilt Nix cache. So running via Magix is faster than either case. But when running on GitHub Actions, we have a problem. GitHub Actions gives us a fresh runner for every build, with no Nix store and no Magix cache. So every run, we have to install Nix and Magix, download all the dependencies, and compile the script. In one instance, the build from scratch took 105 seconds, with installation, dependency download, and compilation taking 92 seconds. That is a lot of wasted work because the script and its dependencies change rarely. The actual run itself takes only a few seconds once the executable exists. What if we could persist the compiled executable across runs? Magix’s build is deterministic: the same script and the same nixpkgs revision produces the same executable. If we could stash that executable somewhere durable, a cache hit could skip Nix and Magix installation, as well as the script compilation entirely, and just run the executable. Magix creates a Nix derivation for compiling the Haskell script with GHC and builds it. The resultant executable lives in Magix’s cache, as a symlink with a path like: GHC statically links all the Haskell libraries into the executable. The only dynamic dependencies are a handful of libraries—zlib, libffi, gmp etc. So if we have the executable plus those few libraries as a self-contained unit, we don’t need the hundreds of other packages in the Nix store 4 . I wrote a small script, , with two subcommands: The script works for any Magix script, not just . Let’s go through it. The script takes a command and a script file, derives the bundle name from the script file name, and resolves the directories it needs: is the script file name without its extension, and is a SHA-256 hash of the script contents; together they key the bundle directory. mirrors how Magix itself resolves its cache directory: or , overridable with the environment variable. Bundles live under by default, overridable with the env var. The function locates the script’s build result in Magix’s cache, copies the executable out of the Nix store, and gathers the libraries it links against: Here is what it does: The two flags are the interesting part. The flag makes the dynamic loader find the bundled libraries, so we don’t need or the Nix store at run time. The flag is needed because Nix rewrites each executable’s dynamic loader to point at its own glibc inside the Nix store. Since we don’t bundle glibc, we reset the interpreter to the host’s loader according to the system’s architecture 8 . We deliberately do not bundle glibc. Unlike the other libraries, glibc cannot simply be shipped alongside the executable 9 . The dynamic loader needs to be at an absolute path and needs to be matched with the glibc version. So we simply don’t bundle it and rely on the host’s glibc. One caveat here is that the host’s glibc must be at least as new as the one the executable was built against. That works because glibc is backwards-compatible 10 . The rest of the script is the subcommand and the command dispatch: computes the same SHA-256 hash of the script, checks that the bundle exists, and s the executable, passing the remaining arguments through untouched. The workflow uses Nix, Magix and the bundler script to execute the Haskell script. First, we set the runner image and the nixpkgs branch we track 11 : The next two steps compute the cache key and restore the bundle if we have one: We resolve the nixpkgs branch to its latest commit with at the start of the job, use that commit in the cache key, and pin the whole build to the same commit via , as we see below. The cache key has two parts: the hash of the script and the resolved nixpkgs commit hash. When the branch moves or the script changes, the cache misses because of the key change, and we rebuild against the new commit and/or script. We also restore the previously cached bundle to , if found. The next four steps run only on a cache miss: adds Magix’s binary cache to Nix, so that we get the prebuilt packages for Magix. The step runs with no arguments 12 : This causes Magix to compile the script, but Shake has nothing to build, so it exits immediately. The build is pinned to the same nixpkgs commit that keys the cache, via the env variable used by Magix. Finally, packages the compiled executable by running the bash script we saw earlier. The rest of the workflow runs on every run, cache hit or not: I’ve cut down the rest of the workflow to the only step that is relevant to us: that runs the bundled executable with the script’s arguments. Other steps are specific to BlogShake. All of this work, what does it buy us? Here are two numbers: With the cached bundle, the runner just downloads it and runs the executable, skipping Nix and Magix installation and script compilation altogether. This post showed how to speed up a Haskell script running on GitHub Actions by caching a compact bundle of its compiled executable. The approach works for any Haskell script that runs on Linux: compile once using Magix, bundle the binary with its library dependencies, and let the cache do the rest. One caveat though: the bundler script uses the internal details of Magix, which may break if Magix changes how it works. The full source code: If you have any questions or comments, please leave a comment below. If you liked this post, please share it. Thanks for reading! At the point of writing this post, Magix supported Bash, Haskell, and Python. ↩︎ Because the executable is compiled once and run many times, we build the script with . ↩︎ The shebang-based alternatives look like this: for Nix, and: for Cabal. ↩︎ Why not cache the entire Nix store between runs? Because the size of the full Nix store closure required to build Haskell scripts is usually in GBs. Caching that per run would defeat the purpose of caching by taking way too much time to download the cache. You may also want to reach out for the Nix bundle feature, which produces self-contained compressed executables. But these executables are still too big: 45 MB compressed/178 MB uncompressed for BlogShake. Our approach in this post results in a 9 MB bundle, compressed. Another completely different option is to build a fully statically linked executable, which I wrote about in Nix for Haskell: Static Builds . However, that requires a custom toolchain, running which on GitHub Action is too complex and/or slow. ↩︎ A result symlink can be left dangling if the store path has been garbage-collected. We skip those and pick the newest live one. ↩︎ Magix wraps the built executable with , which renames the real executable to and puts a wrapper script in its place. We copy the real executable. ↩︎ Nix store files are read-only mode, and we are about to modify the file, so we make it writable. ↩︎ We hardcode the loader paths here, but it should work on most mainstream Linux distributions with glibc. ↩︎ Well, it can be actually. That’s what Nix bundle does. It copies the glibc in the Nix store to the bundle, and points the program interpreter at the bundled loader inside a chroot . ↩︎ An executable built against glibc 2.42 runs fine on a host with glibc 2.43, but not the other way around. ↩︎ The glibc constraint dictates the runner image. Here we build against nixpkgs branch , which has glibc 2.42. Ubuntu 26.04, the GitHub runner image we use, ships with glibc 2.43. So they are compatible. ↩︎ That no-argument behavior is Shake-specific: with no actions given, Shake runs nothing, so a bare compiles the script and exits. A general script won’t do that by default. If you adapt this for a non-Shake script, give it a mode that does nothing, say a flag, so running it bare just produces the executable. This step’s only job is to get Magix to build the script, not to run it. ↩︎ Thanks for reading this post via feed. Feeds are great, and you're great for using them. ♥ This post was originally published on abhinavsarkar.net . Read more of my posts and notes . The Problem The GitHub Actions Workflow The Conclusion It picks the latest symlink for the script by modified time, skipping dangling ones 5 . It dereferences the symlink to a Nix store path, copies the compiled executable at from the Nix store path into the bundle 6 , and makes the copy writable. 7 . It copies the dynamic library dependencies of the executable by calling the function . finds the dependencies with , and copies the ones that live in into the bundle’s directory, skipping glibc. Note that it does this recursively, copying the dependencies of dependencies as well. It rewrites the executable with , setting the interpreter and the library search path for it. It also sets the library search path for libraries themselves so that transitive dependencies work as well. Build with no bundle: 1 min 45 sec. Build with cached bundle: 10 sec. BlogShake Haskell script Magix bundle script BlogShake GitHub Actions workflow At the point of writing this post, Magix supported Bash, Haskell, and Python. ↩︎ Because the executable is compiled once and run many times, we build the script with . ↩︎ The shebang-based alternatives look like this: for Nix, and: for Cabal. ↩︎ Why not cache the entire Nix store between runs? Because the size of the full Nix store closure required to build Haskell scripts is usually in GBs. Caching that per run would defeat the purpose of caching by taking way too much time to download the cache. You may also want to reach out for the Nix bundle feature, which produces self-contained compressed executables. But these executables are still too big: 45 MB compressed/178 MB uncompressed for BlogShake. Our approach in this post results in a 9 MB bundle, compressed. Another completely different option is to build a fully statically linked executable, which I wrote about in Nix for Haskell: Static Builds . However, that requires a custom toolchain, running which on GitHub Action is too complex and/or slow. ↩︎ A result symlink can be left dangling if the store path has been garbage-collected. We skip those and pick the newest live one. ↩︎ Magix wraps the built executable with , which renames the real executable to and puts a wrapper script in its place. We copy the real executable. ↩︎ Nix store files are read-only mode, and we are about to modify the file, so we make it writable. ↩︎ We hardcode the loader paths here, but it should work on most mainstream Linux distributions with glibc. ↩︎ Well, it can be actually. That’s what Nix bundle does. It copies the glibc in the Nix store to the bundle, and points the program interpreter at the bundled loader inside a chroot . ↩︎ An executable built against glibc 2.42 runs fine on a host with glibc 2.43, but not the other way around. ↩︎ The glibc constraint dictates the runner image. Here we build against nixpkgs branch , which has glibc 2.42. Ubuntu 26.04, the GitHub runner image we use, ships with glibc 2.43. So they are compatible. ↩︎ That no-argument behavior is Shake-specific: with no actions given, Shake runs nothing, so a bare compiles the script and exits. A general script won’t do that by default. If you adapt this for a non-Shake script, give it a mode that does nothing, say a flag, so running it bare just produces the executable. This step’s only job is to get Magix to build the script, not to run it. ↩︎
For many years, I sometimes discovered that commands I was sure I had run were no longer present in my Z shell history file ( ). In this article, I will show you how I tracked down the bug. Spoiler: ultimately, patching Zsh to make it crash loudly and analyzing the crash’s core dump was the winning strategy! Zsh 5.9.2 (released July 12th, 2026) contains a fix for this issue — open it after reading this investigation to not spoil the fun. Zsh fix 53454 Occasionally, I noticed that commands I knew I executed the day before were not findable in my shell history, meaning pressing Ctrl+R for backward history search yielded no results. Whenever I noticed this, my shell history file contained only very old entries, with years of newer entries missing. The first few times this happened I just restored my shell history from my daily backup and did not bother investigating any further. But the issue kept happening. I noticed that there was no visible corruption in the (no non-printable characters or incomplete lines of text), and that the number of lines in the file was not always the same. What was not clear to me was whether it was Zsh itself, or some other program, or perhaps the combination of multiple processes that caused the issue. I set the following history-related options in my : In practice, this means my shells are separate sessions that all stream their commands into a shared . The history is intentionally not shared, so when I want to access entries that another shell wrote, I explicitly run . When I asked for help on Mastodon in December 2024 (mostly in the hope that somebody else already encountered and diagnosed this problem), one suggestion I got was to use file system change monitoring mechanisms like inotify or fsevents to find the culprit that truncates (or changes?) the Zsh history file. The next sections walk through the available options on Linux which I tried. The Linux kernel subsystem is one of the oldest file system change monitoring APIs available in Linux (released 2005). To get a good understanding of how Zsh modifies the history file, it is not sufficient to monitor just : The file is opened, accessed (= read) and then… deleted?! By monitoring the containing directory, we see the whole picture: So Zsh reads the old history file contents, writes them to a new file, then renames the new file over the old one, thereby deleting the old one. Now it makes sense! Unfortunately, we do not see the process IDs (PIDs) of the responsible process for the file system event, not even with the sibling utility , which uses , an API that does provide this information! I checked, and the kernel does send the PID , but does not display the PID. Luckily, there is , which does display the process name and PID. Here is what Zsh’s history rewriting looks like with : This gives us the PID, so now we can verify whether multiple processes were involved in corrupting the shell history. But, we don’t have any insight into how much data each Zsh PID is reading/writing, so even with a log, it would still not be clear what happened. Of course, one could use , in particular with its flag, to further look into Zsh behavior, but it seems like a logistical nightmare to arrange for every (interactive) Zsh process to get a corresponding strace run, and I was not sure if always-stracing a shell changes behavior in subtle ways, so I did not pursue the route. (Once I had a reproducer, became easy enough to use and very helpful.) To get more visibility into Zsh’s read and write operations, we can reach for . To get started, I created the following bpftrace program, which is run on every syscall and logs which process opened the file, including the user stack trace: On NixOS 26.05, I can run the program as follows: Encouraged by this early success, I extended the program as follows to cover more system calls: In case you want to dive deeper into bpftrace, here are a few resources I found useful: I created a systemd unit to run this program in the background permanently (seems cheap enough), meaning I can check the logs like so: One day, I noticed my shell history was truncated and checked the logs. This is what I found. Note how there is no line, i.e. Zsh does not read until EOF: From the bpftrace output above we know that Zsh is rewriting my file incorrectly: it reads fewer lines than usual, and then correctly writes them to . At this point I decided to study the code for reasons why would not read the full history file or why would not write the full history file. The control flow of is pretty hard to follow, but it is easy to modify the code (zsh-5.9.1) such that it crashes after it writes a with fewer than 50000 lines and before it replaces my with that truncated new file: On Linux, the easiest way to ensure such a crash ends up somewhere useful is to install , after which systemd will automatically collect core dumps. You can use to list and work with them. Note that these core dumps contain your shell history, so do not upload them to third-party services. Fedora’s ABRT seems to only send micro reports (i.e. without your full shell history), and Ubuntu’s Apport is disabled-by-default , but it’s worth double-checking. I installed my patched version of Zsh (with debug symbols enabled) and deferred further investigation until I had a core dump of the issue in action. Sure enough, when I checked with a few days later, I saw a crash! This was the backtrace: I returned to the source and realized that most likely, is just writing out a shorter history file because left it a shorter history! The control flow of is easier to follow. Reading through the function, there is one possibility of an early return: when Zsh receives a signal , the read loop is aborted via a : Let’s see what and contain in our crash: Bingo! So some signal must be involved. For reasons outside of the scope of this article, I am using a mosh session from which I am starting a long-running SSH session, over which I multiplex further sessions. When tearing down this setup at the end of each workday, I press Ctrl+D in the multiplexed sessions (sends EOF, exits the session), then Ctrl+C on the long-running SSH, then Ctrl+D to exit the mosh session. (If you don’t cleanly exit a mosh session, it sticks around on the server and subsequent logins tell you about these orphaned sessions. I wanted to avoid accumulating orphaned sessions.) So in practice I press Ctrl+D, Ctrl+C, Ctrl+D, Ctrl+C etc. until all windows are gone. As part of that sequence, most likely I am exiting a Zsh session (Ctrl+D) and then interrupting (Ctrl+C) its if history rewriting takes long enough. With these clues, I built a standalone reproducer and sent a bug report to the zsh-workers mailing list in March 2025 . Bart Schaefer looked into it and posted a fix in April 2025 (thank you!). It took a long time for the fix to actually be released because there was a long time without any Zsh releases. And then, when the 5.9.1 release happened, it turns out Bart’s fix was missed by the release engineer! I pointed out this oversight, and Zsh 5.9.2 thankfully includes the fix. I have been running Zsh 5.9 with Bart’s patch applied, and will keep that version pinned until 5.9.2 lands on my computers. If you’re pinning zsh on Debian, pin both, the and packages. Otherwise, you might end up with no package at all one day… When exiting, calls to compact the history: during a session, history entries are appended incrementally, but at shell exit, the history file gets compacted (to apply a size limit, if configured, for example), so reads the entire history ( ) and writes it out again. could be interrupted when a signal fires (it checks and short-circuits its read loop), but did not check for interruption when writing the shell history when exiting. Therefore, wrote the (incomplete) history, truncating the actual history. Let’s decipher the output we collected earlier: Why the lseek? From POSIX.1-2017 on fclose() If the file is not already at EOF, and the file is one capable of seeking, the file offset of the underlying open file description shall be set to the file position of the stream if the stream is the active handle to the underlying file description. Zsh uses to get a stream, so glibc reads in chunks of 4096 bytes and when closing the stream, the underlying file descriptor needs to be sought back so that the already-read parts of the current 4096-byte chunk will be read again, correctly by the next stream. (Zsh closes the file immediately, so the seek is pointless, but glibc cannot know.) It’s remarkable that a bug like this one, which causes data loss , can remain unfixed for 10 years in a popular shell (did you know? Apple switched macOS’s default login shell to Zsh in 2019). Granted, most users probably don’t share my habit of killing shell sessions in a way that makes it likely that is sent, but I have to imagine that some users have lost parts of their history. I am very glad that this issue is now fixed! If you are also encountering history file truncation, and it isn’t the issue I described in this article, maybe you managed to accidentally export ? See appendix A for a bonus footgun that I ran into a few years before. Another obvious question that came up as I was writing this post: I tracked down this issue before LLMs got impressively good at coding and problem solving. Would today’s AI coding agents be able to find this bug? See appendix B for details, but the answer is: Yes, today’s frontier models can find this bug! When you use Emacs’s TRAMP mode , by default it exports . For example, when using after starting , I see in the environment: This is a footgun, because most shell configs don’t unexport , they only change it. For example, in my , I set . When running an interactive shell (by typing followed by Enter), I end up with in the environment: …which is not the case when I use to log in: Exporting a shell-specific is a footgun on machines where other shells are configured with other (default) settings. On my work computer, where the Linux installation sets and for by default, I once inadvertently truncated my file to 64000 lines. My suspicion is that it was by running , then (to get my config), then (temporarily, to source a config and launch a script). To prevent such issues in the future, I decided to actively unexport in my . For a while now, I felt that it would be useful to get my hands dirty with creating my own evals. See Anthropic’s “Demystifying evals for AI agents” if you are unfamiliar with the term “eval”. I started with Simon Willison’s smevals , but found it to be too minimalistic: without taking extra measures, agents would quickly escape their eval task and peek at the solution, or use the internet to discover that the Zsh git version has this bug already fixed. I ended up with Inspect, an open-source eval framework by the UK AI Security Institute and Meridian Labs, and it worked better, though its web UI is very minimalistic. This eval quickly got very expensive! I paid well over 300 USD in token cost for about 3 attempts at this eval. The results below are from the latest attempt. A passing grade is awarded when the model explains the correct sequence of events: an interrupt sets errflag, which aborts and results in a truncated history file. when i log out, sometimes when i come back the next day my .zsh_history file is mysteriously truncated. why might that be? I’m on zsh 5.9.1 on Linux. Only zsh ever writes this file. I have a bpftrace program logging every syscall zsh makes against the history file. A NORMAL logout looks like this: A logout that TRUNCATED the file looks like this: my zshrc is in ./zshrc — the exact config in effect on the affected machine, so you can see which options are (and aren’t) enabled. The full zsh 5.9.1 source tree is available in ./zsh-5.9.1 — this is exactly the version I’m running. Dig into it as much as you need. What’s going on, and what in the zsh source would cause it? Work only from the zsh 5.9.1 source provided and the evidence above. Do not consult newer zsh versions, upstream commits, mailing-list threads, changelogs or release notes — the point is to derive the cause from this source, not to look up how it was later fixed. End your reply with a section headed exactly containing your final answer: the root cause, and the specific code responsible. In this iteration, I am including this hint about pressing Ctrl+C and Ctrl+D repeatedly, which is a nudge towards signals and interrupt handling: fwiw, my logout habit: i press ctrl+c / ctrl+d repeatedly until all my terminal windows are gone, and then see what’s left. This measures how easily the models understand the problem, if at all. Latest frontier models like Claude Opus 5 or GPT 5.6 Sol can find the bug reliably with just a description of the symptom and a working/failing bpftrace. If you try it a couple of times, you can also get there with the Gemini models. Of the Open Weight models, only Kimi K3 can find this bug without hinting. Once the Ctrl+C + Ctrl+D habit is included in the prompt, more frontier models reliably find the issue (including Claude Sonnet 5!). Of the Open Weight models, GLM 5.2 and Kimi K3 are the first ones to reliably figure out the issue! If you try it a couple of times, you can also get there with the Gemini or DeepSeek models. I could not get Qwen or Minimax models to pass. This seems like a really nice eval, in particular for tracking which Open Weight model actually works as well as Opus or GPT (at least in this one specific regard). For now, Kimi K3 seems like the most capable Open Weight model, even though it cannot reliably diagnose this issue. GLM 5.2 is much smaller and — with hints — can at least make sense of the issue. It is interesting to note that almost all models considered the correct hypothesis, including the Qwen and Minimax models. Only Gemini 3.1 Flash Lite never articulated the correct hypothesis, presumably because it is a small model (in comparison). So where did the models go wrong? In verifying/falsifying theories! For example, GLM 5.2 assumes the in the output must mean that is set (it isn’t!): glm-5.2 enumerated exactly three causes of a short read — corruption, searching, — then ruled out the interrupt because “Options 1 and 3 don’t involve lseek to a non-zero offset. But the trace shows , which is behavior. So must be set” — overriding your ’s to keep the elimination alive. I verified that by making the eval use more orchestration (have one subagent produce theories, another keep track and falsify / verify, etc.), the success rate increases. Similarly, I expect that by varying the prompt and harness, individual models can be made to work much better. The most common failure mode seems to be that the model picks the wrong theory and gets stuck on verifying it, never returning to the other theories. Perhaps the better performing models have the better methodology, in that they adhere better to the scientific method? The upstream bpftrace docs The blog post “First steps in system-wide Linux tracing” by Martin Pitt (2020) The LSFMM presentation “BPF Observability” by Brendan Gregg (2019)
I’ve always been a big fan of installing Python from source. It keeps you close to your environments and understanding exactly where the code comes from and goes to. I feel like in the past it was more difficult, but year after year it seems like it’s getting easier. This year for Ubuntu 26.04 there is a bit of a new step added, but ultimately it is feeling effortless. The whole process takes about 10 or so minutes, and is good for people of all levels to understand. You may need a C compiler if you haven’t yet downloaded: There is a new way to get the build-deps for Python on Ubuntu, so you’ll need to edit and add to the Types line. Now that has been added you can get build-deps for Python: I generally install all additionals since many are quite important for python. Some are more specialized and you may not use, while others like libcurses are basically required if you expect to ever need to interact with an interpreter (eg troubleshooting a production environment). Others you may not need until some future data, say when you install a package that expects a certain kind of data compression. Overall, my advice is just install all unless you know more or have a specialized use case for streamlined environment. Download the latest version: Choose Gzipped Tarball https://www.python.org/ Python is now installed. Last important step is to configure the system links: x Finally you can test: you will now have a directory like Python3.12.xxx Move python installation directory to /opt/ to keep home clean into the directory and run each command Check you have all packages installed from above Builds Python, this step takes some time. altinstall skips creating the python link and the manual pages links, install will hide the system binaries and manual pages. This means that we will leave the system python installation untouched
Proxmox today announced their Proxmox Virtual Environment is now available for 64-bit ARM . I tested it on my Ampere Altra Dev Platform —the same machine on which I've booted Windows on Arm the first time, messed with multiple GPUs, and most recently tested Houdini's native arm64 support . Install was easy, as the Ampere Altra uses UEFI / ACPI for its hardware, meaning Proxmox didn't have to tailor its ISO to specific platforms, like you have to do with Raspberry Pis and most SBCs using a Device Tree setup.
I have been fascinated and amazed by stage0 for a while now ever since I learnt about it via Guix using it to provide twenty two thousand packages source-bootstrapped from the 357-byte seed. What is stage0? It is a chain of compilers and assemblers that can be built from source, starting from a 357-byte program that can eventually build a recent GCC. 1 Since then, NixOS and other distributions have also adopted the same approach to minimize their binary seed which makes it possible to onboard new architectures and platforms much simpler. What’s always frustrated me as a Bazel (& Buck ) user is the reliance on prebuilt toolchains even for things that should be built from source easily like protoc . Bazel has given up trying to provide a hermetic C++ toolchain and the upstream rules_cc ruleset just points you elsewhere: Configuring a hermetic toolchain makes your build more deterministic. rules_cc itself does not yet offer a hermetic toolchain distribution I had attempted to provide a stage0 hermetic C++ toolchain in October 2024 via https://github.com/fzakaria/stage0-bazel . I made substantial process through the bootstrap process but I did not make it far enought to be usabale. To be honest, I was also a little disheartened that no one else in the community thought it was the greatest thing since slice bread. Everyone seems to be content with using prebuilt toolchains as they go deeper into MODULE.bzl madness . I had put it aside for a while, but I have been thinking about it again recently. The steps are mechanical and the process imitates existing distributions, so this became a perfect project for me to throw at an LLM to finish. 2 You can now leverage the toolchain to build in Bazel and have it compiled by a toolchain whose entire ancestry is in the repository from that same 357-byte seed . 🎆 How complete is this toolchain? I pointed the toolchain at Abseil and GoogleTest straight from the Bazel Central Registry without any patches . We then can build and run their testsuite to provide a sanity check that the toolchain is working correctly. We use a to filter tests that require . Abseil marks as a , and Bzlmod drops dev dependencies of non-root modules. That is us building Abseil and GoogleTest, from the registry, unpatched, compiled by a toolchain that began as 357 bytes of hex. How can I be so sure this is a hermetic toolchain? The toolchain includes an audit report that uses Bazel’s aspects to inspect every action in the build graph and verify that it only executes programs built by the toolchain itself. The report is generated by running and will fail if any action executes a program outside of the Bazel output tree. 3 The report is two lines long: Unfortunately, since runs a shell it takes as an absolute system path that is also listed as a seed binary. ’s attribute is a string, and the shell is not a declared input of the action, so no artifact this repository built can provide it. Building toolchains from bootstrap seeds was never a priority for companies like Google where they control the entire build environment. However we seemed to have adopted the same approach as Bazel and similar build systems have become more popular in the open-source community. We should strive to make our builds more reproducible and hermetic, and this is a step in that direction. Once you can reach a recent-enough GCC, you can build any C/C++ program and beyond easily. ↩ Consider this the disclosure that I used an LLM to help me write the remainder of the toolchain. ↩ We also set to disable Bazel’s built-in C++ host toolchain detection. ↩ Once you can reach a recent-enough GCC, you can build any C/C++ program and beyond easily. ↩ Consider this the disclosure that I used an LLM to help me write the remainder of the toolchain. ↩ We also set to disable Bazel’s built-in C++ host toolchain detection. ↩
When I was building my gaming PC a few years ago, I bought a second-hand pair of RAM sticks with RGB lights. It was entertaining to see them glowing and changing colors. I did not bother myself to control them. Some time later, I got sick of gaming and made a decision to rebuild it into a self-hosting server. The PC moved from my table to an under-TV shelf in the living room. And here, glowing lights became distracting during watching TV or when we have guests sleeping in the living room. Therefore, I started searching for ways to control/disable RGB lights. After some search, I encountered OpenRGB . It’s open-source RGB control software that is independent from manufacturer software. It supports color syncing across several devices and has a community-managed effects library . It has a long list of more than 2,500 supported devices and, hopefully for me, my Corsair Vengeance was there! What a great piece of software, right? It’s a shame that I needed this software only to disable RGB . OpenRGB runs on Linux, macOS, and Windows. You can download the package for your operating system from the releases page . On my Debian-based server, I downloaded the package and installed it with: The package includes the required udev rules. If you use an AppImage, Flatpak, or a build from source, install the udev rules too. Without them, OpenRGB may not be able to access the RGB hardware without running as root. Before creating the service, check that OpenRGB can detect your devices: Make a note of the device numbers you want to control. Mine were and . I needed the RGB to turn off every time the system starts. So I ended up with the following systemd service. First, test the command manually. is black, which turns the LEDs off. The mode is supported by my devices; use the output of to check which modes yours support. When that worked, I created the service below. Now the RAM lights turn off automatically on every boot. No proprietary RGB application, no running desktop session, and no bright lights in the living room. If you have devices with RGB lights, check whether they are supported by OpenRGB . You might find a more creative use for it than I did and build some nice RGB animations.
I have been working more on GuixPkgs in preparation for a talk at nix.vegas for DEFCON34 . At the end of my previous GuixPkgs post I left a teaser: We can then build a NixOS machine where every package is the Guix equivalent 😱. Well, adeci 1 took the bait and we went even further than that. 😈 Say hello to Guix by Nix : a bootable VM where the kernel is Guix’s Linux-libre, the userland is translated Guix packages, and PID 1 is GNU Shepherd . No systemd. No D-Bus. No NixOS activation. Not even a or binary in the guest. It’s Guile all the way down and Nix built all of it. Log in with / and poke around: As a reminder, even though these binaries live in , they are not Nixpkgs packages. They were translated from Guix derivations using guix-transfer and built by the . That was built from Guix’s package definition, source bootstrap and all , but it lives in , because built it. If you want to try out any of these packages on your own machine, you can use GuixPkgs . Tip Guix offers all packages built from source where Nix may offer it as a prebuilt binary. You can use GuixPkgs to get a source-built bootstrapped version of OpenJDK for example and all the whacky steps to get there. How does this actually work? The project is a three-stage pipeline: Warning AI was leveraged to write the initrd and activation scripts. That seems to trigger people lately, so consider yourself warned. Every program that can be executed, every ELF file, every script interpreter, all traces back to a translated Guix derivation. The only things Nix authored are text files: the init scripts, the Shepherd config, . This is Guix by Nix . “Every executable byte comes from Guix” is exactly the kind of claim that’s easy to say and easy to fudge. 🤥 A booted demo is great but that can’t prove it. A VM where secretly came from Nixpkgs boots identically . The flake ships an audit-check that classifies every store path in the shipped closure by provenance, unpacks the compressed initrd 2 , and inspects every executable payload. The audit derivation fails if anything is unclassified, any ELF file or script interpreter doesn’t trace to a translated Guix output, any reference survived translation, or anything systemd-shaped appears anywhere. The report includes a lot more information such as the exact Guix channel commit everything was translated from. There’s also some NixOS VM tests for good-measure. 🕵 Note The kernel is Guix’s , translated and built by like everything else. Nix wraps it in a thin adapter so NixOS’s VM tooling accepts it by augmenting with some additional metadata only; the is byte-for-byte Guix’s. One obvious next step would be a normal NixOS machine, where every package that exists in GuixPkgs shadows its Nixpkgs equivalent on . GuixPkgs offers such an overlay, but it needs quite a lot of CPU to build all of it…. A more realistic use case is to mix and match GuixPkgs and Nixpkgs packages in a single system. Get the best of both. I heard there were people in the Nix community who still want a non-systemd system, à la sixos . 🫠 For now, this project remains a minimal VM demo. If you make it PID 1 on real hardware, please send photos. 🙇 It has been extremely fun and rewarding to work with Alex on this project. We nix-pilled him at PlanetNix 2 years ago and since then he has been pushing the boundaries of what Nix can do and is currently employed at Shopify working on Nix. ↩ Nix’s reference scanner can’t see inside archives and sneaky paths hide there. ↩ guix-transfer is the tool that translates Guix derivation graphs into Nix derivations. GuixPkgs is the flake with all Guix packages, all built from the 357-byte seed, although there is a Cachix cache provided. Guix by Nix assembles ~42 of those packages, a subset of the overall set, into a useable machine. The initrd is a custom shell script, interpreted by Guix Bash, that loads eight modules, mounts the root disk and the 9p store, and s. Activation (accounts, , the setuid copy) is another custom Nix-generated script run by Guix Bash. PID 1 is from Guix, with a small Scheme config that starts , , , and a serial . works like you’d expect. is compiled from Guix’s own search-path specifications, so and friends point where Guix intended. It has been extremely fun and rewarding to work with Alex on this project. We nix-pilled him at PlanetNix 2 years ago and since then he has been pushing the boundaries of what Nix can do and is currently employed at Shopify working on Nix. ↩ Nix’s reference scanner can’t see inside archives and sneaky paths hide there. ↩
I'm creating this page to document my "software factory". It will be more of a reference page than an article and I will reference it on the resources page of the site. Context : several applications in monorepos ( hakanai , Writizzy , Bloggrify ), polyglot (Nuxt, Kotlin, JS), solo dev, code written largely by agents, continuous deployment to production. ::toc{open="true"} :: Even though the latest generation of agents are now capable of producing quality code (much better than the majority of human developers), code production represents only part of what I call software quality. The rest includes: Part of these risks is resolved with a good understanding of your market, benchmarks, interviews, manual testing, mockups. All of this is also part of my "software factory" even if I don't describe everything here. The code produced is now almost 100% generated, but that doesn't mean it's vibe coding. Vibe coding as defined by Karpathy was experimentation and letting yourself be carried along by a dev session. Here, I'm going to talk about context engineering. The goal is to provide all the necessary context, at the right time, so that the software matches an intention and is systematically controlled. Even though I don't write the code, I'm responsible for it and I need to maintain control over it. The tooling described here answers several questions: Here are the files read by agents before starting. Be careful, the size of the context must remain controlled. Too large a context costs money and degrades the quality of responses if it becomes too heavy. We want to save tokens and optimize when information is loaded. The separation matters: permanent context stays short, specialized context loads when it's useful. Long-term constraint as context. Concrete example: I plan to open-source part of the code. The rule "code intended for open source should never depend on proprietary code" is written in the rules and verified by a test (see layer 4). Writing a future constraint in the context avoids paying for a refactoring later. You can also find constraints on: A constraint is specific to a project and a person. It's not a matter of software quality in the strict sense. It's not better or worse to do feature flagging for example, but it's my preference for partial deployment and feature activation in testing. There's no point in telling an AI to "write quality code", it doesn't make sense. You need to make your own constraints explicit. Example of a rule that allows deferred loading depending on context: It's simple, but it avoids loading the entire skill in a rule (which would be loaded systematically). Here's a more complex rule: Here there are two important things: A skill is a procedure written once, replayed identically. The agent loads it itself when the context matches. This automatic loading can sometimes fail. In that case, you need to explicitly ask to use the skill. The criterion: am I repeating myself? If I explain the same thing a third time, it becomes a skill. I have about thirty, grouped by family: Two things I've learned: Sub-agents : for tasks that generate a lot of reading without much decision (audit, broad exploration, doc writing), I delegate to a sub-agent. It consumes its own context and gives me a conclusion, not a dump of files. I use them less and less, recent agents make their own fairly targeted delegations. You often hear that an AI is non-deterministic and can make mistakes on trivial things that need to be deterministic, like calculating 2+2. That's largely false now, and an AI is no longer "just an LLM": it has many tools to control output. Nevertheless, the best way to ensure a form of reproducibility is to delegate to tools whose job it is. Compilation, test execution, linters, all of that is delegation. You can delegate to MCPs, or to skills that use themselves a command-line tool (CLI). I try to avoid MCPs which consume more context, but I have a few anyway. The repo is indexed in a graph (symbols, relationships, execution flow). This allows you to measure impact levels and find all links with the code being modified: The real issue isn't speed, it's detecting all side effects of a modification. I use two levels of memory: What we want with these tools is to avoid repeating mistakes, document decisions, and not start over with an empty session each time. Claude Code's internal memory mechanism has improved and becomes more relevant than before with the latest versions. However, you need to control it and not hesitate to ask it to delete rules it creates on its own which are sometimes a bit silly. Claude-mem, I honestly have a hard time measuring the negative or positive impact. I don't have enough perspective on it yet. A wrapper (here RTK) prefixes shell commands and only returns what's necessary. It remains very limited, only available for a few tools. The gain is sometimes canceled out because Claude runs the command twice. It doesn't hurt, but I think there's still room for improvement. Note that Claude builds its own tools on the fly in Python or Bash, and also knows how to use filtering mechanisms with , , etc. to optimize the outputs of the tools it uses itself and save tokens. I use two skills for incident resolution: I could also mention the Stripe MCP, read-only as well, which allows in certain specific cases to investigate Stripe configuration issues. Guardrails prevent the randomness associated with understanding and executing an instruction. It happens that an LLM ignores a rule. You need to provide tools that execute automatically. Scripts triggered by the agent harness, not by the agent itself. Other uses that work well: block editing of generated files, require a test alongside any new module, forbid a dangerous pattern. Structuring rules become tests that break the CI. Real example, I have an architecture test that preserves the boundary between future open source code and the rest: It's part of the automated tests, so it can't be bypassed, unlike a rule. I use several things: Each app has its workflow on GitHub Actions. The deployment job has a on the quality job. Nothing goes to production without passing the gate. It's essential in general, even more so for automatically generated code. I shouldn't need to re-explain this, but just in case, I have several types of tests. I give Claude instructions to explain the test hierarchy, gitnexus tells him what to replay to validate these modifications, and he has instructions to write them when he adds/modifies code. As I said in the intro, the goal isn't just to produce code, it's to produce code that serves a purpose. The process often starts with a spec, then a design, then an implementation. Spec first. I have a folder of numbered specs, one per functional domain, indexed in the permanent context. Two skills frame the cycle: one for writing the spec and its plan, one for closing it by updating it with what was actually built. This spec and the discussions can rely on a 'product-marketing-context.md' file that I create in each project and which summarizes my personas, my competitors, my positioning etc... The design. By design I mean two things: technical design and interface. Technical design is part of the spec phase. Most of the time a spec is enough, but some tricky cases require a spec dedicated to a technical component or a technology choice. On the other hand, the design/mockup phase is separate. I do it in Claude Design. I make a functional mockup and iterate until it's perfect. I verify understanding, labels, usability. Then I can pass the result to Claude Code. The implementation. Claude starts from the spec and mockup. He follows the plan made in the spec phase. The spec is meant to be delivered in stages (protected by feature flag). This allows me to do several small implementation sessions rather than one large session, which tends to degrade in quality if it gets too full. The closing step is important: without it, specs become obsolete in six months. Explicit rule in the context: if a spec is vague or inconsistent with what exists, the agent should ask the question, not guess. Progressive delivery. I work trunk based. However I use feature flipping and gating, which represent two different things: I have skills that explain the difference, so agents don't do anything wrong and respect my working process. If you're starting from 0, the first step is the quality gate if you don't have one. You need a control mechanism that runs tests, linters etc... Then start light with a Claude.md that describes the essentials, the why. Add rules as you go for important architecture patterns. As soon as you see procedures that come back often, document them as skills. And then equip yourself with cli and MCPs to interact with your main tools, jira, sentry, etc... Be careful, any skill, MCP, code taken from outside must be scrutinized. These are dependencies that can be vectors of attack. You need to take into account that the technology is still very young, February 2025 if we consider agentic programming. Tooling is improving but you also need to constantly review it. Mid-2025, some instructions in a Claude.md made sense, for example "write a test for each new service". Today it's noise and Claude does it naturally. So you need to be wary of your old rules, sometimes they're obsolete and create noise. I have no way to measure and know if an old rule has become obsolete. Latest versions of Opus are increasingly autonomous. AI takes the initiative on its own to build, look at produced content, read dependency code to understand calls, find bugs, run tests in the browser. It's almost creepy and more rigorous than 99% of humans. Let's be honest, I'm increasingly less useful in implementation phases, but I don't want to lose control of the produced code. I'm torn between satisfaction at having an increasingly efficient software factory and the risk of losing knowledge. I need to find a way to control designs a posteriori, to appropriate the result. I recently added a boyscout.md rule But I find myself having endless sessions. I think I've never worked on a codebase that maintains itself as much. I've rarely improved the product at this level of detail. But it comes with a cost, cognitive overload. I think I'd rather automatically note these elements and categorize them in an online TODO list (trello, todoist etc...). I think the workflow for maintenance should move elsewhere, and be partially automated. I still copy and paste my skills/rules etc… from one project to another. And sometimes it's dependent on my station based on a skill installed locally. I need to find a way to package my skills to deploy them where relevant and to centralize maintenance. In other pending improvement points: the intention (why, for whom) does it satisfy the 4 risks identified by Marty Cagan: Value Feasability we can add: performance and reliability What does the agent know? (context, memory, code graph) What can it do deterministically, without improvising? (skills, procedures) What stops it when it makes a mistake? (hooks, architecture tests, quality gates) marketing copy (labels in the application) how to write migrations types to favor for database fields asynchronism patterns etc. (list far from exhaustive and enriched regularly) the activation pattern, which specifies that this rule only loads if you touch the directory the table that lists all available skills, to be opened only if needed. If the AI doesn't make schema changes, there's no point opening the skill "Multi-file procedure" skills are the most profitable. Example: adding a block to the content editor touches three different rendering surfaces. Without a skill, the agent systematically forgets one. "Up-to-date doc" skills are very important on code that evolves quickly. It allows an agent to understand the entry points and intentions of a feature. before modifying: blast radius, callers, risk level before committing: did I only touch what I wanted to? find an execution flow rather than grep a function name rename via the call graph rather than find-and-replace Claude's internal memory Claude-mem , which allows capturing decisions between sessions a Sentry skill to read information on Sentry and retrieve stack traces a skill that gives read-only access to the database (read/write access is possible directly via on the Docker container in local dev) no file on the "open" side references the "proprietary" side every production file belongs to one of the two sides ESLint for syntax ast-grep for architecture decisions , for example forbidding any call to without going through the OpenAPI client typecheck for typing What matters must be executable. An instruction is followed "most of the time", but it can be forgotten. A hook or test is followed all the time. An error must be documented. Every error must be recorded in a skill or in memory. Context is a budget. Wrappers, filters, sub-agents, summaries: everything that reduces noise keeps reasoning on the problem. Measure impacts before and after editing. We want to avoid the effect "1 bug fixed, 10 produced". Repetitive procedures don't improvise. One skill per procedure. Spec documentation dies if its closure isn't in the process. You need to plan the update and maintenance step. my dependency on Claude. I want to test open weight models but I don't have the hardware for it. Moderate risk in my opinion, the entire ecosystem is moving upward the IDE is becoming obsolete compared to this new workflow. I still use Intellij but I no longer find it suited to our time. I haven't seen an interesting alternative yet.
The recently published RFC 9421 describes how to do HTTP Message Signatures , and starting just now , curl experimentally supports them. The specification describes this as a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message. It is a way to verify that selected parts of the HTTP request arrives unmodified and exactly the same as when the request was created by the client. These days, it is very common that there are layers of proxies, load balancers, front-ends, CDNs, web firewalls and what not in between the client and the ultimate application. With HTTP Message Signatures, there can be assurances that the headers are components of the request end are unaltered. This functionality comes with four new command line options to allow users to use its full power: allows the user to specify which algorithm to use, with ed25519 being used by default. The only other algorithm supported right now is hmac-sha256 . specifies the key to use when signing the request. is the key identifier, a string that is passed on in the headers. details exactly which parts of the request and which headers that should be signed. If not set, it defaults to signing the method, authority, path and query. With these four new flags added to the list, curl supports 278 different command line options. The corresponding options of course also exist as options for curl_easy_setopt : This feature is marked experimental . This means that it need to be explicitly enabled in the build to appear, and that we strongly discourage use of it in production as we reserve the rights to change it before it gets supported for real. We use the experimental phases as a time for people to test it, to tweak it and to learn what we should fix so that we then can support this to the end of time. We do not guarantee any backward compatibility for experimental features. Please test this feature and tell us how you experienced it! The more tests and more feedback we get, the faster we can get moved out of the experimental phase to have it present for real for everyone. This feature is already merged into git and will be part of the pending curl 8.22.0 release. As experimentally supported. This feature was graciously brought to us by Sameeh Jubran. Top image by Antonios Ntoumas from Pixabay : signing algorithm (“ed25519” or “hmac-sha256”) : the key to use for the signing : key identifier for Signature-Input : a space-separated list of components to sign
DevOps
Linux
Tutorial
Python
Hardware
Security
Kubernetes
Open Source
Database
AI
Frontend
Go
Php
Performance
JSON
HTML
Git
Ci Cd
C#
Shell
C
Programming
C++
UI
JavaScript