Jim Nielsen Today

We Need All the Disciplines

From where I sit, figuring out what to build, building it, and shipping it are all part of the art of making software. Many disciplines have emerged through the years to facilitate this work: product, design, engineering, devops, etc. We need all these disciplines to make great, dependable software (granted, in different doses). Each brings different values and judgments; each sees things the others don’t. And including all might mean going a bit slower. Human collaboration may seem less “efficient” than handing the work over entirely to the machines. But inefficiency may be the cost of collaborating with humans, rather than extracting an output from a machine. We gain not only from their judgment and expertise in a particular discipline, but from the relationship, trust, and shared understanding that blossoms from making something together. As the old proverb says: “If you want to go fast, go alone. If you want to go far, go together.” Reformulated for today: If you want to go fast, go with the LLM. If you want to go far, go with your fellow humans. Reply via: Email · Mastodon · Bluesky

Career Business DevOps

daniel.haxx.se Today

OSI approved the curl license

curl started out GPL licensed, as direct inheritance from its httpget days, but in October 1998 we switched to the Mozilla Public License (MPL). In 2001, we added the MIT license as an additional license to optionally use in addition to MPL. We made curl dual licensed because clearly the MPL license caused problems for some users – primarily because it was deemed not compatible with GPL. In 2002 we dropped the MPL option and curl was from then on licensed only under the MIT license. Kind of. Because of reasons we cannot recall anymore, I decided to ever so slightly edit the language in the MIT license text around that time. (I strongly discourage anyone else from doing this, and if my time machine was functional I would go back and correct this decision.) The license text we then kept referring to and thinking about as an MIT license, was in fact not exactly MIT anymore. No one objected at the time. I can’t remember anyone really considering the implications at the time. Maybe because not that many people noticed or cared. It was in essence still the MIT license. Decades later, the SPDX project quite correctly and ruthlessly made the distinction between a proper MIT and a modified MIT matter: they created a separate and unique identified to the license curl uses. It may be almost like MIT but they deemed the differences big enough to say that the curl license is a special beast. It is since then known as the curl license. I have over the years casually kept referring to the license as MIT since it helps users understand even though it is technically incorrect. Open Source people in general tend to know the MIT license and the implications of using it. A drawback with having our own license is that it was not immediately recognized and known as Open Source or Free Software; not acknowledged and verified by FSF nor OSI. On April 17, 2026. The Open Source Initiative approved the curl license . It is since then officially arguably slightly more Open Source than before. I don’t think anyone ever doubted or that it was actually much of a drawback in adoption or usage, but it feels good to finally have this checkmark. I write about this now (many months after the fact) because I totally forgot that this was happening and I only recently realized OSI approved it back in April.

Programming DevOps

Martin Fowler Today

When Software Meets Hardware

When software meets hardware, engineering principles remain largely the same, but the sensible defaults often change. Kuldeep Singh describes our experience modernizing a large IoT fleet of battery-powered sensors. Our familiar principles remained important; what changed was how we applied them. These practices emerged through experiments, failures, and operational experience—each challenge helping us replace a reasonable assumption with a more sensible default.

Hardware DevOps Iot

W. Jason Gilmore 2 days ago

Technical Due Diligence - Configuring a Secret GitHub Team

Add hidden users to your GitHub organization

DevOps Security Github

anders murphy 3 days ago

Lockstep to pipeline in almost one line

DevOps

Farid Zakaria 1 weeks ago

Nix wrote half of my debugger

A few days ago I wrote about Rewind VM , a deterministic VM where every run of a Nix build is a pure function of its inputs, thread schedule included! I have been using it to find, reproduce and solve numerous race conditions in Nix builds, but it’s been making me feel a little crazy . What is this superpower, and why is nobody else using it? The tool has quickly grown a source panel, stack frames, bookmarks, a “Compare” tab, a lot more gdb support, thread lanes, which show who held the CPU at every step, and “Check from here”, which helps find the exact step where a race condition happens. I went into each new feature thinking it would be a big code-lift but I kept running into the same thing: the hard part of each feature was already done, and Nix had done it. A debugger for instance, needs the exact inputs of the program, its debug symbols, its sources, the sources of every library under it, and a way for someone else to get all of that on their machine. That’s exactly what a derivation is, and Nix makes that easy. A classic simple example of a race condition is two threads depositing into one bank account. Each deposit reads the balance, writes a line to the ledger, then stores the balance plus the deposit. If one teller runs between the other’s read and store, it writes a stale balance over the other’s deposits. 💥 A teller that runs between the other’s read and store writes a stale balance over the other’s deposits. 1 teller 1 balance teller 2 read 150 read 150 store 200 store 150 + 10 160 teller 1's last deposit is gone Rewind’s VM has one CPU, so its first run passes too. runs the build again under perturbed schedules, each asking the guest kernel to reschedule at different steps, and narrows the first failure down to one step: Running this check on my laptop took 11 seconds. The two runs are the same machine, exit for exit, until step 3237, where only the failing one gets a reschedule. The argument to is a derivation and can be a flake reference. The beauty of Nix is that it knows the inputs of a derivation, so that’s all we need to make sure the run is reproducible. The derivation’s inputs are the source, the compiler, the libraries, the kernel, and the VM’s configuration realises the derivation’s inputs, packs the closure into a read-only erofs image, and boots the VM on it. The run’s id is a hash of its inputs, the way a store path is, and prints the command that makes it again: When the build succeeds, the guest reports each output’s NAR hash, and checks it against the host’s copy and against every binary cache Nix substitutes from, by fetching only the : This helps us validate that the build within the VM is the same as the build on my laptop, and that the VM’s run is reproducible by Nix. Debugging code is much simpler when you are looking at the source. A new panel in the Rewind app or in the terminal shows the source of the program at the playhead, and the stack frames that called it. The debugger needs to know where the sources are, and Nix gives us that for free too. nixpkgs builds the packages with , and the debug info is cached on cache.nixos.org as a output. We can leverage debuginfod to fetch the debug info and the sources by build ID, so we can see the source of any binary in the VM, including the Linux kernel! 😲 Sometimes though, the source is not enough, and we need to see the state of the program. opens gdb on a fork of the run at the playhead, with every thread of the process. The debugger can set breakpoints, watchpoints, and inspect memory, registers and variables. At step 3249, just before teller 2 gets the CPU back, we can watch the variable and continue until it changes. Nothing gdb does changes the recording, and we can rewind to the same step and fork again, or fork at any other step, and gdb will see the same state. If gdb is not enough, opens a shell in the VM at a step with any package from nixpkgs on its . It is one more closure packed into one more image. Rewind makes it very easy to compare two runs. The Compare tab in the app, or in the terminal, shows the last shared events of both runs, and the first event that differs. The Compare tab puts both runs’ events side by side from just before they part, with what differs marked. Teller 2 starts from 150 in the failing run and from 200 in the passing one: Rewind’s VM has one CPU, so at any step exactly one thread is running. A race is a question of order: which thread ran when. The trace does not answer that directly. It records what threads did, a write, an open, a fork, an exit, a signal, but not who was running in the gaps between those events. Rewind can fill in the gaps without recording anything new. Every run replays exactly, so it can walk any stretch of a run one step at a time and, at each step, ask the guest kernel which thread is on the CPU. In the failing run, teller 1 (Thread 140) is interrupted at step 3238, halfway through a deposit: it has read the balance but not stored it. Teller 2 (Thread 141) gets the CPU for a single step, 3239, long enough to read the balance, and then teller 1 gets it back and finishes. 2 Finding one bad interleaving raises the next question: how likely is it? Is the passing run the normal case, or did it get lucky? normally answers that by building the whole derivation again under many schedules. With , it starts from a run you already have instead, at any step you pick_. It forks the run there once per schedule, each fork taking a different order of threads from that step on, and counts how many end differently. Everything before the step stays exactly as it was. We can apply this technique to the bank example, starting from the step where the two runs diverged, 3,221. When we run 16 schedules from there, all 16 lose money 3 : We can do the same thing from the Rewind App with “Check from here” in the context menu of the playhead. It forks the run at the playhead and runs each fork under a different schedule, counting how many end differently. bookmarks the playhead’s step with a note. Bookmarks are kept with the run, and they travel in its export, so the person I hand the run to opens it with my notes on the timeline: Each one is a derivation in the flake, with one bug and one fix: Most of the hard work of a debugger is already done by Nix. Rewind adds a little more, but the rest is already there: the inputs, the sources, the debug symbols, and a way for someone else to get all of that on their machine. When you start with something hermetic like a Nix derivation, you can get a debugger for free. On my 16-core laptop, lost money in 396 of 1,000 runs. Pinned to a single core with , it lost money in none of 1,000: one core alone rarely switches threads in the middle of a deposit, which is why Rewind perturbs the schedule.  ↩ It is a little hard to see since the event from Teller 2 is tucked underneath the Orange bar.  ↩ Schedule 0 is the run itself, which passed. Every other line is a fork, and is failing because the balance came up short.  ↩ read 150 read 150 store 200 store 150 + 10 160 teller 1's last deposit is gone Rewind’s VM has one CPU, so its first run passes too. runs the build again under perturbed schedules, each asking the guest kernel to reschedule at different steps, and narrows the first failure down to one step: Running this check on my laptop took 11 seconds. The two runs are the same machine, exit for exit, until step 3237, where only the failing one gets a reschedule. Free thing one: the inputs The argument to is a derivation and can be a flake reference. The beauty of Nix is that it knows the inputs of a derivation, so that’s all we need to make sure the run is reproducible. The derivation’s inputs are the source, the compiler, the libraries, the kernel, and the VM’s configuration realises the derivation’s inputs, packs the closure into a read-only erofs image, and boots the VM on it. The run’s id is a hash of its inputs, the way a store path is, and prints the command that makes it again: When the build succeeds, the guest reports each output’s NAR hash, and checks it against the host’s copy and against every binary cache Nix substitutes from, by fetching only the : This helps us validate that the build within the VM is the same as the build on my laptop, and that the VM’s run is reproducible by Nix. Free thing two: every symbol and every source Debugging code is much simpler when you are looking at the source. A new panel in the Rewind app or in the terminal shows the source of the program at the playhead, and the stack frames that called it. The debugger needs to know where the sources are, and Nix gives us that for free too. nixpkgs builds the packages with , and the debug info is cached on cache.nixos.org as a output. We can leverage debuginfod to fetch the debug info and the sources by build ID, so we can see the source of any binary in the VM, including the Linux kernel! 😲 gdb, on a fork of any step Sometimes though, the source is not enough, and we need to see the state of the program. opens gdb on a fork of the run at the playhead, with every thread of the process. The debugger can set breakpoints, watchpoints, and inspect memory, registers and variables. At step 3249, just before teller 2 gets the CPU back, we can watch the variable and continue until it changes. Nothing gdb does changes the recording, and we can rewind to the same step and fork again, or fork at any other step, and gdb will see the same state. If gdb is not enough, opens a shell in the VM at a step with any package from nixpkgs on its . It is one more closure packed into one more image. Compare two runs Rewind makes it very easy to compare two runs. The Compare tab in the app, or in the terminal, shows the last shared events of both runs, and the first event that differs. The Compare tab puts both runs’ events side by side from just before they part, with what differs marked. Teller 2 starts from 150 in the failing run and from 200 in the passing one: Who had the CPU Rewind’s VM has one CPU, so at any step exactly one thread is running. A race is a question of order: which thread ran when. The trace does not answer that directly. It records what threads did, a write, an open, a fork, an exit, a signal, but not who was running in the gaps between those events. Rewind can fill in the gaps without recording anything new. Every run replays exactly, so it can walk any stretch of a run one step at a time and, at each step, ask the guest kernel which thread is on the CPU. In the failing run, teller 1 (Thread 140) is interrupted at step 3238, halfway through a deposit: it has read the balance but not stored it. Teller 2 (Thread 141) gets the CPU for a single step, 3239, long enough to read the balance, and then teller 1 gets it back and finishes. 2 Check from here Finding one bad interleaving raises the next question: how likely is it? Is the passing run the normal case, or did it get lucky? normally answers that by building the whole derivation again under many schedules. With , it starts from a run you already have instead, at any step you pick_. It forks the run there once per schedule, each fork taking a different order of threads from that step on, and counts how many end differently. Everything before the step stays exactly as it was. We can apply this technique to the bank example, starting from the step where the two runs diverged, 3,221. When we run 16 schedules from there, all 16 lose money 3 : We can do the same thing from the Rewind App with “Check from here” in the context menu of the playhead. It forks the run at the playhead and runs each fork under a different schedule, counting how many end differently. Bookmarks bookmarks the playhead’s step with a note. Bookmarks are kept with the run, and they travel in its export, so the person I hand the run to opens it with my notes on the timeline: Other examples to try Each one is a derivation in the flake, with one bug and one fix: : the deadlock from the last post. : the lost update above. : a SIGCHLD that arrives between a flag check and , so the parent sleeps until ’s ten second timeout kills it. : one process rewrites a config file in place while another rereads it. On many cores it fails nearly every time; on one CPU only some schedules land the reader between the truncate and the last write. On my 16-core laptop, lost money in 396 of 1,000 runs. Pinned to a single core with , it lost money in none of 1,000: one core alone rarely switches threads in the middle of a deposit, which is why Rewind perturbs the schedule.  ↩ It is a little hard to see since the event from Teller 2 is tucked underneath the Orange bar.  ↩ Schedule 0 is the run itself, which passed. Every other line is a fork, and is failing because the balance came up short.  ↩

Shell DevOps

Ruud van Asseldonk 1 weeks ago

Hosting without hyperscalers

How I host my blog has evolved throughout the years. In this post I explain how and why I moved my blog from American hyperscalers to European infrastructure.

Cloud DevOps

Chris Coyier 1 weeks ago

WordPress MCP

I’ve still got all my WordPress sites on Pressable and happily so; I think they do a good job! They do everything you’d expect a WordPress host to do. The things I specifically like are: On that last point, I appreciated this little one-clicker: With that on (and presumably because I also use Jetpack and thus WordPress.com?), I can auth an agent like Claude Code that supports MCPs and get it connected. Now I can, for instance, tell my agent to go make a draft blog post quick about something I’m thinking about, so I won’t forget about it. Just to be meta, I’ll have it look at this blog post and make sure I don’t say anything weird about elephants. And it easily caught me! I think I’m more likely to use this for very rote tasks like making sure my headers are Title Case, and things are formatted like I want them to be, matching a style guide. I think it pairs nicely with having a “SKILL.md” file for it, which can be home-base for a style guide like that. The skill could have stuff in it like “only ever touch draft blog posts”, “make sure blocks of code are labelled for what type of code they are”, “make sure images are self-hosted”, “make sure anything I’m linking to looks correct and the link is valid”. I friggin’ hate AI slop writing, so please know I have no interest in automating anything around that. But as someone who has authored and edited one billion blog posts, some intelligent automation of certain rote tasks is appealing. Customer support is good/fast, including free movement of sites Deploying from a GitHub repo is a first-class citizen Perfectly good DNS management No need for extra security, performance, or update plugins Things are generally with-the-grain of the WordPress ecosystem

Wordpress AI DevOps

Farid Zakaria 1 weeks ago

Rewind VM: a flaky build you only catch once

If a test fails on CI and nobody can reproduce it, did it really fail? 🧘 Nix gives me a build that is a function of its inputs via the extensional model. The same derivation, produces the same store path, and if I am lucky, the same bytes. What Nix does not give me is the same run . 1 A test suite with a race in it may pass on my laptop, fail once on CI, and when I rebuild it to look, it passes again. If you have experienced bugs like this, you know that it can be frustrating. You are effectively at times trying to find the needle in the haystack . I wrote recently that the Nix sandbox is a hidden input to a derivation. The same is true of the thread schedule. The order the kernel happens to run your processes in decides whether some builds pass, and nothing within the derivation records it. These are the hidden inputs to a build that can make it flaky. Are we left to hoping that we will find the needle in the haystack? Or is there a way to make the run a function of its inputs too? I built Rewind VM to make the schedule an input too. It runs a Nix build, a test suite or any Linux command inside a KVM virtual machine whose every run is a function of its inputs. The same inputs give the same run, at the same steps, every time . You can replay a failure, scrub through it, read any file as it was at any point, and fork it under a different thread interleaving. ✨ Confused? Yes it sounds like magic. The best way to explain it is with short demo. Let’s investigate the dining philosophers problem. The problem is that the five philosophers sit at a round table with a fork between each of them. A philosopher needs both forks beside them to eat. A philosopher may pick up one fork at a time. 2 f0 f1 f2 f3 f4 P0 eats P1 eats P2 eats P3 eats P4 eats five philosophers, five forks P0 picks up f0 P0 picks up f1 and eats P0 puts both down P2 picks up f2 P2 picks up f3 and eats P2 puts both down everyone reaches for the left fork each waits on a neighbor's fork nobody can ever eat: deadlock The philosophers take turns, until all five pick up their left fork at once. If all five pick up their left fork before any of them reaches for a right one, every fork is taken and every philosopher waits on a neighbor who is also waiting. Deadlock . 3 On my laptop 14 of 100 rebuilds deadlocked. 💣 builds the same derivation the way the Nix sandbox would, inside the deterministic VM. Oh darn, it passed! That means it will pass forever right? Not quite. The VM is deterministic, but the guest kernel’s scheduler is not. It can reschedule threads at different points in the program, and that can change the outcome. To find the other interleavings, runs the build again under perturbed schedules , we effectively ask the guest kernel to reschedule at different steps which causes a different sequence of events. runs one VM per core by default and stops after the first batch of schedules where the exit code differs. Note A failing schedule on its own is not that super helpful. Schedule 1 which had deadlocked likely perturbs every step from the start of the build to the end, and most of those perturbations have nothing to do with the deadlock. To help with this, narrows it: it shrinks the window of steps the schedule may perturb, first pulling in the end and then the start, reruns the build for each candidate window, and keeps the smallest one that still deadlocks. For our deadlock problem, it might be easier to see the last few lines of the log and see that all five philosophers have picked up their left fork and are waiting for the right one. We can also inspect the events, which are the same as the log but with timestamps and thread IDs. What if I’m not familiar with the VM? Can we look around? Yes! drops you into a shell inside the VM at any step, in a process’s working directory, with the build’s environment, while everything else in the VM stays stopped. We can use to bring gdb into that shell, and gdb can attach to the stuck process. 4 All five philosophers are on line 27, waiting for their second fork. f0 f1 f2 f3 f4 P0 P1 P2 P3 P4 left fork first every wait is on a held fork: a ring , and each work on a “throwaway” fork of the run at a step, so nothing they do changes the recording. You can use to do a “real” fork: it branches a run at a step under another schedule. A recording does not have to stay on the machine that made it. packs a run into a single file, with the VM’s kernel, its input image and the keyframes, so another machine with the same CPU vendor can replay it. 5 You are no longer beholden to a random flake on CI. Run the tests under on a CI machine with KVM, upload the failing run’s as a build artifact, and you can reproduce the bug perfectly on your machine. How do we fix the deadlock? We number the forks and always pick up the lower numbered one first. The last Philosopher now reaches for fork 0 before fork 4, so the waits can never cause a deadlock. f0 f1 f2 f3 f4 P0 P1 P2 P3 eats P4 lower numbered fork first P4 holds nothing, so f4 stays free We can then run to test the fix. Before the fix, 9 of the same 64 schedules deadlocked. Rewind VM includes some tutorials with more examples of using to find and fix bugs if you want to explore further. The VM has a single vCPU on stock KVM, so guest code runs on the real CPU at close to native speed. 6 What breaks determinism in a normal VM is everything that reaches the guest from outside its instruction stream: timer interrupts, clocks, random numbers, I/O completions, and any other event Rewind removes each of those or replaces it with a value it controls. Keeping account of every event and the step it happened at, it can replay the same run. If you squint, a run is a derivation. Its inputs are the kernel, the initramfs, a root filesystem (a Nix closure packed into a read-only image), the command, a seed and a schedule. Change any of them and you get a different run. Change none of them and you get the same one. The command is open source under the MIT license. 7 There is also a desktop app that gives a friendlier view of a recorded run. You can drag the playhead, view the build log, the process tree and the files at any event. “Open shell” and “Attach gdb” open a terminal pane on a fork at the playhead. I pointed Rewind at some tools I use every day to see what we can find. Each of these is reported upstream with a fix. On NixOS there is a module: If you have a test that fails on CI once a week, I would like to hear whether Rewind catches it and helped you debug it. Don’t just add a and paper over your concurrecy failures anymore, replay them. 🔁 A run is the sequence of events that happen in a process to produce a result. A build is a run of a derivation.  ↩ The world is a metaphor for the thread scheduler. Each philosopher is a thread and each fork a mutex.  ↩ The program stops making progress and never exits, so the check phase runs it under , which kills it and exits with status 124.  ↩ There is actually native support for gdb in the VM already. You can also use to bring in any other tool you want to use.  ↩ Without , writes only the run’s trace, its events and output, and leaves out the kernel, the input image and the keyframes. The file is much smaller and enough to read the run with and . It can’t be replayed or forked, though, so , and need the full export.  ↩ GNU hello build from nixpkgs takes 11.7 s in the VM vs. 14.3 s without it on the same laptop.  ↩ The guest kernel patch is GPL-2.0 alongside Linux.  ↩ The NixOS module can do it automatically for you and without the setting Rewind falls back to a coarser clock.  ↩ f0 f1 f2 f3 f4 P0 eats P1 eats P2 eats P3 eats P4 eats five philosophers, five forks P0 picks up f0 P0 picks up f1 and eats P0 puts both down P2 picks up f2 P2 picks up f3 and eats P2 puts both down everyone reaches for the left fork each waits on a neighbor's fork nobody can ever eat: deadlock The philosophers take turns, until all five pick up their left fork at once. If all five pick up their left fork before any of them reaches for a right one, every fork is taken and every philosopher waits on a neighbor who is also waiting. Deadlock . 3 On my laptop 14 of 100 rebuilds deadlocked. 💣 builds the same derivation the way the Nix sandbox would, inside the deterministic VM. Oh darn, it passed! That means it will pass forever right? Not quite. The VM is deterministic, but the guest kernel’s scheduler is not. It can reschedule threads at different points in the program, and that can change the outcome. To find the other interleavings, runs the build again under perturbed schedules , we effectively ask the guest kernel to reschedule at different steps which causes a different sequence of events. runs one VM per core by default and stops after the first batch of schedules where the exit code differs. Note A failing schedule on its own is not that super helpful. Schedule 1 which had deadlocked likely perturbs every step from the start of the build to the end, and most of those perturbations have nothing to do with the deadlock. To help with this, narrows it: it shrinks the window of steps the schedule may perturb, first pulling in the end and then the start, reruns the build for each candidate window, and keeps the smallest one that still deadlocks. For our deadlock problem, it might be easier to see the last few lines of the log and see that all five philosophers have picked up their left fork and are waiting for the right one. We can also inspect the events, which are the same as the log but with timestamps and thread IDs. What if I’m not familiar with the VM? Can we look around? Yes! drops you into a shell inside the VM at any step, in a process’s working directory, with the build’s environment, while everything else in the VM stays stopped. We can use to bring gdb into that shell, and gdb can attach to the stuck process. 4 All five philosophers are on line 27, waiting for their second fork. f0 f1 f2 f3 f4 P0 P1 P2 P3 P4 left fork first every wait is on a held fork: a ring , and each work on a “throwaway” fork of the run at a step, so nothing they do changes the recording. You can use to do a “real” fork: it branches a run at a step under another schedule. A recording does not have to stay on the machine that made it. packs a run into a single file, with the VM’s kernel, its input image and the keyframes, so another machine with the same CPU vendor can replay it. 5 You are no longer beholden to a random flake on CI. Run the tests under on a CI machine with KVM, upload the failing run’s as a build artifact, and you can reproduce the bug perfectly on your machine. How do we fix the deadlock? We number the forks and always pick up the lower numbered one first. The last Philosopher now reaches for fork 0 before fork 4, so the waits can never cause a deadlock. f0 f1 f2 f3 f4 P0 P1 P2 P3 eats P4 lower numbered fork first P4 holds nothing, so f4 stays free We can then run to test the fix. Before the fix, 9 of the same 64 schedules deadlocked. Rewind VM includes some tutorials with more examples of using to find and fix bugs if you want to explore further. What is Rewind VM? The VM has a single vCPU on stock KVM, so guest code runs on the real CPU at close to native speed. 6 What breaks determinism in a normal VM is everything that reaches the guest from outside its instruction stream: timer interrupts, clocks, random numbers, I/O completions, and any other event Rewind removes each of those or replaces it with a value it controls. Keeping account of every event and the step it happened at, it can replay the same run. If you squint, a run is a derivation. Its inputs are the kernel, the initramfs, a root filesystem (a Nix closure packed into a read-only image), the command, a seed and a schedule. Change any of them and you get a different run. Change none of them and you get the same one. The command is open source under the MIT license. 7 There is also a desktop app that gives a friendlier view of a recorded run. You can drag the playhead, view the build log, the process tree and the files at any event. “Open shell” and “Attach gdb” open a terminal pane on a fork at the playhead. Footguns One vCPU. Threads interleave but never run in parallel, so races that need two cores at once are out of reach. No preemption between system calls. A thread spinning on a flag without yielding stalls the VM. AMD needs once per boot for the exact clock. 8 A run replays only on the CPU vendor it was made on , AMD from Zen 2 on. No network besides loopback, and x86_64 Linux hosts with KVM only. Nix : dies of when exits between two writes. It never failed in 20,000 runs on my laptop and failed on the first run in Rewind. #16546 , fixed by #16547 ( case study ). Nix : several processes creating a new store at once fail with “database is busy”, as seen on Hydra. #15987 , fixed by #16554 . nixd : formatter output over 64 KiB hangs the language server. #899 , fixed by #900 . jujutsu : three tests fail about half the time on tmpfs, because operations ending in the same millisecond are ordered by a random id. #10306 , fixed by #10307 . A run is the sequence of events that happen in a process to produce a result. A build is a run of a derivation.  ↩ The world is a metaphor for the thread scheduler. Each philosopher is a thread and each fork a mutex.  ↩ The program stops making progress and never exits, so the check phase runs it under , which kills it and exits with status 124.  ↩ There is actually native support for gdb in the VM already. You can also use to bring in any other tool you want to use.  ↩ Without , writes only the run’s trace, its events and output, and leaves out the kernel, the input image and the keyframes. The file is much smaller and enough to read the run with and . It can’t be replayed or forked, though, so , and need the full export.  ↩ GNU hello build from nixpkgs takes 11.7 s in the VM vs. 14.3 s without it on the same laptop.  ↩ The guest kernel patch is GPL-2.0 alongside Linux.  ↩ The NixOS module can do it automatically for you and without the setting Rewind falls back to a coarser clock.  ↩

Programming DevOps Shell

Simon Willison 1 weeks ago

We're going to need default hard budget caps on pretty much everything

Here's a product feature which the world is going to need a whole lot more of over the coming months and years: default hard budget caps . I'm talking about the feature of pay-by-usage services and APIs that lets you say "after $X/month, cut this thing off and return errors". These need to be hard limits. Soft caps, "after $X/month, send me a warning email", will not cut it. Coding agents, and personal agents (coding agents wrapped in a less threatening UI), greatly reduce the friction of spinning up code that can do useful things. Sometimes those things cost money - calls to paid APIs, or hosted web applications, or systems that can bill for additional storage and compute. Nobody wants to wake up to an email sent at midnight warning about a budget limit and find that, while they slept, their rogue service had consumed several hundred (or several thousand) more dollars of usage. An argument against this is that businesses don't want their hosted applications to start throwing errors because some budget was exceeded. I expect that most businesses and individuals would prefer errors to a surprise $10,000+ bill. I think hard budget caps need to be the default. If someone wants to live dangerously they should be able to do that, but it needs to be on an opt-in basis. Have a nice, clear checkbox somewhere prominent: Remove the budget cap. My application will not be shut down if I exceed the configured budget limit, and I will be responsible for subsequent charges. The service I most want to see this from is AWS. I've heard plenty of stories from people who refuse to use AWS for personal projects out of (justified) fear that a runaway service might bankrupt them. I've also heard stories from people who didn't anticipate this and ended up seriously burned. ... and it turns out AWS finally launched spending limits a few weeks ago! From their announcement New AWS experience helps builders get started and ship faster on 16th September: When you're ready to upgrade to a paid plan, you can set a monthly spend limit for your project based on your usage patterns so that you stay within your budget. If a project's usage reaches its spend limit, your project is paused for that month. See also Create a spend limit in AWS Settings , though that page warns that "We're currently releasing our new experience to a limited number of customers." Here's hoping that hits general availability for existing accounts soon. Google Cloud launched a similar feature in July, called Spend Caps, which lets you "set a monthly financial cap on specific services within a project". Looks like this is becoming a trend! In an ideal world, our agents could help with this. It would be great if agents started biasing towards recommending providers with hard budget caps, and warning new and inexperienced builders against deploying applications using uncapped services that might get them into trouble. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options .

Cloud Finance DevOps

neilzone 1 weeks ago

Replacing our SIP doorbell with a plain, old-fashioned, doorbell

Just over six years ago, I installed a shiny Fanvil i10 SIP doorbell. It was little more than a SIP phone in a doorbell box (as the configuration options made clear), but I wanted something which would ring my phone and Sandra’s phone when someone rang it, wherever we were, and I wanted it to go as SIP/RTP, rather than using a proprietary app. So I installed it, popped it on its own VLAN (because I had to run an ethernet cable to the outside of the front door), configured it to talk to our PBX, and went with it. I didn’t bother with the video side of things. And, for those six years, it has been fine, ringing our phones whenever the doorbell was pressed. Recently, calls from the doorbell have been intermittent in their success in calling Sandra’s phone, whether terminating via SIP client, or routing a call to Sandra’s cellular network. And, sure, I could probably spend the time debugging it, but I was trying to solve a problem which didn’t really need solving: we just need a button for someone to press, and for it to result in a noise in the house, and in my office. We have a parcel box, which is very obvious and next to the door. Delivery drivers are great at dropping a parcel in there, and leaving without ringing the doorbell, which is absolutely fine. There’s usually an emailed delivery notification and, if we are that desperate, we can always open the box and check (although I did consider fitting a sensor the parcel box door, to trigger Home Assistant, but in the end I did not…). We are not - currently, anyway - in a position in which we need to know exactly who is at the day (or, rather, who they claim to be) before we answer the door. So perhaps I didn’t need a “smart” doorbell after all… A £18 Tecknet wireless doorbell has solved this just fine, and has meant that I can get rid of the ethernet cabling across the front of the house. It has no problem transmitting through the house, across the garden, and to the alarm box in my office and, if I want, I can change the alert mode from noise and light to just light. It claims to be IP65-rated, and in any case, it is under a porch - the Fanvil device was just fine there.

DevOps AI Hardware

マリウス 2 weeks ago

Updates 2026/Q3

This post includes personal updates and some open source project updates. もしもし、マリウスです, faxing at 9600 baud directly from Tokyo, Japan . After a short hop over to Hong Kong and Shenzhen , I’m back in the beautiful neighborhood of Shimokitazawa , where I’m spending almost the entire rest of the year until right before Xmas, when I’ll be heading back to Shenzhen. First things first, a warning: This is a very long post as it includes a metric ton of project updates. If you’re interested in the stuff that I’m building then this update is definitely for you. However, let’s start off with some infrastructure-related things. A few major updates happened in the past quarter with regard to my infrastructure. Probably the most important one is the migration from individual VPS instances to my own bare metal running a Proxmox “datacenter” . While I had been relatively satisfied with Vultr in the past, with an ever-increasing number of virtual server instances, for projects like MSG.TAXI , Hyperuplink , tty.fail , and everything related to this very website, my cloud service bill became overly expensive, all while contributions sadly decreased over the past months, making it unsustainable to run cloud infrastructure, especially to that extent. It just so happened that Vultr had more maintenance windows than usual over the past months, which prompted me to finally deal with the task of moving my VPS instances onto my own hardware. By migrating to bare metal, I was able to slash costs by approximately 30%, by discarding the resource safety buffer I had accounted for within each individual VPS, for a more dynamically allocating approach across individual instances. With rising hardware costs, however, one trade-off that I had to make concerns high availability. A full hardware failure on the current setup will result in all uncached services becoming unavailable for as long as the repairs take. Unless prices for hardware go down again, which is unlikely to happen , or contributions pick up again, which in the current market climate is equally unlikely, I won’t be building out the setup any further. As for some general stats about this specific website, over the past six months it generated at its peak over 270GB of traffic per month, which is considerable, given its compact size and its GTmetrix rating of (100% performance, 98% structure, 450ms LCP, 0ms TBT, 0.02 CLS), with the first contentful paint after 344ms, the speed index at 366ms, the time to interactive at 436ms, and an average page size (including images) of a few megabytes at most. Because I stopped running any form of analytics software, due to privacy reasons at first, but ultimately because of the realization that it has become pointless with all the robots LARPing as legitimate human traffic, I cannot tell you the number of visitors this website or any of the related projects have, and frankly I don’t even care. Not because I don’t care about actual humans being interested in the things I’m publishing, but because over the past months I have been receiving an increasing amount of feedback via e-mail, as well as welcoming a surprising number of new members to the community channel , so that I don’t need analytics software to show me that visitors appreciate the work I’m putting in and decide to stick around. Speaking of sticking around: The SimpleX group that existed next to the primary XMPP channel is no more. Neither the platform, nor the group itself developed in a favorable way over the past years, despite its almost 500 members. SimpleX turned from an interesting privacy platform into yet-another-Telegram-type messenger and is on the way to enshittification with VC investments from e.g. ACP, who is an investor in K2 Space (ELINT), OnScreen.ai (tracking), Zorus (employee/network monitoring), as well as Jack Dorsey. To make matters worse, at the end of September Evgeny Poberezkin reached out to me via e-mail with the following request: Thank you very much for this article: https://xn--gckvb8fzb.com/an-overview-of-privacy-focused-decentralized-instant-messengers/ (we shared it here: https://simplex.chat/ <redacted>). We launched equity crowdfunding on Wefunder in August ( https://simplex.chat/ <redacted>), to give users the opportunity to get a stake in SimpleX Chat and benefit from its growth - 150 people have already invested. Please help us spread the word - maybe you could write about it and some other recent news: the foundation, channels, public names, and supporter badges, which we launched recently ( https://simplex.chat/ <redacted>). It would be great to connect - SimpleX link is below - please send any questions! Thank you again and all the best Evgeny P.S. If you write something, could you please share with us before publishing so it complies with Regulation Crowdfunding rules? We cannot promote (and won’t be able to re-share with our community) if it mentions valuation, security type, how much is or left to raise (saying how many people invested is allowed), and how the funds will be used. We can mention perks we provide though. Thank you! – Evgeny Poberezkin SimpleX Chat, Founder Despite my article clearly stating, quote: Red flag: VC funded, specifically by Jack Dorsey , since Aug 14, 2024, hence I do not recommend it any longer. … Evgeny (or, more likely, his team that didn’t properly vet every organic site linking/mentioning SimpleX) sent me this fairly generic looking e-mail, to which I replied by pointing at the red flag . Long story short, the fact that SimpleX has now seemingly become so much of a product in the literal sense that they are trying to get approved organic content out for their funding campaign shows once more that it isn’t a suitable option for the privacy/decentralization community anymore. I have therefore deleted the SimpleX group, as well as removed all mentions of the messenger on this website, treating it the same way as I treat e.g. Telegram, WhatsApp or any other commercial platform. Anyway, a few days ago I shut down the last Vultr instance, an OpenBSD machine that had been serving this website for years. It outlived the rest of my cloud infrastructure by a couple of days. The clearnet version of this website had moved from the instance to a Bunny storage zone with a pull zone in front of it earlier in September, which happened due to the increased traffic by bots and maybe humans , and which means that there is no origin server at all any longer. The Tor and I2P sites still needed a machine of their own, since Bunny doesn’t offer a way to serve either of them, hence both are now served from a virtual machine on my own hardware. That machine has no public IP address, which neither the Tor daemon nor i2pd needs in order to publish a site. Unfortunately it does appear to make the I2P site less reliable, presumably because an I2P router that nobody can connect to depends on other routers to introduce it, and in a measurement about four hours after the move, 4 out of 8 requests to the eepsite succeeded, while all 8 requests to the onion service did. If the eepsite doesn’t load for you, simply try again, since a second attempt usually works. The clearnet side has a trade-off of its own, namely that this website now depends on a provider being there, where the VPS used to be able to serve the entire thing by itself if I ever took the CDN away. What makes me comfortable with it is that the build is still a folder of plain files that any web server can serve, hence going back would require not much more than a server, a workflow step and a few changes in the DNS, but not a rebuild. There’s a dedicated write-up on the whole publishing pipeline in the works, so I’m keeping it at this for now. It’s been almost three months since I got the Lenovo X1 Carbon Gen 14 Aura and switched from the StarBook Mk VI to it , and I haven’t regretted it so far. The laptop performs as well as I had expected it to while offering very decent battery life. At times I’m getting dangerously close to its 32GB RAM limit, which is something I had expected before purchasing it, but most of my workloads still have plenty of breathing room. Shortly after upgrading to the X1 Carbon I also decided to get an external (portable) 19" screen, and so I snapped up the Uperfect GR19BU. So far my experience with the display has been very decent, but I won’t get ahead of myself as I’m going to release a dedicated write-up/review on it soon. However, with several other new additions to my setup, like this display, the Mudi 7 , and the things below , it’ll probably make sense to also start preparing a follow-up on the travel desk setup from earlier this year. As usual with many of the posts on this site, however, it takes between one and three months from the first line to the finished and published post, so don’t count on it until the end of the year or maybe even early next year. After having had a few minor issues with the uni USB-C hub (8-in-1 with USB-C power), and slowly running out of USB ports, I decided that it was time to get a proper Thunderbolt 4 dock for my X1 Carbon , and what option would be better than the official Lenovo ThinkPad Thunderbolt 4 Smart Dock Gen 2 7500, for which I happened to find a pretty sweet deal. As a matter of fact the deal was so good that I didn’t realize at the time of buying that the dock comes with A) its own external 135 Watt power adapter and cannot be powered through a regular USB-C PD charger, which is sort of messing with my minimal yet productive travel desk setup , because I’m already carrying a PSU that weighs over 800g, and B) with cLoUd CoNnEcTiViTy . The dock itself is 590g, its PSU (plus power cord) another 350g, which amounts to 940g total added weight for only the dock. Clearly the dock isn’t meant to be a travel-friendly option, but it seems like no Thunderbolt dock really is considering the (even heavier) alternatives, like the CalDigit TS-4. However, with regular USB-C hubs not providing enough connectivity, performance and, most importantly, stability, it was time to try a different class of devices. Sadly the dock only lasted a single day before it stopped working and had a slight smell of burnt electronics to it. Hence I had almost three weeks of back and forth with Lenovo’s customer service in order to get it RMA’d/replaced, which was a frustrating experience to put it mildly. Then again, I don’t think the lackluster performance of Lenovo in the specific geographic region that I had to deal with them in is a fair representation of Lenovo’s global customer service, but very likely typical customer service across most companies in that region. Anyhow, I eventually received the replacement and I’m finally able to use it. The hardware is fairly decent, I haven’t encountered any odd messages in my log yet, and I finally have enough USB-A/-C ports to connect all sorts of peripherals, like my keyboard, my mouse, the portable monitor , and more. However, I’d wish that these docks would replace their HDMI and DisplayPort connectors with more USB-A/-C ports and maybe even a halfway decent sound card with various audio connectors (3.5mm amongst others). I never needed that many display outputs on a computer, let alone on a portable device. But I guess I’m the exception, as pretty much every dock manufacturer seems to follow this quadruple/sextuple/octuple display-setup trend. The only meh part of the dock seems to be the integrated Ethernet card/port, that appears to be a Realtek RTL8156B ( ), which is a bit of a potato. There are plenty of user reports online that describe all sorts of issues with this specific NIC under Linux. While it does offer 2.5G, matching my new switch perfectly, it does seem to come with a few issues with regard to reliability and actual performance. However, with the amount of USB ports on the dock I can connect one of the many 1G Ethernet adapters that I have and be done with it, in case the integrated NIC should ever give me a hard time. So far, however, it had been working without issues. In an ongoing effort to USB-C-ify all of my hardware I also pulled the trigger on a new portable switch, namely the Ubiquiti Flex Mini 2.5G. I’ve been using the blocky Netgear 1GbE switches forever, but I’ve always struggled to power them, as they come with a barrel connector and a dedicated PSU. While I did at some point get a USB-A-to-barrel-connector cable, judging from the power output of the PSU those switches are not intended to be run off of a USB-A port. It works, but it’s probably not the best idea. Short story long, I went for the Flex Mini primarily because it is powered via USB-C, so that I don’t have to carry another PSU or worry about damaging the device. As an added benefit, the switch now offers 2.5G links, which the Lenovo dock , as well as the Mudi 7 and the Slate 7 can benefit from. Ideally my portable NAS should also benefit from the higher speeds, but sadly that probably won’t happen with the current hardware. Speaking of which … The Ultra-Portable Data Center (v2) has had its second birthday and has held up extremely well, considering all the travel and, at times, incredibly dusty, humid, and/or hot environments that it went through. More importantly, though, even two years later there is still not a single comparable, commercial product on the market that would make me want to replace the existing system with it. Every piece of hardware that I’ve stumbled upon to date appears to come with major hard- or software headaches, be it the UnifyDrive UT2, the Beelink ME Mini, the CWWK x86 P6, or even the significantly larger/heavier AOOSTAR R7. The closest in terms of software freedom, hardware reliability and physical footprint to date appears to be the QNAP TBS-464, which is an ultra-thin and lightweight, portable 4-bay M.2 NVMe NAS introduced back in 2021. Because I had to further reduce the weight and size of the items that I carry with me on my travels, however, at the beginning of September I began working on the next iteration of my ultra-portable NAS, the UPDC v3. I haven’t yet found the time to pour the endeavor into a dedicated write-up, but I am happy to say that I’ve managed to reduce the build’s footprint from initially around 1.48 liters down to as little as 0.68 liters in volume. I won’t spoil too much, yet, but it’s fair to say that I’m relatively happy with the end result and that it allowed me to make the UPDC a fixed part of my carry-on luggage . Not much happened on the keyboard front this quarter, apart from a set of PBS Black on Black that I was gifted, and that has since been on my Kunai . It has become one of my favorite keycap sets and it is pretty much what I was hoping for . The PBS keycaps are however not as rough/textured as I was hoping them to be, which is a bit of a bummer. Anyway, I might post a dedicated long-form review in a few months when I’ll be able to better judge long-term use. I invested some time in pursuing my open source projects in the past quarter, hence there are a few updates to share. Several of them shared the same two things, which were a move to the SEGV-1.1 license , and a vanity import path under my own domain. As some of you might remember, I had an unpleasant encounter last year with one of the Alacritty developers, which prompted me to do something that was long overdue, namely switch to Ghostty. Along the way I implemented the exact feature I initially wanted for Alacritty and made my code public, on a dedicated branch in my fork of Ghostty. Over a year later, I’m still maintaining that patch. If you’re using my patch, make sure to update your local version. Neon Modem Overdrive , my BBS-style command line client, received a whole new backend this quarter. It can now connect to Hyperuplink, the bulletin board of the future, to which I’ll get in just a second . Neon Modem is the first dedicated client implementation for the relatively young and not-yet-well-thought-out REST API. Additionally, a shared prompt package now handles the server URL and credential input for that system as well as for Discourse and Lemmy, instead of each one keeping its own copy. I also tracked down a longstanding source of UI lag. The post-create and post-show windows were doing more work on every keystroke than they needed to, and reworking their handlers and rendering made the interface responsive again. A new release with all of this is available on tty.fail , with binaries on GitHub . Kopi , the command line coffee journal, had its two heaviest dependencies replaced. The SQLite driver moved from the -based to the pure-Go , so Kopi now builds without a C toolchain and ships as a fully static binary on every platform. The OCR helper, which reads the details off a photo of a coffee bag, moved from an Ollama-specific client to the official OpenAI Go SDK , so it can point at any OpenAI-compatible endpoint rather than only an Ollama instance. The Go module was renamed to a vanity import path under my own domain ( ), and the usual dependency, workflow and GoReleaser housekeeping came with it. On top of this, Kopi received the first contribution , which was applied to it via at the end of July. reader , the command line web page reader, had a productive quarter. The biggest change is the removal of the journalist dependency, whose crawling logic is now part of reader’s own package rather than pulled in from a separate module. As I don’t maintain journalist any longer it made sense to pull out the bits that I need for reader. Additionally, reader gained proxy support, so it honors the standard proxy environment variables (#23). It now also correctly handles the URL scheme when deciding whether a source is an HTTP resource (#35), and it got a fix for an -related issue (#37), which I worked out despite the reporter never supplying the example files I asked for. Note: Don’t be that guy . If you open an issue in any piece of software that you don’t pay for and that someone else dedicates their time to free of charge, consider upfront if it’s really something that you’re going to use, and if it’s an issue you’re willing to dive into and help fix. If either of these are a clear nope , then simply don’t report it at all. usbec , the USB Equipment Commander daemon that runs commands when USB devices are plugged in or removed, had its one core dependency reimplemented in-tree and enhanced. The module was dropped and its Linux netlink listener rebuilt inside usbec’s own package, so the daemon no longer depends on an outside module for that. The license moved from GPLv3 to the SEGV-1.1 license , the module was renamed to the vanity import path, and the release workflow and GoReleaser config were fixed along with it. In addition, the USB Equipment Commander is now a USB & Network Equipment Commander, with the introduction of NetworkManager (via D-Bus). This allows you to have usbec run commands whenever the network changes in any of the various ways it can change. My motivation was to run a script that queries my public IP and the inferred location via and shows it as a desktop notification whenever NetworkManager (re-)connects. zpoweralertd , my Zig rewrite and drop-in replacement of poweralertd, which is a UPower-powered power (.. power, POWER, POWERRRRR) alerter, has received a deep and thorough refresh. The biggest change is the C bindings, which I have replaced with handwritten s, just the way I had used them for the first time in ssh-askpass-zigtk , but more on that new tool below. The second big change is the Zig compatibility, which now requires at least 0.16, and which I have tested against the current master/future 0.17 release as well, to make sure that everything will continue working. With this, I also largely refactored the code, for which I initially took inspiration from poweralertd. The restructuring allowed me to clean up the codebase and find a handful of memory-related issues along the way. A new version of zpoweralertd has been released that contains these enhancements and, for the first time, it is available as pre-built binaries directly from the releases page. cexec , my small cached-exec wrapper that runs a command and caches its output for a given amount of time so that re-running it returns the stored output instead of executing it again, was rewritten from Go to Zig this quarter. The new version is a drop-in replacement for the previous one, but it changed a few things in the background, one of which is the storage. Previously cexec used BuntDB, which turned out to be a bad choice for a process that can theoretically run multiple times in parallel and access/write to the same database. The new version, instead, uses plain files that are scoped to the actual commands that are being executed. In addition, the new Zig version supports encryption for the cached runs, with an optional key from or the environment variable, which makes using cexec suitable even for output that contains sensitive data. On top of that, everything cexec does is now also available as a Zig module, so that the same caching can be used from another program without having to shell out to the cexec binary. New this quarter is , a GTK4 helper written in Zig . OpenSSH runs an askpass program to prompt for a passphrase or a yes/no confirmation when it has no controlling terminal. The reason it exists is my own issue with X11, which I explained here . The reference askpass, and most others, link GTK against X11, so they fail to build on my Gentoo system with the global USE flag. does not require X11 dependencies, so it builds and runs on an X11-less GTK4. It works on Linux and (hopefully) on the BSDs alike. Also new this quarter is Switchyard , a lightweight bridge that accepts e-mail over SMTP and forwards each message to XMPP. You might have read about it before in the dedicated post that I had published at the beginning of August, but if you haven’t I recommend going through it if a bridge for SMTP to XMPP sounds like something you could have a use for. For years I had a small shell script in my dotfiles that popped up a bemenu list whenever I clicked a link, so that I could decide which browser it opened in. This quarter I turned that idea into a proper GTK4 application written in Zig, namely Browser Select . It registers itself as a web browser, hence every link another application opens goes to it first, and a small popup then lists the browsers installed on the system for you to pick one with the mouse or the keyboard. Just like the script before, that used the Go version of cexec as a binary to cache the choice for a few seconds, Browser Select also uses the new Zig cexec library for remembering a choice for a while, so that clicking a burst of links doesn’t lead to repeated asking for a browser. The Flipper BUSY Bar review that I posted this quarter came with a little goodie: busybar.zig . It is a Zig 0.16 client library and command line tool for the Flipper BUSY Bar, implementing the device’s current OpenAPI specification as closely as possible, so that you can drive it over its HTTP API from Zig code or straight from the shell. It uses nothing but Zig’s library, which means it builds for every target Zig supports, including macOS and Windows. Darkwing Ducky is a BadUSB (“Rubber Ducky”) firmware for the PicoUSB, an RP2040-based board, built in Zig 0.16 with MicroZig . I’ve struggled to finish this for about ten months or so, primarily because implementing the required USB HID code that acts like an external keyboard wasn’t all that straightforward with MicroZig. Anyway, Darkwing Ducky announces itself to the host as an ordinary keyboard and then types out a predefined sequence, which can be useful for automating the setup of a machine that you have physical access to, or, well, for other things. While the PicoUSB comes with its own CircuitPython-based software, I wanted to try Zig on hardware, especially with MicroZig for a while and so this device was the perfect excuse to do so. If you’re curious, the repository has the details on the payload format and how to flash it. The internet bulletin board software that I had been writing about as ▓▓▓▓▓▓▓▓▓▓▓ in the previous updates has a name, and it’s out. Hyperuplink went public at the end of August, together with a dedicated post that covers what it is, why it exists and how to run it. If a JavaScript-free forum as a single binary sounds like your kind of thing, read that one first and come back here afterwards. Otherwise you might as well skip this part. Most of the quarter went into the bits and pieces that the board needed before anyone other than me could run it. Attachments and profile pictures with local or S3-compatible storage, board-wide settings and an administration UI for all of them, a search, soft-deletion for categories, forums and topics, and a manual that is embedded in the binary and available under Help -> Manual . On top of that came a whole set of themes and color schemes, as well as the packaging, which now covers Docker and Podman images, Quadlets, a Helm chart, a nixpkg with a NixOS module, an ebuild for Gentoo, and service files for systemd, OpenRC and rc.d. And if that wasn’t enough already there is the REST API, which is what Neon Modem connects to. If you’d like to have a look at Hyperuplink yourself without any commitment, there’s a demo instance that you can check out. You can find its login details on the official website . Account creation is disabled on the demo instance (for now). Go compiles down to a single static binary, which is why it is a great choice for Hyperuplink. However it has nothing along the lines of Django, Rails or Phoenix that takes care of the web parts, hence a good share of Hyperuplink was never about the bulletin board at all, but about routing, controllers, models, migrations, background jobs, translations and configuration. At the end of July I moved that share out of the repository and into Go on Glides , a web application framework for Go that includes everything needed to build database-backed web applications, on top of Fiber v3 and pgx , with embedded migrations, an asynchronous job queue, cron jobs, i18n, local and S3-compatible storage, and helpers that each project would otherwise have to reimplement. The reason for the split is that I’m building a second service on the same foundation, namely Maya , which meant that Hyperuplink’s internals had to become something that more than one program can use. Inca is the spiritual successor of addrb and caldr , my two earlier attempts at the same problem, this time as a single command line client for CalDAV and CardDAV that synchronizes calendars, tasks and contacts into a local database for offline use. It is built on top of the “framework” that I built and make use of in zeit , and therefore it shares a lot of zeit’s UI aesthetic. On the other end is Maya , a CalDAV and CardDAV server as a single Go binary with PostgreSQL underneath it, which (sadly) uses a patched hard-fork of go-webdav , and which exists because neither Radicale nor Baïkal ever made me happy. It’s the second program built on Glides . I’ve been dogfooding Maya myself since early September, and as of now it still isn’t publicly available because it’s not yet at a point that I’m comfortable releasing it, in particular due to its hard-fork of go-webdav, which was a requirement because the upstream project is sadly pretty much dead at this point. However, given the absolute mess that DAV protocols are, it’s not even surprising that the maintainer seemingly lost interest in this project, and that the ecosystem is so bad. Anyway, I have both, Inca and Maya, at a point that I can daily-drive them and, more importantly, that other clients (Android, iOS, desktop) can participate without too many hiccups along the way, but I’m not at all happy with the results for reasons that I will get into if I ever choose to release Maya. Also new this quarter, and admittedly not something one might call a modest undertaking , is Netrunner , a web browser built on WebKitGTK and GTK4, written in Zig . I’m calling it the hacker’s browser as it’s meant for people who prefer a lightweight browser over Chromium and Firefox and who care more about configurability and about built-in integrations than about endless features and add-ons. Netrunner deliberately leaves out a number of things that other browsers have, bookmarks being probably the most obvious one, because the people it is built for tend to have a system for those features in place already. Netrunner is configured through a single TOML file that holds the static settings as well as the runtime state, e.g. which sites are allowed to run JavaScript, play audio or send notifications. The session (i.e. open windows and their tabs) is also persisted in that one TOML file when the browser quits, and restored on the next start. Two other features that Netrunner comes with built-in are Onion addresses and Eepsites, which are native protocols for the browser. By default, whenever you open an URL or an link Netrunner opens a connection via Tor or i2pd and keeps it alive for as long as you browse sites within the Onion/I2P network. As soon as your browsing activity goes back to the clearnet, Netrunner automatically disconnects from Tor/I2P again. To the user these connects/disconnects happen completely transparently and every network gets a of its own, so a tab that follows a link onto another network moves over and keeps its back and forward history. In plain English this means that and URLs are indistinguishable from any other URL and you don’t need to do anything special to be able to open them. Obviously Netrunner is not a replacement for the official Tor Browser that the Tor Project releases and that is somewhat hardened to make sure you stay as anonymous and safe as possible. Instead, Netrunner’s Tor and I2P integrations are intended for the casual user who enjoys the freedom of browsing those networks as if they were part of the “normal internet”. Speaking about the normal internet, one thing that can’t be missed, especially these days, is content filtering. While WebKit sadly doesn’t support running a full-blown uBlock Origin, its own content filter engine is nevertheless a relatively decent option for basic ad blocking and annoyance filtering. As a matter of fact, Netrunner uses the same AdBlock and uBlock Origin lists that popular browser extensions use, but it does so by downloading and converting them into the WebKit content blocker format via adblock-webkit-convert.zig , one of two small Zig libraries that came out of this browser experiment. Sadly, the WebKit content blocker format does not support all the trickery that e.g. uBlock is able to do within its browser extension, which means that no, you won’t be able to avoid YouTube ads within Netrunner, at least for now. You will however still score above 62% on AdBlocker test sites , which is fairly okay. The other library that came out of this experiment is osdetect , which does what its name suggests: It detects with some amount of confidence what operating system a program is running on, and it gives you specifics like whether it’s e.g. a Debian or a Gentoo machine, or not even a Linux at all. I won’t tell what my main motivation for building this library was, but if I should ever get to the point that I have a 0.1 version of Netrunner ready for release you will be able to the source code for to find out. Which brings me to the release. Not only to that of Netrunner or Maya, but to the more general question of open source and how I intend to contribute to the ecosystem moving forward. Despite having a landing page online, Netrunner is a fun project that I don’t take too seriously and that I’m currently dogfooding, just like I do with Maya , to find out whether this is a project that I can imagine maintaining long-term, or whether it is one that will eventually outgrow my availability. After two decades of putting the software that I build for myself online, I have come to realize one or two things. The first thing is that “just putting it out there” won’t benefit the software and it certainly won’t benefit me. My main motivation for building all these things is my own curiosity about how stuff works and, ultimately, my own needs. I need a proper CalDAV/CardDAV server that is pleasant to administer/maintain and that can talk to all of the software that I’m using, hence I’m building Maya (and Inca) . I need a proper TUI client for Keebtalk and a few other forums, hence I’m building Neon Modem Overdrive . Speaking of which, I needed a forum and I was sick of the cruft that is phpBB or Discourse, so I built Hyperuplink . And it’s also me who is annoyed by the utterly insane bloat of modern web browsers, which is why I started building Netrunner. Publishing any of them inevitably changes the software from “what I need” to “what I need and what others seem to need” , and there is no guarantee, even if you go the extra mile to implement what others ask for, that those people will stick around or ever start contributing themselves. Zeit is one example where I experienced exactly that, multiple times, and every time it left a bitter taste in my mouth and, what’s worse, hundreds of lines of code that other people had requested but that nobody ended up using long term. At some point Zeit v0.x had become so bloated with other people’s needs that I rewrote it from scratch as the v1 release that it is today, with only the features that I believe make sense to support long term. The second thing that I realized is that “putting it out there” in hopes of finding like-minded people to collaborate with is a pipe dream. The open source “community” has been broken for a long time and, in most cases, was never much of a community to begin with, but rather the developer-equivalent of a one-man band . Not that there are no exceptions, however, my projects usually aren’t those exceptions. Kopi received its first contribution only this quarter, two years after I initially released it, and it was a one-time submission rather than something that materialized into a long-term collaboration. reader , in the very same quarter, collected a handful of issues, one of which I fixed without ever receiving any feedback on it whatsoever. And if you look through the few “Issues” sections on GitHub that I haven’t disabled yet, you’ll see that it’s the same story across the other projects as well, and you might understand why most of the repos lack the “Issues” tab these days. A single patch against two or three dozen issue reports is roughly the ratio that every single one of my projects has had over the past two decades, and I don’t believe that this is because my software is particularly unattractive to contributors, despite it being intentionally niche, but simply because writing a patch for someone else’s program is work, while filing a request is at most a two-minute annoyance. This is also why I, contrary to many other voices in the industry, do not regard opening an issue report as an actual contribution to a project. With all due respect, I don’t want your issue reports, keep them, especially the drive-by ones for something that you’re unlikely to end up using anyway. What a public repository attracts, hence, is an audience rather than a community , and many times that audience has a very short attention span. All of this eventually turns a maintainer into more of an unpaid support desk, which is not what I would like for myself. On top of that, LLMs have taken away the one argument that publishing open source software still had going for it. That argument wasn’t that a stranger would show up and maintain your software for you , but that someone out there might one day need the exact same thing badly enough to extend what you had already built, instead of writing it from scratch all by themselves. In 2026, however, that someone doesn’t need your repository any longer, because they’ll describe what they want to a model and get back something that compiles. Whether what comes out the other end is any good depends on the person directing the LLM and on the effort, or, dare I say, the money they’re willing to throw at it, rather than on the LLM itself. Code itself has become relatively cheap . As a matter of fact, if you want a specific feature implemented in any of my years-old tools, you might as well ask the machine to do it for you, so you can have what you need for the time being. And frankly, I’m quite happy about that! I’m happy that a random passer-by can get the feature that they thought they needed during their euphoric discovery phase of a tool they just found out about, mainly because I also know that the moment the honeymoon phase ends and the person loses interest in the tool, their interest in that feature will vanish along with it, and it would be left up to the maintainer to continue supporting it. Don’t get me wrong, I’m not arguing that collaboration never works, because examples like Linux, curl and PostgreSQL prove the opposite, but I am arguing that for small programs that a single person can hold in their head there are very few reasons for others to actively contribute, and even fewer in a world of automated code generation. Going back to where I started, I’m not sure yet whether I’ll ever publish any of the newer tools and programs that I’m building, like Maya or Netrunner, simply because I don’t see much of a reason to do so. Especially when, for projects like Netrunner, I know of far more popular examples that still died the moment their maintainer stopped investing time into them. And maybe that’s a good thing. Maybe these kinds of projects, just like the ones that I’m publishing, are simply not relevant enough in the grand scheme of things. Maybe it’s just Darwinism at work, and maybe that’s just how things should be after all. There he goes. One of God’s own prototypes. Some kind of high-powered mutant never even considered for mass production. Too weird to live, and too rare to die. – Raoul Duke

Go JavaScript Hardware DevOps Python

Jeff Geerling 2 weeks ago

I'm starting HomelabFest (in St. Louis, Sep 2027)

I've been working on my homelab for years, and I enjoy sharing what I learn through this blog and my YouTube channel. Over the past few years, I realized the homelab community (which isn't really a defined thing—that's okay) hasn't had its own gathering, the way makers, software engineers, embedded developers, and HPC professionals do. There are many events that have crossover into the homelab realm, but nothing focused on energy-efficient servers, privacy-respecting self-hosted software, keeping old hardware useful, or cable management and rackmounting on a budget.

DevOps Hardware Open Source

Farid Zakaria 2 weeks ago

Every package is already installed

tl;dr; omnibin is a FUSE filesystem that puts every binary nixpkgs ever shipped on your . Nothing is installed. Nothing needs building. 0 bytes on disk until something actually reads a file. 😈 It’s 2026, why am I still installing packages individually? 1 Why must I go through the ritual of adding a package to my , running or succumb to the hellscape of . Nix gives us the power of having packages installed side-by-side without conflict. Why do I have to pick which ones I want to install? Why can’t I just have them all? What if the machine just had all of them? That is over fifty thousand 2 top-level binaries available on my , from 2013 to 2026 built by Nixpkgs , available on-demand, without installing anything. This is the magic 🧙‍♂️ of Nix , but it’s not restricted to Nix. Everyone seems to still love Docker and OCI , why am I still picking which base image to use? Why can’t I just have them all? Is this the ultimate agent harness? It’s a container with everything in it, right from the start. Try it at fmzakari/omnibin . Of course, I cannot forget our NixOS friends. You no longer have to curate your or , you can just have them all. What is “package management” if every package is already installed? Turns out that Hydra writes a file next to every single narinfo on cache.nixos.org that describes the contents of the archive as JSON: That metadata turns out to be the perfect index for a FUSE filesystem that can lazily fetch the NARs from the cache and unpack them on-demand. 🤓 None of this would mean anything without nixpkgs-multiverse , which already resolves any in nixpkgs history to the store path Hydra built for it on cache.nixos.org . When you combine the two, you get a filesystem that can answer the question “where is ” and then fetch it from the cache and unpack it for you, all without ever having to install it. I crawled all of it the files in under twelve minutes. 🤯 Once you have that, the filesystem writes itself: That is CPython 3.6.2, from 2017. That downloaded nothing , it is answered from the pre-crawled index. Agents are “a thing”. Making them useful is a thing. Making them useful without installing anything is a thing. If your agent tried to and stat every single entry, it would have a really bad time. There are 881,933 binaries in the tree, and it would take a long time to stat them all. To help the agents out a bit, lists only the bare names, one per executable, each resolving to the newest package that provides it. The versioned forms all resolve, but they are not listed. For example, resolves to the latest Python 3, which is 3.14.6 at the time of writing, but resolves to the 2017 version. For everything else there is the index, which is sitting right there in the mount: That UX is a little rough, so you can also use the CLI to query the index: Lastly, there is a /omnibin/README.md whose entire job is to tell whatever is exploring the filesystem to stop exploring the filesystem and query the database instead. 🤖 At this point it should be obvious, but you pay for this on startup for the first access. The first run took 2.7 seconds to fetch the NARs and unpack them, the second run was instantaneous because the store paths were already present. Other than that? Not really, which is pretty amazing. For any long-lived machine, you would expect your to already be warmed up with the packages you need, so the first access penalty is not a big deal. I remember one of the first things that blew my mind and sold me on Nix, was seeing a demo by @burke on comma . The capability to test a package, at a single nixpkgs revision, without “installing it”; revolutionary! I believe this to be a spiritual successor and I hope to imbue others with the same sense of wonder and amazement that I felt back then as a beacon of the power of Nix. The repo is at github.com/fzakaria/omnibin . Please responsibly. Yes, I am a little inspired after watching DHH’s keynote at RailsConf 2026 . I feel the same way about package management.  ↩ There are actually 881,933 binaries in the tree, but only lists the latest version of each binary. The versioned forms are still available, but they are not listed.  ↩ Yes, I am a little inspired after watching DHH’s keynote at RailsConf 2026 . I feel the same way about package management.  ↩ There are actually 881,933 binaries in the tree, but only lists the latest version of each binary. The versioned forms are still available, but they are not listed.  ↩

DevOps Linux Open Source Python JSON

Entropic Thoughts 3 weeks ago

Shrinking a Fedora virtual disk

I have a weird setup for my laptop. In summary, I have been irrationally scared of running Fedora on bare, dual-GPU metal, so I run it in a virtual machine with Hyper-V on Windows as a hypervisor. The idea was to let Windows manage GPUs and power management stuff like hibernation, and then use Fedora as my main UI to the computer. This has worked surprisingly well, but that’s only because my expectations were low to begin with. The VM can barely access the hardware of the underlying machine: it can’t use the GPU, it can’t use the front-facing camera, Linux starts thrashing weirdly when the VM runs out of memory, among other quirks I’ve learned to work around. Thus, I’ve wanted to re-install on bare metal for a while, but! problem! Come on a journey with me. It’s got a hero with hubris, mistakes, tension, and a cliffhanger. (Continue reading the full article on the web.)

Programming DevOps Linux

alikhil 3 weeks ago

My MikroTik workaround for LaLiga Cloudflare blocks

Spain loves football. Spain also has plenty of people watching football on pirate streams. Those streams often sit behind Cloudflare, where one IP address can belong to a lot of completely unrelated websites. LaLiga spent a while trying to get Cloudflare to take the pirate streams down. That did not work out. Then LaLiga got a court order that lets Spanish ISPs block IP addresses during matches. So they block the IPs used by the streams, and legitimate sites behind Cloudflare go down with them. Message shown by a Spanish ISP when you try to load a website hosted on an IP address it has blocked. The result is annoying. A random site does not load. An API starts timing out. You can spend an hour debugging your own setup before noticing that there is a LaLiga match on. I used to fix it by enabling a rule on my MikroTik that sends Cloudflare traffic through a VPN. Every match, same small ritual. Recently I found ¿Hay ahora fútbol? , a site with one useful question: is football on right now, and is Cloudflare probably broken again? It also exposes its signal over DNS. That was enough to automate the whole thing. The router keeps a list of Cloudflare IP ranges. When Hayahora reports a large set of blocked IPs, it enables one disabled mangle rule. New connections from my LAN to Cloudflare then go through my existing VPN. When the DNS answer goes back to normal, the rule turns itself off. Only Cloudflare traffic takes the detour. The VPN is not my default route, and the rest of the network keeps using my normal ISP connection. My setup uses RouterOS 7 and an existing policy-based IPsec connection marked . If your VPN uses a different connection mark, replace below. Get the VPN working first; this post only adds the automatic switch. First, refresh a firewall address list named every day. I use Davie3’s list , but any maintained Cloudflare IP-list import will do. Then add this rule. It starts disabled on purpose. Finally, this script checks Hayahora every five minutes. More than ten DNS answers is my signal that a blocking event is active. The same threshold is used by the TRMNL LaLiga plugin . You might wonder why this does an HTTP request to Google DNS instead of using RouterOS’s native . returns only one record. I need the full answer set to keep the threshold, so one record does not send all Cloudflare traffic through the VPN by accident. This is a blunt workaround. While the rule is enabled, every new connection from my LAN to a Cloudflare IP goes through the VPN. That is fine for me during a match, but it sends all Cloudflare traffic through the VPN, not just traffic to the site that failed. The DNS data is an observation, not an official blocklist. And connections that already exist keep their old route until they reconnect. Still, it has saved me from one more round of “why is this website broken only tonight?” If football is on and the block is active, I am French automatically.

DevOps Linux Tutorial

Farid Zakaria 3 weeks ago

A build graph that rolls dice

We are right around the corner from NixCon 2026 . Another year where I sadly won’t be present. I looked at the lineup and saw quite a few talks on dynamic derivations, along with some recent other posts on the topic such as cargo-dyndrv which had me thinking about revisting the topic since my earlier posts on the subject. I wanted to better understand the implications of dynamic derivations and how they change the build graph. Originally, I was focused on how it could replace the lang2nix style of build graph generation, but I realized that the implications are much broader than that. The graph does not need to be known up front. It can be defined as the build progresses. 🧐 What do I mean? Traditionally, Nix requires you to ask it to build anything and it will tell you exactly the steps it will take ahead of time as the “derivations” that comprise the build graph. This is an important property of Nix. It is what makes it possible to reason about builds without running them by understanding what will be built. Many Nix tools rely on this property via . For instance, nix-diff tells you why two closures differ without building either . Why does dynamic derivations change this? In functional languages it’s very easy to get abstract and use fancy words like “applicative” and “monadic” to describe the difference between types of computation. The difference is subtle, but it is profound. Nix traditionally was an “applicative” build system. Dynamic derivations make it a “monadic” build system. The difference is that in an applicative build system, the entire build graph is known up front, while in a monadic build system, the graph can be defined as the build progresses. 1 One way to think about the difference is to look at the type signatures of the two operations that define them: (applicative) takes a value known ahead of time and returns the result . You can see the entire graph before you run it. (monadic) takes , a function , to return . You cannot see the entire graph before you run it. Applicative can be thought of writing the shopping list before you leave the house. You read the recipe, you write down every ingredient, you drive to the store once. The list is a function of the recipe and nothing else. Monadic is a recipe with a step that says taste it, and if it is too salty, go buy a potato . You cannot write that shopping list up front. Whether the potato is on it depends on the saltiness, and the saltiness does not exist until you have already done some of the cooking. Nix used to be solely the former, dynamic derivations add the latter. Okay, I guess I should have said “Nix is now monadic in the scheduler”. Nix has always had bind in the evaluator. We have been doing monadic builds for years. We call it import from derivation . on a derivation output is a bind in exactly the sense above: what to build next is a function of a value that does not exist yet. The Nix evaluator cannot produce the graph without that value, and the only way to get it is to stop evaluating and run a builder. Unfortunately, this has a lot of footguns, such as causing to take ten seconds , and why nixpkgs bans the technique outright . Whereas import from derivation is a bind in the evaluator, dynamic derivations now adds bind in the scheduler. The difference is that the scheduler can run builders in parallel, ship them to remote machines, and it can substitute their results from a cache. The evaluator cannot do any of that. Dynamic derivations do not add the bind. The bind was always there. What changes is which layer performs it. 🤓 Many of the examples of dynamic derivations have been focused on build graph generation via lang2nix tooling, but I wanted to explore the implications of dynamic derivations in a more general sense. In the true monadic sense, the build graph can be defined as the build progresses. The next step in the build graph can depend on the result of a previous step. Here is a really simple example, & , that rolls a die and either stops or continues the chain by adding a new derivation step to the build graph. The depth of the chain is random, and the result of the build is how deep we got. Only the leftmost box exists when you run . Everything to the right of it is written by a builder, while the build is already underway. The general idea of this derivation is: Either way the file copied into is a , so works the same on a chain that stopped and a chain that kept going. 2 We can see the build graph grow in the denominator as the build progresses. If we try to introspect the graph with or , we get nothing. Since the graph is not known yet, our tools cannot see it. Dynamic derivations introduce a second graph that exists only after the build, and the ecosystem has no way to introspect it yet. I ran the build 900 times and logged the length of the chain. Rolling a six-sided die is one of the most classic ways to simulate geometric decay and we can see the results in the histogram below. The mean is 6.18, median 4, longest 46. The math maths. Every dynamic derivations demo so far, mine included, has been a build system: MakeNix for C, NpmNix for node, cargo-dyndrv for Rust, nix-ninja for ninja. That is a reasonable place to start but it does not capture the full power of the primitive. What other ideas can we explore? Mario. In Super Mario Derivations the attribute path is the button sequence and I have to supply the press count. The dynamic version emulates until Mario dies. The stopping condition is data, discovered mid-build, and the run is however long it turns out to be. Searching a state space. Swap the die for a predicate and you have a search where each frontier node is a derivation. Model checking, puzzle solvers, a fuzzer that only expands inputs which found new coverage. States you reach twice collapse onto one store path, and they survive a reboot. Crawling. Fetch a page, parse the links, emit one derivation per link. The frontier is discovered rather than declared, the crawl resumes because the store remembers every page already fetched, and the dependency graph of the result is the link graph. What are the bounds of this primitive? I don’t know. The size of the graph now is only limited by whether Nix stops emitting successors. There is no depth limit for the store layer, no equivalent. My longest honest chain in the die-roll was 46 deep, but I had rigged examples that went to 500. How far can it go? I am so brainwashed to thinking about making a plan before I start for my build systems, that the idea of making it up as I go is a little scary. 😨 The paper Build Systems à la Carte is the defacto read on this topic.  ↩ We actually need a depth parameter to avoid infinite recursion and so that the store-path of the derivations are different since they are content-addressed.  ↩ roll a six and we write a derivation that echoes the depth. Ordinary, nothing dynamic about it. This terminates the build graph. roll anything else and we write a passthrough : a derivation whose only job is , where is . The paper Build Systems à la Carte is the defacto read on this topic.  ↩ We actually need a depth parameter to avoid infinite recursion and so that the store-path of the derivations are different since they are content-addressed.  ↩

Linux Rust DevOps Programming

Massively Parallel Procrastination 3 weeks ago

Superpowers 6.4

I'm pleased to announce the release of Superpowers 6.4. This release adds new support for three coding agents: Meta's Muse, OpenCode 2.0 and Qwen Code. Every new model behaves just a bit differently than the ones before it. As the frontier models get more capable, they also develop new quirks. A lot of the work in this release is meant to help the latest models use Superpowers effectively. We've done some work on and to help steer your agents away from skipping steps or doing significant unrequested work. Superpowers offers two plan execution styles. One is "Subagent-Driven Development", which uses the primary agent session as an orchestrator, dispatching all implementation and test tasks to right-sized subagents with limited context and limited mandates. The other is "Native (inline) Execution", which dates to the earliest days of Superpowers. It was originally intended for a human-in-the-loop flow where you acted as a go-between for two agentic sessions. As the frontier models have improved, they've gotten better at executing work inline without getting distracted or losing the plot. Superpowers 6.4 reinvents Native Execution (the skill) for today's more capable models. We've been running extensive evals comparing Subagent-Driven Development, Native Execution and "bare metal" implementations with Claude Code. With a frontier model, Native Execution is about twice as fast and half as expensive as Subagent-Driven Development. Building without Superpowers gets you results faster and cheaper, but in our testing, even with Astra and Fable, builds without Superpowers implementation skills are significantly buggier than builds using Superpowers. We are, of course, working hard to make Superpowers faster and cheaper to use. We're currently testing out some fairly big changes to the planning skills that we hope to debut soon. 6.4 also introduces a new skill, which you can use when a session has done something you didn't expect. It knows how to dig into your coding agent's history to understand what Superpowers was doing and can help you put together a good bug report. We've done a little bit of work to tune the code reviewer prompts to apply a bit more "what would a reasonable person think?" judgment. This should help overzealous models calm down and converge rather than work themselves into aggressive review loops. You can read the full release notes at https://github.com/obra/superpowers/blob/main/RELEASE-NOTES.md If you install Superpowers from the Superpowers Marketplace ( https://github.com/obra/superpowers-marketplace ) or from OpenAI Codex, 6.4 is available in your coding agent today. Anthropic and other providers update their plugin marketplaces on their own schedules.

AI DevOps Go Programming

Aran Wilkinson 4 weeks ago

Software factories

When I first came across the term software factory in discussions about coding agents, I assumed it meant writing a task and handing it to an agent. The agent would gather the context, write the code, run the tests, open a pull request, and perhaps even deploy it. It sounded like a new, agentic way of building software. Then I saw Dex Horthy’s diagram , and it changed how I understood the term. I realised I had been a part of software factories for years. The task tracker, repository, pull request, CI system, deployment process, monitoring, incident response, and users were already connected. A factory was not a new thing that used agents. It was the system I was already using to turn a problem into a change in a running product. An agent changes one part of that system. It may take on more of the implementation, testing, investigation, or operational work, but it does not create the queues, hand-offs, or feedback loops around that work. Those were already there. I just had not been calling them a software factory. I have also been thinking about this in practical terms. What would it take to build a small software factory for my own projects, with agents helping me prototype ideas, build them, maintain them, and investigate problems once they are running? Looking at the whole system seems more useful than starting with a particular agent or tool. Reading Donella Meadows’s Thinking in Systems helped me make sense of this. One lesson I took from it is that a system’s behaviour comes from how its parts interact, not from the parts themselves, and that pushing harder on one part rarely fixes the system. CI is just one machine in the factory. The factory is the whole system: people and components connected by flows of work and information, operating towards some purpose. Here is the diagram that changed my view of it. It shows the software factory around implementation: users, product managers, engineers, pull requests, tests, deployments, monitoring, incidents, and feature requests. Dex Horthy, “Why Software Factories Fail” . What I find useful about the diagram is that it does not stop at deployment. It shows how information from users and production feeds back into what gets built next. Before seeing Dex’s diagram, I would probably have described software delivery as a series of separate pipelines. A product decision became a task, the task became code, the code passed its checks, and the change was deployed. That work would lead to the next task or another change, but I thought of those as separate pieces of work. The diagram gave me a different way to look at something I already understood. It made the connection explicit. What happens after deployment feeds back into the next decision. I had always been part of that process. I just had not thought of it as a loop, or seen all of its parts as one software factory. The first loop is the delivery loop: a decision becomes a change, the change is built, reviewed, tested, and deployed. The second starts with the product rather than the code. Users, monitoring, incidents, and feature requests produce information that feeds back into the next decision. I sketched the two connected loops together as a single flow below. Production is not the end of the process. It is one of the places where we find out what the change actually did. These loops meet when somebody decides what to build next. The delivery loop asks, “Can we change the software safely?” The product loop asks, “Is this the change we should make?” Those are separate questions. A team can get very good at shipping the wrong thing. It can also understand the product problem clearly and still be unable to deliver a safe change without a long queue of manual work. Seeing the work as two loops changed what I thought the factory was producing. The obvious output is a deployable artefact: a change to the software that can be released to users. But the factory also produces information. That information comes from different parts of the factory. Tests tell us about behaviours we thought to check. Static analysis finds certain classes of mistake. A review can expose a confusing design or a missing operational detail. Monitoring tells us what happened after deployment. Users tell us whether the change was useful in the first place. None of these signals answers the whole question. A green build does not mean that users are happy. A successful deployment does not mean that the system is healthy. A passing unit test does not mean that the change belongs in the product. That was the part I had missed when I thought of a software factory as an agent processing a task. I was focused on the change it produced, rather than the information the wider system produced about that change. The factory is useful when those signals make it back to the people deciding what to do next. That was my original mistake when I thought software factories were mainly about agents. I was thinking about how an agent could produce a change, but not about how the wider system would tell it whether the change was useful, safe, or worth keeping. The question is not simply how much code an agent can write. It is what happens to the rest of the factory when an agent takes on part of the work. Dex’s agentic diagram helped me see that the change may be local. An agent can replace or expand the implementation step, but the rest of the factory still has to review, test, deploy, and observe the change. Dex Horthy, “Why Software Factories Fail” . That seems obvious when written down, but it is easy to forget when an agent produces a working pull request quickly. The pull request is only where the change enters the rest of the factory. It is not the end of the work. A diagram published by The Pragmatic Engineer illustrates a more elaborate version, based on interviews about OpenAI’s internal system: The OpenAI example pushes further. It suggests that agents can remain involved after the merge, gathering production context, watching for regressions, and helping with incidents. I find that interesting, but I do not see it as a reference architecture to copy. It raises a more practical question for me: which parts of that loop would actually help a small project, and which would only add another layer of machinery? This is reporting about one company’s internal setup, not a reference architecture for every team. Here is a generic version of that flow: In the generic flow, an agent gathers context from the code, documentation, and internal tools, writes a change, and keeps fixing issues until the build and the checks pass. Specialist review agents inspect the result, and a risk classification decides whether it follows an automated path or needs human review. In the system described by The Pragmatic Engineer, the loop continues after the merge. An agent helps roll the change out to production, watches the relevant signals, and can create monitoring dashboards for the change. A performance system looks for regressions and proposes fixes. An incident-response bot gathers context and suggests mitigations when something goes wrong. Dex’s response is to turn the lights back on , not to abandon automation. If agents make implementation much faster, review becomes the next bottleneck. Removing human review can increase throughput for a while, but it also removes one of the few places where somebody might notice that a change is making the system harder to understand or modify. In the second part of his series, he argues for moving human judgement earlier in the process: agree the product requirements, system architecture, and shape of the program before asking an agent to implement it, then work in small vertical slices that can be checked as they are built. That is a much more ambitious factory than “ask an agent to write some code”. It gives agents access to the context around the code, lets them act on feedback from the build and review systems, and keeps them involved after deployment. If an agent can reach the wider factory only through a human at a terminal, the human becomes the integration layer. They have to notice a failed CI job, collect the logs or metrics, decide what is relevant, and bring that context back to the agent. This makes each loop slower and means the agent sees only the part of the problem the human thought to provide. An agent connected to the factory can query those systems itself: inspecting CI results, reading production logs, comparing metrics, and gathering the context it needs before proposing what to do next. That connection does not require unrestricted access or permission to act autonomously. Access can be read-only and limited to the systems relevant to the task, with human approval still required for consequential actions. An agentic software factory does not need to be fully automated. A person might start an agent manually, mark a feature request as ready for an agent to pick up, or route a failed build, production regression, or incident directly to one. How an agent gets kicked off, manually or automatically, is a design choice, not part of the definition. What matters is that the agent takes part in the factory’s flows of work and information: it can gather the relevant context, act within its permissions, and respond to the results of its work rather than merely suggest the next line of code. For businesses already experimenting with coding agents, I think the next step is to optimise the software factory around them. It is not simply a matter of giving every developer an agent. Businesses need to look at their existing software factory and decide how it should work when agents become participants: which queues they can take work from, what context and tools they can access, how their output is checked, and where people need to approve an action. Most software delivery systems have been built around human teams, so adding agents means reconsidering those interfaces and hand-offs rather than dropping a coding tool into an otherwise unchanged process. Agency does not mean removing people from the system either. A human still needs to be in the loop at the points where judgement or accountability matters. Depending on the risk, that might mean deciding when to start the agent, reviewing its proposed change, approving an action it wants to take, or approving its deployment. The boundary will differ between teams and between changes; the important part is that it is deliberate rather than whatever the tooling happens to allow. Deliberate boundaries also make the human responsibilities easier to see. People still need to define useful outcomes, make the relevant context available, decide what counts as low risk, ensure the tests describe acceptable behaviour, choose the signals that matter, and decide where an agent must stop and ask for help. Automation changes the boundary of the work; it does not remove judgement from the system. A change marked as low risk can still be wrong, so the classification and the automated path need their own checks. The most important detail in the diagram is the feedback path from production back to the coding agent. An agent that can write code but cannot observe its consequences is just a faster way to create a queue of changes. An agent that can see failures, understand their context, and propose a fix starts to look more like a participant in the factory. The agent does not need to remain active or control every stage. The factory could route the results of a failed build, deployment, or production alert to an agent with the relevant context. What matters is whether that path exists. Without it, the wider feedback loop is available to the human but closed to the agent. Even when those feedback paths exist, there is another limit: an agent can only respond to signals the factory can produce. This is part of Dex Horthy’s wider argument. Tests and production metrics can quickly reveal certain failures, but maintainability has a much slower feedback loop. A design that makes the next six changes harder may not reveal itself until months later. This is not unique to agents, but faster implementation allows more of those design decisions to accumulate before their consequences become visible. A factory can have plenty of automation and still be slow. If every change waits several days for a review, producing the initial change faster will not fix the queue. If deployments are frequent but nobody looks at production behaviour, the factory is producing changes without learning from them. If feature requests arrive faster than the team can understand and prioritise them, adding more implementation capacity will not fix that decision queue. The slowest part of the system is often not the compiler or the test suite. It is a hand-off: This is why the arrows in the diagram are as important as the boxes. A box describes an activity. An arrow describes how work and information move between activities. There is a risk in taking the factory analogy too literally. Software is not a pile of identical widgets. The work includes discovery, judgement, and learning. The thing being built often changes shape while we are building it. Optimising only for throughput can make a team worse at its actual job. We might measure the number of pull requests merged, deployments made, or tickets closed because those numbers are easy to count. That does not tell us whether the product is becoming more useful or whether the system is becoming easier to change. The useful part of the analogy is not “make engineers work like machines”. It is “look at the complete system”. Where does work enter? Where does it wait? Which checks happen automatically? Which decisions need a person? How quickly does information from production get back to the team? The factory should make good changes easier to make and bad changes easier to notice. It should not turn every decision into a race for a higher throughput number. I have been thinking about what a small software factory might look like for my own projects. I want to experiment with agents not only as a way to implement tasks, but as part of a system that helps me with the same four jobs I mentioned at the start: prototyping ideas, building them, keeping them maintained, and investigating failures. Headcode is an obvious place to start. In Splitting Headcode into a polyrepo workspace , I described separating it into smaller repositories with clearer boundaries, repository-local checks, and a separate place for agent work plans. I made those changes partly to give agents bounded contexts. That workspace is not yet a software factory, but it gives me a practical system in which to explore how work moves between agents, repositories, checks, deployments, and production feedback. I do not mean a lights-off factory that can be left to ship changes on its own. The interesting part is deciding which loops I can automate safely, where I still need to make decisions, and what context and signals an agent would need to do useful work. To start with, I will keep most of the agent use manual: running an agent when I choose to, rather than wiring it to a trigger. Starting that way keeps costs predictable while I work out which loops are worth automating, and it keeps me close to what the agents are actually doing before I let them run on their own. That raises more useful questions than simply choosing an agent: Adding another tool is not the same as improving the factory. A new tool only helps if it removes a real delay, improves a useful signal, or makes a risky decision safer. I have not built that system yet. In follow-up posts, I want to explore what an agentic software factory looks like in practice, starting with personal projects where I can experiment freely and document what I learn. I am also interested in seeing how these ideas could apply to the systems I work with professionally, where I have permission to experiment. Thinking of it as a factory gives me a better place to start: not “which agent should I use?”, but “which loop am I trying to improve?” Did the change solve the problem? Did it create a new failure mode? Did users understand it? Did it make the system harder to operate? What should we change next? an engineer waits for somebody who understands the product decision; a pull request waits for a reviewer who is also busy building something else; a deployment waits for somebody to perform a manual check; an incident waits for information that is spread across several systems; a user report waits for somebody to turn it into a reproducible problem. Where does work enter the factory: a Linear ticket, a failed build, a production alert, or a note to myself? How do I describe the outcome rather than only the implementation task? What code, documentation, and operational context does the agent need? Which checks provide meaningful feedback rather than merely producing a green build? Which changes are safe to automate, and where should the agent stop? How does information from maintenance, failures, and actual use make it back into the next change?

DevOps Ci Cd

Dangling Pointers 1 months ago

Presto: A Match-Action TCP Stack for the Terabit Era

Presto: A Match-Action TCP Stack for the Terabit Era Rajath Shashidhara, Antoine Kaufmann, and Simon Peter SIGCOMM'26 This paper presents Presto, a Goldilocks implementation of the TCP protocol. It is efficient and yet does not require fixed-function TCP-specific networking hardware. The paper is a tour-de-force in the way it isolates the specific problems that make TCP processing hard to pipeline, and describing clever solutions to these problems. The Reconfigurable Match-Action Table architecture one specific flavor of programmable network accelerator. Here are two previous paper summaries that reference the RMT architecture. At its core, the RMT architecture is a feed-forward pipeline through which network packets flow. Each pipeline stage has a content addressable memory, and a limited amount of compute. The hard part about mapping an application to the RMT architecture is that there is very limited communication between pipeline stages. Network packets flow forward through the pipeline. The one escape hatch is the pipeline can decide to recirculate a packet, which can cause information to be sent from the tail of the pipeline to the front. This paper which, implements a key-value store with RMT leans heavily on this recirculation. Mapping the various steps in TCP protocol handling onto the RMT architecture requires distributing the state associated with a connection across the RMT pipeline. The size of per-connection state at each pipeline stage is fixed. The hardest TCP feature to map onto RMT is segment reassembly. Segment reassembly is the task of tracking and handling received segments (i.e., packets), which may arrive out of order. The receive side of a TCP connection must track the start and end of a window of packets that may be accepted. For example, if the packet with sequence number 4 has been processed, and the window size is 10, then the sender is free to send packets [5, 6, …, 15]. The paper describes three segment reassembly designs, I’ll illustrate one (OOO-1) here. Fig. 4 illustrates a continuous stream of packets with monotonically increasing sequence numbers. is the lowest sequence number of packets that have not yet been received (i.e., the start of the TCP window). defines the end of the TCP window. and define a contiguous set of packets that have been received and are in the TCP window. Note that this design happily accepts these packets. Source: https://dl.acm.org/doi/10.1145/3789240.3829111 Fig. 3 illustrates the 4 pipeline stages that implement TCP receive window tracking. Note that each of the 4 state variables described above is tracked in a different pipeline stage. For example, say that and , and . This means that the next expected sequence number is 4, and no packets in the TCP window have arrived. Say that packet 6 arrives next. Presto will accept this packet and set and . If packet 5 arrives next, then will be set to 5. Finally, when packet 4 arrives, will be set to 4. At this moment (ooo-head-1 is equal to next-seq), the packets 4, 5, and 6 can be sent down the pipeline. This is accomplished with recirculation: a dummy packet is injected into the pipeline which flows through all stages and updates state variables as expected. Source: https://dl.acm.org/doi/10.1145/3789240.3829111 Results Fig. 9 shows throughput vs latency curves for Presto and TAS (a software TCP stack based on kernel bypass): Source: https://dl.acm.org/doi/10.1145/3789240.3829111 Fig. 10 shows power consumption: Source: https://dl.acm.org/doi/10.1145/3789240.3829111 Dangling Pointers It is a shame that Intel has discontinued the Tofino chips. The literature shows that the RMT architecture is flexible enough to efficiently implement a wide range of applications (e.g., key-value store, TCP protocol acceleration). Thanks for reading Dangling Pointers! Subscribe for free to receive new posts.

Web DevOps Hardware