GreatReads - Blog Aggregator · Phoenix Framework

Testing

C++

0 views

Farid Zakaria 1 weeks ago

Nix wrote half of my debugger

A few days ago I wrote about Rewind VM , a deterministic VM where every run of a Nix build is a pure function of its inputs, thread schedule included! I have been using it to find, reproduce and solve numerous race conditions in Nix builds, but it’s been making me feel a little crazy . What is this superpower, and why is nobody else using it? The tool has quickly grown a source panel, stack frames, bookmarks, a “Compare” tab, a lot more gdb support, thread lanes, which show who held the CPU at every step, and “Check from here”, which helps find the exact step where a race condition happens. I went into each new feature thinking it would be a big code-lift but I kept running into the same thing: the hard part of each feature was already done, and Nix had done it. A debugger for instance, needs the exact inputs of the program, its debug symbols, its sources, the sources of every library under it, and a way for someone else to get all of that on their machine. That’s exactly what a derivation is, and Nix makes that easy. A classic simple example of a race condition is two threads depositing into one bank account. Each deposit reads the balance, writes a line to the ledger, then stores the balance plus the deposit. If one teller runs between the other’s read and store, it writes a stale balance over the other’s deposits. 💥 A teller that runs between the other’s read and store writes a stale balance over the other’s deposits. 1 teller 1 balance teller 2 read 150 read 150 store 200 store 150 + 10 160 teller 1's last deposit is gone Rewind’s VM has one CPU, so its first run passes too. runs the build again under perturbed schedules, each asking the guest kernel to reschedule at different steps, and narrows the first failure down to one step: Running this check on my laptop took 11 seconds. The two runs are the same machine, exit for exit, until step 3237, where only the failing one gets a reschedule. The argument to is a derivation and can be a flake reference. The beauty of Nix is that it knows the inputs of a derivation, so that’s all we need to make sure the run is reproducible. The derivation’s inputs are the source, the compiler, the libraries, the kernel, and the VM’s configuration realises the derivation’s inputs, packs the closure into a read-only erofs image, and boots the VM on it. The run’s id is a hash of its inputs, the way a store path is, and prints the command that makes it again: When the build succeeds, the guest reports each output’s NAR hash, and checks it against the host’s copy and against every binary cache Nix substitutes from, by fetching only the : This helps us validate that the build within the VM is the same as the build on my laptop, and that the VM’s run is reproducible by Nix. Debugging code is much simpler when you are looking at the source. A new panel in the Rewind app or in the terminal shows the source of the program at the playhead, and the stack frames that called it. The debugger needs to know where the sources are, and Nix gives us that for free too. nixpkgs builds the packages with , and the debug info is cached on cache.nixos.org as a output. We can leverage debuginfod to fetch the debug info and the sources by build ID, so we can see the source of any binary in the VM, including the Linux kernel! 😲 Sometimes though, the source is not enough, and we need to see the state of the program. opens gdb on a fork of the run at the playhead, with every thread of the process. The debugger can set breakpoints, watchpoints, and inspect memory, registers and variables. At step 3249, just before teller 2 gets the CPU back, we can watch the variable and continue until it changes. Nothing gdb does changes the recording, and we can rewind to the same step and fork again, or fork at any other step, and gdb will see the same state. If gdb is not enough, opens a shell in the VM at a step with any package from nixpkgs on its . It is one more closure packed into one more image. Rewind makes it very easy to compare two runs. The Compare tab in the app, or in the terminal, shows the last shared events of both runs, and the first event that differs. The Compare tab puts both runs’ events side by side from just before they part, with what differs marked. Teller 2 starts from 150 in the failing run and from 200 in the passing one: Rewind’s VM has one CPU, so at any step exactly one thread is running. A race is a question of order: which thread ran when. The trace does not answer that directly. It records what threads did, a write, an open, a fork, an exit, a signal, but not who was running in the gaps between those events. Rewind can fill in the gaps without recording anything new. Every run replays exactly, so it can walk any stretch of a run one step at a time and, at each step, ask the guest kernel which thread is on the CPU. In the failing run, teller 1 (Thread 140) is interrupted at step 3238, halfway through a deposit: it has read the balance but not stored it. Teller 2 (Thread 141) gets the CPU for a single step, 3239, long enough to read the balance, and then teller 1 gets it back and finishes. 2 Finding one bad interleaving raises the next question: how likely is it? Is the passing run the normal case, or did it get lucky? normally answers that by building the whole derivation again under many schedules. With , it starts from a run you already have instead, at any step you pick_. It forks the run there once per schedule, each fork taking a different order of threads from that step on, and counts how many end differently. Everything before the step stays exactly as it was. We can apply this technique to the bank example, starting from the step where the two runs diverged, 3,221. When we run 16 schedules from there, all 16 lose money 3 : We can do the same thing from the Rewind App with “Check from here” in the context menu of the playhead. It forks the run at the playhead and runs each fork under a different schedule, counting how many end differently. bookmarks the playhead’s step with a note. Bookmarks are kept with the run, and they travel in its export, so the person I hand the run to opens it with my notes on the timeline: Each one is a derivation in the flake, with one bug and one fix: Most of the hard work of a debugger is already done by Nix. Rewind adds a little more, but the rest is already there: the inputs, the sources, the debug symbols, and a way for someone else to get all of that on their machine. When you start with something hermetic like a Nix derivation, you can get a debugger for free. On my 16-core laptop, lost money in 396 of 1,000 runs. Pinned to a single core with , it lost money in none of 1,000: one core alone rarely switches threads in the middle of a deposit, which is why Rewind perturbs the schedule. ↩ It is a little hard to see since the event from Teller 2 is tucked underneath the Orange bar. ↩ Schedule 0 is the run itself, which passed. Every other line is a fork, and is failing because the balance came up short. ↩ read 150 read 150 store 200 store 150 + 10 160 teller 1's last deposit is gone Rewind’s VM has one CPU, so its first run passes too. runs the build again under perturbed schedules, each asking the guest kernel to reschedule at different steps, and narrows the first failure down to one step: Running this check on my laptop took 11 seconds. The two runs are the same machine, exit for exit, until step 3237, where only the failing one gets a reschedule. Free thing one: the inputs The argument to is a derivation and can be a flake reference. The beauty of Nix is that it knows the inputs of a derivation, so that’s all we need to make sure the run is reproducible. The derivation’s inputs are the source, the compiler, the libraries, the kernel, and the VM’s configuration realises the derivation’s inputs, packs the closure into a read-only erofs image, and boots the VM on it. The run’s id is a hash of its inputs, the way a store path is, and prints the command that makes it again: When the build succeeds, the guest reports each output’s NAR hash, and checks it against the host’s copy and against every binary cache Nix substitutes from, by fetching only the : This helps us validate that the build within the VM is the same as the build on my laptop, and that the VM’s run is reproducible by Nix. Free thing two: every symbol and every source Debugging code is much simpler when you are looking at the source. A new panel in the Rewind app or in the terminal shows the source of the program at the playhead, and the stack frames that called it. The debugger needs to know where the sources are, and Nix gives us that for free too. nixpkgs builds the packages with , and the debug info is cached on cache.nixos.org as a output. We can leverage debuginfod to fetch the debug info and the sources by build ID, so we can see the source of any binary in the VM, including the Linux kernel! 😲 gdb, on a fork of any step Sometimes though, the source is not enough, and we need to see the state of the program. opens gdb on a fork of the run at the playhead, with every thread of the process. The debugger can set breakpoints, watchpoints, and inspect memory, registers and variables. At step 3249, just before teller 2 gets the CPU back, we can watch the variable and continue until it changes. Nothing gdb does changes the recording, and we can rewind to the same step and fork again, or fork at any other step, and gdb will see the same state. If gdb is not enough, opens a shell in the VM at a step with any package from nixpkgs on its . It is one more closure packed into one more image. Compare two runs Rewind makes it very easy to compare two runs. The Compare tab in the app, or in the terminal, shows the last shared events of both runs, and the first event that differs. The Compare tab puts both runs’ events side by side from just before they part, with what differs marked. Teller 2 starts from 150 in the failing run and from 200 in the passing one: Who had the CPU Rewind’s VM has one CPU, so at any step exactly one thread is running. A race is a question of order: which thread ran when. The trace does not answer that directly. It records what threads did, a write, an open, a fork, an exit, a signal, but not who was running in the gaps between those events. Rewind can fill in the gaps without recording anything new. Every run replays exactly, so it can walk any stretch of a run one step at a time and, at each step, ask the guest kernel which thread is on the CPU. In the failing run, teller 1 (Thread 140) is interrupted at step 3238, halfway through a deposit: it has read the balance but not stored it. Teller 2 (Thread 141) gets the CPU for a single step, 3239, long enough to read the balance, and then teller 1 gets it back and finishes. 2 Check from here Finding one bad interleaving raises the next question: how likely is it? Is the passing run the normal case, or did it get lucky? normally answers that by building the whole derivation again under many schedules. With , it starts from a run you already have instead, at any step you pick_. It forks the run there once per schedule, each fork taking a different order of threads from that step on, and counts how many end differently. Everything before the step stays exactly as it was. We can apply this technique to the bank example, starting from the step where the two runs diverged, 3,221. When we run 16 schedules from there, all 16 lose money 3 : We can do the same thing from the Rewind App with “Check from here” in the context menu of the playhead. It forks the run at the playhead and runs each fork under a different schedule, counting how many end differently. Bookmarks bookmarks the playhead’s step with a note. Bookmarks are kept with the run, and they travel in its export, so the person I hand the run to opens it with my notes on the timeline: Other examples to try Each one is a derivation in the flake, with one bug and one fix: : the deadlock from the last post. : the lost update above. : a SIGCHLD that arrives between a flag check and , so the parent sleeps until ’s ten second timeout kills it. : one process rewrites a config file in place while another rereads it. On many cores it fails nearly every time; on one CPU only some schedules land the reader between the truncate and the last write. On my 16-core laptop, lost money in 396 of 1,000 runs. Pinned to a single core with , it lost money in none of 1,000: one core alone rarely switches threads in the middle of a deposit, which is why Rewind perturbs the schedule. ↩ It is a little hard to see since the event from Teller 2 is tucked underneath the Orange bar. ↩ Schedule 0 is the run itself, which passed. Every other line is a fork, and is failing because the balance came up short. ↩

0 views

Farid Zakaria 1 weeks ago

Rewind VM: a flaky build you only catch once

If a test fails on CI and nobody can reproduce it, did it really fail? 🧘 Nix gives me a build that is a function of its inputs via the extensional model. The same derivation, produces the same store path, and if I am lucky, the same bytes. What Nix does not give me is the same run . 1 A test suite with a race in it may pass on my laptop, fail once on CI, and when I rebuild it to look, it passes again. If you have experienced bugs like this, you know that it can be frustrating. You are effectively at times trying to find the needle in the haystack . I wrote recently that the Nix sandbox is a hidden input to a derivation. The same is true of the thread schedule. The order the kernel happens to run your processes in decides whether some builds pass, and nothing within the derivation records it. These are the hidden inputs to a build that can make it flaky. Are we left to hoping that we will find the needle in the haystack? Or is there a way to make the run a function of its inputs too? I built Rewind VM to make the schedule an input too. It runs a Nix build, a test suite or any Linux command inside a KVM virtual machine whose every run is a function of its inputs. The same inputs give the same run, at the same steps, every time . You can replay a failure, scrub through it, read any file as it was at any point, and fork it under a different thread interleaving. ✨ Confused? Yes it sounds like magic. The best way to explain it is with short demo. Let’s investigate the dining philosophers problem. The problem is that the five philosophers sit at a round table with a fork between each of them. A philosopher needs both forks beside them to eat. A philosopher may pick up one fork at a time. 2 f0 f1 f2 f3 f4 P0 eats P1 eats P2 eats P3 eats P4 eats five philosophers, five forks P0 picks up f0 P0 picks up f1 and eats P0 puts both down P2 picks up f2 P2 picks up f3 and eats P2 puts both down everyone reaches for the left fork each waits on a neighbor's fork nobody can ever eat: deadlock The philosophers take turns, until all five pick up their left fork at once. If all five pick up their left fork before any of them reaches for a right one, every fork is taken and every philosopher waits on a neighbor who is also waiting. Deadlock . 3 On my laptop 14 of 100 rebuilds deadlocked. 💣 builds the same derivation the way the Nix sandbox would, inside the deterministic VM. Oh darn, it passed! That means it will pass forever right? Not quite. The VM is deterministic, but the guest kernel’s scheduler is not. It can reschedule threads at different points in the program, and that can change the outcome. To find the other interleavings, runs the build again under perturbed schedules , we effectively ask the guest kernel to reschedule at different steps which causes a different sequence of events. runs one VM per core by default and stops after the first batch of schedules where the exit code differs. Note A failing schedule on its own is not that super helpful. Schedule 1 which had deadlocked likely perturbs every step from the start of the build to the end, and most of those perturbations have nothing to do with the deadlock. To help with this, narrows it: it shrinks the window of steps the schedule may perturb, first pulling in the end and then the start, reruns the build for each candidate window, and keeps the smallest one that still deadlocks. For our deadlock problem, it might be easier to see the last few lines of the log and see that all five philosophers have picked up their left fork and are waiting for the right one. We can also inspect the events, which are the same as the log but with timestamps and thread IDs. What if I’m not familiar with the VM? Can we look around? Yes! drops you into a shell inside the VM at any step, in a process’s working directory, with the build’s environment, while everything else in the VM stays stopped. We can use to bring gdb into that shell, and gdb can attach to the stuck process. 4 All five philosophers are on line 27, waiting for their second fork. f0 f1 f2 f3 f4 P0 P1 P2 P3 P4 left fork first every wait is on a held fork: a ring , and each work on a “throwaway” fork of the run at a step, so nothing they do changes the recording. You can use to do a “real” fork: it branches a run at a step under another schedule. A recording does not have to stay on the machine that made it. packs a run into a single file, with the VM’s kernel, its input image and the keyframes, so another machine with the same CPU vendor can replay it. 5 You are no longer beholden to a random flake on CI. Run the tests under on a CI machine with KVM, upload the failing run’s as a build artifact, and you can reproduce the bug perfectly on your machine. How do we fix the deadlock? We number the forks and always pick up the lower numbered one first. The last Philosopher now reaches for fork 0 before fork 4, so the waits can never cause a deadlock. f0 f1 f2 f3 f4 P0 P1 P2 P3 eats P4 lower numbered fork first P4 holds nothing, so f4 stays free We can then run to test the fix. Before the fix, 9 of the same 64 schedules deadlocked. Rewind VM includes some tutorials with more examples of using to find and fix bugs if you want to explore further. The VM has a single vCPU on stock KVM, so guest code runs on the real CPU at close to native speed. 6 What breaks determinism in a normal VM is everything that reaches the guest from outside its instruction stream: timer interrupts, clocks, random numbers, I/O completions, and any other event Rewind removes each of those or replaces it with a value it controls. Keeping account of every event and the step it happened at, it can replay the same run. If you squint, a run is a derivation. Its inputs are the kernel, the initramfs, a root filesystem (a Nix closure packed into a read-only image), the command, a seed and a schedule. Change any of them and you get a different run. Change none of them and you get the same one. The command is open source under the MIT license. 7 There is also a desktop app that gives a friendlier view of a recorded run. You can drag the playhead, view the build log, the process tree and the files at any event. “Open shell” and “Attach gdb” open a terminal pane on a fork at the playhead. I pointed Rewind at some tools I use every day to see what we can find. Each of these is reported upstream with a fix. On NixOS there is a module: If you have a test that fails on CI once a week, I would like to hear whether Rewind catches it and helped you debug it. Don’t just add a and paper over your concurrecy failures anymore, replay them. 🔁 A run is the sequence of events that happen in a process to produce a result. A build is a run of a derivation. ↩ The world is a metaphor for the thread scheduler. Each philosopher is a thread and each fork a mutex. ↩ The program stops making progress and never exits, so the check phase runs it under , which kills it and exits with status 124. ↩ There is actually native support for gdb in the VM already. You can also use to bring in any other tool you want to use. ↩ Without , writes only the run’s trace, its events and output, and leaves out the kernel, the input image and the keyframes. The file is much smaller and enough to read the run with and . It can’t be replayed or forked, though, so , and need the full export. ↩ GNU hello build from nixpkgs takes 11.7 s in the VM vs. 14.3 s without it on the same laptop. ↩ The guest kernel patch is GPL-2.0 alongside Linux. ↩ The NixOS module can do it automatically for you and without the setting Rewind falls back to a coarser clock. ↩ f0 f1 f2 f3 f4 P0 eats P1 eats P2 eats P3 eats P4 eats five philosophers, five forks P0 picks up f0 P0 picks up f1 and eats P0 puts both down P2 picks up f2 P2 picks up f3 and eats P2 puts both down everyone reaches for the left fork each waits on a neighbor's fork nobody can ever eat: deadlock The philosophers take turns, until all five pick up their left fork at once. If all five pick up their left fork before any of them reaches for a right one, every fork is taken and every philosopher waits on a neighbor who is also waiting. Deadlock . 3 On my laptop 14 of 100 rebuilds deadlocked. 💣 builds the same derivation the way the Nix sandbox would, inside the deterministic VM. Oh darn, it passed! That means it will pass forever right? Not quite. The VM is deterministic, but the guest kernel’s scheduler is not. It can reschedule threads at different points in the program, and that can change the outcome. To find the other interleavings, runs the build again under perturbed schedules , we effectively ask the guest kernel to reschedule at different steps which causes a different sequence of events. runs one VM per core by default and stops after the first batch of schedules where the exit code differs. Note A failing schedule on its own is not that super helpful. Schedule 1 which had deadlocked likely perturbs every step from the start of the build to the end, and most of those perturbations have nothing to do with the deadlock. To help with this, narrows it: it shrinks the window of steps the schedule may perturb, first pulling in the end and then the start, reruns the build for each candidate window, and keeps the smallest one that still deadlocks. For our deadlock problem, it might be easier to see the last few lines of the log and see that all five philosophers have picked up their left fork and are waiting for the right one. We can also inspect the events, which are the same as the log but with timestamps and thread IDs. What if I’m not familiar with the VM? Can we look around? Yes! drops you into a shell inside the VM at any step, in a process’s working directory, with the build’s environment, while everything else in the VM stays stopped. We can use to bring gdb into that shell, and gdb can attach to the stuck process. 4 All five philosophers are on line 27, waiting for their second fork. f0 f1 f2 f3 f4 P0 P1 P2 P3 P4 left fork first every wait is on a held fork: a ring , and each work on a “throwaway” fork of the run at a step, so nothing they do changes the recording. You can use to do a “real” fork: it branches a run at a step under another schedule. A recording does not have to stay on the machine that made it. packs a run into a single file, with the VM’s kernel, its input image and the keyframes, so another machine with the same CPU vendor can replay it. 5 You are no longer beholden to a random flake on CI. Run the tests under on a CI machine with KVM, upload the failing run’s as a build artifact, and you can reproduce the bug perfectly on your machine. How do we fix the deadlock? We number the forks and always pick up the lower numbered one first. The last Philosopher now reaches for fork 0 before fork 4, so the waits can never cause a deadlock. f0 f1 f2 f3 f4 P0 P1 P2 P3 eats P4 lower numbered fork first P4 holds nothing, so f4 stays free We can then run to test the fix. Before the fix, 9 of the same 64 schedules deadlocked. Rewind VM includes some tutorials with more examples of using to find and fix bugs if you want to explore further. What is Rewind VM? The VM has a single vCPU on stock KVM, so guest code runs on the real CPU at close to native speed. 6 What breaks determinism in a normal VM is everything that reaches the guest from outside its instruction stream: timer interrupts, clocks, random numbers, I/O completions, and any other event Rewind removes each of those or replaces it with a value it controls. Keeping account of every event and the step it happened at, it can replay the same run. If you squint, a run is a derivation. Its inputs are the kernel, the initramfs, a root filesystem (a Nix closure packed into a read-only image), the command, a seed and a schedule. Change any of them and you get a different run. Change none of them and you get the same one. The command is open source under the MIT license. 7 There is also a desktop app that gives a friendlier view of a recorded run. You can drag the playhead, view the build log, the process tree and the files at any event. “Open shell” and “Attach gdb” open a terminal pane on a fork at the playhead. Footguns One vCPU. Threads interleave but never run in parallel, so races that need two cores at once are out of reach. No preemption between system calls. A thread spinning on a flag without yielding stalls the VM. AMD needs once per boot for the exact clock. 8 A run replays only on the CPU vendor it was made on , AMD from Zen 2 on. No network besides loopback, and x86_64 Linux hosts with KVM only. Nix : dies of when exits between two writes. It never failed in 20,000 runs on my laptop and failed on the first run in Rewind. #16546 , fixed by #16547 ( case study ). Nix : several processes creating a new store at once fail with “database is busy”, as seen on Hydra. #15987 , fixed by #16554 . nixd : formatter output over 64 KiB hangs the language server. #899 , fixed by #900 . jujutsu : three tests fail about half the time on tmpfs, because operations ending in the same millisecond are ordered by a random id. #10306 , fixed by #10307 . A run is the sequence of events that happen in a process to produce a result. A build is a run of a derivation. ↩ The world is a metaphor for the thread scheduler. Each philosopher is a thread and each fork a mutex. ↩ The program stops making progress and never exits, so the check phase runs it under , which kills it and exits with status 124. ↩ There is actually native support for gdb in the VM already. You can also use to bring in any other tool you want to use. ↩ Without , writes only the run’s trace, its events and output, and leaves out the kernel, the input image and the keyframes. The file is much smaller and enough to read the run with and . It can’t be replayed or forked, though, so , and need the full export. ↩ GNU hello build from nixpkgs takes 11.7 s in the VM vs. 14.3 s without it on the same laptop. ↩ The guest kernel patch is GPL-2.0 alongside Linux. ↩ The NixOS module can do it automatically for you and without the setting Rewind falls back to a coarser clock. ↩

0 views

Farid Zakaria 2 weeks ago

Hiding my AI slop from my Nix friends

Warning If you are pissed off reading this, you can start here and work your way back. I am using AI a lot . I am public about my use of it and when required make the necessary disclosures: “Assited-By” trailers or having to say “I used an AI to help write this software” on every Discourse post. 1 For those of you in the “Anti-AI” crowd, it might surprise you to learn that those of us who enjoy using it also don’t want to read “slop”. I also don’t want anything I create to read like I used AI a lot. Commit messages, READMEs, code comments. I’m reading it also! Being a Nix person, my instructions to the agent are of course declarative. My (and Codex’s ) is generated by home-manager from agent-settings.nix . It has a whole section titled “Writing style: no LLM tells” that I stitched together from Wikipedia’s Signs of AI writing and Simon Willison’s LLM cliche highlighter . The code-style tries to ban many AI-isms as they emerge such as: “it’s not X, it’s Y”, “dvelve”, “byte-order”, minimal em-dashes and so forth. I even came across Fabien Sanglard ’s code style rules from this HackerNews thread that I have enjoyed incorporating as well. Why not have my agent write code that mimics engineers I admire!? Despite this, I still found my agents continuing to use these banned words and phrases. 😒 I was a little annoyed I had to keep re-prompting my preferred writing style but I often reflect on the bitter lesson and generally try to avoid any additional work while I wait for the model’s capabilities to improve. I was however surprised when upon one such occasion I realized that the agent itself had itself gotten frustrated at forgetting my writing style and turned it into a flake check. During the creation of omnibin.dev , I noticed the agent had created tools/check-prose.py to check for banned words and phrases in the source tree. It is a simple Python script that walks the repo and fails on a banned word, phrase, or a single em dash anywhere. I had begun setting up a distinct pattern of how I like my repositories laid out so it was funny to see it automatically include this check in the CI of my build. Despite the bitter lesson , I might try leveraging this check in my other repos. What I like about deterministic checks rather than a markdown context-file is that it’s verifyable and reproducible. I’m not left guessing whether the agent is following my style or not. I can run the check myself and see if it passes or fails. Let’s be honest with myself and do some introspection. How much has AI-isms crept into my writing? I applied the same check to my own blog posts. Clearly I was not a fan of em dashes in the last five years, then suddenly I found 21 uses in two, which happen to coincide with the proliferation of AI. 2 🤔 We seem to be all worried about how many em dashes are in our text. Goodhart’s law teaches us that at that point, that is no longer a good measure of AI slop. Treat each other kindly. Seems like whether or not you use AI, I feel more compelled just to disclose I did to avoid the “you didn’t disclose it” argument. Such is life. ↩ The top offender is at 51 hits. I used it in 2021 and 2022 too. Some tells are just mine. ↩ Seems like whether or not you use AI, I feel more compelled just to disclose I did to avoid the “you didn’t disclose it” argument. Such is life. ↩ The top offender is at 51 hits. I used it in 2021 and 2022 too. Some tells are just mine. ↩

0 views

Farid Zakaria 2 weeks ago

Every package is already installed

tl;dr; omnibin is a FUSE filesystem that puts every binary nixpkgs ever shipped on your . Nothing is installed. Nothing needs building. 0 bytes on disk until something actually reads a file. 😈 It’s 2026, why am I still installing packages individually? 1 Why must I go through the ritual of adding a package to my , running or succumb to the hellscape of . Nix gives us the power of having packages installed side-by-side without conflict. Why do I have to pick which ones I want to install? Why can’t I just have them all? What if the machine just had all of them? That is over fifty thousand 2 top-level binaries available on my , from 2013 to 2026 built by Nixpkgs , available on-demand, without installing anything. This is the magic 🧙‍♂️ of Nix , but it’s not restricted to Nix. Everyone seems to still love Docker and OCI , why am I still picking which base image to use? Why can’t I just have them all? Is this the ultimate agent harness? It’s a container with everything in it, right from the start. Try it at fmzakari/omnibin . Of course, I cannot forget our NixOS friends. You no longer have to curate your or , you can just have them all. What is “package management” if every package is already installed? Turns out that Hydra writes a file next to every single narinfo on cache.nixos.org that describes the contents of the archive as JSON: That metadata turns out to be the perfect index for a FUSE filesystem that can lazily fetch the NARs from the cache and unpack them on-demand. 🤓 None of this would mean anything without nixpkgs-multiverse , which already resolves any in nixpkgs history to the store path Hydra built for it on cache.nixos.org . When you combine the two, you get a filesystem that can answer the question “where is ” and then fetch it from the cache and unpack it for you, all without ever having to install it. I crawled all of it the files in under twelve minutes. 🤯 Once you have that, the filesystem writes itself: That is CPython 3.6.2, from 2017. That downloaded nothing , it is answered from the pre-crawled index. Agents are “a thing”. Making them useful is a thing. Making them useful without installing anything is a thing. If your agent tried to and stat every single entry, it would have a really bad time. There are 881,933 binaries in the tree, and it would take a long time to stat them all. To help the agents out a bit, lists only the bare names, one per executable, each resolving to the newest package that provides it. The versioned forms all resolve, but they are not listed. For example, resolves to the latest Python 3, which is 3.14.6 at the time of writing, but resolves to the 2017 version. For everything else there is the index, which is sitting right there in the mount: That UX is a little rough, so you can also use the CLI to query the index: Lastly, there is a /omnibin/README.md whose entire job is to tell whatever is exploring the filesystem to stop exploring the filesystem and query the database instead. 🤖 At this point it should be obvious, but you pay for this on startup for the first access. The first run took 2.7 seconds to fetch the NARs and unpack them, the second run was instantaneous because the store paths were already present. Other than that? Not really, which is pretty amazing. For any long-lived machine, you would expect your to already be warmed up with the packages you need, so the first access penalty is not a big deal. I remember one of the first things that blew my mind and sold me on Nix, was seeing a demo by @burke on comma . The capability to test a package, at a single nixpkgs revision, without “installing it”; revolutionary! I believe this to be a spiritual successor and I hope to imbue others with the same sense of wonder and amazement that I felt back then as a beacon of the power of Nix. The repo is at github.com/fzakaria/omnibin . Please responsibly. Yes, I am a little inspired after watching DHH’s keynote at RailsConf 2026 . I feel the same way about package management. ↩ There are actually 881,933 binaries in the tree, but only lists the latest version of each binary. The versioned forms are still available, but they are not listed. ↩ Yes, I am a little inspired after watching DHH’s keynote at RailsConf 2026 . I feel the same way about package management. ↩ There are actually 881,933 binaries in the tree, but only lists the latest version of each binary. The versioned forms are still available, but they are not listed. ↩

Linux

0 views

Farid Zakaria 3 weeks ago

A build graph that rolls dice

We are right around the corner from NixCon 2026 . Another year where I sadly won’t be present. I looked at the lineup and saw quite a few talks on dynamic derivations, along with some recent other posts on the topic such as cargo-dyndrv which had me thinking about revisting the topic since my earlier posts on the subject. I wanted to better understand the implications of dynamic derivations and how they change the build graph. Originally, I was focused on how it could replace the lang2nix style of build graph generation, but I realized that the implications are much broader than that. The graph does not need to be known up front. It can be defined as the build progresses. 🧐 What do I mean? Traditionally, Nix requires you to ask it to build anything and it will tell you exactly the steps it will take ahead of time as the “derivations” that comprise the build graph. This is an important property of Nix. It is what makes it possible to reason about builds without running them by understanding what will be built. Many Nix tools rely on this property via . For instance, nix-diff tells you why two closures differ without building either . Why does dynamic derivations change this? In functional languages it’s very easy to get abstract and use fancy words like “applicative” and “monadic” to describe the difference between types of computation. The difference is subtle, but it is profound. Nix traditionally was an “applicative” build system. Dynamic derivations make it a “monadic” build system. The difference is that in an applicative build system, the entire build graph is known up front, while in a monadic build system, the graph can be defined as the build progresses. 1 One way to think about the difference is to look at the type signatures of the two operations that define them: (applicative) takes a value known ahead of time and returns the result . You can see the entire graph before you run it. (monadic) takes , a function , to return . You cannot see the entire graph before you run it. Applicative can be thought of writing the shopping list before you leave the house. You read the recipe, you write down every ingredient, you drive to the store once. The list is a function of the recipe and nothing else. Monadic is a recipe with a step that says taste it, and if it is too salty, go buy a potato . You cannot write that shopping list up front. Whether the potato is on it depends on the saltiness, and the saltiness does not exist until you have already done some of the cooking. Nix used to be solely the former, dynamic derivations add the latter. Okay, I guess I should have said “Nix is now monadic in the scheduler”. Nix has always had bind in the evaluator. We have been doing monadic builds for years. We call it import from derivation . on a derivation output is a bind in exactly the sense above: what to build next is a function of a value that does not exist yet. The Nix evaluator cannot produce the graph without that value, and the only way to get it is to stop evaluating and run a builder. Unfortunately, this has a lot of footguns, such as causing to take ten seconds , and why nixpkgs bans the technique outright . Whereas import from derivation is a bind in the evaluator, dynamic derivations now adds bind in the scheduler. The difference is that the scheduler can run builders in parallel, ship them to remote machines, and it can substitute their results from a cache. The evaluator cannot do any of that. Dynamic derivations do not add the bind. The bind was always there. What changes is which layer performs it. 🤓 Many of the examples of dynamic derivations have been focused on build graph generation via lang2nix tooling, but I wanted to explore the implications of dynamic derivations in a more general sense. In the true monadic sense, the build graph can be defined as the build progresses. The next step in the build graph can depend on the result of a previous step. Here is a really simple example, & , that rolls a die and either stops or continues the chain by adding a new derivation step to the build graph. The depth of the chain is random, and the result of the build is how deep we got. Only the leftmost box exists when you run . Everything to the right of it is written by a builder, while the build is already underway. The general idea of this derivation is: Either way the file copied into is a , so works the same on a chain that stopped and a chain that kept going. 2 We can see the build graph grow in the denominator as the build progresses. If we try to introspect the graph with or , we get nothing. Since the graph is not known yet, our tools cannot see it. Dynamic derivations introduce a second graph that exists only after the build, and the ecosystem has no way to introspect it yet. I ran the build 900 times and logged the length of the chain. Rolling a six-sided die is one of the most classic ways to simulate geometric decay and we can see the results in the histogram below. The mean is 6.18, median 4, longest 46. The math maths. Every dynamic derivations demo so far, mine included, has been a build system: MakeNix for C, NpmNix for node, cargo-dyndrv for Rust, nix-ninja for ninja. That is a reasonable place to start but it does not capture the full power of the primitive. What other ideas can we explore? Mario. In Super Mario Derivations the attribute path is the button sequence and I have to supply the press count. The dynamic version emulates until Mario dies. The stopping condition is data, discovered mid-build, and the run is however long it turns out to be. Searching a state space. Swap the die for a predicate and you have a search where each frontier node is a derivation. Model checking, puzzle solvers, a fuzzer that only expands inputs which found new coverage. States you reach twice collapse onto one store path, and they survive a reboot. Crawling. Fetch a page, parse the links, emit one derivation per link. The frontier is discovered rather than declared, the crawl resumes because the store remembers every page already fetched, and the dependency graph of the result is the link graph. What are the bounds of this primitive? I don’t know. The size of the graph now is only limited by whether Nix stops emitting successors. There is no depth limit for the store layer, no equivalent. My longest honest chain in the die-roll was 46 deep, but I had rigged examples that went to 500. How far can it go? I am so brainwashed to thinking about making a plan before I start for my build systems, that the idea of making it up as I go is a little scary. 😨 The paper Build Systems à la Carte is the defacto read on this topic. ↩ We actually need a depth parameter to avoid infinite recursion and so that the store-path of the derivations are different since they are content-addressed. ↩ roll a six and we write a derivation that echoes the depth. Ordinary, nothing dynamic about it. This terminates the build graph. roll anything else and we write a passthrough : a derivation whose only job is , where is . The paper Build Systems à la Carte is the defacto read on this topic. ↩ We actually need a depth parameter to avoid infinite recursion and so that the store-path of the derivations are different since they are content-addressed. ↩

Linux

Rust

0 views

Farid Zakaria 4 weeks ago

Visualizing Nix closures

tl;dr seenix.dev lays every byte of a Nix closure out on a map, one pixel per byte, and lets you zoom from a whole NixOS system down to the hex of . Try hello , firefox or a GNOME desktop . Nothing runs on a server. With the advent of LLMs I keep tugging at any crazy question I ask myself. I know there is the anti-AI crowd and they will happily proclaim anything pursued in this vein as “slop” but I am feeling fortunate to be able to explore these questions. My recent itch was to ask “what does a Nix closure look like?” and to answer it in a way that is interactive and visual . I wanted to see the bytes, not just the store paths. I had come across binvis.io on Hacker News and I found it a compelling way to look at data. I personally never found a need for it, but I found it fascinating none-the-less. 1 The timing for this itch was perfect. I noticed a trending thread on X where a Python binary seemingly includes and . 🤷 I built that tool. You can check it out at seenix.dev . It is a single-page web app that runs entirely in your browser, with no server. It fetches the narinfos of a closure and lays them out on a map, one pixel per byte, and lets you zoom in to see the bytes themselves. We can visualize the closure of that binary, , and see if it really does include those two packages. Turns out it does not. The closure is 41 store paths and 234 MiB, with no and no among them. Turns out those dependencies are build-time and are not included in the final runtime closure. We can visualize much larger closures. Here is a GNOME desktop: 1,324 store paths and 5.3 GiB, each colour one package. That picture needed zero NAR downloads. It was laid out in 3 ms from the narinfos alone. 🤯 The “trick” I learned to make this visualization possible, is the Hilbert curve . A Hilbert curve is a single, unbroken line that folds back and forth such that it completely fills up a flat square. It is a fractal . Every store path in the closure is sorted by name (the root first) and their NARs are concatenated into one long line of bytes. The Hilbert curve folds that line into a square, so byte n is pixel n along the curve. The Hilbert curve has two properties that lend itself nicely to visualize binaries and as a result Nix closures: Bytes that are near each other in a file stay near each other on the map. A NAR is a single contiguous range of bytes, so a store path is a single contiguous region on the map. A file inside that store path is a smaller contiguous region, and a section inside that file is smaller still and so forth. Squares are just byte ranges. Here’s a tiny 4×4 map. Each number is the byte that lands on that pixel: That means we can easily place a store path on the map by knowing its starting byte and its size. That’s what makes the map cheap to draw. 2 The layout only needs each path’s , which every narinfo carries, so the whole map exists before a single NAR is downloaded. Hovering already tells you which store path you are pointing at, its size, its retained size (the bytes that would leave the closure without it) and a “why is this here” chain back to the root. As you zoom in, the NARs on screen are fetched from the cache and the color fills in. Here is ’s closure, most of which is glibc: Blue is printable ASCII, red is high bytes, green is control bytes and black is . The speckled top is machine code. The big solid blue area at the bottom is glibc’s locale data, which is plain text. Keep zooming and every pixel becomes a byte you can read. Hovering names the file inside the NAR, and for ELF files, the section. That is of , in your browser tab, fetched from cache.nixos.org , without any server . 😈 Does everything need a purpose? Sometimes something is fun to make and to use with no real purpose. For fun, I even added a Save PNG button, and it saves the view at the canvas’s full resolution. The ultimate ricing of your NixOS system: a pixel image of your desktop closure. Can Omarchy do that? 😎 Anything you can export works: Drop the file and see the map. You can provide additional Nix binary caches to fetch NARs from as well. The source is at github.com/fzakaria/seenix . Go look at something big. Build without purpose. Have fun. Aldo Cortesi’s writing on visualising binaries is a great resource on this. ↩ This is why the world is always a power of four bytes. hello’s closure is 36 MiB, which fills a bit over half of a 64 MiB square, and the rest is drawn as background. ↩ Aldo Cortesi’s writing on visualising binaries is a great resource on this. ↩ This is why the world is always a power of four bytes. hello’s closure is 36 MiB, which fills a bit over half of a 64 MiB square, and the rest is drawn as background. ↩

Web

0 views

Farid Zakaria 1 months ago

Orange Site Vanity

“Curiosity is only vanity. We usually only want to know something so that we can talk about it” – Blaise Pascal, Pensées I enjoy writing. Most of the time I write for myself, or that is what I tell myself. The act of writing is me trying to deeply understand something and then recording my thought process. It has paid dividends already as I have gone back numerous times to reference myself. When I am honest with myself though, I deeply enjoy knowing when others read my work as well. Knowing that something I found interesting and insightful landed for someone else too is incredibly satisfying. If I could have helped someone understand something better while having done so for myself, pure joy. The peak of that vanity seems to be when the Hacker News crowd has deemed your content “worthy” to have made it on the front page . There is a sort of inner satisfaction when someone else messages me to let me know one of my posts has made it onto Mount Olympus. I have for years added Google Analytics tracking to my site to understand engagement but I rarely went any deeper with the metrics to understand it, until now! 🤓 I have put my vanity on public display by collecting metrics pertaining to my readership . 🪞 The numbers deflate the myth a little. As of writing, my writing has been submitted to Hacker News 127 times, and 26 of those reached the front page. Those 26 bought me 121 hours up there in total, under five hours each 1 , and exactly one ever touched #1. Mount Olympus turns out to be crowded, and difficult to climb. Turns out building the vanity site was itself rewarding. I got a better understanding of the metrics I am collecting through Google Analytics & Search Console. I also tied my writings to submissions to Reddit , Lobsters & Hacker News . The data is fetched offline and periodically updated via a GitHub Actions workflow and included in the site, of course, as a Nix derivation. Pascal was probably right. I tell myself I write to understand things, and that part is true, but I have now built a daily pipeline whose only job is to tell me who else was listening. Curiosity is only vanity. My curiosity now has a dashboard. A goal of mine is to have Fareed Zakaria mistaken for me instead of the other way around. 😅 A mean, which I have previously argued means nothing. ↩ A mean, which I have previously argued means nothing. ↩

Analytics

Writing

0 views

Farid Zakaria 1 months ago

A Nix store is three functions

While building trynix I needed somewhere to host a store-path that did not exist on cache.nixos.org . 1 I wanted to demonstrate that non-Nixpkgs store paths could be booted just as easily. The only requirement seemed to be a lenient Cross-Origin Resource Sharing (CORS) policy, , because the fetch happens in JavaScript. Turns out that GitHub Pages sets that header on every file it serves. 😈 I committed the output of to my Git repository and voilà, I have a free Nix substituter. I seem to be late to the party on this discovery. tomberek’s github-store is a cache assembled out of GitHub release assets. 2 GitHub Pages or Releases are a static file server. It has no idea what Nix is. If a humble file server can be a Nix binary cache, what else could we use? Turns out that in order to be a Nix binary cache, you must implement only three simple functions. The Nix client does not care what medium you use to implement them although HTTP is the most common and included by default in CppNix 3 . Anything that can answer those three requests can be used as a remote Nix store . 4 The reason we can be this careless about transport is that Nix does not trust it. A narinfo’s signature ( ) field covers , , and . It does not cover , , or . Once the archive is fetched, Nix decompresses it and checks that the matches. This is the special sauce of how packages that were signed by cache.nixos.org can be fetched from any other binary cache as an intermediary, and the signature still validates. The field does not even have to be on the same host as the narinfo. It can be anywhere on the internet, and it can be a different protocol than HTTP. Nix does not care. The only thing that matters is that the archive fetched from has the same as the narinfo. For protocols that are not included by default in the Nix client, you can always write an HTTP proxy that translates the three functions to whatever medium you want. In research for this post, I found a few interesting ones. gachix : puts the archives in git’s object database. Git content-addresses and delta-compresses blobs already, so the store dedupes itself; the author reports roughly 82% smaller than the equivalent plain cache. DNS : I wrote a proof-of-concept that puts the narinfo and 4 KiB slices of the archive in TXT records. The narinfo is small enough to fit on one record but the archive needs to be chunked. pastebin : a pastebin can hold the narinfo and the archive. The narinfo is small enough to fit on one paste, but the archive needs to be chunked. Many pastebins have an expiry policy which acts as a natural garbage collector. nixcache-oci : uses an OCI registry to store Nix archives. infinite storage glitch : encodes data within a video and uploads it to YouTube. “Everything is available on npm” – Some person on the internet Unsurprisingly, npm is a great binary cache and it has some interesting properties for release management we can ab use. emits a directory and npm publishes directories: a match made in heaven. 💑 Let’s walk through a small example. It is dynamically linked against glibc, so the closure is five paths and roughly ~36 MiB: We copy it to a local cache, signed with our own key, and add the one file npm needs ( ): then dutifully packages our complete closure for us: is now a real package on the public npm registry. It is now a substituter you can point Nix at directly: Note We have to use to run the binary because is a chroot store and all the are still under . If we had relocatable binaries we could run it directly. That is Nix fetching the complete closure from npm and running it. 🤯 We can distribute Nix packages to non-Nix users, let the infection spread! As an added bonus, similar to Nixpkg and NixOS we can get nice “channel” semantics by using npm’s dist-tags. The tag is mutable and points to the latest version, while each version is immutable and points to a specific store path. The major downside of this approach is that npm ahs no incremental publishing. Every version is a whole tarball, so fifty closures sharing glibc upload glibc fifty times. We could fix that by publishing each store path as a separate package, and then having a small index package that points at them. Each store path would then be uploaded exactly once. I won’t build that though as it’s not in good faith to the npm ecosystem. What other store implementations can we find? I was also waiting for @domenkozar to enable CORS on cache.nixos.org so I could use it. ↩ In order to be a Nix binary cache, the prefix is stripped from the field in the narinfo, because GitHub releases are a flat namespace. ↩ You can write a Nix plugin to implement a new protocol if you wanted. ↩ We will see that that they must not all be all on the same medium, protocol or domain even! ↩ I was also waiting for @domenkozar to enable CORS on cache.nixos.org so I could use it. ↩ In order to be a Nix binary cache, the prefix is stripped from the field in the narinfo, because GitHub releases are a flat namespace. ↩ You can write a Nix plugin to implement a new protocol if you wanted. ↩ We will see that that they must not all be all on the same medium, protocol or domain even! ↩

JavaScript

0 views

Farid Zakaria 1 months ago

Review a pull request by booting it

tl;dr; trynix-preview is a GitHub action that comments a link on a pull request which lets you boot the PR’s build in the browser using https://trynix.dev . No servers, just browsers. I ended my earlier trynix post with a list of ideas I think we could accomplish now that we can boot arbitrary paths in the browser. The most obvious one was to let a reviewer boot a pull request’s build in the browser for testing, validation and feedback. That is now real. 🤯 A demo is worth a 1000 words: here is a pull request ( PR#31 ) against my sqlelf project, from a fork , with the comment our action left on it: Click the link and a Linux machine boots in your tab with that PR’s on . You did not clone anything, you did not build anything. No servers, no SSH, no VPN, no Docker, no VM, no cloud. Just a browser and a link. 😈 As with any GitHub action, it’s just a few lines to add to your workflow. The caveat is that you must have built and cached the path already, so the action can link to it. The action does not build or cache anything. The action publishes and builds nothing. Whatever already fills your cache keeps doing it, and the action’s whole job is to simply provide the store paths via and hand the cache’s URL and public key to the browser. It is not Nix cache provider specific, but I do recommend Cachix because it is free for open source up to 5GiB. 1 You can checkout my trynix.yaml workflow for the full example. You have to set in the step because the workflow runs on a fork’s pull request, and that has security implications. 2 If that is not your cup of tea, there is a version where a maintainer types on the pull request which kicks off the workflow. In either case, the workflow runs on the default branch and checks out the pull request’s code, so a fork cannot edit the workflow that builds it. Did I just upend all CI products by easily letting reviewers boot a PR? Unfortunately, no. 🥲 The performance for large binaries is pretty bad. Even with many of the improvements I AI-assisted into the engine, large binaries can still take 1-2 minutes to execute. 3 Nevertheless, this is still a pretty amazing workflow and showcases the power of Nix. Maybe as we get closer to AGI, our AI overlords will be able to optimize the engine to execute large binaries in a few seconds, but for now, the action is best suited for small to medium-sized binaries. You should definitely sign up for Cachix but you can test this out without it since the free tier is very generous. ↩ I recommend a private segregated cache for pull request builds, so that a fork cannot push to your main cache. ↩ I added a benchmark page, https://trynix.dev/bench/ , to the site with a lot of rich data on boot and run times for various applications. ↩ You should definitely sign up for Cachix but you can test this out without it since the free tier is very generous. ↩ I recommend a private segregated cache for pull request builds, so that a fork cannot push to your main cache. ↩ I added a benchmark page, https://trynix.dev/bench/ , to the site with a lot of rich data on boot and run times for various applications. ↩

Github

Ci Cd Yaml

0 views

Farid Zakaria 1 months ago

How safe is follows?

I recently wrote about omniflake , accessible via https://omniflake.com , which indexes over 12,000 flakes and makes them available through a single flake input. One of the earliest requests I had was to support something similar to my tool nix-auto-follows to consolidate every flake’s inputs to a single coherent set. This is the same pattern you see often in every guide to flakes, often highlighting the attribute without much explanation of what it does or why it is used. The attribute consolidates a flake’s inputs to a single source, often used for . This reduces the number of distinct nixpkgs revisions in the dependency graph, which can speed up evaluation and reduce overall closure size. Those with more experience with flakes have a strong opinion about . While it improves evaluation speed and closure size, it can introduce subtle breakages and deviates from the original author’s intent which is counter to the philosophy of Nix which deems “reproducibility” as a core principle. I wanted to dig into the question of how safe it is to use in practice. The folklore is that it is dangerous, but I wanted to see if that was true. Turns out that https://omniflake.com is the perfect tool to try and answer this question. As I mentioned above, lets you consolidate a flake’s inputs to a single source. In the case of nixpkgs, this means that you are asking a flake to build against a nixpkgs revision that its author may never have evaluated it with. This can lead to breakages if the flake relies on specific behavior or attributes of the original nixpkgs revision. Often the delta between the two revisions is a good measure of how likely it is to break. If the nixpkgs revision you are following is only a few days or weeks newer than the original, it is likely to be safe. If it is several months or years newer, the risk of breakage increases. We can look at every flake within omniflake and see how old the nixpkgs revision is that it locks. 11,936 flakes are indexed in the omniflake index, 10,754 of them lock a nixpkgs and between them they name 3,261 distinct nixpkgs revisions . Here is every distinct revisions, placed by the date it was locked by the number of flakes that pinned it. Quite a lot of revisions are pinned by only a single flake. We can also visualize how old each nixpkgs revision is as of August 31, 2026. Half of all flakes are pinned to a nixpkgs older than 506 days , nearly a year and a half, and a twentieth are past 1,768 days (five years). The channels follow the distribution one would largely expect: (red line) pins are the freshest, which is unsurprising for a channel that moves daily. The darwin channel seems scary to replace with , barely 5% of those pins are under a year old What does this all mean? If we consider the median, when you add a line you are, typically, asking a flake to replace it with a nixpkgs likely a year and a half newer than anything its author ever tested. One time in twenty you are asking it to consolidate across five years. 😬 Not all nixpkgs are equal. A flake pinned to a nixpkgs that was a channel bump was built by Hydra and therefore received testing and is available for substitution from https://cache.nixos.org . We can audit the nix release archive to determine whether a given nixpkgs was ever a channel bump. Every channel bump ever published is a directory under , named for the commit it pointed at: If we enumerate all 74 channels, every and channel, we discover 30,850 (commit, channel) pairs to check against. 1 The majority of revisions, four in five, were a published channel bump. That is better than I expected and means that most flakes are pinned to a nixpkgs that was built by Hydra and therefore received testing and is available for substitution from the cache. Few flakes are pinned to a nixpkgs that was never a channel bump. The most common revisions that are shared amongst the flakes are relatively recent as well. The top 8 revisions are all from 2026, and the oldest revision in the top 8 is from May 4, 2026. It was good to see that the most common revisions across flakes are relatively recent. Personally, I enjoy running a release or two behind my personal NixOS laptop to avoid unecessary churn while all the kinks are worked out. How common is that? For each flake, we can take the gap between when the flake was last touched and the date of the nixpkgs it locks. That is how stale the pin was at the moment its author chose it , this is very different than the earlier graph which is how stale it is as of today . The original stateleness is depicted by the green line. It is the age of Nixpkgs the day it was written whereas red is the same pin today. Half of every flake in the index picked a nixpkgs less than three week old, and a quarter picked one inside three days! The shades region is simply time passing. It is how stale the pin has become over time. It is a frequency of how often a flake is updated. So essentially nobody deliberately pins an old nixpkgs. They run , get whatever was current that week, and stop . The ecosystem’s year and a half median is the result of a year and a half of neglect on each flake. Not realy but it was fun to look at the data. omniflake includes an easy way to unify all flake inputs . We could use it to see how many flakes would break if we replaced their nixpkgs with a single revision, but that would only tell us whether it could evaluate. A flake can evaluate perfectly and then fail to build. That’s a lot harder to measure. For now, I still recommend being mindful when you use . It is a powerful primitive that can improve evaluation speed and closure size, but it can also introduce subtle breakages, especially if you rely on . 🤫 The data for this post came from omniflake’s own index, joined against the release archive. It is one file, on a gist using a shebang and the standard library, so it needs nothing installed. Beware of bugs in the above code; I have only proved it correct, not tried it – Donald Knuth Earlier releases used shorter revisions and has no channel directory, its releases sit under . ↩ Earlier releases used shorter revisions and has no channel directory, its releases sit under . ↩

0 views

Farid Zakaria 1 months ago

Stamping build info in constant memory

This is a fun little trick I came across at . I did not invent it, but I thought it was interesting enough to understand better and share. At we build with buck2 and we stamp our executables with build information: build-id, timestamp, author, the usual suspects using as a step after the link. The reason it is a separate step is caching. If the build info was generated at link time then everytime we link the binary it would produce different bytes causing it to not be bit-reproducible. When something is bit-reproducible, it is safe to cache it, and the build system can apply early cut-off optimizations. That works, until the binaries get big. We noticed that ’s memory use scales with the size of the file it is stamping. Stamping is exactly the kind of step that runs massively parallel at the end of a build so this can cause a lot of memory pressure. Why is the stamping step reading the binary at all? 🤔 Let’s measure the claim that the memory use of scales with the size of the file. We will attach a JSON build info blob to an increasingly large synthetic executable and measure the peak RSS of the stamping step. The graph confirms the claim. The memory use of scales linearly with the size of the file being stamped. Surprisingly, the slope is two . The peak RSS is roughly twice the size of the file being stamped irrespective of the size of the build info being attached. 1 I am helping to shepherd a PR open against LLVM to stream the ELF output rather than materialize it, which roughly halves the peak. That is a definite improvement but the problem remains that in order to add a tiny section to a large binary, the whole binary has to be read into memory. The memory use is still linear in the size of the file. The problem is not poor implementation on the part of . Adding a section to an ELF touches three separate things: 0x78 bytes in .interp\0 .buildinfo\0 .rela… 0x70 0x78 0x83 the section's bytes 88 bytes of JSON, at 0x401021 The entry holds neither the name nor the bytes. It only refers to them — and only one of those two references can be repointed in place. In order to account for the new section, the section header table has to grow by one entry, and has to grow by the length of the new name. The current model for is to read the whole file into memory, add the new section, and write the whole file back out. That is why the memory use scales with the size of the file. How can we avoid having to rebuild the whole file just to add a tiny section? The trick is to pay the cost at link time rather than at stamp time. We can have the linker emit a placeholder section with the right name and a single byte of content. The section header table entry is already there, and the name is already in . The post-link stamping step can then append the payload to the end of the file and update the section header entry to point to it. 💡 We make linker emit the section during the normal build. It does not need to hold anything; it just needs to exist so that it owns a name and a header. Our “stamp” step is now incredibly simple. It does not need to read the file at all, it just needs to write the new payload and update the section header entry. Nothing that already exists moves. does not move, the section header table does not move, no other changes. The edit is sixteen bytes , at a file offset you can compute from the ELF header, plus a . read into a model, serialized again — 1,238 bytes of it actually differ reserve one byte, then append 16 bytes + a tail ehdr phdrs .text .rodata … .shstrtab section headers payload the reserved byte sh_offset, sh_size Note Why 1 byte? Turns out that and GNU disagree on whether an empty section is a valid ELF. The one byte is a cheap way to make both linkers happy. The payload lands after the section header table, which looks alarming the first time you see it but is completely legal. Nothing in ELF says section contents must precede the section header table. The kernel also never looks at section headers also, it loads segments out of the program headers, which we do not touch. I wrote a small C version to benchmark it in contrast, please be mindful that this graph is log-log. Our trick works! The “append + 16 bytes” approach is constant memory. Not only is the peak RSS constant, but the wall time is also constant and much faster by avoiding the read and write of the whole file. It is often easy to reach for general-purpose tools like as they are a swiss-army knife for manipulating object files. What I like about this trick though is that there are meaningful improvements to be made by writing special purpose tools and that does not mean we have to accrue large maintenance costs. In this case it was a tiny 200-line C program. The economics of these tools is also changing with the rise of LLMs in our workflow. While many are concerned about the influx of generated code, I remain optimistic that we we can use them to find such opportunities. Don’t be afraid to write a small tool to solve a specific problem. For those thinking this is an LLVM specific issue, GNU exhibits the same behavior. ↩ the section’s bytes , somewhere in the file a 64-byte entry in the section header table describing where those bytes are the section’s name , which is not in the entry itself but rather the entry holds a offset into , so the name has to be appended to that string table 0x70 0x78 0x83 the section's bytes 88 bytes of JSON, at 0x401021 The entry holds neither the name nor the bytes. It only refers to them — and only one of those two references can be repointed in place. In order to account for the new section, the section header table has to grow by one entry, and has to grow by the length of the new name. The current model for is to read the whole file into memory, add the new section, and write the whole file back out. That is why the memory use scales with the size of the file. Pay the byte at link time How can we avoid having to rebuild the whole file just to add a tiny section? The trick is to pay the cost at link time rather than at stamp time. We can have the linker emit a placeholder section with the right name and a single byte of content. The section header table entry is already there, and the name is already in . The post-link stamping step can then append the payload to the end of the file and update the section header entry to point to it. 💡 We make linker emit the section during the normal build. It does not need to hold anything; it just needs to exist so that it owns a name and a header. Our “stamp” step is now incredibly simple. It does not need to read the file at all, it just needs to write the new payload and update the section header entry. append the payload to the end of the file write the new and into the placeholder’s section header entry the reserved byte sh_offset, sh_size Note Why 1 byte? Turns out that and GNU disagree on whether an empty section is a valid ELF. The one byte is a cheap way to make both linkers happy. The payload lands after the section header table, which looks alarming the first time you see it but is completely legal. Nothing in ELF says section contents must precede the section header table. The kernel also never looks at section headers also, it loads segments out of the program headers, which we do not touch. Benchmark I wrote a small C version to benchmark it in contrast, please be mindful that this graph is log-log. elfstamp.c Our trick works! The “append + 16 bytes” approach is constant memory. Not only is the peak RSS constant, but the wall time is also constant and much faster by avoiding the read and write of the whole file. One trick pony It is often easy to reach for general-purpose tools like as they are a swiss-army knife for manipulating object files. What I like about this trick though is that there are meaningful improvements to be made by writing special purpose tools and that does not mean we have to accrue large maintenance costs. In this case it was a tiny 200-line C program. The economics of these tools is also changing with the rise of LLMs in our workflow. While many are concerned about the influx of generated code, I remain optimistic that we we can use them to find such opportunities. Don’t be afraid to write a small tool to solve a specific problem. For those thinking this is an LLVM specific issue, GNU exhibits the same behavior. ↩

Performance

Ci Cd

0 views

Farid Zakaria 1 months ago

Your executable is a SQLite database

I have been probably obsessed with two things in the last few years: Nix as a tool to explore innovative ideas that require the capability to rebuild the world and replacing ELF with SQLite as an executable format. You might have noticed that these two ideas are well suited to each other. I explored the idea during my PhD thesis but found feedback from others unmotivating. Radical ideas are hard to sell, as you are working against the inertia of the established solution. One of the end results of that exploration was sqlelf , a tool that lets you explore an ELF file declaratively using SQL. 1 instead of fiddling with and . It was remarkably simple by leveraging virtual tables over the ELF: however I found it to be a refreshing improvement to explore the ELF file format. I knew however that there is still something much bigger to be done. I never let the idea go and with the recent improvements with LLMs, I find it compelling to revisit these ideas to explore further. Specifically, can we replace ELF with SQLite as an executable format? 🤔 Not “a database that describes an executable”, but the actual file you and run. I developed a pretty fleshed out prototype. It is called SELF , the Structured Executable & Linkable Format , because I am unoriginal. It is on GitHub if you are interested. I’m surprised about all the interesting things that fall out of this idea. Working through my PhD, I realized something that bugged me. ELF is already a database. It just implements many database primitives by hand, along with a surprising number of data structures for performance, like a bloom filter for symbol lookup. If you ever have to analyze or parse ELF, the kernel, , binutils, LIEF, goblin, , you are re-implementing the same parser over and over again. Every producer re-implements the same serializer. The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it. SQLite is the counter-example. They are a self-describing format that is extremely stable. It is designed to be extended to support new features without breaking existing consumers and supporting a wide range of queries performantly. If we were to replace ELF with SQLite, what would fall out and can all of the necessary information be represented in a SQLite database? The answer is yes, and it is surprisingly simple. A SELF file needs two tables to run: is the ELF header as key/value pairs and is the load image, one row per program header with the bytes in a : A single table for the symbol table replaces many of the ELF sections and the index. It is a single table with a single index: Our capability to include an index is equivalent to and in ELF, but it is a proper b-tree index maintained by SQLite instead of a hand-rolled bloom filter. 2 Surprisingly a lot more falls out as well: is gone, because is and SQLite already interns strings, symbol versioning is a column, not the / contraption and there is no need for a table. Other tables exist as well for metadata which exist for tooling: , , . Delete them and the program still runs, which means is a transaction: All the tools that operate on ELF files for reading, reduce to queries over the database. Any tool that modifies an ELF file, like , can operate on the database within a transaction rather than performing fragile offset surgery: is a and . is an . Any information missing from the schema can be easily exposed via a view. For example, is a query over the table, which is a join of the table with the table to find the sonames of the libraries needed by the program. SQLite reserves a 4-byte at byte offset 68 of its header, for exactly this purpose. We stamp it , so an ordinary SQLite database never matches: We can now leverage binfmt_misc , the subsystem that allows you to invoke any binary as if it were native. We need only to register the magic to trigger on and an interpreter that will invoke our new file format. On NixOS the registration is a few lines matching the SQLite magic at offset 0 and at 68: For now, I have a small tool that converts an ELF file into a SELF file. It is a simple hook you can opt into per package on NixOS. The tool reads the ELF, extracts the program headers and symbol table, and writes them into the SQLite database. We could look at extending or to emit SELF directly, but for now this is a simple way to explore the idea. is the interpreter. It is a small C program linked against . Its implementation is remarkably similar to that of but it fetches the program headers and symbol table from the database instead of reading them from the ELF file. It maps the loadable segments into memory, relocates them, and jumps to the entry point. Note has to stay an ELF file. An interpreter that also matches the registration recurses straight into . Running a static program was quick and easy but boring and unimaginative. The interesting part is dynamic linking, which is where the database shines. I explored two different ways to do dynamic linking. The first is to keep and just replace the lookup with a SQL query via rtld-audit interface, to quickly iterate on the design. The second is to replace entirely with a new dynamic linker that does the entire lookup and binding in SQL. glibc’s rtld-audit interface lets an audit library intercept every shared object lookup ( ) before any filesystem search happens, included. The audit library can then answer the question “which library satisfies this symbol?” with a SQL query instead of walking the and . Stock maps and relocates it, so the full gamut of glibc features work: lazy PLT, IFUNCs, TLS and symbol versioning, while library storage are rows and library lookups are queries. I was curious what a fully SQL dynamic linker would look like, so I prototyped one. It is called and it is a small C program that implements the dynamic linker entirely in SQL. It is a proof-of-concept, but it works. It maps every object’s segments, publishes their exports, and for each relocation patches the GOT and jumps to the start. The two things that often matter when replacing a well-established format are size and latency. How much bigger is a SELF file than an ELF file, and how much slower is it to run? Size. A SELF file carries SQLite’s b-tree overhead and lands at roughly double the ELF. Similar to ELF binaries, most of that is recoverable, because the overhead is mostly the optional tables for debugging and tooling. Stripping them and deleting them is a transaction. A stripped SELF is 1,794,048 B against the ELF’s 1,768,632 B, that is within 1% . We will see though that there are interesting ways to amortise the overhead even more which I found very unique and interesting. Latency. I benchmarked various binaries from a 15 KiB to a 42 MiB linking 47 libraries: There is a fixed ~5 ms to open SQLite and start the interpreter, plus a copy proportional to the image. That copy is worse than it looks, because the b-tree pages are not mapped into memory. Two processes running the same SELF binary do not share text pages the way a normally- ‘d ELF does, because the bytes are copied out of the b-tree rather than mapped. 3 A SQLite database though need not merely be a single executable. It can be a closure , a single file that contains a program and all of its transitive dependencies. The output of a program is ambiguous: it only lists the sonames of the libraries it needs, not the specific files that satisfy those needs. Nix improves upon this by explicitly resolving every edge to a specific store path via the use of . 4 We can do the same in SELF by storing the resolved path of each edge in the database: packs a binary and its transitive dependencies into one database with those edges filled in. Shared library resolution stops being a guess and becomes a foreign key and becomes a 🤯: This single database is a closure of the executable and its five libraries: six objects, segment bytes and all, in one 4.8 MiB file. There is no soname ambiguity inside a closure, because a closure by construction contains exactly one provider per edge. I hope you’ve been with me so far, because this is where it gets really interesting. We can go even further and pack multiple closures into a single database. I pointed at every ELF binary on this system’s : 723 executables, which pull in 400 distinct shared libraries. 1,123 objects, 346,386 symbols, 3,808 dependency edges, all as one SQLite file . Turns out when you do that, the database is much smaller than you would expect. 611.9 MiB of database against 644.4 MiB of ELF files. The whole userland, as one queryable file, is smaller than the files it came from. The b-tree cost that doubled a single amortises to nearly nothing across 1,123 objects and is roughly 6% over the actual program bytes. The libraries and closure are shared across the executables very similar to how Nix might share them across multiple closures, if the store-path was the same. If every root shipped its own private closure (i.e. the AppImage model), the same 723 programs would come to 5.53 GiB but the deduplication of libraries and symbols falls out naturally from the database schema. Many common idioms we use in ELF immediately fall out of the database. For example, is a row in a table rather than an environment variable. The table is a list of objects to map last, so their exports win. This means that turning on and off is a transaction. We were able to accomplish an atomic across a whole userland in one file, “interpose a tracing everywhere, then ” is a single transaction. 😈 The format is done and round-trips between ELF and SELF losslessly. The tooling is done and can query, modify, and pack closures. Lookup through SQL works on unmodified glibc programs perfectly and the native-SQL loader works enough to explore it as a possibility for ideas. The whole thing is at fzakaria/selfdb . boots a NixOS VM where is a SQLite database. 🙌 Nix lets us explore radical ideas like this. We can rebuild the world down to the Linux kernel if needed. We need not be constrained by the existing decisions and constraints of the past. We can explore new ideas and see what falls out. I hope you find this idea as interesting as I do. I wrote a paper, arXiv:2405.03883 , that I failed to get published and a follow-up post on querying with it . ↩ is a bloom filter plus bucket chains, laid out so can reject a miss without touching the chain during symbol discovery. ↩ You might notice that (274 KiB, 27 libraries) starts slower than ELF (4.6 MiB, 5 libraries). That is doing work proportional to the number of objects rather than the number of bytes, which I have complained about before . ↩ I have written about on Nix before such as making it redundant or speeding it up . ↩ I wrote a paper, arXiv:2405.03883 , that I failed to get published and a follow-up post on querying with it . ↩ is a bloom filter plus bucket chains, laid out so can reject a miss without touching the chain during symbol discovery. ↩ You might notice that (274 KiB, 27 libraries) starts slower than ELF (4.6 MiB, 5 libraries). That is doing work proportional to the number of objects rather than the number of bytes, which I have complained about before . ↩ I have written about on Nix before such as making it redundant or speeding it up . ↩

Sql

Database

Programming SQLite

0 views

Farid Zakaria 1 months ago

Three ways to smuggle SQLite into Nix

The core of nixpkgs-multiverse , when you strip away the Nix API and the CLI, is an index. It is a map from to the revision that shipped it as a JSON file. 1 As of 9cc0209 , is 5.3 MiB and is 7.5MiB covering 305,492 package versions across 31,904 packages and 1,534 revisions. The Nix API loads the JSON files lazily and are all read via : I would like to enrich the data with even more information however it comes at a cost: mo’data, mo’problems. The goal of the project is to minimize the number of Nixpkgs that are downloaded. If we merely swap fetching huge Nixpkgs for huge JSON, it’s not a clear win. For now we have to be judicious about what we store in the JSON files and think of clever encoding schemes to make the data small and compact. If we were not constrained to the Nix , we would leverage established technologies to efficiently encode our dataset that allow multiple query access patterns: databases! Let’s say we were not restricted to JSON, do we have any other options? Why are large JSON files so problematic? is eager . There is no lazy JSON in Nix, no streaming parse (i.e. “just give me this one key”). The moment you touch the result you have parsed all 5.3 MB and materialised all 305,492 values on the Nix heap. In the case of the multiverse, asking for one package costs the same as what asking for all of them. Note The lookup itself is not the problem. Nix attribute sets are a sorted array, so access is a binary search, not a scan. The cost is entirely in the JSON parse and in allocating the values and downloading a large file. If we want to do alternate questions over the index, we have to make sure we keep the answers efficiently stored to better match the access pattern. What we want is obvious. We want a way to efficiently encode the data and a declarative way to define queries: we want SQLite! 2 Nix by default cannot do this. Unfortunately there is no , although I think there should be… Turns out though there are knobs we can touch or sources we can patch to get what we want anyways, albeit each one has a caveat. 😈 I was surprised I did not know about this , and it has been around since release 1.11.9 in April 2017. It is the ultimate escape hatch for a variety of use-cases when you simply can’t get them done with what’s available. takes a list of strings, runs the program, and parses its stdout as a Nix expression . It is gated behind a setting that makes it clear it’s unsafe. For integration, SQLite is perfectly capable of printing the Nix syntax. We never need a serialisation format in between as we make SQLite emit the attrset directly: The caveat is that every query is now a , an , a process image of SQLite, and a re-parse of the output through the Nix parser. If you do not plan to execute many queries that overhead is likely acceptable given the simplicity of the integration. From researching , I stumbled upon . It takes a path to a shared object and a symbol name, s it, and calls that symbol. It landed in 1.8 , December 2014. 3 The shared object must implement the following signature: We can define a new native function that returns the versions for our input: The implementation is ordinary C++ using the Nix API. Below is a snippet of the implementation, making sure to cache our handles to avoid the same startup penalty as : Using it looks like this: Determinate Systems shipped a third option in March of 2026: , which calls a function inside a WebAssembly module. 4 The motivation was similar to wanting to extend Nix surface area but avoid expanding . Wasm is sandboxed and deterministic, so unlike the two builtins above, the goal is to provide a safe escape-hatch . WebAssembly is a binary instruction format for a stack-based virtual machine. The claim is that it is well suited for Nix because it has deterministic execution , which is a lot more restrained than a backdoor . A module needs to export , an initialiser called , and the entry point. Nixpkgs already includes the target for cross-compilation, so making one is pretty straightforward: You call back into the evaluator through the Nix API functions, so a wasm module builds real Nix values, similar to minus the footgun. SQLite ships an official wasm build , so the pieces seem to be sitting right there and the gears in my mind began to turn. Initial attempts to try and load a SQLite database with the traditional Nix were a bit of a failure as Nix strings cannot contain NULL bytes. Thankfully, with the help of some additional due-diligence by LLMs, we discovered that one of the Nix API functions is not in the blog post: is specifically designed for this problem. This function allows a WASM module to pull arbitrary raw-bytes off disk into its memory. Unfortunately, it’s a little too broad in that it reads the complete file which is kind of overkill and what we are trying to avoid from our initial JSON solution. In the pursuit of exploration, let’s patch the implementation and augment the API to allow random access and partial read of a file. Turns out the patch to add is relatively small and straightforward. Now we have everything we need to hook up SQLite and a custom virtual filesystem (VFS) layer to read from the provided path entry. We build a WASM target of SQLite and we set . That flag removes SQLite’s entire VFS layer and requires us to supply one. We provide the build a simple implementation of the API which is a call-back into the Nix evaluator via that newly exposed function. Everything else is stubs. Note Unfortunately gives every call a fresh instance . This is deliberate from the implementation, meaning we pay some startup code each time although not quite as drastic as a & Using it looks like this: 5 That is a real full SQLite with all the bells and whistles: prepared statement, bound parameter, b-tree descent through an index, executing inside the Nix evaluator. All through WebAssembly. 🤯 How do these four approaches compare? Here are all four approaches answering the same question: “which revisions shipped this package?” against the same 22 MB SQLite build of the index. As we initially complained, is a flat line in the wrong place. It is 0.29s whether you ask one question or two hundred, because the 5.3 MB parse happens once and dominates everything after it. starts the cheapest and climbs , roughly 3.8 ms per query of + + Nix-parsing the output. It crosses somewhere around eighty queries. is flat and nearly free , 0.05s across the whole range since we reuse SQLite instantiations across multiple invocations. The database is opened once for the entire evaluation and the pages stay warm. Unfortunately, SQLite in wasm is dominated by a fixed cost , roughly 2.5 s before the first query, then about 7 ms each query thereafter. That 2.5 s is Cranelift compiling 1.1 MB of SQLite. Right now that is a limitation of the WASM implementation however Eelco has mentioned that the generated code could be cached on disk in the future across invocations. For a lock file pinning thirty packages, still wins outright at the current index size. None of these three is right for shipping the multiverse index, and I am not going to make depend on . Asking people to run their evaluator with native code loading enabled so my flake can be faster is not a worthwhile request at the moment . For now, the index stays JSON and I’m holding back on some of the more loftier ideas I have that require a lot more data . Although philosophically I only use CppNix , I was a little intrigued and impressed with what the ecosystem could unlock with WASM. There are definitely some warts however such as waiting for it to JIT and the developer-experience of maybe having checked-in compiled blobs but there is definitely potential to unlock a variety of problems. There are actually a few other files that drive other features such as the statistics or “fast mode” , but they are all JSON as well. ↩ nixpkgs-multiverse already exports a SQLite database as a package to help others explore this data. ↩ The C++ field was originally called and was renamed to for . ↩ Eelco gave a talk about this at SCALE 23x . ↩ Don’t forget that this is needs our patched version of Determiante System’s Nix . ↩ There are actually a few other files that drive other features such as the statistics or “fast mode” , but they are all JSON as well. ↩ nixpkgs-multiverse already exports a SQLite database as a package to help others explore this data. ↩ The C++ field was originally called and was renamed to for . ↩ Eelco gave a talk about this at SCALE 23x . ↩ Don’t forget that this is needs our patched version of Determiante System’s Nix . ↩

C++

0 views

Farid Zakaria 2 months ago

DEFCON34 wrap-up

I recently came back from DEFCON34 and the nix.vegas community. The talks I gave are now online if you are interested in watching them. 🙌 Many thanks to all the organizers of DEFCON34 and nix.vegas. This is our, the Nix community and mine specifically, second year at DEFCON34 and it was a blast. To be honest, I barely interacted with the rest of DEFCON because I was so busy with the Nix community. The talks, the hallway conversations, and the in-chance encounters were all amazing. One particular story, was that Carl Dong happen to be walking by the Nix Vegas village as I was giving my talk on Guix by Nix . He was a Bitcoin core developer and was one of the contributors responsible for the Bitcoin Core reproducible builds project that leverges Guix . 1 For those that don’t know: nix.vegas is the Nix community that runs within DEF CON in Las Vegas, hosted by the SoCal NixOS User Group and Distractions, Inc. This was its second year: DEF CON 33 ran under the banner “Rebuild the World” , and this year’s theme was “Escape Your Fate” . The full playlist is on YouTube . Note If the sound is a bit off or weird, this year DEF CON experimented with “silent” talks. Each talk was broadcasted and attendees had to wear headphones to listen. It was a bit weird giving talks to a quiet room. 🤷 Summary : Nix’s absolute paths buy us reproducibility, but costs us the ability to put the store anywhere else. You can change the store prefix today, but it changes the hash of every single derivation in the closure down to , so you get to rebuild the world before you get to run . How can we circumvent this? The talk walks through in and upstreaming support in the Linux kernel via a eBPF-based solution. Further reading: Linux kernel will support $ORIGIN, sort of . Summary : What was meant to be a lightning talk on guix-transfer and GuixPkgs but went a little over. This is our project on rewriting Guix derivations into Nix derivations so that every Guix package becomes buildable by Nix. This lets us include their source-bootstrapped JDK for instance, which nixpkgs does not have. Further reading: Guix by Nix and GuixPkgs: every Guix package, as a Nix flake Summary : This talk is a bit of a rant, but it is given in good faith with a dose of humor. The core claim is that we optimize Nix and nixpkgs for social comfort and broad appeal, and we pay for it in technical ambition. Further reading: How to piss off your Nix friends . Looking forward to next year. Three talks in two days was a little ambitious, but I would do it again. Everything lives on my talks page alongside their slides and the rest of my talks. He was pleasantly surprised and happy to hear that Nix also has reproducible builds that start from stage0 . ↩ He was pleasantly surprised and happy to hear that Nix also has reproducible builds that start from stage0 . ↩

Security

0 views

Farid Zakaria 2 months ago

nixpkgs-multiverse: fast mode

“The fastest evaluation is the one that never happens.” – Sun Tzu, The Art of Evaluation nixpkgs-multiverse gives you every version of every package that ever shipped in Nixpkgs from a single flake input. Note It continues to blow my mind that this is even possible. It feels like it suddenly unlocks a new dimension of Nixpkgs, and I am still trying to understand what it means. I think this capability is a fundamental change to the way we think about Nixpkgs, and it is not just a new feature. It is a new way of thinking about the entire ecosystem. There was always a penalty at the center of it. Asking for a specific version of , such as , meant fetching the whole ~378 MB Nixpkgs tree from 2021 and evaluating it to determine the . What if we could skip that evaluation? What if we could just ask for the path directly, and have Nix fetch it from the cache if it is there? This is a common idiom if you have ever used . That requires knowing the store path upfront. nixpkgs-multiverse now has a attribute that does exactly that: it gives you the store path for every indexed version of every package. This lets you skip the download and evaluation of Nixpkgs and get the store path straight from the cache. No Nixpkgs is fetched. Nothing is evaluated. No experimental features and no needed for this to work. The complete Nix API, except for releases, works with this fast path. If you want to learn more read the docs about the feature. Every channel bump published a listing of every path Hydra built for it: , or a for back in the pre-2017 era. These files are still available, and they are the source of the multiverse index. The listing is a map from derivation name to store path. The multiverse index is a map from to the revision that shipped it. By joining the two, every historical version gets a concrete address: Knowing the path is not enough, especially in the Nix language. We need to convince Nix that a string that looks like a store path actually is a store path. exists but it is an impure function and requires to work. How do we get around this? We attach “context” to the String. Context is the invisible baggage a String carries in Nix. When you interpolate a derivation into a String, the result remembers where it came from, and that is what makes realise the dependency instead of writing a dangling path into a script. lets you attach it by hand. The identifies that “this String names a store path that must exist,” which is exactly what produces for a path already in your store, except this works for a path that is not in your store yet and is not in this evaluation’s input closure either. Loopole! 👿 We then wrap that in an attrset that resembles like a derivation and the Nix CLI is satisfied: This is tomberek ’s trick from fastpkgs , and it is an amazing trick to circumvent needing . Everything about this remains pure evaluation, and the resulting graph is gauranteed to be bit-for-bit identical to what Nixpkgs would have produced if it had been evaluated. The only difference is that we skip the evaluation of Nixpkgs itself, and instead use the store path directly. The eval path derives the address, the fast path remembers it. A “fake” ( ) derivation has no , because there is no behind it. Nothing can build it and it can only be substituted. The CLI often wants a though when you hand it a derivation attrset, so we must make sure to append the output (i.e. ): and need a real derivation. Every fake derivation carries a lazy that is the real, revision-exact derivation: In the spirit of trying to keep my index small, is empty, so there is not additional information about the package. You can still get the from the real derivation by using as well. This scheme rests on cache.nixos.org still serving thirteen-year-old paths, which thankfully it does and with the same signing key. To demonstrate that the cache is offering nearly every path that Nixpkgs ever built, I ran a census of every indexed version of every package and asked the cache if it was still alive. As of August 14 2026, all 271,187 of them are alive . All of them, down to every NAR payload file. That is 14.8 TB of unpacked software from 2013 onward, one fast command away. 1 There’s some other data on nixmultiverse.com about the census, dependency graphs and additional features. Check it out! All of this is also available via the mvs command line tool as well for offline use. I guess now there is a caveat: there is now a trick in the multiverse. It remains mostly an index, some JSON, and a behind a memo table. The clever trick is tomberek ’s, and it is three important lines. The NixOS infrastructure has never garbage collected the binary cache. It is an S3 bucket that only grows, and the bill is paid by the NixOS Foundation and its sponsors. ↩ The NixOS infrastructure has never garbage collected the binary cache. It is an S3 bucket that only grows, and the bill is paid by the NixOS Foundation and its sponsors. ↩

Performance

0 views

Farid Zakaria 2 months ago

nixpkgs-multiverse is audacitymaxxing

Every package manager on earth picks one version for you. Nixpkgs picked one too. It never had to. I shared nixpkgs-multiverse recently: one flake input that hands you every version of every package that ever shipped in Nixpkgs. I love how unbelievable audacious Nix lets me be, audacitymaxxing . As of this writing, you have access to 31,783 packages and 304,484 distinct package version pairs pulled from 1,537 revisions. 🤯 The fact most distributions only give you one version of each package is not a bug. It is often considered a feature: a single self-consistent set of software that boots and runs together. It falls directly out of a shared global filesystem, the filesystem hierarchy standard (FHS), like , and . The purpose and existence of Nix is to eschew from that convention and allow multiple versions of the same package to coexist. Nixpkgs is a distribution built on that capability, and yet, it has been doing the same thing as every other distribution: picking one version of everything. nixpkgs-multiverse only supports, at the moment , top-level attributes that are packages but already the sheer volume of installable software dwarfs . 1 Nix’s answer to the FHS was audacious in 2003 and is still audacious now. A package lives at , where the hash is derived from every input that went into building it: the intensional model . How audacious are we? How about 246 distinct CPython versions, from 2.6.8 forward, all installable side by side, all built and cached, all addressable by version number instead of commit hash. 2 To re-iterate, these are distinct versions of CPython, including their transitive dependencies. There is no or or that is shared between them. 3 They work just as reliably as when they were first released, and they are all still installable today and can be substituted from the cache. People want to pin to a version. Upgrading software can be disruptive, and some people have to stay on a particular version but that should not impede the rest of the world from moving forward. The nixpkgs-multiverse helped solve one of the oldest devenv.sh issues, cachix/devenv#16 , the desire to pin a specific package. “It is not really practical to pin a separate version of nixpkgs for every different version of a tool needed in a dev environment. Normally we have at least 20-30 different tools all with a specific pinned version that we would want to specify.” – itpropro The issue, “Pinning a specific package”, was opened on 2022-11-10 and is now closed. devenv now documents the multiverse as the solution. 💪 The audacity of the multiverse is not technical. Nix took care of that. There is no clever trick in here; it’s 5 MB of JSON, about 200 lines of Nix and a behind a memo table. The audacity is in the premise. Two smaller things landed that I like and which was driven by feedback from the community. A soak period. gives you the whole of as it stood N days before an anchor, a cooldown window, in the spirit of Determinate Systems’ cooldowns , except the anchor can be any selector takes. Provenance. Every package set carries where it came from, so a you were handed can be interrogated rather than guessed at. The data was fetched from the Repology repository size map. ↩ The data for other distributions was fetched from Repology . ↩ Unless they happen to dedupe due to their hash. ↩ The data was fetched from the Repology repository size map. ↩ The data for other distributions was fetched from Repology . ↩ Unless they happen to dedupe due to their hash. ↩

Linux

0 views

Farid Zakaria 2 months ago

nixpkgs-multiverse: every version that ever existed

Enter the Nixpkgs multiverse. All the versions that ever existed, all in one place. I bumped the release for my NixOS configuration to refresh many of my packages and found that a package I depended on at a particular version is no longer available. The package was “version bumped forward” in a way that broke some of my tooling. It’s late and I don’t want to fix it, so I just add another input pinned to the commit that had the version I want. This works, but it is miserable in a way that compounds. The need for the most recent package is so common that I had keep an overlay that would inject as a package set for me to easily pull from. If I have a need for a particular version of a package and it’s not present in my current , I am left searching for the commit and pinning it. 1 Every pin is a whole extra in the file. Flake inputs are fetched eagerly even if not used. A flake with three inputs whose output references only the first are all materialised. Nix lets us easily create a closure that reproduces a specific version of a package, but Nixpkgs makes it hard to hold one package still while everything else moves. Each Nixpkgs input to a flake is a distinct universe. If we can have multiple Nixpkgs as input to achieve fetching a particular package, why not have every version that ever existed always available ? 🤯 nixpkgs-multiverse is one flake input that gives you all of them at once. We can query the flake for all the versions of a package that ever existed in Nixpkgs . If we want a specific complete revision of Nixpkgs we can use the function. That is access to all the versions of all the packages that ever existed in Nixpkgs. You can mix them all together in one shell, one package or a build environment. How is it possible to have multiple Python versions? That is the whole point of Nix itself. Every package immaculately describes its dependencies using a hash via the intensional model . 2 Nixpkgs already supports multiple versions of a package in a single revision (i.e. , , ) as separate attributes. We took this to its logical conclusion of making them all available easily. Our deliberately has no inputs: . Inputs are fetched eagerly, and we have 1,393 of them. We need to fetch them lazily, only when something actually references a revision. To do this, we fetch revisions with , pinned by , only when needed. Two files do all the work: and . is one ordered array of every revision from Nixpkgs , 1,393 as of this writing, from 2017 to 2026. 3 We limit our commits to those that were actually built and cached by Hydra, so we only include commits that were either a release or a channel bump. How do we know which revisions to pick for the ones? We rely on the nix-releases S3 bucket to tell us which commits actually became published builds. The S3 bucket uses the commit hash as the directory name, so we can list the bucket and get a complete list of all revisions that were actually built. is the map from (attribute, version) to a revision: That integer is an offset into . It is the most recent revision that shipped that version. At this many revisions, it turns out that how you encode the data matters a lot. My first encoding stored every revision a version appeared in. Although it was simple, it was a disaster in terms of size for these JSON files. As you might expect, most versions of most packages are unchanged across many revisions. The size of our file was growing linearly with the number of revisions. By storing only the newest revision that shipped a version, we can keep the file small and still answer the question “which revision had this version”. Here is how it actually grows as revisions get indexed: 5.18 MB covering 1,393 revisions and 289,521 distinct (attribute, version) pairs. The key design rule for our flake: Cost is per revision touched , not per package. If we were to add revisions as inputs, evaluating our flake would explode. Each flake in our measurement below has N inputs and an output that references only the first one ; the timing is how long before that output evaluates. 4 Five pins that are not used cost 26 seconds before the output evaluates. Each input costs about 5 seconds, and the input is fetched and materialised even if never used. In contrast, the green line is with 1,393 revisions available , which is a flat 0.20s to parse the JSON. 🤩 Revisions are memoised, so pulling 3 packages out of one revision costs the same as pulling one. That concept that the can hold many graphs of the same package is core to understanding Nix. The popularity and rise of flakes made it even more apparent that we can mix multiple revisions of Nixpkgs together. The thing I keep coming back to is that Nixpkgs history already is the multiverse. Every version that ever existed is already built, already cached, already reachable. It was just addressed by commit hash instead of by version number, which is exactly backwards from how anyone thinks about it. The whole project is 5 MB of JSON and about 200 lines of Nix. It does not build anything, mirror anything, or host anything. It is a phone book. Thankfully sites like nixhub.io or lazamar’s search make this a little easier. ↩ The hash is a unique identifier for the exact set of inputs that were used to build it. If you change any input, the hash changes and you get a new package. ↩ A NixOS release is not special; it is a commit that happens to carry a label. ↩ Everything is against a local clone, so there is no network latency. ↩ Thankfully sites like nixhub.io or lazamar’s search make this a little easier. ↩ The hash is a unique identifier for the exact set of inputs that were used to build it. If you change any input, the hash changes and you get a new package. ↩ A NixOS release is not special; it is a commit that happens to carry a label. ↩ Everything is against a local clone, so there is no network latency. ↩

0 views

Farid Zakaria 2 months ago

Super Mario Derivations

One of the most surprising aspects of the Nix language is that it is lazy , especially if you have never used a lazy language before. This laziness is what makes much of Nixpkgs possible, and its complexity. One of the simplest ways to observe the laziness is by understanding that only the attributes you access are evaluated. The more whackier version of this is you can have endless recursion in an attribute set. Nixpkgs is filled with these bottomless attribute sets: The same store path every time. contains itself, and so does every package set inside it. 🤯 If laziness is what lets a recursive attribute set terminate, then the recursion doesn’t have to bottom out at all : That attribute set is infinitely deep. Indexing three levels into it costs exactly three levels of evaluation, and the rest of the infinite tree is never built because nobody asked. So an attribute path is a walk through a lazily-generated tree. Which made me wonder: what if the attribute path were input to something ? 🤔 I decided to take that idea and make the attribute path a sequence of button presses in Super Mario Bros. 3 . Each node in the tree is a frame of the game, and each child is a button press that produces a new frame. Game states are recursive by nature. is right + B, which in Super Mario Bros. 3 is “run right”. is run and jump. The output is the frame you’d be looking at if you’d pressed those buttons in that order, on real hardware, in that game. 1 Append anywhere along the path and you get the whole run stitched into a recording: The coolest thing though is that every one of those frames is a separate derivation in my store . The code is at fzakaria/nes-nix . It is generalized and the ROM is a flake input you point wherever you like for any other game. The flake computes a derivation based on the attribute path such that each press is its own derivation, and it takes the previous press’s savestate as an input . Each derivation never re-emulates its ancestors’ frames. 2 The practical consequence is that the store becomes the emulator’s savestate history: Branching off the middle of a hundred-press run costs one press as does appending to the end of it. We can look at it the other way. The dependency graph is the input sequence, so we can ask Nix what buttons produced a frame: So what is actually doing? Almost nothing. Every frame along the path is already sitting in the store as the output of its own press, so the recording never emulates anything. It is a directory of symlinks to the frames for to process. How far can we take this input-sequence game input idea? Nix by default gives out at around 2,400 presses, with: defaults to 10,000 and evaluating each press costs roughly four nested calls. It’s a guard against runaway recursion, not a structural limit, and we can raise it to 10 million and get 20,000 presses: 20,000 presses, takes roughly fourteen seconds to evaluate on my laptop. The cost is linear in the number of presses, and it is roughly 0.7ms “per press”. The next bottleneck though is that the kernel gives out at 21,845 presses on my machine. An attribute path is a single element, and Linux caps the size of the argument list in total and individual arguments. The per-argument limit is 131,072 bytes ( ), and each press is six bytes long ( ), so 21,845 presses is the maximum that can be passed to as a single argument. The escape hatch is to stop passing the run as an argument. and we can feed in the input-sequence as from a file: This produces the byte-identical derivation to the equivalent attribute path, so a run kept in a file still shares the same store paths. All of this was to simply evaluate the Nix expression. Now we have to build it. Although Nix is great at building derivations in parallel, the recursion here is tail-recursive and therefore serial. I benchmarked the build time of a growing list of button presses and the cost is also linear, as we would expect, with the number of presses. The cost per press is roughly 1.27 seconds with substituters enabled and 0.28 seconds with them disabled. The round-trips cost for checking whether the derivation is in the cache costs noticeably more than emulating the frames does. 3 We’re used to the attribute path being a name , simply a coordinate into a catalogue of things that exist. Laziness means it’s really a program : a sequence of steps the evaluator walks, generating whatever it needs as it goes. Nixpkgs happens to use that machinery to describe software, but nothing about it requires that the tree be a catalogue at all. Coupled with the fact that the store turns out to be a decent persistence layer for reproducible state-machines, makes a our “package manager” reasonable to use for playing Mario. 🍄 The prefix is a precanned sequence of button presses that gets you to the start of level 1-1. ↩ A screenshot of the frame is also produced, which is used when we want to stitch a video sequence together. ↩ We can set or if we want to avoid this cost. ↩ The prefix is a precanned sequence of button presses that gets you to the start of level 1-1. ↩ A screenshot of the frame is also produced, which is used when we want to stitch a video sequence together. ↩ We can set or if we want to avoid this cost. ↩

0 views

Farid Zakaria 2 months ago

A C++ toolchain from 357 bytes, in Bazel

I have been fascinated and amazed by stage0 for a while now ever since I learnt about it via Guix using it to provide twenty two thousand packages source-bootstrapped from the 357-byte seed. What is stage0? It is a chain of compilers and assemblers that can be built from source, starting from a 357-byte program that can eventually build a recent GCC. 1 Since then, NixOS and other distributions have also adopted the same approach to minimize their binary seed which makes it possible to onboard new architectures and platforms much simpler. What’s always frustrated me as a Bazel (& Buck ) user is the reliance on prebuilt toolchains even for things that should be built from source easily like protoc . Bazel has given up trying to provide a hermetic C++ toolchain and the upstream rules_cc ruleset just points you elsewhere: Configuring a hermetic toolchain makes your build more deterministic. rules_cc itself does not yet offer a hermetic toolchain distribution I had attempted to provide a stage0 hermetic C++ toolchain in October 2024 via https://github.com/fzakaria/stage0-bazel . I made substantial process through the bootstrap process but I did not make it far enought to be usabale. To be honest, I was also a little disheartened that no one else in the community thought it was the greatest thing since slice bread. Everyone seems to be content with using prebuilt toolchains as they go deeper into MODULE.bzl madness . I had put it aside for a while, but I have been thinking about it again recently. The steps are mechanical and the process imitates existing distributions, so this became a perfect project for me to throw at an LLM to finish. 2 You can now leverage the toolchain to build in Bazel and have it compiled by a toolchain whose entire ancestry is in the repository from that same 357-byte seed . 🎆 How complete is this toolchain? I pointed the toolchain at Abseil and GoogleTest straight from the Bazel Central Registry without any patches . We then can build and run their testsuite to provide a sanity check that the toolchain is working correctly. We use a to filter tests that require . Abseil marks as a , and Bzlmod drops dev dependencies of non-root modules. That is us building Abseil and GoogleTest, from the registry, unpatched, compiled by a toolchain that began as 357 bytes of hex. How can I be so sure this is a hermetic toolchain? The toolchain includes an audit report that uses Bazel’s aspects to inspect every action in the build graph and verify that it only executes programs built by the toolchain itself. The report is generated by running and will fail if any action executes a program outside of the Bazel output tree. 3 The report is two lines long: Unfortunately, since runs a shell it takes as an absolute system path that is also listed as a seed binary. ’s attribute is a string, and the shell is not a declared input of the action, so no artifact this repository built can provide it. Building toolchains from bootstrap seeds was never a priority for companies like Google where they control the entire build environment. However we seemed to have adopted the same approach as Bazel and similar build systems have become more popular in the open-source community. We should strive to make our builds more reproducible and hermetic, and this is a step in that direction. Once you can reach a recent-enough GCC, you can build any C/C++ program and beyond easily. ↩ Consider this the disclosure that I used an LLM to help me write the remainder of the toolchain. ↩ We also set to disable Bazel’s built-in C++ host toolchain detection. ↩ Once you can reach a recent-enough GCC, you can build any C/C++ program and beyond easily. ↩ Consider this the disclosure that I used an LLM to help me write the remainder of the toolchain. ↩ We also set to disable Bazel’s built-in C++ host toolchain detection. ↩

C++