Farid Zakaria 5 days ago

Stamping build info in constant memory

This is a fun little trick I came across at . I did not invent it, but I thought it was interesting enough to understand better and share. At we build with buck2 and we stamp our executables with build information: build-id, timestamp, author, the usual suspects using as a step after the link. The reason it is a separate step is caching. If the build info was generated at link time then everytime we link the binary it would produce different bytes causing it to not be bit-reproducible. When something is bit-reproducible, it is safe to cache it, and the build system can apply early cut-off optimizations. That works, until the binaries get big. We noticed that ’s memory use scales with the size of the file it is stamping. Stamping is exactly the kind of step that runs massively parallel at the end of a build so this can cause a lot of memory pressure. Why is the stamping step reading the binary at all? 🤔 Let’s measure the claim that the memory use of scales with the size of the file. We will attach a JSON build info blob to an increasingly large synthetic executable and measure the peak RSS of the stamping step. The graph confirms the claim. The memory use of scales linearly with the size of the file being stamped. Surprisingly, the slope is two . The peak RSS is roughly twice the size of the file being stamped irrespective of the size of the build info being attached. 1 I am helping to shepherd a PR open against LLVM to stream the ELF output rather than materialize it, which roughly halves the peak. That is a definite improvement but the problem remains that in order to add a tiny section to a large binary, the whole binary has to be read into memory. The memory use is still linear in the size of the file. The problem is not poor implementation on the part of . Adding a section to an ELF touches three separate things: 0x78 bytes in .interp\0 .buildinfo\0 .rela… 0x70 0x78 0x83 the section's bytes 88 bytes of JSON, at 0x401021 The entry holds neither the name nor the bytes. It only refers to them — and only one of those two references can be repointed in place. In order to account for the new section, the section header table has to grow by one entry, and has to grow by the length of the new name. The current model for is to read the whole file into memory, add the new section, and write the whole file back out. That is why the memory use scales with the size of the file. How can we avoid having to rebuild the whole file just to add a tiny section? The trick is to pay the cost at link time rather than at stamp time. We can have the linker emit a placeholder section with the right name and a single byte of content. The section header table entry is already there, and the name is already in . The post-link stamping step can then append the payload to the end of the file and update the section header entry to point to it. 💡 We make linker emit the section during the normal build. It does not need to hold anything; it just needs to exist so that it owns a name and a header. Our “stamp” step is now incredibly simple. It does not need to read the file at all, it just needs to write the new payload and update the section header entry. Nothing that already exists moves. does not move, the section header table does not move, no other changes. The edit is sixteen bytes , at a file offset you can compute from the ELF header, plus a . read into a model, serialized again — 1,238 bytes of it actually differ reserve one byte, then append 16 bytes + a tail ehdr phdrs .text .rodata … .shstrtab section headers payload the reserved byte sh_offset, sh_size Note Why 1 byte? Turns out that and GNU disagree on whether an empty section is a valid ELF. The one byte is a cheap way to make both linkers happy. The payload lands after the section header table, which looks alarming the first time you see it but is completely legal. Nothing in ELF says section contents must precede the section header table. The kernel also never looks at section headers also, it loads segments out of the program headers, which we do not touch. I wrote a small C version to benchmark it in contrast, please be mindful that this graph is log-log. Our trick works! The “append + 16 bytes” approach is constant memory. Not only is the peak RSS constant, but the wall time is also constant and much faster by avoiding the read and write of the whole file. It is often easy to reach for general-purpose tools like as they are a swiss-army knife for manipulating object files. What I like about this trick though is that there are meaningful improvements to be made by writing special purpose tools and that does not mean we have to accrue large maintenance costs. In this case it was a tiny 200-line C program. The economics of these tools is also changing with the rise of LLMs in our workflow. While many are concerned about the influx of generated code, I remain optimistic that we we can use them to find such opportunities. Don’t be afraid to write a small tool to solve a specific problem. For those thinking this is an LLVM specific issue, GNU exhibits the same behavior.  ↩ the section’s bytes , somewhere in the file a 64-byte entry in the section header table describing where those bytes are the section’s name , which is not in the entry itself but rather the entry holds a offset into , so the name has to be appended to that string table 0x70 0x78 0x83 the section's bytes 88 bytes of JSON, at 0x401021 The entry holds neither the name nor the bytes. It only refers to them — and only one of those two references can be repointed in place. In order to account for the new section, the section header table has to grow by one entry, and has to grow by the length of the new name. The current model for is to read the whole file into memory, add the new section, and write the whole file back out. That is why the memory use scales with the size of the file. Pay the byte at link time How can we avoid having to rebuild the whole file just to add a tiny section? The trick is to pay the cost at link time rather than at stamp time. We can have the linker emit a placeholder section with the right name and a single byte of content. The section header table entry is already there, and the name is already in . The post-link stamping step can then append the payload to the end of the file and update the section header entry to point to it. 💡 We make linker emit the section during the normal build. It does not need to hold anything; it just needs to exist so that it owns a name and a header. Our “stamp” step is now incredibly simple. It does not need to read the file at all, it just needs to write the new payload and update the section header entry. append the payload to the end of the file write the new and into the placeholder’s section header entry the reserved byte sh_offset, sh_size Note Why 1 byte? Turns out that and GNU disagree on whether an empty section is a valid ELF. The one byte is a cheap way to make both linkers happy. The payload lands after the section header table, which looks alarming the first time you see it but is completely legal. Nothing in ELF says section contents must precede the section header table. The kernel also never looks at section headers also, it loads segments out of the program headers, which we do not touch. Benchmark I wrote a small C version to benchmark it in contrast, please be mindful that this graph is log-log. elfstamp.c Our trick works! The “append + 16 bytes” approach is constant memory. Not only is the peak RSS constant, but the wall time is also constant and much faster by avoiding the read and write of the whole file. One trick pony It is often easy to reach for general-purpose tools like as they are a swiss-army knife for manipulating object files. What I like about this trick though is that there are meaningful improvements to be made by writing special purpose tools and that does not mean we have to accrue large maintenance costs. In this case it was a tiny 200-line C program. The economics of these tools is also changing with the rise of LLMs in our workflow. While many are concerned about the influx of generated code, I remain optimistic that we we can use them to find such opportunities. Don’t be afraid to write a small tool to solve a specific problem. For those thinking this is an LLVM specific issue, GNU exhibits the same behavior.  ↩

Performance Programming JSON Ci Cd

▲ 0

0 views

Farid Zakaria 1 weeks ago

Your executable is a SQLite database

I have been probably obsessed with two things in the last few years: Nix as a tool to explore innovative ideas that require the capability to rebuild the world and replacing ELF with SQLite as an executable format. You might have noticed that these two ideas are well suited to each other. I explored the idea during my PhD thesis but found feedback from others unmotivating. Radical ideas are hard to sell, as you are working against the inertia of the established solution. One of the end results of that exploration was sqlelf , a tool that lets you explore an ELF file declaratively using SQL. 1 instead of fiddling with and . It was remarkably simple by leveraging virtual tables over the ELF: however I found it to be a refreshing improvement to explore the ELF file format. I knew however that there is still something much bigger to be done. I never let the idea go and with the recent improvements with LLMs, I find it compelling to revisit these ideas to explore further. Specifically, can we replace ELF with SQLite as an executable format? 🤔 Not “a database that describes an executable”, but the actual file you and run. I developed a pretty fleshed out prototype. It is called SELF , the Structured Executable & Linkable Format , because I am unoriginal. It is on GitHub if you are interested. I’m surprised about all the interesting things that fall out of this idea. Working through my PhD, I realized something that bugged me. ELF is already a database. It just implements many database primitives by hand, along with a surprising number of data structures for performance, like a bloom filter for symbol lookup. If you ever have to analyze or parse ELF, the kernel, , binutils, LIEF, goblin, , you are re-implementing the same parser over and over again. Every producer re-implements the same serializer. The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it. SQLite is the counter-example. They are a self-describing format that is extremely stable. It is designed to be extended to support new features without breaking existing consumers and supporting a wide range of queries performantly. If we were to replace ELF with SQLite, what would fall out and can all of the necessary information be represented in a SQLite database? The answer is yes, and it is surprisingly simple. A SELF file needs two tables to run: is the ELF header as key/value pairs and is the load image, one row per program header with the bytes in a : A single table for the symbol table replaces many of the ELF sections and the index. It is a single table with a single index: Our capability to include an index is equivalent to and in ELF, but it is a proper b-tree index maintained by SQLite instead of a hand-rolled bloom filter. 2 Surprisingly a lot more falls out as well: is gone, because is and SQLite already interns strings, symbol versioning is a column, not the / contraption and there is no need for a table. Other tables exist as well for metadata which exist for tooling: , , . Delete them and the program still runs, which means is a transaction: All the tools that operate on ELF files for reading, reduce to queries over the database. Any tool that modifies an ELF file, like , can operate on the database within a transaction rather than performing fragile offset surgery: is a and . is an . Any information missing from the schema can be easily exposed via a view. For example, is a query over the table, which is a join of the table with the table to find the sonames of the libraries needed by the program. SQLite reserves a 4-byte at byte offset 68 of its header, for exactly this purpose. We stamp it , so an ordinary SQLite database never matches: We can now leverage binfmt_misc , the subsystem that allows you to invoke any binary as if it were native. We need only to register the magic to trigger on and an interpreter that will invoke our new file format. On NixOS the registration is a few lines matching the SQLite magic at offset 0 and at 68: For now, I have a small tool that converts an ELF file into a SELF file. It is a simple hook you can opt into per package on NixOS. The tool reads the ELF, extracts the program headers and symbol table, and writes them into the SQLite database. We could look at extending or to emit SELF directly, but for now this is a simple way to explore the idea. is the interpreter. It is a small C program linked against . Its implementation is remarkably similar to that of but it fetches the program headers and symbol table from the database instead of reading them from the ELF file. It maps the loadable segments into memory, relocates them, and jumps to the entry point. Note has to stay an ELF file. An interpreter that also matches the registration recurses straight into . Running a static program was quick and easy but boring and unimaginative. The interesting part is dynamic linking, which is where the database shines. I explored two different ways to do dynamic linking. The first is to keep and just replace the lookup with a SQL query via rtld-audit interface, to quickly iterate on the design. The second is to replace entirely with a new dynamic linker that does the entire lookup and binding in SQL. glibc’s rtld-audit interface lets an audit library intercept every shared object lookup ( ) before any filesystem search happens, included. The audit library can then answer the question “which library satisfies this symbol?” with a SQL query instead of walking the and . Stock maps and relocates it, so the full gamut of glibc features work: lazy PLT, IFUNCs, TLS and symbol versioning, while library storage are rows and library lookups are queries. I was curious what a fully SQL dynamic linker would look like, so I prototyped one. It is called and it is a small C program that implements the dynamic linker entirely in SQL. It is a proof-of-concept, but it works. It maps every object’s segments, publishes their exports, and for each relocation patches the GOT and jumps to the start. The two things that often matter when replacing a well-established format are size and latency. How much bigger is a SELF file than an ELF file, and how much slower is it to run? Size. A SELF file carries SQLite’s b-tree overhead and lands at roughly double the ELF. Similar to ELF binaries, most of that is recoverable, because the overhead is mostly the optional tables for debugging and tooling. Stripping them and deleting them is a transaction. A stripped SELF is 1,794,048 B against the ELF’s 1,768,632 B, that is within 1% . We will see though that there are interesting ways to amortise the overhead even more which I found very unique and interesting. Latency. I benchmarked various binaries from a 15 KiB to a 42 MiB linking 47 libraries: There is a fixed ~5 ms to open SQLite and start the interpreter, plus a copy proportional to the image. That copy is worse than it looks, because the b-tree pages are not mapped into memory. Two processes running the same SELF binary do not share text pages the way a normally- ‘d ELF does, because the bytes are copied out of the b-tree rather than mapped. 3 A SQLite database though need not merely be a single executable. It can be a closure , a single file that contains a program and all of its transitive dependencies. The output of a program is ambiguous: it only lists the sonames of the libraries it needs, not the specific files that satisfy those needs. Nix improves upon this by explicitly resolving every edge to a specific store path via the use of . 4 We can do the same in SELF by storing the resolved path of each edge in the database: packs a binary and its transitive dependencies into one database with those edges filled in. Shared library resolution stops being a guess and becomes a foreign key and becomes a 🤯: This single database is a closure of the executable and its five libraries: six objects, segment bytes and all, in one 4.8 MiB file. There is no soname ambiguity inside a closure, because a closure by construction contains exactly one provider per edge. I hope you’ve been with me so far, because this is where it gets really interesting. We can go even further and pack multiple closures into a single database. I pointed at every ELF binary on this system’s : 723 executables, which pull in 400 distinct shared libraries. 1,123 objects, 346,386 symbols, 3,808 dependency edges, all as one SQLite file . Turns out when you do that, the database is much smaller than you would expect. 611.9 MiB of database against 644.4 MiB of ELF files. The whole userland, as one queryable file, is smaller than the files it came from. The b-tree cost that doubled a single amortises to nearly nothing across 1,123 objects and is roughly 6% over the actual program bytes. The libraries and closure are shared across the executables very similar to how Nix might share them across multiple closures, if the store-path was the same. If every root shipped its own private closure (i.e. the AppImage model), the same 723 programs would come to 5.53 GiB but the deduplication of libraries and symbols falls out naturally from the database schema. Many common idioms we use in ELF immediately fall out of the database. For example, is a row in a table rather than an environment variable. The table is a list of objects to map last, so their exports win. This means that turning on and off is a transaction. We were able to accomplish an atomic across a whole userland in one file, “interpose a tracing everywhere, then ” is a single transaction. 😈 The format is done and round-trips between ELF and SELF losslessly. The tooling is done and can query, modify, and pack closures. Lookup through SQL works on unmodified glibc programs perfectly and the native-SQL loader works enough to explore it as a possibility for ideas. The whole thing is at fzakaria/selfdb . boots a NixOS VM where is a SQLite database. 🙌 Nix lets us explore radical ideas like this. We can rebuild the world down to the Linux kernel if needed. We need not be constrained by the existing decisions and constraints of the past. We can explore new ideas and see what falls out. I hope you find this idea as interesting as I do. I wrote a paper, arXiv:2405.03883 , that I failed to get published and a follow-up post on querying with it .  ↩ is a bloom filter plus bucket chains, laid out so can reject a miss without touching the chain during symbol discovery.  ↩ You might notice that (274 KiB, 27 libraries) starts slower than ELF (4.6 MiB, 5 libraries). That is doing work proportional to the number of objects rather than the number of bytes, which I have complained about before .  ↩ I have written about on Nix before such as making it redundant or speeding it up .  ↩ I wrote a paper, arXiv:2405.03883 , that I failed to get published and a follow-up post on querying with it .  ↩ is a bloom filter plus bucket chains, laid out so can reject a miss without touching the chain during symbol discovery.  ↩ You might notice that (274 KiB, 27 libraries) starts slower than ELF (4.6 MiB, 5 libraries). That is doing work proportional to the number of objects rather than the number of bytes, which I have complained about before .  ↩ I have written about on Nix before such as making it redundant or speeding it up .  ↩

Sql Database Programming SQLite

▲ 0

0 views

Farid Zakaria 1 weeks ago

Three ways to smuggle SQLite into Nix

The core of nixpkgs-multiverse , when you strip away the Nix API and the CLI, is an index. It is a map from to the revision that shipped it as a JSON file. 1 As of 9cc0209 , is 5.3 MiB and is 7.5MiB covering 305,492 package versions across 31,904 packages and 1,534 revisions. The Nix API loads the JSON files lazily and are all read via : I would like to enrich the data with even more information however it comes at a cost: mo’data, mo’problems. The goal of the project is to minimize the number of Nixpkgs that are downloaded. If we merely swap fetching huge Nixpkgs for huge JSON, it’s not a clear win. For now we have to be judicious about what we store in the JSON files and think of clever encoding schemes to make the data small and compact. If we were not constrained to the Nix , we would leverage established technologies to efficiently encode our dataset that allow multiple query access patterns: databases! Let’s say we were not restricted to JSON, do we have any other options? Why are large JSON files so problematic? is eager . There is no lazy JSON in Nix, no streaming parse (i.e. “just give me this one key”). The moment you touch the result you have parsed all 5.3 MB and materialised all 305,492 values on the Nix heap. In the case of the multiverse, asking for one package costs the same as what asking for all of them. Note The lookup itself is not the problem. Nix attribute sets are a sorted array, so access is a binary search, not a scan. The cost is entirely in the JSON parse and in allocating the values and downloading a large file. If we want to do alternate questions over the index, we have to make sure we keep the answers efficiently stored to better match the access pattern. What we want is obvious. We want a way to efficiently encode the data and a declarative way to define queries: we want SQLite! 2 Nix by default cannot do this. Unfortunately there is no , although I think there should be… Turns out though there are knobs we can touch or sources we can patch to get what we want anyways, albeit each one has a caveat. 😈 I was surprised I did not know about this , and it has been around since release 1.11.9 in April 2017. It is the ultimate escape hatch for a variety of use-cases when you simply can’t get them done with what’s available. takes a list of strings, runs the program, and parses its stdout as a Nix expression . It is gated behind a setting that makes it clear it’s unsafe. For integration, SQLite is perfectly capable of printing the Nix syntax. We never need a serialisation format in between as we make SQLite emit the attrset directly: The caveat is that every query is now a , an , a process image of SQLite, and a re-parse of the output through the Nix parser. If you do not plan to execute many queries that overhead is likely acceptable given the simplicity of the integration. From researching , I stumbled upon . It takes a path to a shared object and a symbol name, s it, and calls that symbol. It landed in 1.8 , December 2014. 3 The shared object must implement the following signature: We can define a new native function that returns the versions for our input: The implementation is ordinary C++ using the Nix API. Below is a snippet of the implementation, making sure to cache our handles to avoid the same startup penalty as : Using it looks like this: Determinate Systems shipped a third option in March of 2026: , which calls a function inside a WebAssembly module. 4 The motivation was similar to wanting to extend Nix surface area but avoid expanding . Wasm is sandboxed and deterministic, so unlike the two builtins above, the goal is to provide a safe escape-hatch . WebAssembly is a binary instruction format for a stack-based virtual machine. The claim is that it is well suited for Nix because it has deterministic execution , which is a lot more restrained than a backdoor . A module needs to export , an initialiser called , and the entry point. Nixpkgs already includes the target for cross-compilation, so making one is pretty straightforward: You call back into the evaluator through the Nix API functions, so a wasm module builds real Nix values, similar to minus the footgun. SQLite ships an official wasm build , so the pieces seem to be sitting right there and the gears in my mind began to turn. Initial attempts to try and load a SQLite database with the traditional Nix were a bit of a failure as Nix strings cannot contain NULL bytes. Thankfully, with the help of some additional due-diligence by LLMs, we discovered that one of the Nix API functions is not in the blog post: is specifically designed for this problem. This function allows a WASM module to pull arbitrary raw-bytes off disk into its memory. Unfortunately, it’s a little too broad in that it reads the complete file which is kind of overkill and what we are trying to avoid from our initial JSON solution. In the pursuit of exploration, let’s patch the implementation and augment the API to allow random access and partial read of a file. Turns out the patch to add is relatively small and straightforward. Now we have everything we need to hook up SQLite and a custom virtual filesystem (VFS) layer to read from the provided path entry. We build a WASM target of SQLite and we set . That flag removes SQLite’s entire VFS layer and requires us to supply one. We provide the build a simple implementation of the API which is a call-back into the Nix evaluator via that newly exposed function. Everything else is stubs. Note Unfortunately gives every call a fresh instance . This is deliberate from the implementation, meaning we pay some startup code each time although not quite as drastic as a & Using it looks like this: 5 That is a real full SQLite with all the bells and whistles: prepared statement, bound parameter, b-tree descent through an index, executing inside the Nix evaluator. All through WebAssembly. 🤯 How do these four approaches compare? Here are all four approaches answering the same question: “which revisions shipped this package?” against the same 22 MB SQLite build of the index. As we initially complained, is a flat line in the wrong place. It is 0.29s whether you ask one question or two hundred, because the 5.3 MB parse happens once and dominates everything after it. starts the cheapest and climbs , roughly 3.8 ms per query of + + Nix-parsing the output. It crosses somewhere around eighty queries. is flat and nearly free , 0.05s across the whole range since we reuse SQLite instantiations across multiple invocations. The database is opened once for the entire evaluation and the pages stay warm. Unfortunately, SQLite in wasm is dominated by a fixed cost , roughly 2.5 s before the first query, then about 7 ms each query thereafter. That 2.5 s is Cranelift compiling 1.1 MB of SQLite. Right now that is a limitation of the WASM implementation however Eelco has mentioned that the generated code could be cached on disk in the future across invocations. For a lock file pinning thirty packages, still wins outright at the current index size. None of these three is right for shipping the multiverse index, and I am not going to make depend on . Asking people to run their evaluator with native code loading enabled so my flake can be faster is not a worthwhile request at the moment . For now, the index stays JSON and I’m holding back on some of the more loftier ideas I have that require a lot more data . Although philosophically I only use CppNix , I was a little intrigued and impressed with what the ecosystem could unlock with WASM. There are definitely some warts however such as waiting for it to JIT and the developer-experience of maybe having checked-in compiled blobs but there is definitely potential to unlock a variety of problems. There are actually a few other files that drive other features such as the statistics or “fast mode” , but they are all JSON as well.  ↩ nixpkgs-multiverse already exports a SQLite database as a package to help others explore this data.  ↩ The C++ field was originally called and was renamed to for .  ↩ Eelco gave a talk about this at SCALE 23x .  ↩ Don’t forget that this is needs our patched version of Determiante System’s Nix .  ↩ There are actually a few other files that drive other features such as the statistics or “fast mode” , but they are all JSON as well.  ↩ nixpkgs-multiverse already exports a SQLite database as a package to help others explore this data.  ↩ The C++ field was originally called and was renamed to for .  ↩ Eelco gave a talk about this at SCALE 23x .  ↩ Don’t forget that this is needs our patched version of Determiante System’s Nix .  ↩

AI Programming C++ JSON

▲ 0

0 views

Farid Zakaria 2 weeks ago

DEFCON34 wrap-up

I recently came back from DEFCON34 and the nix.vegas community. The talks I gave are now online if you are interested in watching them. 🙌 Many thanks to all the organizers of DEFCON34 and nix.vegas. This is our, the Nix community and mine specifically, second year at DEFCON34 and it was a blast. To be honest, I barely interacted with the rest of DEFCON because I was so busy with the Nix community. The talks, the hallway conversations, and the in-chance encounters were all amazing. One particular story, was that Carl Dong happen to be walking by the Nix Vegas village as I was giving my talk on Guix by Nix . He was a Bitcoin core developer and was one of the contributors responsible for the Bitcoin Core reproducible builds project that leverges Guix . 1 For those that don’t know: nix.vegas is the Nix community that runs within DEF CON in Las Vegas, hosted by the SoCal NixOS User Group and Distractions, Inc. This was its second year: DEF CON 33 ran under the banner “Rebuild the World” , and this year’s theme was “Escape Your Fate” . The full playlist is on YouTube . Note If the sound is a bit off or weird, this year DEF CON experimented with “silent” talks. Each talk was broadcasted and attendees had to wear headphones to listen. It was a bit weird giving talks to a quiet room. 🤷 Summary : Nix’s absolute paths buy us reproducibility, but costs us the ability to put the store anywhere else. You can change the store prefix today, but it changes the hash of every single derivation in the closure down to , so you get to rebuild the world before you get to run . How can we circumvent this? The talk walks through in and upstreaming support in the Linux kernel via a eBPF-based solution. Further reading: Linux kernel will support $ORIGIN, sort of . Summary : What was meant to be a lightning talk on guix-transfer and GuixPkgs but went a little over. This is our project on rewriting Guix derivations into Nix derivations so that every Guix package becomes buildable by Nix. This lets us include their source-bootstrapped JDK for instance, which nixpkgs does not have. Further reading: Guix by Nix and GuixPkgs: every Guix package, as a Nix flake Summary : This talk is a bit of a rant, but it is given in good faith with a dose of humor. The core claim is that we optimize Nix and nixpkgs for social comfort and broad appeal, and we pay for it in technical ambition. Further reading: How to piss off your Nix friends . Looking forward to next year. Three talks in two days was a little ambitious, but I would do it again. Everything lives on my talks page alongside their slides and the rest of my talks. He was pleasantly surprised and happy to hear that Nix also has reproducible builds that start from stage0 .  ↩ He was pleasantly surprised and happy to hear that Nix also has reproducible builds that start from stage0 .  ↩

Open Source DevOps Security

▲ 0

0 views

Farid Zakaria 2 weeks ago

nixpkgs-multiverse: fast mode

“The fastest evaluation is the one that never happens.” – Sun Tzu, The Art of Evaluation nixpkgs-multiverse gives you every version of every package that ever shipped in Nixpkgs from a single flake input. Note It continues to blow my mind that this is even possible. It feels like it suddenly unlocks a new dimension of Nixpkgs, and I am still trying to understand what it means. I think this capability is a fundamental change to the way we think about Nixpkgs, and it is not just a new feature. It is a new way of thinking about the entire ecosystem. There was always a penalty at the center of it. Asking for a specific version of , such as , meant fetching the whole ~378 MB Nixpkgs tree from 2021 and evaluating it to determine the . What if we could skip that evaluation? What if we could just ask for the path directly, and have Nix fetch it from the cache if it is there? This is a common idiom if you have ever used . That requires knowing the store path upfront. nixpkgs-multiverse now has a attribute that does exactly that: it gives you the store path for every indexed version of every package. This lets you skip the download and evaluation of Nixpkgs and get the store path straight from the cache. No Nixpkgs is fetched. Nothing is evaluated. No experimental features and no needed for this to work. The complete Nix API, except for releases, works with this fast path. If you want to learn more read the docs about the feature. Every channel bump published a listing of every path Hydra built for it: , or a for back in the pre-2017 era. These files are still available, and they are the source of the multiverse index. The listing is a map from derivation name to store path. The multiverse index is a map from to the revision that shipped it. By joining the two, every historical version gets a concrete address: Knowing the path is not enough, especially in the Nix language. We need to convince Nix that a string that looks like a store path actually is a store path. exists but it is an impure function and requires to work. How do we get around this? We attach “context” to the String. Context is the invisible baggage a String carries in Nix. When you interpolate a derivation into a String, the result remembers where it came from, and that is what makes realise the dependency instead of writing a dangling path into a script. lets you attach it by hand. The identifies that “this String names a store path that must exist,” which is exactly what produces for a path already in your store, except this works for a path that is not in your store yet and is not in this evaluation’s input closure either. Loopole! 👿 We then wrap that in an attrset that resembles like a derivation and the Nix CLI is satisfied: This is tomberek ’s trick from fastpkgs , and it is an amazing trick to circumvent needing . Everything about this remains pure evaluation, and the resulting graph is gauranteed to be bit-for-bit identical to what Nixpkgs would have produced if it had been evaluated. The only difference is that we skip the evaluation of Nixpkgs itself, and instead use the store path directly. The eval path derives the address, the fast path remembers it. A “fake” ( ) derivation has no , because there is no behind it. Nothing can build it and it can only be substituted. The CLI often wants a though when you hand it a derivation attrset, so we must make sure to append the output (i.e. ): and need a real derivation. Every fake derivation carries a lazy that is the real, revision-exact derivation: In the spirit of trying to keep my index small, is empty, so there is not additional information about the package. You can still get the from the real derivation by using as well. This scheme rests on cache.nixos.org still serving thirteen-year-old paths, which thankfully it does and with the same signing key. To demonstrate that the cache is offering nearly every path that Nixpkgs ever built, I ran a census of every indexed version of every package and asked the cache if it was still alive. As of August 14 2026, all 271,187 of them are alive . All of them, down to every NAR payload file. That is 14.8 TB of unpacked software from 2013 onward, one fast command away. 1 There’s some other data on nixmultiverse.com about the census, dependency graphs and additional features. Check it out! All of this is also available via the mvs command line tool as well for offline use. I guess now there is a caveat: there is now a trick in the multiverse. It remains mostly an index, some JSON, and a behind a memo table. The clever trick is tomberek ’s, and it is three important lines. The NixOS infrastructure has never garbage collected the binary cache. It is an S3 bucket that only grows, and the bill is paid by the NixOS Foundation and its sponsors.  ↩ The NixOS infrastructure has never garbage collected the binary cache. It is an S3 bucket that only grows, and the bill is paid by the NixOS Foundation and its sponsors.  ↩

Open Source Performance Programming JSON

▲ 0

0 views

Farid Zakaria 2 weeks ago

nixpkgs-multiverse is audacitymaxxing

Every package manager on earth picks one version for you. Nixpkgs picked one too. It never had to. I shared nixpkgs-multiverse recently: one flake input that hands you every version of every package that ever shipped in Nixpkgs. I love how unbelievable audacious Nix lets me be, audacitymaxxing . As of this writing, you have access to 31,783 packages and 304,484 distinct package version pairs pulled from 1,537 revisions. 🤯 The fact most distributions only give you one version of each package is not a bug. It is often considered a feature: a single self-consistent set of software that boots and runs together. It falls directly out of a shared global filesystem, the filesystem hierarchy standard (FHS), like , and . The purpose and existence of Nix is to eschew from that convention and allow multiple versions of the same package to coexist. Nixpkgs is a distribution built on that capability, and yet, it has been doing the same thing as every other distribution: picking one version of everything. nixpkgs-multiverse only supports, at the moment , top-level attributes that are packages but already the sheer volume of installable software dwarfs . 1 Nix’s answer to the FHS was audacious in 2003 and is still audacious now. A package lives at , where the hash is derived from every input that went into building it: the intensional model . How audacious are we? How about 246 distinct CPython versions, from 2.6.8 forward, all installable side by side, all built and cached, all addressable by version number instead of commit hash. 2 To re-iterate, these are distinct versions of CPython, including their transitive dependencies. There is no or or that is shared between them. 3 They work just as reliably as when they were first released, and they are all still installable today and can be substituted from the cache. People want to pin to a version. Upgrading software can be disruptive, and some people have to stay on a particular version but that should not impede the rest of the world from moving forward. The nixpkgs-multiverse helped solve one of the oldest devenv.sh issues, cachix/devenv#16 , the desire to pin a specific package. “It is not really practical to pin a separate version of nixpkgs for every different version of a tool needed in a dev environment. Normally we have at least 20-30 different tools all with a specific pinned version that we would want to specify.” – itpropro The issue, “Pinning a specific package”, was opened on 2022-11-10 and is now closed. devenv now documents the multiverse as the solution. 💪 The audacity of the multiverse is not technical. Nix took care of that. There is no clever trick in here; it’s 5 MB of JSON, about 200 lines of Nix and a behind a memo table. The audacity is in the premise. Two smaller things landed that I like and which was driven by feedback from the community. A soak period. gives you the whole of as it stood N days before an anchor, a cooldown window, in the spirit of Determinate Systems’ cooldowns , except the anchor can be any selector takes. Provenance. Every package set carries where it came from, so a you were handed can be interrogated rather than guessed at. The data was fetched from the Repology repository size map.  ↩ The data for other distributions was fetched from Repology .  ↩ Unless they happen to dedupe due to their hash.  ↩ The data was fetched from the Repology repository size map.  ↩ The data for other distributions was fetched from Repology .  ↩ Unless they happen to dedupe due to their hash.  ↩

Open Source Linux DevOps JSON

▲ 0

0 views

Farid Zakaria 3 weeks ago

nixpkgs-multiverse: every version that ever existed

Enter the Nixpkgs multiverse. All the versions that ever existed, all in one place. I bumped the release for my NixOS configuration to refresh many of my packages and found that a package I depended on at a particular version is no longer available. The package was “version bumped forward” in a way that broke some of my tooling. It’s late and I don’t want to fix it, so I just add another input pinned to the commit that had the version I want. This works, but it is miserable in a way that compounds. The need for the most recent package is so common that I had keep an overlay that would inject as a package set for me to easily pull from. If I have a need for a particular version of a package and it’s not present in my current , I am left searching for the commit and pinning it. 1 Every pin is a whole extra in the file. Flake inputs are fetched eagerly even if not used. A flake with three inputs whose output references only the first are all materialised. Nix lets us easily create a closure that reproduces a specific version of a package, but Nixpkgs makes it hard to hold one package still while everything else moves. Each Nixpkgs input to a flake is a distinct universe. If we can have multiple Nixpkgs as input to achieve fetching a particular package, why not have every version that ever existed always available ? 🤯 nixpkgs-multiverse is one flake input that gives you all of them at once. We can query the flake for all the versions of a package that ever existed in Nixpkgs . If we want a specific complete revision of Nixpkgs we can use the function. That is access to all the versions of all the packages that ever existed in Nixpkgs. You can mix them all together in one shell, one package or a build environment. How is it possible to have multiple Python versions? That is the whole point of Nix itself. Every package immaculately describes its dependencies using a hash via the intensional model . 2 Nixpkgs already supports multiple versions of a package in a single revision (i.e. , , ) as separate attributes. We took this to its logical conclusion of making them all available easily. Our deliberately has no inputs: . Inputs are fetched eagerly, and we have 1,393 of them. We need to fetch them lazily, only when something actually references a revision. To do this, we fetch revisions with , pinned by , only when needed. Two files do all the work: and . is one ordered array of every revision from Nixpkgs , 1,393 as of this writing, from 2017 to 2026. 3 We limit our commits to those that were actually built and cached by Hydra, so we only include commits that were either a release or a channel bump. How do we know which revisions to pick for the ones? We rely on the nix-releases S3 bucket to tell us which commits actually became published builds. The S3 bucket uses the commit hash as the directory name, so we can list the bucket and get a complete list of all revisions that were actually built. is the map from (attribute, version) to a revision: That integer is an offset into . It is the most recent revision that shipped that version. At this many revisions, it turns out that how you encode the data matters a lot. My first encoding stored every revision a version appeared in. Although it was simple, it was a disaster in terms of size for these JSON files. As you might expect, most versions of most packages are unchanged across many revisions. The size of our file was growing linearly with the number of revisions. By storing only the newest revision that shipped a version, we can keep the file small and still answer the question “which revision had this version”. Here is how it actually grows as revisions get indexed: 5.18 MB covering 1,393 revisions and 289,521 distinct (attribute, version) pairs. The key design rule for our flake: Cost is per revision touched , not per package. If we were to add revisions as inputs, evaluating our flake would explode. Each flake in our measurement below has N inputs and an output that references only the first one ; the timing is how long before that output evaluates. 4 Five pins that are not used cost 26 seconds before the output evaluates. Each input costs about 5 seconds, and the input is fetched and materialised even if never used. In contrast, the green line is with 1,393 revisions available , which is a flat 0.20s to parse the JSON. 🤩 Revisions are memoised, so pulling 3 packages out of one revision costs the same as pulling one. That concept that the can hold many graphs of the same package is core to understanding Nix. The popularity and rise of flakes made it even more apparent that we can mix multiple revisions of Nixpkgs together. The thing I keep coming back to is that Nixpkgs history already is the multiverse. Every version that ever existed is already built, already cached, already reachable. It was just addressed by commit hash instead of by version number, which is exactly backwards from how anyone thinks about it. The whole project is 5 MB of JSON and about 200 lines of Nix. It does not build anything, mirror anything, or host anything. It is a phone book. Thankfully sites like nixhub.io or lazamar’s search make this a little easier.  ↩ The hash is a unique identifier for the exact set of inputs that were used to build it. If you change any input, the hash changes and you get a new package.  ↩ A NixOS release is not special; it is a commit that happens to carry a label.  ↩ Everything is against a local clone, so there is no network latency.  ↩ Thankfully sites like nixhub.io or lazamar’s search make this a little easier.  ↩ The hash is a unique identifier for the exact set of inputs that were used to build it. If you change any input, the hash changes and you get a new package.  ↩ A NixOS release is not special; it is a commit that happens to carry a label.  ↩ Everything is against a local clone, so there is no network latency.  ↩

AI Python UI Shell JSON

▲ 0

0 views

Farid Zakaria 3 weeks ago

Super Mario Derivations

One of the most surprising aspects of the Nix language is that it is lazy , especially if you have never used a lazy language before. This laziness is what makes much of Nixpkgs possible, and its complexity. One of the simplest ways to observe the laziness is by understanding that only the attributes you access are evaluated. The more whackier version of this is you can have endless recursion in an attribute set. Nixpkgs is filled with these bottomless attribute sets: The same store path every time. contains itself, and so does every package set inside it. 🤯 If laziness is what lets a recursive attribute set terminate, then the recursion doesn’t have to bottom out at all : That attribute set is infinitely deep. Indexing three levels into it costs exactly three levels of evaluation, and the rest of the infinite tree is never built because nobody asked. So an attribute path is a walk through a lazily-generated tree. Which made me wonder: what if the attribute path were input to something ? 🤔 I decided to take that idea and make the attribute path a sequence of button presses in Super Mario Bros. 3 . Each node in the tree is a frame of the game, and each child is a button press that produces a new frame. Game states are recursive by nature. is right + B, which in Super Mario Bros. 3 is “run right”. is run and jump. The output is the frame you’d be looking at if you’d pressed those buttons in that order, on real hardware, in that game. 1 Append anywhere along the path and you get the whole run stitched into a recording: The coolest thing though is that every one of those frames is a separate derivation in my store . The code is at fzakaria/nes-nix . It is generalized and the ROM is a flake input you point wherever you like for any other game. The flake computes a derivation based on the attribute path such that each press is its own derivation, and it takes the previous press’s savestate as an input . Each derivation never re-emulates its ancestors’ frames. 2 The practical consequence is that the store becomes the emulator’s savestate history: Branching off the middle of a hundred-press run costs one press as does appending to the end of it. We can look at it the other way. The dependency graph is the input sequence, so we can ask Nix what buttons produced a frame: So what is actually doing? Almost nothing. Every frame along the path is already sitting in the store as the output of its own press, so the recording never emulates anything. It is a directory of symlinks to the frames for to process. How far can we take this input-sequence game input idea? Nix by default gives out at around 2,400 presses, with: defaults to 10,000 and evaluating each press costs roughly four nested calls. It’s a guard against runaway recursion, not a structural limit, and we can raise it to 10 million and get 20,000 presses: 20,000 presses, takes roughly fourteen seconds to evaluate on my laptop. The cost is linear in the number of presses, and it is roughly 0.7ms “per press”. The next bottleneck though is that the kernel gives out at 21,845 presses on my machine. An attribute path is a single element, and Linux caps the size of the argument list in total and individual arguments. The per-argument limit is 131,072 bytes ( ), and each press is six bytes long ( ), so 21,845 presses is the maximum that can be passed to as a single argument. The escape hatch is to stop passing the run as an argument. and we can feed in the input-sequence as from a file: This produces the byte-identical derivation to the equivalent attribute path, so a run kept in a file still shares the same store paths. All of this was to simply evaluate the Nix expression. Now we have to build it. Although Nix is great at building derivations in parallel, the recursion here is tail-recursive and therefore serial. I benchmarked the build time of a growing list of button presses and the cost is also linear, as we would expect, with the number of presses. The cost per press is roughly 1.27 seconds with substituters enabled and 0.28 seconds with them disabled. The round-trips cost for checking whether the derivation is in the cache costs noticeably more than emulating the frames does. 3 We’re used to the attribute path being a name , simply a coordinate into a catalogue of things that exist. Laziness means it’s really a program : a sequence of steps the evaluator walks, generating whatever it needs as it goes. Nixpkgs happens to use that machinery to describe software, but nothing about it requires that the tree be a catalogue at all. Coupled with the fact that the store turns out to be a decent persistence layer for reproducible state-machines, makes a our “package manager” reasonable to use for playing Mario. 🍄 The prefix is a precanned sequence of button presses that gets you to the start of level 1-1.  ↩ A screenshot of the frame is also produced, which is used when we want to stitch a video sequence together.  ↩ We can set or if we want to avoid this cost.  ↩ The prefix is a precanned sequence of button presses that gets you to the start of level 1-1.  ↩ A screenshot of the frame is also produced, which is used when we want to stitch a video sequence together.  ↩ We can set or if we want to avoid this cost.  ↩

Programming

▲ 0

0 views

Farid Zakaria 4 weeks ago

A C++ toolchain from 357 bytes, in Bazel

I have been fascinated and amazed by stage0 for a while now ever since I learnt about it via Guix using it to provide twenty two thousand packages source-bootstrapped from the 357-byte seed. What is stage0? It is a chain of compilers and assemblers that can be built from source, starting from a 357-byte program that can eventually build a recent GCC. 1 Since then, NixOS and other distributions have also adopted the same approach to minimize their binary seed which makes it possible to onboard new architectures and platforms much simpler. What’s always frustrated me as a Bazel (& Buck ) user is the reliance on prebuilt toolchains even for things that should be built from source easily like protoc . Bazel has given up trying to provide a hermetic C++ toolchain and the upstream rules_cc ruleset just points you elsewhere: Configuring a hermetic toolchain makes your build more deterministic. rules_cc itself does not yet offer a hermetic toolchain distribution I had attempted to provide a stage0 hermetic C++ toolchain in October 2024 via https://github.com/fzakaria/stage0-bazel . I made substantial process through the bootstrap process but I did not make it far enought to be usabale. To be honest, I was also a little disheartened that no one else in the community thought it was the greatest thing since slice bread. Everyone seems to be content with using prebuilt toolchains as they go deeper into MODULE.bzl madness . I had put it aside for a while, but I have been thinking about it again recently. The steps are mechanical and the process imitates existing distributions, so this became a perfect project for me to throw at an LLM to finish. 2 You can now leverage the toolchain to build in Bazel and have it compiled by a toolchain whose entire ancestry is in the repository from that same 357-byte seed . 🎆 How complete is this toolchain? I pointed the toolchain at Abseil and GoogleTest straight from the Bazel Central Registry without any patches . We then can build and run their testsuite to provide a sanity check that the toolchain is working correctly. We use a to filter tests that require . Abseil marks as a , and Bzlmod drops dev dependencies of non-root modules. That is us building Abseil and GoogleTest, from the registry, unpatched, compiled by a toolchain that began as 357 bytes of hex. How can I be so sure this is a hermetic toolchain? The toolchain includes an audit report that uses Bazel’s aspects to inspect every action in the build graph and verify that it only executes programs built by the toolchain itself. The report is generated by running and will fail if any action executes a program outside of the Bazel output tree. 3 The report is two lines long: Unfortunately, since runs a shell it takes as an absolute system path that is also listed as a seed binary. ’s attribute is a string, and the shell is not a declared input of the action, so no artifact this repository built can provide it. Building toolchains from bootstrap seeds was never a priority for companies like Google where they control the entire build environment. However we seemed to have adopted the same approach as Bazel and similar build systems have become more popular in the open-source community. We should strive to make our builds more reproducible and hermetic, and this is a step in that direction. Once you can reach a recent-enough GCC, you can build any C/C++ program and beyond easily.  ↩ Consider this the disclosure that I used an LLM to help me write the remainder of the toolchain.  ↩ We also set to disable Bazel’s built-in C++ host toolchain detection.  ↩ Once you can reach a recent-enough GCC, you can build any C/C++ program and beyond easily.  ↩ Consider this the disclosure that I used an LLM to help me write the remainder of the toolchain.  ↩ We also set to disable Bazel’s built-in C++ host toolchain detection.  ↩

Open Source C++ DevOps Shell

▲ 0

0 views

Farid Zakaria 1 months ago

Nix finally has a source-bootstrapped OpenJDK

One of the earliest requested issues we had opened on GuixPkgs was to add more packages, specifically OpenJDK. “Specifically I would like to see openjdk translated. openjdk is not bootstrapped from source code in nixpkgs” [ issue#3 ] Ever since I learned about the stage0 bootstrap chain and how Guix announced full-source bootstrap for all packages in 2023, I was in awe. They provided a package graph of more than 22,000 nodes rooted in a 357-byte program, including the JDK. 1 GuixPkgs can now build OpenJDK 25 🎉 Does it really work? Let’s take it for a spin. lives in the output, since Guix splits the package. is written in Java. is C++, but the class library it needs is Java, and the compiler that compiles the class library is Java, and it runs on a JVM that needs a class library… and so on. This is a bootstrapping problem. Every distribution resolves this the same way in practice: download a JDK and use it to build your JDK. Debian documents the pain , and the Bootstrappable project has a whole page on it. They are fun reads, I highly recommend them. What makes JDK special is that there is no actively maintained JDK that can be built from source without a JDK. The authors had to go back quite a few years to find one, and bring it back to life. Nixpkgs does the same. is built by : a 135 MB prebuilt-tarball. Nixpkgs makes it pretty easy to audit in the meta.sourceProvenance of the package. What does the source provenance of GuixPkgs’ look like? It’s a little whacky but the overall build chain in Guix is the following: Nineteen complete JDK builds, from a C++ program. We can use to emit the entire closure as a dot file to visualize the differences. I restyled the nodes and edges: dots instead of labelled boxes, and the red to highlight the derivations that exist only because of the JDK bootstrap. The upper panel’s entire Java story is those two red dots and the one edge between them: → . The lower panel’s is the 33-node constellation. Why is the Nix graph still so big if I claimed it was from a binary distribution? It turns out that a JDK closure is mostly not Java. It is a large C++ program that wants X11, cups, fontconfig, freetype, alsa and zlib, sitting on a C toolchain that both sides bootstrap from source. That sub-graph is the same in both and it swamps everything. I was curious to zoom-in and see the source-bootstrap portion of the derivation graph. Since that shared sub-graph is drowning the signal, I chose to filter it out. Which derivations exist in this closure only because of how the JDK is bootstrapped? 🤔 That is a reachability query. We delete the Java-provenance nodes from the graph, see what is still reachable from the root, and whatever is left are the derivations that exist only because of the JDK bootstrap. This lines up with our intuition. Nixpkgs only needs two derivations to bootstrap the JDK, while GuixPkgs needs 876 . Note Interestingly, the bootstrap build for JDK has in its closure: IcedTea 8 wants GTK 2, which wants its own Mesa, which wants 😱 What started off as a fun art project , started during TacoSprint 2026 , has now found relevance to those interested in bootstraping and reproducible builds. Nixpkgs has made a lot of progress on this front as well, however it is still not fully bootstrapped and relies on plenty of prebuilt binaries.  ↩ jikes 1.22 (C++) compiles GNU Classpath 0.93 , a free reimplementation of the Java class library. Classpath is enough to bring up JamVM 1.5.1 , a JVM written in C. Now something can run Java. Now we can finally use Java tools: Ant and ecj , the Eclipse Compiler for Java. Then it doubles back and rebuilds all of that with the newer versions to get more modern JDK features. That stack finally compiles IcedTea 2.6.13 (OpenJDK 7) → IcedTea 3.19.0 (OpenJDK 8) → OpenJDK 9 . After which it is one rung at a time: 9 builds 10, 10 builds 11, …, 24 builds 25. Nixpkgs has made a lot of progress on this front as well, however it is still not fully bootstrapped and relies on plenty of prebuilt binaries.  ↩

C Programming C++ Java

▲ 0

0 views

Farid Zakaria 1 months ago

Guix by Nix

I have been working more on GuixPkgs in preparation for a talk at nix.vegas for DEFCON34 . At the end of my previous GuixPkgs post I left a teaser: We can then build a NixOS machine where every package is the Guix equivalent 😱. Well, adeci 1 took the bait and we went even further than that. 😈 Say hello to Guix by Nix : a bootable VM where the kernel is Guix’s Linux-libre, the userland is translated Guix packages, and PID 1 is GNU Shepherd . No systemd. No D-Bus. No NixOS activation. Not even a or binary in the guest. It’s Guile all the way down and Nix built all of it. Log in with / and poke around: As a reminder, even though these binaries live in , they are not Nixpkgs packages. They were translated from Guix derivations using guix-transfer and built by the . That was built from Guix’s package definition, source bootstrap and all , but it lives in , because built it. If you want to try out any of these packages on your own machine, you can use GuixPkgs . Tip Guix offers all packages built from source where Nix may offer it as a prebuilt binary. You can use GuixPkgs to get a source-built bootstrapped version of OpenJDK for example and all the whacky steps to get there. How does this actually work? The project is a three-stage pipeline: Warning AI was leveraged to write the initrd and activation scripts. That seems to trigger people lately, so consider yourself warned. Every program that can be executed, every ELF file, every script interpreter, all traces back to a translated Guix derivation. The only things Nix authored are text files: the init scripts, the Shepherd config, . This is Guix by Nix . “Every executable byte comes from Guix” is exactly the kind of claim that’s easy to say and easy to fudge. 🤥 A booted demo is great but that can’t prove it. A VM where secretly came from Nixpkgs boots identically . The flake ships an audit-check that classifies every store path in the shipped closure by provenance, unpacks the compressed initrd 2 , and inspects every executable payload. The audit derivation fails if anything is unclassified, any ELF file or script interpreter doesn’t trace to a translated Guix output, any reference survived translation, or anything systemd-shaped appears anywhere. The report includes a lot more information such as the exact Guix channel commit everything was translated from. There’s also some NixOS VM tests for good-measure. 🕵 Note The kernel is Guix’s , translated and built by like everything else. Nix wraps it in a thin adapter so NixOS’s VM tooling accepts it by augmenting with some additional metadata only; the is byte-for-byte Guix’s. One obvious next step would be a normal NixOS machine, where every package that exists in GuixPkgs shadows its Nixpkgs equivalent on . GuixPkgs offers such an overlay, but it needs quite a lot of CPU to build all of it…. A more realistic use case is to mix and match GuixPkgs and Nixpkgs packages in a single system. Get the best of both. I heard there were people in the Nix community who still want a non-systemd system, à la sixos . 🫠 For now, this project remains a minimal VM demo. If you make it PID 1 on real hardware, please send photos. 🙇 It has been extremely fun and rewarding to work with Alex on this project. We nix-pilled him at PlanetNix 2 years ago and since then he has been pushing the boundaries of what Nix can do and is currently employed at Shopify working on Nix.  ↩ Nix’s reference scanner can’t see inside archives and sneaky paths hide there.  ↩ guix-transfer is the tool that translates Guix derivation graphs into Nix derivations. GuixPkgs is the flake with all Guix packages, all built from the 357-byte seed, although there is a Cachix cache provided. Guix by Nix assembles ~42 of those packages, a subset of the overall set, into a useable machine. The initrd is a custom shell script, interpreted by Guix Bash, that loads eight modules, mounts the root disk and the 9p store, and s. Activation (accounts, , the setuid copy) is another custom Nix-generated script run by Guix Bash. PID 1 is from Guix, with a small Scheme config that starts , , , and a serial . works like you’d expect. is compiled from Guix’s own search-path specifications, so and friends point where Guix intended. It has been extremely fun and rewarding to work with Alex on this project. We nix-pilled him at PlanetNix 2 years ago and since then he has been pushing the boundaries of what Nix can do and is currently employed at Shopify working on Nix.  ↩ Nix’s reference scanner can’t see inside archives and sneaky paths hide there.  ↩

Shell Programming DevOps Bash

▲ 0

0 views

Farid Zakaria 1 months ago

Seriously, what is the large code-model even for?

I have been working on making massive binaries possible at . One of the Hail Marys that you should be able to rely on is the large code-model ( ) as it makes no assumptions about size and distance of relocations. Large code model : The large code model makes no assumptions about addresses and sizes of sections. [ cite ] In a previous post , I documented how I hit simple performance bottlenecks that made me believe that the code-model is largely theoretical in practice. In those cases, the fixes were evident and relatively small; however their omission was a hint at how no one uses the code-model because without them the performance penalty was a non-starter. I hinted at some other ways I found the large code-model to still be lacking however I had not yet fully understood the pain of those failure modes. I left a small teaser near the bottom about Thread Local Storage ( / ) being one of the “fun failure modes”. Turns out that it is worse than I thought. The instruction sequences the compiler emits for TLS are 32-bit by construction , and has nothing to swap them for. 🤦🏻‍♂️ To put that very bluntly, is incapable of building complex large binaries despite its stated goal. The annoying part of this whole line of research is producing a test subject. Emitting 2GiB of real instructions into is genuinely painful, you’d have to generate and assemble billions of instructions, and the object file is enormous. Beyond that the cardinality of the problem space grows when you consider whether the code is position-independent, do we use a procedure-linkage-table (PLT), TLS, GOT and all the various nuances each compiler brings with how they layout and order sections in their default linker scripts. But relocation overflow isn’t about how many bytes are on disk, it’s about virtual-address distance between a reference and its target. So we have a few options. Uninitialized globals land in , which is a section: it occupies virtual address space but zero bytes on disk (the same sparse-file trick I wrote about in massively huge fake files ). So a few thousand 1MiB arrays gives us gigabytes of address space essentially for free. We can synthetically produce this pretty easily with the following script. Link it with the (default) small code-model and it falls over exactly where you’d expect, at the 2GiB signed-32-bit boundary: sits at = precisely 2GiB. Recompile the same source with and it links cleanly: The large model did its job: it replaced the 32-bit references with 64-bit sequences: a of a 64-bit offset + an . The trick is great for exercising the linker, but it’s a little unsatisfying and overly synthetic. I often want to mimic relocation failures as they traverse a large segment. We can leverage the assembler’s directive to repeat instructions. We can generate a tiny source of N functions spaced 1MiB apart with a sea between them, plus one dispatcher that s every function. This requires each to have a relocation and the calls to functions past the 2GiB mark overflow. 🔥 Now let’s do the exact same thing, but using . When you access a thread-local variable, the compiler doesn’t just load an address, it emits one of four access models , from fastest/least-flexible to slowest/most-flexible: Notice the pattern: the value that reaches the thread pointer can be 64-bit (i.e. it lives in a GOT slot), but every instruction that participates in a TLS access uses a 32-bit field . , , , are all 32-bit. 4GiB of , still tiny on disk. Now look at the relocations the large code-model generated: Every single TLS access is which is 32-bit , even under . 🫣 The exact same large code-model that just happily linked 4GiB of ordinary fails on 4GiB of . This isn’t a GNU quirk either. LLVM ( & ) does exactly the same thing, albeit with ’s diagnostic being a bit friendly by printing the actual offset and the window it has to fit in: is exactly , sitting at the far bottom of the TLS block, and is the signed 32-bit window has to live in. even parks large-model code in a section. I was a little confused at first: exists . Why doesn’t the compiler use it, especially with ? A relocation type is glued to the specific field it patches and is determined by the code-sequence emitted by the compiler. Here’s the local-exec sequence emits: The offset is applied with . The x86-64 encoding for that instruction has only a 32-bit immediate field . There is no 64-bit form of it. The only relocation that can physically patch that field is a 32-bit one: . So where is used? It’s not in the code ( ) at all. It lives as a dynamic relocation on the GOT slot used by initial-exec . The code side only holds the 32-bit that points at the slot: patches the 8-byte (64bit) data word in the GOT. The instruction that reaches that word ( ) is a 32-bit PC-relative relocation. So even initial-exec has a 32-bit link in the chain: the GOT slot must sit within 2GiB of the code. 😭 local-exec is the mode you want for a big statically-linked executable’s own thread-locals. It’s the fast path: the thread-pointer offset is a link-time constant, so the access is a direct with no memory load, no GOT, no indirection. Unfortunately, is limited to 32bits and overflows. I would argue it’s also the mode most likely to need a large binary, since it’s what a giant static executable uses for its own TLS. What if we fall back to a slower mode? Unfortunately, each one of them has a 32-bit field too: The use of seems especially confusing. The large code-model already reaches the GOT at 64 bits everywhere else: GOTPC64 for the base, GOT64 for the slot, so the omission seems surprising. The 64-bit machinery is right there. TLS just doesn’t use it. 🥲 After having dug into it, it’s not simply a bug in GCC or LLVM. Look again at what emits for a single local-exec access under : It’s a plain . There is nothing stopping clang from emitting a 64-bit form instead: could patch that immediate, exactly the way already patches a immediate for ordinary data. Oddly, the relocation type is not the missing piece. What’s missing is a code sequence in the x86-64 psABI that uses a 64-bit TLS offset as an instruction immediate. The ABI simply never defined a large code-model TLS access model, so / only ever appear in GOT slots, never in code, and no toolchain can emit what the spec doesn’t describe. The large code-model “exists”, but it doesn’t actually deliver arbitrarily-large binaries. For thread-local storage it can’t, because the specification never defined how to. This is one more reason I’m working toward making massive binaries possible. If you want to follow along, the discussion is over in the x86-64-abi google-group where I’ve posted an RFC, and we have started an LLVM Massive Binaries working group and hold monthly meetings. initial-exec is 32-bit RIP-relative reach to the GOT slot. general-dynamic is 32-bit RIP-relative.

Backend Programming Performance

▲ 0

0 views

Farid Zakaria 1 months ago

Small site changes and new design

I have been wanting to refresh my site for a while now. I have even posted a few times on LinkedIn and X offering to pay if somoene was interested in taking a stab at it. I am sure this will put people off from reading my blog just stating this, but I decided to take a stab at some design changes with AI and some minor site improvements that I have always wanted to do. If you think the site looks much worse, please let me know . Some small improvements and quality-of-life enhacements: Look at me! I’m sidebar content I will undoubtedly abuse. I continued to try and give it my personal flair to avoid it looking like AI-slop but 🤷 A small entry in the footer to showcase the site is built with Nix. A Jekyll plugin that automatically creates source-sets for the images I use in my blog posts. A Jekyll plugin to create a short content hash for the stylesheets to avoid the browsers caching stale CSS. A Jekyll plugin that generates a little curved line for the masthead that is the of the page. Redesigned the layout to allow side-bar content.

Web AI Programming CSS

▲ 0

0 views

Farid Zakaria 1 months ago

Linux kernel will support $ORIGIN, sort of

For some reason, during TacoSprint 2026 I decided to see if we could tackle relocatable binaries in Nix. I enjoy these lofty goals to push Nix and the surrounding ecosystem forward. I am bold if not stupid . I left the last earlier post with one potential idea of how to get there: We could patch the Linux kernel so that $ORIGIN is supported in PT_INTERP and the shebang. I waded through the complexity of sending patches over email (turns out I actually enjoy this workflow!), and sent a proposal to the Linux kernel mailing list. My first attempt here proposed simply adding direct support for in the Virtual File System (VFS) subsystem. I waited nervously. I was expecting the result from what I had come to read about online; someone non-politely telling me to F$#CK OFF because there is something I missed, misunderstood or did not consider. 🤬 The result was completely different. 😲 Christian Brauner , the maintainer for VFS responded to me in good faith, asking for the rationale for the change and eventually proposing some ways in which such a support could make it into the subsystem. Note It definitely helped having someone like John Ericson chime in and advocate why having a non-fixed interpreter ( ) is useful to Nix and other use-cases (i.e. Buck & Bazel). He offered that potentially we could leverage eBPF as a programmable way to select an interpreter through binfmt_misc . I wanted to merely allow but a programmable selection could let us do anything! The idea must have really intrigued him because soon-after, on his vacation , Christian offered the first draft of such a solution. We went back and forth a little over the mailing list and the end result is a patch series that will make its way into branch in the near future. If you don’t know what eBPF is or , WTF did we just collaborate on? Let’s take a look! I won’t do eBPF justice, and there are plenty of articles online about it as it’s quite in-vogue at the moment. tl;dr; You can write programs in a C subset that gets compiled to an instruction set whose virtual machine is running within the kernel . Shouldn’t the kernel be super fast? Yes, the programs are jitted to their native CPU architecture and the programs have a fixed-time slice. Isn’t this some crazy vulnerability for the kernel? Before any code is loaded it is “verified” to be safe. Checkout this guide for more info. We can now support with a relatively simple eBPF program: Once the above program is loaded and registered into the kernel, we then ask the subsystem to trigger it. Checkout this thread if you want to see the complete example. What does that mean? It means that every binary now triggers the function above, in this case any file, but it could be executables with a new segment like , and the kernel will ask to determine the interpreter to use dynamically. Our special BPF program has support for 💥 What else could you do? Well we can now even completely replace the traditional QEMU registration script with a BPF program now like this one . What else can we do? Since we can now programmatically select our interpreter based on anything in the file, we can do quite a lot. I’m keen to hear your suggestions and ideas 💡. Some of the smaller items are that we can even support in the shebangs ( ) very easily as seen here : we simply look at the first 256 bytes of the file and look for to trigger. One downside or side-effect of the traditional hand-off was that the way in which the desired final binary was invoked was non-transparent . The registered interpreter becomes the process. It owns the entire process identity, and the binary you actually asked to run gets demoted to an argument. For or that’s acceptable as they are emulators but for a per-binary BPF loader that might pick a traditional it does not make much sense. This leaks in a few painful ways but the simplest are : Christian sent a large patch series for this as well. His latest patch series adds two new dispatch modes that close the gap from opposite ends and covers a few other gotchas that these modes can fix. The loader substitition is the one I’m most excited about for Nix. With the flag, the kernel executes the matched binary natively as the main image, and merely substitutes the registered interpreter for the loader named in the binary’s . stops being a hand-off and becomes a plain override. There’s no contract and no identity to reconstruct, so a stock dynamic loader works unchanged . Where does this leaves us? I’ll be tracking the Linux kernel releases and, once this lands in and ships in a tagged release, I plan to upstream a NixOS module that registers the support at boot. 🎉 The plan is to gate it on a new segment rather than matching every file. That keeps things backwards compatible : the BPF handler only kicks in for binaries that explicitly opt-in by carrying the new segment. This means Nix produced binaries continue to work without the BFP handler but those that have it may elevate themselves to relocatable status . A ship in harbor is safe, but that is not what ships are built for. — John A. Shedd and show the interpreter invocation, not what you executed. names the interpreter. Relocatable programs commonly locate themselves through , and instead they find the dynamic linker. 😩

Open Source Linux DevOps

▲ 0

0 views

Farid Zakaria 1 months ago

How to piss off your Nix friends

Warning If you are pissed off reading this blog post, I guess, mission accomplished ? Try not to take life too seriously. Unfortunately, there’s a lot worse things in life than my opinions on Nix. Seems like it’s all too easy to get people in Nix flustered, angry and out with their pitchforks. All it takes is someone proposing a markdown file for the community to lose its mind. Having been a member, exposed to and part of the Nix/NixOS community for many years, I thought I would share some personal opinions , some which are self-evident and others that are purely philosophical. Nix is brilliant and deeply flawed. Both things are true at the same time. The documentation is infamously terrible (although getting better!), the language is foreign to read for many and there are still growing pains from the governance changes. You should be free to say all of this and yet still believe Nix to be the best idea in the industry, at least this decade. The emperor has no clothes. “If we just fix the documentation and the onboarding, then there’s no stopping Nix & NixOS” - Most Nix users Nix is “having a moment”. Growth on any measurable metric is growing. There are endless posts about it. Part of human nature is the desire to win. As a result, there is a persistent fantasy that if we just fix the documentation, just smooth the onboarding, just ship a friendlier installer and so on, then everyone, even your grandmother , can be using NixOS. Nix is not meant for everyone. Nix is a power tool. Power tools have a learning curve and can occasionally take a finger. I continue to use a non-NixOS Linux machine in addition and guess what, it works well enough. It is surprisingly stable and despite what Nix leads us to believe, not everything is on fire. Our obsession with mass adoption has warped priorities and diluted the amazing possibilities Nix could allow by having to make it more palatable for broader appeal. Nix originated in Europe. It should be no surprise that it skews heavily European. We can easily affirm this from the 2025 community survey . Europeans are different from Americans. We hold different cultural values and priorities. Both have traits that I wish the other emulated. Unfortunately, where they clash however is often a point of contention. Americans are capitalist maximalists. The idea of the “American dream” is tied to it. We also have the largest military budget in the world. You’re rarely more than a degree or two from a through-line between a business and the military. This clashes with much of the European worldview. As a result, there’s a bit of an undercurrent where being an American or an American corporation is problematic in the community. Your position is suspect from the start, assumed to carry ulterior motives and it curdles into a purity test. I am clearly in the “AI is useful” camp. I have written before about how much LLMs have unlocked for me personally. The Nix ecosystem is probably the best poised for the AI-wave. I have found a newfound joy and love for my NixOS machine now with LLMs. All those weird quirks that bugged me I have been able to resolve, and declaratively reproduce for future generations. One-off AI written tools can be written and stored in my NixOS configuration with a sense of assurance that they will not collide or interfere with the rest of the system. Unfortunately there’s a loud contingent that treats AI output at best as technically unsound and at worst as some moral failing of the user. Despite nixpkgs offering AI tools, an AGENTS.md file was seen as heresy. I particularly enjoyed Linus Torvalds, on a kernel mailing list articulating better than me Linux’s position on AI: AI is a tool, just like other tools we use. And it’s clearly a useful one. […] Anybody who doubts that clearly hasn’t actually used it. These are tools we can use to push Nix & nixpkgs further. The democratic process is great for society but a software project originated from someone . Someone had the vision, birthed the idea, worked tirelessly on it and then attracted others to contribute to their ongoing vision. In the case of Nix, Eelco created Nix in 2003 as part of his PhD research. The NixOS foundation didn’t exist until 2015. That is over a decade of work towards a project driven by his own vision as the Benevolent Dictator for Life (BDFL). A non-democratic model works especially well in open-source because you are free to fork the software and try your hand at your own ideas if you disagree – the same cannot be said for our shared geography. A clear vision, whether you agree with it or not, is refreshing. Someone who can say yes or no and is not simply stonewalled by design-by-committee. DHH had said it poignantly well: “Using open source software does not entitle you to a vote on the direction of the project.” [ cite ]. Every hour spent making Nix pleasant on macOS is an hour not spent making Nix extraordinary on Linux. You would never catch an iOS developer working on Linux and yet it pains me to see those who target the Linux platform working on a Mac. For Nix, Darwin support is a bottomless tax. Closed toolchains, an SDK that shifts under you every release, a sandbox that fights you. All to chase an OS whose entire philosophy (opaque, proprietary, convention over purity) is the antithesis of Nix. When we have to target solutions that cover wildly different platforms, the end result is muddled and limited. Beauty, elegance and innovation emerge when you apply constraints and restrict a problem. Flakes are here to stay and yet their adoption is constantly brought up in order to validate its existence. I default to using it for my new projects, mostly because it’s easier at this point and it seems annoyingly tied to the new CLI format. Upon reflection though, I don’t feel like I have gained anything really over or . Subjectively my evaluations feel slower as now I’m fetching many more or jump through hoops to make every flake follow each other defeating the whole purpose of separate trees. Unless you are sharing a laptop with your family or are using a mainframe from 1980, you won’t have more than a single user on your machine. Despite this, the default installation we guide users towards is one designed for multiple users. The multiple-user install adds unnecessary complexity many don’t need or won’t understand: a build daemon, pool of users, systemd service, etc… In contrast, the single-user install is radically simpler to run, operate and triage. I don’t have to remember whether I am setting configuration for my “client” or the “daemon” and that there is a difference. Multi-user is the right default for a shared build farm. It is overkill for the single human it’s actually installed on the majority of the time. Nix is the closest thing we have to a solution where we can rebuild the entire world reproducibly. It is used by the Software Heritage Foundation as a way to reliably collect, preserve, and share all software that is publicly available in source code form [ cite ]. Some of the most amazing technology that exists in this world exists when one can make changes at multiple layers throughout the stack. This is the secret sauce to many of the hyperscalers of today. This is table-stakes for NixOS. You can implement a solution that requires new: application, compiler, library, runtime and kernel all within a single commit . Despite the ability to wildly diverge from traditional distributions, innovate and differentiate, we largely replicate the status-quo – albeit more reproducible. Whether it’s constraints imposed by needing to accommodate alternate platforms (e.g. macOS) or fearing alienating more novice-users we limit the potential of what Nix could do. Are you pissed off? Let us still be friends.

Open Source DevOps Linux

▲ 0

0 views

Farid Zakaria 1 months ago

Who does Anubis actually stop?

I have been working on a patch to the Linux kernel to support for the interpreter ( ) via bpf in [ thread ]. Of course I’m leveraging an LLM to help me do this! To pre-seed the context of the LLM, I asked it to read the https://lore.kernel.org/ thread. Uh oh. Looks like they have adopted Anubis , which is an HTTP proxy that requires proof-of-work before allowing access to the resource. Did this really do anything? Unfortunately, no. My AI diligently came up with anubis-fetch , which you can find at https://github.com/fzakaria/anubis-fetch . The tool tries to natively solve the proof of work or, as a last resort, will launch Chromium to visit the URL. This tool also impersonates a real Chrome TLS/JA3 fingerprint natively via req so it clears passive Cloudflare blocking too. ☝️ So who did we stop? The exact adversary Anubis targets defeats it trivially. The whole use of Anubis feels regressive and marginalizes those without access to “good” AI. For a scraper, solving the Anubis challenge is a one-time, amortized-to-zero cost since the cookie can be cached and reused. For a human, it’s seconds of spinner, battery drain on every fresh visit. They can’t amortize anything amongst each other. This “regressive tax” is paid even more so by those with weaker devices or who access the content on their phone. Clients that don’t leverage JavaScript (e.g., text browsers (w3m/lynx), screen readers, RSS readers) are completely left out. Did deploying Anubis stop any of the aforementioned bot-farms or are they mildly inconvenienced when they had to augment their bots to support a new proof of work solution briefly? The irony is that Anubis’s goal is to stop AI but it was incredibly easy for AI to circumvent it and yet the cost to humans and an open web remains. With the presumption Anubis is now a regressive tax, how much does it cost us? Every number here is a rough estimate. This is not a environmental argument at all since the bot-farmers and AI tools themselves are using many orders of magnitude more energy. Nevertheless, it’s interesting to see how much time is spent doing proof-of-work challenges that marginalize people. Difficulty is the number of leading zero hex characters the hash must have, so the expected work per solve is hashes. Difficulty 4 is the common default. Rates assumed: ~50 MH/s native (Go), ~0.5 MH/s in-browser JS; “felt” wall-clock includes page load, the worker, and the reload. Let be the number of Anubis challenge-solves per day, worldwide. Assume a felt time of and device energy per solve (screen + CPU). Collectively we are wasting an impressive amount of time waiting for access to websites; time we didn’t spend before the AI era. As a human, time is precious and finite to me, whereas to a robot it is not. Human-time / year = Energy / year (kWh) =

Security JavaScript Open Source

▲ 0

0 views

Farid Zakaria 2 months ago

A TacoSprint 2026 Retrospective

This is my retrospective of TacoSprint 2026 that took place in June 2026 at La Saladita, Guerrero, Mexico. For a while now, I have watched from the sidelines as Nixers around the world gathered for sprints: OceanSprint , ThaigerSprint , SaltSprint , TransylvaniaSprint , AuroraSprint and NixCamp . Wow, we Nixers sure do like our sprints! All of them also happen to be in Europe or East-Asia. With an upcoming fourth baby on the way, I figured it was now-or-never to put words into action. I messaged @domen , who organizes OceanSprint every year, and asked if he’d be interested in helping me set up the first sprint in North America. Domen is an avid surfer, a recurring theme in his OceanSprints, so I appealed to his inner-surfer and we spec’d out some places in North America that were both cost-effective and had ample, amazing surf 🏄. I had already been to the Troncones and the La Saladita area, so my prior experience removed a large vector of the unknown. It seemed like a no-brainer. Domen was already going to be in South America in June, so the timing lined up nicely (+ summer is the swell season there!). We set to work standing up a website and trying to attract sponsorship and attendees. This was probably the hardest part of organizing a brand-new sprint. We had far lower turnout for registration and sponsorship than I foresaw. Several people responded on our application form, or told us directly, that they were unsure about the safety of visiting Mexico, since the US Department of State had it under a travel advisory. Despite my best efforts to soothe everyone’s fears, it remained a real hindrance. Note For those still on the fence for next year: the area felt extremely safe. We rented a house in a fairly secluded stretch that caters almost entirely to surfers. At no point did anyone feel uncomfortable or unsafe. Getting there was its own small adventure. Flights were unusually challenging to book thanks to the World Cup soaking up demand across the region. The most dramatic casualty was Alex ( @adeci ), who managed to completely miss his connecting flight and arrived three days late. To his credit, he showed up in great spirits and slotted right back in to hacking with the group like nothing happened. Once everyone was settled, we fell into a rhythm that I can only describe as suspiciously sustainable : It was amazing to bookend each day with a surf at La Saladita’s left point break. Surfing for me lets me enter flow state very similar to when I am deep in thought hacking-away. It helped clear through a lot of built-up gunk and I often returned back with a clear intention or solution to a problem I had been working on. One of the more unexpectedly wonderful parts of the trip was the meal preparation from Gladys, our local cook, who pretty much cooked for us three times a day. We were extremely well-fed, which let us focus on the Nix-hacking and motivated me to make sure I kept up with the surfing to put off any weight gain 🫠. The website will be updated to have a more formal summary of every contribution we managed to put forward and their current status however it was amazing to see how much work a group of nine people can put forward in a single week with a combined mission and passion for an ecosystem. Our work spanned dynamic linking, package relocatability, peer-to-peer remote builds, faster module systems, shrinking the OCaml runtime closure and cross-distribution packaging. A few of my own threads, if you want to go deeper: LLM-based agents featured prominently throughout. We were fortunate to have Geoff Huntley with us, who is quite the AI-maximizer , spiritually guiding us and offering us some SOTA insight in how we might want to explore leveraging AI. Alan ( @alurm ), had the greatest idea for us to put together an academic style trip report. We worked together on the paper and the result is Attention, Nix and Tacos Is All You Need , a loving parody of a certain famous paper. An arXiv submission is coming, but in the meantime you can read it below or download it here . Your browser doesn't support embedded PDFs. You can download it here instead. We already agreed to organize the same sprint next year. I can’t wait. This was literally the most enjoyable thing I’ve ever done as it combined my two passions (surfing & hacking) in a way I honestly did not think was possible all while producing a ton of value to the Nix ecosystem. For a different vantage point, please check out the retrospectives from my fellow attendees! GuixPkgs: every Guix package, as a Nix flake Hijacking ELF entry points for NixOS compatibility, or wtf is wrap-buddy Nix needs relocatable binaries Alan Urmancheev Jared Siegel

Ocaml Writing Culture Business

▲ 0

0 views

Farid Zakaria 2 months ago

Hijacking ELF entry points for NixOS compatibility or WTF is wrap-buddy?

We are part-way through TacoSprint 2026 and a project that has inspired me has been the long-standing pursuit of producing relocatable binaries in Nix. This is something I’ve been discussing since as early as 2022 . We’ve made pretty great headway! 🥳 I posted a proposal to the Linux kernel mailing list to add support for to , which will allow for resolving the interpreter relatively. I also submitted PR#534339 to nixpkgs which improves the generation and shrinking by modifying them to leverage as well. This needs no new Linux kernel support and will make Nix derivations a teeny bit more relocatable. Throughout this investigation, I was informed about similar efforts via wrap-buddy by the venerable Mic92 . I opened the GitHub project and I have to admit, I did not quite understand it. Jörg is an amazingly prolific and technical developer, and despite my knowledge of the space, it took me a while to understand the craziness beauty of what was being done. So, wtf is wrap-buddy ? Nix is all about explicit dependencies and it leverages this with techniques like on the ELF binary. This all works for newly minted code, but if you try to download any precompiled binary on your NixOS machine, you’ll hit an error for a myriad of reasons. One of the biggest being that the dynamic linker/interpreter, , does not exist on NixOS. We would love to compile everything from source, but the reality is that plenty of software people want to use is closed . In order to allow that to work on NixOS machines, derivations may patch the ELF files with patchelf setting things like and to Nix-friendly paths. In some rare cases, however, that doesn’t work. The documentation in claims: autoPatchelfHook can be error-prone and may break binaries that, have unusual ELF layouts. In these pathological cases, is an alternative that takes over the startup of the binary to modify it at runtime. 🤯 Let’s take a look with a small example. We can build a small C program linked against two shared libraries, and , forcing a non-NixOS interpreter path: If we run this binary, it fails immediately because doesn’t exist or it can’t resolve . Now we patch it using pointing to our library paths: Now if we run our binary, , we see that it works: What did it do? 🤔 First off, it copies the first 416 bytes of our program code into a hidden file named . Let’s peek at the original binary and the instructions for : saves those starting 416 bytes to the hidden file . The configuration file format starts with a 22-byte header, followed by the interpreter string (83 bytes) and string (442 bytes), placing our saved original instructions at offset 547 ( ): Next, it clears our to so the Linux kernel thinks it’s a statically linked binary and boots it directly: Lastly, it overwrites our entrypoint with that small stub (416 bytes). We can see in the disassembly that immediately redirects and calls now: Why all this complexity? What is doing? The goal of is to find a known custom loader, , which will help us finish all the dynamic linking. The custom loader gets even more nuanced and low-level. It would be a disservice to try and completely go over everything it does, and at this point the README does a fairly good job. At a high level: The NixOS dynamic linker takes over, uses the to resolve and . We can now run the application using the restored original entry point with everything resolved. Magic. Wizard. Mic92 . 🧙 It reads the saved original bytes from the file and copies the original bytes back over our stub in memory. To any observer, the binary is now completely clean and resembles the original. It injects the custom by creating a brand new dynamic section in memory and populates it with the containing our library search paths that we stored in . It loads the real NixOS interpreter into memory. It rewrites the kernel’s stack metadata (auxiliary vector pointers like , , and ) to trick the native loader ( ) into believing it was loaded natively by the kernel. Finally, it jumps to the entry point of the NixOS interpreter.

Programming

▲ 0

0 views

Farid Zakaria 2 months ago

Nix needs relocatable binaries

This is my problem statement and proposal for a TacoSprint 2026 project 🏄. Nix, or store-based systems , are a class of package managers that use a well-defined prefix to store all packages. This can be for Nix or for Guix. This is simple. It makes rewriting paths to binaries or libraries easy. Derivations only need to the strings with the full store-path; becomes for instance. What if you wanted a different path, one not prefixed at the root ? This could be desirable if you don’t have Nix installed already or are missing necessary permissions – “rootless Nix”. Well, Nix already lets you specify a different store-path today but there is a catch! Let’s take a look at a simple example. We can build two different ways. The first command builds and installs at and the second at using and mount namespaces. Notice both have the same hash . This is important. By keeping the hash the same, we can leverage the precomputed derivations from binary substituters like https://cache.nixos.org . Ok, so what’s missing? If you are using tools like Bazel or Buck2 they likely already employ their own sandboxing via namespacing for builds. Integrating Nix into these ecosystems becomes incredibly impractical because we run into nested user namespace and mount restrictions. We can ask to use an alternate store prefix, without chroot and mount namespaces but it has a big gap. The hash is now 😭 It’s even more disastrous. Changing this simple string cascade-invalidates the entire dependency graph. You are now waiting 4 hours for GCC to compile just so you can print “Hello World” from a different folder. 🫠 This means we cannot leverage the public cache. This gap is called out by the Nix documentation today. Does it have to be that way? What if we could install Nix binaries anywhere , without using namespacing or . Can we have our cake and eat it too? 🍰 Nix needs relocatable binaries . The problem is that the store-prefix is part of the derivation itself so it affects the hash calculation. We don’t have to specify the full store-prefix everywhere. What if we used relative paths ? 🤔 Let’s look at one place the full paths are written today in the binary via . When this program runs, the dynamic linker looks at to find its shared dependencies. The loader in Linux however natively supports the variable which translates to “the directory containing the executable.” [ ref ] We could instead write the to be . If we did that then changing the store would cause no hashes to change. No recompilation. 🥳 Okay, so are we done? Well, like most things the devil is in the details. 😈 Before the dynamic linker can read the to find the necessary libraries, the Linux kernel has to load the dynamic linker itself. This path is stored in a different ELF header called (Program Interpreter). Unfortunately, the Linux Kernel does not support in this field as of today . We run into the exact same kernel limitation with the shebang line in scripts as well. When we execute a script, the kernel parses the (shebang) and expects an absolute path. Support for is also lacking as as of today . We cannot use relative paths reliably here unless they are relative to the current working directory, which breaks the moment you run the script from anywhere else. To achieve true relocatable binaries, we need to bypass these kernel limitations. historically would never make sense for in the Linux kernel because “Why would you want your dynamic linker to be found relative to the file!?”. Nix has changed that assessment. There are a few ways we could attack this: I believe augmenting support in the Linux kernel is the right approach. The beauty of Nix is we can even patch the kernel today in any NixOS machine for this support. As a final cherry on top, we can include additional metadata on every derivation whether it’s relocatable . 🍒 We could patch the Linux kernel so that is supported in and the shebang. We wrap every binary with a small static binary that computes its own location and then invokes the dynamic linker. We need to replace file locations to also leverage language-specific features for relative paths. For instance, in Python we can leverage to access files relative to itself similar to .

Python DevOps Open Source Linux

▲ 0

0 views

Farid Zakaria 2 months ago

A trillion dollars

This is not meant to be a political post. Headlines recently have been projecting Elon Musk’s net worth to hit $1 trillion USD. Working in software, you inevitably come up against Jeff Dean’s latency numbers every programmer should know . The original post wasn’t just insightful for sharing the time costs of common I/O operations. By using the scale of one access pattern to contextualize the next order of magnitude, it really helped entrench the cost of these increasing access patterns in my mind. Scale at increasing orders of magnitude is often difficult to comprehend. Although logarithmic graphs are useful for showing exponential growth or displaying vastly different scales on a single chart, they are easily misunderstood. If we were to map 1 million USD to 1 ns , what are the matching parallels to I/O access patterns? I have seen similar graphics for representing wealth, but I decided to make my own – because why not . Check it out: https://fzakaria.github.io/trillion-wut/ – how fast can you scroll to the bottom? You can find the source available at https://github.com/fzakaria/trillion-wut .

Programming

▲ 0

0 views