GreatReads - Blog Aggregator · Phoenix Framework

0 views

Farid Zakaria 2 weeks ago

A TacoSprint 2026 Retrospective

This is my retrospective of TacoSprint 2026 that took place in June 2026 at La Saladita, Guerrero, Mexico. For a while now, I have watched from the sidelines as Nixers around the world gathered for sprints: OceanSprint , ThaigerSprint , SaltSprint , TransylvaniaSprint , AuroraSprint and NixCamp . Wow, we Nixers sure do like our sprints! All of them also happen to be in Europe or East-Asia. With an upcoming fourth baby on the way, I figured it was now-or-never to put words into action. I messaged @domen , who organizes OceanSprint every year, and asked if he’d be interested in helping me set up the first sprint in North America. Domen is an avid surfer, a recurring theme in his OceanSprints, so I appealed to his inner-surfer and we spec’d out some places in North America that were both cost-effective and had ample, amazing surf 🏄. I had already been to the Troncones and the La Saladita area, so my prior experience removed a large vector of the unknown. It seemed like a no-brainer. Domen was already going to be in South America in June, so the timing lined up nicely (+ summer is the swell season there!). We set to work standing up a website and trying to attract sponsorship and attendees. This was probably the hardest part of organizing a brand-new sprint. We had far lower turnout for registration and sponsorship than I foresaw. Several people responded on our application form, or told us directly, that they were unsure about the safety of visiting Mexico, since the US Department of State had it under a travel advisory. Despite my best efforts to soothe everyone’s fears, it remained a real hindrance. Note For those still on the fence for next year: the area felt extremely safe. We rented a house in a fairly secluded stretch that caters almost entirely to surfers. At no point did anyone feel uncomfortable or unsafe. Getting there was its own small adventure. Flights were unusually challenging to book thanks to the World Cup soaking up demand across the region. The most dramatic casualty was Alex ( @adeci ), who managed to completely miss his connecting flight and arrived three days late. To his credit, he showed up in great spirits and slotted right back in to hacking with the group like nothing happened. Once everyone was settled, we fell into a rhythm that I can only describe as suspiciously sustainable : It was amazing to bookend each day with a surf at La Saladita’s left point break. Surfing for me lets me enter flow state very similar to when I am deep in thought hacking-away. It helped clear through a lot of built-up gunk and I often returned back with a clear intention or solution to a problem I had been working on. One of the more unexpectedly wonderful parts of the trip was the meal preparation from Gladys, our local cook, who pretty much cooked for us three times a day. We were extremely well-fed, which let us focus on the Nix-hacking and motivated me to make sure I kept up with the surfing to put off any weight gain 🫠. The website will be updated to have a more formal summary of every contribution we managed to put forward and their current status however it was amazing to see how much work a group of nine people can put forward in a single week with a combined mission and passion for an ecosystem. Our work spanned dynamic linking, package relocatability, peer-to-peer remote builds, faster module systems, shrinking the OCaml runtime closure and cross-distribution packaging. A few of my own threads, if you want to go deeper: LLM-based agents featured prominently throughout. We were fortunate to have Geoff Huntley with us, who is quite the AI-maximizer , spiritually guiding us and offering us some SOTA insight in how we might want to explore leveraging AI. Alan ( @alurm ), had the greatest idea for us to put together an academic style trip report. We worked together on the paper and the result is Attention, Nix and Tacos Is All You Need , a loving parody of a certain famous paper. An arXiv submission is coming, but in the meantime you can read it below or download it here . Your browser doesn't support embedded PDFs. You can download it here instead. We already agreed to organize the same sprint next year. I can’t wait. This was literally the most enjoyable thing I’ve ever done as it combined my two passions (surfing & hacking) in a way I honestly did not think was possible all while producing a ton of value to the Nix ecosystem. For a different vantage point, please check out the retrospectives from my fellow attendees! GuixPkgs: every Guix package, as a Nix flake Hijacking ELF entry points for NixOS compatibility, or wtf is wrap-buddy Nix needs relocatable binaries Alan Urmancheev Jared Siegel

Ocaml

Writing

Culture

Business

0 views

Farid Zakaria 3 weeks ago

Hijacking ELF entry points for NixOS compatibility or WTF is wrap-buddy?

We are part-way through TacoSprint 2026 and a project that has inspired me has been the long-standing pursuit of producing relocatable binaries in Nix. This is something I’ve been discussing since as early as 2022 . We’ve made pretty great headway! 🥳 I posted a proposal to the Linux kernel mailing list to add support for to , which will allow for resolving the interpreter relatively. I also submitted PR#534339 to nixpkgs which improves the generation and shrinking by modifying them to leverage as well. This needs no new Linux kernel support and will make Nix derivations a teeny bit more relocatable. Throughout this investigation, I was informed about similar efforts via wrap-buddy by the venerable Mic92 . I opened the GitHub project and I have to admit, I did not quite understand it. Jörg is an amazingly prolific and technical developer, and despite my knowledge of the space, it took me a while to understand the craziness beauty of what was being done. So, wtf is wrap-buddy ? Nix is all about explicit dependencies and it leverages this with techniques like on the ELF binary. This all works for newly minted code, but if you try to download any precompiled binary on your NixOS machine, you’ll hit an error for a myriad of reasons. One of the biggest being that the dynamic linker/interpreter, , does not exist on NixOS. We would love to compile everything from source, but the reality is that plenty of software people want to use is closed . In order to allow that to work on NixOS machines, derivations may patch the ELF files with patchelf setting things like and to Nix-friendly paths. In some rare cases, however, that doesn’t work. The documentation in claims: autoPatchelfHook can be error-prone and may break binaries that, have unusual ELF layouts. In these pathological cases, is an alternative that takes over the startup of the binary to modify it at runtime. 🤯 Let’s take a look with a small example. We can build a small C program linked against two shared libraries, and , forcing a non-NixOS interpreter path: If we run this binary, it fails immediately because doesn’t exist or it can’t resolve . Now we patch it using pointing to our library paths: Now if we run our binary, , we see that it works: What did it do? 🤔 First off, it copies the first 416 bytes of our program code into a hidden file named . Let’s peek at the original binary and the instructions for : saves those starting 416 bytes to the hidden file . The configuration file format starts with a 22-byte header, followed by the interpreter string (83 bytes) and string (442 bytes), placing our saved original instructions at offset 547 ( ): Next, it clears our to so the Linux kernel thinks it’s a statically linked binary and boots it directly: Lastly, it overwrites our entrypoint with that small stub (416 bytes). We can see in the disassembly that immediately redirects and calls now: Why all this complexity? What is doing? The goal of is to find a known custom loader, , which will help us finish all the dynamic linking. The custom loader gets even more nuanced and low-level. It would be a disservice to try and completely go over everything it does, and at this point the README does a fairly good job. At a high level: The NixOS dynamic linker takes over, uses the to resolve and . We can now run the application using the restored original entry point with everything resolved. Magic. Wizard. Mic92 . 🧙 It reads the saved original bytes from the file and copies the original bytes back over our stub in memory. To any observer, the binary is now completely clean and resembles the original. It injects the custom by creating a brand new dynamic section in memory and populates it with the containing our library search paths that we stored in . It loads the real NixOS interpreter into memory. It rewrites the kernel’s stack metadata (auxiliary vector pointers like , , and ) to trick the native loader ( ) into believing it was loaded natively by the kernel. Finally, it jumps to the entry point of the NixOS interpreter.

0 views

Farid Zakaria 3 weeks ago

Nix needs relocatable binaries

This is my problem statement and proposal for a TacoSprint 2026 project 🏄. Nix, or store-based systems , are a class of package managers that use a well-defined prefix to store all packages. This can be for Nix or for Guix. This is simple. It makes rewriting paths to binaries or libraries easy. Derivations only need to the strings with the full store-path; becomes for instance. What if you wanted a different path, one not prefixed at the root ? This could be desirable if you don’t have Nix installed already or are missing necessary permissions – “rootless Nix”. Well, Nix already lets you specify a different store-path today but there is a catch! Let’s take a look at a simple example. We can build two different ways. The first command builds and installs at and the second at using and mount namespaces. Notice both have the same hash . This is important. By keeping the hash the same, we can leverage the precomputed derivations from binary substituters like https://cache.nixos.org . Ok, so what’s missing? If you are using tools like Bazel or Buck2 they likely already employ their own sandboxing via namespacing for builds. Integrating Nix into these ecosystems becomes incredibly impractical because we run into nested user namespace and mount restrictions. We can ask to use an alternate store prefix, without chroot and mount namespaces but it has a big gap. The hash is now 😭 It’s even more disastrous. Changing this simple string cascade-invalidates the entire dependency graph. You are now waiting 4 hours for GCC to compile just so you can print “Hello World” from a different folder. 🫠 This means we cannot leverage the public cache. This gap is called out by the Nix documentation today. Does it have to be that way? What if we could install Nix binaries anywhere , without using namespacing or . Can we have our cake and eat it too? 🍰 Nix needs relocatable binaries . The problem is that the store-prefix is part of the derivation itself so it affects the hash calculation. We don’t have to specify the full store-prefix everywhere. What if we used relative paths ? 🤔 Let’s look at one place the full paths are written today in the binary via . When this program runs, the dynamic linker looks at to find its shared dependencies. The loader in Linux however natively supports the variable which translates to “the directory containing the executable.” [ ref ] We could instead write the to be . If we did that then changing the store would cause no hashes to change. No recompilation. 🥳 Okay, so are we done? Well, like most things the devil is in the details. 😈 Before the dynamic linker can read the to find the necessary libraries, the Linux kernel has to load the dynamic linker itself. This path is stored in a different ELF header called (Program Interpreter). Unfortunately, the Linux Kernel does not support in this field as of today . We run into the exact same kernel limitation with the shebang line in scripts as well. When we execute a script, the kernel parses the (shebang) and expects an absolute path. Support for is also lacking as as of today . We cannot use relative paths reliably here unless they are relative to the current working directory, which breaks the moment you run the script from anywhere else. To achieve true relocatable binaries, we need to bypass these kernel limitations. historically would never make sense for in the Linux kernel because “Why would you want your dynamic linker to be found relative to the file!?”. Nix has changed that assessment. There are a few ways we could attack this: I believe augmenting support in the Linux kernel is the right approach. The beauty of Nix is we can even patch the kernel today in any NixOS machine for this support. As a final cherry on top, we can include additional metadata on every derivation whether it’s relocatable . 🍒 We could patch the Linux kernel so that is supported in and the shebang. We wrap every binary with a small static binary that computes its own location and then invokes the dynamic linker. We need to replace file locations to also leverage language-specific features for relative paths. For instance, in Python we can leverage to access files relative to itself similar to .

Python

Linux

0 views

Farid Zakaria 1 months ago

A trillion dollars

This is not meant to be a political post. Headlines recently have been projecting Elon Musk’s net worth to hit $1 trillion USD. Working in software, you inevitably come up against Jeff Dean’s latency numbers every programmer should know . The original post wasn’t just insightful for sharing the time costs of common I/O operations. By using the scale of one access pattern to contextualize the next order of magnitude, it really helped entrench the cost of these increasing access patterns in my mind. Scale at increasing orders of magnitude is often difficult to comprehend. Although logarithmic graphs are useful for showing exponential growth or displaying vastly different scales on a single chart, they are easily misunderstood. If we were to map 1 million USD to 1 ns , what are the matching parallels to I/O access patterns? I have seen similar graphics for representing wealth, but I decided to make my own – because why not . Check it out: https://fzakaria.github.io/trillion-wut/ – how fast can you scroll to the bottom? You can find the source available at https://github.com/fzakaria/trillion-wut .

0 views

Farid Zakaria 1 months ago

Relocatable Derivations

The earlier post on guix-transfer highlighted how we can use the tool to transfer derivations from to . It is always delightful when someone offers deeper insights into an idea I had put forward. I was so focused on the transfer from derivations from Guix, I failed to see the larger applicability of the tool. @tomberek shared with me the insight that the tool can be generalized to: “transfer derivations between realms” . Relocatable derivations. 💥 What does that mean? Perhaps the clearest concept to apply it to are deployments . You might have a derivation that you want to propagate through cascading deployment tiers: alpha, beta, and prod. You might have needed to painstakingly apply some logical firewall if all three realms used as their prefix to gate your deployment. By changing the prefix of each one, i.e. or , they are naturally segregated in Nix. How do we promote derivations from one realm to another? We could re-evaluate the Nix expression again against each new store or we can leverage guix-transfer . Why is this better than doing against these new store directories? What is our source of truth? The Nix files or the derivations? I posit that the derivations themselves are the source of truth. Furthermore, evaluation could be slow and requires the full source code (Nix expressions) and the entire evaluation environment (i.e. Nixpkgs, plugins, and overlays). By relocating at the derivation level, we move from Evaluation-based deployment (which is slow, requires source access, and may be prone to evaluation-time impurities potentially) to a Plan-based deployment . We now treat the build graph, via the derivations, as a portable artifact that can be relocated into any realm, regardless of whether that realm has the source code, the right version of Nixpkgs, or in the case of Guix, even speaks the same front-end language.

0 views

Farid Zakaria 1 months ago

The Guix Nix Abomination: Leveraging Guix derivations in Nix

Nix and Guix look like rival ecosystems, but under the hood they’re the same “Input Output Machine”. Need proof? 🕵 How about we build a Guix derivation with Nix. First let’s create a super basic derivation in Guix: Hello world . We then ask Nix to build it. 🪄 We ask to use as the Nix store and have it write its state, database and log files in alternate directories, so it does not collide or mess with Guix. Note It’s slightly more complicated. Nix happens to check its SQLite database for the derivation, so we need to register it first. The version of Guix (v1.5.0) I’m using leverages a user that runs inside a private mount namespace where is writable, but everyone else (including me) sees it as read-only. The creates a new private mount namespace so I can mount it as read-write and run the Nix command against it. We just built a Guix derivation using Nix. 🔥 How is that possible? Both take a language frontend, Nix or Guile (Scheme), that compiles to a derivation (recipe) and pass that onto a builder (daemon) that executes it to produce an output. What makes them both special is they both promise the same thing: hermetic builds . Everything needed to build the output is declared in the recipe: sources, environment variables, dependencies, etc. “Under Nix, a build process will only find resources that have been declared explicitly as dependencies. There’s no way it can build until everything it needs has been correctly declared. If it builds, you will know you’ve provided a complete declaration.” – Nix OS Website Guix, specifically the daemon, was forked from Nix early on, and as a result the two are very similar; they both share the same derivation format, ATerm , for instance. Guix is based on the Nix package manager – Guix Website That’s why our earlier example of building the Guix derivation with Nix was possible without much translation. What if we could leverage an existing recipe from Guix in Nix in its traditional ? If we could convert from one recipe file to the other, we could use the existing recipes from Guix in Nix and vice versa. Turns out this is far more feasible than you would think, because Guix is Nix or at least a superset of it. I, with the help of Claude, built a tool to do just that: guix-transfer 🤯. guix-transfer is a CLI tool for performing bottom-up translation of GNU Guix derivations into Nix. Confused? Let us see it in action: Note When you unpack a tarball, tar restores each file’s original permissions, including setuid/setgid bits. Nix’s sandbox installs a seccomp filter that blocks any call that sets these bits, returning “Operation not permitted”. Guix’s early bootstrap uses a Scheme-based (gash-utils) that treats this error as fatal, unlike GNU tar which silently skips it. The fix is , which disables the filter. If it’s not clear what we just did: we took a Guix derivation and all of its dependencies (down to the bootstrap seeds), translated it to a Nix derivation, and built it with Nix. 😲 What is this abomination and how was this possible!? It’s important to revisit what a derivation is, and how it’s used in Nix and Guix to better understand how this is possible. Let’s look at the same basic derivation from earlier, Hello World . You might want to check out my other post on Nix derivations by hand if this interests you 🤓. When we evaluate (nix-instantiate) this derivation, we get a path to a file that contains the derivation in the ATerm format: If we look at the contents of the file, we can see the ATerm representation of the derivation: This has all the information we need to build the output by the builder. At this point, it’s really not Nix specific anymore. The same applies for the Guix derivations. The derivations do not “know” whether they came from Scheme or Nix. It’s a recipe. The insight then is if we rewrite the store paths from to , and swap some builtins (i.e. for ), we can get to build it identically . 💡 The only difference in more complex derivations is that they have dependencies, which are also derivations, and the builder references them so it forms a graph of derivations, each built by the builder in topological order. The leaves of this tree for any non-trivial derivation are the bootstrap seeds: , , , etc. Guix is famous for bootstrapping itself from a 357-byte binary as source [ ref ]. Since at no point do the bootstrap seeds depend on being the prefix, the translated chain builds identically under Nix. walks a Guix graph in post-order and for each derivation: Guix’s is replaced with Nix’s . Same idea, different name. Source files are added to the Nix store, with embedded paths rewritten to their equivalents. Every reference: input drvs, builder path, args, env vars are rewritten to the mapped path. Output paths are blanked as Nix recomputes them via . The result is serialised as JSON and registered with . That’s it. No Nix expressions are generated. No . No mapping of Guix packages to nixpkgs equivalents. The Guix derivation graph is translated faithfully , and builds it. Note Interestingly, takes exactly one URL and cannot fall back. Guix derivations carry lists of mirrors, many of which are flaky or dead. Similar to Nix, Guix operates a content-addressed mirror at that serves any source its CI has ever seen. We leverage this for the instead of the original source URL. Now that we have a way to slurp Guix packages into Nix, we can start to do some diabolical combinations by combining native Nix and Guix packages together! We can take our package we built in Nix and leverage it in a Nix derivation. Nix automatically scans your derivations for anything prefixed with and tracks it as an input dependency. This is similar to how store paths are interpolated when you do something like . If writing the paths raw in the Nix expression is a little too raw for you, we can build something more ergonic pretty easily as well. has an mode that instead will emit the Nix expression for the translated . Let’s look at a slightly more complex example that uses Guix’s to build a derivation with dependencies: We can now convert this to a Nix expression with . We realise the derivation with or we can the Nix expression. Please notice that both produce the exact same hash : . We can now use this Guix derivation like any normal Nix expression, such as the ones you might encounter in Nixpkgs. That means we could even build a that is all of Guix packages available for use. My mind is blown. 🤯 Nixpkgs is known as the world’s largest package repository, and now we have made a way for it suddenly to become even larger by borrowing any derivation from Guix! The real power behind Nix are the derivations and that they are hermetic, declaring any dependency needed. We’ve seen that we can transfer these recipes to any store-based system that has similar qualities and preserve the reproducibility. Guix’s is replaced with Nix’s . Same idea, different name. Source files are added to the Nix store, with embedded paths rewritten to their equivalents. Every reference: input drvs, builder path, args, env vars are rewritten to the mapped path. Output paths are blanked as Nix recomputes them via . The result is serialised as JSON and registered with .

Linux

JSON

0 views

Farid Zakaria 1 months ago

Every byte matters

I have spent a large portion of my career working in Java. In that time, you get used to huge classes. New functionality? Just add a new method and field to the class. The cost of each new field is rarely considered. Performance is often considered from a classic computer science perspective by considering asymptotic analysis of the algorithms and data structures in-use. Turns out that even within a growth scale for your algorithm, such as a simple for-loop , time can vary dramatically if we have a little deeper understanding of the underlying hardware. First, let’s understand our current machine. Let’s take a peek at our cache line and page sizes. The instances number is a reflection of how the caches are shared amongst CPUs. If I had 10 CPUs, each one has their own cache, whereas two of them would share an cache. Our cache line size is 64 bytes . When you read a single byte from memory, the hardware will fill the surrounding 64 bytes into the cache line. The idea being that data is often temporal and spatially located, meaning data is often accessed near each other and close in time to each other. We can reference Jeff Dean’s famous “Latency numbers every programmer should know” , however a quick recap with the values from our particular machine is the following: The sizes for each cache, is the number returned by divided by the number of cores or instances; i.e. 352 KiB ÷ 10 instances = ~35 KiB. We then determine the number of cache lines by dividing this number by 64; i.e. 35 KiB ÷ 64 bytes = 560 cache lines. How does this all matter ? 🤔 Let’s consider an example where we want to iterate over a single struct and pull out the to filter them. We create our struct, and in this particular example we need 64 bytes to represent a single Monster. If we had an array of Monsters and we iterate over them, the cache line would fill up like so. Each cache line would fill with a single monster, and we would fetch only the byte. This is often referred to as “Array of Structs”. If we instead normalize the data such that each field is in it’s own list, we can pack the cache lines much tighter. This type of layout is referred to as “Struct of Arrays”. How much of an impact can this have? We can observe up to 30x improvements when the Monster struct is 1KiB 🤯 The delta is less observable when the struct is small because multiple Monster structs can still be fetched within a single cache-line. This data access is incredibly hot though. Your CPU pre-fetcher knows it’s going sequentially and fetches the next cache line before you need it. You never actually have to wait for the memory to be fetched. What about random access patterns? Not all access patterns are sequential. Hash maps, trees, graph traversal, and pointer-heavy data structures jump to unpredictable locations. The CPU can’t prefetch what it can’t predict. With random access, the CPU needs the entire array to be present in the cache in order to avoid stalls due to memory lookup. This means the total size of your collection determines your performance tier. Doubling the struct from 64B to 128B doubles the working set for the same number of monsters, pushing the data into slower cache levels. At just 512 monsters, a 64B struct fits in L1d at ~3 ns — but a 128B struct has already spilled to L2 at ~11 ns. We can observe this with a pointer-chasing benchmark. We allocate N monster-sized nodes, wire them into a random order, and chase pointers. Each hop lands at an unpredictable address, defeating the CPU’s prefetcher entirely. Rather than graph it logarithmically, which I find sometimes is easy to miss, I have included a zoomed in graph. We can see that all struct sizes hit the same staircase like pattern as they go through the various cache levels however the larger struct sizes are shifted left , meaning they hit the increase earlier. This means for random access patterns, if you can keep tight control on your total working set size, you can drastically affect the time. Knowing your struct and working set size can make a substantial difference.

Java

0 views

Farid Zakaria 1 months ago

AI is a Boon for the Anal-Retentive

I guess it’s my turn to write an “AI article”. 🫠 It’s Saturday evening and I’m late into my 3rd project, one where I would never have gotten so far without the help of recent advancements in AI, all thanks to LLMs. The realization that I’ve been able to accomplish personal projects that evaded me for so long was really remarkable and I feel fortunate…at least for now. I can see the knobs slowly dialing back, the amount of tokens we are given pro bono is quickly diminishing. A typical project, B.O. (Before Opus 4.6), was one where I think of an idea that inevitably needs a frontend component. I then spend 3-7 days researching the latest on frontend build patterns, frameworks and trends. I’m then mired in choice and complexity only to lose focus, or lose interest in bit-twiddling CSS (something I never quite truly learned). My latest project, Zephyr , is my attempt to build a weather wind station powered by battery & solar that can send me updates via cellular network. The project involves: writing firmware via Rust to an ESP32 board, sending commands to the modem to initiate the HTTPS requests, connecting the anemometer and wind vane to the device, working with a breadboard (I can do hardware now too!) and writing a frontend website to display the results. To say this project is outside of my wheelhouse is an understatement . However, I have been able to get surprisingly far asking for advice and guidance, and leaning on the LLM to bootstrap some code. I have had to be incredibly thoughtful in guiding it to relevant schematics, examples and manuals since writing the firmware for the board was incredibly tricky with all the various knobs that can be tuned. The project prior to that was one dedicated to recording surfing entries: http://surfing.exe.xyz/ . The project prior to that was https://checkthisdealforme.com/ , a small website where you can quickly appraise items you either want to buy or sell. You might notice a theme that all these projects leverage https://exe.dev/ . It’s a platform I’ve enjoyed that has similarly taken all the infrastructure molasses out when working on small projects. All these projects would never have made it this far. I’m far too anal-retentive. That quality is part of my personality. It has often been a strength when dealing with the mire of complexity at $DAYJOB$ but has been a burden for things whose complexity I am happy to relinquish. Thank you, large language models, for freeing me of my anal-retentiveness when necessary. While others use the term “slop” as a form of insult, here it’s been a boon and a welcomed one.

Rust

AI

CSS

0 views

Farid Zakaria 1 months ago

Leaving performance on the table

I have been working with LLVM at , and I have gotten to become familiar with the benefits of optimizing your workloads. I tend to think of optimizing my binaries as thinking about whether I have attached to my compiler flags; maybe if I’m particularly advanced that day I’ll sprinkle in some (link time optimziation) and call it a day. Turns out though that’s leaving lots of performance on the table. Compilers work under the assumption that every branch is is equally taken, unless you are hints like ( ref ). If we can feed the compilers more information about the likely path that our workloads often take, then they can produce much more performant code. There are two primary ways to optimize a binary: instrumented or statistical. When we instrument our binary, we run our workload with an instrumented binary and capture the exact paths that are executed. We will then optimize the binary perfectly tuned to that workload. If our workloads however are varied, we can collect profiles via over a length of time and create an optimized binary based on the statistical occurence of call graphs. Both approaches have their benefits however let’s start with the instrumented variant first, as it’s a little easier to follow and understand. Let’s look at a very simple benchmark. We will calculate fibonocci using SQL in sqlite3 . This is an ideal workload because it’s purely CPU-bound and ripe for optimizing. We will compile from source by downloading it. We can compile a “traditional” optimized binary that merely has and also a version that has LTO enabled since I was also keen to see how much LTO itself adds. Ok, so it looks like our program takes roughly 14-15 seconds to run. Sounds ok? How much better can we do…. 🤔 Next, we compile our program again but we instrument the binary , which effectively injects counters into the program to count invocations of functions. We get very accurate counts of our calls but the binary itself now runs much slower, which can be a problem if your workload was already very slow. Luckily for us, we are in a time domain (~15 seconds), where that is ok. After we have our instrumented binary, we run our workload again to generate the profile data and rebuild the binary with that data. The last step will be to optimize with BOLT, which is a post-link optimizer, which requires us to keep relocations so I’ve also added . When we run our workload with the final optimized binary, we see massive improvement already! 🤯 We’ve cut our workload time down to ~10 seconds which is a nearly a 1.5x improvement. Now let’s optimize the final binary with LLVM’s BOLT . BOLT is a post-link optimizer designed for “large applications”. What this means, is that it largely works by shuffling code around the binary to keep code-paths that have high temporal locality near each other (spatial locality). This can have positive impact on performance due to the instruction cache for instance. Looks like it was a little faster but not much. That makes sense since itself is a pretty small binary (~6MB), but nontheless was good to run through. Running a more thorough benchmark with we can get a final tally of our results. Looks like the I got from the Fedora ecosystem was the slowest . When all the optimizations were applied I was able to get a maximum of 1.38x faster than what was available. These optimizations would be even more dramatic for code-bases that are a sprawl and can heavily vary. Don’t worry also about getting the profile perfectly tuned to your workloads. I have a coworker who often cites that even poor profiles are still much better than no profile at all.

Sql

0 views

Farid Zakaria 3 months ago

Does anyone actually use the large code-model?

I have been focused lately on trying to resolve relocation overflows when compiling large binaries in the small & medium code-models. Often when talking to others about the problem, they are quick to offer the idea of using the large code-model. Despite the performance downsides of using the large code-model from the instructions generated, it’s true that its intent was to support arbitrarily large binaries. However does anyone actually use it? Turns out that large binaries do not only affect the instructions generated in the section but may also have effects on other sections within the ELF file such as (exception handling information), (optimized binary search table for ), and even . Let’s take and as an example. They specifically allow various encodings for the data within them ( or for 4 bytes and 8 bytes respectively) irrespective of the code-model used. However, it looks like the userland has terrible support for it! If we look at the format, we can see how these encodings are applied in practice. The entries in this column are the ones that actually resolve to specific DWARF exception header encoding formats (like , , , etc.) depending on the values provided in the preceding fields. format [ ref ]: Note: The values for and dictate their byte size and format. For example, if is set to , the field will be processed as an (signed 4-byte) value. Up until very recently ( pull#179089 ), LLVM’s linker would crash if it tried to link exception data ( ) beyond 2GiB. This section is always generated to help stack searching algorithms avoid linear search. Once we fix that though, it looks like ( gcc-patch@ ) and ( pull#964 ) explicitly either crash on or avoid the binary search table completely reverting back to linear search. How devasting is linear search here? If you have a lot of exceptions, which you theoretically might for the large code-model, I had benchmarks that started at ~13s improve to ~18ms for a ~700x speedup . Other fun failure modes that exist: Note: Don’t let confuse you, it’s actually 32bit: It seems like the large code-model “exists” but no one is using it for it’s intended purpose which was to build large binaries. I am working to make massive binaries possible without the large code-model while retaining much of the performance characteristics of the small code-model. You can read more about it in x86-64-abi google-group where I have also posted an RFC.

0 views

Farid Zakaria 4 months ago

Nix is a lie, and that’s ok

When Eelco Dolstra , father of Nix, descended from the mountain tops and enlightened us all, one of the main commandments for Nix was to eschew all uses of the Filesystem Hierarchy Standard (FHS) . The FHS is the “find libraries and files by convention” dogma Nix abandons in the pursuit of purity. What if I told you that was a lie ? 😑 Nix was explicitly designed to eliminate standard FHS paths (like or ) to guarantee reproducibility. However, graphics drivers represent a hard boundary between user-space and kernel-space. The user-space library ( ) must match the host OS’s kernel module and the physical GPU. Nearly all derivations do not bundle with them because they have no way of predicting the hardware or host kernel the binary will run on. What about NixOS? Surely, we know what kernel and drivers we have there!? 🤔 Well, if we modified every derivation to include the correct it would cause massive rebuilds for every user and make the NixOS cache effectively useless. To solve this, NixOS & Home Manager introduce an intentional impurity, a global path at where derivations expect to find . We’ve just re-introduced a convention path à la FHS. 🫠 Unfortunately, that leaves users who use Nix on other Linux distributions in a bad state which is documented in issue#9415 , that has been opened since 2015. If you tried to install and run any Nix application that requires graphics, you’ll be hit with the exact error message Nix was designed to thwart: There are a couple of workarounds for those of us who use Nix on alternate distributions: For those of us though who cling to the beautiful purity of Nix however it feels like a sad but ultimately necessary trade-off. Thou shall not use FHS, unless you really need to. nixGL , a runtime script that injects the library via manually hacking creating your own and symlinking it with the drivers from

0 views

Farid Zakaria 4 months ago

Linker Pessimization

In a previous post , I wrote about linker relaxation : the linker’s ability to replace a slower, larger instruction with a faster, smaller one when it has enough information at link time. For instance, an indirect through the GOT can be relaxed into a direct plus a . This is a well-known technique to optimize the instructions for performance. Does it ever make sense to go the other direction ? 🤔 We’ve been working on linking some massive binaries that include Intel’s Math Kernel Library (MKL) , a prebuilt static archive. MKL ships as object files compiled with the small code-model ( ), meaning its instructions assume everything is reachable within ±2 GiB. The included object files also has some odd relocations where the addend is a very large negative number (>1GiB). The calculation for the relocation value is S + A - P : the symbol address plus the addend minus the instruction address. WIth a sufficiently large negative addend, the relocation value can easily exceed the 2 GiB limit and the linker fails with relocation overflows. We can’t recompile MKL (it’s a prebuilt proprietary archive), and we can’t simply switch everything to the large code model. What can we do? 🤔 I am calling this technique linker pessimization : the reverse of relaxation. Instead of shrinking an instruction, we expand one to tolerate a larger address space. 😈 The specific instructions that overflow in our case are (Load Effective Address) instructions. In x86_64, performs pure arithmetic: it computes and stores the result in without accessing memory. The is a 32-bit signed integer embedded directly into the instruction encoding, and the linker fills it in via an relocation. The relocation formula is S + A - P . Let’s look at an example with a large addend. A 32-bit signed integer can only represent ±2,048 MB (±2 GiB). Our value of −2,062 MB exceeds that range and the linker rightfully complains 💥: Note These instructions appear in MKL because the library uses them as a way to compute an address of a data table relative to the instruction pointer. The large negative addend ( ) is intentional ; it’s an offset within a large lookup table. The core idea is delightful because often as an engineer we are trained to optimize systems, but in this case we want the opposite. We swap the for a that reads through a nearby pointer. Recall from the relaxation post : relaxation shrinks instructions (e.g. indirect -> direct ). Here we do the opposite: we make the instruction do more work (pure arithmetic -> memory load) in exchange for a reachable displacement. That’s why I consider it a pessimization or reverse-relaxation . Both instructions use the same encoding length (7 bytes with a REX prefix), so the patch is a single byte change in the opcode. 🤓 The difference in behavior is critical: Original — the must reach across the entire binary: Pessimized — the reads a nearby pointer that holds the full address: We’ve traded one direct computation for an indirect through a pointer, and we make sure the displacement is now tiny. The 64-bit pointer slot can reach any address in the virtual address space. 👌 For each problematic relocation, three changes are needed in the object file: 1. Opcode Patch : In , change byte to (1 byte). This converts the (compute address) into a (load from address). The rest of the instruction encoding (ModR/M byte, REX prefix) stays identical because both instructions use the same operand format. 2. New Pointer Slot — Create a new section ( ) containing 8 zero bytes per patch site, plus a new relocation pointing to the original symbol with the original addend. is a 64-bit absolute relocation. Its formula is simply , no subtraction of . There is no 32-bit range limitation; it can address the entire 64-bit address space. This is the key insight that makes the fix work. 3. Retarget the Original Relocation — In the entry for the patched instruction, change the symbol to point at the new pointer slot in and update the type to . The addend becomes a small offset (the distance from the instruction to the fixup slot), which is guaranteed to fit. Note Because both and with a operand are exactly the same length (7 bytes with a REX prefix), we don’t shift any code, don’t invalidate any other relocations, and don’t need to rewrite any other parts of the object file. It’s truly a surgical patch. The pessimized now performs a memory load where the original did pure register arithmetic. That’s an extra cache line fetch and a data dependency. If this instruction is in a tight loop, it could be a performance hit. Optimization is the root of all evil, what does that make pessimization? 🧌 LEA : (arithmetic, no memory access). must encode the entire distance to the far-away data. This overflows. MOV : (memory load). points to a nearby 8-byte pointer slot. The pointer slot holds the full 64-bit address. This never overflows.

0 views

Farid Zakaria 5 months ago

Creating massively huge fake files and binaries

I was writing a test case for to support “thunks” [ llvm#180266 ] which uses a linker script to place two sections very far apart (8GiB) in the virtual address space. After linking a trivially small assembly file, I ran on the resulting binary was confused 8 GiB . For what amounts to a handful of instructions. 😲 What’s going on? And where did all that space come from? Turns out reports the logical (apparent) size of the file, which is simply an integer stored in the inode metadata. It represents the offset of the last byte written. Since lives at (~8 GiB), the file’s logical size extends out that far even though the actual code is tiny. The real story is told by : 12 KiB on disk. The file is sparse . 🤓 A sparse file is one where the filesystem doesn’t bother allocating blocks for regions that are all zeros. The filesystem (ext4, btrfs, etc.) stores a mapping of logical file offsets to physical disk blocks in the inode’s extent tree . For a sparse file, there are simply no extents for the hole regions. For our 8 GiB binary, the extent tree looks something like: We can use to also see the same information, albeit a little more condensed. When something reads the file: You don’t need a linker to create sparse files. will do it: A 1 PiB file that takes zero bytes on disk. with works too: Both produce the same result: a file whose logical size is 1 PiB but whose on-disk footprint is effectively nothing. The virtual filesystem (VFS) receives at some offset The filesystem looks up the extent tree for that offset If extent found then read from the physical disk block If no extent (hole) then the kernel fills the buffer with zeros, no disk I/O

0 views

Farid Zakaria 5 months ago

Crazy shit linkers do: Relaxation

I have been looking into linkers recently and I’ve been amazed at all the crazy options and optimizations that a linker may perform. Compilers are a well understood domain, taught in schools with a plethora of books but few resources exist for linkers aside from what you may find on some excellent technical blogs such as Lance Taylor’s series on writing the gold linker and Fangrui Song’s, also known as MaskRay, very in-depth blog . I wanted to write down in my own style, concepts I’m learning from first principles . Recently, I came across a term “relaxation” as I was fuddling around LLVM’s . What is it? 🤔 Note Relaxation looks to be relatively new , and the original RFC to the x86-64-abi google group was proposed in 2015. Well, let’s look at a super simple example to understand what it is and why we want it. If you want to follow along take a look at this godbolt example. If we compile this with we see the following disassembly in the object file using . Specifically, the compile has left a “note” for the linker in the form of a relocation , specifically . You can see that the address in the emitted code is after compilation. The linker needs to replace that value with the address of the function from the GOT relative to the register (instruction pointer). This works great and is necessary for shared libraries but what if we are building a final static binary? 🤓 Turns out that in some cases, this instruction can be further simplified by the linker since when producing the final executable binary it has all the information. We will have to see the actual instruction-code to understand this further. If we look at the hexcode for that assembly we see the following: This indirect call ( ) via the GOT address is 6 bytes long with 2 bytes for the opcode & 4 bytes belonging to the offset to the GOT entry. Note Understanding x86-64 is its own whole can of worms. The ISA is incredibly dense and complex, but if you want you can reference it here . x86-64 though has other types ( ), that operate in a direct mode where it calls the address relative to the bytes presented. This direct-mode type is only 5 bytes long with 1 byte for the opcode and 4 bytes for the offset to the function. If we knew the location of the function ahead of time, it would be nice if we could skip checking the GOT completely and just go to where we want to be. Why would we want to do this? Well it’s more efficient to directly jump to the address we want to end up directly. The CPU doesn’t have to load the memory stored at the GOT before jumping to it. When building a static binary the linker should know all the final relative addresses of all the functions, so going through the GOT is no longer necessary. Since the number of bytes is nearly equal, the linker can effectively patch the binary without disrupting other relative calculations, provided it can fill the small gap. We only need to find a single byte to pad our more-efficient ! 🕵️ Turns out, the operation is only a single byte . 👌 We then get the equality: This is what the relocation indicates. It tells the linker it is safe to “relax” and modify the instructions to the more performant variation. When we enable relaxation, we now generate the same code as above but with this new relocation type instructing the linker to perform the optimization if possible. Note Why not just always optimize when possible and forgo introducing a new relocation? My own guess is that it’s important to be backwards compatible and you wouldn’t want the emitted code to vary depending on the linker version but I would be interested to hear something more concrete if you know! Interestingly that many linkers, optimize this even further! Rather than generating a instruction, the linker instead prefixes the with ( ). On x86-64, ( ) usually implies 32-bit addressing for the operand. However, for a relative instruction, it acts as a benign prefix that effectively ignores the override but also consumes exactly 1 byte. If we go back to our example and enable relaxation, and produce a final binary, we can disassemble it to see whether it was relaxed. Here we can see that in fact our was relaxed since we can see 🥳. As it happens, you can do this same “relaxation” optimization for a few other instructions such as , and but the basic premise is the same.

0 views

Farid Zakaria 6 months ago

Bespoke software is the future

At Google, some of the engineers would joke, self-deprecatingly , that the software internally was not particularly exceptional but rather Google’s dominance was an example of the power of network effects: when software is custom tailored to work well with each other. This is often cited externally to Google, or similar FAANG companies, as indulgent “NIH” (Not Invented Here) syndrome; where the prevailing practice is to pick generalized software solutions, preferably open-source, off-the shelf. The problem with these generalized solutions is that, well, they are generalized and rarely fit well together. 🙄 Engineers are trained to be DRY (Don’t Repeat Yourself), and love abstractions. As a tool tries to solve more problems, the abstraction becomes leakier and ill-fitting. It becomes a general-purpose tax. If you only need 10% of a software solution, you pay for the remaining 90% via the abstractions they impose. 🫠 Internally to a company, however, we are taught that unused code is a liability. We often celebrate negative pull-requests as valuable clean-up work with the understanding that smaller code-bases are simpler to understand, operate and optimize. Yet for our most of our infrastructure tooling, we continue to bloat solutions and tout support despite miniscule user bases. This is probably one of the areas I am most excited about with the ability to leverage LLM for software creation. I recently spent time investigating linkers in previous posts such as LLVM’s lld . I found LLVM to be a pretty polished codebase with lots of documentation. Despite the high-quality, navigating the codebase is challenging as it’s a mass of interfaces and abstractions in order to support: multiple object file formats, 13+ ISAs, a slough of features (i.e. linker scripts ) and multiple operating systems. Instead, I leveraged LLMs to help me design and write µld , a tiny opinionated linker in Rust that only targets ELF, x86_64, static linking and barebone feature-set. It shouldn’t be a surprise to anyone that the end result is a codebase that I can audit, educate myself and can easily grow to support additional improvements and optimizations. The surprising bit, especially to me, was how easy it was to author and write within a very short period of time (1-2 days). That means smaller companies, without the coffer of similar FAANG companies, can also pursue bespoke custom tailored software for their needs. This future is well-suited for tooling such as Nix . Nix is the perfect vehicle to help build custom tooling as you have a playground that is designed to build the world similar to a monorepo. We need to begin to cut away legacy in our tooling and build software that solves specific problems. The end-result will be smaller, easier to manage and better integrated. Where this might have seemed unattainable for most, LLMs will democratize this possibility. I’m excited for the bespoke future.

AI

Rust

0 views

Farid Zakaria 6 months ago

Huge binaries: papercuts and limits

In a previous post , I synthetically built a program that demonstrated a relocation overflow for a instruction. However, the demo required I add to disable some additional data that might cause other overflows for the purpose of this demonstration. What’s going on? 🤔 This is a good example that only a select few are facing the size-pressure of massive binaries. Even with which already is beginning to articulate to the compiler & linker: “Hey, I expect my binary to be pretty big.”; there are surprising gaps where the linker overflows. On Linux, an ELF binary includes many other sections beyond text and data necessary for code execution. Notably there are sections included for debugging (DWARF) and language-specific sections such as which is used by C++ to help unwind the stack on exceptions. Turns out that even with you might still run into overflow errors! 🤦🏻‍♂️ Note Funny enough, there is a very recent opened issue for this with LLVM #172777 ; perfect timing! For instance, assumes 32-bit values regardless of the code model. There are similar 32-bit assumptions in the data-structure of as well. I also mentioned earlier about a pattern about using multiple GOT, Global Offset Tables, to also avoid the 31-bit (±2GiB) relative offset limitation. Is there even a need for the large code-model? How far can that take us before we are forced to use the large code-model? Let’s think about it: First, let’s think about any limit due to overflow accessing the multiple GOTs. Let’s say we decide to space out our duplicative GOT every 1.5GiB. That means each GOT can grow at most 500MiB before there could exist a instruction from the code section that would result in an overflow. Each GOT entry is 8 bytes, a 64bit pointer. That means we have roughly ~65 million possible entries. A typical GOT relocation looks like the following and it requires 9 bytes: 7 bytes for the and 2 bytes for . That means we have 1.5GiB / 9 = ~178 million possible unique relocations. So theoretically, we can require more unique symbols in our code section than we can fit in the nearest GOT, and therefore cause a relocation overflow. 💥 The same problem exists for thunks, since the thunk is larger than the relative call in bytes. At some point, there is no avoiding the large code-model, however with multiple GOTs, thunks and other linker optimizations (i.e. LTO, relaxation), we have a lot of headroom before it’s necessary. 🕺🏻

Backend

C++

0 views

Farid Zakaria 6 months ago

Huge binaries: I thunk therefore I am

In my previous post , we looked at the “sound barrier” of x86_64 linking: the 32-bit relative instruction and how it can result in relocation overflows. Changing the code-model to fixes the issue but at the cost of “instruction bloat” and likely a performance penalty although I had failed to demonstrate it via a benchmark 🥲. Surely there are other interesting solutions? 🤓 First off, probably the simplest solution is to not statically build your code and rely on dynamic libraries 🙃. This is what most “normal” software-shops and the world does; as a result this hasn’t been such an issue otherwise. This of course has its own downsides and performance implications which I’ve written about and produced solutions for (i.e., Shrinkwrap & MATR ) via my doctorate research. Beyond the performance penalty induced by having thousands of shared-libraries, you lose the simplicity of single-file deployments. A more advanced set of optimizations are under the umbrella of “LTO”; Link Time Optimizations. The linker at the final stage has all the information necessary to perform a variety of optimizations such as code inlining and tree-shaking. That would seem like a good fit except these huge binaries would need an enormous amount of RAM to perform LTO and cause build speeds to go to a crawl. Tip This is still an active area of research and Google has authored ThinLTO . Facebook has its own set of profile guided LTO optimizations as well via Bolt . What if I told you that you could keep your code in the fast, 5-byte small code-model, even if your binary is 25GiB for most callsites. 🧐 Turns out there is prior art for “Linker Thunks” [ ref ] within LLVM for various architectures – notably missing for with a quote: “i386 and x86-64 don’t need thunks” [ ref ] What is a “thunk” ? You might know it by a different name and we use them all the time for dynamic-linking in fact; a trampoline via the procedure linkage table (PLT). A thunk (or trampoline) is a linker-inserted shim that lives within the immediate reach of the caller. The caller branches to the thunk using a standard relative jump, and the thunk then performs an absolute indirect jump to the final destination. LLVM includes support for inserting thunks for certain architectures such as AArch64 because it is a fixed-size instruction set (32-bit), so the relative branch instruction is restricted to 128MiB. As this limit is so low, has support for thunks out of the box. If we cross-compile our “far function” example for AArch64 using the same linker script to synthetically place it far away to trigger the need for a thunk, the linker magic becomes visible immediately. We can now see the generated code with . Instead of branching to at , it branches to a generated thunk at (only 16 bytes away). The thunk similar to the large code-model, loads with the absolute address, stored in the , and then performs an absolute jump ( ). What if supported this? Can we now go beyond 2GiB? 🤯 There would be some more similar thunks that would need to be fixed beyond instructions. Although we are mostly using static binaries, some libraries such as may be dynamically loaded. The access to the methods from these shared libraries are through the GOT, Global Offset Table, which gives the address to the PLT (which is itself a thunk 🤯). The GOT addresses are also loaded via a relative offset so they will need to changed to be either use thunks or perhaps multiple GOT sections; which also has prior art for other architectures such as MIPS [ ref ]. With this information, the necessity of code-models feels unecessary. Why trigger the cost for every callsite when we can do-so piecemeal as necessary with the opportunity to use profiles to guide us on which methods to migrate to thunks. Furthermore, if our binaries are already tens of gigabytes, clearly size for us is not an issue. We can duplicate GOT entries, at the cost of even larger binaries, to reduce the need for even more thunks for the PLT . What do you think? Let’s collaborate.

0 views

Farid Zakaria 6 months ago

Failing interviews

My blog has been a little quiet. I recently accepted a new role at Meta and it’s been keeping me busy! Once the onboarding phase is done I hope to get back to my Nix contributions. Accepting the position at Meta has had me reflecting on my journey to this current role. People often share their highlights of accepting a new role but rarely their lowlights . I wanted to share a brief look at what interviewing might be like in the software industry. People are often discouraged by failure but it’s part of the process. I remember having done interview training at Google where they discussed most interviewers decide on the outcome of the interview within the first-five minutes. That story is not to totally discourage oneself from the process but rather to demonstrate there is a portion that is out of your control. Going through my emails to get an accurate accounting is challenging, however I found threads as early as 2011 interviewing for Facebook. I am actually sure I had interviewed ealier through my co-ops at University of Waterloo, but I don’t have access anylonger to those emails. 😩 Some rough dates I had found: 2011, 2014, 2015, 2018, 2019, 2020, 2021, 2022, 2023*, 2024, 2025. * This interview round was long and was for 3 distinct roles. Across those years, the level I interviewed at was different and sometimes the role too (IC vs EM). Don’t be discouraged from failure.

Career

1 views

Farid Zakaria 8 months ago

Nix derivation madness

I’ve written a bit about Nix and I still face moments where foundational aspects of the package system confounds and surprises me. Recently I hit an issue that stumped me as it break some basic comprehension I had on how Nix works. I wanted to produce the build and runtime graph for the Ruby interpreter. I have Ruby but I don’t seem to have the derivation, , file present on my machine. No worries, I think I can it and download it from the NixOS cache. I guess the NixOS cache doesn’t seem to have it. 🤷 This was actually perplexing me at this moment. In fact there are multiple discourse posts about it. My mental model however of Nix though is that I must have first evaluated the derivation (drv) in order to determine the output path to even substitute. How could the NixOS cache not have it present? Is this derivation wrong somehow? Nope. This is the derivation Nix believes that produced this Ruby binary from the database. 🤨 What does the binary cache itself say? Even the cache itself thinks this particular derivation, , produced this particular Ruby output. What if I try a different command? So I seem to have a completely different derivation, , that resulted in the same output which is not what the binary cache announces. WTF? 🫠 Thinking back to a previous post, I remember touching on modulo fixed-output derivations . Is that what’s going on? Let’s investigate from first principles. 🤓 Let’s first create which is our fixed-output derivation . ☝️ Since this is a fixed-output derivation (FOD) the produced path will not be affected to changes to the derivation beyond the contents of . Now we will create a derivation that uses this FOD. The for the output for this derivation will change on changes to the derivation except if the derivation path for the FOD changes. This is in fact what makes it “modulo” the fixed-output derivations. Let’s test this all out by changing our derivation. Let’s do this by just adding some garbage attribute to the derivation. What happens now? The path of the derivation itself, , has changed but the output path remains consistent. What about the derivation that leverages it? It also got a new derivation path but the output path remained unchanged. 😮 That means changes to fixed-output-derivations didn’t cause new outputs in either derivation but it did create a complete new tree of files. 🤯 That means in nixpkgs changes to fixed-output derivations can cause them to have new store paths for their but result in dependent derivations to have the same output path. If the output path had already been stored in the NixOS cache, then we lose the link between the new and this output path. 💥 The amount of churn that we are creating in derivations was unbeknownst to me. It can get even weirder! This example came from @ericson2314 . We will duplicate the to another file whose only difference is the value of the garbage. Let’s now use both of these in our derivation. We can now instantiate and build this as normal. What is weird about that? Well, let’s take the JSON representation of the derivation and remove one of the inputs. We can do this because although there are two input derivations, we know they both produce the same output! Let’s load this modified derivation back into our and build it again! We got the same output . Not only do we have a trait for our output paths to derivations but we can also take certain derivations and completely change them by removing inputs and still get the same output! 😹 The road to Nix enlightenment is no joke and full of dragons.

Ruby