Posts in Ci-cd (20 found)
Farid Zakaria 1 weeks ago

The intensional model in disguise

Despite how different they look on the surface, Nix, Bazel and buck2 are remarkably similar under the hood. Don’t believe me? I built a prototype in one store, two frontends to prove it. Turns out though that the similarities are even deeper than I realized. Upon further reflection, I realized that the intensional model is an even more substantial common denominator. What is the intensional model? TIP I’ve written before about Nix’s intensional model , better known as content-addressed (CA) derivations. It has been “experimental” in Nix for years. 🧪 In the intensional model, outputs are addressed by their content , and actions refer to their inputs by content as well. The extensional model instead addresses an output by the recipe and inputs that produced it. Nix also calls this “content-addressed derivations” (CA derivations). Bazel and buck2 don’t name their model, but they use content-addressing as well. Consider this coincidence: Eelco’s PhD thesis was published in 2006, the same year that Google started development on Blaze, the predecessor of Bazel [ cite ]. Let’s take a closer look at the three systems with a simple C example and see how all three implement the intensional model. 🥸 In the traditional extensional model (input-addressed), a change anywhere upstream of a build action invalidates the action’s output. This occurs even if the change is irrelevant to the output, such as a comment change in a source file. In contrast, the intensional model (content-addressed) only invalidates an action’s output if the content of the output itself changes. In our C example, if we were to change the comment in from to , the compiled object file would still be byte-for-byte identical, as the comment does not affect the compiled output. In the ideal world, the linking of would not need to be redone, as its inputs have not changed in content. By default, Nix uses the extensional (input-addressed) model. If we edit the comment and rebuild: Everything rebuilt. 😩 The store path in the extensional model is a hash of the recipe , and the recipe for includes the recipe for . Recursion is fun! Change one byte anywhere upstream and every path downstream changes with it. 🌀 Warning The two binaries are not even byte-identical: the cc-wrapper puts in the , so the binary contains its own store path. The extensional model leaks into the bytes. 🙃 If we enable the build to use content-addressed derivations we observe rebuilt, but did not. 🎉 You can read demystifying Nix’s intensional model for a more detailed explanation of how CA derivations work, but the short version is that Nix rewrites to point at ’s content-addressed output instead of . The resolved is the same one we built before the edit. Since no “input” changed, Nix avoids the rebuild. This is the same behavior as Bazel and buck2, which are intensional by default. If we edit the comment and rebuild with Bazel or buck2, we see that the compile ran, but the link was an action cache hit . Bazel and buck2 do not have an input-addressed mode. They always use the intensional model. An action’s key is a hash of its command line (to force rebuilds) and the content digests of its inputs. A change only travels as far as the bytes actually change. This optimization is called early cutoff , and it is the whole selling point of the intensional model. There is a clear mapping between Nix’s CA derivations and Bazel’s action cache. A resolved derivation is a Bazel/buck2 action. 🤯 You may see the output paths in Bazel and buck2 and think they are extensional (i.e. ), but they are just a convenient place on disk. The real addresses are the digests. can even make this explicit with . Until dynamic derivations stabilize, people will continue to reach for Bazel and buck2 for the granularity on large builds. The intensional model is a big part of why they are so fast, and Nix has the same model. The two implementations are nearly identical in concept. Nix described the intensional model in 2006. Google started building on it the same year. Twenty years later, maybe it’s time they compared notes. 🥸

0 views
Blog System/5 3 weeks ago

BazelCon 2026 recap

We are right in the middle of fall, and that means BazelCon 2026 just took place. This time around it was hosted in Amsterdam and spanned three days: one day of tutorials and an overlapping Buildbarn meetup that I wasn’t aware of, and two days of main-conference talks and events. And as the event is behind us, it’s time for the now-traditional recap. Welcome to BazelCon 2026 sign. What follows is a recollection of my notes arranged by topic and intermixed with my own commentary and thoughts. Some sections and topics are more detailed than others because in some cases I didn’t have a seat and couldn’t type enough, and in others I got distracted in the hallways chatting with people. Speaking of people, several of you approached me to praise previous editions of these recaps, so I now feel obliged to write another one. Thanks for the kind comments! As we will see, “scale” was a central theme of the conference: AI is multiplying concurrent builds and tests, artifacts and graphs are growing at an accelerated pace due to new features like configuration transitions, and the ecosystem is responding by making execution, caching, and analysis more efficient. There is a lot to cover but here are a few things that felt memorable to me : The BUILD Foundation is now a reality, giving the Bazel ecosystem a vendor-neutral home. Its goal is to act as a proxy between the community and Google to help Google understand what to prioritize and to fund projects. If your company can spare “a few” dollars, please donate! AI-driven development is having a broad impact on the industry and, in particular, everyone is seeing their CI systems become overloaded. Well, news at 11: I’m dealing with this day in and day out. But what was new to me was that we at Snowflake are not the only ones struggling with local system overload. Lots of people out there have likewise “discovered” Git worktrees as a way to run multiple agents in parallel, which in turn means multiple concurrent Bazel servers. Bazel 10 is shaping up to be a substantial release that will help with the problem right above due to its many performance improvements and memory consumption optimizations. Have you struggled to keep Bazel up-to-date before? These days, if you want to get any platform-related work funded you have to spin it in terms of AI, so this release will be an easy sell. Let’s “Rewrite it in Rust”? I’m impressed. Google is seriously evaluating a progressive rewrite of Bazel in Rust, and this is probably the most exciting news I got out of the conference. You know I’ve been critical of Bazel’s usage of Java since day one so, if this takes off, I’ll be the first in line to evaluate the results (and it will likely push me to actually contribute to Bazel again). Fast and correct sandboxing on macOS remains a problem. This is “interesting” to me because it’s a problem I tried to fix almost ten years ago with the original sandboxfs, but I failed because of unexpected bugs in macOS’s FUSE support and eventually abandoned the idea when FUSE became a dead end on Apple’s platforms. Funnily enough, other folks have now reinvented sandboxfs—and even given it the same name (ugh)—and there is even another implementation of a similar idea in the form of actiondFS. Types are coming to Starlark, but Bazel’s late arrival to the party means that multiple competing approaches now exist. We have at least two LSPs with different annotation mechanisms and Buck 2’s own syntax. Hopefully they can all converge. Platforms and configurations remain difficult to model at scale. Careless use of transitions and build settings can produce an explosion of configured actions. Apparently, as one speaker noted, AI agents love transitions for some reason, which is interesting because they are not a popular feature and they are not particularly easy to use. Let’s dive into the full report now. You’ll find my notes arranged by talk but I’ve tried to group them into higher-level themes for easier navigation. These conference summaries take effort to write, and some of you are waiting for them. Show your true support by subscribing! The two keynotes are usually the most information-packed events but… as you can imagine they are also the most popular. The venue this year was not roomy enough to host BazelCon’s attendance (Helen quoted that we got 600 registrations), so I had to stand for these two events. The recordings and slides are not out yet, so consequently, I can only tell you a few details from memory. The conference opened with the usual Bazel Community Update , presented by Helen Altshuler from EngFlow and Chuck Grindel from Glydways. The biggest announcement is that the BUILD Foundation is now a reality. The effort had been discussed at the previous two BazelCons and likely various meetups, but it had remained mostly invisible while its organizers worked through the legal and financial details. The foundation now provides a neutral organization to support the broader build ecosystem (in their words this is not just for Bazel), and it is looking for both members and concrete proposals to fund. If you want to get involved, get your company pumped up and email [email protected] . Documentation continues to receive attention as it has always been a lowlight in previous surveys. Work is underway to make the docs easier for the community to own and improve by decoupling them from Google’s internal systems. But one must wonder: what exactly is the biggest problem with documentation in today’s world? AI agents used to do terribly with Bazel just a year ago and nowadays do… fine as we shall see in the summary. Maybe the work on documentation needs a different spin now. And as usual, a reminder to fill the annual Bazel user survey . Your input is necessary to quantify ecosystem-wide pain points and influence future work, especially now that the foundation exists. John Field and Tobias Werth from Google followed with the State of the Union , which presents a recap of major improvements during the year and the path ahead for Bazel 10 and beyond. Performance remains a major theme. Google reported improvements of about 15% in some workloads together with reductions in memory consumption. Some of these speedups come from upgrading from Java 25 to 27, while others come from correct action rewinding, optimizations to Skyframe cache pruning, and content-defined chunking. These changes are very relevant because Bazel’s large memory consumption gets in the way of scaling CI and prevents using multiple AI agents in parallel. And here is where we heard about Skymetal: the codename for the Google-internal exploration of rewriting Bazel in Rust. The rationale for this rewrite is obviously performance, and I’m guessing it’s now a real consideration because AI can make such large-scale changes feasible. Tobias acknowledged that Java was a reasonable choice back in 2006 when Blaze started and that the JVM can deliver great throughput performance at scale… but it is not the best choice for a tool like Bazel in this day and age, especially when optimizing for memory efficiency is critical. I called it in 2015. I have been in the BazelCon Program Committee (PC) for 3 years now. I remember that, during last year’s review, we encountered a few AI talks but I think none made the cut: they all looked too hype-y and out of place. But it was impossible to ignore the topic this year, and it was actually important to bring AI-related talks on board to not sound tone deaf. AI is a reality in our industry, and Bazel is a key component to survive the transition. Nicolas Lopez from EngFlow and I, representing Snowflake, presented AI Development: Challenges and Early Learnings for Bazel . Agentic workflows have expanded rapidly throughout 2026, and the effects are visible well beyond code generation. At Snowflake, we were fortunate enough to finish our Bazel migration and to stabilize remote execution by moving from our self-hosted open-source deployment to EngFlow’s fully-managed service before all of this happened. And I say fortunate because the increase in CI load that AI workflows brings can only be handled with a good build system and caching story. But none of that is news, so the core of the talk was to look at how Bazel usage looks different in agent-initiated environments. The key points from my side: Agents run more builds more frequently than humans, discover and invoke targets that were often “internal only”, and explore several approaches in parallel. Your intuition about how to evaluate which builds matter and how to measure them is probably wrong by now. Bazel and IDEs have never played great together and maybe that just doesn’t matter anymore. We are seeing a sharp decline in IDE usage, although developers still want a way to browse code changes and debug them. This means LSPs are key now to support “leaner” IDEs, and maybe standalone debuggers. Short build and test latency is critical when a human must remain focused on a coding task but, in an agentic workflow, the constraints are different. You know that, after you hit “execute this plan”, the agent will take many minutes to return so minimal build latency is not as important. And actually… maybe you now want to trade some latency in exchange for extra validations so that agents have to take fewer turns to converge. And because humans are now waiting for agents to finish for long periods of time, they have to fill that time with other tasks, which has driven them to adopt Git worktrees en masse. This is leading to scalability problems on dev machines because multiple concurrent Bazel server processes are not great for memory and disk usage. The rest of the talk, presented by Nicolas, focused on trying to understand behavioral changes by looking at anonymized backend data across companies. One specific insight was that people’s work days have lengthened over the year. Cynically, the promise of “working less because of AI” has not materialized: in fact, we are all seemingly working longer hours and with fewer chunks of focus time, so the dangers of burnout are higher. Maggie Lou from BuildBuddy presented Taking Humans Out of the Build Loop . The premise was that coding agents have improved quickly, but people still spend too much time copy/pasting logs, triggering builds, explaining failures, and reviewing the results. In her presentation, we witnessed many new features in BuildBuddy to allow agents (and humans!) to interact and diagnose remote builds with ease: to download filtered logs for a build. to answer questions such as “Why was my build slow?” which are extremely common in the Bazel ecosystem. to start a remote Firecracker VM and run Bazel inside of it. The VMs are snapshotted for quick reuse, and the runners can execute arbitrary commands rather than Bazel alone. to diagnose a failed build and either create a pull request or push a commit with the proposed correction. to perform a review and post comments to GitHub. The need for these features is unsurprising this late in 2026, but what’s really useful in their offering is how these features integrate with build data. Inspecting the BEP, profiles, remote execution metadata, and logs in unison via an agent can yield much better diagnostics and automated improvements than staring at just a few lines of console output. One thing that surprised me in chatting with the presenter is that BuildBuddy is not particularly big in terms of headcount but they churn out features at a very high rate—and that was true even before AI. Impressive! Armando Montanez (and Ulf Adams, but he didn’t make it) from EngFlow presented Proving Equivalence in any2bazel Build System Migrations . The talk described a structured way to use AI agents for build system migrations and started with a reflection: AI agents used to be terrible at Bazel less than a year ago, but now they do pretty damn well—especially if you can close the loop with automated validations. However, even if agents do well at writing Bazel builds, the fidelity of a migration leaves a lot to be desired if left unattended. And here is where any2bazel comes into play: it turns the process into a guess-and-verify loop. The initial focus of the tool are CMake-to-Bazel migrations, and the tool comes with a script that extracts reference compiler and linker actions from CMake in a deterministic manner. The agent then writes Bazel build files, runs , and feeds the resulting actions to a differ tool that compares those to the originals. The loop continues until the two action sets are equivalent. The equivalence checks are what’s hard though: there are compiler flags that may have no impact on the result, for example, so they need to be “ignored” to let the agent experiment with different Bazel-native approaches until it gets to bit-for-bit identical outputs. It’s important to focus on comparing outcomes and not internals so that the generated Bazel builds do not necessarily follow ancient idioms, like e.g. a Maven-like project structure in a Bazel build. In the end, though, this approach works surprisingly well. But the question is: “How good are the generated build files?” The presenters evaluated them along four axes: Correctness: Good, because equivalence is the central constraint of the loop. Hermeticity: Good, but you might need to do extra upfront work to remove host-specific behavior from the reference build before starting. Efficiency: Usually good when standard rules can model the build. Complex projects with custom build rules may end up with overly large Bazel actions though. Maintainability: Unsurprisingly, the weakest point. Agents tend to model variation with configuration flags instead of proper platforms and toolchains, which is not idiomatic to Bazel. Expert review of AI outputs remains critical, which is, again, not surprising. As a side note, it seems like the team likes pain because they tried to migrate an MSBuild project too. MSBuild is more painful than others because its binary log requires running through a real build: there is no dry-run mode, so it might be difficult to extract a truly clean log of actions, and the process is slow. Fortunately, the reference needs to be extracted only once before the comparison agentic loop can start, and it works: we saw a POC migration of the Color Picker tool that comes from PowerToys. When optimizing a distributed build system, it’s easy to focus on the costs of the remote execution workers: how many there are, what kinds of hardware they use, how well they auto-scale… but a significant driver in cost comes from the storage systems required to support those workers. In other words: the remote cache is expensive. Let’s look at a few topics that deal with this problem. Tyler French from BuildBuddy and Benjamin Ingberg from Meroton presented Content-Defined Chunking for Bazel . This was one of the most immediately-practical talks for anyone operating a remote cache, and it surely caught my eye because I had just been troubleshooting a problem at work that CDC would have prevented in the first place. The problem is that Bazel has traditionally treated each file as one indivisible blob. If an action produces a one-gigabyte binary (not uncommon these days huh) and this binary only changes by a few bytes from a previous build, the client or remote worker must re-upload the entire new file to the CAS and the CAS must store a full copy of those bytes. This is inefficient network- and storage-wise because the cache has to be overprovisioned so that these very large, rarely reused blobs don’t prematurely evict valuable cache entries. Content-defined chunking (CDC) divides a file into variable-sized pieces that can be cached independently. To compute the chunks, a rolling hash scans the contents and chooses boundaries when the hash matches a particular condition. Because the boundaries depend on the content rather than fixed offsets, inserting a few bytes in the middle of a file shifts only the nearby chunks instead of invalidating every subsequent one. With CDC, Bazel represents the original file as a list of chunk offsets and digests instead of a single digest, and each chunk is uploaded or downloaded independently. The presenters reported average reuse of chunks of 36%, chunks of 40%, chunks of 96%, chunks of 98%, and chunks of 99.5%. Furthermore, BuildBuddy estimates that CDC saves 35% of bytes written, deduplicates 60% of chunked bytes, and avoids about 400 TB of traffic per day across its fleet. Finally, the median project sees about 70% less remote storage, while local disk caches shrink by roughly 40%. Impressive numbers. Almost all of these benefits can be achieved by enabling CDC on the backend alone without changes to Bazel because most savings come from worker-to-CAS communication. But there are improvements to be made on the client too, of course. CDC fits bb-clientd especially well because chunks can be reused across output bases and, to materialize a file, the client needs only its chunk map. This can be much cheaper than reconstructing the whole file on every incremental build. For existing users, the starting point is together with a server that supports remote CDC. You’ll want to use the latest Bazel 8.x or 9.x releases for best results, as earlier versions didn’t perform the necessary validations to ensure that CDC didn’t corrupt data. The talk ended with a plea for tool authors to rethink binary formats: files that contain absolute pointers in them are poison for caching even with CDC because small changes in a file can have cascading effects across it. Wherever possible, relative pointers are better. Lore is Epic Games’ open-source, content-addressed version control system for very large, asset-heavy monorepos. Mattias Jansson from Epic Games came on stage to talk about it. But this was not just a presentation of the version control system. This was also a presentation of how Lore integrates with Bazel to prevent ferrying large assets between Lore repositories, the machine running Bazel, and the remote execution/cache cluster. The insight is that Lore and Bazel’s remote cache are both backed by a CAS… so if they can share the same storage, artifacts don’t need to move around. The idea relies on the fact that Bazel-the-client does not need to know the content of source files. All Bazel needs to know are the digests of those files to detect file changes across builds and to compose Remote Execution (RE) API requests. You’d imagine that computing digests requires reading the files, and you’d be right… but the server (Lore) already has those files, can compute their digests once, and can expose those to Bazel for consumption. So, what Epic Games has built is a translation layer between the RE API and Lore’s chunked content addresses, plus a virtual file system that exposes the precomputed digests to Bazel. In this way, Bazel talks to just one storage system and can leverage digests without computing them itself. Of interest was a brief note that Bazel can now use BLAKE3 instead of SHA-256 for content digests, which can likely be a big deal for performance because the former excels in speed due to parallel chunk processing. TIL. Caleb Froese from Canva presented Diffing Millions of Screenshots a Day, Hermetically . The premise of the talk is that, when developing a UI, you really want to have regression tests for how the UI looks —and doing that at scale is difficult. The idea is simple, though: make the code render a part of your UI, capture a screenshot, and compare it against a known-good image. Check in the golden image into source control, and call it a day. Except… if you do that at Canva’s scale, you end up growing the Git repository by about 20 GB per day , which is obviously untenable due to how Git works. A different approach is to store the images elsewhere and expose them to tests through an external repository rule. This integrates well with Bazel in the sense that the downloads are out of the execution path, but it also scales poorly: too much data must be downloaded to each client just for it to be re-uploaded right away to remote executors. And it also creates friction for developers when defining the build graph. In the end, Canva’s solution is to use the Build Event Service and the remote CAS to keep the assets outside of Git while preserving hermetic tests. This pattern applies to any test suite with large golden artifacts, not screenshots alone. Similar story to what Lore tries to fix. There is one extra wrinkle to worry about: GPU-based rendering introduces tiny output differences that make pixel-perfect image comparisons difficult. Canva’s solution is to pin these tests to a specific hardware class by selecting an execution pool through the execution property. That trades scheduling flexibility for reproducible images, which is a good compromise for their workload. Optimizing the way Bazel handles large artifacts helps with performance, but there are many other things that do too. There were a large number of talks around this topic, so let’s dive into some of them. Guillaume Maudoux from Tweag presented Taming Configurations Explosion , a practical talk addressing problems they’ve seen while supporting builds that target physical hardware boards. Such builds grew to have more than three million configured actions even though only about 500,000 distinct actions needed to run. The root problem is that configuration settings accumulate. A project may begin with a setting for the target platform, then add the board’s operating system, its physical layout, its capabilities, and so on. Every setting creates a new dimension in the configuration space and, because each configuration gets a distinct output-tree hash, apparently-different actions may in fact do identical work. Before you know it, you have blown up the action graph and possibly your disk space too. In the example from the talk, 12 parameters end up representing 72 million possible configurations. Only 856 of those configurations matter, and a particular source file has just 39 meaningful ways to be compiled. The rest is duplication introduced by the model. There are three main tools to fight this: Reset transitions: These discard irrelevant configuration values as early as possible in the graph. The art is to move decisions up the dependency tree so that downstream targets do not carry unnecessary dimensions. The trade-off is that ordinary output can become unmanageable, forcing users toward . But then… requires a configuration, so some targets may not be queryable at all without one. Path mapping: If two actions cannot be made identical during analysis, Bazel can rewrite configuration-specific paths during execution and replace their hashes with a stable component. This should be seen as a measure of last resort, though, because subtle problems arise. For example, if the same logical path comes from different files from different configurations, Bazel disables path mapping silently. Tweag fixed Bazel so that different copies of the same file don’t disable path mapping, but debuggability continues to be a problem when this condition does not apply. Flag scopes: This new feature allows a flag to declare where in the graph it applies and to automatically reset to its default everywhere else. This is similar to a reset transition but requires less manual graph plumbing. On the other hand, describing the correct scope can sometimes be harder than writing the transition directly. Han-Wen Nienhuys from EngFlow presented Horapha: Your Personal Warm Bazel , a clever experiment to avoid repeatedly starting Bazel servers and rebuilding their in-memory state. Several CI systems already offer similar features for efficiency, but Horapha targets developer-initiated workflows. The idea is to snapshot a warmed-up Bazel JVM with a new command and save that to disk. A later either starts a normal server when no checkpoint exists or restores the snapshot and continues from the saved state. The implementation relies on CRIU (Checkpoint/Restore In Userspace) , a Linux-only technology that can serialize a running process to disk. Correct restoration requires preserving process IDs, which can be achieved by placing Bazel in a PID namespace. In other words: a container but without pulling in the full weight of Docker. The prototype is available in the Horapha repository with the corresponding Bazel changes in the horapha branch of Han-Wen’s Bazel fork . Han-Wen is now trying to carve a path to get this integrated into upstream Bazel. Did you know that Emacs works similarly? Your executable is actually a snapshot of the LISP machine computed during the build process and stored to disk by the portable dumper introduced in Emacs 27. John Cater and Antonio Di Stefano from EngFlow presented Persistent Test Runners . For a long time, Bazel has had persistent workers for compilation actions but supporting tests has remained elusive—yet persistent workers for tests promise to increase test performance significantly by reducing JVM startup times, sharing heavyweight fixtures across tests, and sharing state such as container images. The reason I said elusive is because Bazel has tried to solve this before. Twice. Google introduced a persistent test feature back in 2019 but it was rolled back that same day. The roll forward was attempted again a month later, already in 2020, but this time separating the feature flag from the runner. Unfortunately, the whole mechanism was scrapped in 2022 because it never worked reliably. The old design suffered from classpath problems after incremental builds, leaked state between tests, and was tightly coupled to Bazel’s native rule—an unacceptable constraint due to the desire to move all rules to Starlark. More fundamentally, the test protocol and the persistent worker protocol modeled very different things. The new proposal from EngFlow introduces an API that explicitly separates the settings and inputs used to create the runner from the data needed by one test request. That distinction is important because the runner infrastructure must be able to decide whether an existing JVM is still compatible with a new test. The team explored several alternatives along the way: Pre-warming the next JVM instead of keeping the current one alive. This provided more control over when and why to restart a test in a new JVM, but suffered from significant coordination overhead. Forking a JVM. This did not work because, due to POSIX semantics, only the calling thread survives a fork call. The JVM is heavily multithreaded, so if you end up losing all but one thread, the new JVM is just not going to work. Android’s Zygote. This provides a pre-warmed runtime ready to be forked ( , anyone?), but it exists only for ART, not the standard JVM, and it’s just not reasonable to expect a switch to ART. Checkpointing. Instead of forking a running process, why not snapshot it to disk so that it can be re-created again? This might have worked, like Horapha does for Bazel startup, but raised difficult questions about when to capture the process and how to handle state outside the JVM. In the end, the only piece that survived these experiments was the coordinator that asks the runner whether a JVM can be reused. The decision considers the Java executable, JVM options, driver code, plugin configuration, and the digests of all loaded JARs. Configurable hooks run at JVM start/stop events as well as before/after each test. When running Bazel’s own test suite, this approach reduced total test time by 46%. The prototype is available in the branches of TheGrizzlyDev/bazel and TheGrizzlyDev/rules_java . It can be tried with a local module override and . Unconference discussion topic proposals and votes for the second day. Remote execution BoF Chi Wang and Tiago Quelhas led the Remote Execution Birds of a Feather . I only stayed for half of it, and so I only captured the discussion around remote persistent workers. Persistent workers are already part of several commercial remote execution offerings, including BuildBuddy and EngFlow, but their implementation was derived from Bazel’s local persistent worker support, which is not… particularly well designed. For example, we have at least three different variants of the persistent worker protocol: one for “regular” persistent workers, another for multiplex persistent workers, and yet another for multiplex persistent workers that support cancellation. And the protocol does not clearly specify which flags and tools belong to the worker and which belong to the action being executed. In fact, these distinctions are currently determined heuristically, with the worker assuming that flag files correspond to the action and other flags to the worker. There was consensus that this should be cleaned up, but the unsatisfying conclusion is that rehashing these complaints and discussions is not going to lead anywhere until someone sits down and presents a specific proposal for a new API. So, if you are reading this and have… opinions… volunteer to do this. Helen Altshuler from EngFlow and Nikita Chepanov from Plaid presented Where Bazel Clusters Break at Scale: Field Notes from Plaid and the EngFlow Fleet . That’s quite a mouthful to say “load testing”. And the reason is none other than AI once again: Plaid’s build volume doubled during the year due to AI-assisted coding and is on track to reach four times its early-2026 baseline. With a predictable increase in load comes a predictable outage. So, instead of waiting for it to happen organically, Plaid deliberately chose to load-test their CI system. The goal was to reach multiples of the normal load and discover each bottleneck under controlled conditions. They had to go through six rounds of tests and fixes until they were able to sustain their desired target. The previous five uncovered problems in network bursts, AWS quotas, source control, CI autoscaling, and cache infrastructure. The solutions were not magical: find and raise limits, cache everything possible, and reduce demand where it does not add value. Examples of the last category include skipping tests for pull requests in the middle of a PR stack and avoiding uploads for actions when they are unlikely to be reused. An interesting observation is that Plaid runs EngFlow in Plaid’s own AWS account, so the cache cluster is co-managed. But careful: shared ownership can easily turn into ambiguous ownership. The two companies addressed this by agreeing on health metrics and thresholds upfront and putting them on a shared dashboard. Both sides now know who must act before developers notice an outage. Ah, one of my favorite topics. Back in my Google days, I spent at least three years of my so-called 20% time working on improving the performance of the Bazel sandbox on macOS. Those efforts were painful: creating a writable FUSE file system is difficult, and diagnosing mis-interactions with macOS was a horrifying exercise. Today, I’m not scared anymore: I’ve been recently using AI to write yet another file system at work and agents do exceedingly well at handling corner cases. Let’s see how modern solutions do better than what I originally tried. David Zbarsky from OpenAI presented actiond: The Quest for a Better Sandbox , which was the first of at least two talks on improving sandboxing performance and strictness. The project started from frustration with local sandbox escapes on macOS and eventually grew into a custom file system, a local virtual machine, and a Bazel-built Linux kernel. actiond pretends to be a remote executor, but the “remote” worker is a Linux VM running locally on the Mac. Bazel communicates with it over a and actiond implements both execution and a content-addressable store. Funnily enough, leveraging this VM for action execution results in faster Bazel builds on macOS than running actions natively. Inside the VM, actiondFS mounts a directory described by a protocol buffer and creates the action sandbox without building a forest of links. Same idea as my original sandboxfs , but the key difference is that actiondFS is a kernel module rather than a FUSE file system so it solves all performance and deployment problems I originally faced with FUSE. A mind-blowing mention was linux.bzl : a Bazel-native build of the Linux kernel, used in this project to build the actiond VM end-to-end with Bazel. This reminds me that I’ve been saying, since the inception of Bazel, that this build system is the perfect pairing for a monorepo like NetBSD’s or FreeBSD’s; and, in fact, I recently wanted a Bazel-native build of NetBSD to put the EndBOX together. I wonder how difficult it would be to hack something up these days with AI… Oh, and: . This flag finally replaces the super-complex script that Bazel uses to run a test with a native binary. This resolves some compatibility issues across platforms and should result in measurably faster test execution. Borja Lorente from Byte Bard Software presented The Sandbox Is a Filesystem: Content-Addressed Bazel Builds on macOS with FSKit , yet another talk on sandboxing and macOS. The immediate problem they tried to address was sandbox escapes in TypeScript builds. You might not be aware, but there happen to be some tools out there that, when faced with a symlink, decide to resolve the target of the symlink and then perform extra operations on that location, like finding sibling files and the like. The compiler is one such tool, and consequently, escapes the symlink-based sandbox used on macOS and behaves incorrectly. Aspect originally patched the JavaScript rules to intercept Node.js file-system calls and redirect them, but that solution stopped working when was rewritten in Go. Apple’s new FSKit framework should in theory be usable to solve this, as it’s “just” a new way of writing file systems in user space. If we could leverage FSKit, the idea then becomes simple: represent the sandbox as a file system and mount a content-addressed view of the action’s inputs within it. Identical subtrees across actions share nodes, contents are fetched lazily, and outputs write through to the host volume. You know, exactly like my own sandboxfs did back in the day with FUSE. Unfortunately, FSKit comes with its own set of problems: An FSKit extension ships as an application, which must be packaged and notarized. This might be OK, but what’s bad is that, because this comes from an application, macOS displays a TCC (Transparency, Consent, and Control) popup notification when an app such as the Terminal tries to access the FSKit volume. Not cool for automated (CI) deployments. Mounting takes hundreds of milliseconds, making a separate mount per action too expensive. This can be solved by reusing the same mount across actions and making the file system create different subtrees per sandbox. Again, like the original sandboxfs. User space file systems have… unavoidable overheads as most operations have to cross the kernel/user space boundary at least twice. These may or may not be acceptable. With my own FUSE-based solution years ago, I just couldn’t fix some performance issues due to some oddities in OSXFUSE. AMFI and Gatekeeper inspect each executable once , adding substantial latency on first execution. And because any executable exposed through a separate file system looks brand new to the kernel… performance tanks. My original sandboxfs mitigated this by keeping inode numbers stable across different instances of the same file across different sandboxes, but that’s what ultimately led to the blockers I encountered with OSXFUSE. Because of all these problems, FSKit was not the clear choice for a new sandboxfs, so the team evaluated alternatives for performance like pure file copying, the old symbolic links, hard links, and APFS’ . FSKit was best at creating the initial directory structure, but later operations such as , , and were slower than with the alternatives. The conclusion was that there is no perfect answer to a fast and secure sandbox on macOS. Symlinks are fastest but allow some tools to escape. Hard links are the second-best option but they cannot protect source files against writes and only work within one volume. is efficient but limited to APFS and, again, one volume. FSKit provides the right abstraction with frustrating deployment and runtime costs. Because of this, the outcome of the talk is a proposal for a new sandboxfs protocol , not a specific implementation. Namely: a way for Bazel to send a manifest describing the inputs plus settings such as network access to a sandbox service. With that, the service can create the sandbox directory and return its path; Bazel can execute the action; and the service can then report the outputs and tear the sandbox down. Different backends can implement the same protocol using the best mechanism for each machine. I like the proposal and the fact that what was old is new again, but there were two interesting questions from the audience. One was: “What about an NFSv4 server?”, which is how Buildbarn exposes efficient virtual file systems on macOS; and the other was: “Can we not implement a new protocol and instead reuse the RE API?” Switching gears to the more theoretical aspects behind Bazel, we can look at changes to a core piece of the ecosystem: the Starlark language. The star (wink, wink) of the show in this area were type systems. Alexandre Rostovtsev from Google presented Static types in Starlark , an update on the type system first discussed during an unconference session at BazelCon 2025. Bazel 8.8 and 9.3 parse the new annotations but otherwise treat them as no-ops, allowing rulesets to adopt the syntax without breaking those releases. Bazel 10 will check annotated files and type-checking should be enabled by default by the time the final release ships. The syntax is a subset of Python’s type annotations with extensions to support Starlark’s native types like providers and attributes, both of which require typed exhaustive dictionaries. This should make large rulesets easier to navigate and maintain, and it lays the foundation for better automated tooling. Right after Google’s update on native Starlark types, Tamir Duberstein from OpenAI presented Type-checking Starlark without waiting for Bazel . I originally read this title as “type-checking without starting a slow Bazel process”—but no, the “waiting for Bazel” part here refers to the fact that the original type-system feature request was filed six (!!) years ago, not addressed in a timely manner, and the ecosystem could not simply stand still until the Bazel team decided to do something. During these six years, Buck 2 has come up with type annotations and a full-featured LSP for their Starlark variant. On the Bazel side, starpls implements another language server and type checker, but its annotations live in comments and the resulting experience is not ideal. OpenAI’s approach is sty , which combines ideas from with ty , the Python type checker developed by Astral. This new project provides fast cross-file checking, resolution, editor integration, and go-to-definition support while avoiding dependencies on a particular Bazel release. One difficulty is that, even if Starlark looks like Python, it isn’t Python: Bazel gives the language several domain-specific semantics. For example, a rule’s fields have types, mandatory status, and default values derived from the rule declaration. A generic Python checker knows none of this, so has to model it explicitly. By “pure” coincidence, OpenAI’s choices here do not diverge much from Google’s official typing system, so there is a path towards reconciliation. The project also introduces files, which act like external stub files and describe types for dependencies without modifying their source. These are particularly useful for rulesets that must continue to support Bazel versions that cannot parse type annotations, because the annotations can live out of sight from Bazel. The last topic I have notes on are talks about rulesets, but I do not have a lot. Let’s just cover a couple of the talks I did attend and take notes from. David Zbarsky from OpenAI presented : {Fast, Hermetic, Ergonomic} — Pick 3 , a new Rust ruleset built to address the pain points that OpenAI encountered with at scale. The usual dependency flow takes and , creates a synthetic workspace, runs , and generates a Bazel-specific . Repinning is slow—roughly O(n^2) in the size of the dependency graph—and Bazel has to repeatedly ask Cargo to contact registries and download metadata. rules_rs avoids that architecture. Bazel already has a downloader and lockfile, so the new ruleset leverages those directly and performs dependency discovery only once per platform. At OpenAI, repins dropped from about ten minutes to 100 milliseconds and can happen as part of an ordinary . Build configuration is read from , avoiding a separate Bazel lock file and the need to keep generated configuration in sync with Cargo. Rust build scripts remain a challenge because they can perform arbitrary work but appear to Bazel as one opaque action. For example, OpenAI replaced the build script in with a dedicated bzlmod module so that Bazel could see and schedule the underlying work. The ruleset also builds the Rust standard library from source, which enables Tier 3 targets and tools such as Miri . Combined with a hermetic LLVM toolchain, can cross-compile projects such as Selenium to Linux, macOS, and Windows. The question from the audience was simple: “What is the carrot for choosing over Cargo?” And the answer was simple as well: at OpenAI, they observed that everything in CI that did not use Bazel was falling over under agent-generated load. Antonio Di Stefano and Yannic Bonenberger from EngFlow presented No Bytes Left Untracked—Bazel Supply Chain Security . The key message was that, while we have always worried about correct builds and Bazel delivers on them, that’s just not enough anymore. We must also be able to explain where every byte comes from and enforce policy over their provenance in order to defend against security threats. Supply-chain attacks are becoming more frequent, and frameworks such as SLSA cover the complete software development lifecycle rather than compilation alone. They include requirements around review, provenance, and Software Bills of Materials (SBOMs). Bazel has always been in a unique position to provide this provenance information because it has a precise dependency graph and runs actions in controlled environments. But a common difficulty is that, outside of Google, Bazel interacts with other ecosystems: third-party package managers bring their own manifests and lock files and coordinating those with Bazel is difficult. There is no uniform representation of package identity, licensing, or provenance. The supply-chain ruleset addresses this with a standard target. The goal is for every package in the Bazel Central Registry to expose its identity and attributes, including licensing information, through a small JSON representation. The presenters asked for help in achieving this, as annotating packages is tedious and, ultimately, the responsibility of their owners. The processing model looks like this: A “gather” aspect walks the target graph of the artifact that needs to be shipped. Each dependency contributes metadata through a transitive provider, effectively producing a serialized view of the relevant graph. Tools convert that data into standard SBOM formats such as SPDX and CycloneDX. Validation actions enforce organization-specific policy, such as banning a particular license from entering a product. This is useful, but the target graph alone cannot fully account for all bytes in an output. True end-to-end attestation also has to inspect the action graph and the environment in which those actions run. Questions like these are to be expected: Which tools transformed the inputs? Where did the actions run? Did those environments have known kernel vulnerabilities? The project roadmap includes rules, module extensions, macro definitions, and stricter build-level enforcement. The 1.0 release was scheduled to happen during BazelCon, but unfortunately they’ll be a few days late due to last-minute issues with documentation and the like. And that’s all I captured this year. There were many other sessions—including talks on hermetic LLVM, Zig package integration, Gazelle v2, CUDA compilation, and richer platforms—but I either did not attend them or did not take enough notes to write anything of substance. Apologies if I did not cover yours here. Check the official BazelCon 2026 schedule for the complete program and stay tuned for the production-grade recordings . Photo while running towards Amsterdam Centraal. The common thread across the conference was scale. AI agents are multiplying the number of builds, tests, and changes that our infrastructure must process, while Bazel users continue to push larger graphs through increasingly-specialized execution environments. The good news is that the ecosystem is responding: Bazel 10 brings meaningful core improvements, remote execution keeps getting smarter, and the BUILD Foundation gives community work a place to grow. But even if the ecosystem is responding, you must too. Bazel is positioned to be a great answer to scalability issues in development workflows thanks to its safe cross-machine, cross-user caching features. And Bazel is also a good way to remain secure amid the unprecedented number of changes to your third-party dependencies. If you have not yet considered Bazel, the best time to do so is today. Stay tuned for the next article! I got some really cool swag from one of the vendors and I promised them I’d write something up once I had a chance to evaluate it. Welcome to BazelCon 2026 sign. What follows is a recollection of my notes arranged by topic and intermixed with my own commentary and thoughts. Some sections and topics are more detailed than others because in some cases I didn’t have a seat and couldn’t type enough, and in others I got distracted in the hallways chatting with people. Speaking of people, several of you approached me to praise previous editions of these recaps, so I now feel obliged to write another one. Thanks for the kind comments! As we will see, “scale” was a central theme of the conference: AI is multiplying concurrent builds and tests, artifacts and graphs are growing at an accelerated pace due to new features like configuration transitions, and the ecosystem is responding by making execution, caching, and analysis more efficient. My highlights There is a lot to cover but here are a few things that felt memorable to me : The BUILD Foundation is now a reality, giving the Bazel ecosystem a vendor-neutral home. Its goal is to act as a proxy between the community and Google to help Google understand what to prioritize and to fund projects. If your company can spare “a few” dollars, please donate! AI-driven development is having a broad impact on the industry and, in particular, everyone is seeing their CI systems become overloaded. Well, news at 11: I’m dealing with this day in and day out. But what was new to me was that we at Snowflake are not the only ones struggling with local system overload. Lots of people out there have likewise “discovered” Git worktrees as a way to run multiple agents in parallel, which in turn means multiple concurrent Bazel servers. Bazel 10 is shaping up to be a substantial release that will help with the problem right above due to its many performance improvements and memory consumption optimizations. Have you struggled to keep Bazel up-to-date before? These days, if you want to get any platform-related work funded you have to spin it in terms of AI, so this release will be an easy sell. Let’s “Rewrite it in Rust”? I’m impressed. Google is seriously evaluating a progressive rewrite of Bazel in Rust, and this is probably the most exciting news I got out of the conference. You know I’ve been critical of Bazel’s usage of Java since day one so, if this takes off, I’ll be the first in line to evaluate the results (and it will likely push me to actually contribute to Bazel again). Fast and correct sandboxing on macOS remains a problem. This is “interesting” to me because it’s a problem I tried to fix almost ten years ago with the original sandboxfs, but I failed because of unexpected bugs in macOS’s FUSE support and eventually abandoned the idea when FUSE became a dead end on Apple’s platforms. Funnily enough, other folks have now reinvented sandboxfs—and even given it the same name (ugh)—and there is even another implementation of a similar idea in the form of actiondFS. Types are coming to Starlark, but Bazel’s late arrival to the party means that multiple competing approaches now exist. We have at least two LSPs with different annotation mechanisms and Buck 2’s own syntax. Hopefully they can all converge. Platforms and configurations remain difficult to model at scale. Careless use of transitions and build settings can produce an explosion of configured actions. Apparently, as one speaker noted, AI agents love transitions for some reason, which is interesting because they are not a popular feature and they are not particularly easy to use. Agents run more builds more frequently than humans, discover and invoke targets that were often “internal only”, and explore several approaches in parallel. Your intuition about how to evaluate which builds matter and how to measure them is probably wrong by now. Bazel and IDEs have never played great together and maybe that just doesn’t matter anymore. We are seeing a sharp decline in IDE usage, although developers still want a way to browse code changes and debug them. This means LSPs are key now to support “leaner” IDEs, and maybe standalone debuggers. Short build and test latency is critical when a human must remain focused on a coding task but, in an agentic workflow, the constraints are different. You know that, after you hit “execute this plan”, the agent will take many minutes to return so minimal build latency is not as important. And actually… maybe you now want to trade some latency in exchange for extra validations so that agents have to take fewer turns to converge. And because humans are now waiting for agents to finish for long periods of time, they have to fill that time with other tasks, which has driven them to adopt Git worktrees en masse. This is leading to scalability problems on dev machines because multiple concurrent Bazel server processes are not great for memory and disk usage. to download filtered logs for a build. to answer questions such as “Why was my build slow?” which are extremely common in the Bazel ecosystem. to start a remote Firecracker VM and run Bazel inside of it. The VMs are snapshotted for quick reuse, and the runners can execute arbitrary commands rather than Bazel alone. to diagnose a failed build and either create a pull request or push a commit with the proposed correction. to perform a review and post comments to GitHub. Correctness: Good, because equivalence is the central constraint of the loop. Hermeticity: Good, but you might need to do extra upfront work to remove host-specific behavior from the reference build before starting. Efficiency: Usually good when standard rules can model the build. Complex projects with custom build rules may end up with overly large Bazel actions though. Maintainability: Unsurprisingly, the weakest point. Agents tend to model variation with configuration flags instead of proper platforms and toolchains, which is not idiomatic to Bazel. Expert review of AI outputs remains critical, which is, again, not surprising. Reset transitions: These discard irrelevant configuration values as early as possible in the graph. The art is to move decisions up the dependency tree so that downstream targets do not carry unnecessary dimensions. The trade-off is that ordinary output can become unmanageable, forcing users toward . But then… requires a configuration, so some targets may not be queryable at all without one. Path mapping: If two actions cannot be made identical during analysis, Bazel can rewrite configuration-specific paths during execution and replace their hashes with a stable component. This should be seen as a measure of last resort, though, because subtle problems arise. For example, if the same logical path comes from different files from different configurations, Bazel disables path mapping silently. Tweag fixed Bazel so that different copies of the same file don’t disable path mapping, but debuggability continues to be a problem when this condition does not apply. Flag scopes: This new feature allows a flag to declare where in the graph it applies and to automatically reset to its default everywhere else. This is similar to a reset transition but requires less manual graph plumbing. On the other hand, describing the correct scope can sometimes be harder than writing the transition directly. Pre-warming the next JVM instead of keeping the current one alive. This provided more control over when and why to restart a test in a new JVM, but suffered from significant coordination overhead. Forking a JVM. This did not work because, due to POSIX semantics, only the calling thread survives a fork call. The JVM is heavily multithreaded, so if you end up losing all but one thread, the new JVM is just not going to work. Android’s Zygote. This provides a pre-warmed runtime ready to be forked ( , anyone?), but it exists only for ART, not the standard JVM, and it’s just not reasonable to expect a switch to ART. Checkpointing. Instead of forking a running process, why not snapshot it to disk so that it can be re-created again? This might have worked, like Horapha does for Bazel startup, but raised difficult questions about when to capture the process and how to handle state outside the JVM. Unconference discussion topic proposals and votes for the second day. Remote execution BoF Chi Wang and Tiago Quelhas led the Remote Execution Birds of a Feather . I only stayed for half of it, and so I only captured the discussion around remote persistent workers. Persistent workers are already part of several commercial remote execution offerings, including BuildBuddy and EngFlow, but their implementation was derived from Bazel’s local persistent worker support, which is not… particularly well designed. For example, we have at least three different variants of the persistent worker protocol: one for “regular” persistent workers, another for multiplex persistent workers, and yet another for multiplex persistent workers that support cancellation. And the protocol does not clearly specify which flags and tools belong to the worker and which belong to the action being executed. In fact, these distinctions are currently determined heuristically, with the worker assuming that flag files correspond to the action and other flags to the worker. There was consensus that this should be cleaned up, but the unsatisfying conclusion is that rehashing these complaints and discussions is not going to lead anywhere until someone sits down and presents a specific proposal for a new API. So, if you are reading this and have… opinions… volunteer to do this. Where Bazel clusters break at scale Helen Altshuler from EngFlow and Nikita Chepanov from Plaid presented Where Bazel Clusters Break at Scale: Field Notes from Plaid and the EngFlow Fleet . That’s quite a mouthful to say “load testing”. And the reason is none other than AI once again: Plaid’s build volume doubled during the year due to AI-assisted coding and is on track to reach four times its early-2026 baseline. With a predictable increase in load comes a predictable outage. So, instead of waiting for it to happen organically, Plaid deliberately chose to load-test their CI system. The goal was to reach multiples of the normal load and discover each bottleneck under controlled conditions. They had to go through six rounds of tests and fixes until they were able to sustain their desired target. The previous five uncovered problems in network bursts, AWS quotas, source control, CI autoscaling, and cache infrastructure. The solutions were not magical: find and raise limits, cache everything possible, and reduce demand where it does not add value. Examples of the last category include skipping tests for pull requests in the middle of a PR stack and avoiding uploads for actions when they are unlikely to be reused. An interesting observation is that Plaid runs EngFlow in Plaid’s own AWS account, so the cache cluster is co-managed. But careful: shared ownership can easily turn into ambiguous ownership. The two companies addressed this by agreeing on health metrics and thresholds upfront and putting them on a shared dashboard. Both sides now know who must act before developers notice an outage. Sandboxing and local execution Ah, one of my favorite topics. Back in my Google days, I spent at least three years of my so-called 20% time working on improving the performance of the Bazel sandbox on macOS. Those efforts were painful: creating a writable FUSE file system is difficult, and diagnosing mis-interactions with macOS was a horrifying exercise. Today, I’m not scared anymore: I’ve been recently using AI to write yet another file system at work and agents do exceedingly well at handling corner cases. Let’s see how modern solutions do better than what I originally tried. actiond: the quest for a better sandbox David Zbarsky from OpenAI presented actiond: The Quest for a Better Sandbox , which was the first of at least two talks on improving sandboxing performance and strictness. The project started from frustration with local sandbox escapes on macOS and eventually grew into a custom file system, a local virtual machine, and a Bazel-built Linux kernel. actiond pretends to be a remote executor, but the “remote” worker is a Linux VM running locally on the Mac. Bazel communicates with it over a and actiond implements both execution and a content-addressable store. Funnily enough, leveraging this VM for action execution results in faster Bazel builds on macOS than running actions natively. Inside the VM, actiondFS mounts a directory described by a protocol buffer and creates the action sandbox without building a forest of links. Same idea as my original sandboxfs , but the key difference is that actiondFS is a kernel module rather than a FUSE file system so it solves all performance and deployment problems I originally faced with FUSE. A mind-blowing mention was linux.bzl : a Bazel-native build of the Linux kernel, used in this project to build the actiond VM end-to-end with Bazel. This reminds me that I’ve been saying, since the inception of Bazel, that this build system is the perfect pairing for a monorepo like NetBSD’s or FreeBSD’s; and, in fact, I recently wanted a Bazel-native build of NetBSD to put the EndBOX together. I wonder how difficult it would be to hack something up these days with AI… Oh, and: . This flag finally replaces the super-complex script that Bazel uses to run a test with a native binary. This resolves some compatibility issues across platforms and should result in measurably faster test execution. The sandbox is a filesystem Borja Lorente from Byte Bard Software presented The Sandbox Is a Filesystem: Content-Addressed Bazel Builds on macOS with FSKit , yet another talk on sandboxing and macOS. The immediate problem they tried to address was sandbox escapes in TypeScript builds. You might not be aware, but there happen to be some tools out there that, when faced with a symlink, decide to resolve the target of the symlink and then perform extra operations on that location, like finding sibling files and the like. The compiler is one such tool, and consequently, escapes the symlink-based sandbox used on macOS and behaves incorrectly. Aspect originally patched the JavaScript rules to intercept Node.js file-system calls and redirect them, but that solution stopped working when was rewritten in Go. Apple’s new FSKit framework should in theory be usable to solve this, as it’s “just” a new way of writing file systems in user space. If we could leverage FSKit, the idea then becomes simple: represent the sandbox as a file system and mount a content-addressed view of the action’s inputs within it. Identical subtrees across actions share nodes, contents are fetched lazily, and outputs write through to the host volume. You know, exactly like my own sandboxfs did back in the day with FUSE. Unfortunately, FSKit comes with its own set of problems: An FSKit extension ships as an application, which must be packaged and notarized. This might be OK, but what’s bad is that, because this comes from an application, macOS displays a TCC (Transparency, Consent, and Control) popup notification when an app such as the Terminal tries to access the FSKit volume. Not cool for automated (CI) deployments. Mounting takes hundreds of milliseconds, making a separate mount per action too expensive. This can be solved by reusing the same mount across actions and making the file system create different subtrees per sandbox. Again, like the original sandboxfs. User space file systems have… unavoidable overheads as most operations have to cross the kernel/user space boundary at least twice. These may or may not be acceptable. With my own FUSE-based solution years ago, I just couldn’t fix some performance issues due to some oddities in OSXFUSE. AMFI and Gatekeeper inspect each executable once , adding substantial latency on first execution. And because any executable exposed through a separate file system looks brand new to the kernel… performance tanks. My original sandboxfs mitigated this by keeping inode numbers stable across different instances of the same file across different sandboxes, but that’s what ultimately led to the blockers I encountered with OSXFUSE. A “gather” aspect walks the target graph of the artifact that needs to be shipped. Each dependency contributes metadata through a transitive provider, effectively producing a serialized view of the relevant graph. Tools convert that data into standard SBOM formats such as SPDX and CycloneDX. Validation actions enforce organization-specific policy, such as banning a particular license from entering a product.

0 views
Aran Wilkinson 2 months ago

Software factories

When I first came across the term software factory in discussions about coding agents, I assumed it meant writing a task and handing it to an agent. The agent would gather the context, write the code, run the tests, open a pull request, and perhaps even deploy it. It sounded like a new, agentic way of building software. Then I saw Dex Horthy’s diagram , and it changed how I understood the term. I realised I had been a part of software factories for years. The task tracker, repository, pull request, CI system, deployment process, monitoring, incident response, and users were already connected. A factory was not a new thing that used agents. It was the system I was already using to turn a problem into a change in a running product. An agent changes one part of that system. It may take on more of the implementation, testing, investigation, or operational work, but it does not create the queues, hand-offs, or feedback loops around that work. Those were already there. I just had not been calling them a software factory. I have also been thinking about this in practical terms. What would it take to build a small software factory for my own projects, with agents helping me prototype ideas, build them, maintain them, and investigate problems once they are running? Looking at the whole system seems more useful than starting with a particular agent or tool. Reading Donella Meadows’s Thinking in Systems helped me make sense of this. One lesson I took from it is that a system’s behaviour comes from how its parts interact, not from the parts themselves, and that pushing harder on one part rarely fixes the system. CI is just one machine in the factory. The factory is the whole system: people and components connected by flows of work and information, operating towards some purpose. Here is the diagram that changed my view of it. It shows the software factory around implementation: users, product managers, engineers, pull requests, tests, deployments, monitoring, incidents, and feature requests. Dex Horthy, “Why Software Factories Fail” . What I find useful about the diagram is that it does not stop at deployment. It shows how information from users and production feeds back into what gets built next. Before seeing Dex’s diagram, I would probably have described software delivery as a series of separate pipelines. A product decision became a task, the task became code, the code passed its checks, and the change was deployed. That work would lead to the next task or another change, but I thought of those as separate pieces of work. The diagram gave me a different way to look at something I already understood. It made the connection explicit. What happens after deployment feeds back into the next decision. I had always been part of that process. I just had not thought of it as a loop, or seen all of its parts as one software factory. The first loop is the delivery loop: a decision becomes a change, the change is built, reviewed, tested, and deployed. The second starts with the product rather than the code. Users, monitoring, incidents, and feature requests produce information that feeds back into the next decision. I sketched the two connected loops together as a single flow below. Production is not the end of the process. It is one of the places where we find out what the change actually did. These loops meet when somebody decides what to build next. The delivery loop asks, “Can we change the software safely?” The product loop asks, “Is this the change we should make?” Those are separate questions. A team can get very good at shipping the wrong thing. It can also understand the product problem clearly and still be unable to deliver a safe change without a long queue of manual work. Seeing the work as two loops changed what I thought the factory was producing. The obvious output is a deployable artefact: a change to the software that can be released to users. But the factory also produces information. That information comes from different parts of the factory. Tests tell us about behaviours we thought to check. Static analysis finds certain classes of mistake. A review can expose a confusing design or a missing operational detail. Monitoring tells us what happened after deployment. Users tell us whether the change was useful in the first place. None of these signals answers the whole question. A green build does not mean that users are happy. A successful deployment does not mean that the system is healthy. A passing unit test does not mean that the change belongs in the product. That was the part I had missed when I thought of a software factory as an agent processing a task. I was focused on the change it produced, rather than the information the wider system produced about that change. The factory is useful when those signals make it back to the people deciding what to do next. That was my original mistake when I thought software factories were mainly about agents. I was thinking about how an agent could produce a change, but not about how the wider system would tell it whether the change was useful, safe, or worth keeping. The question is not simply how much code an agent can write. It is what happens to the rest of the factory when an agent takes on part of the work. Dex’s agentic diagram helped me see that the change may be local. An agent can replace or expand the implementation step, but the rest of the factory still has to review, test, deploy, and observe the change. Dex Horthy, “Why Software Factories Fail” . That seems obvious when written down, but it is easy to forget when an agent produces a working pull request quickly. The pull request is only where the change enters the rest of the factory. It is not the end of the work. A diagram published by The Pragmatic Engineer illustrates a more elaborate version, based on interviews about OpenAI’s internal system: The OpenAI example pushes further. It suggests that agents can remain involved after the merge, gathering production context, watching for regressions, and helping with incidents. I find that interesting, but I do not see it as a reference architecture to copy. It raises a more practical question for me: which parts of that loop would actually help a small project, and which would only add another layer of machinery? This is reporting about one company’s internal setup, not a reference architecture for every team. Here is a generic version of that flow: In the generic flow, an agent gathers context from the code, documentation, and internal tools, writes a change, and keeps fixing issues until the build and the checks pass. Specialist review agents inspect the result, and a risk classification decides whether it follows an automated path or needs human review. In the system described by The Pragmatic Engineer, the loop continues after the merge. An agent helps roll the change out to production, watches the relevant signals, and can create monitoring dashboards for the change. A performance system looks for regressions and proposes fixes. An incident-response bot gathers context and suggests mitigations when something goes wrong. Dex’s response is to turn the lights back on , not to abandon automation. If agents make implementation much faster, review becomes the next bottleneck. Removing human review can increase throughput for a while, but it also removes one of the few places where somebody might notice that a change is making the system harder to understand or modify. In the second part of his series, he argues for moving human judgement earlier in the process: agree the product requirements, system architecture, and shape of the program before asking an agent to implement it, then work in small vertical slices that can be checked as they are built. That is a much more ambitious factory than “ask an agent to write some code”. It gives agents access to the context around the code, lets them act on feedback from the build and review systems, and keeps them involved after deployment. If an agent can reach the wider factory only through a human at a terminal, the human becomes the integration layer. They have to notice a failed CI job, collect the logs or metrics, decide what is relevant, and bring that context back to the agent. This makes each loop slower and means the agent sees only the part of the problem the human thought to provide. An agent connected to the factory can query those systems itself: inspecting CI results, reading production logs, comparing metrics, and gathering the context it needs before proposing what to do next. That connection does not require unrestricted access or permission to act autonomously. Access can be read-only and limited to the systems relevant to the task, with human approval still required for consequential actions. An agentic software factory does not need to be fully automated. A person might start an agent manually, mark a feature request as ready for an agent to pick up, or route a failed build, production regression, or incident directly to one. How an agent gets kicked off, manually or automatically, is a design choice, not part of the definition. What matters is that the agent takes part in the factory’s flows of work and information: it can gather the relevant context, act within its permissions, and respond to the results of its work rather than merely suggest the next line of code. For businesses already experimenting with coding agents, I think the next step is to optimise the software factory around them. It is not simply a matter of giving every developer an agent. Businesses need to look at their existing software factory and decide how it should work when agents become participants: which queues they can take work from, what context and tools they can access, how their output is checked, and where people need to approve an action. Most software delivery systems have been built around human teams, so adding agents means reconsidering those interfaces and hand-offs rather than dropping a coding tool into an otherwise unchanged process. Agency does not mean removing people from the system either. A human still needs to be in the loop at the points where judgement or accountability matters. Depending on the risk, that might mean deciding when to start the agent, reviewing its proposed change, approving an action it wants to take, or approving its deployment. The boundary will differ between teams and between changes; the important part is that it is deliberate rather than whatever the tooling happens to allow. Deliberate boundaries also make the human responsibilities easier to see. People still need to define useful outcomes, make the relevant context available, decide what counts as low risk, ensure the tests describe acceptable behaviour, choose the signals that matter, and decide where an agent must stop and ask for help. Automation changes the boundary of the work; it does not remove judgement from the system. A change marked as low risk can still be wrong, so the classification and the automated path need their own checks. The most important detail in the diagram is the feedback path from production back to the coding agent. An agent that can write code but cannot observe its consequences is just a faster way to create a queue of changes. An agent that can see failures, understand their context, and propose a fix starts to look more like a participant in the factory. The agent does not need to remain active or control every stage. The factory could route the results of a failed build, deployment, or production alert to an agent with the relevant context. What matters is whether that path exists. Without it, the wider feedback loop is available to the human but closed to the agent. Even when those feedback paths exist, there is another limit: an agent can only respond to signals the factory can produce. This is part of Dex Horthy’s wider argument. Tests and production metrics can quickly reveal certain failures, but maintainability has a much slower feedback loop. A design that makes the next six changes harder may not reveal itself until months later. This is not unique to agents, but faster implementation allows more of those design decisions to accumulate before their consequences become visible. A factory can have plenty of automation and still be slow. If every change waits several days for a review, producing the initial change faster will not fix the queue. If deployments are frequent but nobody looks at production behaviour, the factory is producing changes without learning from them. If feature requests arrive faster than the team can understand and prioritise them, adding more implementation capacity will not fix that decision queue. The slowest part of the system is often not the compiler or the test suite. It is a hand-off: This is why the arrows in the diagram are as important as the boxes. A box describes an activity. An arrow describes how work and information move between activities. There is a risk in taking the factory analogy too literally. Software is not a pile of identical widgets. The work includes discovery, judgement, and learning. The thing being built often changes shape while we are building it. Optimising only for throughput can make a team worse at its actual job. We might measure the number of pull requests merged, deployments made, or tickets closed because those numbers are easy to count. That does not tell us whether the product is becoming more useful or whether the system is becoming easier to change. The useful part of the analogy is not “make engineers work like machines”. It is “look at the complete system”. Where does work enter? Where does it wait? Which checks happen automatically? Which decisions need a person? How quickly does information from production get back to the team? The factory should make good changes easier to make and bad changes easier to notice. It should not turn every decision into a race for a higher throughput number. I have been thinking about what a small software factory might look like for my own projects. I want to experiment with agents not only as a way to implement tasks, but as part of a system that helps me with the same four jobs I mentioned at the start: prototyping ideas, building them, keeping them maintained, and investigating failures. Headcode is an obvious place to start. In Splitting Headcode into a polyrepo workspace , I described separating it into smaller repositories with clearer boundaries, repository-local checks, and a separate place for agent work plans. I made those changes partly to give agents bounded contexts. That workspace is not yet a software factory, but it gives me a practical system in which to explore how work moves between agents, repositories, checks, deployments, and production feedback. I do not mean a lights-off factory that can be left to ship changes on its own. The interesting part is deciding which loops I can automate safely, where I still need to make decisions, and what context and signals an agent would need to do useful work. To start with, I will keep most of the agent use manual: running an agent when I choose to, rather than wiring it to a trigger. Starting that way keeps costs predictable while I work out which loops are worth automating, and it keeps me close to what the agents are actually doing before I let them run on their own. That raises more useful questions than simply choosing an agent: Adding another tool is not the same as improving the factory. A new tool only helps if it removes a real delay, improves a useful signal, or makes a risky decision safer. I have not built that system yet. In follow-up posts, I want to explore what an agentic software factory looks like in practice, starting with personal projects where I can experiment freely and document what I learn. I am also interested in seeing how these ideas could apply to the systems I work with professionally, where I have permission to experiment. Thinking of it as a factory gives me a better place to start: not “which agent should I use?”, but “which loop am I trying to improve?” Did the change solve the problem? Did it create a new failure mode? Did users understand it? Did it make the system harder to operate? What should we change next? an engineer waits for somebody who understands the product decision; a pull request waits for a reviewer who is also busy building something else; a deployment waits for somebody to perform a manual check; an incident waits for information that is spread across several systems; a user report waits for somebody to turn it into a reproducible problem. Where does work enter the factory: a Linear ticket, a failed build, a production alert, or a note to myself? How do I describe the outcome rather than only the implementation task? What code, documentation, and operational context does the agent need? Which checks provide meaningful feedback rather than merely producing a green build? Which changes are safe to automate, and where should the agent stop? How does information from maintenance, failures, and actual use make it back into the next change?

0 views
Farid Zakaria 2 months ago

Review a pull request by booting it

tl;dr; trynix-preview is a GitHub action that comments a link on a pull request which lets you boot the PR’s build in the browser using https://trynix.dev . No servers, just browsers. I ended my earlier trynix post with a list of ideas I think we could accomplish now that we can boot arbitrary paths in the browser. The most obvious one was to let a reviewer boot a pull request’s build in the browser for testing, validation and feedback. That is now real. 🤯 A demo is worth a 1000 words: here is a pull request ( PR#31 ) against my sqlelf project, from a fork , with the comment our action left on it: Click the link and a Linux machine boots in your tab with that PR’s on . You did not clone anything, you did not build anything. No servers, no SSH, no VPN, no Docker, no VM, no cloud. Just a browser and a link. 😈 As with any GitHub action, it’s just a few lines to add to your workflow. The caveat is that you must have built and cached the path already, so the action can link to it. The action does not build or cache anything. The action publishes and builds nothing. Whatever already fills your cache keeps doing it, and the action’s whole job is to simply provide the store paths via and hand the cache’s URL and public key to the browser. It is not Nix cache provider specific, but I do recommend Cachix because it is free for open source up to 5GiB. 1 You can checkout my trynix.yaml workflow for the full example. You have to set in the step because the workflow runs on a fork’s pull request, and that has security implications. 2 If that is not your cup of tea, there is a version where a maintainer types on the pull request which kicks off the workflow. In either case, the workflow runs on the default branch and checks out the pull request’s code, so a fork cannot edit the workflow that builds it. Did I just upend all CI products by easily letting reviewers boot a PR? Unfortunately, no. 🥲 The performance for large binaries is pretty bad. Even with many of the improvements I AI-assisted into the engine, large binaries can still take 1-2 minutes to execute. 3 Nevertheless, this is still a pretty amazing workflow and showcases the power of Nix. Maybe as we get closer to AGI, our AI overlords will be able to optimize the engine to execute large binaries in a few seconds, but for now, the action is best suited for small to medium-sized binaries. You should definitely sign up for Cachix but you can test this out without it since the free tier is very generous.  ↩ I recommend a private segregated cache for pull request builds, so that a fork cannot push to your main cache.  ↩ I added a benchmark page, https://trynix.dev/bench/ , to the site with a lot of rich data on boot and run times for various applications.  ↩ You should definitely sign up for Cachix but you can test this out without it since the free tier is very generous.  ↩ I recommend a private segregated cache for pull request builds, so that a fork cannot push to your main cache.  ↩ I added a benchmark page, https://trynix.dev/bench/ , to the site with a lot of rich data on boot and run times for various applications.  ↩

0 views
Farid Zakaria 2 months ago

Stamping build info in constant memory

This is a fun little trick I came across at . I did not invent it, but I thought it was interesting enough to understand better and share. At we build with buck2 and we stamp our executables with build information: build-id, timestamp, author, the usual suspects using as a step after the link. The reason it is a separate step is caching. If the build info was generated at link time then everytime we link the binary it would produce different bytes causing it to not be bit-reproducible. When something is bit-reproducible, it is safe to cache it, and the build system can apply early cut-off optimizations. That works, until the binaries get big. We noticed that ’s memory use scales with the size of the file it is stamping. Stamping is exactly the kind of step that runs massively parallel at the end of a build so this can cause a lot of memory pressure. Why is the stamping step reading the binary at all? 🤔 Let’s measure the claim that the memory use of scales with the size of the file. We will attach a JSON build info blob to an increasingly large synthetic executable and measure the peak RSS of the stamping step. The graph confirms the claim. The memory use of scales linearly with the size of the file being stamped. Surprisingly, the slope is two . The peak RSS is roughly twice the size of the file being stamped irrespective of the size of the build info being attached. 1 I am helping to shepherd a PR open against LLVM to stream the ELF output rather than materialize it, which roughly halves the peak. That is a definite improvement but the problem remains that in order to add a tiny section to a large binary, the whole binary has to be read into memory. The memory use is still linear in the size of the file. The problem is not poor implementation on the part of . Adding a section to an ELF touches three separate things: 0x78 bytes in .interp\0 .buildinfo\0 .rela… 0x70 0x78 0x83 the section's bytes 88 bytes of JSON, at 0x401021 The entry holds neither the name nor the bytes. It only refers to them — and only one of those two references can be repointed in place. In order to account for the new section, the section header table has to grow by one entry, and has to grow by the length of the new name. The current model for is to read the whole file into memory, add the new section, and write the whole file back out. That is why the memory use scales with the size of the file. How can we avoid having to rebuild the whole file just to add a tiny section? The trick is to pay the cost at link time rather than at stamp time. We can have the linker emit a placeholder section with the right name and a single byte of content. The section header table entry is already there, and the name is already in . The post-link stamping step can then append the payload to the end of the file and update the section header entry to point to it. 💡 We make linker emit the section during the normal build. It does not need to hold anything; it just needs to exist so that it owns a name and a header. Our “stamp” step is now incredibly simple. It does not need to read the file at all, it just needs to write the new payload and update the section header entry. Nothing that already exists moves. does not move, the section header table does not move, no other changes. The edit is sixteen bytes , at a file offset you can compute from the ELF header, plus a . read into a model, serialized again — 1,238 bytes of it actually differ reserve one byte, then append 16 bytes + a tail ehdr phdrs .text .rodata … .shstrtab section headers payload the reserved byte sh_offset, sh_size Note Why 1 byte? Turns out that and GNU disagree on whether an empty section is a valid ELF. The one byte is a cheap way to make both linkers happy. The payload lands after the section header table, which looks alarming the first time you see it but is completely legal. Nothing in ELF says section contents must precede the section header table. The kernel also never looks at section headers also, it loads segments out of the program headers, which we do not touch. I wrote a small C version to benchmark it in contrast, please be mindful that this graph is log-log. Our trick works! The “append + 16 bytes” approach is constant memory. Not only is the peak RSS constant, but the wall time is also constant and much faster by avoiding the read and write of the whole file. It is often easy to reach for general-purpose tools like as they are a swiss-army knife for manipulating object files. What I like about this trick though is that there are meaningful improvements to be made by writing special purpose tools and that does not mean we have to accrue large maintenance costs. In this case it was a tiny 200-line C program. The economics of these tools is also changing with the rise of LLMs in our workflow. While many are concerned about the influx of generated code, I remain optimistic that we we can use them to find such opportunities. Don’t be afraid to write a small tool to solve a specific problem. For those thinking this is an LLVM specific issue, GNU exhibits the same behavior.  ↩ the section’s bytes , somewhere in the file a 64-byte entry in the section header table describing where those bytes are the section’s name , which is not in the entry itself but rather the entry holds a offset into , so the name has to be appended to that string table 0x70 0x78 0x83 the section's bytes 88 bytes of JSON, at 0x401021 The entry holds neither the name nor the bytes. It only refers to them — and only one of those two references can be repointed in place. In order to account for the new section, the section header table has to grow by one entry, and has to grow by the length of the new name. The current model for is to read the whole file into memory, add the new section, and write the whole file back out. That is why the memory use scales with the size of the file. Pay the byte at link time How can we avoid having to rebuild the whole file just to add a tiny section? The trick is to pay the cost at link time rather than at stamp time. We can have the linker emit a placeholder section with the right name and a single byte of content. The section header table entry is already there, and the name is already in . The post-link stamping step can then append the payload to the end of the file and update the section header entry to point to it. 💡 We make linker emit the section during the normal build. It does not need to hold anything; it just needs to exist so that it owns a name and a header. Our “stamp” step is now incredibly simple. It does not need to read the file at all, it just needs to write the new payload and update the section header entry. append the payload to the end of the file write the new and into the placeholder’s section header entry the reserved byte sh_offset, sh_size Note Why 1 byte? Turns out that and GNU disagree on whether an empty section is a valid ELF. The one byte is a cheap way to make both linkers happy. The payload lands after the section header table, which looks alarming the first time you see it but is completely legal. Nothing in ELF says section contents must precede the section header table. The kernel also never looks at section headers also, it loads segments out of the program headers, which we do not touch. Benchmark I wrote a small C version to benchmark it in contrast, please be mindful that this graph is log-log. elfstamp.c Our trick works! The “append + 16 bytes” approach is constant memory. Not only is the peak RSS constant, but the wall time is also constant and much faster by avoiding the read and write of the whole file. One trick pony It is often easy to reach for general-purpose tools like as they are a swiss-army knife for manipulating object files. What I like about this trick though is that there are meaningful improvements to be made by writing special purpose tools and that does not mean we have to accrue large maintenance costs. In this case it was a tiny 200-line C program. The economics of these tools is also changing with the rise of LLMs in our workflow. While many are concerned about the influx of generated code, I remain optimistic that we we can use them to find such opportunities. Don’t be afraid to write a small tool to solve a specific problem. For those thinking this is an LLVM specific issue, GNU exhibits the same behavior.  ↩

0 views
ENOSUCHBLOG 3 months ago

GitHub Actions needs OIDC audience constraints

TL;DR : GitHub Actions should allow end-users to express audience constraints , to make it harder for an attacker to pivot across services that use independent OIDC-bearing jobs. They could do this with relatively small syntax tweak, although the backend implications are probably nontrivial. Like many CI/CD providers, GitHub Actions provides verifiable machine identities 1 via OpenID Connect (OIDC). These are awesome for a lot of reasons, not least of which is that they allow workflows running on GitHub Actions to federate with other (third-party) services without GitHub having to intermediate and pre-bless every interaction. This is the backbone of how both Trusted Publishing and Sigstore work: an individual workflows on GitHub Actions presents its machine identity (via an OIDC token) to an external service, which then authenticates and for some purpose (uploading to PyPI or signing artifacts, respectively). Unfortunately, GitHub’s mechanism for exposing OIDC tokens in workflows contains a significant weakness, one that (in my opinion) will present an increasingly serious security risk over time. This post is about that weakness. At the core of all “OIDC in CI/CD” implementations is some mechanism that allows the workflow (pipeline definition, etc.) to request or otherwise be pre-loaded with an OIDC identity. Here’s how that looks in GitLab CI/CD: and here’s the equivalent in GitHub Actions: The difference between these two is small but important: GitLab requires the OIDC audience (the ) to be declared up-front and statically , while GitHub requires the workflow to dynamically request a token with an audience selected at runtime (the parameter in the HTTP request). First, a very quick foray into OIDC. Under the hood, OIDC is (mostly) just OAuth 2.0 , and OIDC identity tokens are just JSON Web Tokens (JWTs), with some additional 2 constrains on the claims they should express. The most important claim in an OIDC ID token is arguably , since it identifies the principal (the “subject”) 3 . However, the second most important claim is , for the audience . The audience is critical because it constrains who honors the token : services that accept ID tokens should only do so when they recognize the audience as matching theirs. In other words: the claim prevents an ID token that’s intentionally been issued for a specific service from being stolen by the attacker and mis-applied to another service. This is intended as a defense-in-depth: even if an attacker manages to exfiltrate an OIDC credential, they should not be able to pivot to other services it. It’s a flimsy defense but one that’s generally effective 4 , unless you give the attacker the ability to control the claim as well. Unfortunately, that’s exactly what GitHub Actions enables 5 : gives the job (or entire workflow) the ability to mint any ID token it pleases, with any audience. This matters a great deal in a world (our world) where jobs that are given also run a lot of third-party code: any vulnerability (or malware) in that code has the potential to ask for new ID tokens for audiences that it isn’t supposed to have access to. We thought about this problem when designing Trusted Publishing, and came to the conclusion that the machine identity that Trusted Publishing uses must include the workflow name, preventing an attacker from impersonating by inducing an ID token from with . However, this constraint is not suitable for all possible use cases: many integrations want to use just the slug as a sufficient identity, meaning that all workflows are effectively co-equal when issuing ID tokens. Add constraints! Ideally, something like this: The basic idea here is to constrain what audiences the job can request ID tokens for. In the example above, attempting to request a token for would cause an error, preventing a job that’s intended only for publishing to PyPI from serving as a pivot to AWS. There are, of course, some potential downsides to this. For example, some services might (inadvisedly) require dynamic information in their audience, meaning that a value can’t be statically pre-declared. I think this is rare enough in practice to not be worth blocking a security improvement and, when they do occur, GitHub could continue to allow the form as a (less secure!) catch-all. More precisely, “workload identities.” But the distinction is not relevant here.  ↩ Not very stringent.  ↩ In practice, the claim is often a mess, since it’s typically an opaque string that uses or similar as a contentional delimiter. This causes all kinds of security bugs (e.g. when attacker-controlled components of the subject can also contain the delimiter), which is why in practice Trusted Publishing and Sigstore both emphasize IdP-specific claims that represent each part of the subject’s identity rather than a bespoke composite form.  ↩ Services can (and do) have bugs, like neglecting to check the audience or allowing it to vary with other potentially attacker-controlled claims. But we’re assuming services that do properly check the audience here, like PyPI’s Trusted Publishing does.  ↩ Others appear to do the same thing, although I’m less familiar with other CI/CD providers. For example, BuildKite’s OIDC flow involves invoking , which involves the same level of runtime control of the audience.  ↩ More precisely, “workload identities.” But the distinction is not relevant here.  ↩ Not very stringent.  ↩ In practice, the claim is often a mess, since it’s typically an opaque string that uses or similar as a contentional delimiter. This causes all kinds of security bugs (e.g. when attacker-controlled components of the subject can also contain the delimiter), which is why in practice Trusted Publishing and Sigstore both emphasize IdP-specific claims that represent each part of the subject’s identity rather than a bespoke composite form.  ↩ Services can (and do) have bugs, like neglecting to check the audience or allowing it to vary with other potentially attacker-controlled claims. But we’re assuming services that do properly check the audience here, like PyPI’s Trusted Publishing does.  ↩ Others appear to do the same thing, although I’m less familiar with other CI/CD providers. For example, BuildKite’s OIDC flow involves invoking , which involves the same level of runtime control of the audience.  ↩

0 views
Justin Duke 5 months ago

Self-hosting our CI, three months later

A couple of folks have written in over the past month asking how my experiments in self-hosting GitHub Actions runners have held up. And three months seems like a reasonable amount of time to post a follow-up. I'll also treat this as a bit of a first draft for a longer engineering post on the Buttondown blog itself — so please write in if there's something you wish I'd covered here. The short version: the experiment graduated into the plan. That March post ended on a half-joking aside — "maybe we're just going to self-host the entire CI suite" — and, reader, we did. When I wrote that post back in March, I called out two things that were true at the time: Neither is quite true anymore, and — as you might expect — the way each stopped being true turns out to be somewhat commensal. Around a month or so ago, on a lark and the back of a Blacksmith incident, I decided to try running the entirety of CI on Pythia just to see what would happen. As is often the case, experiments are really just preludes to decisions; what seemed fairly promising — a 30% slowdown in performance on a quote-unquote hot server, but saving a hundred or so bucks a month in exchange — felt good enough to commit to. To put some numbers on it: that January Blacksmith bill was $300, and by the time I got around to this it had crept up toward $400 as the team merged more. The backend suite was the single fattest line item — around $100/month on its own. Moving it over was the last domino; our Blacksmith bill now rounds to zero. The 30% is less scary than it sounds in the body of an actual workday. The backend suite that used to go green in around six minutes on Blacksmith's 8-CPU runner now lands closer to eight. You notice it; you do not particularly care. The flip side is that it's very easy to unintentionally starve yourself. Just like in the initial post, we have five GitHub Actions runners running in parallel on a single box. This works well for our smaller jobs, which are IO-bound and don't really use multi-threading in any real capacity. But we do still have very big jobs, and poor Pythia suffers the consequences when we try to run three or so backend test jobs simultaneously. GitHub Actions helpfully provides a parameter, but it's a bit finicky to get working in exactly the right way, because it applies at an overall workflow level and not a job level. To enforce the concurrency I actually want, I'd have to isolate the backend tests into their own workflow. All of this is a very in-the-weeds way of saying: migrating the whole gestalt of our CI onto a dinky little box in my office has resulted in tiny amounts of pain that could probably go away if I spent an afternoon thinking really hard about the problem. And then there's the objection everyone reaches for first: you've put all of your CI behind a single point of failure, and that single point of failure is a box in your office. Which is true! But that single point of failure resides in my office, which — notably — is not the same thing as residing in my head. The total number of times I have had to bike over to the office and restart the server over the past 90 days has been exactly one.[^bike] [^bike]: It would be disingenuous not to also cop to the other time this happened — except that one doesn't count, because I was already at the office. But the fact that I've been able to save the team and myself almost $400 a month — with very, very little work, and no marginal cost — has been a bit of a revelation. I highly encourage you to spend an afternoon giving it a shot yourself. We were continuing to use Blacksmith for our heaviest suite of jobs — the backend test suite. We had touched essentially nothing since setting it up; it just ran.

0 views
blog.philz.dev 7 months ago

What is Buildkite?

If you're starting a new project, just skip the misery of GitHub actions and move on. Buildkite mostly gets it. The core Buildkite noun is a Pipeline, and, as traditional for an enterprise software company, their docs don't really tell you what's what. The point is that your pipeline should be: Pipelines can add steps to themselves . So, you can write a script to generate your pipeline (or just store it in your repo), and cat it into the command, and that's how the rest of your steps are discovered. Pipeline steps are each executed in their own clean checkout of what you're building. So, if you want to run the playwright tests in parallel with the backend tests (or whatever), you just declare that as two different steps, but they're part of the same thing. Pipelines have a dependency graph between steps that's conceptually similar to . (Perhaps was the Make replacement that I first heard of that did "generate the ninja graph"?) The agents seem pretty good at manipulating Buildkite once you give them an API key. They also seem to not in-line shell scripts into the yaml, which is Obviously Good. The ways to speed up the build is always the same: cache and parallelize. A 16-core machine for 1 minute costs the same as an 2-core machine for 8 minutes, and I know which one I'd rather wait for! Buildkite makes parallelism pretty easy. Anyway, it's pretty good. Thanks, Buildkite.

0 views
devansh 8 months ago

More egress filtering bypasses in harden-runner

Egress filtering in CI/CD is supposed to keep secrets in and attacker callbacks out. If your build step only needs to talk to , it shouldn't be able to reach anything else. StepSecurity's Harden-Runner does this for GitHub Actions — you give it a list of allowed endpoints, set , and everything else gets denied. The filtering is domain-based. It watches outbound connections, checks the destination, and blocks anything not on the list. That works fine as long as data leaves through a visible connection to a visible domain. DNS gives you two ways around that. The first is DNS over TCP. Harden-Runner doesn't adequately restrict DNS queries sent over TCP, so a single command bypasses the entire policy. The second is DNS over HTTPS (DoH), where a DNS query gets wrapped inside a normal HTTPS request to a public resolver like . The resolver acts as a proxy — Harden-Runner sees a legitimate HTTPS connection and lets it through. It never looks at what's inside. Both issues affect Harden-Runner v2.13.3 (community version) and likely all prior versions. Harden-Runner enforces egress policies on GitHub runners by filtering outbound connections at the network layer. When is enabled with a restrictive list, all non-compliant outbound traffic should be denied. A typical hardened workflow: After this step, any outbound connection to a domain or port not on the whitelist should be blocked. DNS queries usually go over UDP on port 53. But DNS also works over TCP on the same port — it's standard, well-documented, and tools like support it with a flag. It's commonly used for large responses and zone transfers, nothing exotic. Harden-Runner doesn't adequately restrict DNS queries sent over TCP. While it may handle standard UDP DNS and HTTPS connections, TCP-based DNS queries to an external resolver go through without being blocked or detected. That means an attacker can run: The query goes over TCP to Google's resolver, gets forwarded to 's nameserver, and the attacker reads the data out of the subdomain labels. Harden-Runner's dashboard shows zero outbound destinations and zero detections. is pre-installed on every Ubuntu runner. Harden-Runner shows 0 outbound destinations , 0 HTTPS events , and 0 detections . The Burp Collaborator server receives the DNS query. The entire egress block policy bypassed with a single command. Step Security Dashboard (our C2 domain is not getting flagged) Data exfiltration to our C2 DNS over HTTPS (DoH) wraps DNS queries inside standard HTTPS requests. From the network's perspective, a DoH request to looks like a normal HTTPS connection to a well-known endpoint. Harden-Runner sees it and lets it through. The idea is simple, use as a proxy. An attacker crafts a DNS query where the domain being resolved is , sends it as an HTTPS POST to , and Google's resolver forwards it to the attacker's authoritative nameserver. The data is encoded in the subdomain labels, the runner's hostname, environment variables, secrets, whatever fits. The egress filter sees an HTTPS connection to a legitimate destination. It never inspects what's inside. The attacker's domain never appears in any connection log. The workflow sets up Harden-Runner with a restrictive policy (only ), then runs a Python script that exfiltrates the runner's hostname via DoH through Google's resolver: The script builds a DNS wire-format query with the runner's hostname encoded as a subdomain, then sends it as a DoH request to : PoC Execution Step Security Dashboard (our C2 domain is not getting flagged) Data exfiltration to our C2 Harden-Runner shows 1 destination ( ), 0 HTTPS events , and 0 detections . The Collaborator server receives the DNS query with the runner's hostname. Data exfiltrated cleanly through a permitted HTTPS connection — acted as an unwitting proxy. Both vulnerabilities were reported to the StepSecurity team in early December 2025 via GitHub Security Advisories. The 90-day disclosure timeline was applied naturally, with the blog post date set for mid-March, 2026. StepSecurity acknowledged the reports over email, but my last follow-up message on the advisories asking for updates went unanswered and no CVE was issued. These were tested against the community version of Harden-Runner (v2.13.3). It's possible this was addressed in step-security/agent#469 , but I have not tested it. Since the 90-day disclosure window has passed and I did not hear back about a CVE, I am disclosing now. CVEs are pending. What is Harden-Runner's Egress Policy? Egress Block Policy Bypass via DNS over TCP Proof of Concept Egress Policy Bypass via DNS over HTTPS — Proxying DNS queries using Google's resolver Proof of Concept Disclosure Timeline Discovery & Report (DNS over TCP) : 4th December 2025 ( GHSA-g699-3x6g-wm3g ) Discovery & Report (DNS over HTTPS) : 5th December 2025 ( GHSA-46g3-37rh-v698 ) Vendor Contact : 4th–5th December 2025 Vendor Acknowledgment : Acknowledged over email Possible Fix : step-security/agent#469 — not tested Public Disclosure : 15th March 2026

0 views
matklad 9 months ago

CI In a Box

I wrote , a thin wrapper around ssh for running commands on remote machines. I want a box-shaped interface for CI: That is, the controlling CI machine runs a user-supplied script, whose status code will be the ultimate result of a CI run. The script doesn’t run the project’s tests directly. Instead, it shells out to a proxy binary that forwards the command to a runner box with whichever OS, CPU, and other environment required. The hard problems are in the part: CI discourse amuses me — everyone complains about bad YAML, and it is bad, but most of the YAML (and associated reproducibility and debugging problems) is avoidable. Pick an appropriate position on a dial that includes What you can’t just do by writing a smidgen of text is getting the heterogeneous fleet of runners. And you need heterogeneous fleet of runners if some of the software you are building is cross-platform. If you go that way, be mindful that The SSH wire protocol only takes a single string as the command, with the expectation that it should be passed to a shell by the remote end. In other words, while SSH supports syntax like , it just blindly intersperses all arguments with a space. Amusing to think that our entire cloud infrastructure is built on top of shell injection ! This, and the need to ensure no processes are left behind unintentionally after executing a remote command, means that you can’t “just” use SSH here if you are building something solid. One of them is not UNIX. One of them has licensing&hardware constraints that make per-minute billed VMs tricky (but not impossible, as GitHub Actions does that). All of them are moving targets, and require someone to do the OS upgrade work, which might involve pointing and clicking . writing a bash script, writing a script in the language you already use , using a small build system , using a medium-sized one like or , or using a large one like or .

0 views
Martin Alderson 1 years ago

How I make CI/CD (much) faster and cheaper

Why GitHub Actions runners are slow and how bare metal servers can make your CI/CD 2-10x faster while costing 10x less

0 views
ENOSUCHBLOG 1 years ago

Dear GitHub: no YAML anchors, please

TL;DR : for a very long time, GitHub Actions lacked support for YAML anchors. This was a good thing . YAML anchors in GitHub Actions are (1) redundant with existing functionality, (2) introduce a complication to the data model that makes CI/CD human and machine comprehension harder, and (3) are not even uniquely useful because GitHub has chosen not to support the one feature (merge keys) that lacks a semantic equivalent in GitHub Actions. For these reasons, YAML anchors are a step backwards that reinforces GitHub Actions’ status as an insecure by default CI/CD platform. GitHub should immediately remove support for YAML anchors, before adoption becomes widespread. GitHub recently announced that YAML anchors are now supported in GitHub Actions. That means that users can write things like this: On face value, this seems like a reasonable feature: the job and step abstractions in GitHub Actions lend themselves to duplication, and YAML anchors are one way to reduce that duplication. Unfortunately, YAML anchors are a terrible tool for this job. Furthermore (as we’ll see) GitHub’s implementation of YAML anchors is incomplete , precluding the actual small subset of use cases where YAML anchors are uniquely useful (but still not a good idea). We’ll see why below. Pictured: the author’s understanding of the GitHub Actions product roadmap. The simplest reason why YAML anchors are a bad idea is because they’re redundant with other more explicit mechanisms for reducing duplication in GitHub Actions. GitHub’s own example above could be rewritten without YAML anchors as: This version is significantly clearer, but has slightly different semantics: all jobs inherit the workflow-level . But this, in my opinion, is a good thing : the need to template environment variables across a subset of jobs suggests an architectural error in the workflow design. In other words: if you find yourself wanting to use YAML anchors to share “global” configuration between jobs or steps, you probably actually want separate workflows, or at least separate jobs with job-level blocks. In summary: YAML anchors further muddy the abstractions of workflows, jobs, and steps, by introducing a cross-cutting form of global state that doesn’t play by the rules of the rest of the system. This, to me, suggests that the current Actions team lacks a strong set of opinions about how GitHub Actions should be used, leading to a “kitchen sink” approach that serves all users equally poorly. As noted above: YAML anchors introduce a new form of non-locality into GitHub Actions. Furthermore, this form of non-locality is fully general : any YAML node can be anchored and referenced. This is a bad idea for humans and machines alike: For humans: a new form of non-locality makes it harder to preserve local understanding of what a workflow, job, or step does: a unit of work may now depend on any other unit of work in the same file, including one hundreds or thousands of lines away. This makes it harder to reason about the behavior of one’s GitHub Actions without context switching. It would only be fair to note that GitHub Actions already has some forms of non-locality: global contexts, scoping rules for blocks, dependencies, step and job outputs, and so on. These can be difficult to debug! But what sets them apart is their lack of generality : each has precise semantics and scoping rules, meaning that a user who understands those rules can comprehend what a unit of work does without referencing the source of an environment variable, output, &c. For machines: non-locality makes it significantly harder to write tools that analyze (or transform) GitHub Actions workflows. The pain here boils down to the fact that YAML anchors diverge from the one-to-one object model 1 that GitHub Actions otherwise maps onto. With anchors, that mapping becomes one-to-many: the same element may appear once in the source, but multiple times in the loaded object representation. In effect, this breaks a critical assumption that many tools make about YAML in GitHub Actions: that an entity in the deserialized object can be mapped back to a single concrete location in the source YAML. This is needed to present reasonable source locations in error messages, but it doesn’t hold if the object model doesn’t represent anchors and references explicitly. Furthermore, this is the reality for every YAML parser in wide use: all widespread YAML parsers choose (reasonably) to copy anchored values into each location where they’re referenced, meaning that the analyzing tool cannot “see” the original element for source location purposes. I feel these pains directly: I maintain zizmor as a static analysis tool for GitHub Actions, and makes both of these assumptions. Moreover, ’s dependencies make these assumptions: (like most other YAML parsers) chooses to deserialize YAML anchors by copying the anchored value into each location where it’s referenced 2 . One of the few things that make YAML anchors uniquely useful is merge keys : a merge key allows a user to compose multiple referenced mappings together into a single mapping. An example from the YAML spec, which I think tidily demonstrates both their use case and how incredibly confusing merge keys are: I personally find this syntax incredibly hard to read, but at least it has a unique use case that could be useful in GitHub Actions: composing multiple sets of environment variables together with clear precedence rules is manifestly useful. Except: GitHub Actions doesn’t support merge keys ! They appear to be using their own internal YAML parser that already had some degree of support for anchors and references, but not for merge keys. To me, this takes the situation from a set of bad technical decisions (and lack of strong opinions around how GitHub Actions should be used) to farce : the one thing that makes YAML anchors uniquely useful in the context of GitHub Actions is the one thing that GitHub Actions doesn’t support. To summarize, I think YAML anchors in GitHub Actions are (1) redundant with existing functionality, (2) introduce a complication to the data model that makes CI/CD human and machine comprehension harder, and (3) are not even uniquely useful because GitHub has chosen not to support the one feature (merge keys) that lacks a semantic equivalent in GitHub Actions. Of these reasons, I think (2) is the most important: GitHub Actions security has been in the news   a great deal recently , with the overwhelming consensus being that it’s too easy to introduce vulnerabilities in (or expose otherwise latent vulnerabilities through ) GitHub Actions workflow. For this reason, we need GitHub Actions to be easy to analyze for humans and machine alike. In effect, this means that GitHub should be decreasing the complexity of GitHub Actions, not increasing it. YAML anchors are a step in the wrong direction for all of the reasons aforementioned. Of course, I’m not without self-interest here: I maintain a static analysis tool for GitHub Actions, and supporting YAML anchors is going to be an absolute royal pain in my ass 3 . But it’s not just me: tools like actionlint , claws , and poutine are all likely to struggle with supporting YAML anchors, as they fundamentally alter each tool’s relationship to GitHub Actions’ assumed data model. As-is, this change blows a massive hole in the larger open source ecosystem’s ability to analyze GitHub Actions for correctness and security. All told: I strongly believe that GitHub should immediately remove support for YAML anchors in GitHub Actions. The “good” news is that they can probably do so with a bare minimum of user disruption, since support has only been public for a few days and adoption is (probably) still primarily at the single-use workflow layer and not the reusable action (or workflow) layer. That object model is essentially the JSON object model, where all elements appear as literal components of their source representation and take a small subset of possible types (string, number, boolean, array, object, null).  ↩ In other words: even though YAML itself is a superset of JSON, users don’t want YAML-isms to leak through to the object model. Everybody wants the JSON object model, and that means no “anchor” or “reference” elements anywhere in a deserialized structure.  ↩ To the point where I’m not clear it’s actually worth supporting anchors to any meaningful extent, and instead immediately flagging them as an attempt at obfuscation.  ↩ For humans: a new form of non-locality makes it harder to preserve local understanding of what a workflow, job, or step does: a unit of work may now depend on any other unit of work in the same file, including one hundreds or thousands of lines away. This makes it harder to reason about the behavior of one’s GitHub Actions without context switching. It would only be fair to note that GitHub Actions already has some forms of non-locality: global contexts, scoping rules for blocks, dependencies, step and job outputs, and so on. These can be difficult to debug! But what sets them apart is their lack of generality : each has precise semantics and scoping rules, meaning that a user who understands those rules can comprehend what a unit of work does without referencing the source of an environment variable, output, &c. For machines: non-locality makes it significantly harder to write tools that analyze (or transform) GitHub Actions workflows. The pain here boils down to the fact that YAML anchors diverge from the one-to-one object model 1 that GitHub Actions otherwise maps onto. With anchors, that mapping becomes one-to-many: the same element may appear once in the source, but multiple times in the loaded object representation. In effect, this breaks a critical assumption that many tools make about YAML in GitHub Actions: that an entity in the deserialized object can be mapped back to a single concrete location in the source YAML. This is needed to present reasonable source locations in error messages, but it doesn’t hold if the object model doesn’t represent anchors and references explicitly. Furthermore, this is the reality for every YAML parser in wide use: all widespread YAML parsers choose (reasonably) to copy anchored values into each location where they’re referenced, meaning that the analyzing tool cannot “see” the original element for source location purposes. I feel these pains directly: I maintain zizmor as a static analysis tool for GitHub Actions, and makes both of these assumptions. Moreover, ’s dependencies make these assumptions: (like most other YAML parsers) chooses to deserialize YAML anchors by copying the anchored value into each location where it’s referenced 2 . That object model is essentially the JSON object model, where all elements appear as literal components of their source representation and take a small subset of possible types (string, number, boolean, array, object, null).  ↩ In other words: even though YAML itself is a superset of JSON, users don’t want YAML-isms to leak through to the object model. Everybody wants the JSON object model, and that means no “anchor” or “reference” elements anywhere in a deserialized structure.  ↩ To the point where I’m not clear it’s actually worth supporting anchors to any meaningful extent, and instead immediately flagging them as an attempt at obfuscation.  ↩

0 views
Blog System/5 1 years ago

Bazel and glibc versions

Imagine this scenario: your team uses Bazel for fast, distributed C++ builds. A developer builds a change on their workstation, all tests pass, and the change is merged. The CI system picks it up, gets a cache hit from the developer’s build, and produces a release artifact. Everything looks green. But when you deploy to production, the service crashes with a mysterious error: . What went wrong? The answer lies in the subtle but dangerous interaction between Bazel’s caching, remote execution, and differing glibc versions across your fleet. In previous posts in this series, I’ve covered the fundamentals of action non-determinism , remote caching , and execution execution . Now, finally, we’ll build on those to tackle this specific problem. This article dives deep into how glibc versions can break build reproducibility and presents several ways to fix it—from an interesting hack (which spawned this whole series) to the ultimate, most robust solution. Before moving on and getting captivated by the intricate details of the problem, take a moment to support Blog System/5. Suppose you have a pretty standard (corporate?) development environment like the following: Developer workstations (WS). This is where Bazel runs during daily development, and Bazel can execute build actions both locally and remotely. A CI system. This is a distributed cluster of machines that run jobs, including PR merge validation and production release builds. These jobs execute Bazel too, who in turn executes build actions both locally and remotely. The remote execution (RE) system. This is a distributed cluster of worker machines that execute individual Bazel build actions remotely. The key components we want to focus on today are the AC, the CAS, and the workers—all of which I covered in detail in the previous two articles. The production environment (PROD). This is where you deploy binary artifacts to serve your users. No build actions run here. All of the systems above run some version of Linux, and it is tempting to wish to keep such version in sync across them all. The reasons would include keeping operations simpler and ensuring that build actions can run consistently no matter where they are executed. However, this wish is misguided and plain impossible. It is misguided because you may not want to run the same Linux distribution on all three environments: after all, the desktop distribution you run on WS may not be the best choice for RE workers, CI nodes, nor production. And it is plain impossible because, even if you aligned versions to the dot, you would need to take upgrades at some point: distributed upgrades must be rolled out over a period of time (weeks or even months) for reliability, so you’d have to deal with version skew anyway. To make matters more complicated, the remote AC is writable from all of WS, CI, and RE to maximize Bazel cache hits and optimize build times. This goes against best security practices (so there are mitigations in place to protect PROD), but it’s a necessity to support an ongoing onboarding into Bazel and RE. The question becomes: can the Linux version skew among all machines involved cause problems with remote caching? It sure can because C and C++ build actions tend to pick up system-level dependencies in a way that Bazel is unaware of (by default), and those influence the output the actions produce. Here, look at this: The version of glibc leaks into binaries and this is invisible to Bazel’s C/C++ action keys . glibc versions its symbols to provide runtime backwards compatibility when their internal details change, and this means that binaries built against newer glibc versions may not run on systems with older glibc versions. How is this a problem though? Let’s take a look by making the problem specific. Consider the following environment: In this environment, developers run Bazel in WS for their day-to-day work, and CI-1 runs Bazel to support development flows (PR merge-time checks) and to produce binaries for PROD. CI-2 sometimes runs builds too. All of these systems can write to the AC that lives in RE. As it so happens, one of the C++ actions involved in the build of , say , has a tag which forces the action to bypass remote execution. This can lead to the following sequence of events: A developer runs a build on a WS. has changed so it is rebuilt on the WS. The action uses the C++ compiler, so the object files it produces pick up the dependency on glibc 2.28. The result of the action is injected into the remote cache. CI-1 schedules a job to build for release. This job runs Bazel on a machine with glibc 2.17 and leverages the RE cluster which also contain glibc 2.17. Many C++ actions get rebuilt but is reused from the cache. The production artifact now has a dependency on symbols from glibc 2.28. Release engineering picks the output of CI-1, deploys the production binary to PROD, and… boom, PROD explodes: The fact that the developer WS could write to the AC is very problematic on its own, but we could encounter this same scenario if we first ran the production build on CI-2 for testing purposes and then reran it on CI-1 to generate the final artifact. So, what do we do now? In a default Bazel configuration, C and C++ action keys are underspecified and can lead us to non-deterministic behavior when we have a mixture of host systems compiling them. Let’s start with the case where you aren’t yet ready to strictly restrict writes to the AC from RE workers, yet you want to prevent obvious mistakes that lead to production breaks. The idea here is to capture the glibc version that is used in the local and remote environments, pick the higher of the two, and make that version number an input to the C/C++ toolchain. This causes the version to become part of the cache keys and should prevent the majority of the mistakes we may see. WARNING: This is The Hack I recently implemented and that drove me to writing this article series! Prefer the options presented later, but know that you have this one up your sleeve if you must mitigate problems quickly. To implement this hack, the first thing we have to do is capture the local glibc version. We can do this with: One important tidbit here is the use of the file, indirectly via the requirement of stamping. This is necessary to force this action to rerun on every build because we don’t want to hit the case of using an old tree against an upgraded system. As a consequence, we need to modify the script pointed at by script (you have one, right?) to emit the glibc version: The second thing we have to do is capture the remote glibc version. This is… trickier because there is no tag to force Bazel to run an action remotely. Even if we assume remote execution, features like the dynamic spawn strategy or the remote local fallback could cause the action to run locally at random. To prevent problems, we have to detect whether the action is running within RE workers or not, and the way to do that will depend on your environment: The third part of the puzzle is to select the highest glibc version between the two that we collected. We can do this with the following , leveraging ’s flag to compare versions. This flag is a GNU extension… but we are talking about glibc anyway here so I’m not going to be bothered by it : And, finally, we can go to our C++ toolchain definition and modify it to depend on the produced by the previous action: Ta-da! All of our C/C++ actions now encode the highest possible glibc version that the outputs they produce may depend on. And, while not perfect, this is an easy workaround to guard against most mistakes. But can we do better? Of course. Based on the previous articles, what we should think about is plugging the AC hole and forcing build actions to always run on the RE workers. In this way, we would precisely control the environment that generates action outputs and we should be good to go. Unfortunately, we can still encounter problems! Remember how I said that, at some point, you will have to upgrade glibc versions? What happens when you are in the middle of a rolling upgrade to your RE workers? The worker pool will end up with different “partitions”, each with a different glibc version, and you will still run into this issue. To handle this case, you would need to have different worker pools, one with the old glibc version and one with the new version, and then make the worker pool name be part of the action keys. You would then have to migrate from one pool to the other in a controlled manner. This would work well at the expense of reducing cache effectiveness, causing a a big toll on operations, and making the rollout risky because the switch from one pool to another is a all-or-nothing proposition. The real solution comes in the form of sysroots. The idea is to install multiple parallel versions of glibc in all environments and then modify the Bazel C/C++ toolchain to explicitly use a specific one. In this way, the glibc version becomes part of the cache key and all build outputs are pinned to a deterministic glibc version. This allows us to roll out a new version slowly with a code change, pinning the version switch to a specific code commit that can be rolled back if necessary, and keeping the property of reproducible builds for older commits. This is the solution outlined at the end of Picking glibc versions at runtime and is the only solution that can provide you 100% safety against the problem presented in this article. It is difficult to implement, though, because convincing GCC and clang to not use system-provided libraries is tricky and because this solution will sound alien to most of your peers. The problem presented in this article is far from theoretical, but it’s often forgotten about because typical build environments don’t present significant skew across Linux versions. This means that facing new glibc symbols is unlikely, so the chances of ending up with binary-incompatible artifacts are low. But they can still happen, and they can happen at the worst possible moment. Therefore, you need to take action. I’d strongly recommend that you go towards the sysroot solution because it’s the only one that’ll give you a stable path for years to come, but I also understand that it’s hard to implement. Therefore, take the solutions in the order I gave them to you: start with the hack to mitigate obvious problems, follow that up with securing the AC, and finally go down the sysroot rabbit hole. As for the glibc 2.17 mentioned en-passing above, well, it is ancient by today standards at 13 years of age, but it is what triggered this article in the first place. glibc 2.17 was kept alive for many years by the CentOS 7 distribution—an LTS system used as a core building block by companies and that reached EOL a year ago, causing headaches throughout the industry. Personally, I believe that relying on LTS distributions is a mistake that ends up costing more money/time than tracking a rolling release, but I’ll leave that controversial topic for a future opinion post. Imagine this scenario: your team uses Bazel for fast, distributed C++ builds. A developer builds a change on their workstation, all tests pass, and the change is merged. The CI system picks it up, gets a cache hit from the developer’s build, and produces a release artifact. Everything looks green. But when you deploy to production, the service crashes with a mysterious error: . What went wrong? The answer lies in the subtle but dangerous interaction between Bazel’s caching, remote execution, and differing glibc versions across your fleet. In previous posts in this series, I’ve covered the fundamentals of action non-determinism , remote caching , and execution execution . Now, finally, we’ll build on those to tackle this specific problem. This article dives deep into how glibc versions can break build reproducibility and presents several ways to fix it—from an interesting hack (which spawned this whole series) to the ultimate, most robust solution. Before moving on and getting captivated by the intricate details of the problem, take a moment to support Blog System/5. The scenario Suppose you have a pretty standard (corporate?) development environment like the following: Developer workstations (WS). This is where Bazel runs during daily development, and Bazel can execute build actions both locally and remotely. A CI system. This is a distributed cluster of machines that run jobs, including PR merge validation and production release builds. These jobs execute Bazel too, who in turn executes build actions both locally and remotely. The remote execution (RE) system. This is a distributed cluster of worker machines that execute individual Bazel build actions remotely. The key components we want to focus on today are the AC, the CAS, and the workers—all of which I covered in detail in the previous two articles. The production environment (PROD). This is where you deploy binary artifacts to serve your users. No build actions run here. All of the systems above run some version of Linux, and it is tempting to wish to keep such version in sync across them all. The reasons would include keeping operations simpler and ensuring that build actions can run consistently no matter where they are executed. However, this wish is misguided and plain impossible. It is misguided because you may not want to run the same Linux distribution on all three environments: after all, the desktop distribution you run on WS may not be the best choice for RE workers, CI nodes, nor production. And it is plain impossible because, even if you aligned versions to the dot, you would need to take upgrades at some point: distributed upgrades must be rolled out over a period of time (weeks or even months) for reliability, so you’d have to deal with version skew anyway. To make matters more complicated, the remote AC is writable from all of WS, CI, and RE to maximize Bazel cache hits and optimize build times. This goes against best security practices (so there are mitigations in place to protect PROD), but it’s a necessity to support an ongoing onboarding into Bazel and RE. The problem The question becomes: can the Linux version skew among all machines involved cause problems with remote caching? It sure can because C and C++ build actions tend to pick up system-level dependencies in a way that Bazel is unaware of (by default), and those influence the output the actions produce. Here, look at this: The version of glibc leaks into binaries and this is invisible to Bazel’s C/C++ action keys . glibc versions its symbols to provide runtime backwards compatibility when their internal details change, and this means that binaries built against newer glibc versions may not run on systems with older glibc versions. How is this a problem though? Let’s take a look by making the problem specific. Consider the following environment: In this environment, developers run Bazel in WS for their day-to-day work, and CI-1 runs Bazel to support development flows (PR merge-time checks) and to produce binaries for PROD. CI-2 sometimes runs builds too. All of these systems can write to the AC that lives in RE. As it so happens, one of the C++ actions involved in the build of , say , has a tag which forces the action to bypass remote execution. This can lead to the following sequence of events: A developer runs a build on a WS. has changed so it is rebuilt on the WS. The action uses the C++ compiler, so the object files it produces pick up the dependency on glibc 2.28. The result of the action is injected into the remote cache. CI-1 schedules a job to build for release. This job runs Bazel on a machine with glibc 2.17 and leverages the RE cluster which also contain glibc 2.17. Many C++ actions get rebuilt but is reused from the cache. The production artifact now has a dependency on symbols from glibc 2.28. Release engineering picks the output of CI-1, deploys the production binary to PROD, and… boom, PROD explodes:

0 views
ENOSUCHBLOG 1 years ago

Bypassing GitHub Actions policies in the dumbest way possible

TL;DR : GitHub Actions provides a policy mechanism for limiting the kinds of actions and reusable workflows that can be used within a repository, organization, or entire enterprise. Unfortunately, this mechanism is trivial to bypass . GitHub has told me that they don’t consider this a security issue (I disagree), so I’m publishing this post as-is. Update 2025-06-13 : GitHub has silently updated the actions policies documentation to note the bypass in this post: Policies never restrict access to local actions on the runner filesystem (where the path start with ). GitHub Actions is GitHub’s CI/CD offering. I’m a big fan of it, despite its spotty security track record . Because a CI/CD offering is essentially arbitrary code execution as a service , users are expected to be careful about what they allow to run in their workflows, especially privileged workflows that have access to secrets and/or can modify the repository itself. That, in effect, means that users need to be careful about what actions and reusable workflows they trust. Like with other open source ecosystems, downstream consumers (i.e., users of GitHub Actions) retrieve their components (i.e., action definitions) from an essentially open index (the “Actions Marketplace” ​ 1 ). To establish trust in those components, downstream users perform all of the normal fuzzy heuristics: they look at the number of stars, the number of other user, recency of activity, whether the user/organization is a “good” one, and so forth. Unfortunately, this isn’t good enough along two dimensions: Even actions that satisfy these heuristics can be compromised. They’re heuristics after all, not verifiable assertions of quality or trustworthiness. The recent tj-actions attack typifies this: even popular, widely-used actions are themselves software components, with their own supply chains (and CI/CD setups). This kind of acceptance scheme just doesn’t scale , both in terms of human effort and system complexity: complex CI/CD setups can have dozens (or hundreds) of workflows, each of which can contain dozens (or hundreds) of jobs that in turn employ actions and reusable workflows. These sorts of large setups don’t necessarily have a single owner (or even a single team) responsible for gating admission and preventing a the introduction of unvetted actions and reusable workflows. The problem (as stated above) is best solved by eliminating the failure mode itself: rather than giving the system’s committers the ability to introduce new actions and reusable workflows without sufficient review, the system should prevent them from doing so in the first place . To their credit, GitHub understands this! They have a feature called “Actions policies 2 ” that does exactly this. From the Manage GitHub Actions settings documentation: You can restrict workflows to use actions and reusable workflows in specific organizations and repositories. Specified actions cannot be set to more than 1000. (sic) To restrict access to specific tags or commit SHAs of an action or reusable workflow, use the same syntax used in the workflow to select the action or reusable workflow. For an action, the syntax is . For example, use to select a tag or to select a SHA. For more information, see Using pre-written building blocks in your workflow. For a reusable workflow, the syntax is . For example, . For more information, see Reusing workflows. You can use the wildcard character to match patterns. For example, to allow all actions and reusable workflows in organizations that start with , you can specify . To allow all actions and reusable workflows in repositories that start with , you can use . For more information about using the wildcard, see Workflow syntax for GitHub Actions. Use to separate patterns. For example, to allow and , you can specify . GitHub also provides special “preset” cases for this functionality, such as allowing only actions and reusable workflows that belong to the same organization namespace as the repository itself. Here’s what that looks like on a dummy organization and repository of mine: …and here’s what happens when I try to violate that policy, e.g. by using in a workflow: This is fantastic, except that it’s trivial to bypass. Let’s see how. To understand how we’re going to bypass this, we need to understand a few of the building blocks underneath actions and reusable workflows. In particular: These four aspects of GitHub Actions compose together into the world’s dumbest policy bypass : instead of doing , the user can (or otherwise fetch) the repository into the runner’s filesystem, and then use to run the very same action. Here’s what that looks like in practice: (The actual block of the step is inconsequential — I just used that repository for the demo, but anything would work.) And naturally, it works just fine: The fix for this bypass is simple, if potentially somewhat painful: GitHub Actions could consider “local” references to be another category for the purpose of policies, and reject them whenever the policy doesn’t permit them. This would seal off the entire problem, since would just stop working. The downside is that it would potentially break existing users of policies who also use local actions and reusable workflows, assuming there are significant numbers of them 4 . The other option would be to leave it the way it is, but explicitly document local references as a limitation of this policy mechanism. I honestly think this would be perfectly fine; what matters is that users 5 are informed of a feature’s limitations, not necessarily that the feature lacks limitations. First, I’ll couch this again: this is not exactly fancy stuff. It’s a very dumb bypass, and I don’t think it’s critical by any means. At the same time, I think this matters a great deal : ineffective policy mechanisms are worse than missing policy mechanisms, because they provide all of the feeling of security through compliance while actually incentivizing malicious forms of compliance . In this case, the maliciously complying party is almost certainly a developer just trying to get their job done: like most other developers who encounter an inscrutable policy restriction, they will try to hack around it such that the policy is satisfied in name only. For that reason alone I think GitHub should fix this bypass, either by actually fixing it or at least documenting its limitations. Without either of those, projects and organizations are likely to mistakenly believe that these sorts of policies provide a security boundary where none in fact exists . Technically “publishing” an action to the Actions Marketplace is not required; anybody can do to fetch the action defined in even if it isn’t published. All publishing does is give the action a marketplace page and the potential for a little blue checkmark of unclear security value.  ↩ Actually, I don’t know what this feature is called. It’s titled “Policies” under the “Actions” section of the repo/org/enterprise settings and is documented under “Github Actions policies” in the Enterprise documentation, but I’m not sure if that’s an umbrella term or not. I’m just going to keep calling it “Actions policies” for now.  ↩ Where “owner” is an individual owner or an organization, which in turn might be controlled by an enterprise. But that last bit isn’t visible in the namespace.  ↩ I honestly have no idea how widely used this policy feature is.  ↩ Here, policy authors and enforcers.  ↩ Even actions that satisfy these heuristics can be compromised. They’re heuristics after all, not verifiable assertions of quality or trustworthiness. The recent tj-actions attack typifies this: even popular, widely-used actions are themselves software components, with their own supply chains (and CI/CD setups). This kind of acceptance scheme just doesn’t scale , both in terms of human effort and system complexity: complex CI/CD setups can have dozens (or hundreds) of workflows, each of which can contain dozens (or hundreds) of jobs that in turn employ actions and reusable workflows. These sorts of large setups don’t necessarily have a single owner (or even a single team) responsible for gating admission and preventing a the introduction of unvetted actions and reusable workflows. For an action, the syntax is . For example, use to select a tag or to select a SHA. For more information, see Using pre-written building blocks in your workflow. For a reusable workflow, the syntax is . For example, . For more information, see Reusing workflows. Actions and reusable workflows share the same namespace as the rest of GitHub, i.e. 3 ; When a user writes something like in a workflow, GitHub resolves that reference to mean “the file defined at tag in the repository”; keywords can also refer to relative paths on the runner itself. For example, runs the step with the in the current directory. Relative paths from the runner are not inherently part of the repository state itself: the runner is can contain any state introduced by previous steps within the same job. Technically “publishing” an action to the Actions Marketplace is not required; anybody can do to fetch the action defined in even if it isn’t published. All publishing does is give the action a marketplace page and the potential for a little blue checkmark of unclear security value.  ↩ Actually, I don’t know what this feature is called. It’s titled “Policies” under the “Actions” section of the repo/org/enterprise settings and is documented under “Github Actions policies” in the Enterprise documentation, but I’m not sure if that’s an umbrella term or not. I’m just going to keep calling it “Actions policies” for now.  ↩ Where “owner” is an individual owner or an organization, which in turn might be controlled by an enterprise. But that last bit isn’t visible in the namespace.  ↩ I honestly have no idea how widely used this policy feature is.  ↩ Here, policy authors and enforcers.  ↩

0 views
usher.dev 1 years ago

Minimal GitOps-like Deployment Tool

However, it's easy to forget that there's a class of service/business where this doesn't make sense. Many small businesses (if they're not using a PaaS like Fly), don't have the resources to run a Kubernetes cluster and 95% of the time, just need an app to run on a single VPS. Sometimes, maybe during an event or a promotional period, it might make sense to add another server to the mix, load balancing traffic somewhere (DigitalOcean load balancers are pretty effective). Assuming we have some nice way to spin up more servers running the app (a nice Ansible Playbook perhaps?), and assuming we have some nice CD pipeline to deploy the app, we need some way to make sure all the servers are running the latest version. One pattern I like for this is a simplified GitOps pull model (think ArgoCD or Flux but held together with duct tape) where we: This way, your deployment pipeline doesn't need to know anything about the servers it's deploying to - it just needs to know how to update the repo with a reference to the new release. In one case, I have a repo called 'ops' which along with all the ansible playbooks, has a set of files, one for each app which contain (along with other settings), a line like: I can then run a script like the following on each server, which pulls down the repo, checks each app and updates the systemd service file to point to the latest release. For cases where Kubernetes and a full GitOps solution is overkill, this is a nice way to get a simple deployment pipeline setup. Have a repo which indicates what app releases should currently be deployed. Get your servers to query this repo to determine what it should be running and automatically update the app on the server to match. Make your deployment pipeline update the repo with a reference to the new release.

0 views
The Tymscar Blog 3 years ago

How I deploy private GitHub projects to local self-hosted servers (CI/CD)

I have a lot of experience with massive CI/CD pipelines that deploy private code to public servers. I’ve also worked with pipelines that deploy public repositories to private servers, such as my homelab. However, I never experimented with a pipeline that takes a private GitHub repo, builds it, and deploys it to a server on the LAN. That’s precisely what I needed for a project I’m currently working on that isn’t yet public.

0 views
Alex Molas 3 years ago

Automate your static blogroll.

In this post I explain how I built my automatic blogroll using Github Action and Github Pages.

0 views

Speeding up CircleCI Builds with Caching

Redownloading dependencies for every step in your CI/CD pipeline can be time consuming. You can dramatically speed up the build time of your application with caching, making your team more responsive to breaking changes and ultimately more productive. Here's how to do it.

0 views