BazelCon 2026 recap
We are right in the middle of fall, and that means BazelCon 2026 just took place. This time around it was hosted in Amsterdam and spanned three days: one day of tutorials and an overlapping Buildbarn meetup that I wasn’t aware of, and two days of main-conference talks and events. And as the event is behind us, it’s time for the now-traditional recap. Welcome to BazelCon 2026 sign. What follows is a recollection of my notes arranged by topic and intermixed with my own commentary and thoughts. Some sections and topics are more detailed than others because in some cases I didn’t have a seat and couldn’t type enough, and in others I got distracted in the hallways chatting with people. Speaking of people, several of you approached me to praise previous editions of these recaps, so I now feel obliged to write another one. Thanks for the kind comments! As we will see, “scale” was a central theme of the conference: AI is multiplying concurrent builds and tests, artifacts and graphs are growing at an accelerated pace due to new features like configuration transitions, and the ecosystem is responding by making execution, caching, and analysis more efficient. There is a lot to cover but here are a few things that felt memorable to me : The BUILD Foundation is now a reality, giving the Bazel ecosystem a vendor-neutral home. Its goal is to act as a proxy between the community and Google to help Google understand what to prioritize and to fund projects. If your company can spare “a few” dollars, please donate! AI-driven development is having a broad impact on the industry and, in particular, everyone is seeing their CI systems become overloaded. Well, news at 11: I’m dealing with this day in and day out. But what was new to me was that we at Snowflake are not the only ones struggling with local system overload. Lots of people out there have likewise “discovered” Git worktrees as a way to run multiple agents in parallel, which in turn means multiple concurrent Bazel servers. Bazel 10 is shaping up to be a substantial release that will help with the problem right above due to its many performance improvements and memory consumption optimizations. Have you struggled to keep Bazel up-to-date before? These days, if you want to get any platform-related work funded you have to spin it in terms of AI, so this release will be an easy sell. Let’s “Rewrite it in Rust”? I’m impressed. Google is seriously evaluating a progressive rewrite of Bazel in Rust, and this is probably the most exciting news I got out of the conference. You know I’ve been critical of Bazel’s usage of Java since day one so, if this takes off, I’ll be the first in line to evaluate the results (and it will likely push me to actually contribute to Bazel again). Fast and correct sandboxing on macOS remains a problem. This is “interesting” to me because it’s a problem I tried to fix almost ten years ago with the original sandboxfs, but I failed because of unexpected bugs in macOS’s FUSE support and eventually abandoned the idea when FUSE became a dead end on Apple’s platforms. Funnily enough, other folks have now reinvented sandboxfs—and even given it the same name (ugh)—and there is even another implementation of a similar idea in the form of actiondFS. Types are coming to Starlark, but Bazel’s late arrival to the party means that multiple competing approaches now exist. We have at least two LSPs with different annotation mechanisms and Buck 2’s own syntax. Hopefully they can all converge. Platforms and configurations remain difficult to model at scale. Careless use of transitions and build settings can produce an explosion of configured actions. Apparently, as one speaker noted, AI agents love transitions for some reason, which is interesting because they are not a popular feature and they are not particularly easy to use. Let’s dive into the full report now. You’ll find my notes arranged by talk but I’ve tried to group them into higher-level themes for easier navigation. These conference summaries take effort to write, and some of you are waiting for them. Show your true support by subscribing! The two keynotes are usually the most information-packed events but… as you can imagine they are also the most popular. The venue this year was not roomy enough to host BazelCon’s attendance (Helen quoted that we got 600 registrations), so I had to stand for these two events. The recordings and slides are not out yet, so consequently, I can only tell you a few details from memory. The conference opened with the usual Bazel Community Update , presented by Helen Altshuler from EngFlow and Chuck Grindel from Glydways. The biggest announcement is that the BUILD Foundation is now a reality. The effort had been discussed at the previous two BazelCons and likely various meetups, but it had remained mostly invisible while its organizers worked through the legal and financial details. The foundation now provides a neutral organization to support the broader build ecosystem (in their words this is not just for Bazel), and it is looking for both members and concrete proposals to fund. If you want to get involved, get your company pumped up and email [email protected] . Documentation continues to receive attention as it has always been a lowlight in previous surveys. Work is underway to make the docs easier for the community to own and improve by decoupling them from Google’s internal systems. But one must wonder: what exactly is the biggest problem with documentation in today’s world? AI agents used to do terribly with Bazel just a year ago and nowadays do… fine as we shall see in the summary. Maybe the work on documentation needs a different spin now. And as usual, a reminder to fill the annual Bazel user survey . Your input is necessary to quantify ecosystem-wide pain points and influence future work, especially now that the foundation exists. John Field and Tobias Werth from Google followed with the State of the Union , which presents a recap of major improvements during the year and the path ahead for Bazel 10 and beyond. Performance remains a major theme. Google reported improvements of about 15% in some workloads together with reductions in memory consumption. Some of these speedups come from upgrading from Java 25 to 27, while others come from correct action rewinding, optimizations to Skyframe cache pruning, and content-defined chunking. These changes are very relevant because Bazel’s large memory consumption gets in the way of scaling CI and prevents using multiple AI agents in parallel. And here is where we heard about Skymetal: the codename for the Google-internal exploration of rewriting Bazel in Rust. The rationale for this rewrite is obviously performance, and I’m guessing it’s now a real consideration because AI can make such large-scale changes feasible. Tobias acknowledged that Java was a reasonable choice back in 2006 when Blaze started and that the JVM can deliver great throughput performance at scale… but it is not the best choice for a tool like Bazel in this day and age, especially when optimizing for memory efficiency is critical. I called it in 2015. I have been in the BazelCon Program Committee (PC) for 3 years now. I remember that, during last year’s review, we encountered a few AI talks but I think none made the cut: they all looked too hype-y and out of place. But it was impossible to ignore the topic this year, and it was actually important to bring AI-related talks on board to not sound tone deaf. AI is a reality in our industry, and Bazel is a key component to survive the transition. Nicolas Lopez from EngFlow and I, representing Snowflake, presented AI Development: Challenges and Early Learnings for Bazel . Agentic workflows have expanded rapidly throughout 2026, and the effects are visible well beyond code generation. At Snowflake, we were fortunate enough to finish our Bazel migration and to stabilize remote execution by moving from our self-hosted open-source deployment to EngFlow’s fully-managed service before all of this happened. And I say fortunate because the increase in CI load that AI workflows brings can only be handled with a good build system and caching story. But none of that is news, so the core of the talk was to look at how Bazel usage looks different in agent-initiated environments. The key points from my side: Agents run more builds more frequently than humans, discover and invoke targets that were often “internal only”, and explore several approaches in parallel. Your intuition about how to evaluate which builds matter and how to measure them is probably wrong by now. Bazel and IDEs have never played great together and maybe that just doesn’t matter anymore. We are seeing a sharp decline in IDE usage, although developers still want a way to browse code changes and debug them. This means LSPs are key now to support “leaner” IDEs, and maybe standalone debuggers. Short build and test latency is critical when a human must remain focused on a coding task but, in an agentic workflow, the constraints are different. You know that, after you hit “execute this plan”, the agent will take many minutes to return so minimal build latency is not as important. And actually… maybe you now want to trade some latency in exchange for extra validations so that agents have to take fewer turns to converge. And because humans are now waiting for agents to finish for long periods of time, they have to fill that time with other tasks, which has driven them to adopt Git worktrees en masse. This is leading to scalability problems on dev machines because multiple concurrent Bazel server processes are not great for memory and disk usage. The rest of the talk, presented by Nicolas, focused on trying to understand behavioral changes by looking at anonymized backend data across companies. One specific insight was that people’s work days have lengthened over the year. Cynically, the promise of “working less because of AI” has not materialized: in fact, we are all seemingly working longer hours and with fewer chunks of focus time, so the dangers of burnout are higher. Maggie Lou from BuildBuddy presented Taking Humans Out of the Build Loop . The premise was that coding agents have improved quickly, but people still spend too much time copy/pasting logs, triggering builds, explaining failures, and reviewing the results. In her presentation, we witnessed many new features in BuildBuddy to allow agents (and humans!) to interact and diagnose remote builds with ease: to download filtered logs for a build. to answer questions such as “Why was my build slow?” which are extremely common in the Bazel ecosystem. to start a remote Firecracker VM and run Bazel inside of it. The VMs are snapshotted for quick reuse, and the runners can execute arbitrary commands rather than Bazel alone. to diagnose a failed build and either create a pull request or push a commit with the proposed correction. to perform a review and post comments to GitHub. The need for these features is unsurprising this late in 2026, but what’s really useful in their offering is how these features integrate with build data. Inspecting the BEP, profiles, remote execution metadata, and logs in unison via an agent can yield much better diagnostics and automated improvements than staring at just a few lines of console output. One thing that surprised me in chatting with the presenter is that BuildBuddy is not particularly big in terms of headcount but they churn out features at a very high rate—and that was true even before AI. Impressive! Armando Montanez (and Ulf Adams, but he didn’t make it) from EngFlow presented Proving Equivalence in any2bazel Build System Migrations . The talk described a structured way to use AI agents for build system migrations and started with a reflection: AI agents used to be terrible at Bazel less than a year ago, but now they do pretty damn well—especially if you can close the loop with automated validations. However, even if agents do well at writing Bazel builds, the fidelity of a migration leaves a lot to be desired if left unattended. And here is where any2bazel comes into play: it turns the process into a guess-and-verify loop. The initial focus of the tool are CMake-to-Bazel migrations, and the tool comes with a script that extracts reference compiler and linker actions from CMake in a deterministic manner. The agent then writes Bazel build files, runs , and feeds the resulting actions to a differ tool that compares those to the originals. The loop continues until the two action sets are equivalent. The equivalence checks are what’s hard though: there are compiler flags that may have no impact on the result, for example, so they need to be “ignored” to let the agent experiment with different Bazel-native approaches until it gets to bit-for-bit identical outputs. It’s important to focus on comparing outcomes and not internals so that the generated Bazel builds do not necessarily follow ancient idioms, like e.g. a Maven-like project structure in a Bazel build. In the end, though, this approach works surprisingly well. But the question is: “How good are the generated build files?” The presenters evaluated them along four axes: Correctness: Good, because equivalence is the central constraint of the loop. Hermeticity: Good, but you might need to do extra upfront work to remove host-specific behavior from the reference build before starting. Efficiency: Usually good when standard rules can model the build. Complex projects with custom build rules may end up with overly large Bazel actions though. Maintainability: Unsurprisingly, the weakest point. Agents tend to model variation with configuration flags instead of proper platforms and toolchains, which is not idiomatic to Bazel. Expert review of AI outputs remains critical, which is, again, not surprising. As a side note, it seems like the team likes pain because they tried to migrate an MSBuild project too. MSBuild is more painful than others because its binary log requires running through a real build: there is no dry-run mode, so it might be difficult to extract a truly clean log of actions, and the process is slow. Fortunately, the reference needs to be extracted only once before the comparison agentic loop can start, and it works: we saw a POC migration of the Color Picker tool that comes from PowerToys. When optimizing a distributed build system, it’s easy to focus on the costs of the remote execution workers: how many there are, what kinds of hardware they use, how well they auto-scale… but a significant driver in cost comes from the storage systems required to support those workers. In other words: the remote cache is expensive. Let’s look at a few topics that deal with this problem. Tyler French from BuildBuddy and Benjamin Ingberg from Meroton presented Content-Defined Chunking for Bazel . This was one of the most immediately-practical talks for anyone operating a remote cache, and it surely caught my eye because I had just been troubleshooting a problem at work that CDC would have prevented in the first place. The problem is that Bazel has traditionally treated each file as one indivisible blob. If an action produces a one-gigabyte binary (not uncommon these days huh) and this binary only changes by a few bytes from a previous build, the client or remote worker must re-upload the entire new file to the CAS and the CAS must store a full copy of those bytes. This is inefficient network- and storage-wise because the cache has to be overprovisioned so that these very large, rarely reused blobs don’t prematurely evict valuable cache entries. Content-defined chunking (CDC) divides a file into variable-sized pieces that can be cached independently. To compute the chunks, a rolling hash scans the contents and chooses boundaries when the hash matches a particular condition. Because the boundaries depend on the content rather than fixed offsets, inserting a few bytes in the middle of a file shifts only the nearby chunks instead of invalidating every subsequent one. With CDC, Bazel represents the original file as a list of chunk offsets and digests instead of a single digest, and each chunk is uploaded or downloaded independently. The presenters reported average reuse of chunks of 36%, chunks of 40%, chunks of 96%, chunks of 98%, and chunks of 99.5%. Furthermore, BuildBuddy estimates that CDC saves 35% of bytes written, deduplicates 60% of chunked bytes, and avoids about 400 TB of traffic per day across its fleet. Finally, the median project sees about 70% less remote storage, while local disk caches shrink by roughly 40%. Impressive numbers. Almost all of these benefits can be achieved by enabling CDC on the backend alone without changes to Bazel because most savings come from worker-to-CAS communication. But there are improvements to be made on the client too, of course. CDC fits bb-clientd especially well because chunks can be reused across output bases and, to materialize a file, the client needs only its chunk map. This can be much cheaper than reconstructing the whole file on every incremental build. For existing users, the starting point is together with a server that supports remote CDC. You’ll want to use the latest Bazel 8.x or 9.x releases for best results, as earlier versions didn’t perform the necessary validations to ensure that CDC didn’t corrupt data. The talk ended with a plea for tool authors to rethink binary formats: files that contain absolute pointers in them are poison for caching even with CDC because small changes in a file can have cascading effects across it. Wherever possible, relative pointers are better. Lore is Epic Games’ open-source, content-addressed version control system for very large, asset-heavy monorepos. Mattias Jansson from Epic Games came on stage to talk about it. But this was not just a presentation of the version control system. This was also a presentation of how Lore integrates with Bazel to prevent ferrying large assets between Lore repositories, the machine running Bazel, and the remote execution/cache cluster. The insight is that Lore and Bazel’s remote cache are both backed by a CAS… so if they can share the same storage, artifacts don’t need to move around. The idea relies on the fact that Bazel-the-client does not need to know the content of source files. All Bazel needs to know are the digests of those files to detect file changes across builds and to compose Remote Execution (RE) API requests. You’d imagine that computing digests requires reading the files, and you’d be right… but the server (Lore) already has those files, can compute their digests once, and can expose those to Bazel for consumption. So, what Epic Games has built is a translation layer between the RE API and Lore’s chunked content addresses, plus a virtual file system that exposes the precomputed digests to Bazel. In this way, Bazel talks to just one storage system and can leverage digests without computing them itself. Of interest was a brief note that Bazel can now use BLAKE3 instead of SHA-256 for content digests, which can likely be a big deal for performance because the former excels in speed due to parallel chunk processing. TIL. Caleb Froese from Canva presented Diffing Millions of Screenshots a Day, Hermetically . The premise of the talk is that, when developing a UI, you really want to have regression tests for how the UI looks —and doing that at scale is difficult. The idea is simple, though: make the code render a part of your UI, capture a screenshot, and compare it against a known-good image. Check in the golden image into source control, and call it a day. Except… if you do that at Canva’s scale, you end up growing the Git repository by about 20 GB per day , which is obviously untenable due to how Git works. A different approach is to store the images elsewhere and expose them to tests through an external repository rule. This integrates well with Bazel in the sense that the downloads are out of the execution path, but it also scales poorly: too much data must be downloaded to each client just for it to be re-uploaded right away to remote executors. And it also creates friction for developers when defining the build graph. In the end, Canva’s solution is to use the Build Event Service and the remote CAS to keep the assets outside of Git while preserving hermetic tests. This pattern applies to any test suite with large golden artifacts, not screenshots alone. Similar story to what Lore tries to fix. There is one extra wrinkle to worry about: GPU-based rendering introduces tiny output differences that make pixel-perfect image comparisons difficult. Canva’s solution is to pin these tests to a specific hardware class by selecting an execution pool through the execution property. That trades scheduling flexibility for reproducible images, which is a good compromise for their workload. Optimizing the way Bazel handles large artifacts helps with performance, but there are many other things that do too. There were a large number of talks around this topic, so let’s dive into some of them. Guillaume Maudoux from Tweag presented Taming Configurations Explosion , a practical talk addressing problems they’ve seen while supporting builds that target physical hardware boards. Such builds grew to have more than three million configured actions even though only about 500,000 distinct actions needed to run. The root problem is that configuration settings accumulate. A project may begin with a setting for the target platform, then add the board’s operating system, its physical layout, its capabilities, and so on. Every setting creates a new dimension in the configuration space and, because each configuration gets a distinct output-tree hash, apparently-different actions may in fact do identical work. Before you know it, you have blown up the action graph and possibly your disk space too. In the example from the talk, 12 parameters end up representing 72 million possible configurations. Only 856 of those configurations matter, and a particular source file has just 39 meaningful ways to be compiled. The rest is duplication introduced by the model. There are three main tools to fight this: Reset transitions: These discard irrelevant configuration values as early as possible in the graph. The art is to move decisions up the dependency tree so that downstream targets do not carry unnecessary dimensions. The trade-off is that ordinary output can become unmanageable, forcing users toward . But then… requires a configuration, so some targets may not be queryable at all without one. Path mapping: If two actions cannot be made identical during analysis, Bazel can rewrite configuration-specific paths during execution and replace their hashes with a stable component. This should be seen as a measure of last resort, though, because subtle problems arise. For example, if the same logical path comes from different files from different configurations, Bazel disables path mapping silently. Tweag fixed Bazel so that different copies of the same file don’t disable path mapping, but debuggability continues to be a problem when this condition does not apply. Flag scopes: This new feature allows a flag to declare where in the graph it applies and to automatically reset to its default everywhere else. This is similar to a reset transition but requires less manual graph plumbing. On the other hand, describing the correct scope can sometimes be harder than writing the transition directly. Han-Wen Nienhuys from EngFlow presented Horapha: Your Personal Warm Bazel , a clever experiment to avoid repeatedly starting Bazel servers and rebuilding their in-memory state. Several CI systems already offer similar features for efficiency, but Horapha targets developer-initiated workflows. The idea is to snapshot a warmed-up Bazel JVM with a new command and save that to disk. A later either starts a normal server when no checkpoint exists or restores the snapshot and continues from the saved state. The implementation relies on CRIU (Checkpoint/Restore In Userspace) , a Linux-only technology that can serialize a running process to disk. Correct restoration requires preserving process IDs, which can be achieved by placing Bazel in a PID namespace. In other words: a container but without pulling in the full weight of Docker. The prototype is available in the Horapha repository with the corresponding Bazel changes in the horapha branch of Han-Wen’s Bazel fork . Han-Wen is now trying to carve a path to get this integrated into upstream Bazel. Did you know that Emacs works similarly? Your executable is actually a snapshot of the LISP machine computed during the build process and stored to disk by the portable dumper introduced in Emacs 27. John Cater and Antonio Di Stefano from EngFlow presented Persistent Test Runners . For a long time, Bazel has had persistent workers for compilation actions but supporting tests has remained elusive—yet persistent workers for tests promise to increase test performance significantly by reducing JVM startup times, sharing heavyweight fixtures across tests, and sharing state such as container images. The reason I said elusive is because Bazel has tried to solve this before. Twice. Google introduced a persistent test feature back in 2019 but it was rolled back that same day. The roll forward was attempted again a month later, already in 2020, but this time separating the feature flag from the runner. Unfortunately, the whole mechanism was scrapped in 2022 because it never worked reliably. The old design suffered from classpath problems after incremental builds, leaked state between tests, and was tightly coupled to Bazel’s native rule—an unacceptable constraint due to the desire to move all rules to Starlark. More fundamentally, the test protocol and the persistent worker protocol modeled very different things. The new proposal from EngFlow introduces an API that explicitly separates the settings and inputs used to create the runner from the data needed by one test request. That distinction is important because the runner infrastructure must be able to decide whether an existing JVM is still compatible with a new test. The team explored several alternatives along the way: Pre-warming the next JVM instead of keeping the current one alive. This provided more control over when and why to restart a test in a new JVM, but suffered from significant coordination overhead. Forking a JVM. This did not work because, due to POSIX semantics, only the calling thread survives a fork call. The JVM is heavily multithreaded, so if you end up losing all but one thread, the new JVM is just not going to work. Android’s Zygote. This provides a pre-warmed runtime ready to be forked ( , anyone?), but it exists only for ART, not the standard JVM, and it’s just not reasonable to expect a switch to ART. Checkpointing. Instead of forking a running process, why not snapshot it to disk so that it can be re-created again? This might have worked, like Horapha does for Bazel startup, but raised difficult questions about when to capture the process and how to handle state outside the JVM. In the end, the only piece that survived these experiments was the coordinator that asks the runner whether a JVM can be reused. The decision considers the Java executable, JVM options, driver code, plugin configuration, and the digests of all loaded JARs. Configurable hooks run at JVM start/stop events as well as before/after each test. When running Bazel’s own test suite, this approach reduced total test time by 46%. The prototype is available in the branches of TheGrizzlyDev/bazel and TheGrizzlyDev/rules_java . It can be tried with a local module override and . Unconference discussion topic proposals and votes for the second day. Remote execution BoF Chi Wang and Tiago Quelhas led the Remote Execution Birds of a Feather . I only stayed for half of it, and so I only captured the discussion around remote persistent workers. Persistent workers are already part of several commercial remote execution offerings, including BuildBuddy and EngFlow, but their implementation was derived from Bazel’s local persistent worker support, which is not… particularly well designed. For example, we have at least three different variants of the persistent worker protocol: one for “regular” persistent workers, another for multiplex persistent workers, and yet another for multiplex persistent workers that support cancellation. And the protocol does not clearly specify which flags and tools belong to the worker and which belong to the action being executed. In fact, these distinctions are currently determined heuristically, with the worker assuming that flag files correspond to the action and other flags to the worker. There was consensus that this should be cleaned up, but the unsatisfying conclusion is that rehashing these complaints and discussions is not going to lead anywhere until someone sits down and presents a specific proposal for a new API. So, if you are reading this and have… opinions… volunteer to do this. Helen Altshuler from EngFlow and Nikita Chepanov from Plaid presented Where Bazel Clusters Break at Scale: Field Notes from Plaid and the EngFlow Fleet . That’s quite a mouthful to say “load testing”. And the reason is none other than AI once again: Plaid’s build volume doubled during the year due to AI-assisted coding and is on track to reach four times its early-2026 baseline. With a predictable increase in load comes a predictable outage. So, instead of waiting for it to happen organically, Plaid deliberately chose to load-test their CI system. The goal was to reach multiples of the normal load and discover each bottleneck under controlled conditions. They had to go through six rounds of tests and fixes until they were able to sustain their desired target. The previous five uncovered problems in network bursts, AWS quotas, source control, CI autoscaling, and cache infrastructure. The solutions were not magical: find and raise limits, cache everything possible, and reduce demand where it does not add value. Examples of the last category include skipping tests for pull requests in the middle of a PR stack and avoiding uploads for actions when they are unlikely to be reused. An interesting observation is that Plaid runs EngFlow in Plaid’s own AWS account, so the cache cluster is co-managed. But careful: shared ownership can easily turn into ambiguous ownership. The two companies addressed this by agreeing on health metrics and thresholds upfront and putting them on a shared dashboard. Both sides now know who must act before developers notice an outage. Ah, one of my favorite topics. Back in my Google days, I spent at least three years of my so-called 20% time working on improving the performance of the Bazel sandbox on macOS. Those efforts were painful: creating a writable FUSE file system is difficult, and diagnosing mis-interactions with macOS was a horrifying exercise. Today, I’m not scared anymore: I’ve been recently using AI to write yet another file system at work and agents do exceedingly well at handling corner cases. Let’s see how modern solutions do better than what I originally tried. David Zbarsky from OpenAI presented actiond: The Quest for a Better Sandbox , which was the first of at least two talks on improving sandboxing performance and strictness. The project started from frustration with local sandbox escapes on macOS and eventually grew into a custom file system, a local virtual machine, and a Bazel-built Linux kernel. actiond pretends to be a remote executor, but the “remote” worker is a Linux VM running locally on the Mac. Bazel communicates with it over a and actiond implements both execution and a content-addressable store. Funnily enough, leveraging this VM for action execution results in faster Bazel builds on macOS than running actions natively. Inside the VM, actiondFS mounts a directory described by a protocol buffer and creates the action sandbox without building a forest of links. Same idea as my original sandboxfs , but the key difference is that actiondFS is a kernel module rather than a FUSE file system so it solves all performance and deployment problems I originally faced with FUSE. A mind-blowing mention was linux.bzl : a Bazel-native build of the Linux kernel, used in this project to build the actiond VM end-to-end with Bazel. This reminds me that I’ve been saying, since the inception of Bazel, that this build system is the perfect pairing for a monorepo like NetBSD’s or FreeBSD’s; and, in fact, I recently wanted a Bazel-native build of NetBSD to put the EndBOX together. I wonder how difficult it would be to hack something up these days with AI… Oh, and: . This flag finally replaces the super-complex script that Bazel uses to run a test with a native binary. This resolves some compatibility issues across platforms and should result in measurably faster test execution. Borja Lorente from Byte Bard Software presented The Sandbox Is a Filesystem: Content-Addressed Bazel Builds on macOS with FSKit , yet another talk on sandboxing and macOS. The immediate problem they tried to address was sandbox escapes in TypeScript builds. You might not be aware, but there happen to be some tools out there that, when faced with a symlink, decide to resolve the target of the symlink and then perform extra operations on that location, like finding sibling files and the like. The compiler is one such tool, and consequently, escapes the symlink-based sandbox used on macOS and behaves incorrectly. Aspect originally patched the JavaScript rules to intercept Node.js file-system calls and redirect them, but that solution stopped working when was rewritten in Go. Apple’s new FSKit framework should in theory be usable to solve this, as it’s “just” a new way of writing file systems in user space. If we could leverage FSKit, the idea then becomes simple: represent the sandbox as a file system and mount a content-addressed view of the action’s inputs within it. Identical subtrees across actions share nodes, contents are fetched lazily, and outputs write through to the host volume. You know, exactly like my own sandboxfs did back in the day with FUSE. Unfortunately, FSKit comes with its own set of problems: An FSKit extension ships as an application, which must be packaged and notarized. This might be OK, but what’s bad is that, because this comes from an application, macOS displays a TCC (Transparency, Consent, and Control) popup notification when an app such as the Terminal tries to access the FSKit volume. Not cool for automated (CI) deployments. Mounting takes hundreds of milliseconds, making a separate mount per action too expensive. This can be solved by reusing the same mount across actions and making the file system create different subtrees per sandbox. Again, like the original sandboxfs. User space file systems have… unavoidable overheads as most operations have to cross the kernel/user space boundary at least twice. These may or may not be acceptable. With my own FUSE-based solution years ago, I just couldn’t fix some performance issues due to some oddities in OSXFUSE. AMFI and Gatekeeper inspect each executable once , adding substantial latency on first execution. And because any executable exposed through a separate file system looks brand new to the kernel… performance tanks. My original sandboxfs mitigated this by keeping inode numbers stable across different instances of the same file across different sandboxes, but that’s what ultimately led to the blockers I encountered with OSXFUSE. Because of all these problems, FSKit was not the clear choice for a new sandboxfs, so the team evaluated alternatives for performance like pure file copying, the old symbolic links, hard links, and APFS’ . FSKit was best at creating the initial directory structure, but later operations such as , , and were slower than with the alternatives. The conclusion was that there is no perfect answer to a fast and secure sandbox on macOS. Symlinks are fastest but allow some tools to escape. Hard links are the second-best option but they cannot protect source files against writes and only work within one volume. is efficient but limited to APFS and, again, one volume. FSKit provides the right abstraction with frustrating deployment and runtime costs. Because of this, the outcome of the talk is a proposal for a new sandboxfs protocol , not a specific implementation. Namely: a way for Bazel to send a manifest describing the inputs plus settings such as network access to a sandbox service. With that, the service can create the sandbox directory and return its path; Bazel can execute the action; and the service can then report the outputs and tear the sandbox down. Different backends can implement the same protocol using the best mechanism for each machine. I like the proposal and the fact that what was old is new again, but there were two interesting questions from the audience. One was: “What about an NFSv4 server?”, which is how Buildbarn exposes efficient virtual file systems on macOS; and the other was: “Can we not implement a new protocol and instead reuse the RE API?” Switching gears to the more theoretical aspects behind Bazel, we can look at changes to a core piece of the ecosystem: the Starlark language. The star (wink, wink) of the show in this area were type systems. Alexandre Rostovtsev from Google presented Static types in Starlark , an update on the type system first discussed during an unconference session at BazelCon 2025. Bazel 8.8 and 9.3 parse the new annotations but otherwise treat them as no-ops, allowing rulesets to adopt the syntax without breaking those releases. Bazel 10 will check annotated files and type-checking should be enabled by default by the time the final release ships. The syntax is a subset of Python’s type annotations with extensions to support Starlark’s native types like providers and attributes, both of which require typed exhaustive dictionaries. This should make large rulesets easier to navigate and maintain, and it lays the foundation for better automated tooling. Right after Google’s update on native Starlark types, Tamir Duberstein from OpenAI presented Type-checking Starlark without waiting for Bazel . I originally read this title as “type-checking without starting a slow Bazel process”—but no, the “waiting for Bazel” part here refers to the fact that the original type-system feature request was filed six (!!) years ago, not addressed in a timely manner, and the ecosystem could not simply stand still until the Bazel team decided to do something. During these six years, Buck 2 has come up with type annotations and a full-featured LSP for their Starlark variant. On the Bazel side, starpls implements another language server and type checker, but its annotations live in comments and the resulting experience is not ideal. OpenAI’s approach is sty , which combines ideas from with ty , the Python type checker developed by Astral. This new project provides fast cross-file checking, resolution, editor integration, and go-to-definition support while avoiding dependencies on a particular Bazel release. One difficulty is that, even if Starlark looks like Python, it isn’t Python: Bazel gives the language several domain-specific semantics. For example, a rule’s fields have types, mandatory status, and default values derived from the rule declaration. A generic Python checker knows none of this, so has to model it explicitly. By “pure” coincidence, OpenAI’s choices here do not diverge much from Google’s official typing system, so there is a path towards reconciliation. The project also introduces files, which act like external stub files and describe types for dependencies without modifying their source. These are particularly useful for rulesets that must continue to support Bazel versions that cannot parse type annotations, because the annotations can live out of sight from Bazel. The last topic I have notes on are talks about rulesets, but I do not have a lot. Let’s just cover a couple of the talks I did attend and take notes from. David Zbarsky from OpenAI presented : {Fast, Hermetic, Ergonomic} — Pick 3 , a new Rust ruleset built to address the pain points that OpenAI encountered with at scale. The usual dependency flow takes and , creates a synthetic workspace, runs , and generates a Bazel-specific . Repinning is slow—roughly O(n^2) in the size of the dependency graph—and Bazel has to repeatedly ask Cargo to contact registries and download metadata. rules_rs avoids that architecture. Bazel already has a downloader and lockfile, so the new ruleset leverages those directly and performs dependency discovery only once per platform. At OpenAI, repins dropped from about ten minutes to 100 milliseconds and can happen as part of an ordinary . Build configuration is read from , avoiding a separate Bazel lock file and the need to keep generated configuration in sync with Cargo. Rust build scripts remain a challenge because they can perform arbitrary work but appear to Bazel as one opaque action. For example, OpenAI replaced the build script in with a dedicated bzlmod module so that Bazel could see and schedule the underlying work. The ruleset also builds the Rust standard library from source, which enables Tier 3 targets and tools such as Miri . Combined with a hermetic LLVM toolchain, can cross-compile projects such as Selenium to Linux, macOS, and Windows. The question from the audience was simple: “What is the carrot for choosing over Cargo?” And the answer was simple as well: at OpenAI, they observed that everything in CI that did not use Bazel was falling over under agent-generated load. Antonio Di Stefano and Yannic Bonenberger from EngFlow presented No Bytes Left Untracked—Bazel Supply Chain Security . The key message was that, while we have always worried about correct builds and Bazel delivers on them, that’s just not enough anymore. We must also be able to explain where every byte comes from and enforce policy over their provenance in order to defend against security threats. Supply-chain attacks are becoming more frequent, and frameworks such as SLSA cover the complete software development lifecycle rather than compilation alone. They include requirements around review, provenance, and Software Bills of Materials (SBOMs). Bazel has always been in a unique position to provide this provenance information because it has a precise dependency graph and runs actions in controlled environments. But a common difficulty is that, outside of Google, Bazel interacts with other ecosystems: third-party package managers bring their own manifests and lock files and coordinating those with Bazel is difficult. There is no uniform representation of package identity, licensing, or provenance. The supply-chain ruleset addresses this with a standard target. The goal is for every package in the Bazel Central Registry to expose its identity and attributes, including licensing information, through a small JSON representation. The presenters asked for help in achieving this, as annotating packages is tedious and, ultimately, the responsibility of their owners. The processing model looks like this: A “gather” aspect walks the target graph of the artifact that needs to be shipped. Each dependency contributes metadata through a transitive provider, effectively producing a serialized view of the relevant graph. Tools convert that data into standard SBOM formats such as SPDX and CycloneDX. Validation actions enforce organization-specific policy, such as banning a particular license from entering a product. This is useful, but the target graph alone cannot fully account for all bytes in an output. True end-to-end attestation also has to inspect the action graph and the environment in which those actions run. Questions like these are to be expected: Which tools transformed the inputs? Where did the actions run? Did those environments have known kernel vulnerabilities? The project roadmap includes rules, module extensions, macro definitions, and stricter build-level enforcement. The 1.0 release was scheduled to happen during BazelCon, but unfortunately they’ll be a few days late due to last-minute issues with documentation and the like. And that’s all I captured this year. There were many other sessions—including talks on hermetic LLVM, Zig package integration, Gazelle v2, CUDA compilation, and richer platforms—but I either did not attend them or did not take enough notes to write anything of substance. Apologies if I did not cover yours here. Check the official BazelCon 2026 schedule for the complete program and stay tuned for the production-grade recordings . Photo while running towards Amsterdam Centraal. The common thread across the conference was scale. AI agents are multiplying the number of builds, tests, and changes that our infrastructure must process, while Bazel users continue to push larger graphs through increasingly-specialized execution environments. The good news is that the ecosystem is responding: Bazel 10 brings meaningful core improvements, remote execution keeps getting smarter, and the BUILD Foundation gives community work a place to grow. But even if the ecosystem is responding, you must too. Bazel is positioned to be a great answer to scalability issues in development workflows thanks to its safe cross-machine, cross-user caching features. And Bazel is also a good way to remain secure amid the unprecedented number of changes to your third-party dependencies. If you have not yet considered Bazel, the best time to do so is today. Stay tuned for the next article! I got some really cool swag from one of the vendors and I promised them I’d write something up once I had a chance to evaluate it. Welcome to BazelCon 2026 sign. What follows is a recollection of my notes arranged by topic and intermixed with my own commentary and thoughts. Some sections and topics are more detailed than others because in some cases I didn’t have a seat and couldn’t type enough, and in others I got distracted in the hallways chatting with people. Speaking of people, several of you approached me to praise previous editions of these recaps, so I now feel obliged to write another one. Thanks for the kind comments! As we will see, “scale” was a central theme of the conference: AI is multiplying concurrent builds and tests, artifacts and graphs are growing at an accelerated pace due to new features like configuration transitions, and the ecosystem is responding by making execution, caching, and analysis more efficient. My highlights There is a lot to cover but here are a few things that felt memorable to me : The BUILD Foundation is now a reality, giving the Bazel ecosystem a vendor-neutral home. Its goal is to act as a proxy between the community and Google to help Google understand what to prioritize and to fund projects. If your company can spare “a few” dollars, please donate! AI-driven development is having a broad impact on the industry and, in particular, everyone is seeing their CI systems become overloaded. Well, news at 11: I’m dealing with this day in and day out. But what was new to me was that we at Snowflake are not the only ones struggling with local system overload. Lots of people out there have likewise “discovered” Git worktrees as a way to run multiple agents in parallel, which in turn means multiple concurrent Bazel servers. Bazel 10 is shaping up to be a substantial release that will help with the problem right above due to its many performance improvements and memory consumption optimizations. Have you struggled to keep Bazel up-to-date before? These days, if you want to get any platform-related work funded you have to spin it in terms of AI, so this release will be an easy sell. Let’s “Rewrite it in Rust”? I’m impressed. Google is seriously evaluating a progressive rewrite of Bazel in Rust, and this is probably the most exciting news I got out of the conference. You know I’ve been critical of Bazel’s usage of Java since day one so, if this takes off, I’ll be the first in line to evaluate the results (and it will likely push me to actually contribute to Bazel again). Fast and correct sandboxing on macOS remains a problem. This is “interesting” to me because it’s a problem I tried to fix almost ten years ago with the original sandboxfs, but I failed because of unexpected bugs in macOS’s FUSE support and eventually abandoned the idea when FUSE became a dead end on Apple’s platforms. Funnily enough, other folks have now reinvented sandboxfs—and even given it the same name (ugh)—and there is even another implementation of a similar idea in the form of actiondFS. Types are coming to Starlark, but Bazel’s late arrival to the party means that multiple competing approaches now exist. We have at least two LSPs with different annotation mechanisms and Buck 2’s own syntax. Hopefully they can all converge. Platforms and configurations remain difficult to model at scale. Careless use of transitions and build settings can produce an explosion of configured actions. Apparently, as one speaker noted, AI agents love transitions for some reason, which is interesting because they are not a popular feature and they are not particularly easy to use. Agents run more builds more frequently than humans, discover and invoke targets that were often “internal only”, and explore several approaches in parallel. Your intuition about how to evaluate which builds matter and how to measure them is probably wrong by now. Bazel and IDEs have never played great together and maybe that just doesn’t matter anymore. We are seeing a sharp decline in IDE usage, although developers still want a way to browse code changes and debug them. This means LSPs are key now to support “leaner” IDEs, and maybe standalone debuggers. Short build and test latency is critical when a human must remain focused on a coding task but, in an agentic workflow, the constraints are different. You know that, after you hit “execute this plan”, the agent will take many minutes to return so minimal build latency is not as important. And actually… maybe you now want to trade some latency in exchange for extra validations so that agents have to take fewer turns to converge. And because humans are now waiting for agents to finish for long periods of time, they have to fill that time with other tasks, which has driven them to adopt Git worktrees en masse. This is leading to scalability problems on dev machines because multiple concurrent Bazel server processes are not great for memory and disk usage. to download filtered logs for a build. to answer questions such as “Why was my build slow?” which are extremely common in the Bazel ecosystem. to start a remote Firecracker VM and run Bazel inside of it. The VMs are snapshotted for quick reuse, and the runners can execute arbitrary commands rather than Bazel alone. to diagnose a failed build and either create a pull request or push a commit with the proposed correction. to perform a review and post comments to GitHub. Correctness: Good, because equivalence is the central constraint of the loop. Hermeticity: Good, but you might need to do extra upfront work to remove host-specific behavior from the reference build before starting. Efficiency: Usually good when standard rules can model the build. Complex projects with custom build rules may end up with overly large Bazel actions though. Maintainability: Unsurprisingly, the weakest point. Agents tend to model variation with configuration flags instead of proper platforms and toolchains, which is not idiomatic to Bazel. Expert review of AI outputs remains critical, which is, again, not surprising. Reset transitions: These discard irrelevant configuration values as early as possible in the graph. The art is to move decisions up the dependency tree so that downstream targets do not carry unnecessary dimensions. The trade-off is that ordinary output can become unmanageable, forcing users toward . But then… requires a configuration, so some targets may not be queryable at all without one. Path mapping: If two actions cannot be made identical during analysis, Bazel can rewrite configuration-specific paths during execution and replace their hashes with a stable component. This should be seen as a measure of last resort, though, because subtle problems arise. For example, if the same logical path comes from different files from different configurations, Bazel disables path mapping silently. Tweag fixed Bazel so that different copies of the same file don’t disable path mapping, but debuggability continues to be a problem when this condition does not apply. Flag scopes: This new feature allows a flag to declare where in the graph it applies and to automatically reset to its default everywhere else. This is similar to a reset transition but requires less manual graph plumbing. On the other hand, describing the correct scope can sometimes be harder than writing the transition directly. Pre-warming the next JVM instead of keeping the current one alive. This provided more control over when and why to restart a test in a new JVM, but suffered from significant coordination overhead. Forking a JVM. This did not work because, due to POSIX semantics, only the calling thread survives a fork call. The JVM is heavily multithreaded, so if you end up losing all but one thread, the new JVM is just not going to work. Android’s Zygote. This provides a pre-warmed runtime ready to be forked ( , anyone?), but it exists only for ART, not the standard JVM, and it’s just not reasonable to expect a switch to ART. Checkpointing. Instead of forking a running process, why not snapshot it to disk so that it can be re-created again? This might have worked, like Horapha does for Bazel startup, but raised difficult questions about when to capture the process and how to handle state outside the JVM. Unconference discussion topic proposals and votes for the second day. Remote execution BoF Chi Wang and Tiago Quelhas led the Remote Execution Birds of a Feather . I only stayed for half of it, and so I only captured the discussion around remote persistent workers. Persistent workers are already part of several commercial remote execution offerings, including BuildBuddy and EngFlow, but their implementation was derived from Bazel’s local persistent worker support, which is not… particularly well designed. For example, we have at least three different variants of the persistent worker protocol: one for “regular” persistent workers, another for multiplex persistent workers, and yet another for multiplex persistent workers that support cancellation. And the protocol does not clearly specify which flags and tools belong to the worker and which belong to the action being executed. In fact, these distinctions are currently determined heuristically, with the worker assuming that flag files correspond to the action and other flags to the worker. There was consensus that this should be cleaned up, but the unsatisfying conclusion is that rehashing these complaints and discussions is not going to lead anywhere until someone sits down and presents a specific proposal for a new API. So, if you are reading this and have… opinions… volunteer to do this. Where Bazel clusters break at scale Helen Altshuler from EngFlow and Nikita Chepanov from Plaid presented Where Bazel Clusters Break at Scale: Field Notes from Plaid and the EngFlow Fleet . That’s quite a mouthful to say “load testing”. And the reason is none other than AI once again: Plaid’s build volume doubled during the year due to AI-assisted coding and is on track to reach four times its early-2026 baseline. With a predictable increase in load comes a predictable outage. So, instead of waiting for it to happen organically, Plaid deliberately chose to load-test their CI system. The goal was to reach multiples of the normal load and discover each bottleneck under controlled conditions. They had to go through six rounds of tests and fixes until they were able to sustain their desired target. The previous five uncovered problems in network bursts, AWS quotas, source control, CI autoscaling, and cache infrastructure. The solutions were not magical: find and raise limits, cache everything possible, and reduce demand where it does not add value. Examples of the last category include skipping tests for pull requests in the middle of a PR stack and avoiding uploads for actions when they are unlikely to be reused. An interesting observation is that Plaid runs EngFlow in Plaid’s own AWS account, so the cache cluster is co-managed. But careful: shared ownership can easily turn into ambiguous ownership. The two companies addressed this by agreeing on health metrics and thresholds upfront and putting them on a shared dashboard. Both sides now know who must act before developers notice an outage. Sandboxing and local execution Ah, one of my favorite topics. Back in my Google days, I spent at least three years of my so-called 20% time working on improving the performance of the Bazel sandbox on macOS. Those efforts were painful: creating a writable FUSE file system is difficult, and diagnosing mis-interactions with macOS was a horrifying exercise. Today, I’m not scared anymore: I’ve been recently using AI to write yet another file system at work and agents do exceedingly well at handling corner cases. Let’s see how modern solutions do better than what I originally tried. actiond: the quest for a better sandbox David Zbarsky from OpenAI presented actiond: The Quest for a Better Sandbox , which was the first of at least two talks on improving sandboxing performance and strictness. The project started from frustration with local sandbox escapes on macOS and eventually grew into a custom file system, a local virtual machine, and a Bazel-built Linux kernel. actiond pretends to be a remote executor, but the “remote” worker is a Linux VM running locally on the Mac. Bazel communicates with it over a and actiond implements both execution and a content-addressable store. Funnily enough, leveraging this VM for action execution results in faster Bazel builds on macOS than running actions natively. Inside the VM, actiondFS mounts a directory described by a protocol buffer and creates the action sandbox without building a forest of links. Same idea as my original sandboxfs , but the key difference is that actiondFS is a kernel module rather than a FUSE file system so it solves all performance and deployment problems I originally faced with FUSE. A mind-blowing mention was linux.bzl : a Bazel-native build of the Linux kernel, used in this project to build the actiond VM end-to-end with Bazel. This reminds me that I’ve been saying, since the inception of Bazel, that this build system is the perfect pairing for a monorepo like NetBSD’s or FreeBSD’s; and, in fact, I recently wanted a Bazel-native build of NetBSD to put the EndBOX together. I wonder how difficult it would be to hack something up these days with AI… Oh, and: . This flag finally replaces the super-complex script that Bazel uses to run a test with a native binary. This resolves some compatibility issues across platforms and should result in measurably faster test execution. The sandbox is a filesystem Borja Lorente from Byte Bard Software presented The Sandbox Is a Filesystem: Content-Addressed Bazel Builds on macOS with FSKit , yet another talk on sandboxing and macOS. The immediate problem they tried to address was sandbox escapes in TypeScript builds. You might not be aware, but there happen to be some tools out there that, when faced with a symlink, decide to resolve the target of the symlink and then perform extra operations on that location, like finding sibling files and the like. The compiler is one such tool, and consequently, escapes the symlink-based sandbox used on macOS and behaves incorrectly. Aspect originally patched the JavaScript rules to intercept Node.js file-system calls and redirect them, but that solution stopped working when was rewritten in Go. Apple’s new FSKit framework should in theory be usable to solve this, as it’s “just” a new way of writing file systems in user space. If we could leverage FSKit, the idea then becomes simple: represent the sandbox as a file system and mount a content-addressed view of the action’s inputs within it. Identical subtrees across actions share nodes, contents are fetched lazily, and outputs write through to the host volume. You know, exactly like my own sandboxfs did back in the day with FUSE. Unfortunately, FSKit comes with its own set of problems: An FSKit extension ships as an application, which must be packaged and notarized. This might be OK, but what’s bad is that, because this comes from an application, macOS displays a TCC (Transparency, Consent, and Control) popup notification when an app such as the Terminal tries to access the FSKit volume. Not cool for automated (CI) deployments. Mounting takes hundreds of milliseconds, making a separate mount per action too expensive. This can be solved by reusing the same mount across actions and making the file system create different subtrees per sandbox. Again, like the original sandboxfs. User space file systems have… unavoidable overheads as most operations have to cross the kernel/user space boundary at least twice. These may or may not be acceptable. With my own FUSE-based solution years ago, I just couldn’t fix some performance issues due to some oddities in OSXFUSE. AMFI and Gatekeeper inspect each executable once , adding substantial latency on first execution. And because any executable exposed through a separate file system looks brand new to the kernel… performance tanks. My original sandboxfs mitigated this by keeping inode numbers stable across different instances of the same file across different sandboxes, but that’s what ultimately led to the blockers I encountered with OSXFUSE. A “gather” aspect walks the target graph of the artifact that needs to be shipped. Each dependency contributes metadata through a transitive provider, effectively producing a serialized view of the relevant graph. Tools convert that data into standard SBOM formats such as SPDX and CycloneDX. Validation actions enforce organization-specific policy, such as banning a particular license from entering a product.
Culture
Books
Business
DevOps
Ci Cd
AI
Go
Java
Security
Php
Web Development
Design
Hardware
Performance
Frontend
Career
Programming
Entertainment
Media
Data
Testing
Agile
HTML
JavaScript
Shell
TypeScript
Music
JSON
Rust
Tutorial
Open Source
Ruby