GreatReads - Blog Aggregator · Phoenix Framework

0 views

Anton Zhiyanov 6 days ago

Go-flavored concurrency in C

Go's concurrency is one of the main reasons people like the language. You write , send values through channels, and the runtime scheduler runs thousands of goroutines on just a few OS threads. It feels effortless. None of that machinery exists in C. Which made me wonder: how close can you get to Go's concurrency model using only POSIX threads? Obviously, native OS threads can't match the efficiency of lightweight goroutines, but what is the actual cost, when does it become a problem, and is there any way to at least partially avoid it? I ran into these questions while adding concurrency to Solod (So), a strict subset of Go that translates to plain C, with no runtime and no garbage collector. In the end, I came to the conclusion that you can do quite a lot with pthreads — as long as you're honest about the tradeoffs. This post is about the POSIX threads-based concurrency model I chose, the benefits it offers, and its limitations. Mutex/Cond • Atomics • Pool • Channel • Performance • Design • Wrapping up Everything in So's concurrency stack is built on two basic POSIX primitives: the mutex and the condition variable. is a thin wrapper around : Since So translates to C, this is basically a struct that holds a and a function that calls . Here's the transpiler output: That is the whole translation — the generated C is a near-mechanical mirror of the So code, only noisier. From here on, I'll mainly show the So version, but I'll also provide the C code for those who are interested. There's nothing exciting here: is a pthread mutex wrapper that panics if something goes wrong (which is rare). The companion primitive is , a wrapper around . It's the standard "wait until a condition holds" tool, associated with a mutex: These two types — and — are the foundation. Other concurrency tools — , the thread pool, channels — are built using a mutex and one or more condition variables. This has several effects on performance, as we'll see later. Not everything needs a lock. So's mirrors Go's: , , , , , and a generic , all with , , , and methods. The nice thing is that these don't need pthreads at all. They map directly to the C compiler's builtins — the same hardware instructions that Go's compiler emits. So there's no reason for them to be any slower, and they're not: Each number is the cost of one operation on a single thread. is a good example of using atomics effectively. Its fast path only needs a single atomic load — after the given function runs, every future call to checks a flag and returns: To actually run code concurrently, you need threads. The type wraps and its related functions: Consider this function: Usage example: It might look like , but that's just on the surface. starts an actual OS thread, not a goroutine. You have to eventually call to join or it, or else its resources will leak. Also, OS threads are expensive to create — they're nothing like Go's goroutines, which only need a few kilobytes of stack and start up in nanoseconds. That's exactly why you usually don't want to call inside a loop. For tasks that are short-lived or happen often, it's better to use a pool of long-lived worker threads and send tasks to them. to the rescue: Usage example: The first argument to , , is a memory allocator. Solod avoids hidden allocations, so anything that needs memory takes an allocator explicitly — here it backs the pool's task queue. Under the hood, a is a fixed group of worker threads that pull tasks from a shared queue (a ring buffer). It uses one mutex and a few condition variables: wakes up a worker when there are tasks to do, applies back-pressure when the queue is full, and lets know when everything is finished. It's a classic producer-consumer setup, about 200 lines of code , and there's nothing fancy about it. The heart of the pool is the worker loop. Each thread blocks until a task appears, runs it outside the lock so workers execute in parallel, then records that it finished: This is what separates a pool from a plain queue. bumps as it enqueues; each worker decrements it after running a task, and the last one out broadcasts . sleeps until the count hits zero: The tradeoff is that the number of worker threads is fixed. In Go, a program can handle thousands of concurrent I/O waits because blocked goroutines use very little memory. A So pool can't do this — if all N workers are parked on a blocking syscall, the pool is stalled until one returns. You have to set the pool size based on the workload, instead of letting the runtime manage it for you. Channels are an important part of Go's concurrency model, and So's gives you something quite similar. Just like in Go, it passes values by copy and comes in buffered and unbuffered flavors: is a thin generic shell over one of two engines, picked at creation time: Buffered ( ) is a mutex-guarded ring buffer with and condition variables — like the queue. Senders block when it's full, receivers block when it's empty. The full implementation also checks for , but I left it out for brevity. is the mirror method: block while empty, pop the next value, signal to wake a sender. It also handles the closed channel, returning once the buffer is closed and drained. The rest is this lock-wait-signal core. Buffer source code Unbuffered ( ) is a rendezvous: each send blocks until a receiver takes the value, copying bytes directly from the sender's stack to the receiver's destination without using an intermediate buffer. is the other half: it waits for a published, unclaimed value, copies bytes straight from the sender's stack into (no intermediate buffer), marks it as claimed, and broadcasts to wake the sender back, creating wakeup #2. One hand-off, two wakeups. Copying directly from the sender's stack is safe because of that second wakeup. is a pointer to , which lives on the sender's stack. While the receiver is reading it, the sender is parked in , so its stack frame stays alive. The sender only returns (and reclaims that memory) after the receiver sets and wakes it up. There's no need to copy into a shared buffer because the source is guaranteed to outlive the read. Rendezvous source code As you can see, the API is pretty similar to Go. Now let's look at the numbers. Here's the main tradeoff: pthread-based concurrency primitives are fast when no one has to block, but they get slow when someone does. And it's always for the same reason. Go schedules goroutines in userspace. When one goroutine blocks on a channel and another wakes it up, the runtime moves them between its own queues — no kernel involved. POSIX threads, on the other hand, don't provide a userland scheduler. When a thread blocks on a condition variable, it parks in the kernel, and waking it up requires a syscall. Every hand-off between threads that actually parks pays the cost of a syscall on both ends. You can clearly see the difference in the mutex benchmarks. With 8 competing threads, it all comes down to whether the waiting threads have to park or not: Each number is the average time for a single / pair. The uncontended benchmark runs on one thread, while the contended benchmarks have multiple threads fighting over the same mutex. Notice that So actually wins the first two benchmarks, and for good reason. So's is a plain call with nothing extra, while Go's adds more overhead — like starvation-mode tracking and a runtime that stays involved because a goroutine can be preempted in the middle of a critical section. When nobody parks, that overhead is the main cost, and the thinner wrapper is closer to the hardware. With an empty critical section (the spin benchmark), a waiting thread grabs the lock while still spinning and almost never parks — So wins by 2.8x. The uncontended benchmark (a single thread, no contention) shows the same thing: less code between the call and the lock, so 9ns versus 14ns. The picture flips the moment threads have to park. Give the critical section about a microsecond of real work (the work benchmark) and waiters exhaust their spin budget and park. Now every hand-off costs a wakeup syscall, and So drops to half of Go's throughput. The work is identical in both cases — the difference comes from the parking cost. Condition variables demonstrate this clearly because they always park: Each number is the cost of one rendezvous round: a single broadcast that wakes every waiter and hands control back, with N waiters plus one broadcaster. Pthread-based condition variable is consistently 7-10 times slower. There's no trick to close this gap — it's just the cost of waking up a real OS thread instead of a goroutine. Channels have the same issue because they're built using mutexes and condition variables: Each number is the cost of moving one value through the channel (send plus its matching receive). The number in parentheses is the buffer capacity. The uncontended case fills and drains a buffer from a single thread, so nothing ever blocks — it's just a lock plus a copy, which gives So a slight advantage. But the moment a producer and consumer actually start handing off work, So has to wake up a thread for every transfer that gets parked. It's worst for the unbuffered channel, where every value is a rendezvous with two wakeups: 23x slower. A larger buffer helps a lot — with room for 100 items, most sends go through without waking anyone, and the gap narrows to about 2x. The consequence is that the larger your tasks are, the better pthread-based concurrency works. If you use a channel for fine-grained, value-at-a-time streaming between threads, performance will suffer. But if you use a channel to pass whole work items to a pool, where each item takes tens of microseconds to process, the wakeup cost becomes negligible. The pool benchmarks on realistic workloads confirms this: Each number is the wall-clock time for 8 workers to process the whole batch. Here, So is within 1.1x of Go. The per-task dispatch cost is still present, but it's spread out over real work, and the performance penalty is pretty small. Benchmarking All benchmarks were run on an Apple M1 CPU running macOS. The C code was compiled with Clang 16 using these CFLAGS and mimalloc as the system allocator: The results shown are the medians from several benchmark runs. Each benchmark ran many iterations, following the same logic as Go's own benchmarking. The Go benchmarks used Go 1.26 and . Source code for both So's and Go's benchmarks: conc • sync Here's a summary of the strengths and weaknesses of the pthread-based approach: If you're looking for "thousands of cheap goroutines", the pthread-based approach will let you down. But if you're fine with "a few worker threads handling lots of tasks", it holds up well. Three decisions influenced the way I implemented concurrency in Solod. Pthreads, not fibers . I know there are coroutine/fiber libraries for C that avoid the kernel wakeup cost — single-threaded ones like neco , and multi-threaded ones like libfiber . A userspace scheduler is exactly what would help to match Go in the benchmarks above. I decided not to use one. I wanted something dead simple — an approach I could explain in a paragraph, using tools every C programmer already knows. The trade-off is that you lose some performance with fine-grained blocking, but in many real-world situations, pthreads work fine if you use a worker pool. For me, keeping things simple is more important than saving a few microseconds during task hand-offs. For now, at least. Standard library, not language . Go bakes goroutines, channels, and select right into the language. I decided to keep everything in the stdlib for two reasons. ➀ It follows So's "no hidden allocations" rule. In Go, quietly allocates a goroutine stack, and allocates a buffer. In So, all allocations are explicit: you pass an allocator to and , and you always know exactly where the memory comes from — whether it's the system allocator, an arena, or something else. ➁ A library is more flexible. Since a pool is a regular value, you can have as many as you need, each sized for its specific purpose. In a multi-stage pipeline where each stage needs a different capacity, you can start one pool per stage, each with its own and , instead of being given a single global scheduler. The language stays simple, and the flexibility is in code you can easily read. Timeouts, not select . Go's waits on several channel operations at once and proceeds with whichever is ready first. Implementing it would require a lot of work — a thread has to register interest on multiple channels, block once, and then wake up when any of them is ready — so I left it out. Instead, offers and , which cover two common uses of with a single channel: What's missing is the ability to block on multiple channels at once and continue with whichever one is ready first, as well as the option to mix sends and receives in the same selection. How close can you get to Go's concurrency using only pthreads? Close enough to be useful, but not enough to really match Go. You can wrap real OS threads with familiar APIs — mutexes, condition variables, pools, channels — and the code will look and act a lot like Go, at least until a thread needs to block. But there's no scheduler underneath, so when a thread blocks, it's an actual thread waiting in the kernel, not a goroutine that's paused for free. That's the main limitation of this approach. What you get in return is brutal simplicity. Every primitive is a thin wrapper with no runtime hiding behind it, so the performance is exactly what the OS gives you: fast atomics, fast uncontended locks, and pooled throughput within ~10% of Go on coarse-grained work. But as soon as you switch to fine-grained, one-value-at-a-time hand-offs, the cost of kernel wakeups becomes the main factor, and you'll notice the slowdown. If you think the pthread approach might work for you, I invite you to try Solod . It includes the and packages, along with many others ported from Go's standard library. ➕ Coarse-grained pooled workloads are within about 10% of Go's performance. ➕ Uncontended locks and spin-friendly critical sections perform quite well. ➕ Atomic operations are as fast as in Go. ➕ The implementation is 100x simpler. ➖ Anything that needs to park and wake an OS thread is much slower than Go's userspace scheduler. ➖ The pool can't handle thousands of blocked waiters like goroutines can. "Do this, but give up after a while" (Go's idiom). "Do this only if it won't block" (Go's non-blocking branch).

Performance

0 views

Blog System/5 1 weeks ago

Autoconf’s revenge: ad-hoc shell templates

As powerful as Bazel is, sometimes it’s not featureful enough. When using this build system, it’s common practice to wrap it in a launcher script—and in fact, this is natively supported by Bazelisk , Bazel’s native dispatcher that stands for the binary in the user’s . Bazelisk will first download the version of Bazel requested by the project, and then, if exists, invoke it instead of the downloaded binary. is what’s known as a Bazel wrapper and is the point of today’s article. Well, not quite. The actual point of today’s article is to demonstrate a simple trick I learned from the GNU Autoconf and Automake days to implement full-blown conditionals in an ad-hoc template system. But because such trick is trivial once you see it, I have to present it in the context of a modern real-world scenario. So what I’m going to do is guide you through the creation of your very own Bazel wrapper to customize Bazel’s configuration file in ways that the native Bazel tool doesn’t support. Let’s get started. But wait! Take a moment to subscribe. I’m sure you’ll enjoy future posts, and it’s the only way for me to know that they are worth writing in the first place! Template systems are everywhere. Take any static blog generation system and you’ll find some. Take system management tools like Ansible and you’ll find others. Take a cloud orchestration service like Kubernetes and you’ll find Helm. Heck, even Go’s standard library provides a full blown text template system out of the box. There is clear benefit and appetite for these and, surely enough, it’s tempting to use any pre-existing such system in your own project… but if all you need are a bunch of variable replacements, some of which may be only conditionally applied, you can go a long way by not taking any dependencies. A call to or the substring function of your language of choice is all you need. To put this in context, let’s say you have Bazel’s configuration file, which is not very flexible, and that you need to set some arguments based on dynamic values that depend on the environment. E.g. something like this: If you know a little bit about , however, you may squint at that and say: “That’s silly! Make those flags conditional on a configuration and you’re set!” So you try something like this: And… this does not work. Gotcha! Startup flags cannot be placed behind a configuration so there is no way for you to parameterize the JVM’s max heap value passed in . And having to remember to pass from CI all the time is fragile, because you might forget and not get the desired configuration in place. Solving the above is not difficult if we could parameterize the configuration. We might want to write something like this instead: … and then have and be replaced dynamically depending on some runtime arbitrary logic. We can do that via the Bazel wrapper, and this sort of dynamic configuration is a common thing to do from it. So let’s do this. Let’s start with the template logic: Ugly(?) bash syntax but nothing too complicated: The global hashmap tracks variable names and their replacement values. The function inserts a new key/value pair into . (It’s important that the values given to don’t contain -special characters like , backslashes, or the separator we chose—but we control the generation of those values so we are good.) The function transforms the hashmap into a set of arguments of the form and then calls to process the given input file into the given output file. Then, we can plug everything together into a minimal Bazel wrapper: In this new snippet, the function instantiates the file from the contents of via and then calls the actual Bazel binary provided by Bazelisk in . The complexity here may seem overkill, but it’s necessary : while it’s pointless to invoke Bazel in parallel due to its global lock, users will run Bazel in parallel and you must make sure that the wrapper is reentrant. Otherwise, you’ll definitely run into races. The rest of the script in does the actual work to compute key/value pairs to substitute in your now-templated and then delegates to Bazel via . That’s it. This is a barebones implementation of a text template system using bash—and I had to use bash, not sh, to get the niceties of a hashmap —that serves as a launcher. Go try it. By the way, the and nomenclature are inherited from GNU Autoconf’s AC_SUBST primitive . “Great!” I hear you say in a sarcastic tone. “You have just applied string replacements! But what about conditionals, huh? You CaNnOt Do ThAt So EaSiLy!!11!one!” Ah, but you can , and showing you that trick is the whole point of this short article, remember? The necessary insight is that we can use string replacements to comment out lines in the original file. What if we did this: In here, we are defining different configurations for developer workstations and for CI, like we did earlier, but then we are auto-magically picking the default configuration depending on and . How? Well: will expand to the empty string when running on CI and will expand to , so the corresponding lines will be enabled and disabled. And the opposite replacement values will appear when not running on CI. Ta-da! Conditionals. We can make things nicer with a helper function and meta-programming: Don’t panic about that . Just as with the invocation above where we could have issues with special characters appearing in values, we control the arguments to so the is safe. And note that we can even nest conditionals arbitrarily. There is nothing preventing you from doing: Which corresponds to the conceptual equivalent of: Let’s do loops? Sorry no, can’t do! Well akshually… we could do loops. Not by using simple tricks like above, but we could definitely sketch something like this: However, this is starting to look a lot like a high-level parser, not scripting where you glue simpler components together. And if you are headed that way, you are better off transitioning to a proper programming language and a well-known template system. What do you think? Do you hate this already? You can, but note that the whole world runs on this stuff. All of that foundational code behind Linux systems ends up using GNU Automake and GNU Autoconf, and those packages are full of stuff like this in their and files. And you can get very far with just the above constructs if you treat the shell like a real language . The Bazel wrapper that I maintain at work these days grew to almost 1000 lines of code before I pruned a lot of features that had become unnecessary, but it’s still pretty large. We are now transitioning it to a Go-based wrapper for better readability and maintainability… but as we do this, I’m reminded that well-groomed shell scripts give you some flexibility that no other language can match in just a few lines. So, keep things simple. You can do a lot with just a few primitives. As powerful as Bazel is, sometimes it’s not featureful enough. When using this build system, it’s common practice to wrap it in a launcher script—and in fact, this is natively supported by Bazelisk , Bazel’s native dispatcher that stands for the binary in the user’s . Bazelisk will first download the version of Bazel requested by the project, and then, if exists, invoke it instead of the downloaded binary. is what’s known as a Bazel wrapper and is the point of today’s article. Well, not quite. The actual point of today’s article is to demonstrate a simple trick I learned from the GNU Autoconf and Automake days to implement full-blown conditionals in an ad-hoc template system. But because such trick is trivial once you see it, I have to present it in the context of a modern real-world scenario. So what I’m going to do is guide you through the creation of your very own Bazel wrapper to customize Bazel’s configuration file in ways that the native Bazel tool doesn’t support. Let’s get started. But wait! Take a moment to subscribe. I’m sure you’ll enjoy future posts, and it’s the only way for me to know that they are worth writing in the first place! The context Template systems are everywhere. Take any static blog generation system and you’ll find some. Take system management tools like Ansible and you’ll find others. Take a cloud orchestration service like Kubernetes and you’ll find Helm. Heck, even Go’s standard library provides a full blown text template system out of the box. There is clear benefit and appetite for these and, surely enough, it’s tempting to use any pre-existing such system in your own project… but if all you need are a bunch of variable replacements, some of which may be only conditionally applied, you can go a long way by not taking any dependencies. A call to or the substring function of your language of choice is all you need. To put this in context, let’s say you have Bazel’s configuration file, which is not very flexible, and that you need to set some arguments based on dynamic values that depend on the environment. E.g. something like this: If you know a little bit about , however, you may squint at that and say: “That’s silly! Make those flags conditional on a configuration and you’re set!” So you try something like this: And… this does not work. Gotcha! Startup flags cannot be placed behind a configuration so there is no way for you to parameterize the JVM’s max heap value passed in . And having to remember to pass from CI all the time is fragile, because you might forget and not get the desired configuration in place. Basic string replacements Solving the above is not difficult if we could parameterize the configuration. We might want to write something like this instead: … and then have and be replaced dynamically depending on some runtime arbitrary logic. We can do that via the Bazel wrapper, and this sort of dynamic configuration is a common thing to do from it. So let’s do this. Let’s start with the template logic: Ugly(?) bash syntax but nothing too complicated: The global hashmap tracks variable names and their replacement values. The function inserts a new key/value pair into . (It’s important that the values given to don’t contain -special characters like , backslashes, or the separator we chose—but we control the generation of those values so we are good.) The function transforms the hashmap into a set of arguments of the form and then calls to process the given input file into the given output file.

Shell Bash

DevOps

5 views

Unsung 2 weeks ago

“That’s a big number – by almost any scale other than Google’s.”

Thirteen years ago today, Google killed Google Reader. In 2023, The Verge wrote a great piece about the shutdown : Google’s feed-reading tool offered a powerful way to curate and read the internet and was beloved by its users. Reader launched in 2005, right as the blogging era went mainstream; it made a suddenly huge and sprawling web feel small and accessible and helped a generation of news obsessives and super-commenters feel like they weren’t missing anything. It wasn’t Google’s most popular app, not by a long shot, but it was one of its most beloved. In the essay, Google Reader is presented as a victim of Google+. I was at Google when Google+ was announced and can corroborate the feeling of an end of an era at the company. The first large internal presentation was a shell shock: the arrival of secrecy, bureaucracy, corporate delusion, inevitable sycophants following not-so-inevitable bozos. But perhaps it was the opposite – Google as a company would have changed anyway, and Reader just randomly ended up being among the early beloved things that stood in the way. (I mean, arguably, Google changing for the worse destroyed even Google Search since.) I am worried about the open web , but excited seeing some resurgence in RSS usage, and more and more people wanting to come back to the feeling of control, care, and intentionality that using Reader represented. Just a few months ago, Roger Wong found himself reflecting on Reader, too : What gets me is that the vision Wetherell drew on that whiteboard—a single place to follow everything you care about, organized by your taste, shared with people you trust, and non-algorithmic—still doesn’t fully exist. RSS readers are the closest thing we have, and they’re good enough that I’ve built my entire reading and writing practice around one. But the curation layer Wetherell imagined is still unfinished. I’m introducing a new tag to Unsung, software eulogies , which right now encompasses Aperture and Reader. One has to be careful about nostalgia since it has its own gravity and can corrupt as much as a runaway World of WarCraft virus . “They don’t make them like they used to” is a potent drug that can make us disinvested in shaping the future, but it is also true that, well, we don’t make software like we used to. Part of Unsung is about finding inspiration in history, and while each one of us can miss a certain era of computing, certain machines, and certain software for whatever reasons we choose to – healthy or not – I do believe we collectively miss Aperture and Reader for the right reasons that are worth listening to. #google #software eulogies #software evolution #web

Web Development

Culture

Media

0 views

Ahead of AI 2 weeks ago

Using Local Coding Agents

Many people reached out to me in the past asking about my local agent stack as well as how I set up my local agent stack. So, I thought it might be useful to put together a little tutorial on how to set up a local (coding) agent using open-source tools and open-weight LLMs. Figure 1: Overview of the local stack, that is, a coding agent harness that uses a local model hosted through an inference engine / runtime server. This article is a tutorial on setting up a production-ready coding agent with a fully local stack. We will use a locally served LLM together with a local coding harness that can read files, make edits, run commands, and verify changes as shown in the figure above. Here, we can think of the LLM as the engine that provides the reasoning and code generation. And the surrounding harness provides the operating environment that allows the LLM to do meaningful coding work in our local projects. Why local? For many coding workflows, a local setup is an interesting alternative to proprietary services such as GPT in Codex or Opus in Claude Code. The local setup is transparent, inspectable, and free to run apart from hardware and electricity costs. It also stays fully under your control, and you can modify the coding harness in any way you like. Plus, it’s a lot of fun! By the way, in case you want a bit more background information on coding agent harnesses, I covered the core components of coding agents (and building a coding agent from scratch for learning purposes) here: I have to admit that I still primarily alternate between Codex and Claude Code as my daily drivers, for now (and just to keep up with the new tooling and functions that are constantly being added). Also, the plan limits (especially for Codex) are still so generous that I haven’t had to worry about costs so far. However, I’ve been using local solutions for a while, too, to test things and because it somehow gives me joy to have and use a fully local setup (versus proprietary services). Either way, local solutions become more and more attractive each day. One aspect is the costs. If you have the hardware, they are practically free to run. And then there’s, of course, the privacy angle. For example, for organizing and processing my receipts, I’d be more comfortable with a local model ingesting them rather than sending the data over to OpenAI or Anthropic. (Then, if we keep in mind that Anthropic was recently throttling their flagship model’s performance for LLM research , proprietary services may become more restrictive over time, and it’s maybe a good idea to be comfortable with open-weight alternatives as a backup.) And there are many, many additional reasons and use cases like that. Your motivations for using local LLMs and coding harnesses may include: Predictable, fixed costs if you reach your subscription plan limits, and immunity to API price changes. Reproducibility; sometimes it’s nice if a model is upgraded (e.g., GPT 5.4 -> GPT 5.5 -> GPT 5.6) and it solves all your queries more reliably. However, this can also break existing workflows. Offline use in the classic airplane flight scenario with slow or no internet, or when going on a coding/writing retreat in the cabin in the woods w/o a Starlink subscription. And there are probably several others. So, in this article, we will set up and use popular harnesses like Codex and Claude Code with open-weight models and investigate whether using a model-specific harness (like Qwen-Code for Qwen3.6) brings any additional benefits. (Of course, there are many more harnesses like OpenCode, Cline, Pi, and Noumena Code, but I thought that most people already have muscle memory with either Codex or Claude Code, which makes switching to open-weight models a bit smoother). Most coding agent harnesses follow similar principles and have more or less the same features and functionality. However, the implementation details may differ, and certain LLMs have usually been primarily optimized for a specific harness. Of course, many open-weight LLMs like GLM 5.2, for example, would run Claude Code, etc. However, if an LLM developer also develops a coding harness, it is somewhat safe to assume that their model is optimized for their own harness first (while also supporting others). Here, I am primarily going to use Qwen3.6 with the Qwen-Coder coding client. However, I will also go over other options for using a local LLM with other agent harnesses, for example, Claude Code, Codex, and the increasingly popular Cline, but more on that later. The reason why I am primarily using Qwen-Code when working with Qwen models is that: it is open-source, like Codex ( https://github.com/openai/codex ) but unlike Claude Code; Qwen models have been specifically optimized for the Qwen-Code harness (more information below); I can run both Codex (with the latest GPT model) and Qwen-Code with a local Qwen model side by side on the same machine without having to switch manually back and forth between models. Regarding the second point in the list above, that Qwen models work better in Qwen-Code, Nvidia’s Polar: Agentic RL on Any Harness at Scale paper (May 2026) has a benchmark showing that the Qwen3.5-4B base model has the best coding performance in said Qwen-Code harness (both before and after their Polar-RL training), which I included below. Figure 2: Qwen model performance in different coding harnesses via Polar: Agentic RL on Any Harness at Scale ( https://arxiv.org/abs/2605.24220 ) The benchmark in the table above is for an older Qwen3.5 model, and I am assuming that the latest Qwen3.6 models are even further optimized to do well in Qwen-Code specifically. However, Pi ( https://github.com/earendil-works/pi ) also seems to be a very interesting candidate that I need to play around with in the future. By the way, Qwen3.6 35B-A3B is about 22 GB to download, requires roughly 30-40 GB of RAM, and runs pretty swiftly on both a Mac Mini with M4 and a DGX Spark. Based on the recent benchmarks shared by Cohere earlier in June, it is currently the best local model in its size class. Figure 3: Cohere benchmark from North Mini Code report published in June ( https://huggingface.co/blog/CohereLabs/introducing-north-mini-code ) As seen above, Qwen3.6 35B-A3B dominates all but one benchmark in this size class. However, that being said, Qwen Code is a general harness and also supports other types of models. For instance, we could also connect North Mini Code or Gemma 4 in Qwen Code. Figure 4: Yes, Qwen3.6 35B-A3B is a really good model! (Via x.com/pupposandro/status/2064707907489272147/) Architecture-wise, the Qwen3.6 35B-A3B model has hybrid attention similar to Qwen3-Coder and Qwen3.5. I wrote more about it in Beyond Standard LLMs . Figure 5: Qwen3.6 architecture and fact sheet from my LLM gallery . Alternatively, if you don’t want to use Qwen3.6, Cohere’s North Mini Code is probably the most interesting, capable alternative at this size class right now. I will go over this model in the next local LLM setup section as well. Figure 6: North Mini Code architecture and fact sheet from my LLM gallery . No matter what agent harness we use (Qwen-Code, Codex, or Claude Code), we have to set up a local LLM, such as Qwen3.6 35B-A3B, first. There are several options like Ollama, LM Studio, vLLM, SGLang, MLX, etc to serve models locally. You know from my Build A Large Language Model (From Scratch) and Build A Reasoning Model (From Scratch) projects that I like to code these myself. Implementing a model from scratch has the benefits that we understand the whole stack, plus we can modify and further train and fine-tune it. However, here, we just look for a model serving framework that has been super optimized for inference speed and resource needs since we don’t plan to do any training or fine-tuning at this point. (We could, as an extra step, convert and import our own from-scratch fine-tuned model into these efficient serving stacks, but this is out of the scope for this article.) For this tutorial, we will use Ollama as our efficient model serving engine because it’s relatively easy to install and use from the command line across different operating systems (although LM Studio also added a non-GUI client, but I am less familiar with it). By the way, I am not affiliated with any of the tools mentioned in this article, but one nice thing about Ollama is that they also optionally support open-weight models hosted in the cloud, including the currently strongest open-weight model, GLM 5.2, which is too large to run locally on consumer hardware. (The cloud models are not free, of course, but have similar subscription plans as ChatGPT and Claude; it’s still nice though that this option exists to conveniently test the latest state-of-the-art open-weight models “locally.”) Anyways, setting up Ollama is pretty straightforward, and you can find the official macOS/Linux/Windows download instructions on their download page. After installing, I recommend downloading a model for a quick test run. For instance, on macOS, we can use the ollama app to download models directly via the GUI: Figure 7: Using the Ollama app to find and download models Otherwise, this can be done on the command line as well via By the way, the above-mentioned qwen3.6:35b-mlx is a model using Apple’s Metal performance shaders, i.e., optimized for Macs with Apple silicon chips. I highly recommend using *-mlx versions of models working on Macs (if available). Figure 8: Prefer the MLX version when using a Mac (with an Apple Silicon chip). On a Linux machine, use the non-MLX version: Then, to make sure that it works, you can either use the GUI again or launch Ollama from the command line. Figure 9: Running Ollama in the terminal. You can exit this session via the command. As mentioned before, the currently best alternative to this Qwen3.6 35B-A3B model is North Mini Code 1.0 of similar size. Figure 10: North Mini Code 1.0 as an alternative to Qwen3.6 35B A3B. Before deciding on whether to use an LLM as a local coding agent, it’s usually not a bad idea to run a quick speed and quality assessment. Here, for the speed assessment, I would look for tokens/sec performance. Additionally, I’d also make sure this stays stable for (very) long contexts, which is what we are usually dealing with during agentic coding workflows (as opposed to simpler chatbots). Of course, we also don’t want the memory cost to explode either. You could run my ollama_speed_memory_bench.py script to do a quick check. In a nutshell, it sends different prompts (ranging from 1k to 50k words) to an Ollama model and asks it to generate up to 8k tokens by default. It reports simple statistics like prefill speed from Ollama’s prompt evaluation metrics, generation speed from output-token timing, and memory use from the Ollama process plus NVIDIA GPU memory when available. For example, to evaluate the on macOS, if you downloaded or cloned the scripts from https://github.com/rasbt/local-coding-agent-evals , we can run the following, which takes about 5 minutes: On Linux, we can run: Note that this assumes that you already downloaded the respective model as explained in the previous section. Also, depending on your system, if you have less than 30 GB RAM, you may have to use a smaller model like gemma4:e2b, which uses up to about 8 GB RAM on long contexts. Of course, there are also many smaller models, but in my experience, they make pretty bad local coding agents.) Note that for models, the RSS RAM report is not super accurate on macOS (especially for mlx model variants that utilize the Metal backend), and I suggest keeping an eye on the activity monitor’s RAM usage for Ollama during the run as well. In this case, the RAM usage fluctuated between 20 - 29 GB. Anyways, the bottom line is that for 50k contexts, the Qwen3.6 and North Mini Code models use up to 30 GB RAM and generate output with about 40 tok/sec on a recent Mac Mini and 30 tok/sec on a DGX. Below is a visual summary of the different runs. Figure 11: Quick speed comparison of the different models on different systems. Note that the macOS RAM consumption is not super accurate there. Also, note that the Qwen 35B-A3B model is faster on Mac than on the DGX Spark (which is the other way around for the Gemma 4 E2B model) thanks to the optimized MLX version. Code to reproduce: https://github.com/rasbt/local-coding-agent-evals Another interesting question is how Qwen 35B-A3B compares to the similarly-sized Cohere North Mini model? If we take similarly quantized models into account (above, I was using the Qwen3.6 default), they are pretty similar, although North Mini is perhaps slightly ahead overall, as shown below. Figure 12: Q4-quantized Qwen3.6 35B vs North Mini Code. Code to reproduce: https://github.com/rasbt/local-coding-agent-evals Anyway, the bottom line is that, in my opinion, anything faster than 20-30 tok/sec is pretty reasonable for local agent work. This is about the same speed as GPT 5.5 with “high” reasoning . In this case, both models clear the bar easily. By the way, personally, I run my agents almost exclusively on my DGX Spark because I don’t want my Mac Mini to get too hot and I want to have the RAM available for other tasks. Of course, there are always ways to optimize this more with different frameworks (other than Ollama), quantizations, MTP, and so on. However, Ollama is a good plug & play allrounder with minimal setup time that connects easily to various coding agent frameworks and where it’s super simple to swap and try out different models. After checking that the model is fast enough for convenient local work, I recommend doing a quick modeling performance assessment. Sure, there are many standardized benchmarks out there we could take a look at and even run ourselves. Usually, you can find the numbers for relevant benchmarks in the model’s technical report or model hub page. Usually, I also find it useful to look at a relative comparison with other models on https://artificialanalysis.ai/models/ . Figure 13: Benchmark from https://artificialanalysis.ai/models/ . Average performance (top), coding performance (center), agentic performance (bottom). Based on the figure above, we can see that Qwen3 35B-A3B is much more capable than the Gemma 4 E4B and E2B models, for example. Note that the Artificial Intelligence Index numbers keep changing over time as they swap benchmarks and update the weighting, so there are no “absolute” numbers we could use as a reference point for deciding which model is “good enough”. Rather, I would compare a new, interesting model to a model you used before as an anchor or reference point. Beyond standard benchmarks, I would also curate a personal set of tasks that are relevant to you to do a quick check whether this model is even suitable for any type of work that you might want it to perform. Below are the outputs of a reasoning- and code-related set of questions that also test the tool calling capabilities of the models. Here, the model returns the tool call but doesn’t execute the code itself. For instance, we can say that gets the conceptual debugging and security-review tasks right, but still struggles with agentic judgment around “what file/action first” tasks. is usable but not fully reliable for autonomous tool use. But a harness that constrains actions, adds retries, and maybe gives stronger project context could make it pretty usable. On the other hand, failing is a strong signal that it is less suitable for this kind of tool-use reasoning, even if it is fast. Note that the failures are not just formatting issues. It looks like it chooses the wrong tool, asks for clarification when enough context is present, etc. I would probably not use it as a coding-agent model beyond very narrow or heavily constrained tasks. Now, after this lengthy preamble setting up a local LLM, let’s get back to the main topic, the coding agent harness. As mentioned at the beginning of this article, we will use the qwen-code ( https://github.com/QwenLM/qwen-code ) harness, as Qwen models have been optimized for it. Figure 14: Next, we are trying to connect the locally served model to the coding agent harness. If you are familiar with Claude Code, it’s basically the same thing but fully open-source. However, I will also go over how to connect the local Qwen3.6 model to Codex and Claude Code in the next sections. Note that coding harnesses are much more capable than LLMs by themselves. This is where I recommend being more careful about what you are running and where. For instance, when trying new (coding) agents, I like to Do an audit of the (open-source) agent code base first. Run it on separate hardware (e.g., my DGX Spark) or a separate user account and/or virtual environment on my machine at the very least. Regarding the audit, I recommend looking for data sharing/egress and the default blast radius when it comes to file permissions, as well as some baseline robustness to prompt injection. The figure below attempts to summarize the main points. Figure 15: Practical audit checklist before running an installed coding agent harness. Similar concerns apply to the local model serving engine (e.g., Ollama) as well. However, coding agents require even more attention as they can directly read data from your machine and manipulate files. To do a basic audit, I recommend the following: Clone the repo: Ask a trusted agent you used before (like GPT 5.5 in Codex or Opus 4.8 in Claude Code) to review it with a focused prompt. Something like the following: You are auditing ./qwen-code before I install or run the agent on my machine. Focus only on practical local-machine risk from the installed agent and the code paths that create it: install scripts and package lifecycle hooks shell command execution by the agent file read/write boundaries at runtime secret handling and environment-variable inheritance how repo files, project instructions, and tool output can influence the agent MCP, plugin, extension, or tool integrations network calls and telemetry update mechanisms after installation terminal escape/output handling data egress and data residency Ignoring internet downloads that are strictly required for installation, check whether the installed agent can send prompts, files, telemetry, logs, identifiers, or metadata to remote servers when I use a local model through Ollama. Ignore cloud-model configurations. Do not infer risk from the project owner alone. Identify concrete endpoints, SDKs, default providers, environment variables, config defaults, and docs that control network behavior, including any endpoints operated in foreign countries or by third-party companies. Do not do broad style review. Do not refactor. Produce: high-risk findings with file/line references medium-risk concerns network/data-egress findings, including any foreign, third-party, or China-linked endpoints or defaults commands I should avoid running until reviewed settings or environment variables that reduce local-machine risk a short recommendation: safe to test in sandbox, safe to use, or do not run For each item, say whether it is expected behavior for a coding agent or inherently riskier than Codex or Claude Code. Below is a summary of the main findings (because the full report may be a bit boring and too long for this article): Local execution Qwen Code can run shell commands on our machine through its shell tool but there are strict approval controls unless permissive modes such as are enabled. This is expected for a coding agent, and it’s actually what makes it useful in practice. But of course it becomes risky if run unsandboxed or with a full environment containing secrets. Data egress Even with local Ollama, Qwen Code can send usage telemetry and metadata to Alibaba/Aliyun endpoints unless usage statistics and telemetry are disabled (more on that below). This is riskier than a local-only setup because model prompts may stay local, but session IDs, tool metadata, model info, and local base URL metadata can still leave the machine. But again, this is also common among all kinds of tools (yes, Codex and Claude do that as well). File and secret boundaries Workspace files are readable by default, while writes generally require approval and include some overwrite protections. This is good and standard agent practice. Prompt injection surfaces Repo instructions, tool output, MCP tools, extensions, and project config can influence the agent’s behavior. Prompt injection attacks can be reduced via the approval gates mentioned above. This is normal for coding agents, but untrusted repos should be treated as hostile by default because they can steer the agent toward reading files, running commands, or sending data through approved tools. Regarding the main privacy concerns in point 2, most of it is fixable via a custom with the following contents: The setting is a tradeoff. Security fixes will not be installed automatically, but I prefer having explicit control over when updates happen instead of letting the tool pull and apply new code in the background. By the way, cline ( https://github.com/Cline/Cline ), Codex ( https://github.com/openai/codex ), and Claude Code have similar telemetry data sharing defaults that would need to be disabled explicitly. (Note that Claude Code doesn’t have an official open-source version of their codebase, which makes trusting it even trickier, and it does seem to send data to both Anthropic and Datadog.) Either way, overall, it seems Qwen-Code follows standard practices, and as of this writing, there is no particular concern that is non-standard for coding agents. If we accept the reported findings and risks (personally, I didn’t see any red flags), we can now proceed with the installation and hook up our local Qwen3.6-35B-A3B model to Qwen Code (and Codex and Claude Code in the next sections). As mentioned before, I preferably experiment with and run coding agents, which can read and edit local files, on a separate machine (in my case a DGX Spark, but it could also be a separate Mac or Linux workstation). Alternatively, I would run it in a VM or set up a separate macOS or Linux user account as a practical middle ground. (I heard from some friends that they also rent servers for that, like Linode or Heroku, for tinkering purposes. However, instead of the monthly hosting costs for a somewhat capable machine, I would probably rather get a relatively cheap $200-500 hardware box, or even an old retired laptop, and run a local harness and then use a stronger open-weight model hosted in the cloud via Ollama cloud models, OpenRouter, etc if you are looking for alternatives to GPT or Claude.) Anyways, let’s install Qwen-Code. The listed options include, e.g., However, running the commands above assumes that the published artifacts match the code we just reviewed in the GitHub repo. If we are extra careful/paranoid, we can also build it ourselves from the GitHub repo. Be warned, this is more manual/messier though (I recommend executing them one at a time instead of copy & pasting the whole block into the terminal): After completing the installation, we can now launch the Qwen-Code client via the qwen command from the terminal to complete the setup and connect to the locally served LLM. For this, after running the qwen command, we select “Custom Provider”, as shown below. Figure 16: Choose “Custom Provider,” which lets us connect the Ollama LLM. Ollama uses the OpenAI API standard. So, next, we follow the on-screen setup guide and choose the “OpenAI-compatible” option. Figure 17: Since Ollama follows the OpenAI API standard, we choose “OpenAI-compatible” here. Next, we need to provide the API endpoint of the running Ollama application that serves our local LLM. Usually that’s the local address by default. We enter (including the /v1) since that’s the OpenAI-compatible base URL. Figure 18: Configure Qwen Code to use Ollama’s local OpenAI-compatible endpoint, . Next, we enter as our custom provider. Figure 19: Enter as the API key placeholder for the local custom provider. Next, we can select the available models. These are the ones that we downloaded via . You can enter only a single model or multiple ones separated by commas. You can double-check the list of downloaded models via . By the way, you can always add more models easily later (I’ll explain after completing the setup). Figure 20: Select the local Ollama models that Qwen Code should make available through the custom provider. We are almost done! In step 5/6, we of course select “Enable thinking” mode, which will result in higher token usage but the better resulting problem-solving capabilities are worth it. Figure 21: Enable thinking mode for the local model provider. And that’s basically it. Step 6 is basically a review step that we can confirm by pressing “Enter”. Congratulations, you should now have a working fully-local LLM workflow set up. The usage is pretty much similar to Claude Code, where you can use / commands for various functionality. E.g., you can switch models via the command, as shown below. Figure 22: Use to switch models. By the way, as I mentioned before, it’s relatively easy to add new models from ollama. Once you pull a new model via , you can add it as a new entry in . Here, just copy & paste an existing entry into the file and change the “id” and “name” to that of the Ollama model name. Figure 23: We can add new ollama models by editing the config file. Here, is the name of the ollama model name, e.g., . By the way, to update the qwen-code tool once in a while, if we used the git clone & local build route, we can pull a recent GitHub snapshot and update it as follows: Now that we have a fully working, local coding agent, the question is: how well does it perform, and is it actually good enough for my tasks? Of course, there are benchmarks for this, but in my opinion, nothing beats trying it for yourself on some of your workflow. In other words, this basically means using it for a day or two to decide whether it meets your bar. I also recommend compiling a small set of tasks that reflect your common coding agent usage. And if you come upon a particularly challenging one when working on a given project, it may not be a bad idea to add it to this set to evaluate future models. As an example of what I mean, I shared a relatively small, simple, and general set of tasks we can use to test the agents here on GitHub: https://github.com/rasbt/local-coding-agent-evals/tree/main/agent-problem-pack . This is basically an extension of the tasks from the Local LLM Setup section. The details on how to run these are in the GitHub README: https://github.com/rasbt/local-coding-agent-evals/tree/main/agent-problem-pack#quick-start-running-benchmarks-manually . Below is the outcome for the different LLMs tested in Qwen-Code. Figure 24: Small local agent capability benchmark using Qwen-Code. Code to reproduce: https://github.com/rasbt/local-coding-agent-evals As we can see, both the Qwen3.6 and North Mini Code 35B-A3B models solve 4 out of 5 of these problems. Gemma 4 E2B fails a lot. Out of curiosity, I also added the a bit older Nemotron 3 Nano model. It has a similar size and compute performance as the aforementioned Qwen and North models, and it performs similarly well. Figure 25: Nemotron 3 Nano architecture overview from my LLM Gallery After setting up the local coding agent (and the article exceeding 5000 words), this would probably be a reasonable place to stop. However, as a bonus, I also thought it might be interesting to add brief Codex and Claude Code notes for completeness. Unfortunately, as far as I know, the Codex UI does not support non-OpenAI models, but we can use the Codex CLI to run our Ollama models. If you haven’t installed the OpenAI Codex CLI yet, you can get and install it analogously to qwen-code from their open-source GitHub directory: https://github.com/openai/codex (Yes, the Codex CLI is open source!) I will spare you the lengthy listing of the commands and recommend checking the repo’s README instead for the official instructions. (Cloning the repo and running an audit similar to qwen-code is not a bad idea here, as well.) Then, once installed, there are multiple ways to enable local model use. In my opinion, the most convenient way is to set up a separate config (inside the existing folder) with some default options: Figure 26: Set up a separate Ollama profile for Codex for convenience. Then, we can still use to launch the regular “Codex with GPT 5.5” mode and use our Ollama model via . Figure 27: Launch Codex using a local Ollama model. When rerunning the test cases from the Agent Capability Assessment section, to my surprise, Qwen3.6 does actually perform better via Codex compared to its “native” Qwen-Code coding harness, as shown below. Figure 28: Small local agent capability benchmark in Codex. Even though this is just a small set of benchmarks, it suggests that using Codex as the universal coding agent harness may not be such a bad idea after all. Of course, there is also the popular Claude Code agent harness that we could use as a harness around our local LLMs. While very popular and capable, this is probably my least favorite option for local setups because the codebase is proprietary. That also means we cannot readily inspect and/or disable Anthropic’s data logging practices. To set it up, if you don’t have Claude Code already installed on your machine, I suggest checking the official docs for recommended installation commands: https://code.claude.com/docs/en/quickstart . Claude Code itself does not expose the same local-provider configuration path as Codex. However, Ollama provides an integration via : https://docs.ollama.com/integrations/claude-code I.e., we can execute to run the Claude Code harness with an Ollama model. By the way, this also works for codex via , but I personally prefer the route we discussed earlier, as it gives me a bit more insight and control about how things works etc. Figure 29: Claude Code with a local Qwen3.6 model through Ollama. However, as a user, it feels like Claude Code takes much longer to come up with a solution. It probably has a much higher token usage. So, below, I additionally looked at the token usage of all three harnesses. As we can see, Claude Code uses by far the most tokens on average, Codex the least. Figure 30: Average token usage of the three harnesses for different LLMs. Code to reproduce: https://github.com/rasbt/local-coding-agent-evals When it comes to the little agent capability assessment benchmark, the Qwen and North Mini Code models also get 5/5, and even the small Gemma 4 model does ok! Interestingly, we can also see that the token usage is largely driven by the harness, not the LLM itself. I.e., among all three LLMs that are capable of solving (almost) all 5 tasks, they all use the same number of tokens (e.g., Qwen3.6 uses roughly the same number of tokens as North Mini Code and Nemotron 3 Nano when used inside Claude Code). Only Gemma 4 uses fewer tokens, but it also fails almost all tasks, likely because of insufficient tool-calling capabilities where the tasks interrupt early. For reference, below is again the summarized task-success rate. Figure 31: Summarized task success rates. Anyway, the takeaway here is that if more tokens help the model-harness combination to solve more (and more complex) problems, great! But if we have two harnesses that both have an equal task success rate, a harness that uses 50% fewer tokens (e.g., Codex over Claude Code), then this is a huge win, because it will make tasks run twice as fast. However, the big caveat here is that task correctness is a necessary criterion, but it doesn’t measure code quality and readability, which are hard to assess automatically. PS: I tried to analyze why Claude Code uses more tokens, and it seems that the difference mainly comes from input tokens rather than output tokens. In other words, Claude is not writing twice as much. The logs suggest that Claude is repeatedly feeding more context back into the model across turns, including previous messages, tool calls, command outputs, and file contents. For example, one Claude run used about 578k input tokens but only about 4.5k output tokens across 25 turns. So the likely explanation is that Claude’s harness accumulates or accounts for a larger prompt-side history during multi-step agent runs. So far, all the setups we discussed assumed that we were running the local LLM on the same machine as the coding harness. However, what if we developed some trust in the coding agent harness and want to use it on our main Mac while the model itself is hosted on a different machine, e.g., a DGX Spark? In my opinion, the best (or most convenient) setup is an SSH tunnel from the Mac to the DGX. First, I suggest quitting Ollama on the Mac or changing the to something else below. Assuming we quit the Ollama app on the Mac, check that the following returns an empty output to indicate that Ollama is not available: Then run the following command on that Mac in a terminal window on the Mac side: That command means that we open an SSH connection to as user , which you need to adjust to whatever your username and machine name are. Then, the command forwards the Mac’s local port to on the DGX because of . Note that this is the Ollama address. The terminal running will look like it is hanging. That is normal. Keep it open while you use Qwen Code, Codex, or Claude Code. Press to stop the tunnel. So after it is running, use this on your Mac to see if the Mac can indeed access the ollama models from the DGX: If that returns the DGX models, your Mac tools can use the DGX Ollama server as if it were local. Then, just use Qwen Code and Codex just like above. For Claude via , the key is that the Mac-side command must see the tunneled endpoint. If needed: We focused on Qwen Code, Codex, and Claude Code because they are the most direct fit for coding-agent workflows. OpenClaw and Hermes are also capable, but they are broader agent harnesses. They are better suited when you want one agent to coordinate across tools, apps, browsers, terminals, and longer-running workflows. For coding work, I recommend starting with Qwen Code, Codex, or Claude Code first (and there are also many other interesting coding harnesses like OpenCode, Cline, Pi, and Noumena Code). And I would treat OpenClaw and Hermes as interesting follow-up options for things beyond coding rather than the first baseline for this local coding-agent setup. This was a long article with lots of information and configuration. If there are a few main takeaways, I’d say that it’s not the mechanistic setup pipeline but rather the considerations when running coding agents locally. That is, the most important part is not getting one specific tool installed, but understanding the model-serving layer, the agent harness, the permission model, and how to evaluate whether the setup actually solves coding tasks reliably. Of course, GPT 5.5 and Opus 4.8 are currently better than smaller open-weight models that run on a Mac or DGX Spark. But the newer Mixture-of-Experts models in the 30-35B range (such as Qwen3.6, North Mini Code, and Nemotron 3 Nano) are all very, very capable and really sufficient for a lot of tasks. And yes, they run with the same token speed as GPT 5.5 through a Pro subscription, so it should not necessarily slow down your workflows. The main consideration when setting up local agents, besides the model itself, is also which harness we want to use. The common perception is that models are usually optimized more for a specific harness than others (e.g., Qwen3.6 may work better in Qwen Code than Claude Code, for example). Based on the small agent assessment, this may not necessarily be true, though (this is only a very small benchmark, so take it with a big grain of salt). So, if you are more comfortable with a different harness that you have a lot of muscle memory with, like Codex and Claude Code, maybe it’s not a bad idea to just stick the model into that one and give it a try! Anyways, I hope the article was useful, and it got you interested in doing some tinkering with open-weight models. They are becoming more capable by the day, and it’s for some inexplicable reason just fun to run models locally. If you want to try the benchmarks yourself, the code and small evaluation tasks used in this article are available here: https://github.com/rasbt/local-coding-agent-evals Also, my Build a Reasoning Model (From Scratch) book has now gone to print and started shipping. I wanted to post a picture, but it will be 3 more days until it arrives. Build a Reasoning Model (From Scratch) If you liked my previous Build a Large Language Model (From Scratch) book, this is essentially a sequel implementing inference-time scaling techniques and reinforcement learning algorithms from scratch. And if you want to support future long-form articles like this one, consider becoming a paid subscriber . It helps me keep writing these independent deep dives and sharing the accompanying code, figures, and experiments. Figure 1: Overview of the local stack, that is, a coding agent harness that uses a local model hosted through an inference engine / runtime server. This article is a tutorial on setting up a production-ready coding agent with a fully local stack. We will use a locally served LLM together with a local coding harness that can read files, make edits, run commands, and verify changes as shown in the figure above. Here, we can think of the LLM as the engine that provides the reasoning and code generation. And the surrounding harness provides the operating environment that allows the LLM to do meaningful coding work in our local projects. Why local? For many coding workflows, a local setup is an interesting alternative to proprietary services such as GPT in Codex or Opus in Claude Code. The local setup is transparent, inspectable, and free to run apart from hardware and electricity costs. It also stays fully under your control, and you can modify the coding harness in any way you like. Plus, it’s a lot of fun! By the way, in case you want a bit more background information on coding agent harnesses, I covered the core components of coding agents (and building a coding agent from scratch for learning purposes) here: 1. Intro I have to admit that I still primarily alternate between Codex and Claude Code as my daily drivers, for now (and just to keep up with the new tooling and functions that are constantly being added). Also, the plan limits (especially for Codex) are still so generous that I haven’t had to worry about costs so far. However, I’ve been using local solutions for a while, too, to test things and because it somehow gives me joy to have and use a fully local setup (versus proprietary services). Either way, local solutions become more and more attractive each day. One aspect is the costs. If you have the hardware, they are practically free to run. And then there’s, of course, the privacy angle. For example, for organizing and processing my receipts, I’d be more comfortable with a local model ingesting them rather than sending the data over to OpenAI or Anthropic. (Then, if we keep in mind that Anthropic was recently throttling their flagship model’s performance for LLM research , proprietary services may become more restrictive over time, and it’s maybe a good idea to be comfortable with open-weight alternatives as a backup.) And there are many, many additional reasons and use cases like that. Your motivations for using local LLMs and coding harnesses may include: Predictable, fixed costs if you reach your subscription plan limits, and immunity to API price changes. Reproducibility; sometimes it’s nice if a model is upgraded (e.g., GPT 5.4 -> GPT 5.5 -> GPT 5.6) and it solves all your queries more reliably. However, this can also break existing workflows. Offline use in the classic airplane flight scenario with slow or no internet, or when going on a coding/writing retreat in the cabin in the woods w/o a Starlink subscription. it is open-source, like Codex ( https://github.com/openai/codex ) but unlike Claude Code; Qwen models have been specifically optimized for the Qwen-Code harness (more information below); I can run both Codex (with the latest GPT model) and Qwen-Code with a local Qwen model side by side on the same machine without having to switch manually back and forth between models. Figure 3: Cohere benchmark from North Mini Code report published in June ( https://huggingface.co/blog/CohereLabs/introducing-north-mini-code ) As seen above, Qwen3.6 35B-A3B dominates all but one benchmark in this size class. However, that being said, Qwen Code is a general harness and also supports other types of models. For instance, we could also connect North Mini Code or Gemma 4 in Qwen Code. Figure 4: Yes, Qwen3.6 35B-A3B is a really good model! (Via x.com/pupposandro/status/2064707907489272147/) Architecture-wise, the Qwen3.6 35B-A3B model has hybrid attention similar to Qwen3-Coder and Qwen3.5. I wrote more about it in Beyond Standard LLMs . Figure 5: Qwen3.6 architecture and fact sheet from my LLM gallery . Alternatively, if you don’t want to use Qwen3.6, Cohere’s North Mini Code is probably the most interesting, capable alternative at this size class right now. I will go over this model in the next local LLM setup section as well. Figure 6: North Mini Code architecture and fact sheet from my LLM gallery . 3. Local LLM Setup No matter what agent harness we use (Qwen-Code, Codex, or Claude Code), we have to set up a local LLM, such as Qwen3.6 35B-A3B, first. There are several options like Ollama, LM Studio, vLLM, SGLang, MLX, etc to serve models locally. You know from my Build A Large Language Model (From Scratch) and Build A Reasoning Model (From Scratch) projects that I like to code these myself. Implementing a model from scratch has the benefits that we understand the whole stack, plus we can modify and further train and fine-tune it. However, here, we just look for a model serving framework that has been super optimized for inference speed and resource needs since we don’t plan to do any training or fine-tuning at this point. (We could, as an extra step, convert and import our own from-scratch fine-tuned model into these efficient serving stacks, but this is out of the scope for this article.) For this tutorial, we will use Ollama as our efficient model serving engine because it’s relatively easy to install and use from the command line across different operating systems (although LM Studio also added a non-GUI client, but I am less familiar with it). By the way, I am not affiliated with any of the tools mentioned in this article, but one nice thing about Ollama is that they also optionally support open-weight models hosted in the cloud, including the currently strongest open-weight model, GLM 5.2, which is too large to run locally on consumer hardware. (The cloud models are not free, of course, but have similar subscription plans as ChatGPT and Claude; it’s still nice though that this option exists to conveniently test the latest state-of-the-art open-weight models “locally.”) Anyways, setting up Ollama is pretty straightforward, and you can find the official macOS/Linux/Windows download instructions on their download page. After installing, I recommend downloading a model for a quick test run. For instance, on macOS, we can use the ollama app to download models directly via the GUI: Figure 7: Using the Ollama app to find and download models Otherwise, this can be done on the command line as well via By the way, the above-mentioned qwen3.6:35b-mlx is a model using Apple’s Metal performance shaders, i.e., optimized for Macs with Apple silicon chips. I highly recommend using *-mlx versions of models working on Macs (if available). Figure 8: Prefer the MLX version when using a Mac (with an Apple Silicon chip). On a Linux machine, use the non-MLX version: Then, to make sure that it works, you can either use the GUI again or launch Ollama from the command line. Figure 9: Running Ollama in the terminal. You can exit this session via the command. As mentioned before, the currently best alternative to this Qwen3.6 35B-A3B model is North Mini Code 1.0 of similar size. Figure 10: North Mini Code 1.0 as an alternative to Qwen3.6 35B A3B. 4. Simple Speed Performance Assessment Before deciding on whether to use an LLM as a local coding agent, it’s usually not a bad idea to run a quick speed and quality assessment. Here, for the speed assessment, I would look for tokens/sec performance. Additionally, I’d also make sure this stays stable for (very) long contexts, which is what we are usually dealing with during agentic coding workflows (as opposed to simpler chatbots). Of course, we also don’t want the memory cost to explode either. You could run my ollama_speed_memory_bench.py script to do a quick check. In a nutshell, it sends different prompts (ranging from 1k to 50k words) to an Ollama model and asks it to generate up to 8k tokens by default. It reports simple statistics like prefill speed from Ollama’s prompt evaluation metrics, generation speed from output-token timing, and memory use from the Ollama process plus NVIDIA GPU memory when available. For example, to evaluate the on macOS, if you downloaded or cloned the scripts from https://github.com/rasbt/local-coding-agent-evals , we can run the following, which takes about 5 minutes: On Linux, we can run: Note that this assumes that you already downloaded the respective model as explained in the previous section. Also, depending on your system, if you have less than 30 GB RAM, you may have to use a smaller model like gemma4:e2b, which uses up to about 8 GB RAM on long contexts. Of course, there are also many smaller models, but in my experience, they make pretty bad local coding agents.) Note that for models, the RSS RAM report is not super accurate on macOS (especially for mlx model variants that utilize the Metal backend), and I suggest keeping an eye on the activity monitor’s RAM usage for Ollama during the run as well. In this case, the RAM usage fluctuated between 20 - 29 GB. Anyways, the bottom line is that for 50k contexts, the Qwen3.6 and North Mini Code models use up to 30 GB RAM and generate output with about 40 tok/sec on a recent Mac Mini and 30 tok/sec on a DGX. Below is a visual summary of the different runs. Figure 11: Quick speed comparison of the different models on different systems. Note that the macOS RAM consumption is not super accurate there. Also, note that the Qwen 35B-A3B model is faster on Mac than on the DGX Spark (which is the other way around for the Gemma 4 E2B model) thanks to the optimized MLX version. Code to reproduce: https://github.com/rasbt/local-coding-agent-evals Another interesting question is how Qwen 35B-A3B compares to the similarly-sized Cohere North Mini model? If we take similarly quantized models into account (above, I was using the Qwen3.6 default), they are pretty similar, although North Mini is perhaps slightly ahead overall, as shown below. Figure 12: Q4-quantized Qwen3.6 35B vs North Mini Code. Code to reproduce: https://github.com/rasbt/local-coding-agent-evals Anyway, the bottom line is that, in my opinion, anything faster than 20-30 tok/sec is pretty reasonable for local agent work. This is about the same speed as GPT 5.5 with “high” reasoning . In this case, both models clear the bar easily. By the way, personally, I run my agents almost exclusively on my DGX Spark because I don’t want my Mac Mini to get too hot and I want to have the RAM available for other tasks. Of course, there are always ways to optimize this more with different frameworks (other than Ollama), quantizations, MTP, and so on. However, Ollama is a good plug & play allrounder with minimal setup time that connects easily to various coding agent frameworks and where it’s super simple to swap and try out different models. 5. Simple Benchmark Performance Assessment After checking that the model is fast enough for convenient local work, I recommend doing a quick modeling performance assessment. Sure, there are many standardized benchmarks out there we could take a look at and even run ourselves. Usually, you can find the numbers for relevant benchmarks in the model’s technical report or model hub page. Usually, I also find it useful to look at a relative comparison with other models on https://artificialanalysis.ai/models/ . Figure 13: Benchmark from https://artificialanalysis.ai/models/ . Average performance (top), coding performance (center), agentic performance (bottom). Based on the figure above, we can see that Qwen3 35B-A3B is much more capable than the Gemma 4 E4B and E2B models, for example. Note that the Artificial Intelligence Index numbers keep changing over time as they swap benchmarks and update the weighting, so there are no “absolute” numbers we could use as a reference point for deciding which model is “good enough”. Rather, I would compare a new, interesting model to a model you used before as an anchor or reference point. Beyond standard benchmarks, I would also curate a personal set of tasks that are relevant to you to do a quick check whether this model is even suitable for any type of work that you might want it to perform. Below are the outputs of a reasoning- and code-related set of questions that also test the tool calling capabilities of the models. Here, the model returns the tool call but doesn’t execute the code itself. For instance, we can say that gets the conceptual debugging and security-review tasks right, but still struggles with agentic judgment around “what file/action first” tasks. is usable but not fully reliable for autonomous tool use. But a harness that constrains actions, adds retries, and maybe gives stronger project context could make it pretty usable. On the other hand, failing is a strong signal that it is less suitable for this kind of tool-use reasoning, even if it is fast. Note that the failures are not just formatting issues. It looks like it chooses the wrong tool, asks for clarification when enough context is present, etc. I would probably not use it as a coding-agent model beyond very narrow or heavily constrained tasks. 6. Agent Code Base Audit Now, after this lengthy preamble setting up a local LLM, let’s get back to the main topic, the coding agent harness. As mentioned at the beginning of this article, we will use the qwen-code ( https://github.com/QwenLM/qwen-code ) harness, as Qwen models have been optimized for it. Figure 14: Next, we are trying to connect the locally served model to the coding agent harness. If you are familiar with Claude Code, it’s basically the same thing but fully open-source. However, I will also go over how to connect the local Qwen3.6 model to Codex and Claude Code in the next sections. Note that coding harnesses are much more capable than LLMs by themselves. This is where I recommend being more careful about what you are running and where. For instance, when trying new (coding) agents, I like to Do an audit of the (open-source) agent code base first. Run it on separate hardware (e.g., my DGX Spark) or a separate user account and/or virtual environment on my machine at the very least. Figure 15: Practical audit checklist before running an installed coding agent harness. Similar concerns apply to the local model serving engine (e.g., Ollama) as well. However, coding agents require even more attention as they can directly read data from your machine and manipulate files. To do a basic audit, I recommend the following: Clone the repo: Ask a trusted agent you used before (like GPT 5.5 in Codex or Opus 4.8 in Claude Code) to review it with a focused prompt. Something like the following: install scripts and package lifecycle hooks shell command execution by the agent file read/write boundaries at runtime secret handling and environment-variable inheritance how repo files, project instructions, and tool output can influence the agent MCP, plugin, extension, or tool integrations network calls and telemetry update mechanisms after installation terminal escape/output handling data egress and data residency high-risk findings with file/line references medium-risk concerns network/data-egress findings, including any foreign, third-party, or China-linked endpoints or defaults commands I should avoid running until reviewed settings or environment variables that reduce local-machine risk a short recommendation: safe to test in sandbox, safe to use, or do not run Local execution Qwen Code can run shell commands on our machine through its shell tool but there are strict approval controls unless permissive modes such as are enabled. This is expected for a coding agent, and it’s actually what makes it useful in practice. But of course it becomes risky if run unsandboxed or with a full environment containing secrets. Data egress Even with local Ollama, Qwen Code can send usage telemetry and metadata to Alibaba/Aliyun endpoints unless usage statistics and telemetry are disabled (more on that below). This is riskier than a local-only setup because model prompts may stay local, but session IDs, tool metadata, model info, and local base URL metadata can still leave the machine. But again, this is also common among all kinds of tools (yes, Codex and Claude do that as well). File and secret boundaries Workspace files are readable by default, while writes generally require approval and include some overwrite protections. This is good and standard agent practice. Prompt injection surfaces Repo instructions, tool output, MCP tools, extensions, and project config can influence the agent’s behavior. Prompt injection attacks can be reduced via the approval gates mentioned above. This is normal for coding agents, but untrusted repos should be treated as hostile by default because they can steer the agent toward reading files, running commands, or sending data through approved tools.

Python

0 views

Stone Tools 2 weeks ago

Visual Basic on the PC w/Windows 3.1

If I dig deep into my own heart, really self-reflect, I find I simply don't possess whatever people like Bill Gates and Elon Musk do. I think most of us are content to know we've touched a life or two, helped make someone's existence a bit more pleasant, and can feel gratitude toward the universe for those small miracles. Others seem to know no limit in their acquisition of influence, power, and wealth. For them, it isn't simply enough to guide an industry, they must be the industry. In this zero-sum game, there is no upper limit to their cravings Before Musk became the first (I'm choking on the word) trillionaire , Gates was the world's richest person for a couple of decades. Like Musk, he crossed a specific monetary milestone back in 1999 as the "first person with a net worth exceeding $100 billion," about $200B in 2026 money. How he earned it and what he did with it has been the subject of any number of documentaries , books , movies , interviews , depositions , and damning rumors . I think the media can agree on at least one point relevant to our discussion today: Bill Gates was hellbent on owning the entire personal computing landscape. He said as much, out loud, on stage, to industry professionals, in front of the press. Jacqui Morby recounted the story on The Computer Chronicles . "Gary (Kildall) got up (at the Rosen Forum panel discussion) and talked about what his plans were for CP/M and where the company was going, and then made a comment, 'Well, this is a very large market, and there's room for lots of companies.' Bill Gates interrupted and said, 'No, there'll only be one company.'" He didn't seem particularly interested in creating innovative things, so much as he wanted to make sure that the innovations of others had a Microsoft response. While working with Apple to develop software for the original Macintosh, Andy Hertzfeld recalled a story of Gates digging in for system details that didn't really have anything to do with the business applications being built by Microsoft. Shortly thereafter, Windows 1.0 released, much to Steve Jobs's frustration . Jobs wouldn't be the last to feel screwed over by Microsoft "taking" ideas . Another tactic employed by Gates was absorption, the tried and true fast-track to acquiring toys one lacks. Consider the story of Alan Cooper . Coincidentally the idea for a visual application builder "popped into his head" just as HyperCard debuted, in 1987, triggered by Microsoft's announced adoption of DLLs, dynamic link libraries, which provided easy access to core operating system functions to whomever wanted to tap into them. Cooper saw this as a unique foundation upon which to build a kind of "construction set" for the DOS visual shell of your corporate dreams. Don't like the default Windows shell? Build your own! Microsoft engineer Gabe Newell was super impressed with Cooper's demo of the construction set, then called Tripod, and arranged for a demonstration for Gates. From the excellent article, "Something Pretty Right" by Ryan Lucas. "Why can't we do stuff like this?" is very revealing phrasing, IMHO as an armchair psychologist. Give that line to 1,000 actors and you'll get 1,000 unique performances balancing the tension between frustration and longing. As a Very Rich Guy™, there was nothing Gates wanted that he couldn't have. Like someone who pays others to level up their RPG character , US$1M and a contract later, Tripod (renamed Ruby) was his. While Cooper insists that HyperCard had no influence on the creation of Tripod , Gates most certainly was thinking about it. In his article "The 25th Birthday of BASIC" for BYTE Magazine , October 1989 ( Visual Basic would debut in 1991). Ruby was reformulated into something with but a passing resemblance to Tripod . Its bespoke scripting language was replaced with a variant of BASIC, and the goal of the program was no longer to build shells on top of the Microsoft DLLs, but to build applications for Microsoft's own shell, Windows 3.0. Visual Basic was born, arguably a more profound product than Cooper's original vision. Credit where it's due, Gates saw potential that Cooper himself couldn't see. A while back, I dug into Apple's HyperCard . Visual Basic gives us an interesting opportunity to look at a similar first-party, visual programming solution from Microsoft's perspective. Like HyperCard , Visual Basic had its own dedicated magazine , and inspired legions of developers long after Microsoft ceased support in 2008. As recently as 2023 , Microsoft has had to issue official statements on their support plans for "classic" Visual Basic, which keeps a huge number of bespoke, legacy applications alive, something HyperCard cannot claim. The Microsoft vs. Apple wars of the day almost necessitated taking sides, but in truth each has something it could learn from the other. Visual Basic 3.0 was the last pure 16-bit application in the line, and was the first version to include robust database capabilities. The true potential of the product was unlocked. This particular OS/application combination is much more in keeping with the spirit of this blog, I feel. There's a lot to learn. When I studied HyperCard , I noted the 1,000 page book that awaited me. Visual Basic ships with 3,000 pages, to say nothing of the wealth of 3rd party publications; an industry unto itself. As a man who recently took another annual step toward that great Blue Screen in the sky, every tick of the second hand gently rattles my bones. For large projects like this I have to consider how quickly I can get up to speed. Well, given the temperament of training books of the day, I suppose the proper first consideration is, "How dumb am I?" I refer to myself as a "big dummy" in blog posts, and I stand by that assertation, but I don't like it when others call me dumb. I can handle more complex material, but like I said, I don't have a lot of time. How quickly can I learn Visual Basic ? That seems unabsorbably fast . Maybe if I didn't sleep? I think I'd forget everything by Monday. Also by Tuesday. "Proglaming" sounds like fun, but a week is still too fast for my pace. Getting closer. Perfect. Slow enough for an old man to follow; fast enough to finish with time to spare before involuntary admission into a retirement home. If I weren't 40 years too late, I'd throw my own hat into the publishing ring and combine "I'm a big dummy" with "I want to learn this quickly." It's been a long time since I last touched Windows 3.1. It's funny, my memory of it doesn't match my hands-on experience today. I recall it being far uglier, though it still suffers from absurdly large title bars which don't provide much in the way of information or utility. I dig that (VGA mode) powder blue , though. It's handsome if perhaps uninspired, the result of a collaboration between Microsoft and IBM for OS/2's Presentation Manager (which predates Windows 2.0). Their "Joint Development Agreement" gave pretty broad latitude to both companies to use, without licensing fees, code shared between the two companies. I'm not even tangentially familiar with law, but it does read, in part: That gave Windows 2 and 3 a nice glow-up after the flop of Windows 1.0. Initially, even Microsoft had trouble getting their own developers to build Windows applications. I imagine it must have been a huge relief for Gates to have a tool that not only made it easy to build Windows applications, but that could even be an enjoyable experience. Jumping into Visual Basic , the first impression is, "I can do this." It looks approachable. I can't explain what every button in the toolbar does, but some of the basic stuff is as easy to identify as in HyperCard . Adding a control, like a text field, is a double-click away. The "Properties" panel makes intuitive sense, for tweaking the characteristics of a selected control, something HyperCard lacks. Appending code to a control is as simple as double-clicking its instance in the window. "Properties" is context aware, only showing what can be tweaked on the selected object. For the large part, the industry abandoned this contextual approach. I wonder why? PageMaker was leaning that way with its control panel, and InDesign promptly threw that away in favor of persistent controls for things that aren't even in the current document context. Why do we need text kerning tools on screen when there's not even a text box in the current document, in Affinity for example ? Tools like Figma , Apple's Pages seem to have kept the contextual flame alive, which is nice to see. "Pros want every tool on-screen at all times," a UX consultant once said with a straight face, I guess. The toolbar could stand to be better organized and starts gesturing in the direction of that meme image about Microsoft's love of buttons . They certainly did lean heavily on this UI metaphor crutch, as a catch-all way of cramming in as many features as possible. It's confusing at times (why a "picture box" and also "images?"), but with this version of the program, on this operating system, things haven't gotten completely out of hand yet. We're getting up to speed on the controls and how to interface with them today. Let's consider some nice things about Visual Basic's approach. I am rapidly growing to appreciate the keyboard shortcuts for UI elements, like buttons and sliders. Visual Basic makes it super simple to add a keyboard hook to an on-screen control. Simply label a button with in the confusingly named "caption" property and the following character will become the keyboard shortcut, via . So, an "Exit" button with the "caption" will read and will function identically to a mouse click on that button. When I say "identically" I do mean identically. The button's built-in method will be triggered, the same as if a mouse had done it. We don't have to worry about bifurcating control logic between keyboard and mouse for such interactions. We're then treated to an amuse bouche of off-kilter things to come. Checkboxes and radio buttons both have an on/off state, where any number of checkboxes can be on/off, but only one radio button in a set can be on. When programming with these controls, checkboxes return a value of or to represent unchecked or checked. Radio buttons return a or boolean on each of the options. For now, we'll file this under "Things That Make Me Give a Skeptical Sideways Glance." After spending a couple of days with it, the built-in text editor is driving me crazy, a "feature" Visual Basic shares with HyperCard ; neither is good. I can excuse a lack of autocomplete, a tool that would debut with Visual Basic 5 , as "Something Yet to be Invented." I cannot excuse the lack of indentation assistance and word-wraps, both already common features in word processors of the day. Microsoft has given us a smidge more than the absolute bare-minimum for a text editor. Keeping code tidy and readable requires significant, diligent effort on my part; it's not coming easily to me. I appreciate the auto-capitalization (though Basic is case-insensitive) and coloring on language keywords, but syntax checking and formatting a line of text the instant I've repositioned the cursor is annoying. Unfinished lines throw up modal dialogs warning me of interpreter troubles, triggered as easily as moving the cursor up or down for a moment. It's unwieldy to sketch out a code block to fill in the details later with those constant interruptions. It would be nice to be able to trigger the parser on-demand. We're learning about the mouse and how to handle mouse events. From a programmatic standpoint, this is pretty basic stuff. One of the nice things about the code editor is the pulldown in the top toolbar surfaces all possible functions for a selected UI element. We don't have to try to remember the exact name and spelling of a function; just pick the one you want to edit and get started. A setting that is theoretically interesting is the default unit of measurement for elements. Until now, I'd never heard of "twips": a "twentieth of a point". Where a point is 72/inch, there are 1,440 twips/inch. Windows used this as a device-independent standardized unit of measure. For on-screen, a conversion to pixels was used, and for print a conversion to printer resolution was used. Any form you design in Visual Basic can be trivially sent to the printer with a simple Basic call, and it will print at the resolution of the printer, not your screen. The coolest trick, though, is "edit and continue." Because the program is being constantly interpreted, not compiled, we can run the program, pause it, modify the code, and continue live execution. This is super handy for iterating solutions to annoying bugs. The Microsoft-faithful have really never known a world without this. The Apple-faithful have had this tantalizing fruit dangled before them a couple of times now, never quite delivering on the promise. I like it. In building out WIMP applications , we need to fill out the "M" part of that acronym. Today we learn how to build menus using the "Menu Design Window." The tool is competent, if a bit inelegant. Initially, it is easy to bang out a rough outline of an application's menu structure without taking one's hands off the keyboard; mouse-free is always a welcome option. Type a menu item, hit , type the next, hit , and the next, etc. Then, apply structure to the menu with the on-screen arrow tools for indentation/reordering elements. Alas, we cannot indent at the time of menu item entry, that hierarchy must be set in a separate step later. One disappointing absence is any kind of relationship between menu elements. Moving a menu item with "submenu" items will not move those submenu elements with it. No "outliner" style editing, ala ThinkTank , here. We also cannot multi-select items to edit them as a group, something we can do with form controls. Slow, patient, one-at-a-time editing of menu items is all we get. To be fair, menus can be programmatically generated, which may honestly be a better option in many ways. That pulls us away from the "Visual" in Visual Basic , though, don't it? The design window also forces its vertical editing into a horizontal view, another "Things That Make Me Give a Skeptical Sideways Glance." The example in the screenshot shows a 3-level menu, and I'm nowhere close to filling that horizontal space. It's wasted screen real estate, made more aggravating by the fact that the menu design window cannot be resized . As I think many in the industry have internalized by now, an editor view should place its primary content front and center, with refining elements playing a supporting role. The menu item properties would be much better served filling the right-hand side of the window, giving the menu itself vertical breathing room on the left. It's one of those things that probably gets better over the years, but is conspicuously half-baked for version 3 of the product. "It's OK, but I expected better by version 3," will be a running theme going forward. Now that I've been at this for a week, the angle of approach to visual programming HyperCard and Visual Basic each take has come into sharper focus. Initially, their superficial similarities led me to expect more direct parity between the two. Both provide a visual toolkit for designing interfaces. Both use a more simplistic language than the core language for each platform. Neither is truly "object oriented" (if that's important to you). Both were killed despite amassing a large, passionate following. Even a simple inspection of their toolbars highlights the philosophical difference between the two approaches. Most of the HyperCard toolbox is devoted to drawing pictures, with the controls reduced to buttons and text fields. It is constantly surprising to me how much mileage is squeezed out of such a restricted set of UI controls. Microsoft, on the other hand, offers a toolbar button for each and every thing you might want to add to an application. They take inverted approaches. Where I might add a generic button in HyperCard , then attach a script which invokes the system file browser, Visual Basic gives me a pre-built file browser control to drag into my app. I prefer Visual Basic's approach of "drag out a rectangle to define a control," especially for buttons and text fields; it feels more modern in its UX. HyperCard makes us add controls strictly by pulldown menu, then we have to drag the corners of the button, with no visual indicators, into the new size. Surprisingly awkward. Visual Basic also earns points in offering a grid to snap elements to position, making it much easier than HyperCard to align and scale elements precisely with one another. Gotta do a lot of eyeballin' on the HyperCard side of things; its grid only works in paint mode. Consequently, laying out something like a calculator is much faster and easier in Visual Basic , at the expense (?) of looking exactly like any other Windows program ever made. (Although the demo calculator doesn't look anything like the actual Windows calculator?) Don't get me wrong, conformance to corporate homogeneity may be exactly what you need at times and Visual Basic can generate something "professional looking" in a jiffy. It is, perhaps, devoid of character, but it also creates something a Windows user can look at and trust. Breaking free of those somewhat rigid constraints requires considered effort in Visual Basic , whereas HyperCard practically begs us to go hog wild. We're firmly in "learning Basic" land here; the application itself doesn't have a whole lot else to it. The panel for exporting our .exe files is about as barebones as one could imagine. There's a color palette, but I'm not entirely clear why; colors for controls can be set in the Properties palette via its own popup color palette. I should also give a shout out to the built-in Help system. Though I wish it were context aware, there's an absurd amount of information available right there in Windows without having to crack open the 10 pound manual. HyperCard has Balloon Help, which is nice and cute, but also anemic; we only get as much explanation as fits in a couple of sentences. Visual Basic's help system gives lengthy, detailed explanations of topics with code samples, is searchable, is bookmarkable (!), has tutorials for understanding the principles of the program, and more. It's quite good! The last week of my training book gets intense with discussions on make files, database connectivity, MDI (multiple document interface), DDE (dynamic data exchange), interfacing with DLLs, and so on. We've only been building throw-away toy applications so far, and I honestly don't feel the book has mentally equipped me for these hairier discussions. It's a pretty significant cognitive leap from the simplicity I feel the product promised. The long and the short of it is, I'm learning enough Basic to squeak by and get a sense of its tempo and grammar, but as a first-time user I find it more overwhelming than HyperTalk. HyperCard and Visual Basic each come with a 600+ page language reference guide. Microsoft also throws in three more manuals, another 2,400+ pages, for good measure. Its language guide would expand to 1,000+ pages in Visual Basic 4. Brevity is the very soul of cowards, I guess was their stance. Though their language reference guides are similar length, Microsoft's is a far more dense, dry tome. Apple spends the first 150 pages talking about "What even is programming?" and the last 150 pages getting into topics outside the scope of HyperTalk; a slim 300 pages to describe the language. Let's examine some concrete examples. Here's how to make the system thrice on the click of a button in HyperCard : Here's how to (ostensibly) do that in Visual Basic 3: Full disclosure: this didn't work, even though it is the example given in the "Programmer's Guide." Something is coalescing the three beeps into one. DOSBox-X issue? Because scripts are kind of "embedded" into their respective HyperCard objects, we don't have to disambiguate subroutines with prefixes; any given script is scoped precisely to its associated GUI object. It's the La Croix of object orientation; just a whiff of a hint of that flavor. HyperCard's approach lends itself better to casual tinkering around, but Visual Basic has an edge in surfacing all functions of our application in the code editor. In HyperCard we have to remember which object contains which code block, or hunt through all objects individually, searching for the code we want. Visual Basic's approach requires unique names for all subroutines. This makes it fairly trivial to trigger events across objects. If we want a button to click another button by proxy, we would have to do something like this in HyperTalk: Sometimes I wish HyperTalk would allow dot-syntax for object specifier chains. In Visual Basic, we simply call the uniquely-named function directly: Where HyperTalk takes a gentle, English-like approach to its language, Visual Basic isn't afraid to be far more "programmery." HyperTalk developers can certainly get into their own weeds trying to figure out the precise incantation to sidestep the interpreter and achieve specific goals. Conversely, Visual Basic developers could quickly find themselves in a world of memory management, DLLs, batch files, and make files. Both developers feel some pain, but one is kind of orthogonal to the other. Your preference may depend on which breed of demon you enjoy slaying. As clearly evidenced by the Voyager series of software and MYST , highly professional software was possible with HyperCard . That said, the upper boundary for Visual Basic feels much higher. As a simple example, with the keyword we can reach in and directly call the Windows Kernel (or any existing) DLL; this of course being the killer feature that triggered Alan Cooper to develop the program in the first place. That's impossible to do out-of-the-box with HyperCard ; it cannot access the Macintosh Toolbox so deftly. Likewise with database data, Visual Basic gives us flexibility in what kind of data to bring in, like dBASE or FoxPro . There may be specialized stacks or XCMDs (plugins) to HyperCard that can assist with these tasks, but nothing native to the program. However, HyperCard provides its own built-in database free of charge, requiring no special effort on the developer's part to leverage it. Building something like an address book is simply a matter of adding some text fields to a card. Those will function like fields in a database by default, and actions like saving/loading user data will happen transparently. Adding search, or something similar, takes a few extra steps, but is conceptually simple through a HyperTalk command like Visual Basic provides a "Data Manager" module, which allows us to create simple Access databases for use as the backbone of the application. This is all explained in detail in the supplemental 300+ page "Visual Basic 3.0 Professional Features, Book 2." Once the database is built, interfacing with its records is straightforward using the "Data Control" tool. When the database is linked in properly, controls like images and text fields can be wired up directly to their corresponding fields in the database schema, called "bound controls." The database widget itself provides buttons to step through records and corresponding data will auto-populate the bound layout elements. If "browsing" is the extent of your database needs, you're in good shape. I imagine most will want to do more than that, perhaps adding fields, or doing search queries. You'll want to steel yourself, because it gets gnarly real quick. I'll just say that the book is 300+ pages for a reason, with talk about complex subjects like Dynasets, Snapshots, Tables, the JET engine, SQL queries, and more. It's far more capable than HyperCard , as we can work with multiple databases in our VB application, access remote databases, and more. That power is paired with an equivalent learning curve, one which is thrust upon any developer who needs even a tiny bit more than the drag-and-drop controls provide. Overall, it would be fair to call the IDE "competent." It contains the tools we need, arranged by palette, and makes certain actions (like adding a button) as easy as a double-click. What's not to like? I think what frustrates me about these tools is how they feel like somewhat careless design solutions to their respective problems. Look at the "Properties" palette, for example. This looks, to my eyes, like a developer was told, "The properties for a selected object should be available for editing." The developer iterated them as a literal list, adding some basic editing niceties, like making a color chooser available when a color property is edited. What I find in practice is that the vast majority of the properties go untouched, especially for something like a Form object, and the ones I actually need require scrolling through a long list to find and edit. Later properties in the list, even those which are common to all controls, shift around in position depending on how many properties a given control has. I'm constantly having to read through that list, scanning for the "Name" property, which is where we set the programmatic name for the control. It's arguably the most important property , and it's playing peek-a-boo. When I make a new form (a "form" is a window; I don't know why they call it a "form") I have a few things I need to set right off the bat: the size, the title, and the programming reference name. After that, sometimes I want to set the background color. We'll ignore the fact that property names don't make sense; naming conventions had perhaps not yet been firmly established in an era when the terms UI and UX had not yet become common vernacular. From a pure, "What is the user most likely to need?" point of view, this simple alphabetical list is the laziest solution to the design challenge. Fair point, HyperCard's lack of any properties palette was more lazy, but this is version 3 of this product. I frankly (perhaps unfairly) expect more considered effort from a first-party solution. My frustration extends to the main toolbox as well. It's just a bunch of buttons with no organizational structure applied. Tooltips, similar to what we understand today, were introduced with Macintosh System 7 as "Balloon Help" the same year VB3 released, so I can't fault Microsoft for "failing to implement" them in this release. Still, icon-only is a lazy way to handle it, when the goal is to shove as many icons into the toolbar as possible. Asymetrix Toolbook 3 , a similar visual IDE for Windows development, provides more robust, logically arranged tools for the job. Here's the text editor and object properties panels. Note in particular a few things: Visual Basic itself contains a similar contextual help in other parts of the application, like its "Crystal Reports" tool, making its absence in the main app even more frustrating. This kind of haphazard application of tools and controls feels sloppy, which reminds me of something I wanted to talk about. While going through the official manuals for Visual Basic , something kept bothering me. I couldn't put my finger on it at first, but once I saw it, my eyes were forever cursed . This is a small grievance, "petty" some would say, "a colossal waste of mental resources" others may scoff. But what's a tech blog without a certain level of pedantry? I'm not above pedantry. Here we see the Visual Basic 3 manual is laid out in Helvetica and Times. Man, I'm already bored. Anyway, beyond the utterly pedestrian font choices (in fairness, they did have to lay out 3,000+ pages of this stuff), something seems "off" about it. In particular, that Helvetica looks malformed, with sloppy kerning and unbalanced strokes. Let's take a closer look. Helvetica Neue doesn't match, and Arial (my original suspect) is ruled out by the end caps on the capital "C". Helvetica Condensed is also not right. Wait, I see what's happening. It's the same issue I have with the user interface, manifested in the manual. This isn't Helvetica Condensed, it's Helvetica physically squashed into a fake condensed version. The richest man in the world couldn't afford to buy a proper condensed font for his company? "Or is this indicative of a deeper issue?" he asked, slipping back into his pop-psychology armchair. It smacks of "good enough," never striving for "great." That kind of sums up my feelings toward Windows and Windows applications of this period. The stuff worked, and had obvious success, but never seemed to be borne of thoughtful consideration. Did that inattention to detail come from cost-cutting measures, or perhaps some kind of cultural blindness? Were the deficiencies seen and ignored, or simply not seen at all? And that reminds me of something else I wanted to talk about. In the PBS documentary series, Triumph of the Nerds , Steve Jobs famously said of Microsoft, "They have no taste." I genuinely think Bill Gates could not understand the meaning of Jobs's accusation. Or rather, he couldn't fathom why "taste" should enter into his calculus whatsoever. Having no taste didn't stop him from becoming the richest man in the world. What does "taste" have to do with stockholder value? When Apple teased with a new release of OS X, "Redmond, start your photocopiers," I think Gates was thinking, "Of course we will. Thanks for the free R&D." He bristled at being publicly chastised for copying , but my read on that is he really wanted to say, "So what if we copy Apple? Why shouldn't we? Look at our success and tell me it hasn't been a good strategy." What Jobs saw as creative bankruptcy, Gates saw as business efficiency. Being asked to frame his success on Jobs's terms ruffled Gates's feathers. Jobs said, and I agree, that innovation means saying "no" to 1000 things before saying "yes." "Process" is that very action. "Process" is the pruning of the possibility space. It's the self-awareness to distinguish "good enough" from "great." It's when you step away from your work, give it the critical stink eye, and apply taste . That's an impossible task if one has no taste to begin with. So what's a tasteless corporation to do? While Microsoft may have not cared too much about process, they had manufacturing down cold. Put in PenPoint OS, out pops Windows for Pen Computing. Put in OS X 10.3, out pops Windows Vista. Put in Java, out pops J++. Put in a Dreamcast, out pops an Xbox. Even today, similar "factory production" charges are levied against them. I'm not suggesting they "stole" ideas so much as they simply seemed content to let others do the hard work of saying "no" 1,000 times. While they may have shortcut the creative process, they still had to learn how to manufacture products. In so doing, they accidentally picked up a little taste along the way, which would lead to pretty good stuff from time to time. It's been part of the fabric of the industry for decades, and now the torch of manufacturing tasteless product from the creative work of others has been passed on to generative AI. To scale , no less. The ramifications weigh heavily on my mind, especially when someone publicly calls for the absorption of my work into the generative AI apparatus. I'm both flattered and appalled. On average, how many times do you think I rewrite the introductions to these posts? How many thousands of words have I thrown away to reach something approaching what I wanted to actually say? I tend to rewrite intros 3 or 4 times, and I mean that truly; each rewrite is radically different from the others. In this post alone, I have thrown away some 5,000 words. Some might think those 5,000 words are the cost of the process, but that's not right. They are the process. The unpublished words are the important ones. Those are the words that got me to these words. Knowing that, throw any creative work into the generative wood chipper and it should be obvious why what comes out cannot live up to the original. It's lacking the 1,000 nos. I'm disappointed in the ending of this book. Day 21 comes and goes without even a hint of acknowledgement that we've made it through the gauntlet. At the end of it all, we also haven't built anything of value. Every chapter created little baby programs to illustrate specific concepts; no project built upon a previous project except for a few shallow, superficial glow-ups. Contrast that with HyperCard , where we had a full-fledged address book, with database, search, custom art, and save/load. With Visual Basic , I never felt that same spark I did with HyperCard . Visual Basic seems great for when you have a strong idea of what you want to build. However, its lack of drawing tools and "don't worry about it, I've got you covered" database curtail creative exploration far more than I would have predicted at the beginning of my studies. Not having to worry about those details opens up a wider world of "lemme try something real quick" experimentation and iteration. In an ideal product, I'd combine the prototyping strengths of HyperCard with the professional-strength of Visual Basic . Then, later we could swap out the default database with Access, or export the placeholder drawings as image assets for a professional artist to clean up in another revision. I cannot personally find a place for Visual Basic in my heart, but I can absolutely understand why it took off. It filled a major gap in the programming landscape, helping amateurs and pro-ams build tools for themselves, and even throwing a lifeline to a generation of COBOL engineers needing to transition ASAP. Like Apple with HyperCard , that gap was re-opened by the discontinuation of the product, abandoning a whole fleet of developers and, perhaps just as importantly, potential developers. I suppose nothing lasts forever, but these companies are worth multi (choking on the word again) TRILLIONS of US dollars. At valuations like that, with the fealty they demand from us, I consider it a moral imperative for them to provide excellent tools which empower the widest possible breadth of users' skill levels. Not providing such tools is a choice . Considered from another angle, I'll leave you with this open question. What software do Apple and Microsoft provide today that will be spoken of, with the same reverence as HyperCard and Visual Basic, 25 years from now? Ways to improve the experience, notable deficiencies, workarounds, and notes about incorporating the software into modern workflows (if possible). With Visual Basic 3, 2, 1, and DOS 1.0, the applications you build are 16-bit only and are therefore relegated to running only in virtual environments on 64-bit Windows. If this fits your modus operandi, you're in good shape. If you're hoping to keep it old-school, but still want the option of running your creation on modern hardware, then you'll want to get Visual Basic 6 up and running in Windows 2000? XP? I tried it in Windows 98SE and it wouldn't launch. VB6 builds 32-bit applications as standalone, compiled executables, can connect to the Internet, and produces builds which run on Windows 10/11. Note that Windows 11 promises to run applications built with VB6 , but does not promise to run VB6 itself. However, I gave it a shot and though there were issues with the install, and the IDE acts a little weird, and it complains on launch about missing OLE files, it did run and I was able to build an executable on Windows 11. For funsies, here's Gates and Jobs demonstrating their respective visual programming environments. Gates giving a subdued demo of the just-announced Visual Basic 1.0 . His voice cracking at 0:33 is adorable . Jobs had just returned to Apple after they bought NeXT, and here he's showing the technology Apple has bet its future on. We know it today as Xcode , but it started life as Interface Builder . The line he drew between components in the demo was called a "binding," something that has conceptually resurfaced in SwiftUI. DOSBox-X 2026.01.02, Windows x64 build. CPU set to Pentium DOS reports as v6.22 Host system folder mounted as drive C:\ holds Windows Windows 3.1, basic installation 1024 x 768, 32K colors under DOS reports total RAM, but Free only reports . Good enough for today, but 16-bit Windows should be able to register 4MB, not just 2MB. A few extra applications for comparative/convenience reasons: Toolbook, Actor, ObjectVision, Acrobat Distiller Visual Basic 3.0 Reports 386 Enhanced Mode enabled Reports free RAM In lieu of tooltips, at the bottom of the current window we have a contextual description of the current tool, much like Bank Street Writer and Lotus 1-2-3 . The text editor includes indent/outdent tools, can set our editing font of choice, waits to check syntax until we ask it to, and even includes a simple "build a function" utility to wire up common tasks to common UI events. The properties panel is laid out hierarchically, keeping the most-needed stuff front and center, while demoting less-used options to secondary emphasis. DOSBox-X ran everything smoothly and without issue. I did not install Windows on top of real DOS, though. I relied on DOSBox-X's implementation. This may account for a couple of strange issues, outlined below. I experienced one crash in Visual Basic 3 , when accessing the Help system. Issuing a looped command resulted in only a single system beep. My guess is that something in the emulated environment is suppressing this. I could never get databases to connect, even the ones that ship with Visual Basic , let alone any personal data carried over from previous database explorations. It may be the result of DOSBox-X using an emulated version of . Strangely, I saw it work once and then it stopped working as suddenly as it started and never worked again. An installation of Windows on a proper installation of MS-DOS might fix this problem.

Ruby

Java

0 views

alikhil 3 weeks ago

Goodbye, nix-darwin!

Two years ago, my work laptop was force-updated by the IT team and got broken so badly that I had to set it up from scratch. I was frustrated. As an engineer, I don’t like repeating the same actions multiple times, and this gave me the motivation to set up my laptop as code. Some of my friends and colleagues were using Nix. When I decided to take a look, it looked promising: NixOS as a fully declarative OS, the nixpkgs package repository, reproducible environments, and the idea that almost everything can be described as code. There was also support for macOS: nix-darwin and home-manager, including Homebrew integration. I decided to give it a try. I set up my work laptop with it. Later, when I bought a Mac mini for personal use, I was able to reuse most of the config there too. This was cool. I could keep my shell, packages, editor settings, and many small tools in one repository. It felt like I finally had a proper source of truth for my machines. But it also made simple things more complicated. Before Nix, when I needed to add something new, I could install almost any app by running: With nix-darwin, the same action became a source code change followed by a slow rebuild command. Every change was tracked, reproducible, and easy to reuse on another machine. That was the whole point. But sometimes I just wanted to install a tool, try it for five minutes, and move on. Waiting for a rebuild each time made the whole setup feel heavier than I wanted. Over time, the cost became more visible in three places: updates, breakages after updates, and tools that expected a normal mutable macOS home directory. Updating packages was probably the most annoying part. In my setup, I usually did not update one small app directly. I updated the whole Nix configuration: flake inputs, nixpkgs, nix-darwin, home-manager, and then applied everything with . A small update could turn into a 30, 40, or 50 minute process. Nix had to evaluate the configuration, fetch new package versions, download or build what changed, update Homebrew packages through the generated Brewfile, and activate the new generation. I would start with “I need a newer version of this app” and end up maintaining my whole laptop. A new nixpkgs revision could change package options, rename something, move a config path, or slightly change how a module worked. Home Manager and nix-darwin also had their own options and behavior that changed over time. So after a big update, I often had to read error messages, search through changelogs or GitHub issues, and patch my config before the system could switch to the new generation. The rollback story is nice, and it is one of the strongest parts of Nix, but I still had to spend time understanding why the new generation did not build or activate. Package updates started to feel like small migration projects. nixpkgs itself was also inconvenient for me. It is a huge community-maintained package repository, and I respect the amount of work behind it, but new releases do not always appear there quickly. Sometimes the package I needed was behind the latest version. Sometimes it was missing a feature that had already been released upstream. And sometimes installing the latest version was possible, but required overrides, flakes, or other Nix-specific work that I did not want to do for a simple desktop app. This problem became much more visible during the last half year, when many AI tools started appearing. A lot of these tools are distributed through installer scripts: curl this shell script, run it, let it modify your shell config, add something to , install a binary somewhere, and maybe patch your environment. On a normal macOS setup, this usually works. But my shell config was managed by Nix and Home Manager. Some files were symlinks to generated files from the Nix store, some paths were not supposed to be edited manually, and the installer scripts had no idea about that. They tried to modify files that were managed elsewhere, failed, or produced broken changes. So while everyone else could try a new tool in a minute, I often had to stop and translate its installer into Nix config first. The setup stopped feeling helpful and started feeling like extra work. The breaking point happened when I changed my job. I tried to set up my new corporate laptop with Nix, but the company’s internal tooling wasn’t ready for that at all. Some tools expected the default macOS layout. Some scripts assumed Homebrew paths. Some security and management software did not play nicely with my setup. I spent some time trying to make it work, but it felt like I was fighting the company environment instead of doing my actual job. So I gave up and set up the laptop manually, without Nix. I kept using Nix at home for a while. But after that, my work and personal configurations diverged a lot. The main benefit of my setup was supposed to be reuse between machines. Once that disappeared, I had much less motivation to maintain the Nix config while still suffering from all the drawbacks described above. Eventually I decided that enough was enough. I pointed Codex to my Nix repository and gave it a task to de-Nix my Mac mini. Codex wrote a few migration scripts. It helped me move Homebrew packages out of the Nix-managed setup and back into normal Homebrew. It also helped me extract shell configuration into regular dotfiles. My Homebrew is Nix-free now, and I can directly edit my file again. I’m still scared to remove the directory though. It stays there for now. I have no problem with that. nix-darwin is an interesting tool. I still understand why people like it. Having your machine described as code is a nice concept. But for me, it was not user-friendly enough for daily macOS usage. I don’t want to debug my laptop configuration every time I need a new app. I don’t want to wait for a rebuild just to install a small tool. And I don’t want my personal setup to fight with corporate tooling. At this point, I would rather reinstall my system from scratch manually than spend 30 minutes installing a new Nix package.

DevOps

0 views

Unsung 3 weeks ago

“It’s rare that printing nothing at all is the best default behavior.”

Aanand Prasad, Ben Firshman, Carl Tashian, and Eva Parish put together Command Line Interface Guidelines for people who write command-line tools. I like that it harkens and links back to other writing, and is also pragmatic: shares good parameter-parsing libraries, commonly used options, and so on. Here are some good principles that caught my attention: Display output on success, but keep it brief. Traditionally, when nothing is wrong, UNIX commands display no output to the user. This makes sense when they’re being used in scripts, but can make commands appear to be hanging or broken when used by humans. For example, will not print anything, even if it takes a long time. It’s rare that printing nothing at all is the best default behavior, but it’s usually best to err on the side of less. By default, don’t output information that’s only understandable by the creators of the software. If a piece of output serves only to help you (the developer) understand what your software is doing, it almost certainly shouldn’t be displayed to normal users by default—only in verbose mode. Catch errors and rewrite them for humans. If you’re expecting an error to happen, catch it and rewrite the error message to be useful. Think of it like a conversation, where the user has done something wrong and the program is guiding them in the right direction. Example: “Can’t write to file.txt. You might need to make it writable by running ‘chmod +w file.txt’.” Signal-to-noise ratio is crucial. The more irrelevant output you produce, the longer it’s going to take the user to figure out what they did wrong. If your program produces multiple errors of the same type, consider grouping them under a single explanatory header instead of printing many similar-looking lines. Consider where the user will look first. Put the most important information at the end of the output. The eye will be drawn to red text, so use it intentionally and sparingly. Make it recoverable. If the program fails for some transient reason (e.g. the internet connection went down), you should be able to hit <up> and <enter> and it should pick up from where it left off. There’s a lot more inside . (The document is undated, but I believe the effort started in 2020. It seems to still be updated via GitHub , where you can also send in your suggestions.) #command line

0 views

Blargh 4 weeks ago

I fixed shell pipes

In a previous post I made pipes in unix shells more reliable. Well, it had some drawbacks. I’ll summarize the problem, the failed previous version, and then show the new and improved one. Downstream processes in a unix shell pipe cannot know if the upstream finished successfully, or exited with an error. This means that it can’t know if it should “commit” the data it received. Example uses: In both of these cases you want the right hand side to STOP, and not finalize the upload or commit the transaction. This works fine for simple cases, but doesn’t support or per-command environment variables very well. And I don’t want to invent a complex language, so my replacement took a different path. wp on github . instead wraps the input and/or output with a very minimal encapsulating protocol. This allows normal data to pass through, but still allows the downstream to get as metadata. If the data stream ends before receiving the marker, then do not commit . The wrapped downstream child process sees this as remaining open, and instead it’s getting terminated with a signal. can either encapsulate when it wraps something that o utputs data, with , or decapsulate and receive the EOF marker when it’s handling i nput data, or both.

DevOps

0 views

Giles's blog 4 weeks ago

Flax debugging: making a hash of things

I was debugging an issue with a JAX/Flax NNX training loop the other day, and found a neat little trick to help debug it. Specifically, I wanted to see if the issue was with my model, my loss function, my optimiser settings, or the "plumbing" of the training loop itself -- were gradients actually coming through and being applied to the parameters? I could print out the loss and the gradients, but printing out the parameters to see if they were changing was unhelpful -- any given update might only change a small number of parameters, or might change them such a small amount that I'd not notice -- especially given that the model had 77 million of them! Let's take a look. I am building an LLM from scratch in JAX and Flax NNX, and at this stage I'm trying to get the training loop right. As a simple test, I've just implemented the "shell" of the LLM -- the token embeddings on the input side, and the final linear layer for an output head, wired directly together. My plan was to train that so that given a sequence, instead of predicting next tokens for each position, it would "predict" the sequence itself -- that is, I might train it with the input ...and the target ...rather than the normal setup for an LLM, where you feed it ...and give it targets of So, in LLM terms, I'd be training a model to project from vocab space to a learned embedding space where each token had a distinct-enough embedding for the output head to be able to reliably project back to logits in vocab space. There's a bit of background here if that was all Greek to you . Here's the core part of the code I was working with, the function, which seems to be the traditional JAX name for the JITted part of your code that does the forward pass through the model, works out the gradients, and then applies them to update the model: I'd based it on the "Basic Usage" example that's currently right there on the front page of the Flax site. Seasoned Flax veterans will probably spot the issue right away, but it wasn't obvious to me -- so it was time to dig in. The problem was that loss was not dropping -- indeed, taken to two decimal places, it was stuck at 10.82. The digits to the right of that changed for each batch, but the first four did not. Now, this model was using the GPT-2 tokeniser, and 10.82 is exactly the loss that you'd expect if the model was essentially guessing randomly -- if you convert it to perplexity by calculating e 10.82 , you get about 50,011 -- which is very close to the GPT-2 vocab size of 50,257. Perplexity is, loosely, the number of tokens that the model was trying to choose between for a typical input -- so a perplexity equal to the vocab size is what you'd expect of a random model that is getting it right about one in 50,257 times. That said, getting that loss consistently was a solid validation of my loss function! It's vanishingly unlikely that it would have been getting that specific number so consistently if I'd made a mess of that. The tiny variations I was seeing in the third and subsequent decimal places would make sense, as they could easily be due to the variations in the contents of the different batches. So was it that the gradients were somehow zero, or NaNs, or something else that couldn't be usefully applied to the model by the optimiser? I printed them out in the function (removing the decorator, as otherwise the s would only get executed in the initial JIT pass through the function to compile it -- not when it had actual data 1 ). The result was values like this: Those looked plausible enough -- pretty small, but not so tiny that I'd expect them to have no effect at all with my learning rate of 0.0014. It was time to dig into the training loop's plumbing. The obvious suspect was the update step -- was that call to actually changing the parameters at all? Flax's NNX API is a bit odd compared to the normal JAX functional way of doing things . In vanilla JAX code you would expect to do something like this to apply gradients: That is, you get the new parameters by applying a transformation to the old ones. NNX, by contrast, is more PyTorch-flavoured. It updates the parameters in-place, using a function with a side effect of mutating one of its parameters: ...rather than something more functional like this imaginary API: I could easily imagine that I'd got something wrong that would break that in-place update, as it has the feel of something that would have to be quite delicately implemented on top of a functional system like JAX. But how could I see whether the parameters were changing, when there were 77 million of them and they would be being updated (based on gradients like -2.6879393e-06 and a learning rate of 1.4e-3) in the ninth decimal place or beyond? Printing the arrays out was a non-starter! After a little thought, I realised that the solution was to use hashes. Even tiny changes in the parameters' values would change their hashes drastically. So if the parameters were not being updated, as I suspected, I'd see constant hashes. If they were being updated, even by a minuscule amount, then the hashes would change. This GitHub discussion pointed me in the right direction: if I could get the parameters as pure JAX arrays, I could do this: ...where is just . That would produce a hash that was stable for the life of this run -- the same parameters would always have the same hash, and different ones would differ, just as we want. It could vary from run to run (Python uses different hash seeds in each new interpreter), but that wouldn't matter for this kind of debugging. I wasn't sure what the structure of my Flax model's parameters was, but printing them out in the training loop told me: So, guided by that, I added these lines to the training loop: Obviously copying the arrays around and converting them like that would slow things down, but for debugging purposes, it looked solid. I kicked off the training loop, and the problem was clear: ...and so on. The hashes were not changing, so the model's parameters were not being updated, even by a tiny amount. Gotcha! The problem turned out, as I had suspected, to be related to the in-place updates that NNX does. Like I said earlier, I'd based my training loop on the "Basic Usage" example on the Flax site -- but I'd messed up one important thing. I had this: ...and they had this: You can see a number of differences -- for example, they're baking the inputs and targets into the lambda they're using for the loss function through a lexical closure, and that means that they're only passing in the model to the version of it wrapped in . But none of that matters! The real difference is actually nicely highlighted with a comment, but I'd completely managed to miss it. Right at the start, where I had , they had this: It 100% makes sense that in order to support this kind of non-functional, in-place updating of the model's parameters, you have to have a modified version of the JIT decorator. And I was just using the standard, functional pure-JAX one. Fixing that fixed the problem: The hashes were changing! And even better, if you scroll to the right you'll see that loss was slowly dropping. After 10k or so iterations, I was seeing 0.000: I had my do-nothing "LLM" working. A satisfying debugging journey -- and while I don't think I'll make this specific mistake in the future, I think that the parameter-hashing trick is actually a really useful trick for the toolbox. If you're uncertain as to whether your parameters are being updated, just looking at them probably won't help. But looking at their hashes can help you find out whether anything is changing. And I think that the pattern that I used to zoom in on it is a useful one, too. I always track loss, so it's a good starting point (indeed, seeing that it wasn't falling was what told me that something was going wrong). But checking that it has a sane -- or ideally, as in this case, a meaningful -- value is a nice sanity check that we have a working loss function and a model that isn't doing something completely pathological. Moving on from there to checking that some kind of gradients are flowing through is a solid next move (and might become increasingly interesting with deeper models where they can vanish or explode ). Then finally we can check the parameters -- in particular, are they changing? 2 Let's see how many new tricks I pick up as I work through this LLM project. I always forget that exists -- I could have used that instead, and kept the JIT. ↩ Something's slightly broken in my brain and I keep reading that as "is our parameters changing" in George W. Bush's voice . Maybe I can stop that from happening by inflicting it on my readers instead. You're welcome. ↩ I always forget that exists -- I could have used that instead, and kept the JIT. ↩ Something's slightly broken in my brain and I keep reading that as "is our parameters changing" in George W. Bush's voice . Maybe I can stop that from happening by inflicting it on my readers instead. You're welcome. ↩

Python

0 views

Takuya Matsuyama 1 months ago

I made a Claude Code session manager for tmux

Hi, it's Takuya. I'm happy to introduce a tool for managing multiple Claude Code sessions in tmux. Here is a demo video: First, let me share a bit about the journey that led me to build it. Or, you can just jump right to the repo here: Recently, I've read Christoph Nakazawa's blog post . He emphasizes the importance of the toolchain in boosting feedback loops and developer productivity to work efficiently with coding agents: It inspired me to focus on updating my tool setup and DX in my project. First, I focused on improving tool performance. For example, I recently migrated Inkdrop's build toolchain from webpack + Grunt to (Vite 8 + Rolldown). It made the production build 10x faster, and the dev build now launches almost instantly. Also, I installed , a Go-based rewrite of TypeScript. It's super fast and has improved my AI development pipeline, because the AI often runs a typecheck on every task. Similarly, I migrated my linter from to , and my formatter from to . Next, the review process. I've been using the lazygit integration of snacks in Neovim. When I review changes across a lot of files, I use Yanuo's codediff.nvim . Since I often jump around files, I added an option to automatically open the diff when changing the selection in the explorer without pressing Enter (it got merged into codediff.nvim . Thanks, Yanuo🙏). These improvements have been great for boosting feedback loops in a single project (bonus: without ever leaving the terminal screen or touching the mouse). As Christoph says, working with coding agents is like working in a large organization. It means you have a bunch of engineers to manage. The next pain point in my workflow is managing multiple Claude Code sessions. I usually run multiple Claude sessions simultaneously, because I have a lot of modules and libraries to maintain, e.g., the desktop app, mobile app, theme module, markdown renderer, React Native libraries, etc. I shared a tmux tip on how to run Claude Code in a tmux popup window with persistent sessions . It allows me to have fewer tmux windows instead of separate windows for each Claude Code session. But it's been annoying to check whether any sessions are finished or need my answers by switching windows and opening each session in a popup window. So, I created a tool to manage Claude Code sessions in tmux. I published it as a tpm plugin so you can quickly try it. It gives me a single picker over all my running Claude sessions, so I can see which ones need me and jump straight to them instead of switching windows. In a nutshell, it supports: It's built entirely on tmux primitives and shell scripts — there's no background daemon or extra process to keep alive. Here's the gist: Because everything is stored on the tmux sessions themselves, the state survives detaching, reattaching, and closing the picker — there's no separate database to get out of sync. It's a tpm plugin, so installing it is one line in your : Then hit + I to install. You'll also need fzf (it powers the picker UI) and the CLI — both of which you probably already have. The whole workflow comes down to two keybindings: (Both keys are configurable — see the README if and are already taken in your config.) This is the part I built the tool for. Hit + from anywhere and you get a list of all your running Claude sessions, each with a colored status dot: The sessions that need you (waiting and idle) float to the top, so a glance tells you where to go next. On the right is a live preview of each session's screen, so you can see what Claude is actually doing without leaving the picker. From there: Out of the box, the picker lists, previews, jumps, and kills — but the status dots stay until you wire up Claude Code hooks . The status is the best part, so it's worth the two minutes. The hooks stamp each session's state onto its tmux session as Claude works: You just add a small block to — the exact snippet is in the README . Claude Code picks up hooks dynamically, so there's no restart needed; your running sessions start reporting status on their next event. That's the whole thing: + to start sessions, + to see who needs you, to jump in. That's it. Hope it is useful for your terminal workflow with coding agents! Enjoy AI coding. 🔢 A central picker ( + ) listing every running Claude session. 🟢 Live status per session — / / — driven by Claude Code hooks, so you instantly see which need you. 👁️ A live preview of each session's screen right in the picker. 🎯 Smart jump — selecting a session switches your client to the window it was launched from, then resumes it in a popup over it. 🚀 A launcher ( + ) that opens/attaches a Claude session for the current directory. ❌ Quick kill ( ) of finished sessions from the picker. Each session is a plain tmux session. When you launch one, the plugin starts a detached session named running . Because the name is derived from the path, launching again from the same directory just re-attaches to the existing session instead of spawning a duplicate. State lives on the session itself. The Claude Code hooks stamp a option ( / / ) onto the tmux session whenever Claude changes state. Nothing polls in the background — the status is written the moment it changes. The picker is fzf. It lists every session, reads each one's state for the status dot, and shells out to for the live preview. When you pick one, it switches your client to the window you originally launched it from (remembered in ) and resumes the session in the popup. + — launch. Spins up (or re-attaches to) a Claude session for whatever directory your current pane is in, and drops you straight into it in a popup. Run it once per project, and you've got one session per repo, each named after its path. + — the picker. Opens an fzf popup listing every Claude session you've launched. 🔴 working — busy, leave it alone 🟡 waiting — needs your input (a permission prompt or a question) 🟢 idle — finished its turn, your move jumps to the highlighted session. It switches your client back to the window you originally launched it from, then resumes the session in a popup right there — so you land back in its context, not some random window. kills the highlighted session — handy for clearing out finished ones. Type to filter, / to move around. Standard fzf.

JavaScript

0 views

xenodium 1 months ago

agent-shell 0.55 updates

It's been a little while since my last agent-shell update , so let's go through the latest highlights as of v0.55. agent-shell is a native Emacs mode to interact with AI agents powered by ACP ( Agent Client Protocol ). If you noticed slower project activity in April, this is why . I'm getting better at the new 24-hour job , so I've resumed working on agent-shell . I'm still chipping away at the backlog that built up while I was away, but if there's anything in particular you'd like me to look at, feel free to ping. With Anthropic's SDK subscription support changing , Google's Gemini CLI deprecation , and Antigravity's unclear support for the Agent Client Protocol (ACP) , vendor-neutral tools matter more than ever. Luckily, is built on ACP , which sidesteps the problem. When a vendor changes course, you can swap providers and keep using your preferred tool. No need to reshape that hard-earned muscle memory. On that note, the list of agents supported by continues to grow. Here's a list of the latest agents now supported by . Speaking of vendor-neutral tools being more important than ever, there are a couple of ways to help keep going. Some cost money, others just a click. All are appreciated ;) has been attracting quite a few users. It's nice to hear folks are using on a daily basis. They are often relieved exists as an alternative to AI-tools commonly mandated at work. Those tools have well-funded engineering teams behind them, while is just me, an indie dev ;) Time spent on is time away from work that pays the bills, so if it's useful to you, please consider sponsoring the project. Every individual sponsorship genuinely helps keep the project going. And if your employer benefits from your use, they're typically in a position to contribute at a scale individuals can't, so nudge them to chip in too. Hey, I'm looking at you, folks at Google , GitHub , GitLab , NVIDIA , Oracle , Red Hat , Yelp, Venmo, ARM , Spotify , Augment Code , Hinge, Mercury, Nubank, Veeva… Some of you are using . Nudge your employer ;) Anthropic offers 6 months of free Claude Max 20x for qualifying open-source projects with at least 5,000+ GitHub stars. Starring agent-shell costs nothing and can save me some money. We're only a 5th of the way there ;) so if you don't mind a couple of clicks, the project can really use another GitHub star . Speaking of GitHub stars, is now my most popular Emacs package, recently overtaking chatgpt-shell . agent-shell now ships with a brand new, more performant inline markdown renderer. This is the biggest internal change in some time. Enabled by default via (moving away from the overlay-based renderer in shell-maker). Table content is now accessible. Point can land on any cell, which wasn't possible with the previous overlay implementation. In addition, tables are now also navigable: and move between cells. Source-block syntax highlighting is now on by default. The per-snippet copy button is now keyboard-accessible too (previously mouse-click only, due to the overlay implementation). Blockquotes now render in both shell and viewport. More importantly, you can select text in either a viewport page (or the shell itself), press (for reply) and the selection becomes a blockquote in a fresh prompt. Session restoration got a meaningful overhaul ( #605 by @nhojb ), now exposed via , with four levels: Feature availability is agent-specific, requiring either or request support. degrades as needed, ultimately falling back to creating a new session. Note that anything but verbosity is fairly new, so please report bugs or rough edges . Relatedly, now defaults to , and has been retired. You can now fork the current session, starting a new shell that shares the conversation history so far and diverges from there. Invoke via . You can now restart the current shell anew (drop history) via or reload (keep history) via . The new and commands create agents anchored at either or a temp directory. Both are also reachable via . acp.el #20 by @martenlienen landed support for ACP connections over TRAMP, now making it possible to drive remote agents from . Pair it with agent-shell-tramp for the user-facing integration. Viewport interaction continues to be my primary way to interact with agents. It is focused (see only the latest interaction), fast (single-key bindings: = yes, = continue, = more…), and offers a richer editing experience (dedicated prompt-crafting buffer). The viewport is just a viewport to shell content. You can have your cake and eat it too, by jumping to the related shell buffer if needed. From a viewport, you can press to reply to the latest agent response. In the past, you could only reply to idle agents. You can now press to reply to busy agents too, automatically queuing requests on submission. A new (basic) lets you edit list-style content inside the viewport. Some commands prompt you to pick one of your active shell buffers (e.g. ). The picker now shows extra context for each buffer to help you choose. The same mechanism is now used by the new command. More on the underlying API later. Folding got smarter ( #608 by @codeluggage ): Together they replace the previous , which is now an internal primitive. You can now press from a viewport to quickly send a "continue" request, joining the rest of the single-key reply shortcuts: Tool call status is now rendered as a compact icon-based label by default ( ). ships several alternatives, picked via . To get the previous word-based label back: You can now set a default model and session mode for Codex via and ( #405 by @robjgray ). Both must match an ID from Codex's "Available models" / "Available modes" listings. Headers had a few rendering hiccups on Emacs 31. These are now fixed ( #588 and #590 by @nhojb , #463 by @ftlio ). Warnings from deprecated usage were also cleared. is now a supported clipboard handler for pasting images on Wayland ( #461 by @martenlienen ). Similarly, pasting clipboard images now works on Windows via PowerShell ( #572 by @repelliuss ). The graphical header got minor tweaks here and there. For example, thought level is now displayed in the header. It can be changed via as well as menus ( #601 by @martenlienen ). agent-shell now supports ACP session config options ( #553 by @greggroth and #613 by @catern ). Bind (or call ) to pick from the options the agent advertises. Broadcasted as and available externally via . You can now skip interrupt confirmations by unsetting ( #424 by @emil-e ). You can now resume an existing session by its ID via ( #332 ). Primarily useful for external integrations. You can now use to tag or transform outgoing requests. now broadcasts events ( #509 by @arthurgleckler ). returns the underlying shell buffer for the current context. jumps to the latest prompt/response pair, while returns the interaction at point as data. returns , , or status for any shell buffer. The agent-supplied session title is now exposed via the event (delivered via ) ( #559 by @smagnuso ). Can be handy for buffer names, bookmarks, or recent listings. The family of third-party packages keeps growing. Recent additions: Thank you to all contributors for these improvements! Beyond what's showcased, I've poured much love and effort into polishing the experience. Interested in the nitty-gritty? Have a look through my regular commits . If agent-shell is useful to you, please consider sponsoring the project. I'm now back to working on daily. LLM tokens aren't free, and neither is the time dedicated to building this stuff (especially as an indie dev). I also have bills to pay ;) Unless I can make this work sustainable, I will have to shift my focus to work on something else that is. ✨ Sponsor agent-shell ✨ CodeBuddy ( new by @illidan127 ) Factory Droid GitHub Copilot CLI Hermes ( new by @yitang ) Kimi Code ( new by @nicolaisingh ) Mistral Vibe (default): title only, so restore is fast and quiet (needs support). : render the last prompt turn (needs support). : render the first and last prompt turns (needs support). : replay the whole conversation (needs support). toggles the fragment at or near point (DWIM). cycles globally between all-expanded and all-collapsed. : replies "continue" (new) : replies "yes" : replies "more" : replies "again" … : replies with the corresponding numbered choice : opens the reply compose buffer : same as , with the agent response quoted agent-shell-knockknock : Notifications for via knockknock.el . agent-shell-notifications : Desktop notifications for events. agent-shell-hud : Real-time status overlay via a floating dashboard. agent-shell-pet : Codex-like pets that broadcast agent-shell session states. agent-shell-tramp : Tramp integration for . agent-circus : Run AI coding agents in sandboxed Docker containers. #308 : Fix heartbeat nil value crash in timer and busy indicator ( @ElleNajt ) #340 : Add documentation about the OAuth Anthropic authentication ( @chemtov ) #397 : Add detailed context usage indicator mode ( @emil-e ) #398 : Expose outgoing-request-decorator as a defcustom ( @emil-e ) #405 : Codex defaults for session mode and model ( @robjgray ) #408 : Use diff-command for diffing ( @timfel ) #413 : Track and manage diff buffers for each permission request ( @emil-e ) #418 : Add defvar for agent-shell-mode-hook + test ( @emil-e ) #420 : Replace all references to "Claude Code Agent" with "Claude Agent" ( @jinnovation ) #421 : Add agent-shell-clear ( @Makesesama ) #424 : Add agent-shell-confirm-interrupt option ( @emil-e ) #425 : README: Update all references to Claude Code ( @jinnovation ) #429 : Viewport attachment fixes ( @nhojb ) #438 : Fix for structured input from toolCall.rawInput.plan ( @timfel ) #442 : Add related project to README.org ( @rpoisel ) #445 : Use project-name instead of default-directory in header ( @bcc32 ) #446 : Droid: use native acp client and support default model and mode ( @kohnish ) #450 : Fix restart using wrong default-directory ( @zackattackz ) #453 : Ensure that viewport compiles ( @martenlienen ) #457 : Prefer cache directory over tmp for caching ( @martenlienen ) #460 : Unhandled method returns an error, unblocking client ( @0x6362 ) #461 : Add wl-paste as a Wayland image handler ( @martenlienen ) #463 : Fix header text invisible when font-get :size returns 0 ( @ftlio ) #469 : Do not create a file if no image in Wayland clipboard ( @martenlienen ) #473 : Caching project files completions for improved performance ( @Gleek ) #477 : Handle non-text content in user_message_chunk during session load ( @Gleek ) #483 : Add CodeBuddy agent support ( @illidan127 ) #489 : Snapshot session strategy in any case ( @timfel ) #492 : Update claude-agent-acp repository ( @Gleek ) #498 : Normalize missing MCP transport collection fields for ACP compatibility ( @CsBigDataHub ) #503 : Add agent-shell-macext to related projects ( @cxa ) #509 : Add ( @arthurgleckler ) #513 : Quote reply to the complete response with ( @martenlienen ) #515 : Make detection of binary files more robust ( @martenlienen ) #523 : Add agent-shell-org-transcript to Related projects ( @lllShamanlll ) #528 : Add support for Kimi Code CLI using ACP ( @nicolaisingh ) #529 : Sort the session list based on recency ( @smagnuso ) #530 : Add simple blinking circle as busy indicator ( @rudolf-adamkovic ) #532 : Fix removing queued messages ( @Gleek ) #536 : Add programmatic query API: agent-shell-query and agent-shell-shell-buffer ( @eddof13 ) #539 : Add ob-agent-shell to related projects in readme ( @eddof13 ) #545 : Keep buffer name on agent-shell-reload and -restart ( @timfel ) #550 : Add agent-recall to related projects ( @Marx-A00 ) #551 : Avoid gitignore update for external data dirs ( @Silex ) #552 : Use helper function from shell-maker rather than eob ( @smagnuso ) #553 : Add ACP session config options support ( @greggroth ) #554 : Fix error when cancelling session selection prompt ( @Gleek ) #555 : Add some niceties useful for ( @vermiculus ) #559 : Expose topic in agent-shell ( @smagnuso ) #561 : Update Pi coding agent logo to look more official ( @jeff-phil ) #565 : Enable @ and / completion when reading queued prompts ( @Gleek ) #571 : Fix unkillable buffer after major-mode change ( @Scott-Guest ) #572 : Add clipboard image support for Windows ( @repelliuss ) #574 : Add agent-shell-pet link to README ( @lgmoneda ) #582 : Add a few additional forward declarations ( @tychoish ) #583 : Add Hermes Agent support ( @yitang ) #588 : Fix header foreground color (emacs 31) ( @nhojb ) #589 : Only trim response region when region is active ( @martenlienen ) #590 : Fix context-limit header color (emacs-31) ( @nhojb ) #595 : Fix completion regression in viewport buffers ( @Gleek ) #600 : Do not emit permission-request for auto-handled permissions ( @Gleek ) #601 : Expose thought level (reasoning effort) in header, mode line and keymaps ( @martenlienen ) #602 : Update README with Opencode and Ollama setup instructions ( @dvictori ) #605 : Add defcustom ( @nhojb ) #608 : Toggle folding for everything & comfortably fold at point ( @codeluggage ) #609 : Prevent accidental auto-scrolling in viewport buffers on tool calls ( @martenlienen ) #611 : Fix ( @TamsynUlthara ) #613 : Improve completion for set-session-config-option ( @catern ) #618 : Fix header not showing session mode name ( @deftsp ) #621 : Reindent Elisp files ( @bcc32 ) #623 : Add agent-shell-hud to related packages ( @nohzafk ) #625 : Fix source-block face background not rendering when theme loads after package ( @phairoh ) #627 : Hermes: add default-session-mode-id to match other agents ( @yitang ) #629 : Write to instead of ( @phairoh ) #632 : Render tool call parameters for non-standard tools like MCP calls ( @martenlienen ) #633 : Add GitHub Actions workflow for ERT tests ( @phairoh ) #634 : Update art generated by ( @TamsynUlthara ) #639 : Preserve window position when restarting ( @Gleek ) #202 : Header icon is double the expected size #278 : Heartbeat causes high CPU usage with many agent buffers #366 : directory at risk when switching git branches #400 : errors with "Text is read-only" #401 : Garbled characters output when using non-English languages #412 : Diff buffer management is messy #414 : timing makes subscribing to events difficult #417 : Unhandled notifications with kiro #426 : Starting conversation before agent has initialized leaves "dangling" text #431 : Heartbeat timer keeps running after failed/abandoned authentication in OpenAI #435 : clipboard handler silently saves text as PNG in terminal mode #441 : Background agent notifications treated as stale after response #443 : : Invalid image type on calling agent-shell #455 : agent-shell enters frozen/hanging state when receiving unknown notifications #462 : Header text invisible when returns 0 #465 : Session load crashes on non-text user message chunks (e.g. images) #466 : / now focus the originating shell #468 : breaks pasting text #481 : History input rolls back in editing mode while pressing up/down arrows #485 : Shells created in other workspaces no longer display #493 : Tables rendered in agent-shell break cursor (point) navigation #533 : Can't install agent-shell from MELPA #548 : Copied highlighted text from agent output includes trailing backtick #563 : "Cannot modify map in-place" when starting agents #577 : does not work in an function #587 : Long region preview's "Expand…" button is sent literally to the agent #617 : OpenCode: consent prompt shows empty input when arrives before populates

Programming Powershell

0 views

Kaushik Gopal 1 months ago

OpenCode power user tips

In this post, I’d like to talk about some power user tips for OpenCode - an open source , model agnostic harness that more people should be using. Hopefully some of the advanced use cases convince you to give OpenCode (and OpenChamber ) a shot. intermediate to advanced tips only I am specifically choosing to talk about some advanced tips in this post. If you’ve never used an agent harness or are looking to learn how to use OpenCode, this post can be useful but reader beware. While (Ctrl + P) will list out all the possible commands (and is helpful), OpenCode has the concept of a “leader” key (which defaults to ). The leader key allows you to execute targeted useful commands more quickly and there’s a slew of useful ones pre-defined 1 . People reach for whole terminals and extra tooling to juggle between agent sessions. I too had an overly customized tmux setup that looked like this: OpenCode simplifies this. Just hit and you view current sessions and can instantly switch to that session by just selecting it from the list. The ability to quickly rename a session from this view is a godsend for me and what lets me be organized. session directory filtering you can pass a flag to when launching it, which filters the session list to just this workspace/directory by default. You can alternatively not pass that flag, and the session list will show all sessions. Forking takes the session you’re in and spawns a new one. You branch off into a separate conversation while the main agent keeps grinding on whatever you left it doing. I love this feature and even cobbled my own version with tmux long before most harnesses shipped it. Claude Code, Codex and other harnesses have caught up and support this feature. But OpenCode’s UX is the smoothest. You simply type in your chat. It gives you the option to fork the current chat or from a previous point in the message. You can then rename the forked session right from the list ui, and jump back and forth. The easy session switching again comes in handy here. Need to rewind to an earlier point in the same conversation? In OpenCode, there’s no escape-escape dance. leader g shows you a timeline and you can revert the conversation instantly, fork a new session from there, or just copy the message text. Probably one of the main reasons I find it hard switching away from OpenCode. I can bounce between GPT-5.5, Kimi K2.6, and Opus by just hitting 2 . change model & reasoning + switches the model on the fly. changes the reasoning type. I see a future where we will have smaller models we can run locally. OpenCode can point to that ollama model you have running on your own machine too. Click here if you’re curious about my model choices. Not everyone realizes this but OpenCode ships with LSP servers built-in . This means the coding agents inside OpenCode understand how to navigate different programming languages better. You’ll find less file search and grepping. Anthropic even recommends LSP server integration as an advanced move for making harnesses behave in large codebases. OpenCode gives you much of that for free. The other reason I swear by OpenCode: hit to cycle through custom agents. Here’s a few I use a lot: view the subagent work When an agent fans work out to subagents, + pulls up the subagent view so you can watch them work. Like others, you can use OpenCode for scripting and one-shot reviews: So up until now, I’ve mostly talked about features in the context of the TUI. My good friend YY recently introduced me to OpenChamber and it’s changed a lot of things for me. OpenChamber is an OpenCode GUI wrapper. OpenCode already has a web client btw. But OpenChamber has a lot of nice bells and whistles. But here’s the kicker, it’s using your same OpenCode server. In a previous post I dug into OpenCode’s server-client architecture: you run OpenCode as a server and connect multiple clients to it. A client can be a terminal tab, your phone, a desktop, a browser — each an isolated session pointed at the same server, fully synced. OpenChamber is just another client, but a super powered GUI one. This feature has taken the world by storm; especially since Codex introduced their implementation. OpenChamber gives you this feature for free with a super nice UX. One button click and either using or internally, it opens a secure 3 tunnel that you can connect your phone or another client to. So now, your phone controls OpenChamber and by proxy OpenCode exactly as you would from your computer. This was possible with OpenCode and tailscale too (as I mentioned in my previous post) but OpenChamber’s UX and secure tunnel approach makes this fluid. I almost never take my work laptop with me, when I’m getting out of the house now. Just speaking to my phone and a browser tab that has OpenChamber open. The other OpenChamber feature I lean on: multi-run. You have a prompt and want to try it across several models at once. I think Cursor was the first to introduce this feature. OpenChamber provides a super nice UI for this. This is how I’ve been kicking the tires on Opus 4.8 and updating my model choices . There’s just one caveat to be aware of. OpenChamber by default probes for a running OpenCode server. If it doesn’t find an OpenCode server there, it will silently spawn its own. So if you truly want all your sessions in sync, you should start your OpenCode server on port first, then open OpenChamber regularly and it’ll attach to the one you already have. I have a handy shell alias to just start a background OpenCode server now like so: If you didn’t read this tip in time, and need to kill previous OpenCode server instances, I suggest the handy procs cli command. There’s a lot more to both OpenCode and OpenChamber, but this is the stuff I reach for daily. The bit that’s stuck with me most is the one-server, many-clients setup — run a single OpenCode server and point everything at it: the TUI, OpenChamber, your phone. Steal whatever helps here, and if there’s a tip I’m sleeping on, send it my way. OpenChamber v1.12.0 tunnel bug Heads up: OpenChamber v1.12.0 added a headless web app mode, and remote instance switching now changes the OpenChamber API endpoint without loading the full remote UI. This seems to have busted the remote mobile tunnel setup I describe above. :/ The developer is responsive and working on a fix 🤞. Until then, I recommend sticking to v1.11.7 , which you can download manually. You can also bind commands that don’t have a predefined key. As an example, I bind the “Exit the app” command to so I can quit OpenCode quickly. ↩︎ yes yes, you’re probably nuking your prompt/KV cache, but you shouldn’t have long running conversations anyway. ↩︎ one-time + TTL + revocable connect link ↩︎ + switches the model on the fly. changes the reasoning type. red-team — think differently from the implementer with an independent adversarial lens and hunt for failure modes. ghostwriter — drafts messages, posts with a less AI tropey voice. brainstormer — custom agent that’s explicitly tuned to help me brainstorm ideas, plans etc. pr-reviewer — strict reviewer that ignores past conversation and reviews with fresh eyes. kimi-coder — a coding agent guardrailed to Kimi: fast, cheap implementation. agent-kombat — see my agent-kombat post. I have it wired into a custom agent for quick use. You can also bind commands that don’t have a predefined key. As an example, I bind the “Exit the app” command to so I can quit OpenCode quickly. ↩︎ yes yes, you’re probably nuking your prompt/KV cache, but you shouldn’t have long running conversations anyway. ↩︎ one-time + TTL + revocable connect link ↩︎

Tutorial

0 views

Langur Monkey 1 months ago

Langur Agent

Langur Agent is a simple, open, hackable CLI AI agent for Linux and macOS. It connects to any service providing an OpenAI-compatible endpoint. It features: The source is available in this repository . Langur Agent has been tested on Linux and macOS only. Install the agent with: Run the agent with the default session: If you need an API key to access the endpoint, put it in the file. Langur Agent looks for the file in the following locations, in order: Create the file with the API key: The agent uses to load at startup. The package reads from the environment automatically. You can also set in your shell profile. On first run, the configuration is created in . You can configure the agent interactively with the slash command. The agent works with any OpenAI-compatible endpoint, so LM Studio, Ollama, OpenWebUI, or any other service you configure. Here are the default values: Run the agent, and then you can enter your prompt. You can use the following key bindings during input: During inference, you can cancel the turn and return to the input prompt with Ctrl + c . Use to print information about the available commands, and to configure the agent interactively. Internally, Langur Agent uses sessions to separate different memory histories. Sessions are named by the user. By default, the agent uses the session. You can start in a different session (either create a new one, or restore it if it exists) with the argument: The default session’s name is , so the following two commands are equivalent: You can also list the existing sessions with : Sessions contain: For now, the configuration file is the same for all sessions. Sessions are matched by the directory name in the sessions location ( ). You can rename a session by just renaming the directory! You can enable mode for the current session with the command , or permanently in the configuration . External editor —In mode, exit INSERT mode ( Esc ), then press v to edit your prompt in an external editor (uses your or variable). There are a few commands available to use in the agent loop. You can list them with . Also, use (e.g. ) to show additional help for a command. Persistent memory follows XDG Base Directory spec in : In addition to persistent memory, the agent maintains a chat history of recent user input and assistant output pairs. This provides context that survives beyond the LLM’s context window. Here is how it works: Persistence: Configuration: Langur Agent can be easily customized and extended by adding new tools, commands, and skills. If you create a cool new tool, skill, or slash command, consider contributing it via a pull request! Create a file in or use one of the existing ones. To create a tool, create a method and decorate it with : Tools are auto-discovered on startup. The process is very similar to tools. You need to create your method, preferably in , and decorate it with . A slash command must return, in that order, , , , : Decorated commands are automatically registered, and auto-completed in the input prompt. Add a file in with YAML front matter, following the agentskills.io standard: The front matter and are parsed and shown in the skills list. The body is injected into the system prompt. session management memory management visual candy autocompletion interactive configuration Python 3.13+ for dependency management Current directory, Home directory, Alt + Enter : add a new line Enter : submit the prompt Ctrl + q : quit The input history Chat memory (see chat memory ) Notes (see session memory ) User profile (see session memory ) — user information — persistent notes (added via tool) Memory is loaded into the system prompt each turn tool adds notes during a session tool explicitly persists memory to disk Memory is auto-saved when the agent exits (interactive mode) Each user message and assistant response is stored in memory Reasoning is omitted from chat memory Automatically compacted when exceeding the configured character limit The user can trigger the compaction any time with Chat memory is attached to the system prompt on each turn The agent displays the last 10 exchanges, with long messages truncated Chat history is persisted to Automatically loaded on startup Saved after every exchange (user input or assistant response) Compacted history is also persisted to disk : a indicating if the command succeeded or failed. : an optional short status message. It is printed with or . : an optional with the Python Rich-formatted content, it is printed to the output. : an optional formatted in Markdown, it is printed to the output.

Python

0 views

Michael Stapelberg 1 months ago

How my minimal, memory-safe Go rsync steers clear of vulnerabilities

Back in January 2025, multiple different security researchers published a total of 6 security vulnerabilities in rsync , some of which allow arbitrary code execution and file leaks, so naturally I was wondering whether/how my gokrazy/rsync implementation was affected. Did implementing my own (compatible, but minimal) rsync in Go, a modern and memory-safe programming language, really rule out entire classes of security vulnerabilities? This deep dive article was in the making since January 2025, but was delayed because we uncovered more unpublished vulnerabilities in the process! The “Security Vulnerabilities” section now covers all 12 vulnerabilities from the January 2025 batch and the May 2026 batch. If you are running (upstream, samba) rsync in production, upgrade to version 3.4.3 or newer. If you are running gokrazy/rsync in production, upgrade to version v0.3.3 or newer. Feel free to skip over the nitty-gritty security issue details and jump directly to: For context, I blogged about rsync, how I use it, and how it works back in June 2022. See also all posts tagged “rsync” . The original motivation for writing my own rsync (back then only a server, today all directions are supported) was to provide the software packages of distri, my Linux distribution research project for fast package management , which I wanted to host on router7 , my small home Linux+Go internet router, which in turn is built on gokrazy , my Go appliance platform. I am still running multiple gokrazy/rsync servers for this original purpose, and also many others! Having rsync available as a primitive (that you can link into your Go programs!) is really nice. This article covers the following security vulnerabilities: The first batch of the vulnerabilities above was announced on the oss-security mailing list , but note that the original report has more detail compared to the oss-security summaries! The later vulnerabilities were announced via GitHub Security Advisories on the rsync project . When the checksums are read by the daemon, two different checksums are read: Most importantly, note that field is filled with bytes. always has a size of 16: rsync.h is an attacker-controlled value and can have a value up to bytes, as the next snipper shows: The problem here is that can be larger than 16 bytes, depending on the digest support the binary was compiled with: md-defines.h support is common and sets the value to 64. As a result, an attacker can write up to 48 bytes past the buffer limit. Upstream fix: The upstream fix for CVE-2024-12084 changes the field to a dynamically-allocated field, which is allocated with length, and fixes the bounds check to check against the (checksum length for this transfer’s algorithm). Can Go help prevent this? Yes: Missing or incorrect bounds checks will not result in a heap buffer overflow in Go! Instead, attempting to write out of bounds will result in a panic because the Go runtime performs bounds checks. How does gokrazy/rsync fare? gokrazy/rsync also had insufficient validation! Our issue was different, though: It wasn’t size confusion, we just were not doing any validation of the sum header at all — oops! We can confirm that the Go runtime’s bounds check triggers on an attempt to write out of bounds by changing the code like so and running the tests: As expected, the Go runtime panics with the following message: Of course, crashing the entire server is not the best failure mode, so I added the missing bounds checking to turn the panic into an error . Because of the same lack of validation as in the previous CVE-2024-12084 vulnerability, an attacker could select a checksum algorithm with short checksums (e.g. with 8 byte checksums), but then claim they were sending longer checksums (e.g. 9 bytes), making the victim leak one byte of uninitialized stack content in the response. Leaking one byte of stack content may seem benign, but as the Google Security report puts it: The first pair of vulnerabilities are a Heap Buffer Overflow and an Info Leak. When combined, they allow a client to execute arbitrary code on the machine a Rsync server is running on. The client only requires anonymous read-access to the server. The daemon matches checksums of chunks the client sent to the server against the local file contents in . Part of the function prologue is to allocate a buffer on the stack of bytes: The daemon then iterates over the checksums the client sent and generates a digest for each of the chunks and compares them to the remote digest: Notably, the number of bytes that are compared again are bytes. In this case, the comparison does not go out of bounds since can be a maximum of . However, the local buffer, not to be confused with the attacker-controlled , is a buffer on the stack that is not cleared and thus contains uninitialized stack contents. A malicious client can send a (known) checksum for a given chunk of a file, which leads to the daemon writing 8 bytes to the stack buffer . The attacker can then set to 9 bytes. The result of such a setup would be that the first 8 bytes match and an attacker-controlled 9th byte is compared with an unknown value of uninitialized stack data. An attacker can divide a file into 255 chunks and as a result leak one byte per file download. An attacker can incrementally repeat the process, either in the same connection or by resetting the connection. As a result, they can leak bytes of uninitialized stack data, which can contain pointers to Heap objects, Stack cookies, local variables and pointers to global variables and return pointers. With those pointers they can defeat ASLR. Upstream fix: There are two relevant upstream fixes: Can Go help prevent this? Yes: By design, Go initializes all variables to the zero value. Go programmers do not need to remember to explicitly initialize variables. How does gokrazy/rsync fare? gokrazy/rsync is not affected by this vulnerability: Variables are always initialized in Go. Additionally, selecting checksums other than MD4 was only introduced in protocol version 30 (gokrazy/rsync implements protocol version 27). Description: (quoting the Google Security report ) When the syncing of symbolic links is enabled, either through the or ( ) flags, a malicious server can make the client write arbitrary files outside of the destination directory. A malicious server can send the client a file list such as: Symbolic links, by default, can be absolute or contain characters such as . In practice, the client validates the file list and when it sees the entry, it will look for a directory called , otherwise it will error out. If the server sends as [both, a directory and a symbolic link], [the client] will only keep the directory entry, thus the attack requires some more details to work. In mode, which the server can enable for the client, the server sends the client multiple file lists. The deduplication of the entries happens on a per-file-list basis. As a result, a malicious server can send a client multiple file lists, where: As a result, the directory is created first and is considered a valid entry in the file list. Then, the attacker changes the type of to a symbolic link. When the server then instructs the client to create the file, it will follow the symbolic link and thus files can be created outside of the destination directory. Can Go help prevent this? No. This vulnerability is caused by a logic error: when multiple file lists are used, the merged file list needs to be re-verified. But see Defense in depth: Go’s Upstream fix: The upstream fix for CVE-2024-12087 adds the missing validation. How does gokrazy/rsync fare? gokrazy/rsync is not affected by this vulnerability: gokrazy/rsync does not implement the incremental recursion mode ( ). The trade-off here is implementation complexity vs. resource usage: the incremental recursion mode allows working with the file set in a “windowed” way, as opposed to having to scan the entire file set before any transfer can begin. See also my How does rsync work? blog post. Description: (quoting the Google Security report ) The CLI flag makes the client validate any symbolic links it receives from the server. The desired behavior is that symbolic links target can only be 1) relative to the destination directory and 2) never point outside of the destination directory. The function is responsible for validating these symbolic links. The function calculates the traversal depth of a symbolic link target, relative to its position within the destination directory. As an example, the following symbolic link is considered unsafe: As it points outside the destination directory. On the other hand, the following symbolic link is considered safe as it still points within the destination directory: This function can be bypassed as it does not consider if the destination of a symbolic link contains other symbolic links in the path. For example, take the following two symbolic links: In this case, foo would actually point outside the destination directory. However, the function assumes that is a directory and that the symbolic link is safe. Upstream fix: The upstream fix for CVE-2024-12088 makes stricter by not allowing anywhere within the path, except at the very beginning. Can Go help prevent this? No. This vulnerability is caused by a logic error: the validation function was incorrect. We could have implemented that same bug. But see Defense in depth: Go’s How does gokrazy/rsync fare? gokrazy/rsync is not vulnerable: The feature is not yet implemented in gokrazy/rsync. The rsync receiver (in client mode) did not sanitize file names provided by the rsync sender, or otherwise prevent opening files outside the destination tree. A malicious sender could instruct a receiver to compare checksums of arbitrary files outside the destination tree. By observing the receiver’s reaction to a provided one-byte checksum, a malicious sender can leak arbitrary files. When a client connects to a malicious server the server is able to leak the contents of an arbitrary file on the client’s machine. In the client will read type as well as the from the server if the server sets the appropriate flags. The flag will not be set for the client. The caller ( ) then uses the server provided values to determine a file to compare the incoming data with. In the contents of the file specified by are copied into the destination file. This can be achieved by the server sending a negative token. The server sends a checksum to compare. If they don’t match, a 0 is returned. When the return value is 0 the receiver will then send a to the generator. The generator will then write a message to the server. The server can use this as a signal to determine if the checksum they sent was correct. By starting off with a of 1 a malicious server is able to determine the contents of the target file byte by byte. Upstream fix: The upstream fix for CVE-2024-12086 prevents opening files outside the destination tree by verifying the sender-provided path. Can Go help prevent this? Yes, Go offers an API to prevent this, see Defense in depth: Go’s . How does gokrazy/rsync fare? gokrazy/rsync is not vulnerable: the fuzzy matching feature was introduced with rsync protocol version 29, but gokrazy/rsync implements protocol version 27. Description: (quoting the Red Hat Security Advisory ) A flaw was found in rsync. This vulnerability arises from a race condition during rsync’s handling of symbolic links. Rsync’s default behavior when encountering symbolic links is to skip them. If an attacker replaced a regular file with a symbolic link at the right time, it was possible to bypass the default behavior and traverse symbolic links. Depending on the privileges of the rsync process, an attacker could leak sensitive information, potentially leading to privilege escalation. Upstream fix: The upstream fix for CVE-2024-12747 changes calls in the rsync sender to use the option. The paths are not expected to be symlinks at that point in the algorithm (symlinks would be handled with ). Can Go help prevent this? Yes, Go offers an API to prevent this, see Defense in depth: Go’s . How does gokrazy/rsync fare? gokrazy/rsync was vulnerable before commit , which introduces the same mitigation that upstream rsync uses. To reproduce the issue, use the following steps: Check out gokrazy/rsync v0.2.7: Patch the code as follows to undo the fix and execute the attack: Running the test now shows that the server traversed the symlink: A surprising discovery When I shared a draft of this article with Damien Neil, member of the Go Security Team and the author of the traversal-resistant API , he pointed out: I believe the gokrazy fix for CVE-2024-12747 is insufficient. You’re calling with , but only prevents symlink traversal in the last path component. This is probably still vulnerable to replacing an earlier path component so can be redirected by symlinking to . We reported this to the rsync security contact address in April 2025. In December 2025 I learned that someone else had also independently discovered and reported this issue. Ultimately, this resulted in CVE-2026-29518, published on 2026-05-20. Description: (quoting the rsync 3.4.3 NEWS entry ) TOCTOU symlink race condition allowing local privilege escalation in daemon mode without chroot. An rsync daemon configured with is exposed to a time-of-check / time-of-use race on parent path components. A local attacker with write access to a module can replace a parent directory component with a symlink between the receiver’s check and its open(), redirecting reads (basis-file disclosure) and writes (file overwrite) outside the module. Under elevated daemon privilege this allows privilege escalation. Default is not exposed. Reach: local attacker on the daemon host, write access to a module path, daemon configured with . Upstream fix: The upstream fix for CVE-2026-29518 uses , which is similar to Go’s API. Can Go help prevent this? Yes, Go offers an API to prevent this, see Defense in depth: Go’s . How does gokrazy/rsync fare? gokrazy/rsync was vulnerable until I switched the sender and the receiver to the traversal-resistant API . Description: (quoting the GitHub Security Advisory ) Description: The receiver’s compressed-token decoder accumulated a 32-bit signed counter without overflow checking. A malicious sender can trigger an overflow that, with careful manipulation, leaks process memory contents to the attacker – environment variables, passwords, heap and library pointers – significantly weakening ASLR and facilitating further exploitation. Reach: authenticated daemon connection with compression enabled (the default for protocols >= 30 when both peers advertise it). Disabling compression on the daemon (“refuse options = compress” in rsyncd.conf) is the available workaround. Upstream fix: The upstream fix for CVE-2026-43618 introduces the missing checks. How does gokrazy/rsync fare? gokrazy/rsync is not vulnerable because it does not implement compression. See gokrazy/rsync issue #35 for details on why compression support sounds simple, but is non-trivial. Description: (quoting the GitHub Security Advisory ) The 2025 fix that added a guard in was not applied to the visually-identical block in . A malicious rsync server can drive any connecting client into a deterministic by setting in the compatibility flags, sending a flist whose first sorted entry is not a leading “.” directory (which causes to set ), then sending a transfer record with and a non- iflag word. The receiver reads and dereferences the result. On glibc x86-64 the dereferenced pointer is mmap chunk metadata that lands at an unmapped address, hence a clean ; non-glibc allocators have not been audited. Reach: any rsync client doing a normal pull from an attacker-controlled URL. Works for both rsync:// URLs and remote-shell pulls. is the protocol-30+ default; no special options are required on the victim. Workaround: on the client. Upstream fix: The upstream fix for CVE-2026-43620 adds the guard to as well. How does gokrazy/rsync fare? Just like for CVE-2024-12087 , gokrazy/rsync is not affected by this vulnerability: gokrazy/rsync does not implement the incremental recursion mode ( ). Description: (quoting the GitHub Security Advisory ) Description: Earlier fixes for symlink races on the receiver’s open() call (CVE-2026-29518) missed the same race class on every other path-based system call: chmod, lchown, utimes, rename, unlink, mkdir, symlink, mknod, link, rmdir, lstat. On rsync daemons with “use chroot = no” a local attacker with filesystem access on the daemon host can swap a symlink into a parent directory component between the receiver’s check and one of these syscalls, redirecting it outside the exported module. The fix routes each affected path-based syscall through a parent dirfd opened under RESOLVE_BENEATH-equivalent kernel-enforced confinement (openat2 on Linux 5.6+, O_RESOLVE_BENEATH on FreeBSD 13+ and macOS 15+, per-component O_NOFOLLOW walk elsewhere). Default “use chroot = yes” is not exposed. Reach: local attacker on the daemon host, write access to a module path, daemon configured with use chroot = no. Upstream fix: The upstream fix for CVE-2026-43619 uses the family of syscalls, just like Go’s . Can Go help prevent this? Yes, Go offers an API to prevent this, see Defense in depth: Go’s . How does gokrazy/rsync fare? gokrazy/rsync is not affected, because it uses Go’s API throughout. Description: (quoting the GitHub Security Advisory ) On an rsync daemon configured with the global rsyncd.conf setting, the reverse-DNS lookup of the connecting client was performed after the daemon had chrooted into . If did not contain the files glibc needs for resolution ( , , , NSS service modules), the lookup failed and the connecting hostname was set to “UNKNOWN”. Hostname-based deny rules (“hosts deny = *.evil.example”) therefore could not match, and an attacker controlling their PTR record could connect from a hostname the administrator had intended to deny. IP-based ACLs are unaffected. The per-module setting is unrelated to this issue. Reach: rsync daemon configured with AND hostname-based ACLs AND does not include the libc resolver fixtures. Upstream fix: The upstream fix for CVE-2026-43617 moves the DNS lookup to an earlier point in the protocol. How does gokrazy/rsync fare? gokrazy/rsync is not vulnerable because we only implement IP-based allow/deny lists, not hostname-based allow/deny lists. Description: (quoting the GitHub Security Advisory ) The rsync client’s HTTP proxy support contains an off-by-one out-of-bounds stack write in ( ). After issuing the request, rsync reads the proxy’s first response line one byte at a time into a 1024-byte stack buffer with the bound , so the loop only ever writes . If the proxy (or a man-in-the-middle in front of it) returns 1023+ bytes on the first response line without a terminator, the loop exits with — a slot the loop never wrote, so holds stale stack bytes left there by the earlier that formatted the outgoing request. The post-loop code then does: The lands one byte past the end of the on-stack , corrupting whatever lives in the adjacent stack slot. AddressSanitizer reports at in the frame. Upstream fix: The upstream fix for CVE-2026-45232 validates the attacker-supplied data. How does gokrazy/rsync fare? gokrazy/rsync does not implement such proxy support, so it is not vulnerable. Let’s summarize how Go fares: Aside from being written in Go, another key difference between gokrazy/rsync and the official upstream rsync is that the gokrazy implementation is minimal : Let’s have a look at whether gokrazy/rsync was affected by each CVE at the time of publishing: To be clear: all known vulnerabilities are fixed in gokrazy/rsync! The table above documents what the state was at the time when each CVE was published. In other words: When the January 2025 vulnerabilities were published, gokrazy/rsync panicked (CVE-2024-12084) and was vulnerable to a TOCTOU race (CVE-2024-12747). In the process of fixing the TOCTOU issue, we discovered CVE-2026-29518, which was fixed in gokrazy/rsync before the CVE was published. CVE-2026-43619 was discovered even later, but was also already fixed in gokrazy/rsync with the same fix: using Go’s everywhere. As I was reading the vulnerability reports, I noticed that the reports were slightly misleading by their choice of words: most reports just spoke of “server” and “client”. However, in an rsync transfer, both sides, the rsync client and the rsync server can assume either role: sender (upload files) or receiver (download files)! Some setups come with further restrictions that make certain attacks harder or impossible to pull off. For example, when running in daemon mode, file system access can be restricted to the pre-configured module paths (but not in command mode!). Here is a diagram to give you an overview of the 4 different setups and role/protocol layering: In the context of our vulnerability reports, I would say that the Arbitrary File Leak vulnerability (CVE-2024-12086)’s original title “Server leaks arbitrary client files” can easily be misunderstood. Instead, I would say: The rsync receiver will leak arbitrary files to a malicious sender . I have verified that a malicious client sender can make an unpatched remote rsync open files outside the destination tree (e.g. the system password database) when running in command mode, for example over SSH. (But, when running in daemon mode, the server enables additional path sanitization, which prevents this attack.) Similarly, the Symlink Path Traversal vulnerability (CVE-2024-12087) speaks about a “malicious server”, but again, it should be “malicious sender”, which can be either the client or the server. The OpenBSD project is known for its security focus, so how does openrsync compare? openrsync is not affected by the Heap Buffer Overflow (CVE-2024-12084) and Stack Info Leak (CVE-2024-12085) vulnerabilities because it validates the checksum length and only supports one checksum size/algorithm (MD4). openrsync is not affected by CVE-2024-12086, CVE-2024-12087 and CVE-2024-12088 because it does not implement the relevant features (like gokrazy/rsync). Even if it was vulnerable, openrsync’s defense-in-depth measures like using OpenBSD’s and to restrict file system access would have prevented successful exploitation — at least when running on OpenBSD. openrsync is not affected by CVE-2024-12747 because it used from the very moment they implemented symlink support . But, because is not a sufficient fix for this issue, openrsync is affected by CVE-2026-29518! The above covers the January 2025 batch of vulnerabilities; the May 2026 batch is similar in that most features just are not implemented. Overall, I say: Well done, Kristaps and contributors! By diligently implementing validation, restricting the attack surface and employing defense-in-depth measures, openrsync manages to not be affected by almost all of the reported vulnerabilities. Which APIs and environments can we use on Linux for defense-in-depth measures? I’ll go through the ones supports, ordered by traditional to modern. Within a few weeks after starting the project, I added support for dropping privileges and using mount/pid namespaces on Linux to restrict the file system objects that my rsync server could work with. This approach works very well to mitigate path traversal attacks, but requires privileges, meaning we need to run as or in a Linux user namespace (if enabled on your distribution / system). That limitation makes mount namespaces well-suited for server setups, but usually unavailable for interactive one-off transfers that are typically running under a human’s user account. In the same commit that introduced Linux mount/pid namespace support, I also included a systemd service file that restricted file system access to home directories and encouraged folks in the README to further restrict file system access, depending on what their use-case allows. These file system restrictions, if set up correctly, mitigate the File Leak (CVE-2024-12086) and Path Traversal (CVE-2024-12087) vulnerabilities. The Symlink Race Condition (CVE-2024-12747) relies on privilege escalation through the rsync process, but thanks to the DynamicUser feature, our process has fewer privileges than other users. Similarly to mount namespaces, these measures are great for server setups, but too cumbersome to set up for interactive one-off usages. I stumbled upon Justine’s blog post Porting OpenBSD pledge() to Linux (2022) and was reminded that Linux offers the Landlock API for unprivileged, per-process access control, similar to OpenBSD’s system call, which openrsync uses. The basic idea is that once your program knows the directory it works with, it makes a call like and no longer has access to other file system locations. I had previously heard of Landlock at a Go Meetup, so I knew there was Go support for Landlock. Back in 2022, I enabled Landlock support in the gokrazy kernel images. So I gave it a shot in March 2025 and implemented Landlock support to restrict file system access . It took me a few hours, which seems a little longer than one might expect at first. Making Landlock work (and/or skipping it) in our test environment ran into a couple of road blocks: Our tests had defined many functions that get run in the same process, but when repeatedly adding rulesets, we would exceed the limit of 16 (!) policy layers per process. Once I had it set up just right, it is a beautiful solution. Now we can restrict rsync transfers to their sources (read-only) or destination directories (read-write), even for unprivileged invocations of ! 🎉 The downside to Landlock is that Landlock operates at the process level. This means that Landlock policies must include the files that your program needs, e.g. needs to be able to read for user id lookup, so if the attacker is after the file, Landlock does not help. In February 2025, the Go 1.24 release introduced the API, which is resistant against path traversal, see The Go Blog: Traversal-resistant file APIs (by Damien Neil, March 2025). This API allows more fine-grained control (per file system operation) compared to Landlock. Go 1.25 (released in August 2025) added more methods to , making it a convenient choice for most file system usage. I have converted all of ’s file system usage to use , which is a great fit: users configure input/output directories, but the filenames received over the network are untrusted. That’s exactly what was designed for! When I first looked into using , I thought that some system calls could inherently not be made with this API, like for example to create device node files. Damien explained: It won’t support mknod, though. However, you should be able to use it to enable a safe mknod: If you’re curious how that looks in practice, check out ’s usage in , line 15-29 . Another stumbling block was when I realized that unlike with , Linux only implements , but no (as of Linux 7.0)! Luckily, Lennart Poettering pointed out that there’s a trick to skip path resolution without : you can probably bind to in the meantime… And indeed, this works! Path resolution is skipped because we only specify a basename (last component of a path) after the known-safe , not a path (see line 49-56 ). With these two tips, v0.3.1 and newer are fully using , meaning all file system access is traversal-safe! 🥳 Lacking validation causes vulnerabilities It is interesting to note that aside from the TOCTOU vulnerabilities (CVE-2024-12747, CVE-2026-29518 and CVE-2026-43619), all other vulnerabilities were caused by missing or incorrect input validation. In three cases, there was just no validation to begin with. In another case (CVE-2024-12088), the subject matter of file system path resolution is tricky enough that the existing validation did not cover all edge cases. As the Go verdict section explains in more detail, the most valuable structural fixes are to provide bounds checking (= always-on validation) and safe-by-default APIs like Go’s . Too much complexity A few of the vulnerabilities came from evolution of the rsync protocol: The code used to correctly perform sufficient validation, but then new features were added. For example, when checksum algorithm negotiation was added (protocol version 30), the validation was not correctly updated. When incremental recursion was added (also protocol version 30), the validation that made sense for individual file lists was not updated for the new processing approach of merging incremental file lists. Avoiding complexity avoids vulnerabilities! Both gokrazy/rsync and also openrsync were not vulnerable to 8 out of the 12 security vulnerabilities simply because they do not implement the feature with the vulnerability. Of course, these features were added to rsync because they were valuable to someone at some point, and of course I am not saying that we should just… not develop software any further, ever. But, I consider it ideal to use an implementation whose complexity is appropriate for and proportional to the complexity of the use-case . In other words: for simple use-cases, reach for a simple implementation. Only reach for the fully-featured implementation where needed. The verdict on whether using Go has helped . The verdict on whether a minimal re-implementation like gokrazy/rsync helps . My comparison with OpenBSD’s (written in C). Defense in depth mechanisms one can use on Linux. The conclusion . CVE-2024-12084 to 12088 (original report) CVE-2024-12747 (discovered separately by Aleksei Gorban “loqpa”) CVE-2026-29518 (discovered by Damien Neil and myself! and independently by Nullx3D ) CVE-2026-43617 to 43620 CVE-2026-45232 rsync performed insufficient validation: It read the (attacker-controlled) checksum length from the network and compared the length against . However, rsync’s data structures always declared a 16 byte buffer: is always 16 (bytes), which is sufficient to hold an MD4 or MD5 checksum. used to be 16 (bytes), but can be larger when rsync is compiled with SHA256 or SHA512 checksum support. Hence, the bounds check was ineffective! An attacker could write out of bounds. This issue was introduced with commit in September 2022 , which added SHA256/SHA512 checksum support. A 32-bit Adler-CRC32 Checksum A digest of the file chunk. The digest algorithm is determined at the beginning of the protocol negotiation. The corresponding code can be seen below: sender.c : The “Some checksum buffer fixes” commit prevents this attack because the attacker-controlled can no longer be larger than the transfer’s checksum length. The “prevent information leak off the stack” commit initializes the memory to zero, thereby making any stack leak through impossible. Check out gokrazy/rsync v0.2.7: Patch the code as follows to undo the fix and execute the attack: The Go runtime’s bounds checks turn more serious security issues into a panic. A panic is still a denial-of-service risk, but that’s much preferable. Go initializes memory to zero, making info leaks like CVE-2024-12085 impossible. Go’s API prevents most of the remaining vulnerabilities. Only one out of twelve vulnerabilities (CVE-2026-43617) is a proper bug in the application logic that using Go could not have prevented. gokrazy/rsync is unaffected by many vulnerabilities because it does not implement the feature in question, for example . Like all other wire protocol-compatible rsync implementations, gokrazy/rsync targets protocol version 27, because later protocol versions introduce significant complexity. In some cases, features that would be good to implement come with significant blockers, e.g. compression is tricky, see gokrazy/rsync issue #35 for details. os.Root.OpenFile the parent directory of the target, File.Fd to get the file descriptor for that directory, https://pkg.go.dev/golang.org/x/sys/unix#Mknodat to create the file.

Security

0 views

Corrode 1 months ago

Rust for Linux Live

Hot off the press: this episode is a live recording from Rust Week in Utrecht, just two days ago. On stage with me are two people who hardly need an introduction in the Linux world: Greg Kroah-Hartman , Linux Foundation Fellow, stable kernel maintainer and an embassador for the kernel, and Alice Ryhl , core maintainer of Tokio and one of the driving forces behind Rust for Linux at Google. I have to admit a bit of personal history here: I first wrote about Greg more than 20 years ago for the German online newspaper Pro-Linux . Getting to sit down with him, and with Alice, in front of a live audience to talk about how Rust is reshaping the most important piece of infrastructure on the planet, was a genuine career highlight. We get into the big questions: Why does Alice believe that interop, not rewrites, is how Rust wins inside Linux? How do you carefully weave in Rust while maintaining a 35-million-line C codebase? And what does it actually feel like, day to day, to write kernel code in Rust? “Rust is gonna save the Linux kernel.” — Greg Kroah-Hartman CodeCrafters helps you become proficient in Rust by building real-world, production-grade projects. Learn hands-on by creating your own shell, HTTP server, Redis, Kafka, Git, SQLite, or DNS service from scratch. Start for free today and enjoy 40% off any paid plan by using this link . Rust for Linux is the project bringing the Rust programming language into the Linux kernel. After years of patches, proposals, and heated mailing list threads, Rust is now an officially supported language inside the kernel tree, no longer an experiment. The work spans everything from the build system and the crate to drivers, abstractions over core subsystems and brand-new pieces of infrastructure written entirely in Rust. Greg Kroah-Hartman is a Linux Foundation Fellow, the maintainer of the stable Linux kernel branch, and the maintainer of, among many other things, the USB subsystem , the driver core, sysfs, debugfs, kobject, TTY layer and staging tree. He has been a central figure in Linux for over two decades, has written several books about kernel development, and is convinced Rust belongs in the kernel. Alice Ryhl is a software engineer at Google working on Android and Rust for Linux, and a core maintainer of Tokio , the asynchronous runtime that over 50% of all crates on crates.io directly depends on. Inside the kernel she works on Binder, on async abstractions, and on the bindings that allow Rust drivers to talk safely to the rest of the kernel. Rust Week is an annual conference organized by RustNL. The 2026 edition took place in Utrecht, the Netherlands, from May 18 to May 23. It features talks, workshops, the Rust All Hands, and expert sessions on a wide variety of topics revolving around Rust. This episode was recorded live on stage during the conference. Thanks to the Rust Week team who made this recording possible! Learn more about Rust Week on their website . Linux Docs: USB Subsystem Maintainer - Greg’s first contribution led to him maintaining the USB subsystem, and much more The Register: Happy birthday, Linux: From a bedroom project to billions of devices in 30 years - An interview with Greg celebrating the 30 year anniversary of the Linux kernel Tokio - Another big project maintained by Alice RustWeek: Untrusted data in Linux — How Rust is going to save us - Greg’s talk at RustWeek; Rust is gonna save Linux?! Rust in Production: Rust for Linux - With Danilo, one of the co-maintainers with Greg on the Driver Core subsystem and others Phoronix: New Linux Patch Confirms: Rust Experiment Is Done, Rust Is Here To Stay - The official end of experimental Rust Linux Plumbers Conference - A big conference for all levels of kernel developers std::boxed - The most basic kind of pointer in Rust kernel::list::List - Linux’ linked list Rust binding core lib - The most fundamental parts of the Rust libraries alloc lib - All things in the standard library that only require an allocator, not used by the kernel anymore std lib - The thing most people think of as the standard library, containing things like file access which requires running on a kernel QR code generator for kernel crashes - First Rust code added to the kernel Linux Rust Architecture support - Missing some big platforms like S390 (IBM Mainframes) and MIPS (a lot of consumer networking hardware) sched_ext Schedulers written in Rust - sched_lavd shows promise for video game performance, and servers? Aya - Build eBPF programs with nothing more than Rust and the Linux kernel RustWeek: Completion-based IO - Alice’s talk at RustWeek about IO WE DO NOT BREAK USERSPACE - An e-mail from Linus explaining the mantra in typical Linus fashion Linux clippy config - It’s not pedantic! Rust code style - Coding guidelines of the Linux project for Rust code rustfmt config - Almost vanilla with some ideas for the future clang-format config - Added in 2018 and tabs won! Coccinelle - Semantic code transformation and the reason Greg lives in Europe klint - Custom kernel specific lints, basically a repository of clippy lints for kernel code Rust for Linux Greg Kroah-Hartman on Wikipedia Greg Kroah-Hartman’s homepage (momentarily offline) Greg Kroah-Hartman on Mastodon Alice Ryhl’s website Alice Ryhl on GitHub

Linux

Rust

0 views

Ed Zitron's Where's Your Ed At 1 months ago

AI Is Too Expensive

If you liked this piece, you should subscribe to my premium newsletter. It’s $70 a year, or $7 a month, and in return you get a weekly newsletter that’s usually anywhere from 5,000 to 18,000 words, including vast, detailed analyses of NVIDIA , Anthropic and OpenAI’s finances , and the AI bubble writ large . My Hater's Guides To Private Credit and Private Equity are essential to understanding our current financial system, and my guide to how OpenAI Kills Oracle pairs nicely with my Hater's Guide To Oracle . This week, I’ll publish the second part to my ongoing series (“ What If…We’re In An AI Bubble? ”) about the factors and events that will cause the AI bubble to finally pop. Subscribing to premium is both great value and makes it possible to write these large, deeply-researched free pieces every week. AI is, as it stands, not economically viable for anybody involved other than the construction firms, NVIDIA, and the surrounding hardware companies benefitting from the irrational exuberance of a data center buildout that doesn’t appear to be happening at the speed we believed . Every AI startup loses millions or billions of dollars a year, and nobody appears to have worked out a way to stop hemorrhaging cash. Hyperscalers have invested over $800 billion in the last three years, with plans to add another $700 billion or so in 2026 and another $1 trillion in 2027 , meaning that they need to make at least three trillion dollars in AI specific revenue just to break even , and $6 trillion or more for AI to be anything other than a wash. I went into detail about this (albeit at a lower, pre-2026/2027 capex number) in a premium piece last year . To give you some context, Microsoft made $281 billion , Meta $200 billion , Amazon $716 billion , and Google $402.8 billion in revenue in their most-recent fiscal years for every single product combined, for a total of $1.599 trillion. None of them will talk about their actual AI revenues. Yes, yes, I know Microsoft said that it had $37 billion in AI revenue run rate ($3.08 billion a month or so) and Amazon had $15 billion, or around $1.25 billion a month , but both of these are snapshots of single months that are meant to make it sound like they’re going to make that much in a year but in the end, you don’t actually know anything about how much money they’ve made from AI. We do, however, now know that Microsoft has spent an approximate $100 billion on its OpenAI partnership after testimony from an executive during the otherwise-dull Musk-OpenAI trial, per Bloomberg : This is a fascinating insight for a few reasons: At the end of 2025, OpenAI claimed that it had 1.9GW of capacity (likely referring to total power draw rather than the actual critical IT of the infrastructure at its disposal), which, per analyst estimates, ( $42 to $44 million per megawatt ) works out to around $79.8 billion. This claim was made around six months before the release of Microsoft’s most recent quarterly results. In other words, Microsoft has spent 4 years sinking (either through spending or allocating the capex in advance) nearly $300 billion into…building OpenAI? Okay, fine. Microsoft also has 20 million Microsoft 365 Copilot subscribers for an absolute maximum revenue of $7.2 billion…if every single one were paying $30 a month, which they are most assuredly not as Microsoft has been offering discounts on it for years . Based on my reporting from last year , Microsoft made around $7.5 billion from OpenAI’s inference spend and $761 million from its revenue share in Fiscal Year 2025, a year when it invested (either spent or allocated) around $88.2 billion in capital expenditures. I didn’t report it at the time, but I also had the numbers for all of Microsoft’s revenues for the first three quarters of Fiscal Year 2025 — a total of $8.9 billion of total AI revenue, with around $4.35 billion in revenues when you removed OpenAI’s inference. If we assume that Microsoft’s other AI services grew 10% quarter-over-quarter, I estimate that Microsoft likely made around $17.9 billion in AI revenue in FY2025, or a little under a fifth of its capex. And let’s be clear: none of these numbers include the actual operating expenses. Data centers, after all, need electricity to run, and AI data centers in particular need a lot of electricity. And some — though, admittedly, not many — people to handle the things like maintenance, repairs, and operations. And then there are things like taxes, insurance, and the other day-to-day costs that, when you add them all together, make a big, scary number. You can argue that “actually GPUs are profitable to run” ( I disagree! ), but for any of this to make sense, four things have to happen: All four must be true. If AI revenues don’t explode, capex can stop, margins can be positive, and your best-case scenario is…you maybe broke even. If capex never stops being invested, you need revenues to explode dramatically — to the tune of effectively doubling Microsoft, Meta and Google’s entire businesses, and tripling Amazon Web Services’ annual revenue ( $128 billion ) — and for said revenues to be margin-positive, because if they’re not, eventually other healthy businesses will slow, leaving AI to tear a hole in overall margins. In all cases, AI revenue must stay consistent because, well, you need to get paid . I also cannot find an economic scenario where this pays itself off. Let’s assume that Anthropic is actually at $45 billion in annualized revenue ( I believe it’s doing some very worrisome maths to get there ), or around $3.75 billion a month. On an annualized basis, this would not be enough — assuming it had zero operating expenses (rather than losing billions) — to recover a single year of capital expenditures from Microsoft, Google, Meta, or Amazon from 2024 or 2023. Even if OpenAI’s entire cloud spend ( $50 billion ) for 2026 went to Microsoft and it doubled its Microsoft 365 Copilot revenue (at full cost) to $14.4 billion, it estimates it will invest $190 billion in capital expenditures this year. Amazon’s $15 billion AI run rate, even if it doubled, wouldn’t put much of a dent in its $200 billion in investment plans . While we don’t know Google’s AI revenues, it plans to invest $185 billion in capex this year . These AI revenues have to be completely fucking insane and they need to be that way extremely fucking soon , because otherwise the best they’ll be able to say is “our first few years of capex weren’t particularly useful but the stuff we built after it was,” which still works out to a few hundred billion dollars of waste. Things get even worse when you realize that at least 70% of Microsoft, Google, and Amazon’s compute is dedicated to Anthropic and OpenAI , two companies that burn so many billions of dollars that Microsoft, Google and Amazon have already fed them a combined $54 billion in the last three years, with $28 billion of that coming in the last month and Anthropic due another $50 billion from Google and Amazon if certain performance obligations are met. And there’s no real sign, outside of Anthropic and OpenAI’s compute spend (which is reliant on hyperscaler and venture capital money), of any real explosion in AI revenue. Per The Information (in a chart I love to share!), more than 50% of hyperscalers’ revenue backlogs comes from these companies: If massive, incredible demand for AI existed, wouldn’t these remaining performance obligations be near the trillion mark? Wouldn’t there be other Anthropic or OpenAI sized chunks of revenue? There’s allegedly incredible, unstoppable, insatiable demand for compute. Why isn’t it lining up? Let’s take a look at those RPOs! That was a lot of numbers, so let me make it simpler: outside of OpenAI and Anthropic, these three companies do not appear to be significantly increasing their revenues, and the only way to get that revenue is to feed money to one or both of these companies. Put aside all the theoreticals and hypotheticals and metaphors and imaginary future scenarios and tell me: what, in the next year, are Microsoft, Google and Amazon going to do about this problem? How do they solve it? If we assume the absolute best-case scenario, these companies are making a combined $70 billion in annual revenue on investments that now — including the money invested in the companies themselves — total over $900 billion. Doubling that won’t be enough. Tripling it won’t be enough. In fact, to pay this off, these companies will need to be making over $100 billion each in AI revenue in the next year , because otherwise there is no covering these losses. And it all comes back to a very simple point: AI is too expensive. If the margins were good, they’d be sharing the margins. If the revenues were good, they’d be sharing the revenues (and no, run rates aren’t revenues). If the business was strong, it would be a separate category in their earnings. But LLMs are too expensive! They cost too much to run, and said costs appear to increase linearly with revenues. The more a user uses a product, the more it costs the company to run it, and the more capacity they can take up. The only way to capture any growth is to buy and install GPUs , which in turn requires you to build somewhere to put them, which takes time and money. I’m really struggling to see the argument in favor of continued capex investment. You’re more than $800 billion in the hole with, I estimate, less than half of that resulting in operational GPUs and capacity. Said capacity is mostly taken up by OpenAI and Anthropic, two companies that burn billions of dollars and do not appear to have an answer for how they might stop. The more you build, the more your infrastructure becomes dependent on the continued existence of two perennially-unprofitable ultra-oafs, as your existent AI product lines are, at best, add-ons to products like Google Workspace or Microsoft 365, or further expansion of cloud compute capacity with lower margins and higher up-front costs than anything you’ve ever built. Every quarter is an opportunity to put yourself another $30 billion or so in the hole, all in the hopes that, I assume, OpenAI or Anthropic will pay you $100 billion or $200 billion over the course of a few years, because nobody else in the entire universe is spending that much on compute. You are not recovering these investments without either a massive new product line that doesn’t exist today or three or four Anthropic or OpenAI-sized compute contracts. Put another way, Amazon needs another AWS ($128 billion a year), Microsoft another Azure ( $75 billion a year, including OpenAI’s 2025 compute spend ) and Google a business line at least half the size of search (around $200 billion a year). These businesses have grown to this size by providing extranormally large amounts of value from the very moment they were created and impenetrable monopolies — and while there are quite literally other cloud providers that can physically provide the infrastructure to OpenAI and Anthropic ( Oracle is trying to compete and may die as a result ), the actual “monopoly” here is “being able to deploy hundreds of billions of dollars.” Anthropic proved this when it took 300MW of compute from Elon Musk . In Oracle’s case, as I’ve explained at length , it has to successfully build 7.1GW of capacity, have that capacity actually be margin-positive (doubtful!), and then actually get paid for it by the time it’s built in, oh, I dunno, 2032? Sadly, I have bad news about Oracle, Microsoft, Amazon, and Google’s largest customers. Here’s a fun game: ask an AI booster how OpenAI or Anthropic becomes profitable! Here’s what they’ll say: I must be abundantly clear that nobody has any proof that anyone is profitable on inference, but we have plenty of proof they’re not. They’ll likely cite known liar Sam Altman saying OpenAI is profitable on inference from a party from August 2025 , or Dario Amodei saying ( in a sentence around “stylized facts” that are “not exact” and are specifically “a toy model” and specifically not about Anthropic ) “the inference has some gross margin that’s more than 50%.” Here’s a really simple way to dispute this: Coatue said that Anthropic’s revenues were 85% API calls in 2025 . If it’s profitable on inference, how is it still losing money? You’re gonna say “training,” but that doesn’t actually answer the question: if Anthropic’s process of providing tokens to its models is profitable, how is it losing so much money? Why offer a subscription platform at all? As I’ll get to, Anthropic has companies paying massive amounts for tokens — hundreds of millions a year in some cases — that’s all inference . Why are you bothering with these stinky, nasty monthly subscriptions? The “inference is profitable” argument is a bedtime story told to people that can’t reconcile the logic of a company that allows people to burn between $8 and $13.50 of every dollar of their subscription revenue. Otherwise, you have to reconcile with the fact that both Anthropic and OpenAI are both incinerating money and have no real path to any kind of sustainability other than, well, not doing that. One very, very specific counter-argument people make is that open source models are cheap, and can somehow be compared to OpenAI and Anthropic’s, despite the fact that we have no idea what the actual parameters of Sonnet, GPT, Opus, or any other of their models actually are. What we do know is that both of these companies lose billions of dollars. What we do know is that OpenAI, per The Information , plans to burn $852 billion through the end of 2030, and that as of March 6, 2026 (per CFO Krishna Rao’s sworn affidavit), Anthropic made “exceeding” (sigh) $5 billion in revenue and spent $10 billion on inference and training. Anthropic has done a great deal of work to obfuscate how much it actually makes or spends, but I think it’s likely it burns even more than OpenAI, given the fact that it’s had to raise $75 billion in the last 6 months ( assuming its new $30 billion round closes ), and that’s not including an additional $30 billion from Google and Amazon if certain unknown milestones are hit. Then there’s the issue of those RPOs. Anthropic is now on the hook for $200 billion to Google, $100 billion to Amazon and $30 billion to Microsoft, I assume over the course of the next three or four years. So let’s lay this out. Anthropic — based on its own affidavit from March — appears to have spent $3 to make $1 of revenue on a compute basis, and that’s before you include any and all other costs like staff or electricity or the vocal coach that Dario Amodei uses to add that bass to his voice. Additionally, it needs $330 billion to pay its cloud obligations to Amazon, Google, and Microsoft over the next four years. I’d estimate it needs $5 billion a year for its compute deal xAI (so $20 billion over the total period) and an estimated $30 billion to cover its deal with CoreWeave . That brings us to a total of $380 billion. It’s hard to estimate the actual costs associated with running Anthropic because so much of the reporting no longer makes sense as a result of that affidavit. Nevertheless, I think it’s fair to assume it will need at least $20 billion of operating expenses across that four year period. We don’t even need to play in the realm of “what might Anthropic or OpenAI’s revenues be?” to understand the problem here. Both companies aggressively burn money, and neither of them have any answer as to how they might stop. Numerous reports about how Anthropic will turn “cash flow positive” in either 2027 or 2028 are fantastical, illogical, entirely driven by ridiculous projections, and should never have been reported as anything other than an attempt by companies to mislead their investors. In both cases, reporters should’ve had more asterisks on those numbers than Q*Bert reading Frank’s lines from Blue Velvet . And we have plenty of evidence that they’re losing more money over time. In January 2026, The Information reported that Anthropic’s gross margins were 40% in 2025 — 10% lower than its “optimistic” projections, specifically attributed to “...the costs of running Anthropic models from paying customers, in a process known as inference, on servers from Google and Amazon,” adding that those costs were “23% higher than the company anticipated.” In February, The Information ran another story saying that OpenAI’s gross margins fell from 40% in 2024 to 33% in 2025, a full 13% lower than its projected margins of 46%, all because (and I quote) “...the company having to buy more expensive compute at the last minute in response to higher than expected demand for its chatbots and models.” You know, exactly what Anthropic has had to do. This is what I’ve referred to as the knife-catching problem for compute demand — you either don’t order enough compute and have to rush to buy some last-minute as demand intensifies, or you order too much, and, well, to quote Dario Amodei: And right now, as I’ve covered , there’s not enough compute being built to keep up with Anthropic or OpenAI’s voracious demands, meaning that they will both be bartering to buy whatever’s available at whatever price it’s available at. This naturally will savage their already-negative margins… …and then what? No, really, and then what? One of you fucking AI boosters, answer me, how does this actual reverse course? Because even if Anthropic were making $100 billion in annual revenue, it would probably be losing $300 billion or more to get there. The fact it had to raise $30 billion in February , $15 billion in April, and now $30 billion more in May all while allegedly pulling in more than $3 billion a month in revenue suggests that its COGS are fucking horrendous, and its growth is coming at a terrible financial cost. Let’s say that Anthropic keeps growing and ( as The Information suggests ) hits $100 billion in annualized revenue (around $8.3 billion a month). How, exactly, does it afford to make that much money? Because right now it’s (allegedly) about to hit $45 billion in annualized revenue, and needs so much money that it’s absorbing (along with OpenAI) the majority of venture capital raised this year, and very clearly does not have any path to bring its costs down. The answer is simple: it can’t! There is no mechanism to do so. More compute does not make OpenAI or Anthropic’s services cheaper to offer. There is no magical silicon coming that will make any of this more affordable, and no, Anthropic is not “profitable on inference,” because if it were, that massive revenue growth would have leveled out its margins rather than require it to raise a little less than the combined value of every Major League Baseball team , or more if you add the other $50 billion that Amazon and Google have promised based on secretly-held performance obligations. The same goes for OpenAI, which “raised” $122 billion (around $45 to $50 billion in real cash, with the rest either paid in installments or on it IPOing or reaching (sigh) AGI) in February and is now already considering raising more . Somebody might counter-argue that this is companies raising as a means of boosting their valuations, I think that’s a very convenient way of looking at two extremely problematic companies. I should also ask why neither of them appear to be seriously considering going public. While both were rumoured earlier in the year to be planning to do so in 2026, both appear poised to raise more private capital. I think the answer is simple: their CFOs know that doing so would reveal their actual margins, which are hot dogshit with sprinkles on top. Nobody has a sensible or logical response here. Which leads us right to our next point! One important detail to keep in mind here is that as of a month or two ago, Anthropic moved all enterprise customers to token-based-billing, which will begin, I believe, a true stress-test of the true “value” of AI as costs skyrocket. Just last week I ran the first of a two (or three, potentially) part premium series called “What If We’re In An AI Bubble?” and touched on the gruesome subject of whether organizations could afford to pay for AI long-term : Earlier in the week, carnival barker and Salesforce CEO Marc Benioff said his company would spend $300 million on Anthropic tokens in 2026 , and as I discussed in my premium from Friday , unrestrained AI spending is inflating the revenues of Anthropic and OpenAI in a way that isn’t sustainable for anybody involved: The problem is simple: nobody actually knows how much AI is going to cost them in any given quarter. This means that the current token spend you’re seeing is entirely experimental, which is why organizations keep burning through their tokens so fast. This massive growth in spend is what underpins the “massive” (I have serious questions about its accounting) growth in Anthropic’s revenue. Executives have, across the board, given their engineers free reign to burn as many tokens as they’d like, and while I severely doubt that Anthropic actually hit $50 billion in annualized revenue outside of not-quite-fraudulent non-GAAP measurements, I believe its revenue growth has come from an artificial boost from a tech industry searching for a reason to pay somebody money. To be very clear about what I mean, I think there is currently an AI token binge across both Anthropic and OpenAI. Enterprises do not know the actual value of AI, and do not know how much they should actually be budgeting, which is why Uber and others are running through their token budgets but not, it seems, spending less. We’re currently in an abundance phase — one where nobody is truly thinking about the costs outside of their fear of missing out — but there’s this nasty undercurrent of “wait, how much does that cost?” followed by “oh, fuck, well…you know I love AI but…” Put another way, the current spend on AI tokens is not something that’s indicative of lasting, reliable revenue. In some cases, the pressure to use AI for everything is turning companies’ software stacks into slop. Things are worse elsewhere. Something is wrong at Zillow. Something about LLMs has done something to its technical leadership, something that makes them talk strange and send weird slide decks with confusing, slop-ridden sentences. The real estate tech firm spent over $1 million on AI services in the first quarter of 2026, and in April it spent $749,000 in tokens across Cursor and Anthropic’s services, as well as through AWS Bedrock. As of the end of the month, it was nearly 75% of the way through its annual Cursor token budget of $1.1 million. As of the middle of May, its total AI spend had already crested over $300,000, and its Cursor budget sat dangerously close to the edge at 85%. This is particularly-concerning when you consider that Zillow’s net income for Q1 2026 was $46 million , and ranged from $2 million to $10 million each quarter of 2025. Zillow is currently on course to spend at least $7 million on AI in 2026, and at its current pace might hit as much as $10 million, which would amount to a little less than 50% of its 2025 net income ( $23 million ). You’re probably wondering how Zillow manages to spend so much on AI, and the answer — as I’ll get into in next week’s free newsletter — is that its technical executives appear to have AI psychosis, saying that the short-term goal is for “software engineers to never open a code editor again.” The reality is chaos. In a slide deck that I’ll discuss later, Zillow revealed that while engineering resources have largely stayed the same, outputs requiring human review have increased by nearly 50%. Meanwhile, code deployments and pull requests increased by 39%, and software reviewer load increased by 29,000 hours each month , creating a massive burden on the 1,500 or so engineers working at the company. In simpler terms, that’s about 19 hours of extra work per engineer that’s literally just looking at extra code written by LLMs. On Blind, the anonymous social network for tech workers, Zillow workers complain about Zillow’s code “slowly becoming AI slop,” with “much more code getting approved without guardrails or input due to people not being able to keep up the other’s velocity or just not caring anymore.” One worker claimed that “the slop is job security,” adding that they “don’t want the output to be good or documentation to be clean [as] management will replace [them] with offshore/nearshore/AI agents at the slightest whiff of evidence that the slop cannon is self sustaining.” Another said that they felt “lost in the agentic world,” and that they “didn’t have full grasp of where we are going or what [their] role is,” with a “lot of overlap in what people are doing.” Another said that “people are burning tokens just to hit internal AI adoption targets,” adding that “this is what happens when leadership ties metrics to usage instead of outcomes,” saying that it “literally subsidized busywork.” This is all part of what an internal slide deck viewed by this publication called “AI-Native Engineering,” promising a “path to an agentic Zillow” and “faster outcomes for customers,” though customers are never mentioned in any other slide. The deck — pumped full of AI-generated text — talks about “generic AI being a commodity,” saying that “Zillow-aware AI is a competitive advantage,” and at no point explains what that means. It encourages engineers to go from “AI-Assisted” to “AI-Native,” with “systems enabling org-wide leverage,” with engineers moving from being “soloists” — individual developers with AI tools — to “conductors” that orchestrate AI agents, to “composers” that “define systems AI can safely play,” adding that “2026 is the transition from conductor to composer.” Yet the strangest part is named “2027: A Tuesday,” discussing a theoretical day in the office for whoever is left at the company. This theoretical example is, apparently, a process that would take weeks, but now takes under two hours.” Zillow intends, based on this deck, to sacrifice everything to AI — code review, vulnerability fixes, policy checks, deployments, testing, and basically having agents take over everything , no matter how small, like having an agent do dependency updates and security hotfixes that could be handled with a simple shell script. To quote Zillow: In practice, sources at Zillow tell me that there has been no actual movement toward this vision. Software engineers still open IDEs and review code manually, with one describing Zillow’s “vision” as “nonsense,” adding that “you can’t just throw buzzwords on a slide deck and change how all the engineers do their jobs.” As for why token burn is so high, sources tell me that engineers are actively encouraged to use AI for everything , as much as possible, writing PRD (product requirement documents) in AI, then using the AI to make stuff based on the PRD, then doing a deck with AI, then writing emails with AI, using AI to brainstorm, or create weird, esoteric automations, with some managers pushing workers to have one personal AI “goal” to aspire to. Zillow’s agentic “vision” is apparently a remit from the C-suite. It’s hard to tell if this is AI psychosis or just classic Business Idiot bullshit. Perhaps it’s a little of both. Every organization I’ve talked to has exceeded or is nearing the edge of their annual token budget barely five months into the year, which means that everybody has suddenly given themselves an extra few million dollars’ worth of operating expenses for reasons that escape effectively everybody I’ve talked to. Every engineer tells me the same thing: “I’m being made to do this, I don’t want to do this, my managers do not seem to understand, my bosses seem to understand even less than my managers, and if I don’t use AI somebody is going to fire me.” Put another way, CEOs and CTOs are screeching at their underlings to “use AI as much as possible” to “find its incredible benefits” without anybody really knowing what those are and how much it’ll cost to get there. This might be because Anthropic obfuscates the data that might tell customers the real costs. Per Laura Bratton at The Information , Bratton’s article has numerous quotes from executives saying that Anthropic lacks transparency and granularity into the ways that tokens are being burned across an organization, in a way that I think sounds very, very suspicious, particularly when you add the following: While I’m not accusing Anthropic of anything untoward, massive, multi-million dollar contracts that involve individuals burning thousands or tens of thousands of dollars’ worth of tokens with no service level agreement, transparency or true granularity into the burn is a perfect setup for a company — not saying it’s Anthropic! — to do something dastardly with those numbers. While an individual might be able to monitor their own personal usage, in an organization of hundreds or thousands of engineers, who’s to know if, say, the particular token burn is consistent across every member of the company, or that those costs are actually matching up with what the user is doing? This is a company ostensibly worth $900 billion dollars acting with disregard for the basic measurement of “how much did this cost, and how did it cost so much?” And in the end, how do you even measure it at scale? Say you’ve got 1,500 engineers, and they’re spending a combined $1 million tokens a month. How the fuck do you actually measure the return on investment for that spend? How many tokens does it take to do one thing? Is it consistent across every model? Is it consistent across every employee? Are you even measuring how many tokens a task costs? Because if you’re not, that token budget is basically throwing a dart blindfolded. Okay, now you’ve measured a task, did you make sure to measure it multiple times? Because LLMs can randomly do things differently even with the same prompt and same Claude.MD file and same strictures and same data sources. You’re gonna need at least 10 samples of each task, and you’re gonna need to make sure somebody who actually knows what they’re doing can measure them, because if you get a dimwit, they’re going to say it can do something it can’t. Unless, of course, you can’t actually measure how many tokens a particular task can take with much accuracy, in which case every single AI token budget is bullshit. And each model does things differently depending on many different variables, some of them a result of the user, some of them a result of the AI labs themselves. Alright, well, maybe you just need KPIs — measurements you can aspire toward , and by pursuing them you can start working out how much it costs to do stuff. Wait, which metric works there exactly? In fact, it’s pretty hard to measure anything like “efficiency” or “productivity” in any business, because every metric connected to them can be gamed, leaving managers and executives with the problematic situation where they have to start learning how things work so they can see if they’re good. Before AI, this wasn’t as much of a problem, in the sense that inefficiencies and wasted hours weren’t directly connected to a chatbot that is specifically designed to burn money. Managers and executives could come up with whatever deranged, self-gratifying office bullshit they pleased, wasting hours of people’s time in the process, but doing so didn’t immediately connect to a massive, ever-increasing cost. AI is a perfect storm of failed concepts and organizations, and the apex of the Era of the Business Idiot , an epoch where we’re ruled by people so thoroughly disconnected from the actual workforce that it was inevitable that a technology would be created specifically to grift them. LLMs are dangerous for many, many reasons, but the under-discussed one is how well they play to a certain kind of executive imbecile. Generative AI is — to quote Mo Bitar — really good at doing an impression of work, much like most managers and c-suite executives, and even if it’s completely incapable of doing something, it’ll absolutely say it can and tell you you’re amazing for suggesting it. And that’s why Business Idiots love it. Where regular human beings would say annoying things like “that’s not possible within that timeline” or “we don’t have the resources to do it,” AI will say “of course, right away!” and burn as many tokens as possible. When it makes mistakes, it’ll apologize — as it should because it failed you — but then promise to do better next time, all while costing so much less, at least in theory , than a regular, stinky human being. It’ll create a PRD of a theoretical software project with the confident and vigor that you need to take it immediately to a software engineer and say “build this immediately,” and when the software engineer tells you a bunch of bullshit about it not being possible , it’ll spit out several convincing-sounding responses. Fuck, why even bother talking to that engineer at all? Claude Code can mock up a prototype that you can then shove in their fucking face before you fire them for not using AI to do it themselves. Any executive-level fuckwit you’ve met in your life now has a seemingly-powerful tool that can burp up mimicry of open source software and, if you constantly prompt it, eventually get something half-functional onto some sort of web server. When you face bugs, it’ll try and fix them, sometimes also “fixing” (adding or deleting code) from elsewhere to be helpful, like when Cursor using Anthropic’s Claude Opus 4.6 model deleted an entire production database and all its backups . It will never, ever say no, even if it’s incapable, even if it has no thoughts, even if what you are asking is equal parts impossible and unreasonable in both its timescale and scope. A Business Idiot, given his druthers, can sit there and fuck around and make an LLM spit out something that makes him feel like he’s coding, which in turn makes him feel that you, a lazy and stupid engineer , could do even more with the power of AI. It doesn’t matter that it costs an absolute shit-ton of money, or that there’s no way to measure its efficacy. The Lion does not concern himself with things like “efficacy” or “productivity,” and the Lion is increasingly tired of your whining! The Lion doesn’t even understand what it is you do every day other than not doing what The Lion is asking for! You laugh, but this is genuinely how the majority of managers and executives think and act, and now they have a special chatbot that can fart out functional-enough prototypes to convince a Business Idiot they can do anything, because executives and managers do not regularly do much work and thus have no idea what it looks like other than when they look over your shoulder, which is why they wanted you back in the office! Organizations aren’t burning millions or hundreds of millions of dollars a year on AI because it’s good , they’re doing it because they are run by people who do not know what the fuck they’re doing. In a sane world, randomly adding a massive, ever-expanding operating expense to your business with the express intent of — to quote IT firm Workato’s CIO , “eating the costs while employees experiment” — would have the board blow up your house. In our world, one dominated by disconnected, self-involved and massively-overpaid dullards, many businesses pushing their workers to use AI are doing so because the other guy is doing it, with about as much strategy and forethought as one would expect from somebody who spends 90% of their life reading emails, going to meetings, or going to lunch. The majority of those I see trumpeting the so-called benefits of AI do not appear to do anything of note. I have yet to see one so-called multi-agent orchestrator engineer psychopath ship something remarkable or impressive or even functional. I have yet to see any AI-obsessed boss write or create or author or do anything I can remember. I don’t see any of these fuckwits running a company on their own outside of those who have learned to sell stuff to other AI psychosis victims or executive midwits of varying size. And why oh why is it always the language of inevitability and possessiveness? Nobody who’s this insistent, aggressive and violative with their language of “it’s here and if you don’t adopt it you’re stupid and dead” has ever been right about anything. Nobody this desperate, insistent and forceful has ever had good intentions, good vibes or brought good omens — they are always bearers of some kind of con. Most technology is sold on elevating and ascending human beings. AI cheapens every interaction by creating a work-shaped product from a person that doesn’t respect you enough to give you work that’s barely fit for a human because it wasn’t made for one. You must accept becoming a dogshit dealer that loves accepting and receiving low quality goods. You must celebrate intentionless and decaying slop, and defend it and the machine that made it with your entire being. You must sully yourself — treat its unexceptional, sloppy and unreliable outputs as signs of sentience, or at least the proof that digital sentience is possible. You must defend horrible, abrasive, ugly, loud monoliths of steel full of $50,000 graphics cards. You must say they are necessary, and you must aggressively antagonize those who do not. Every time you defend generative AI you defend a machine of capital that has burned $1 trillion and created one of the most-wasteful products in history. If people disagree with you, you must attempt to harm them somehow — ostracize them, mock them, attack them, denigrate them. You will justify this as moral, because you have been manipulated by a technology built and sold by two of the greatest grifters of all time — Dario Amodei and Sam Altman. Anything less is opposition to an industry with all the trappings of authoritarianism down to the media toadies, the propaganda and the seizure of land in the name of a nebulous “greater good.” But man, these men got people good. Sam Altman helped propagate a technology perfect for conning people with potential, a larger extrapolation of Altman’s own life of taking dogshit — Loopt, for example! — and parlaying it into larger opportunities. It can make a really half-hearted demo of a lot of things, and that’s good enough to sell to Business Idiot. Dario Amodei took this grift and perfected it. Anthropic is a company purpose-built to con people into giving it by money by making people feel smart. LLMs can do work-shaped stuff, sometimes, as long as you debase yourself to accept mediocre and often-broken stuff that you have to keep a vigilant eye on, and either use a subsided product that loses Anthropic money or pay a shit ton of money as an enterprise to Anthropic and they still lose money. These companies were only capable of growing in an economy dominated by the gullible and work-shy. Only a capitalist culture dominated by people who don’t actually do or know stuff have let this get so far. Nobody wants this, nobody wanted it since the beginning, it was forced upon everyone, and to pretend otherwise is laughable and offensive. The amount of people who use this shit a bit and become convinced that we’re mere years from it costing over a trillion dollars to somehow making trillions of dollars and being an entirely different and good product should be aware that they are being manipulated. The more you feel compelled to defend AI the more scrutiny you must show it. I am not your enemy! If you think that I am, you are on the side of a corporation or a product. You can try it, like it, and I don’t really care, but the second I see you trying to be condescending or judgmental or aggressive toward another person for not agreeing with your product choices I immediately feel suspicious. Can’t you see how these people act? Can’t you see how strange it is to defend a thing you pay money for that has terrible economics? If it wasn’t the “in” thing, being an AI person would be considered really weird. I look forward to the day it is. I hope you guys like having the stuff you said since 2022 repeated back to you! I’ve been saving it all. Time is running out for a graceful bow, and you better act quick! If you feel self conscious while other people dunk on AI, that’s weird! I see people say they don’t like Macs all the time. Who gives a fuck! I’m not going to go to the mat for Tim Cook. People can make their own decisions. Those comparing AI to AOL mailing CDs to people should feel ashamed of themselves. This is like if every single time you opened a magazine an AOL CD flew at your head, your boss told you he would replace you with a modem if you didn’t go online, and the news constantly ran segments called “I didn’t receive an email: father forgets son forever because he wasn’t online” or panels with “Internet experts” who said “I am on the Internet superhighway right now, and I’m certain that within 10 years AOL Time Warner will be able to email myself to my dad.” Imagine if Shingy was a billionaire and went on TV every day in 1999 and told you “ the world must get ready, because you’re about to get a ICQ message from The Lord .” Generative AI was purpose-built to grift an economy run by executives and managers who don’t actually do any work. Its success has been driven by a remarkable, society-wide ignorance in the management sect, and its continued proliferation is only possible through the media’s continued trust and faith in the idea that CEOs are busy because they’re actually doing work. Yet even a Business Idiot eventually realizes that too much money is being spent, and the first one of these dimwits to cut their token budget will send the rest of them running for the doors. We should lock them. We should make everybody who obsessed over theoretical ideas about what AI can or will do ashamed for their intellectual deceit or constant ignorance. At the end of the AI era, the only thing that will change the rot at the heart of our economy is the acceptance that the majority of companies are run by lazy, self-involved and ignorant fuckwits, and accountability for those who refused to scrutinize them. Microsoft has spent a total of $293.8 billion in capex since the beginning of Fiscal Year 2023 (which began in the back half of 2022). This means that around 30% of Microsoft’s capex ($87 billion) went to building OpenAI’s infrastructure. Based on discussions with sources familiar with Azure architecture, this is the vast majority of Microsoft’s operational capacity. AI revenues have to explode. Capex has to stop being invested. GPUs need to be margin positive, including both their cost and the debt associated with operationalizing them. AI revenue has to stay consistent both before and after you stop spending that capex. Microsoft’s RPOs jumped from $392 billion to $625 billion between Q1 and Q2 FY26 (or calendar year Q4 2025 and Q1 2026), driven by the $250 billion in “incremental Azure spend” from OpenAI (including already-existent commitments) locked up in October 2025 and the $30 billion promised as part of its deal with Anthropic from November 2025 . Based on Microsoft’s own disclosures , without Anthropic and OpenAI’s additions, RPO would have been effectively flat, as evidenced by the fact that in Q3FY26, remaining performance obligations sat at $627 billion . Amazon’s RPOs jumped from $244 billion in Q4 2025 to $364 billion in Q1 2026, driven by its February 2026 $100 billion expansion of its $38 billion compute deal from November, and its extended partnership with Anthropic for 5GW of compute capacity unattached from any kind of dollar number. Google’s RPOs jumped from $242.8 billion in Q4 2025 to $467.6 billion in Q1 2026, driven by ( per The Information ) $200 billion in committed spend on TPUs and compute from Anthropic, meaning that it has expanded its future revenues by an unremarkable $24.8 billion when you remove Anthropic’s spend, when RPOs had previously jumped $85 billion between Q3 and Q4, likely driven by its compute deal from October 2025 . It’s fair to assume a chunk of the remaining RPOs are from its deal to rent TPUs to Meta , announced in February 2026, which makes it likely that it accounts for the majority of the remaining $24.8 billion. Silicon will get cheaper. They’ll start selling services. They’re profitable on inference. It’s an example of a typical working day. At 8:30AM, the engineer notes that confirmation rates in Dallas dropped 3% overnight. ‘Dallas inventory spiked; buyers went from 3 showings to 7. The agent shows the pattern: we're hitting the same buyer 7 times in 24 hours with "tour confirmed" pings. They're overwhelmed; they're muting us.’ The line before this says: “I don't open the codebase — I open the spec and eval dashboard.” Half an hour later, the engineer changes the spec, which is then tested against previous data, showing an improvement. “The PM and I review diffs, check guardrails, approve.” Diffs are “differences” — essentially comparing two versions of the same document to see which lines have been changed. The code is then rolled out. At 11AM, the senior engineer mentors a junior engineer: ‘A junior engineer's rescheduling agent is failing evals. I ask one question: "What happens if the buyer picks a slot the seller just blocked while the agent is negotiating?" We identify the race condition and add a constraint: "Always re-check availability at confirmation time." She updates the spec and evals. The agent passes.’ It is absolutely adorable they’re pretending that they’ll have junior engineers if this hellscape vision comes to life. You can’t say “burn as many tokens as possible,” because employees will — as happened at both Amazon and Meta — deliberately create ways to burn more tokens using scripts and automations. You can’t say “use AI every day,” because even if they do so, that doesn’t actually set up a success criteria. You can’t tell software engineers to try and “ship more software,” because that, again, emphasizes doing more, not making good stuff , and leads to an increase in velocity rather than how good the stuff is. You can’t say “pull requests” or any other metric a software engineer can manipulate, because in 100% of the situations where you give a software engineer a number to hit they will focus entirely on hitting that number.

Finance

Dart

Business

0 views

The Universe of Joshua Blais 1 months ago

Back to silence

“All of man’s miseries stem from his inability to sit quietly in a room alone.” — Blaise Pascal When I think about the challenge of living in the modern world, the constant that continually crosses my mind is that we are so unable to revel in silence anymore. We fill the potential silence with noise, continual distraction, the inability to sit with our thoughts. Music, podcasts, audio books, we have the content available to never have to sit with ourselves ever again. I have started to, when driving alone, to turn off all music and noise and just be in the silence as I drive. This time is helpful to re-center, to think about that which all of “this” is, and to find some semblance of peace. I could, instead, listen to my 40th audiobook of the year, or that podcast in which the guests are saying nothing of purpose or value for the 3rd time this week. That is the thing - nothing at all is being said . Nothing is being brought up that would make this existence “more” or “better”. In fact, the more that I consume, the less that I feel that I am anyone at all. With children at home, with a family, there is never a quiet moment. But - I love the noise and fun that comes with it. The difference is that there is nowhere we can seemingly go today without the noise - the notifications and the colors and lights and sounds that are designed to have us give over our attention so as to become better consumers. The “noise” that my children produce is nothing in comparison to the algorithmic noise that is meant to keep me from finding and talking to God. In fact, the noise my children produce bring me closer to God . Let us discuss how to get back to the silence. I see some people’s phones and am horrified by that which I see: Notification blobs everywhere, taking up their whole screen and pinging constantly. It is a wonder that people have any attention span at all these days. These applications are designed to be that little bird in your ear, telling you that they are right there so that you can spend your ever decreasing time on this Earth scrolling mindlessly. People literally watch others livestreaming their life - instead of living their own. On my phone, I have simply disabled all notifications that are not phone calls. Even then, I set daily periods of airplane mode between 9pm - 9am in which I am unreachable and offline People know they can reach me by calling me. When I pick up, I say “hello, what’s wrong” - I don’t want to be disturbed. In the future, I will only have a data plan: no phone number, and only authenticated methods such as xmpp for voice will be allowed. My phone will remain in airplane mode for most of the day. The thinking that we should always be accessible is asanine, it is the affliction of the age that must be fought against with ruthlessness. On my computers, I have turned off the notifications center in noctalia shell, so I literally never get any notifications whatsoever. I never seen anyone pinging me, emails coming in, messages on any applications, etc. I check IRC a few times every other hour, but otherwise have no idea if anyone is trying to reach me. I no longer allow unfettered browser usage, and limit the browser to certain hours of the day. I believe that unmetered and uncontrolled access to the internet is not something that is healthy to the human being, yet it is now commonplace with the little computer in our pocket, you would be considered quite “weird” to do limit connectivity - well, so be it! Email is supposed to be asynchronous by default. Instead of replying immediately, I check emails once at 9am and once at 4pm. No more than this. Again, I include in my emails that if anything urgent needs to be handled, to simply call me. I get so many interesting emails on a weekly basis and want to reply to them all, so this batches my time to give quality responses and encourage interaction. I have never understood how people have notifications across a half dozen applications on their devices, but were I to allow social media on my phone (I don’t), I would recommend turning off all notifications and never going back. The best way to mitigate this point of noise is to not install it in the first place. The more that I use technology, the more I want analogue methods of communication and work. So, I defer to my notebook and pen daily. Once an evening, I then organize these notes in my org-mode setup. Face to face communication is preferred over phone calls or texting. The time has always been ripe to reclaim the peace that we are meant to have. Everyone is moving so quickly without thought of where we are going, and it is detrimental to the human being. So, turn it all off. The world wants you to live on fear, of missing out, of missing the proverbial train. But nobody ever asks “where is the train going?” Perhaps it is better to miss the train than to be driven off the cliff with everyone else. As always, God bless, and until next time. If you enjoyed this post, consider Supporting my work , Checking out my book , Working with me , or sending me an Email to tell me what you think.

Culture

0 views

Ed Zitron's Where's Your Ed At 2 months ago

Where Are All The Data Centers?

If you liked this piece, please subscribe to my premium newsletter. It’s $70 a year, or $7 a month, and in return you get a weekly newsletter that’s usually anywhere from 5,000 to 18,000 words, including vast, detailed analyses of NVIDIA , Anthropic and OpenAI’s finances , and the AI bubble writ large . My Hater's Guides To Private Credit and Private Equity are essential to understanding our current financial system, and my guide to how OpenAI Kills Oracle pairs nicely with my Hater's Guide To Oracle . My last piece was a detailed commentary on the circular nature of the AI economy — and how the illusion of AI demand is just that, an illusion. Subscribing to premium is both great value and makes it possible to write these large, deeply-researched free pieces every week. During every bubble there’s one very obvious thing that keeps happening: things are said, these things are repeated, and are then considered fact. Sam Bankman-Fried was the smiling, friendly, “ self-made billionaire ” face of the crypto industry. NFTs were the future of art, and would change the way people think about the ownership of digital media. The actual evidence, of course, never lined up. NFT trading was dominated by wash trading — market manipulation through two parties deliberately buying and selling an asset to raise the price. Cryptocurrency never took off as anything other than a speculative asset, and altcoins are effectively dead . Sam Bankman-Fried was only a billionaire if you counted his billions of illiquid FTX tokens, but that didn’t stop people from saying he wanted to save the world weeks after the collapse of Terra Luna, a stablecoin that he himself had bet against and may have helped collapse . Three months before his arrest, a CNBC reporter would fly to the Bahamas to hear SBF tell the story of how he “ survived the market wreckage and still expanded his empire, ” with the answer being that he had “stashed away ample cash, kept overhead low, and avoided lending,” as opposed to the truth, which was “crime.” The point is that before every scandal is somebody emphatically telling you that everything’s fine. Everything seems real because there’s enough proof, with “enough proof” being a convincing-enough person saying that “most of FTX’s volume comes from customers trading at least $100,000 per day,” when the actual volume was manipulated by FTX itself , and the “$100,000 a day in customer funds” were being used by FTX to prop up its flailing token . In the end, the “proof” that SBF was rich and that FTX was solvent was that nobody had run out of money and that nothing bad had happened to anybody. SBF was a billionaire sixteen times over because enough people had said that it was true. Anyway, one of the most commonly-held parts of the AI bubble is that massive amounts — gigawatts’ worth — of data centers have both already been and continue to be built… …but then you look a little closer, and things start getting a little more vague. While Wood Mackenzie’s report said that there was “ 25GW of data center capacity added to the funnel ” in Q4 2025 does not say how much came online. CBRE said back in February that “net absorption of 2497MW” happened in primary markets in 2025 , with other reports saying that somewhere between 700MW and 2GW of capacity was absorbed every quarter of 2025. At the time, I reached out for any clarity about the methodology in question and received no response. Okay, so, I know data centers are getting built and that they exist . I believe some capacity is coming online. But gigawatts? Or even hundreds of megawatts? How much data center capacity is actually coming online? Why did Anthropic get so desperate it took on a years old data center, xAI’s Colossus-1 , full of even older chips from a competitor — one whose CEO described the company as “evil, ” and that’s currently facing a lawsuit from the NAACP over allegations the facility’s gas turbines are polluting black neighborhoods ? Remember, Colossus-1 is an odd data center, with around 200,000 H100 and H200 GPUs and an indeterminate amount of Blackwell GB200s, weighing in at around 300MW of total capacity… which isn’t really that much if we’re talking about gigawatts being built every quarter, is it? So, I have two very simple questions to ask: how long does it take to build a data center, and how much data center capacity is actually coming online? These simple questions are surprisingly difficult to answer. There exists very little reliable information about in-progress data centers, and what information exists is continually muddied by terrible reporting — claiming that incomplete projects are “operational” because some parts of them have turned on , for example — and a lack of any investor demand for the truth. Hyperscalers do not disclose how many data centers they’ve built, nor do they disclose how much capacity they have available. I find this utterly inexcusable, given the fact that Amazon, Google, Meta and Microsoft have sunk over $800 billion in capex (and more if you count investments into Anthropic and OpenAI) in the last three years . So I went and looked, and what I found was confusing. So, you’re going to hear people say “well Ed , data centers are being built ,” and what I’m talking about is data centers that have been fully constructed and then turned on . It’s really, really easy to find data centers that are under construction , but as I’ve discussed in the past, that can mean everything from a pile of scaffolding to a near-complete data center . Yet finding the latter is very, very difficult. I’ve spent the last week searching for data centers that broke ground in 2023 or 2024 that have actually been finished, and come up surprisingly empty-handed. Some projects are stuck in construction hell, eternally dueling with planning departments over permitting, some are chugging along with no real substantive updates, some, as is the case with Nscale’s Loughton, England data center, have done effectively nothing for the best part of a year , some are perennially adding more capacity to the order as a means of continuing raking in construction bills, and some are claiming their data centers are “operational” as only a single phase has turned on. You should also know that even once construction has finished, the buildings themselves must be fully filled with the necessary cooling, power and compute hardware, at which point it can be configured to meet a client’s specifications (which can take months), at which point the unfortunate soul building the facility can actually start making money. I think it’s also worth revisiting how difficult data center construction is, and how large these new projects are. This starts with a very simple statement: nobody has actually built a 1GW data center (to be clear, it’s usually a campus of multiple buildings networked together) yet. There are campuses — such as Stargate Abilene — which promise to reach 1.2GW, but nearly two years in sit at two buildings at around 103MW of critical IT load each with, based on discussions with sources with direct knowledge of Abilene’s infrastructure , a third building sitting fully-constructed but with barely any gear inside it. It’s fundamentally insane how many different companies are trying to build these things considering how difficult even the simplest data center is to build. Take, for example, American Tower Corporation’s edge data center in Raleigh, North Carolina, which I’ll mention a little later. This is a 1MW facility — or one-thousandth the size of a gigawatt facility — occupying 4000 sq ft of real estate at first and expanding to 16,000 if ATC actually gets it up to 4MW. That’s about two-and-a-bit times larger than the typical American home . And, from ground-breaking to ribbon-cutting , it took eleven months to complete. And that’s not including all the other necessary time-consuming bits, like finding land, securing permits, and so on. That’s a simple one. People want to build data center campuses a thousand times larger than that. Look at how difficult it is. In fact, it’s so difficult that the companies can’t build all of it at once. Larger data center campuses are almost always divided into “phases,” in part because that’s the smartest way to build them, and in part with the express intention of convincing you that they’re “fully operational.” For example, CNBC’s MacKenzie Sigalos reported in October 2025 that Amazon’s Indiana-based (allegedly) 2.2GW Project Rainier data center was “operational,” but only seven out of a planned 30 buildings were actually operational, and her comment of “with two more campuses [of indeterminate capacity] underway.” This comment was buried two videos and 600 words into a piece that declared the data center was “now operational,” with the express intent of making you think the whole thing was operational. To give her credit, at least she didn’t copy-paste the outright lie from Amazon, which claimed that Rainier was “ fully operational ” in a press release the same day. You’ll also note that Amazon never provides any clarity about the actual capacity of Rainier. Sigalos did exactly the same thing when the first (of eight) buildings of Stargate Abilene opened, declaring that “OpenAI’s first data center in $500 billion Stargate project is open in Texas,” burying the comment that only one was operational with another nearly complete several hundred words earlier. These are intentionally attempts to obfuscate the actual progress of the data center buildout, and if I’m honest, I’ve spent months trying to work out why big companies that were supposedly building large swaths of data centers would be trying to do so. Unless, of course, things weren’t going to plan. In its last (Q3 FY26) quarterly earnings call , Microsoft CEO Satya Nadella claimed that “[Microsoft] added another gigawatt of capacity this quarter, and [remained] on track to double [its] overall footprint in two years.” A quarter earlier , he claimed to have added “nearly one gigawatt of total capacity,” with Karl Keirstead of UBS saying that he “...thought the one gigawatt added in the December quarter was extraordinary and hints that the capacity adds are accelerating.” As I’ll discuss below, I can find no evidence of anything more than a few hundred megawatts of Microsoft’s data center capacity coming online. While I’ll humour the idea that it doesn’t announce every new data center, and that there may be colocation and neocloud counterparties ( 67% of CoreWeave’s revenue comes from Microsoft, for example ) that make up the capacity, as I’ll also discuss, I don’t know where the hell that might be. So, to be aggressively fair, I asked Microsoft to answer the following questions on May 4, 2026: A Microsoft representative from WE Communications promised to "circle back" by 5PM ET on Monday May 4th, but did not return further requests for comment via text and email, which is incredibly strange considering the simple and straightforward nature of my questions. That’s probably because the vast majority of its publicly-announced or documented data center capacity doesn’t appear to be getting finished. In September 2025, CEO Satya Nadella claimed that Microsoft had added 2GW of capacity “in the last year,” and acted as if Fairwater, a project with two actively-constructed data centers with one in Wisconsin that broke ground in September 2023 and another in Atlanta that broke ground in July 2024 , was something to be “announced” rather than “a very expensive project that has taken forever.” Nadella also claimed that there are “multiple identical Fairwater datacenters under construction,“ though he neglected to name them. To be clear, “Fairwater” refers to a project where multiple data centers are linked with high-speed networking to make one larger cluster, a project that sounds ambitious because it is , and also unlikely because it’s yet to have been built. Fairwater Atlanta — the latter of the Fairwaters — was “launched” in November 2025 and it’s unclear how much capacity it has. Cleanview claims it’s at 350MW of capacity , and Microsoft’s own community outreach page claims construction would be completed by the beginning of October 2025 , but, as I’ll get to, it’s unclear whether this is just one phase, given that reporting shows multiple other buildings still under construction . I have serious doubts that Microsoft stood up a 350MW data center in less than a year, given everything else I’m about to explain. Fairwater Wisconsin is also a data center of indeterminate size, but Cleanview claims Phase 1 is 400MW , quoting a story from FOX6 News Milwaukee from September 2025 that said that Microsoft was “investing an additional $4 billion to expand the campus,” featuring a video of a very much in construction data center saying the following: So, $3.3 billion — at a rate of around $14 million per megawatt per analyst Jerome Darling of TD Cowen — is about 235MW of capacity, which is a lot lower than 400MW. Seven months later, Satya Nadella said that the Fairwater datacenter in Wisconsin was “going live, ahead of schedule,” a sentence written in the present tense, but also said that it “ will bring together hundreds of thousands of GB200s in a single seamless cluster,” which is in the future tense. It’s a great time to remind you that Microsoft claims that it brought online roughly eight times that capacity (around 2GW) in the past six months. To make matters worse, it doesn’t appear that Fairwater Wisconsin is actually operational. Ricardo Torres of the Milwaukee Journal-Sentinel reports that Microsoft has said it isn’t actually online , and that while there “...is equipment inside the data center conducting start-up opportunities…the company anticipates [they] will continue to happen for the next several weeks.” Epoch AI’s satellite footage of Fairwater Wisconsin — which mentions a completely wrong capacity because it’s uniquely terrible at calculating it ( it claimed Colossus-1 has 425MW capacity, for example) — notes that as of April 2026, one building appeared to be operational, with a second under construction. So, that’s one building in Wisconsin that might be complete, and based on the permitting application from August 2023 dug up by Epoch, the project is designed to have 117MW of capacity, which is a lot lower than 235MW. While Epoch didn’t have permitting for building two, it did for three and four, which are designed to have around 719MW of capacity , and as of April 2026 still appear to be slabs of concrete. In simpler terms, there’s at most around 117MW of capacity running at Fairwater Wisconsin. The Fairwater data centers are Microsoft’s most-publicized data centers, yet they’re shrouded in secrecy, with the Atlanta Journal-Constitution having to file an open records request to find the site being developed by QTS, a data center developer owned by Blackstone . Videos of Fairwater Atlanta from last November show a giant campus with two large buildings and a patch of yet-to-be-developed dirt. DataCenterMap refers to it as “ under construction .” Epoch AI’s satellite footage notes that as of February 2026, building four’s roof was complete and “all mechanical equipment appears to be installed,” but “there is still a lot of construction activity around the building.” Based on air permits filed as part of the project (that Epoch found), it appears that each building is powered by a number of Caterpillar 3516C Generator Sets at around 2.5MW each, with building one having 47 (117.5MW), building two having 13 (32.5MW), building three having 30 (75MW), and building four having 35 (87.5MW). If we’re very generous and assume that three buildings are complete, that means that Fairwater Atlanta is at around 225MW of capacity (not IT load!). So, that’s about 342MW of data center capacity being built by one of the largest companies in the world, in its most-publicized and written-about data centers. Put another way, for Microsoft to come remotely close to its so-called 2GW of capacity in the last six months, it will have had to bring online a little under six times that capacity. I’m calling bullshit. I really did want Microsoft to give me some answers, but I’m very confused as to how it can remotely claim it brought even a gigawatt of capacity online in the last year. I also question whether Microsoft is actually building multiple other “identical” Fairwater data centers, as I can’t find any announcements or pronouncements or mentions or hints as to where they might be. In fact, I’m having a little trouble finding where else Microsoft has been building data centers, and those I can find are extremely suspicious. In Microsoft’s announcement of its Wisconsin data center , it mentioned two other projects — one in Narvik Norway that had already been announced months beforehand by OpenAI , and another with Nscale in Loughton, England that was also announced by OpenAI that very same day as part of the entirely fictional Stargate project . If you’re wondering how those are going, Microsoft had to take over the entire Narvik project (which does not appear to have started construction) from OpenAI , and the Loughton data center ( which OpenAI also backed out of ) is currently a pile of scaffolding . For two straight quarters , Microsoft has said it’s brought on an entire gigwatt of capacity,and I have to ask: where? Because when you actually look at the projects it’s announced, very little appears to have been built, and that which has is nowhere near its theoretical capacity. To be specific about what Microsoft is claiming, it’s saying it’s brought around 4GW of capacity online in the space of two years, and at a 1.35 PUE, that’s about 2.96GW of critical IT load, which works out to the power equivalent of around 284,600 H100 GPUs, which may be possible — after all, Microsoft apparently bought 450,000 H100 GPUs in 2024 — but I can’t find much evidence of data centers that could house that many GPUs, nor that might be in construction. Let’s dig in. Microsoft broke ground on three data centers in Catawba County North Carolina in 2024 — one in Hickory, another in Lyle Creek, and another in Boyd Farms: Alright, maybe I’m being unfair! Maybe it’s just a North Carolina problem. There must be another that broke ground and got built…right? Microsoft also broke ground on a data center in Quebec City, Canada in September 2024 , and as of April 2026 , “generator testing has been completed,” and “civil works will continue until Autumn 2026.” Okay, well, maybe it’s a Canada problem. What about Microsoft’s New Albany, Ohio data center that broke ground in October 2024 ? Well, as of March 2026, “spring activity would resume,” and “beginning soon, soil will be delivered to the site via a designated truck route. I’ll note that Microsoft specifically says that Ames Construction is currently leading it, and that it will “resume the lead role in project communications” once the final phase of construction is done at some unknown time. Alright, well, how about the August 2025 ground breaking in Cheyenne, Wyoming that was allegedly “ due to launch in 2026 ”? Well, Microsoft hasn’t updated its community page since it said there’d be a community meeting planned for November 2025 and that “neighbors within the vicinity will be notified ahead of construction,” which sounds like construction is yet to commence. Not to worry though, it announced on April 14, 2026 that it planned to expand it to “ accelerate innovation and economic growth ” How about that 2023-announced Southwest Hortolândia Brazil data center ? That’s right, the last update was in September 2025 , and the update was “construction activities continue to progress in alignment with local regulations.” A piece from Folha De S.Paulo from March 2026 mentioned that Microsoft “had begun operating its first artificial intelligence data centers in Brazil,” but satellite footage shows that it’s barely finished. What about the Newport, Wales data center it announced in 2022 ? Well, as of November 2025, a politician was standing on a concrete slab saying how many jobs it’ll theoretically bring in , which it won’t. What about Microsoft’s four data centers in Irving, Texas, announced December 2024 ? The best I’ve got for you is a news report about a data center in Irving Texas breaking ground in January 2025 . Its San Antonio data center, announced in July 2024 ? Well, construction was underway as of December 2025 , and it appears that construction will begin in the summer of 2026 on another one in the area. How about the two data centers outside of Cologne, Germany , announced in November 2024? Well, as of September 2025, Microsoft has… plans to build one of them ? …what about the 900 acres of land it bought in June 2024 in Granger, Indiana ? Great news! According to 16NewsNow , Microsoft officials “could break ground on a proposed data center…in late April or early May [2026].” How about Project Ginger West, a data center planned in Des Moines. Iowa since March 2021 ? Hope you like waiting , because Microsoft itself says that it’s estimated to finish construction in Summer 2028 . Ginger East , announced a few months later? Mid-2028 . Project Ruthenium ( announced 2023 )? I don’t have shit for you I’m afraid. Rutheniumkanda Forever! This company claims it’s built four fucking gigawatts of capacity , but when I go and look to see what it’s actually built I’ve failed to find a single announced data center from the last three years that got turned on outside of its Fairwater Atlanta and Wisconsin sites. To be clear, all of these sites are somewhere in the 200MW to 300MW range. For Microsoft to have brought online 4000MW of data center capacity in the last two years would require it to have completed thirteen or more of these projects, all while choosing not to promote them, with every project operating in such a veil of secrecy that no local or national news outlet reported a single one of them. I truly cannot work out how Microsoft has brought on any more than 500MW of capacity in the last year based on my research, and think Microsoft is deliberately obfuscating whether said capacity was contracted rather than actively in-use , much like CoreWeave refers to itself having 3.1GW of “ total contracted power ” but only added 260MW of active power capacity in a single quarter at the end of 2025. However, the exact verbiage used in Microsoft’s earnings transcripts is that it “added another gigawatt of capacity,” which sounds far more like it’s saying it brought them online… …but it didn’t, right? It obviously hasn’t. Where are all the data centers, Satya? Where are they? Why are your PR people too scared to tell me? No, really, where are they? So, to be fair, analyst Ben Bajarin, one of the more friendly pro-AI posters, argues that actually all of that capacity is secretly behind-the-scenes , something I’d humour if there was any kind of paper trail to a bunch of Microsoft data centers that were secretly being built. I’d also be more willing to humour it if any of the data centers that have been publicized as “breaking ground” had actually been finished, or if both Fairwater Atlanta and Wisconsin weren’t so deceptively-marketed. My only devil’s advocate is that Microsoft could, in theory , be working with colocation partners to stand up several gigawatts of capacity through shell corporations and SPVs, but even then , not a single one has any sort of trail to Microsoft? All of that capacity? It’s really, really weird, and the only answers I get are smug statements about how “Fairwater is ahead of schedule.” But if I’m honest, I’m having trouble even making these numbers add up. Considering how loud, offensive and conspicuous the AI bubble has become, it feels like we should have a far, far better understanding of how much actual capacity has been built. I also think it’s time to start being realistic about how long these things are taking to build. For example, I was only able to find a few data centers that for sure, categorically, definitively opened, and for the most part, it appears that a data center takes around 18 months to go from groundbreaking to opening. And these, I add, are all facilities that are relatively modest — at least, when compared to the kinds of gigawatt-scale campuses that are reportedly in active development. Digging deeper, I found a lot of projects stuck in development Hell: While there are absolutely data centers under construction , and some, somewhere , are actually being completed , the vast majority of projects I’ve found are either in a mysterious limbo state or, in most cases, under construction years after breaking ground. Across the board, the message seems to be fairly simple: it takes about 18 to 24 months to build any kind of data center, and the bigger they are, the less likely they are to get completed on schedule. Those that actually “come online” aren’t actually fully constructed, but have brought on a single phase — something I wouldn’t begrudge them if they were anything close to honest about it. In reality, data center companies actively deceive the media and customers about the actual status of projects, most likely because it’s really, really difficult to build a data center. In any case, what I’ve found amounts to a total mismatch between the so-called “rapid buildout” of AI data centers and reality. It also doesn’t make much sense when you factor in how many GPUs NVIDIA sold. In October last year, NVIDIA CEO Jensen Huang told reporters that it had shipped six million Blackwell GPUs in the last four quarters , though it eventually came out that he was counting two cores for every GPU , making the real number three million. I disagree with the framing, I think it’s incoherent and dishonest, but I’ve confirmed this is what NVIDIA meant. In any case, if we assume two cores per GPU, a B200 GPU has a power draw of around 1200W, for around 3.6GW of IT load for 3 million of them. I realize that NVIDIA also sells B100 and B300 GPUs (similar power draw) and NVL72 racks of 72 GB200 GPUs and 36 CPUs, but bear with me. Blackwell GPUs only started shipping with any real seriousness in the first quarter of 2025, which means that a good chunk of these data centers were built with H100 and H200 GPUs in mind. Nevertheless, I can find no compelling evidence that significant amounts — anything over 500,000 GPUs — of Blackwell-based data centers have been successfully brought online. When I say I struggled to find data centers that had been both announced and brought online, I mean that I spent hours looking, hours and hours and hours, and came up short-handed. I want to be clear that I know that there is Blackwell capacity actually being built , and believe that the majority of that capacity is retrofits of previous data centers, such as Microsoft’s extension to its Goodyear Arizona campus which it began building in 2018 that likely houses Blackwell GPUs. But I no longer believe that the majority of Blackwell GPUs are doing anything other than collecting dust in a warehouse. Blackwell GPUs require distinct cooling, a great deal more power than an H100, and cost an absolute shit-ton of money, making it unlikely that a 2023 or early-2024 era data center could handle them without significant modifications. I fundamentally do not believe more than a million — if that! — Blackwell GPUs are actually in service. If that’s the case, NVIDIA is likely pre-selling GPUs years in advance — experimenting with the dark arts of “ bill-and-hold ” — and helping certain partners like Microsoft install the latest generation to create the illusion of utility, availability and viability that does not actually exist. If I’m honest, I also have serious questions about the current status of many H100 and H200 GPUs. Based on what I’ve found, I’d be surprised if more than 3GW of actual capacity was turned on in the last two years, which means that NVIDIA has sold anywhere from double to triple the amount of GPUs that the world can hold. While the Anthropic-Musk compute deal is an obvious sign about xAI’s lack of demand for compute, it’s also, as I mentioned earlier, a clear sign that AI data centers are mostly not getting finished, and those that do get finished are taking two or three years even for smaller builds. While it sounds a little wild, I think in reality only a few hundred megawatts — if that — of actual, usable AI compute capacity is being spun up every quarter. If I was wrong, there’d be significantly more progress on, well, anything I could find. Why can’t Microsoft offer up a data center that isn’t called Fairwater, and why are its Fairwater data centers taking so long? How much actual capacity has Microsoft brought online? Because it certainly isn’t fucking 2GW in six months. I’m willing to believe that Microsoft has a number of collocation agreements with parties that don’t disclose their involvement. I’m also willing to believe that Microsoft doesn’t publicize every single data center it’s building or has built. 2GW of capacity is a lot. It’s nearly ten times the (likely) existing capacity of Fairwater Atlanta. If Microsoft is bringing so much capacity online, why can’t we find it, and why won’t they tell us? And no, this isn’t some super secret squirrel “they’re building secret data centers for the government” thing, it’s very clearly a case where “capacity” refers to “something other than data centers that actually got brought online. Despite their ubiquity in the media, AI data centers are relatively new concepts that are barely five years old. They are significantly more power-intensive than a regular data center, requiring massive amounts of cooling and access to water to the point that the surrounding infrastructure of said data center is often a massive construction project unto itself. For example, OpenAI and Oracle’s Stargate Abilene data center is (in theory) made up of two massive electrical substations , a giant gas power plant and eight distinct data center buildings, each with around 50,000 GB200 GPUs, at least in theory. Every data center requires that power exists — as in it’s being generated in both the manner and capacity necessary to turn it on, either through external or grid-based power — and is accessible at the data center site. This means that every single data center, no matter how big, is its own construction nightmare. You’ve got the power, the labor, the permits, the planning, the construction firm, the power company, the specialist gear, the temporary power (because on-site power is slow ), the backup power (because you can’t just rely on the grid for something you’re charging millions for!), the cooling, the uninterruptible power supplies — endless lists of shit that needs to go very well or else the bloody thing won’t work. These are very difficult and large projects to complete. Edged Computing’s (theoretically) 96MW data center in Illinois is 200,000 square feet in effectively two large squares. For comparison, every single inch of gambling space in Caesar’s Casino Vegas is around 130,000 square feet . These things are fucking huge, fucking difficult, and fucking expensive, and all signs point to capacity not coming online. Let’s go back to Anthropic mopping up Musk’s fallow data center capacity, which stinks of desperation for both companies. If there were modern data centers full of GB200s being turned on and available anywhere in the next month or two, wouldn’t it be more financially prudent to wait for it, even if it’s just on an efficiency level? A franken-center made up of H100s and H200s with some GB200s stapled onto the side feels like a stopgap solution. I have similar questions about the results of adding this capacity — that “...Anthropic plans to use [it] to directly improve capacity for Claude Pro and Claude Max subscribers ,” “doubling” (whatever that means) the 5-hour rate limit and removing the recently-added peak rate limits. What’s the plan here, exactly? Less than a month ago Anthropic’s Head of Growth, Amol Avasare , said that Anthropic was “looking at different options to keep delivering a great experience for users” because Max accounts were created before the era of Claude Code and Cowork . How does adding 300MW of capacity magically resolve that problem? Was that always the plan? Or was this a knee-jerk reaction to the surging popularity of OpenAI’s Codex ? Because the original justification for peak hours was that Anthropic needed to manage “ growing demand for Claude ,” demand that I bet Anthropic claims hasn’t gone anywhere. It’s also important to remember that last year, OpenAI’s margins (which are already non-GAAP), per The Information , were worse than expected because (and I quote) it had to “..to buy more expensive compute at the last minute in response to higher than expected demand for its chatbots and models.” In other words, Anthropic has deliberately tanked its already-negative 2026 gross margins by desperately buying the fallow compute from a company whose CEO threw up the nazi salute , called the company “ misanthropic and evil ,” and has the “right to reclaim the compute” if Anthropic “engages in actions that harm humanity.” Surely you’d wait a few months for some new, less tainted source of compute, right? And surely it wouldn’t be such a big deal, because new data centers get switched on every day, right? So, let’s get to brass tacks. Anthropic and OpenAI have now committed to spending $748 billion across Amazon Web Services, Google Cloud, and Microsoft Azure , accounting for more than 50% of their remaining performance obligations. The very future of hyperscaler revenue depends both on Anthropic and OpenAI’s continued ability to pay and both of them having something to actually pay for. I also think it’s fair to ask why Microsoft’s theoretical gigawatts of new compute aren’t producing tens of billions of dollars of new revenue. Microsoft’s $37 billion in annualized AI run rate (sigh) is mostly taken up by OpenAI’s voracious demands for its :compute , and only ever seems to expand based on OpenAI’s compute demands and the now 20 million lost souls paying for Microsoft 365 Copilot . There’s supposedly incredible, unstoppable demand for AI compute, and Microsoft is apparently sitting on gigawatts’ worth , but somehow those gigawatts don’t seem to be translating into gigabillions , likely because they don’t fucking exist. All of this makes me wonder what Google infrastructure head Amin Vahdat meant last November when he said that Google needed to double its capacity every six months to meet demand . Many took this to mean “Google is doubling its capacity every six months,” but I think it’s far more likely that Google is taking on capacity requests from Anthropic that are making said capacity demands necessary. Similarly, I think CEO Sundar Pichai’s comment that it would have made more money had it had more capacity to sell was a manifestation of a distinct lack of new capacity rather than a result of bringing on swaths of new data centers that immediately got filled. I also need to be blunt on two things: Look, I know it sounds crazy, but I’m telling you: I don’t think very many data centers are coming online! While I keep wanting to hedge my bets and say “I bet a few gigawatts came online,” I cannot actually find any compelling literature that backs up that statement. I’ve spent hours and hours looking, and I’ve come up with a few hundred megawatts delivered in the past two years. Every major project is stuck in the mud, a phase or two in, or facing mounting opposition from locals that don’t want a Godzilla-sized cube making a constant screaming sound 24/7 so that somebody can generate increasingly-bustier Garfields. I’m not even being a hater! It’s just genuinely difficult to find actual data centers that have been announced that have also been fully turned on. So, humour me for a second: if hyperscalers are bringing on hundreds of megawatts of capacity a year, then that means that the ever-growing quarterly chunks of depreciation ripped out of their net income are just a taste of what’s to come. Last quarter, Google’s depreciation jumped $400 million to $6.482 billion, with Microsoft’s jumping nearly a billion dollars from $9.198 billion to $10.167 billion, and Meta’s from $5.41 billion to $5.99 billion. While Amazon’s technically dropped quarter-over-quarter, it still sat at an astonishing $18.94 billion. Remember: depreciation only increases when an item is actually put into service. If Microsoft, Google, Amazon and Meta are sitting on tens of billions of yet-to-be-installed GPUs, and said GPUs are only being installed at a snail’s pace every quarter, that means that these depreciation figures are set to grow dramatically. In fact, year-over-year, Google’s depreciation has jumped 30.7%, Amazon’s 24.7%, Microsoft’s 23.9%, and Meta’s an astonishing 34.9% . And that’s with an extremely slow pace of deployment. I do kind of see why the hyperscalers are sinking capex into these big AI infrastructure gigaprojects now, though. Shareholders are currently tolerating the capex because they think stuff is coming online, and that’s where the “incredible value” is. When a $20 billion or $30 billion a quarter depreciation bill first rears its head — as I said, Amazon is close, reporting $18.945bn in depreciation and amortization expenses in the most recent quarter — it’ll become obvious that the only people seeing value from AI are Jensen Huang and one of the massive construction firms slowly building these projects. Actually, it’s probably important to state that I don’t think the majority of these projects are doing anything untoward I just don’t think any of them realized how difficult it is to build a data center, and unlike basically any other problem the tech industry has ever faced, simply throwing as much money as possible at it doesn’t really change the limits of physical construction. I think every one of these data center projects is its own individual construction nightmare, and thanks to the general market psychosis around the AI bubble, nobody has thought to question the core assumption that these things are actually getting built. With all that being said , I’m not sure that anyone building these things is moving with much urgency either. Perhaps they don’t need to — perhaps hyperscalers are happy, because they can continually string out both the AI narrative and put off those massive blobs of depreciation. But we really do need to reckon with the fact that nearly two years in, Stargate Abilene has only two buildings’ worth of actual, operational, revenue generating capacity, and nobody has given me an answer as to how it doesn’t have even a quarter of the 1.7GW of power it’ll need to turn everything on , if it ever gets fully built. Maybe they can really pick up the pace, but as of early April, barely any actual gear was in the third building. And then we get to the other problem: Oracle. As I’ve discussed before, Oracle is building 7.1GW of total capacity for OpenAI , and keeps — laughably! — saying 2027 or 2028, when at this rate, Stargate Abilene won’t be done until mid-2027, and the rest either never get finished or are done in 2030 or later. This is setting up a horrifying situation where Oracle desperately needs OpenAI to pay it for capacity that doesn’t exist, and if it ever gets built, it’s likely to be years after OpenAI has run out of money, which is the same problem that Microsoft, Google, and Amazon have with their $748 billion of deals with Anthropic and OpenAI, though thanks to the $340 billion or more necessary to build the Stargate data centers, Oracle’s problems are far more existential. I’ve repeatedly — and correctly! — said that the problem is that these companies didn’t have the money to pay for their capacity, but Oracle lacks Microsoft or Google’s existing profitable businesses to fall back on if these data centers are delayed, with its existing business lines plateauing and its only real growth coming from theoretical deals with OpenAI and GPU compute with negative 100% margins . Anthropic’s desperation for new sources of compute also suggests that it’s bonking its head against the limits of its capacity, and will continue to do so as long as it continues to subsidize its users . I also think that the slow pace of construction will eventually lead to OpenAI facing similar problems. These companies need to continue growing to continue to raise the hundreds of billions of dollars in funding necessary to pay Oracle, Google, Microsoft, and Amazon their respective pounds of flesh. It’s now very clear that the whole “inference is profitable” and “most compute is being used for training” myths are dead, because if they weren’t, Anthropic would either need way more compute or way higher-quality compute. Colossus-1 was specifically built as a training cluster, yet its current use is “reduce rate limits for our subsidized AI subscriptions,” which is most decidedly inference provided by three-year-old hardware . Despite writing over 9000 words and driving myself slightly insane trying to find out, I still haven’t got an answer as to how much actual data center capacity has come online. Hyperscalers have clearly been retrofitting old data centers to fit their new chips, and based on my research, I can find no compelling evidence that they’ve added more than a few hundred megawatts a piece since 2023. What I do know is that, across the board, a data center of anything above 50MW (or lower, in some cases) takes anywhere from 18 to 36 months to complete, and nobody has actually built a gigawatt data center despite how many people discuss them. For example, Kevin O’Leary — known as “Mr. Dogshit” to his friends — is allegedly building a 9GW data center in Utah , but he may as well say that he’s building a unicorn that shits Toyota Tacomas, as doing so is far more realistic than a project that will likely cost $396 billion, assuming that locals and bankers don’t drag him to The Other Side like Dr. Facilier . Nobody has built a 1GW data center, so I severely doubt Mr. Dogshit will be able to do anything other than create another scandal and lose a bunch of people’s money. In other words, any time you hear about a “new data center project,” add a year or two to whatever projection they give. If it’s 2027, assume 2029, or that it never gets built. Anything being discussed as “finished in 2030” may as well not exist. In any case, what I’m suggesting is that very, very few data centers are actually getting finished, and if that’s true, NVIDIA has sold years worth of chips that are yet to be digested. And if that’s true, somebody is sitting on piles of them. I’m trying to be fair, so I’ll assume that an unknown amount of data centers got retrofitted to fit Blackwell GPUs. But I also refuse to believe that even half of the three million Blackwell GPUs that got shipped have actually been installed. Where would they go? You can’t use the same racks for them that you would with an H100 or H200, because Blackwell requires so much god damn cooling. Another sign that these things aren’t actually getting installed is Supermicro’s $1.4 billion or so of B200 GPUs left in inventory from a canceled order from Oracle . Why not? Isn’t this meant to be a chip that’s extremely valuable? Isn’t there infinite demand? Is there not a place to put them? Apparently Oracle wanted to use faster GB200 GPUs from Dell , but why aren’t there other customers lining up to buy these things? Also… how was Oracle able to cancel an order of over a billion dollars’ worth of GPUs? Can anybody do that? Because if they can, one has to wonder if this doesn’t start happening as people realize these data centers aren’t getting built. Pick a data center. It’s probably barely under construction, or if it’s “finished” it’s actually “partly done” with no real guide as to when the rest will finish. Remember that $17 billion deal with Microsoft and Nebius signed ? The one that’s a key reason why Nebius’ stock is on a tear? Well, its existence is based on the continued construction of a data center out in Vineland, New Jersey facing massive local opposition, and multiple sources now confirm that construction has been halted due to local planning issues. The data center is horribly behind schedule already, and Microsoft has the option to cancel its entire contract if Nebius fails to meet milestones . That data center is a major reason that people value Nebius’ stock! It cannot make a dollar of revenue without its existence! It has the funds and blessing of Redmond’s finest — the Mandate of Heaven! — and it can’t get things done! This is bad, and indicative of a larger problem in the industry — that it’s really difficult to build data centers, and for the most part, they’re not being fully built! You’ve heard plenty about data centers getting opposed and canceled — how about ones that fully opened? No, really, if you’ve heard about them please get in touch, because it’s really difficult to find them. Why don’t we know? This is apparently the single most important technology movement since whatever the last justification somebody made up was, shouldn’t we have a tangible grasp? Because the way I see it, if these things aren’t coming online at the rate that people think, we have to start asking for fundamental clarity from NVIDIA about where the GPUs are, and when they’re coming online. NVIDIA’s continually-growing valuation is based on the conceit that there is always more demand for GPUs, and perhaps that’s true, but if this demand is based on functionally selling chips two years in advance. That makes NVIDIA’s yearly upgrade cadence utterly deranged. Buy today’s GPUs! They’re the best, for now, at least. By the time you plug them in they’re gonna be old and nasty. But don’t worry, it’ll take two years for you to install the next one too! To be clear, Blackwell GPUs are absolutely being installed! But three million of them? People love to use “enough to power two cities” to illustrate these points, but I actually think it’s better to illustrate in real data center terms. Stargate Abilene has taken two years to build two buildings of around 103MW of critical IT load. 3 million B200 GPUs works out to about 3.6GW of IT load. Do you really think that nearly thirty five Stargate Abilene-scale buildings were built in 2025? If so, where are they, exactly? You may argue that other data centers are smaller, and thus it would be easier to build. So why can’t I find any examples of where they’ve done so? By all means prove me wrong! It’s so easy! Just show me a data center announced or that broke ground in 2023 and find obvious proof it turned on. I’ll even give you credit if it’s partially open! The problem is that I keep finding examples of “partially complete” and those are the only examples of “finished” data centers. Isn’t this a little insane? This is all we’ve heard about for years, everybody is ACTING like these things exist at a scale that I’m not sure is actually true! I expect a fair amount of huffing and “well of course they’re coming online” from the peanut gallery, but come on guys, isn’t this all kind of weird? Even if you want to marry Sandisk and name your children “Western” and “Digital,” why can’t you say with your whole chest several data centers that got finished? We have macro level “proof” but when you try and look at even a shred of the micro you find a bunch of guys with their hands on their hips saying “sorry mate that’ll be another $4 million.” Something doesn’t line up, and it’s exactly the kind of misalignment that happens in a bubble — when infrastructural reality disconnects from the financials. NVIDIA is making hundreds of billions of dollars and it’s unclear how much of it is from GPUs installed in operational data centers. It feels like Jensen Huang might have run the largest preorder campaign of all time. This has massive downstream consequences. Sandisk, Samsung, SK Hynix, Broadcom, AMD, Microsoft, Google, Oracle, and Amazon’s remaining performance obligations total [find] and are dependent on being *able* to sell gigawatts worth of computing gear or compute access. If data centers are not getting built in anything approaching a reasonable timeline, that makes the future of these companies only as viable as the construction projects themselves. Even if you truly believe Anthropic will be a $2 trillion company and a $200 billion customer of Google, the compute capacity has to exist to be bought, and it does not appear to be built or, in many cases, anywhere further than the earliest stages of construction. If they don’t get built in the next few years, there’s no space for that solid state storage or those instinct GPUs. There’s no reason for NVIDIA to have reserved most of TSMC’s capacity , either. There’s also no reason to get excited about Bloom Energy, as it’s not making real revenue on those until Oracle finishes its data centers sometime between the next two years and never . And if they don’t get built, hundreds of billions of dollars have been wasted, with large swaths of those billions funded by private credit, which in turn is funded by pensions, retirements and insurance funds . I’ve got a bad feeling about this. Microsoft claims to have brought around 4GW of data center capacity online in the last two years, but it’s unclear how much actually got built. In an analysis of all announced groundbreakings and land acquisitions, it appears that Microsoft has only finished the first phase of its Atlanta and Wisconsin data centers. It is unclear where this capacity could be. When Mr. Nadella said on his most-recent earnings call that Microsoft had (and I quote) "added another gigawatt of capacity this quarter," did he mean active, revenue-generating capacity? In the event he did not, what did he mean? How much active, revenue-generating capacity has Microsoft brought online in FY2026 so far? Outside of Fairwater Wisconsin and Atlanta, where has that capacity been built? Microsoft’s latest update on the Hickory/Stover site is that it “will” begin “initial site setup and earthwork activities” as of February 2026, and it appears the contractor has changed from Ames Construction to Clayco. The latest Microsoft update on the Boyd Farms site is that it started construction on April 1, 2024. A February 2026 piece from the Charlotte Observer claimed it had started construction again after a 10 month (!) delay. The latest Microsoft update on the Lyle Creek site — which it adds began construction in March 2024 — is that its contractor, Whiting-Turner, “will begin initial site preparation once weather conditions allow” as of February 2026. A press release from a Canadian satellite firm from February 2026 said that it had “identified renewed construction activity at all three of Microsoft’s permitted data center campuses in Catawba County North Carolina.” Novva’s 60MW data center in Reno, Nevada. Announced in May 2023, operational as of July 2025 , or around 26 months. Edged Energy’s 36MW Phoenix, Arizona data center that broke ground in August 2024 and opened in April 2026 , or around 20 months. Duos Edge AI’s 450KW (lol) data center in Corpus Christi, Texas that was announced in July 2025 and opened in May 2026 , or around 10 months. Edge Energy’s 24MW, Columbus, Ohio-based data center that broke ground in August 2024 and opened in September 2025 , or around 13 months. American Tower’s 1MW (scalable to 4MW!) Raleigh, North Carolina data center that broke ground in June 2024 and came online in May 2025 , or around 11 months. EdgeCore’s 36MW Santa Clara, California data center campus that broke ground in January 2023, said it would be “energized in Q1 2024,” and opened in September 2025 , or around 32 months . Edged Energy’s “180MW” data center in Atlanta broke ground in July 2023 , and around 33 months later in April 2026 , it managed to top off a single 42MW building . EdgeCore’s two-building, 216MW campus that broke ground in August 2023 with plans to complete “as early as late 2025” is, as of March 2026, still under construction. Edged Energy broke ground on a 100MW data center in Aurora, Illinois in May 2023 , and has, as of February 2025, successfully opened (per DataCenterDynamics) “phase 1” — 24MW of capacity — but in its own press release from the same day referred to it as 96MW , choosing not to refer to any phases or separate buildings, something it has done since before the 24MW phase was complete. CyrusOne’s 40MW Aurora, Illinois data center broke ground in October 2024 , which was apparently so significant that CyrusOne would announce that it had broken ground a second time on January 28 2025 . Confusingly, CyrusOne has another campus it’s linking to the Bilter Road one on Diehl Road, which may or may not be the same one, and as of May 2026 is still very much under construction . As of March 2026, locals were still opposing the data centers , slowing down the process further. Vantage’s “192MW” OH1 data center in New Albany Ohio broke ground in October 2024 , with its first phase to be due live sometime in 2025. As of August 2025, Vantage had topped off the second building , and per its own website about OH1 , the first building was meant to be operational in December 2025, but it’s unclear whether it actually opened. PowerHouse’s 65MW data center campus in Reno, Nevada broke ground in October 2024 , and its website states that “delivery” will happen in April 2026, with “construction/delivery” due “Q3 2024 to Q2 2026.” Oppidan’s Carol Stream, Illinois data center broke ground in November 2024 , with the “first phase” due live in 2026. Per Clearview, it is still “ planned .” Databank’s 20MW Ashburn, Virginia “IAD4” data center that broke ground in July 2024 was “set to go live in Q1 2026,” and as of May 2026 is still referred to in the future tense on Databank’s website . Aligned’s 96MW “NEO-01” Ohio-based data center that broke ground in May 2024 was “scheduled to be opened by end of this year” as of March 2026 . Aligned’s 72MW Hillsboro. Oregon data center campus broke ground in October 2023 , topped off the first building in July 2024 (Aligned also plans a separate building, too!), and as of May 2026, Cleanview still marks the first one as “planned.” Flexennial broke ground on a Denver-based 22.5MW data center in October 2024 , and as of April 8. 2026, a local Facebook group has said that it will be operational by January 2027 . Flexennial, on the other hand, has been referring to it as “ the new build ” — in terms that make it sound like it was built — as far back as February 2025. If hyperscalers are truly not bringing on that much capacity, they cannot make those hundreds of billions of dollars from Anthropic and OpenAI. The current “AI compute demand is insatiable” narrative is utterly false , and a direct result of a lack of capacity coming online.

Finance

Java

0 views

Nicolas Bustamante 2 months ago

Long Running Agent Engineering

What does it take for an agent to keep working after you leave? Not "answer a long question." Not "use a big context window." I mean actually keep working. Hours. Days. Maybe weeks. Wake up in a fresh session, understand what happened before, choose the next useful thing, make progress, verify it, leave the workspace cleaner than it found it, and do it again. For the last few years we have mostly talked about agents as if the hard thing was autonomy inside one conversation. Give the model tools. Put it in a loop. Let it call bash, edit files, search the web, open a browser, run tests. That loop is real, and it is already enough to change how software gets built. But long running agents expose a different problem. The agent loop is not the product. The harness is. The model does not naturally persist across turns, context windows, sandboxes, process crashes, or days of work. A fresh session is born with amnesia. It has no idea what the last session tried, which tests failed, which files were half edited, which plan is stale, which shortcut was tempting but wrong, or whether the thing it is about to mark done was already marked done three runs ago and later discovered broken. That is the real long running agent problem: handoff across amnesia. The answer emerging across Anthropic, Cursor, OpenAI, Claude Code, Addy Osmani's survey of long running agents , and the Ralph Wiggum community is surprisingly consistent. It is not one magical always awake model. It is not stuffing the whole history into a bigger window. It is a harness that externalizes state into the workspace, restarts agents with fresh context, uses machine verifiable checks as backpressure, and assigns completion judgment to something other than the worker that wants to be done. Here is the punchline up front: Long running agents are not long conversations. They are recoverable workflows. The model is one worker inside that workflow. The durable artifacts are the real continuity layer. It also helps to separate three ideas people collapse into one phrase: long horizon reasoning, long running execution, and persistent agency. A model can reason through a deep task without running for days. A process can run for days without remembering anything useful. An agent can remember the user without owning one large task. Production systems blur the three, but the engineering problems are different. Here's what I'll cover: The naive version of a long running agent is a single agent in a single conversation with a very large context window. This works for small tasks. It fails exactly where long running agents are supposed to matter. The failure is not just that the context window fills. A 200K or 1M token window still becomes a junk drawer if you keep pushing tool outputs, diffs, plans, screenshots, stack traces, and half obsolete reasoning into it. The model does not get a clean working memory. It gets an archaeological site. Anthropic's effective harnesses post frames this cleanly: complex tasks span multiple context windows, but each new agent session begins with no memory unless the environment itself tells the story. They describe two predictable failures. First, the agent tries to one shot too much, runs out of context, and leaves a half implemented mess. Second, a later session looks around, sees progress, and decides the whole project is done. That second failure is the one I keep seeing. The agent is not lazy. It is locally rational. It sees a repo with code, some tests, maybe a UI that loads, maybe a checklist with many items checked. In the absence of a crisp external completion contract, "looks basically done" becomes an attractive stopping point. Long running work makes this worse because every session inherits ambiguity from the previous one. Compaction helps, but compaction is not continuity. A summary can preserve some facts, but it cannot replace a workspace that is structured for recovery. This is the same lesson as agent memory engineering, just at task scale. Memory that lives only in the context window dies when the window dies. Work that lives only in the agent's chain of thought dies when the session dies. If you want continuity, put it somewhere the next worker can read. The architecture that keeps recurring looks like this: There are variations, but the spine is stable. Anthropic uses an initializer agent plus repeated coding agents. The initializer creates the environment future agents need: an , a progress file, a feature list, and a first git commit. Subsequent agents read the state, pick one not yet passing feature, implement it, test it end to end, update the progress log, and commit. The community Ralph Wiggum pattern is the minimal version: The important thing is not the loop. The important thing is what the loop forces. Every iteration starts with fresh context. Every iteration rehydrates from disk. Every iteration must leave disk in a state the next iteration can understand. Blake Crosley's Ralph Loop writeup describes the same pattern through stop hooks: intercept exit attempts, persist state to the filesystem, and restart with a fresh context window until machine verifiable completion criteria are met. Geoffrey Huntley's community guide reduces it to a beautiful primitive: a shell loop feeding a prompt file to the agent, with the implementation plan on disk acting as shared state between otherwise isolated runs. That is the thing people keep underestimating. The loop can be dumb if the workspace is smart. No blackboard server. No bespoke orchestration database. No vector store. No "agent society" with vibes based coordination. Markdown files, git, tests, and a process supervisor. Annoyingly simple. Annoyingly effective. The Ralph loop works because it replaces one degrading conversation with many clean attempts. The agent is not continuous. The workspace is. This flips the unit of autonomy. You stop asking, "Can this one conversation survive for ten hours?" You ask, "Can each session leave enough evidence that the next session can continue without asking me?" That means the agent's job is not only to build. It has to maintain the run state. A good Ralph prompt usually contains four contracts: This is not glamorous. It is project management for an amnesiac coworker. The loop also gives you a natural escape hatch. If the agent goes off track, you edit the plan. If the prompt is too loose, you add a guardrail. If the tests are weak, you strengthen the oracle. If the agent keeps duplicating work, you make completed work more visible. If it keeps touching unrelated files, you narrow the write scope. The prompts you start with are never the prompts you end with. Long running harnesses are tuned by watching failure patterns. That is why Ralph is more than a meme. It is the first pattern that made the correct abstraction obvious: the human sits outside the loop and engineers the environment, not inside the loop approving every step. The roles keep converging: Sometimes these are separate prompts. Sometimes separate models. Sometimes separate processes. Sometimes the judge is a test suite. Sometimes it is a small evaluator model. But the roles are conceptually different, and mixing them is where harnesses get mushy. The initializer is the first agent that touches the task. Its job is not to implement the product. Its job is to make implementation possible across many future sessions. Anthropic's initializer writes a comprehensive feature list. In their clone example, the feature list expanded the user's high level prompt into hundreds of end to end feature requirements, all initially marked failing. This prevents the later worker from inventing a tiny definition of done. A good initializer creates: The initializer is where you spend tokens to save tokens later. Every future worker starts faster because the workspace already has a map. The worker should not be asked to "finish the project." That is how you get giant diffs, brittle code, and fake completion. The worker should be asked to make one bounded unit of progress. The stop matters. A worker that never stops slowly turns into the bad single session architecture. Fresh starts are not overhead. Fresh starts are the mechanism that keeps drift from compounding. The worker should not be the final judge of completion. Workers want to be done. Not emotionally, obviously, but statistically. The completion token is attractive. The model has a strong prior toward wrapping up once the output looks coherent. On long horizon tasks this creates false positives. Claude Code's productizes this separation. You give Claude a completion condition. After each turn, a separate evaluator model checks whether the condition has been met. If the answer is no, the evaluator's reason becomes guidance for the next turn. The worker model is not the only judge of its own success. That one design detail is huge. OpenAI's harness engineering post describes a similar review loop: Codex writes code, reviews its own changes, requests additional agent reviews locally and in the cloud, responds to feedback, and iterates until reviewers are satisfied. They explicitly call this a Ralph Wiggum loop. The pattern generalizes: The judge does not have to be smarter than the worker. It just has to be fresh, narrower, and less invested in the worker's local narrative. Long running agents need durable state, but not all state is the same. If this state lives only in the transcript, the next session has to reconstruct it. If it lives on disk, the next session can read it. Anthropic's scientific computing post is the cleanest non web app example. Claude worked over multiple days on a differentiable cosmological Boltzmann solver and reached sub percent agreement with the reference CLASS implementation. The interesting part is not that the model wrote numerical code. The interesting part is the harness discipline around it: reference implementation, test oracles, persistent notes, git history, and quantifiable progress. Scientific computing makes the verification problem unusually crisp. You can compare your solver to CLASS or CAMB. You can plot error over time. You can watch the agent get closer to a reference implementation. That gives the run a real gradient. Most coding tasks have weaker oracles, so you have to build them. Long running agents magnify weak specs. A human can carry fuzzy intent across a week because humans have common sense, memory, and the ability to ask clarifying questions. An unattended agent will happily optimize the wrong proxy for hours. The more autonomy you grant, the more literal the state layer has to become. A long running agent without verification is just a text generator with file permissions. Verification is what turns motion into progress. This is why end to end tests matter so much. Anthropic observed that Claude would often mark features complete after shallow checks. Once explicitly prompted to use browser automation and test as a human user would, performance improved. That matches my experience. Unit tests are useful, but they are often too close to the implementation. Browser tests force the agent to confront the product surface. The right verification depends on the domain: The best verification is machine checkable and hard to game. The worst verification is asking the same model, in the same context, "are you sure?" That does not mean model judges are useless. They are useful when they judge surfaced evidence against a narrow condition. Claude Code's docs are careful about this: the evaluator does not run commands or read files independently. It judges what Claude has surfaced in the conversation. So the completion condition has to include how the worker should prove it. The judge cannot save you from a vague goal. It can enforce a crisp one. Single worker loops are enough for many tasks. But the moment you want to run hundreds of agents on one codebase for weeks, coordination becomes the whole game. Cursor's scaling agents post is useful because it talks about what failed. Their first approach let agents coordinate as peers through a shared file. Agents would check what others were doing, claim a task, update status, and use locks to prevent duplicate claims. This sounds reasonable. It is also exactly the kind of distributed system that gets weird fast. The problem is not that agents cannot coordinate. The problem is that peer to peer coordination asks every worker to think about the global project while also doing local implementation. That is too much. Cursor moved toward a planner worker judge hierarchy: This is the same role separation again, just scaled out. Workers should not coordinate with other workers if you can avoid it. They should receive a task with a bounded write scope, complete it, and report back. The planner should own the global dependency graph. The judge should decide whether the current state is good enough to continue, merge, or stop. This has a strong human engineering analogue. You do not ask every engineer on a large project to constantly negotiate the whole roadmap with every other engineer. You create ownership boundaries. You run reviews. You integrate. You keep the shared state legible. The hard part is choosing the grain size. Cursor's product follow up, Expanding our long running agents research preview , says long running agents produced substantially larger PRs while keeping merge rates comparable to other agents. That is the product significance. The harness lets agents take on work that previously exceeded the practical size of a single agent session. But "larger PRs with comparable merge rates" is not magic model dust. It is the result of better state, better delegation, better judges, and better recovery. Long running agents need a computer. That computer should be disposable. An agent that can run commands, install packages, edit files, open browsers, and call APIs is powerful enough to be useful and powerful enough to be dangerous. If you run it on your laptop with all your cookies, SSH keys, cloud credentials, and private files, the blast radius is ugly. The long running version makes this worse. A five minute agent can do damage. A five day agent can do creative damage. So the production architecture increasingly separates durable harness state from disposable compute. OpenAI's Agents SDK update points in this direction: model native harnesses, sandbox execution, filesystem tools, memory, manifests, and state rehydration. The key idea is that the agent gets a controlled workspace with the files, tools, and dependencies it needs, while credentials and durable orchestration live outside the sandbox. If the sandbox dies, the run should not die. The harness should rehydrate a fresh sandbox from the last checkpoint, mount the workspace, hand the worker the current state, and continue. This is the same principle again: state must outlive the worker. Sandboxing also changes how you think about tools. In a local interactive agent, giving bash broad access is convenient. In a long running cloud agent, every tool is a capability grant. Network, filesystem, credentials, browser profile, package installation, deploy keys, issue tracker access, email access. Each one needs scope. The Ralph community guide makes this point bluntly: assume the agent environment will be popped at some point, then ask what the blast radius is. That is the right mental model. The best long running harnesses will feel boring operationally: Boring is good. Boring means the agent can be weird without the system becoming weird. There are two product directions converging. The first is the practitioner loop: prompt files, plans, hooks, shell scripts, git commits. This is how power users run agents overnight today. It is messy, flexible, and close to the metal. The second is the productized loop: , cloud agents, background tasks, research previews, SDK harnesses, managed sandboxes. This turns the same patterns into a UX that normal teams can use. The underlying mechanics are more similar than they look. Claude Code's is basically a session scoped Ralph loop with a model judge. Cursor's long running agents are a cloud product built from planner worker judge orchestration. OpenAI's Agents SDK is standardizing the sandbox and filesystem substrate. Anthropic's harness posts are turning the workflow into repeatable environment design. The abstraction is moving up the stack. In 2024, you wrote your own while loop. In 2025, you wrote prompt files and hooks. In 2026, the loop is becoming a product primitive. But the product primitive still has to answer the same questions: The UI can hide the loop. It cannot remove the harness. Long running agents fail differently from short running agents. Short running agents fail by making a bad tool call, hallucinating an answer, editing the wrong file, or stopping too soon. Long running agents fail by accumulating drift. Each failure suggests a harness feature. This is why long running agent engineering looks less like prompt hacking and more like operating a tiny software organization. You need task intake, planning, execution, QA, review, release, rollback, observability, and security. The agent is the worker. The harness is the company. Here are the questions every long running agent system has to answer. My current bias: Fresh sessions beat giant sessions. A fresh context window that reads good state from disk is better than a stale context window carrying ten hours of tool output. Restarting is not giving up. Restarting is garbage collection. The workspace is the memory bus. Plans, progress logs, feature lists, tests, screenshots, git commits, and benchmark outputs are not side effects. They are the continuity layer. If the next worker cannot understand the run from disk, the harness is broken. Judges should be separate from workers. The worker can propose done. Something else should decide done. Ideally tests. Sometimes a model evaluator. Often both. The judge should inspect evidence, not vibes. External verification matters more than longer reasoning. A mediocre plan with a strong oracle will often beat an elegant plan with no backpressure. The agent needs reality to push back. Keep worker scope small. A long running system does not require each worker to do a long task. It requires the whole system to sustain progress across many bounded tasks. Make state disposable and regenerable. Plans rot. Progress logs bloat. Specs change. A good harness can regenerate the plan from the current repo and goal. Treat planning artifacts as useful scaffolding, not sacred truth. Sandbox by default. Long running agents should assume hostile inputs, accidental exfiltration, bad generated code, and runaway loops. Least privilege is not paranoia. It is table stakes. The human's job moves up a level. You stop micromanaging tool calls and start designing the environment: better specs, better evals, better prompts, better ownership boundaries, better recovery points. That last point is the real mindset shift. When code was scarce, the human wrote code. When code became cheap, the human reviewed code. When agents became persistent, the human designs the system in which code keeps getting written after they leave. OpenAI calls this harness engineering, and I think that phrase is going to stick. Harness engineering is the work around the model that makes the model useful over time: This is different from traditional software engineering. You are not only writing deterministic code paths. You are designing an environment that a non deterministic worker can repeatedly enter, understand, act inside, and leave in a better state. That is why the best long running agent harnesses feel weirdly old fashioned. Git. Markdown. Shell scripts. JSON checklists. Test suites. Logs. Small commits. Clear ownership. These are not legacy habits. They are the primitives that survive context death. The future of long running agents is not one immortal session thinking forever. It is many mortal sessions, each with a clean context window, waking up inside a workspace that remembers. So back to the original question: what does it take for an agent to keep working after you leave? Not a bigger prompt. Not just a better model. A durable state layer. A crisp goal. A fresh worker loop. A judge that is not the worker. Tests that push back. Git history that tells the story. Sandboxes that can die without killing the run. Logs that let the human tune the system when it fails. The model is the engine. The harness is the vehicle. And the companies that get this right will not merely have "agents that run longer." They will have agents that can be trusted with larger units of work because the work is recoverable, inspectable, and verifiable. That is the threshold that matters. Not autonomy as theater. Autonomy with a receipt. Why Long Sessions Fail - Context windows rot, agents declare victory early, and half finished work becomes invisible The Architecture That Won - Fresh worker sessions plus durable workspace artifacts The Ralph Loop - Why a dumb restart loop beats a single heroic conversation Initializer, Worker, Judge - The three roles that keep showing up State Outside the Model - Feature lists, progress logs, plans, git history, tests, and notes Verification As Backpressure - Why test oracles matter more than better pep talks Multi Agent Coordination - Why peer to peer locks break and planner worker hierarchies survive Sandboxing and Rehydration - Why long running execution needs disposable compute and durable state What This Means For Agent Design - The checklist every long running harness has to answer Where does state live? What does a new worker read first? How does it choose work? How does it prove progress? Who decides it is done? How do you recover from a bad turn? What happens when the sandbox dies? What is the budget? What is the blast radius?

JSON

Shell Bash