Posts in Performance (20 found)

Remote EngFlow caching meets an old G4

During BazelCon 2026 in early October, the fine folks at EngFlow gave me a very cool piece of swag: a Cache & Carry box or, in other words, a portable remote cache for Bazel builds. The only ask was for me to post a photo on social media, which I’d have gladly done, but… that seemed like too low a price for a nice piece of hardware. So I promised I would write an article instead. The article is late, though, as I have been busy getting ZKeys out the door and trying to find a good idea for what to talk about. I finally got the right inspiration after taking an old PowerBook G4 laptop from 2004 out of the closet to play with Tabby-PPC . That experiment failed, but as the laptop sat on my desk, I thought: could I make it leverage the Cache & Carry and demonstrate the clear benefits of remote caching? The answer was “of course yes” before even trying, but it was not going to be easy: this machine is severely underpowered by today’s standards and won’t run Bazel, so showing remote caching via a modern build system wasn’t going to be possible. I had to find an alternative demonstration path, and find it I did: the recc compiler wrapper . Even then, the benefits weren’t great on the first try, so I had to debug and iterate. In the end, the compilation time for itself dropped from a little over 1 hour to just 3 minutes. Let’s see how. Typing articles like this one, with manually-taken pictures and hand-edited videos, requires a lot of work. Subscribe to Blog System/5 to show your support. EngFlow is a relatively new company that offers a hosted remote caching and execution service for Bazel builds. More specifically, EngFlow is an implementation of the Remote Execution (RE) API . And fun fact: the protocol name is RE, not RBE as many people call it, because RBE was Google’s own RE service that never really saw the light of day. The EngFlow Cache & Carry in its box before unpacking it. Cache & Carry is a little toy that packages the EngFlow remote cache (not execution) software into a Raspberry Pi 4 preconfigured to work as an appliance, enclosed into a branded 3D-printed case and packaged into a fancy cardboard box. Mind you: this is what I wanted to achieve with my own EndBOX last year but never went the last mile to distribute it as a truly finished toy. The EngFlow Cache & Carry Raspberry Pi 4, with its custom branded case. Getting started I’m quite used to EngFlow’s product by now because we use it at Snowflake to power our in-house builds and tests, so using the Cache & Carry was no difficulty. And it shouldn’t be for anyone given that configuration and management are pretty easy. Here is the cluster status page right after plugging in the device and connecting to it, showing how this “cluster” is really just a cache and has no remote execution features: EngFlow's standard Cluster status page in the live dashboard. Using it to run Bazel builds is trivial, really. All it took was to stick these few lines in my file, as the instructions say: And just like that, builds started showing up in the UI: EngFlow's standard Invocations browser in the live dashboard. This is one of the weakest parts of the UI, to be honest, because of its poor search capabilities, but the team is hard at work redesigning it. We can even peek into one of the builds to observe cache statistics: EngFlow's Invocation Details drill-down view in the dashboard. Here we can see the cache usage status and that the whole build was cached and took only 17 seconds to run (compared to the 13 minutes it takes otherwise). One thing that did surprise me about this setup is the ability to carry power and Ethernet over a single USB-C cable between the Raspberry Pi 4 and the host. I didn’t know this was possible, and it’s a great feature: if you just want to use the Cache & Carry with your laptop, all you have to do is plug it in with one cable and it just works. Getting Bazel to work against EngFlow was easy. But… as I told you earlier, EngFlow is not just a remote caching and execution service for Bazel: it is an implementation of the RE API. With some imagination, you would think that there might be clients other than Bazel for this API, and you would be right. Buck 2—Meta’s build system—is an obvious example, but there are more niche clients. And the one I discovered while searching whether supported EngFlow and finding that it didn’t was : a compiler wrapper like and that uses RE for caching and remote execution. Building and installing is easy: But note I installed the binary as , not . The rationale for this is that we have to configure . This is done via environment variables, and as I don’t like to pollute my regular environment with program-specific noise, I created a launcher like the following: With this in place, we can run a single compilation command and verify that the wrapper runs: And, with a repeated invocation, we can ensure that caching is in effect: OK, time to pick up the PowerBook G4 from 2004. This little 12” laptop sports a PowerPC G4 (7447A) running at a whopping 1.5 GHz, with 768 MB of RAM, a super-duper-slow 80 GB spinning drive, and a 100 Mbit Ethernet link. Not a lot, you’d say… but this computer was frigging amazing back in the day. I had the iBook G4 back in the day and was always jealous of the PowerBooks I saw in the wild, unaffordable to me, so when I could grab this one for $40 a few years ago, I just had to. EngFlow Cache & Carry in its box next to the PowerBook G4 12” from 2004 we’ll use as a client. Right now, this machine is running Debian forky/sid, one of the only Linux distros out there that still works out of the box on 32-bit PowerPC. As of 2026-11-09, it reports: With the specs out of the way, let’s go back to the premise of the article: showing you what a massive and unsurprising difference (remote) caching can make during compilations, and how, because RE is “just a protocol”, we can leverage it even from old machines. For this, we’ll measure ’s own build from a clean slate, cached and uncached, because it’s a large enough C++ project that takes about an hour to build out of the box—not too little, but also not so much as to be discouraging. Here are the final numbers of the comparison: Let’s break them down. My first attempt at using remote caching was to build with itself in the “default” configuration for and the toolchain installed by Debian. This meant GCC 16.2 and no custom flags to CMake. The clean, fully-cached build took about 8 minutes to finish. This showed a nice ~7x speedup compared to the clean unwrapped build, but somehow it felt like not enough. Why did we still spend 8 minutes to compile something when everything should be cached? I had two theories, but neither of them was valid: The machine doesn’t have a lot of RAM for C++ compiles. Was causing it to thrash, maybe because it buffers files in memory instead of transferring them in chunks? Looking at the code, the answer was no: implements chunked uploads and downloads. The machine’s Ethernet port is slow. Were network transfers slowing things down? Not really either: the output tree is only ~150MB after a build. (Mind you, with Bazel and remote execution, you’d typically use to avoid ferrying large intermediate object files, but that’s not an option with a compiler wrapper that knows nothing about the end-to-end build graph.) With both theories debunked, I had to look elsewhere: CPU consumption. It is no surprise that compiler wrappers are compute-heavy. One specific CPU-bound operation for C++ wrappers is dependency scanning: these wrappers have to (re)discover on their own what a specific compilation action depends on, because nothing tells them upfront. When the wrapper sees something like , it can tell that depends on … but what about the thousands of other implied dependencies like header files and toolchain binaries involved in that innocent invocation? You must know what those are in order to generate a cache key that can disambiguate different environments correctly. Taking some measurements, dependency scanning was indeed the problem as more than 5 out of the total 8 minutes of the cached build were spent on it. Looking through the codebase, I found that implements support for : a helper tool that ships with Clang and is specifically designed to speed up dependency discovery. Would it help here? I switched to Clang, reran the build and… whoops, still no bueno. The unwrapped build was, interestingly, much faster than the GCC one, but did not help: the logs claimed that it succeeded only for 23 out of 124 scanner runs, and so it didn’t materially change the overall result. Looking further, I found that was not being effective: ’s scanner adds to the compile commands, and because precompiled headers (PCHs) had been built without that option, Clang rejected using it. So… what if we disable PCHs by adding to the CMake invocation? Well, it turns out this made no difference either because CMake emits in this case, and apparently probes for these flags wrong, which in turn causes to fall back to slow dependency scanning. I applied a very hacky, targeted fix to to ensure was effective, reran the measurements, and got these numbers. Focus on the PCH? and Dep scanning columns: Yay! End-to-end clean, fully-cached build times fell from 8 minutes to 3 minutes, and those 3 minutes are pretty much the cost of the entire build and account for local non-compiler work such as protobuf generation and linking. Still not zero, but much better than what we started with. Here is the 1-minute video comparing build times. The difference is pretty massive: Recording of recc's own compilation on a PowerBook G4 without and with remote caching. As you can see, there is nothing magical about remote caching, and EngFlow’s Cache & Carry is a little toy to play with this technology. You can, of course, download an open source alternative such as Builbarn , but playing with a preconfigured appliance was fun and removed a lot of the complexities of setting such a system up. When paired with a modern build system such as Bazel, remote caching (and execution) can be trivially enabled and can achieve near 0-second build times when the whole build is cached: these build systems have precise knowledge of the full build graph and know which paths in it are or are not invalidated for a given build. And because they have this knowledge, they can apply optimizations to the whole build such as not downloading intermediate artifacts. But as we have seen, RE is just a protocol, and if you can speak it, you can leverage it from pretty much any machine (even if ancient and “worthless”) and any build system. You’ll hit some inherent limitations, like a compiler wrapper having only a very narrow view of the build, or you’ll have to deal with strange corner cases that don’t allow maximum efficiency, but you can make it work. Keep caching and carrying! The EngFlow Cache & Carry in its box before unpacking it. Cache & Carry is a little toy that packages the EngFlow remote cache (not execution) software into a Raspberry Pi 4 preconfigured to work as an appliance, enclosed into a branded 3D-printed case and packaged into a fancy cardboard box. Mind you: this is what I wanted to achieve with my own EndBOX last year but never went the last mile to distribute it as a truly finished toy. The EngFlow Cache & Carry Raspberry Pi 4, with its custom branded case. Getting started I’m quite used to EngFlow’s product by now because we use it at Snowflake to power our in-house builds and tests, so using the Cache & Carry was no difficulty. And it shouldn’t be for anyone given that configuration and management are pretty easy. Here is the cluster status page right after plugging in the device and connecting to it, showing how this “cluster” is really just a cache and has no remote execution features: EngFlow's standard Cluster status page in the live dashboard. Using it to run Bazel builds is trivial, really. All it took was to stick these few lines in my file, as the instructions say: And just like that, builds started showing up in the UI: EngFlow's standard Invocations browser in the live dashboard. This is one of the weakest parts of the UI, to be honest, because of its poor search capabilities, but the team is hard at work redesigning it. We can even peek into one of the builds to observe cache statistics: EngFlow's Invocation Details drill-down view in the dashboard. Here we can see the cache usage status and that the whole build was cached and took only 17 seconds to run (compared to the 13 minutes it takes otherwise). One thing that did surprise me about this setup is the ability to carry power and Ethernet over a single USB-C cable between the Raspberry Pi 4 and the host. I didn’t know this was possible, and it’s a great feature: if you just want to use the Cache & Carry with your laptop, all you have to do is plug it in with one cable and it just works. Bypassing Bazel Getting Bazel to work against EngFlow was easy. But… as I told you earlier, EngFlow is not just a remote caching and execution service for Bazel: it is an implementation of the RE API. With some imagination, you would think that there might be clients other than Bazel for this API, and you would be right. Buck 2—Meta’s build system—is an obvious example, but there are more niche clients. And the one I discovered while searching whether supported EngFlow and finding that it didn’t was : a compiler wrapper like and that uses RE for caching and remote execution. Building and installing is easy: But note I installed the binary as , not . The rationale for this is that we have to configure . This is done via environment variables, and as I don’t like to pollute my regular environment with program-specific noise, I created a launcher like the following: With this in place, we can run a single compilation command and verify that the wrapper runs: And, with a repeated invocation, we can ensure that caching is in effect: Let’s go retro OK, time to pick up the PowerBook G4 from 2004. This little 12” laptop sports a PowerPC G4 (7447A) running at a whopping 1.5 GHz, with 768 MB of RAM, a super-duper-slow 80 GB spinning drive, and a 100 Mbit Ethernet link. Not a lot, you’d say… but this computer was frigging amazing back in the day. I had the iBook G4 back in the day and was always jealous of the PowerBooks I saw in the wild, unaffordable to me, so when I could grab this one for $40 a few years ago, I just had to. EngFlow Cache & Carry in its box next to the PowerBook G4 12” from 2004 we’ll use as a client. Right now, this machine is running Debian forky/sid, one of the only Linux distros out there that still works out of the box on 32-bit PowerPC. As of 2026-11-09, it reports: With the specs out of the way, let’s go back to the premise of the article: showing you what a massive and unsurprising difference (remote) caching can make during compilations, and how, because RE is “just a protocol”, we can leverage it even from old machines. For this, we’ll measure ’s own build from a clean slate, cached and uncached, because it’s a large enough C++ project that takes about an hour to build out of the box—not too little, but also not so much as to be discouraging. Here are the final numbers of the comparison: Let’s break them down. First build, slow My first attempt at using remote caching was to build with itself in the “default” configuration for and the toolchain installed by Debian. This meant GCC 16.2 and no custom flags to CMake. The clean, fully-cached build took about 8 minutes to finish. This showed a nice ~7x speedup compared to the clean unwrapped build, but somehow it felt like not enough. Why did we still spend 8 minutes to compile something when everything should be cached? I had two theories, but neither of them was valid: The machine doesn’t have a lot of RAM for C++ compiles. Was causing it to thrash, maybe because it buffers files in memory instead of transferring them in chunks? Looking at the code, the answer was no: implements chunked uploads and downloads. The machine’s Ethernet port is slow. Were network transfers slowing things down? Not really either: the output tree is only ~150MB after a build. (Mind you, with Bazel and remote execution, you’d typically use to avoid ferrying large intermediate object files, but that’s not an option with a compiler wrapper that knows nothing about the end-to-end build graph.)

0 views

My Monitor Was Cutting My Download Speed by 75%

Somehow I am one of those few very lucky people to have the opportunity of owning a 2.5 gig symmetrical line for my internet. But one day, when I wanted to brag to my equally nerdy friends, shock and horror: 600 Mbps down? Luckily my upload speed was still 2.4 Gbps, so I didn’t lose all my bragging rights, but that’s weird, right? If anything, you’d expect the opposite! My ISP is pretty cool. It’s run by this one guy (Hey Keith!), and I know I can always message them and they respond promptly, as they did this time. We tried to run through a few debugging steps, they checked on their end, and everything was normal. I tried my work laptop too, same speed.

0 views
Farid Zakaria 2 weeks ago

A clock made of branches

What is time inside a machine you want to replay? 🧘 When building Rewind , a deterministic VM, you come face-to-face with the idea of how to represent time very quickly. You need a way to measure time inside the guest so that on subsequent replays, the guest sees the same time and makes the same decisions. Time is the ultimate variable. You can’t simply record the wall-clock time of the host because time spent between two interception points depends on things outside the guest, like host load, CPU frequency, caches and other processes. Therefore the same guest work can take a different amount of time on the host. In dafuq is a deterministic VM? I explained how the clock inside Rewind VM moves 5 microseconds per step. More specifically, that clock, exit time , moves only when the guest exits such as from a . It treats everything the program does between two exits as instant . Therefore a program that is CPU bound such as doing nothing but math for a full second between two system calls, sees only 5µs go by. 🤦🏻‍♂️ Rewind has a second clock mode as well, counter time , that tries to account for the work. The clock advances one nanosecond for every conditional branch the guest retired. 1 Aside: New to Rewind? Start with a flaky build you only catch once and dafuq is a deterministic VM? Still confused about what I mean by “free” and “branches”? Let’s look at a simple example to distinguish between the two clocks. The counter time tries to account for the work whereas the exit time only advanced the clock on exits. 2 Why branches, though? We already understand that we cannot rely on the wall-clock time of the host. We need something to measure that is the same for every run of the same program so that we can equate it to time spent. What is the same on every run? The guest’s instruction stream! The CPU is a deterministic machine: the same input, the same registers, the same memory, and the same instructions produce the same next state. We can measure time by measuring progress through the instruction stream. 💡 Turns out that CPU manufacturers even ship a hardware counter for exactly that. Is it reliable? Let’s see. 🧐 Let’s use this simple program: count down from a billion. The assembly equivalent contains a , a billion + (conditional branch) pairs, then three instructions to exit for a total of 2,000,000,004 instructions. Let’s run it on the CPU twice and count the instructions and conditional branches with : The same program and input gives the same conditional branch count, but the instruction count is off by a thousand or so. Why? Although this loop is CPU bound and only decrements a register, it is still not isolated from the effects of the outside world. Timers tick, network packets arrive, other cores send inter-processor interrupts (IPIs). The program can’t see any of that, but the instruction counter we are probing can. I pinned the loop to a single CPU 5 and counted how many interrupts that CPU took during each run, from . The dashed line is one extra count per interrupt , and the instruction counter is almost perfectly on it. 3 The conditional branch counter on the other hand, stays at zero no matter how many interrupts land. Interrupts from the outside-world are not the only thing that can skew the instruction count. The program can also page fault, which is an interrupt too. The instruction counter sees the kernel’s detour to handle the fault, but the program does not. Tip Out of the box, AMD Zen also overcounts conditional branches, a few in tens of millions, from a speculation optimization known as SpecLockMap. The fix is to set a particular bit in the PMU MSR, which Rewind does automatically as does rr#2034 . In order to rewind history, we need a measurement of time that is reliable and deterministic. Our traditional wall-clock time is unreliable due to the effects of the outside world but we do have a reliable measurement of progress through the instruction stream: the conditional branch counter. Using the conditional branch counter as a proxy for time turns out to be a relatively decent approximation. Why do we care about having a “realistic” representation of time? Plenty of programs read the clock and make decisions with it, so the guest’s clock has to be at least somewhat realistic. Under exit time, computing is “free”. A test that checks some work “finishes within 1 s” always passes. If the bug you’re chasing is a timeout, exit time will never reproduce it. 🙃 Races get skewed too with an unrealistic clock. With exit time, a whole burst of work lands in the same instant, which crowds the threads together, so a race can fire far more often in the VM than it ever would on real hardware. Some say time is an illusion. I say it’s a counter. This is the same approach popularized by the rr debugger. The rr paper calls it deterministic time .  ↩ The 5µs was my guess at a reasonable cost of a system call and context switch on current hardware. The 1ns is roughly what a conditional branch costs on a current core.  ↩ The fit is 0.94 rather than 1 because also catches the interrupts that land while and start up and tear down, before and after the loop runs.  ↩ exit time +5 µs +0 +5 µs +0, free! +5 µs +0 +5 µs 20 µs counter time +5 µs +2 µs +5 µs +1 s +5 µs +3 µs +5 µs 1.000025 s exit time: only exits move the clock. Work between two exits is instant. counter time: exits, plus 1 ns for every conditional branch of work. The counter time tries to account for the work whereas the exit time only advanced the clock on exits. 2 Why branches, though? What is time in a replayed machine? We already understand that we cannot rely on the wall-clock time of the host. We need something to measure that is the same for every run of the same program so that we can equate it to time spent. What is the same on every run? The guest’s instruction stream! The CPU is a deterministic machine: the same input, the same registers, the same memory, and the same instructions produce the same next state. We can measure time by measuring progress through the instruction stream. 💡 Turns out that CPU manufacturers even ship a hardware counter for exactly that. Is it reliable? Let’s see. 🧐 Let’s use this simple program: count down from a billion. The assembly equivalent contains a , a billion + (conditional branch) pairs, then three instructions to exit for a total of 2,000,000,004 instructions. Let’s run it on the CPU twice and count the instructions and conditional branches with : The same program and input gives the same conditional branch count, but the instruction count is off by a thousand or so. Why? Although this loop is CPU bound and only decrements a register, it is still not isolated from the effects of the outside world. Timers tick, network packets arrive, other cores send inter-processor interrupts (IPIs). The program can’t see any of that, but the instruction counter we are probing can. I pinned the loop to a single CPU 5 and counted how many interrupts that CPU took during each run, from . The dashed line is one extra count per interrupt , and the instruction counter is almost perfectly on it. 3 The conditional branch counter on the other hand, stays at zero no matter how many interrupts land. Interrupts from the outside-world are not the only thing that can skew the instruction count. The program can also page fault, which is an interrupt too. The instruction counter sees the kernel’s detour to handle the fault, but the program does not. Tip Out of the box, AMD Zen also overcounts conditional branches, a few in tens of millions, from a speculation optimization known as SpecLockMap. The fix is to set a particular bit in the PMU MSR, which Rewind does automatically as does rr#2034 . So, what time is it? In order to rewind history, we need a measurement of time that is reliable and deterministic. Our traditional wall-clock time is unreliable due to the effects of the outside world but we do have a reliable measurement of progress through the instruction stream: the conditional branch counter. Using the conditional branch counter as a proxy for time turns out to be a relatively decent approximation. Clock Time inside the VM Time on the host Exit time 2.1 s 15.7 s Counter time 11.7 s 14.2 s Why do we care about having a “realistic” representation of time? Plenty of programs read the clock and make decisions with it, so the guest’s clock has to be at least somewhat realistic. Under exit time, computing is “free”. A test that checks some work “finishes within 1 s” always passes. If the bug you’re chasing is a timeout, exit time will never reproduce it. 🙃 Races get skewed too with an unrealistic clock. With exit time, a whole burst of work lands in the same instant, which crowds the threads together, so a race can fire far more often in the VM than it ever would on real hardware. Some say time is an illusion. I say it’s a counter. This is the same approach popularized by the rr debugger. The rr paper calls it deterministic time .  ↩ The 5µs was my guess at a reasonable cost of a system call and context switch on current hardware. The 1ns is roughly what a conditional branch costs on a current core.  ↩ The fit is 0.94 rather than 1 because also catches the interrupts that land while and start up and tear down, before and after the loop runs.  ↩

0 views
Martin Alderson 2 weeks ago

Making agents fast - what I've learnt over the last 12 months

I've spent a lot of my career optimising websites and apps to run faster, and it's been interesting to apply some of those principles to agentic workflows. I'll run over some of my key learnings - there's many more I want to get written down at some point, but for brevity I think this is my rough mental model of tackling this. The obvious thing to start with is inference speed. This has two key components, which will be familiar if you have done any backend API optimisation. The first is TTFT (time to first token). This is analogous to TTFB (time to first byte) in web servers, namely how quickly does the first token come after you've done a request. This includes prefill (input token processing), so this scales with how many tokens the agent is sending to the inference provider too - so you should expect to see (potentially significant) variance on this number, much like you'd expect TTFB to be higher on an endpoint that e.g. has to do more database calls to prepare the response. A lower TTFT is better. Prompt caching is, of course, absolutely vital to this. Cached tokens are much faster (and cheaper!) to serve. Most agent harnesses handle this well these days - it's been a while since I've ever seen issues with this. Obviously worth keeping an eye on, but it matters less than I predicted in real world experience. The second is tok/s (tokens per second), which measures how quickly tokens come back once the response starts. This is closer to bandwidth on application servers, where a higher value is better. How relatively important each are depends on what kind of responses you're expecting back from the LLM provider. For example, if you are sending back and forth lots of short responses between turns in your agentic workflow, TTFT is likely to comparatively be of greater importance than tok/s. Equally, if you are having the LLM output longer responses (say, code or documents), then TTFT is likely to be less important than tok/s. Illustrative numbers. TTFT is network, queueing and prefill - the wait before anything comes back. Tok/s is how fast it streams once it starts. Knowing which number you need faster is important when comparing models and inference providers. Smaller models are generally faster than big ones. But between providers of the same model [1] , there is an enormous delta in speed. OpenRouter does provide public data on inference providers, but keep in mind this is an aggregate that may only roughly match your real world use. The main issue that tends to crop up is that providers also tend to vary in speed a lot depending on the time of day and how much load they are under. Big leaps in TTFT (for similar length prompts) can indicate queuing on the provider side, but can also mean your requests are being allocated to slower hardware or more users competing for prefill compute. Equally, slowdowns in tok/s can mean slower underlying hardware or higher batch sizes at the provider. Either way, there is a serious "noisy neighbour" [2] factor on many inference providers. Of course, many providers offer different tiers of service. For example, OpenAI offers a 'flex' tier alongside their standard and 'fast' tiers. These allow you to trade speed for cost. But don't overlook the impact of thinking budgets. A "fast" tier model with thinking set to maximum or xhigh will be vastly slower than a "slow" tier model with low or medium thinking. I'm guilty of constantly overlooking thinking budgets when looking at this stuff, but it makes an enormous difference. It's easy to be drawn to the quantitative tok/s and TTFT values when in reality reducing thinking can completely dwarf it - if suitable for your task. While having fast(er) inference never hurts, it's worth noting that it's often not the biggest part of the speeding up agents, and can be expensive to speed up (paying for faster tiers), or painful to move inference providers (if you have committed spend or contractual requirements). Rough numbers again, but halving inference time only takes 20% off the task when provisioning, tools and suspend/resume stay the same. It's good to start looking at your tool calls in more detail. This is much more classic performance optimisation work. But I've noticed things being slow that are not usually slow. A classic example is an agent running over large amounts of files, which causes hundreds of thousands of new process invocations. This can be painfully slow, especially on sandboxed environments. So while classic software performance engineering applies as per normal to agent tool calls, you need to investigate tool call traces to see what the agent does. It will often chain a bunch of bash commands together to achieve a result and it can be not obvious to see which part is causing slowness. is helpful here, but not ideal to run all the time as it itself has a performance overhead. Tool output size matters too. While most agent harnesses are pretty strict at stopping -style commands from blowing up your context window, as mentioned before uncached input tokens directly impact TTFT. SQL-style data sources have a big advantage if you can give your agent access to these - as it can request exactly what it needs without risk of a broad returning thousands of tokens of unrequired data. Equally, given many agents start a new shell on every command things in your that are not usually an issue can become a real issue. Be careful about what a shell needs to launch - especially if you have nvm or similar in that. Before you know it you can be spending dramatically more time starting the shell than running the command. Building on from the obvious "make this command faster", I think the next step is to think hard about what tooling your agent needs, but is having to "work around". For example, if you have it doing a lot of over markdown files over and over, it is potentially better for that data to be in a database it can call with full text search. This is where things differ a little in my mental model of software optimisation. Usually you look at a request, look at what it's doing, and try and make each part faster, and rinse and repeat. With agentic workflows, you can often just observe what the agent does, and deduce what would work better. So purely looking at tool call traces can be misleading. I've found that open weights models have a massive advantage here as you can read their thinking traces. While CoT traces are well used by the labs trying to work on safety and alignment, it turns out they are great for coming up with ideas to optimise the tools you give the agent. The way I've worked with this is getting a list of slow calls, and looking at the thinking blocks just before this. You'll usually get some blindingly obvious "hmm, could do x, but no y" style thoughts. And you can just give the agent what it wants, and see how things work out. The other thing that can make a big difference is treating loops with as a red flag. This is an indication that the agent wants to be notified of something but is instead having to poll for it. Often agents will poll for something that is quick but they tend to set a 10 or 20s minimum wait on the for loop. This can be solved with prompting sometimes, but often it can be an indication that (usually external) services need a async way to fire back task completion. Finally, parallelisation is a big help in cutting down speed of tool execution. Adding a simple "try and parallelise tool calls in $TASK" can work wonders, but especially with weaker models risks the agent making mistakes and undoing any efficiency gain. As such, providing tools (CLI or MCP) that do the parallel steps for the agent can be worthwhile. In my career, primarily driven by cloud computing platforms, I've often found the easiest quick fix is to just use faster hardware. And by faster hardware I do not mean a bigger AWS instance - dig into the underlying hardware your agents are running on. That means looking at the exact CPU core type you are using (the Zen 4c/5c cores can be much slower than the full sized ones), clock frequency and file IO speeds. This depends heavily on the tasks you're doing with your agents, but if they need to write a lot of custom code or doing a lot of data analysis, making sure the underlying hardware is fast can help a lot. For example, if your agent needs to do quite IO-heavy tasks, you may find a lot of your slowness is coming from networked disks. Having faster (local) NVMe disks may offer many, many orders of magnitude faster IO performance. This is often worse in many sandboxes. With things like copy on write and other snapshotting techniques, there can be a lot of overhead on file IO. Slow underlying IO just makes everything worse. I often see real speed issues with suspend and resume. Not every agentic workflow uses it, but if you (for example) allow agents to pause for a human in the loop approval, you need to be careful with optimising suspend and resume. MicroVMs can often resume on paper extremely quickly, but actually have pages streamed in on first access. So you need to measure not just what the MicroVM says but how quickly new inference requests are starting after resume. Being careful with memory management is the crucial thing here. While you don't want to starve your VMs of RAM and thus cause swap or OOMs, it's worth investigating what is using RAM during a suspend. If you have for example headless browser sessions active, it's almost certainly worth killing them before suspending, as is any running server processes the agent may have built. As always, the key thing is to measure and track RAM usage and break it down over time if you are using a lot of suspend/resume cycles. Finally, check the network. It's very common in my experience to have broken IPv6 in sandbox environments, and you end up with network timeouts while it does a AAAA lookup on DNS, which adds a small overhead to every request. Basic hygiene stuff like this adds up over the course of an agentic session. I think the key things I'd check on any agentic optimisation project are: As always, this space is also evolving fast. Keeping up to date with the field has always been my guiding north star in making sure you can keep stuff fast. I'd love to hear from you if you're working on making agents faster. I'm especially interested if you think I'm missing obvious factors here! Be aware that even the same model may have different quantisation (compression) depending on the provider. You need to compare apples to apples - if one provider serves FP8 and another INT4 (say), it will be faster but at the cost of quality. ↩︎ Noisy neighbour refers to shared compute getting slower when other users of the same hardware are pulling more resources. Typically you'd see this on VM providers with shared CPU resource - it may be fast a lot of the time, but when other customers are running high-load tasks, the hypervisor starts throttling each user. Unlike on VMs, where you can usually see this with 's column, it's very opaque at what load your inference provider is under. ↩︎ Inference speed is nuanced, and knowing what part of inference you rely on most for speed is important. Check thinking budgets! This is overwhelmingly the lowest hanging fruit Read traces, and if you are using open weight models read the CoT thinking traces. They often have so much valuable insight into what the agent is wanting to do Give the agent the tools it wants rather than what you think it needs Don't overlook the basic stuff. Faster hardware still matters, sandboxing and microVMs open a whole world of potential performance footguns, parallelisation helps and classic software performance optimisation is as relevant as ever. Be aware that even the same model may have different quantisation (compression) depending on the provider. You need to compare apples to apples - if one provider serves FP8 and another INT4 (say), it will be faster but at the cost of quality. ↩︎ Noisy neighbour refers to shared compute getting slower when other users of the same hardware are pulling more resources. Typically you'd see this on VM providers with shared CPU resource - it may be fast a lot of the time, but when other customers are running high-load tasks, the hypervisor starts throttling each user. Unlike on VMs, where you can usually see this with 's column, it's very opaque at what load your inference provider is under. ↩︎

0 views
Blargh 2 weeks ago

Fixing SCHED_IDLE for real

I’ve wondered for years why tasks are allowed to preempt other tasks. Nobody seems to know, and I’ve not had the time to dig in and understand the Linux kernel scheduler better. But now I finally have my answer. I want to be able to run a CPU-heavy task in the background, without it stealing any CPU, any at all , from other processes. If you read various documentation, you’d be forgiven if this is as simple as . (and similar statements, e.g. from sched.7 ) But if you try that, you’ll find that it’ll actually preempt other processes, taking little but nonzero CPU. There’s a test program , if you want to try it out. I literally want it to take zero CPU. And not just against , but everything. There appear to be two places that minmax guarantees can show up. Settings with default values: If is enabled in the kernel, then this setting is the maximum limit how much CPU the real-time (RT) classes ( and ) can take, if there are other runnable processes. If is not enabled, then it’s a more hazy tunable, with no preemption guarantees. You can set to , to remove this limit. (though again, it’s not actually a limit unless is enabled) Actually, technically removes the limit, and just sets it to allow taking all of it. I’m not sure what the difference between unconstrained and “constrained to 100%” is. But that’s not the only limit, so even with , the problem is not fixed. So that’s the maximum that a RT process can claim during contention. But it’s not the minimum that other process can take. The non-RT processes are scheduled by the “fair scheduler”, and it gives a minimum of 50ms (50 million nanoseconds) per second that deadlined processes get to make progress in (I think I got the terminology right on that): If you set to , then realtime processes will never be interrupted by lower priority tasks. Note that if your kernel is in lockdown mode (from a SecureBoot), you can neither read nor write this setting. But doing this can make things get wonky. My wifi broke while I was testing this until I killed the test program. Presumably from my test preventing even interrupt handling from completing. So if running realtime classes without limits, you may want to ensure those CPUs have no IRQs assigned to them. Maybe changing in the kernel would fix that aspect, but I’ve not tried it. But this is just for realtime processes. This doesn’t prevent from preempting the default scheduling class, . And this is the main thing I want to do. The scheduler weighting for is not 0. It’s 3 . The default weight is 1024 . This means the supposedly IDLE tasks will always make progress. Another word for “make progress” is “selfishly steal CPU resources that could go to higher priority things”. This is a compile time constant, and you cannot change it at runtime. It’s not even as simple as changing the constant and recompiling, because the math doesn’t work (think divide by zero). You can also change weights between cgroups, but again not to zero: If my goal is not to let the realtime classes dominate, but to prevent from taking even a pinky toe, I’ve actually not solved anything at all. I’m not a kernel programmer. But I know a robot who is. After having it confirm my understanding of the various knobs mentioned, I just said `Make a patched kernel that allows 0 weighting for SCHED_IDLE against SCHED_OTHER. It went brrr for a while, and even wrote scripts to spin up a VM in qemu to test with, and finally produced a patch that adds a scheduler feature that ensures only gets scheduled if there’s nothing non- running. While I have done some minimal kernel coding manually, I’m not competent to evaluate the scheduler changes. But it does work. When you enable the feature, the thread instantly stops being scheduled. This is a patch on top of kernel 6.12.111, with scripts and stuff in this repo . A patched kernel tree in this branch , and the exact patch is this file . Well, in terms of understanding why it was broken, yes. But ideally this scheduler feature should be upstreamed. I need to understand it fully before I’d ask someone else to take a look. But if anybody reading this wants to take that on, go for it! Maybe one day our processes will finally live up to their promise. Kernel docs on realtime scheduler config differences Fixing SCHED_IDLE (it doesn’t)

0 views

TwinString: Preserving String Semantics with Off-Heap Data on the JVM

TwinString: Preserving String Semantics with Off-Heap Data on the JVM Júnior Löff, Daniele Bonetta, and Walter Binder OOPSLA'26 This paper presents an alternative representation of Java strings which significantly reduces JVM/GC overheads, at the cost of potential memory fragmentation. The target applications are ones that process short-lived strings from external sources (e.g., text files). A TwinString is an implementation of . The key optimization is that a TwinString does not contain a byte array, but rather contains a pointer to an immutable set of bytes in a TwinHeap. Fig. 4 illustrates the differences associated with initializing a regular String and a TwinString from bytes contained in a MemorySegment (e.g., bytes loaded from a file): Source: https://dl.acm.org/doi/10.1145/3839503 The standard process illustrated in the top of Fig. 4 involves allocating a byte array, copying string data into it, and then constructing a string wrapper. The TwinString approach illustrated at the bottom of Fig. 4 involves allocating space in the TwinHeap, copying bytes into it, and then constructing a string which contains a pointer to bytes in the TwinHeap. A TwinString contains metadata like the string length and string hash. This enables some methods to execute without having to touch the TwinHeap. TwinHeap is designed to be fast for short-lived strings. A TwinHeap comprises a set of 64KiB pages. Allocation is as simple as incrementing an offset into a thread-local page. Deallocation occurs on a page-by-page basis. During normal JVM garbage collection, a live TwinString causes the associated TwinHeap to stay alive. This means that garbage collection is fast, but fragmentation is a potential issue. A single live string can cause the entire page to not be collected. The design of TwinHeap is simplified by the fact that Java strings are immutable, so there is no need to be able to edit/resize strings after they have been created. Fig. 8 shows that TwinStrings can be initialized much faster than standard strings: Source: https://dl.acm.org/doi/10.1145/3839503 Fig. 9 shows that operations on TwinStrings are about the same speed as regular strings. The “String & TwinHeap” bars are there to try to separate out the costs of TwinString from TwinHeap. Source: https://dl.acm.org/doi/10.1145/3839503 Dangling Pointers The notion of coarse-grained garbage collection seems more generally useful than just for strings, I wonder if it could be extended to other types of objects. Thanks for reading Dangling Pointers! Subscribe for free to receive new posts.

0 views
Den Odell 3 weeks ago

Design Systems Should Be Fast by Default

If a design system’s components are slow to interact with, so are any apps built with them. I timed large dropdowns and checkbox lists in some popular design systems, worked out how to make the slow ones faster, and wrote tests that fail when they’re too slow. Web apps are usually architected and built in layers. The browser is the foundation, often with a framework like React or Lit above that, followed by the company’s shared libraries and design system, and finally the product team’s own code is at the very top. The browser has one main thread for running JavaScript, responding to user events and drawing the screen , so the less work the lower layers do, the more of it is left for the app. If a design system’s equivalent of a dropdown takes a second to open and display its options, it’s not going to be any faster in the apps that use it, however well their teams write the rest of their code. If the design system team takes the time to make that faster, it unlocks better performance for all the apps built with the design system at once, and for everyone using them. I know that design system teams already think this way about accessibility. Using a well-built design system’s button component gives you the correct keyboard support and ARIA roles out of the box, and many design system teams rightfully check their components for accessibility thoroughly before each release. For some teams, though, performance doesn’t always get the same attention. Components usually get tested in isolation with just a few items, nothing intense enough for the cracks to show. They start to slow down once a real app gives them much more to do. The team building that app can feel the page is slow, but because that comes from the design system, they can’t easily fix it themselves. I feel that performance belongs on the design system team’s priority list, next to accessibility. That means testing each component with demanding but realistic amounts of data before it’s released, and making sure its performance doesn’t get quietly worse over time. “Performance belongs on the design system team’s priority list, next to accessibility.” I’m eternally curious, so I took it upon myself to conduct a bit of testing. I wanted to see how some of the most popular open-source design systems fared on performance. I deliberately chose ones built on different frameworks, as I was interested in whether certain frameworks would turn out to affect performance more than others. Here are the ten I picked, with the versions I tested in October 2026: I left out some popular design systems that weren’t actively being developed any more, for a fairer comparison. Also, React Spectrum and Spectrum Web Components obviously share the same design language, but they’re separate libraries with their own code so I wanted to see whether they’d perform the same. The first thing I did was to look through each of their codebases to see if they had any automated performance tests. Only Spectrum Web Components seemed to. It runs a benchmark for each of its components every time a pull request is created. Fluent also has a benchmark for each of its components, but I couldn’t see them running automatically anywhere. The others didn’t appear to test for performance on a component level at all. Ant Design, Material UI and Fluent do check their bundle size. That’s usually what design system teams measure when they consider performance, and it’s definitely important as it affects page load time, but it doesn’t relate to how quickly a component actually responds on the page. I then picked two fairly intensive tasks that real web apps do often and I timed each design system on those. The first was opening a country selector with all 249 countries from the ISO list , like the ones you get on checkout and sign-up forms. The second was selecting 100 checkboxes at once, just like clicking a “select all” checkbox above a list of 100 messages in a webmail app. I realize these two tasks aren’t an extensive review of performance across an entire design system, but they’re where components often get slow, so they’re a good place to start. Before we get to the results, I want to make clear that all of the design systems I tested are well built and are widely used, and they’re written by expert engineers who I respect. The point of this research isn’t to rank them or offer criticism, but to use them as examples to encourage us all to prioritize performance in our design systems. And for any problems I uncover, I’ll suggest a fix, the way I would in any manual code review. I ran these tests with production builds of each design system, using Chrome v155 on my M1 Pro MacBook Pro. A script clicked the country selector or checked all the checkboxes, then measured how long the browser’s main thread stayed busy afterwards, including any animation, and that’s the time shown in the tables. The script slowed the CPU down by 4x, the same as the throttling setting in the Performance panel of Chrome’s developer tools, which I always recommend when developing, because it’s closer to how real users’ devices perform. To avoid any spurious results, I ran each test three times and took the median values. I also measured the longest time the UI stopped responding during each task. That happens when the main thread is blocked, because the browser can’t update the screen or respond to user events, so the page appears to have locked up. Chrome only tells you about frames that take longer than 50ms to draw, so I’ve put “Under 50ms” in the tables wherever it didn’t report any. I’ve put the tests into a design-system-performance-tests repo on GitHub, by the way, in case you want to run them yourself. On to the results, and here’s how long each design system took to open the country selector, from fastest to slowest: The country selectors in Fluent and Carbon opened faster than any of the others. Ionic’s was by far the slowest, taking more than five times as long to open as the next slowest, React Spectrum. To see how the time to open changes as the list gets longer, I opened the country selectors in Ionic and Fluent again with just 50 options. Ionic’s took 203ms with 50 options, but over 1.2 seconds with all 249, and for most of that time the UI stopped responding. Fluent’s barely changed, going from 41ms to 61ms. Now, here’s how long each design system took to select all 100 checkboxes: Ant Design took six times as long as Material UI, even though both are built with React. Selecting the checkboxes didn’t stop the UI responding for any noticeable length of time in any of the design systems, which is encouraging. The longest it stopped responding for was 78ms, which is about five frames the browser skipped drawing at 60fps, far shorter than the slowest country selectors. Both sets of results showed that a design system’s performance didn’t depend on its framework by itself. Carbon, Web Awesome and Spectrum Web Components are all built with Lit, but the country selector in Spectrum Web Components took more than twice as long to open as the one in Carbon. And Ionic had the slowest country selector of all those I tested, but its checkboxes were faster than most of the others, so the same design system can be slow at one task and fairly quick at the other. I didn’t want to leave my testing there without trying to find out the reason for the differences between the results, so I did some profiling on the slowest ones. With the CPU still throttled by 4x, I opened the Performance panel, pressed its Record button, opened the country selector or selected all the checkboxes, and then stopped the recording. The Summary tab at the bottom of the panel shows how the main thread’s time gets split between Scripting, which is JavaScript execution, Rendering, which covers calculating styles and layout, and Painting, which is preparing what to draw on the screen. The “Bottom-Up” tab lists the JavaScript functions that ran, sorted by how long they took, and you can click on a function’s file name to open its source code in the Sources panel and see what it was executing. Going back to Ionic’s results from earlier, its country selector took more than six times as long to open with five times as many options. If each option took the same amount of time to add, it would take about five times as long, or a little less in practice, since showing the dropdown itself takes the same amount of time no matter how many options it has. That means each new option must take longer to add than the last one before it. Figure 1: The DevTools Performance panel after recording Ionic’s country selector opening with 249 options, with the Bottom-Up tab open and the slowest functions at the top. In the Bottom-Up tab for Ionic’s country selector, shown in Figure 1, most of the top rows were functions with minified names, all from the same file. That file also had its own version of , a method the browser normally provides, and running that version took up about half of the time the country selector needed to open. Clicking into those functions, I could see the file contained Stencil’s framework runtime. The button at the bottom of the Sources panel formats minified code so it’s easier to read, and when I compared it with Stencil’s source code on GitHub, I could see that these functions were part of Stencil’s handling of slots. A slot is a placeholder inside a web component that marks where the elements you put inside it are shown, such as the options inside a dropdown. Other design systems built with Stencil might have the same problem. In Ionic’s country selector, each option is a radio button, and they’re all inside a radio group, , which makes sure only one of them can be chosen at a time. Slots are normally handled by the browser, through shadow DOM, which is the feature that gives a web component its own private set of elements, separate from the rest of the page. doesn’t use shadow DOM, though, so Stencil handles the slots instead. It replaces the radio group’s and methods with its own versions that work out where each new option should go. To do that, it checks all the options already in the radio group to find the right slot, then checks them again to work out where the new one should go. Once the new option has been added, Stencil goes through all the options a third time to find out whether the slot’s fallback content should be shown. Fallback content is what a slot displays when it’s empty. Adding the 249th option means checking the 248 options before it three times, while adding the 10th only means checking the 9 before it, which is why each new option takes longer to add than the last one. The setting that replaces is called , and Ionic enables it through another setting, , which turns on all of Stencil’s fixes for slots in components that don’t use shadow DOM, including that one. So the delay comes from that setting together with a radio group that doesn’t use shadow DOM, rather than from Stencil being slow in general. Both settings are by default today, but Stencil’s documentation say they’re meant to become by default in the upcoming v5 release. If that ships as it is, any component that uses slots without shadow DOM will slow down the same way Ionic’s dropdown does, because Stencil will check all of its existing children each time a new one is added. Ionic could fix its dropdown by making use shadow DOM, so the browser handles the slots instead of Stencil, and the browser’s normal function would then be used. That fixes Ionic’s dropdown for all the apps that use it, though it would change how apps can style the radio group, so it would probably need to wait for a major release to do so. Or Stencil could fix the problem by only checking the fallback content of the slot that changed, and skipping that check when the slot already had options in it, because its fallback content would already be hidden. To find out how much difference that would make, I removed the third check from a copy of Stencil and added 2,000 elements, one at a time, to a simple test component. Adding them was about seven times faster without the check. Fixing it directly in Stencil would also fix the same problem in any Stencil-based design system that uses slots without shadow DOM. I’ve raised this with both the Ionic team and the Stencil team . I ran in the Console before and after opening Material UI’s country selector, and the count went up by about 250 elements, one for each option, as soon as its dropdown opened. I ran the same check on Ant Design’s country selector, and the count only went up by about 40. Ant Design’s country selector renders just the options that fit in its dropdown, and renders more as you scroll, which is a form of virtualization. That’s why it opened noticeably faster than the one in Material UI. Virtualization works for long menus and tables too, but it needs care with keyboard navigation and screen readers, because only the options or rows that fit on screen are actually in the DOM. Ant Design’s dropdown is a good example to learn from here. As well as the options you can see, it keeps a hidden copy of the highlighted option in the DOM, so screen readers can still announce it. In the Summary tab, I noticed that Rendering made up about half of the time UI5’s country selector took to open, and Scripting made up much less. So the delay was mostly coming from the browser working out styles and layout, not from executing UI5’s JavaScript functions. In the Elements panel, all 249 options are already in the DOM before the dropdown opens, inside a hidden popover. The browser doesn’t work out styles for elements that are hidden, so when the popover opens, the browser has to style all of them at once. That’s about 1,500 elements, because each of the 249 options has its own shadow DOM with more elements inside. Virtualizing the list, the way Ant Design does, would give the browser far fewer elements to style. React Spectrum’s country selector uses virtualization too, but it was still the second slowest to open, after Ionic’s. I opened it with half as many options, and it took about the same amount of time, so the length of the list wasn’t what made it slow. I profiled it, and near the top of the Bottom-Up tab was a browser function, , called from React Aria, the library React Spectrum is built on. In React Aria’s source code , I found that every time the country selector opens, React Aria goes through the visible options one by one. For each one, it checks that the option is visible, clears its height, and then reads its size, and the browser has to work out the layout of the page again for each of those checks. This is known as layout thrashing. All of that happens straight after the user clicks the country selector, and the UI stops responding until it’s finished. React Aria could clear the heights of all the options first, and then check and measure them all, so the browser only has to work out the layout once instead of again for every option. I’ve raised an issue with the React Aria team about this. I profiled Ant Design’s checkboxes as all 100 were selected, and executing JavaScript made up less than a third of the time it took to select them all. The rest was the browser calculating styles and drawing each checkbox’s animation, 100 times over. I turned the checkboxes’ animations off on the test page, with a CSS rule that sets their transitions and animations to , and selecting all the checkboxes then took less than a third as long. Almost all of the style and drawing work went away, and so did some of the JavaScript, which was tied to the animations. The animation helps when someone checks one box, but users don’t need to watch 100 checkboxes animate at the same time. Ant Design could fix this by only animating a checkbox when someone selects it directly, or the code behind a “select all” checkbox could turn the animations off before checking all the other checkboxes. “Users don’t need to watch 100 checkboxes animate at the same time.” None of this means a design system has to give up its animations, because they can be changed to be less demanding for the main thread. The Elements panel shows that Ant Design’s checkbox styles fade the checkbox’s background and border colors, which the browser has to repaint on the main thread for every frame. Fading the opacity of a filled layer gives the same effect, and the browser can run that on the compositor thread instead. That’s a separate thread that puts together the parts of the page the main thread has already drawn, so it can fade them by itself and keep the animation running even while the main thread is busy. I found these problems by profiling, but each one could have made a performance test fail before the component was released. Most component tests render one copy of the component with a few props, which shows it works, but not how fast it is with 249 options in a country selector or 100 checkboxes being checked at once. In a performance test, you set the component up the way a real app would use it, and then time how long the page takes to respond when you interact with it. Listing 1, below, introduces Playwright performance tests for the country selector, using a fake design system I made up called Nettle. The repo runs the same tests on each of the ten design systems I tested, along with a test that checks all 100 checkboxes at once. The test page renders the country selector with the number of countries set in its URL, such as . In each test, Playwright clicks the country selector to open it and measures how long the browser’s main thread stays busy afterwards, along with the longest time the UI stopped responding. Listing 1: Performance tests for Nettle’s country selector. The function in Listing 1 returns Chrome’s metric, which is a running total of how long the main thread has been busy. If you subtract the total before the click from the total after it, you get the time for everything that happened as a result, including JavaScript execution, rendering and painting to the screen. I ran both tests with the CPU slowed down four times, the same as for the tables above. In my testing, Ionic’s country selector was the only one to go over the first test’s 300ms limit with all 249 options. One long pause that locks up the UI feels worse to a user than several short ones, even when they add up to the same total time. The country selectors in Spectrum Web Components and UI5, for example, took almost the same total time to open, but the one in Spectrum Web Components opened in several smaller steps, so the page only locked up briefly each time. UI5’s one had to style all of its options at once, so the page locked up for longer. That’s why the first test also uses the Long Animation Frames (LoAF) API , which reports any frame that takes over 50ms to draw, so the test can fail if one of them takes 100ms or more. With all 249 options, the country selectors in Ionic, Material UI, React Spectrum and UI5 failed that check, and Ionic’s failed it even with just 50 options. Each limit in these tests is a performance budget , which is the most time something is allowed to take before the test fails. I picked 100ms as the budget for how long the UI can stop responding, because that’s roughly where a response stops feeling instantaneous. In the first test, I gave the total time a higher budget of 300ms, because the test waits for the dropdown to finish animating open, so the total also includes drawing each frame of that animation. You can set your own budgets by running the tests on your components now and setting each one a little above the result, so the tests fail if a later change makes a component slower. The second test in Listing 1 opens the country selector with 250 options, rounded up from 249 so it halves evenly, and then with half as many, the same kind of comparison I made with Ionic’s results earlier. The full list should take no more than about twice as long to open as the half list, so the test fails if it takes much longer than that, with a little room to allow for the times varying from run to run. Every country selector except Ionic’s passed easily, taking well under twice as long with the full list, while Ionic’s took nearly three times as long and failed. If I could only choose one test to run, it would be this one, because it fails when each option takes longer to add than the last one, and you don’t need to change its budget for machines with different specs. After I’d written these tests, I found that Spectrum Web Components already has an automated test for a country selector just like mine, with all 249 countries. But that test only times how long the closed dropdown takes to render on the page, while this one times how long it takes to open when someone clicks it. The same ideas work for other components too. You can adapt the country selector tests for anything that shows a long list, like a menu or a table, and the checkbox test for anything that appears many times on a page, like icons or buttons. If you run these tests automatically on every pull request, with something like GitHub Actions, you’ll know straight away when a change affects a component’s performance. To try the tests on your own design system, you can copy one of the folders in the repo and change it to use your components. The README explains how, and the repo also has a GitHub Actions workflow ready to use. Repository: denodell/design-system-performance-tests — The country selector and checkbox tests from this post, with a folder for each of the design systems I tested. Each one times how long the main thread stays busy and how long the UI stops responding, with the CPU slowed down by 4x. Apart from testing, I think the most useful thing you can do is make the fast behavior the default. A dropdown can switch to virtualization once its list gets long enough to go over its performance budget, and a checkbox might only animate when someone selects it directly, as I suggested for Ant Design. App teams then get the fast version without having to know the problem ever existed. “Make the fast behavior the default.” You can also write in your design system’s documentation how many items each component has been tested with, such as “the dropdown is tested with up to 500 options, and above that an autocomplete is a better fit”. To work out how many options to specify there, you can keep running the tests with more options until the dropdown goes over its performance budget. Once that limit is documented, app teams know how far they can push a component before it gets slow, and you know how many items your tests need to cover. Pages in real apps that use lots of your components are the best indicator of what to fix next, because they show how your components are really used. If you ask app teams now and then to record a profile in the DevTools Performance panel while they use those pages, and send it to you, you can see which of your components might be impacting performance. In React’s development builds, those profiles include a Components track , which is a row in the Performance panel’s timeline showing how long each component took to render. Profiling builds only show it for the parts of the app wrapped in React’s component. Web components don’t get a track like that, but I’ll cover a solution to that in a follow-up post. Each time you release a performance fix, your release notes can give the before and after measurements, such as “the dropdown now opens in 150ms with 500 options, down from 300ms”. App teams can then see exactly what got faster and check the difference in their own apps, the same way a line like “screen readers now announce the selected option” tells them what’s improved for accessibility. A design system is a foundation for the apps built with it, just like the browser is. Browser makers measure performance before every release, and it’s worth doing the same with your design system. This is the idea behind Fast by Default : building performance in from the start, instead of fixing it after users complain. My new book, Fast by Default: Practical Performance Engineering , takes this approach further, for backend and mobile code as well as the web, and explains more about how to set performance budgets and check them on every pull request. It’s in early access from Manning now. Mantine 9.7.1, built with React Ant Design 6.6.5, built with React Material UI 9.5.0, built with React React Spectrum 3.47.6, built with React Carbon 2.65.0, web components built with Lit Fluent 3.1.3, web components built with FAST, Microsoft’s web component library Web Awesome 3.14.0, web components built with Lit UI5 2.27.2, web components built with SAP’s own in-house framework Spectrum Web Components 1.12.4, web components built with Lit Ionic 9.0.7, web components built with Stencil

0 views
Marc Brooker 3 weeks ago

Animations for System Builders: Metastability and Lost Writes

Pretty pictures help us learn, maybe? One of the things I love using LLMs for is helping me understand systems better. In the Strands Decider post I wrote about how I often ask ask me questions to check I understand this . The other technique I’m enjoying right now, and am delighted to see folks around me using too, is building animations. Animations can help us build an intuition for how and why systems work, or how and why they fail. Here’s one example: did you know that a Raft cluster can lose writes if even one of the cluster members has a storage layer that lies about ? When I built Stability Sim , I wanted to help engineers understand metastable failures better. That worked pretty well on an abstract level (e.g. retries and queues make metastability really easy to trigger ), but I still hear doubts that these behaviors happen in real systems. After all, those real systems are production-proven, and wouldn’t we have shaken out the behaviors by now if they existed? Let’s start with Postgres. In this example, captured from traces on real Postgres 16 running in a real MicroVM. Before a load spike, the database is easily able to handle its 20 queries-per-second load, thanks to the whole working set sitting in the page cache. But after the load spike, we’ve filled up a lot of the memory running queries (each takes 12MB), which now run slower because they’re served off the disk rather than the page cache. I like this example, because it shows that even simple, non-distributed systems have these behaviors. What about ? Again, we have load spike triggering the metastable state. Writes, in this case. The overloaded peers look like they’re unavailable, which triggers a storm of application-level resends. I like this example, because is widely-used quality software. Operational experience doesn’t seem to effectively shake out these kinds of issues. When I wrote Garbage Collection and Metastability , a widely-known systems academic told me that it was a fantasy . I’ve seen it happen in the wild in ways I couldn’t share, and I can forgive people for not believing this can happen based on the hand waving in the blog. So it’s nice to have a real world example, picking on again. At the start, we’re ticking along nicely at 500 reads/second, before spiking load to 1000 reads/s. The read spike increases the time spent in GC, increasing average tenure, and making less CPU available for the read workload. This is a self-reinforcing loop, and the system doesn’t recover without intercession. I believe the same thing could happen on a runtime without GC, due to increasing work in the allocator, but haven’t been able to reproduce that yet. The animations here are all generated with Kiro and KiroCrew , mostly with Opus 5.5. The actual results came from a real test environment, which captures the traces that back the animations.

0 views
Unsung 3 weeks ago

“No results” doesn’t mean “no results,” pt. 1

This is something I see time and again in Google Maps when plotting directions: = 3x)" srcset="https://unsung.aresluna.org/_media/no-results-doesnt-mean-no-results-pt-1/1-framed.1600w.avif" type="image/avif"> You might be thinking “this feels like a genuine error – not every two places on earth can be connected,” but that’s not it. I am in a good traffic corridor and, ten seconds later, doing the same exact thing again shows proper directions: = 3x)" srcset="https://unsung.aresluna.org/_media/no-results-doesnt-mean-no-results-pt-1/2-framed.1600w.avif" type="image/avif"> The clue as to what’s going on is elsewhere in both screenshots. My cell signal is low, and the Google Maps misrepresents no results arriving because of slow connection, with no results being there at all . Just to get Google Maps off the hook, here’s a similar thing in a Mastodon client Ivory: = 3x)" srcset="https://unsung.aresluna.org/_media/no-results-doesnt-mean-no-results-pt-1/3-framed.1600w.avif" type="image/avif"> There are 123 posts to be seen here, as the page itself conveys elsewhere. But at this moment, the internet is slow, and Ivory jumps to a wrong “no posts found” conclusion, instead of saying “cannot retrieve the posts because of a bad connection.” I believe what’s happening in these moments is that one kind of “no results” (software asked for posts, but the database results came empty) gets bundled with another (software asked for posts, but the database results didn’t come back in time, or transmission was interrupted). This feels important to me, because the wrong kind of an error can really confuse people and make them jump to the wrong conclusions – I guess Mark Simonson isn’t active on social! I guess I can’t drive there! – or even create bad mental models. Also, while this was a temporary situation for me, for a lot of people bad connection is a constant fact of life, so they might encounter this kind of confusion on a daily basis. Software should differentiate between different ways something can be zero , and in this case, communicate that back to the user. What’s really strange is that Google Maps has a separate “no internet connection” state that you can see in other moments – incl. an extra thoughtful and important manual try again button… = 3x)" srcset="https://unsung.aresluna.org/_media/no-results-doesnt-mean-no-results-pt-1/4-framed.1600w.avif" type="image/avif"> …but apparently still fails to use it in all the cases.

0 views
matduggan.com 3 weeks ago

ClickHouse Will Let You Be Wrong at Scale

So I've posted before how I think ClickHouse is great. Its columnar storage is fast, it has amazing compression. I won't bore you with those details, you can see them all here: https://matduggan.com/clickhouse-is-winning-the-observability-wars/ However all technology is great until you try to do something atypical with it. What I wanted was a realistic torture test: network firewall telemetry, on the order of 2 billion rows a day of mixed-vendor syslog, parsed at ingest into wide typed tables with a 14-day TTL. The dashboards needed to do two things at once in Grafana: fast lookup of individual log lines, and fast visualizations where you can stack ad-hoc filters on top of each other across every panel simultaneously. I learned a lot of things about ClickHouse. Also some things about myself, not all flattering. This ended up being significantly harder than I thought it would be, and part of that is that I made a ton of mistakes. Don't email me mean comments. I'm writing this up as a warning to the people coming behind me, and I'll own every mistake as we go. For the happy OTel path, CH is very easy. Few hundred metrics, couple million logs a day, several thousand traces, not a problem. For the more complex custom route, CH is extremely powerful but it's hard to use. The theme of the whole project is that this stack fails silently. ClickHouse is the most agreeable coworker you'll ever have because it never says you are wrong, it just lets you be super wrong. A query that looks fine reads 84x more rows than it should. A misconfigured filter just doesn't apply. A projection that never routes still returns correct numbers, slowly. Once I understood that ClickHouse will never tell me that I did something wrong and that I had to figure out I did something wrong, we got along fine. You have to build verification habits before you build anything else. is your friend, your bartender, the lovable dog in a horror movie. This is the only thing in the system that says you messed up. Here are the lessons I learned. Hopefully they save someone else from going down the same path I did. So we're not doing anything weird here. My first draft of the dashboard wrote filters the way I'd typically write them in normal SQL, by adding a guard so the query stays valid regardless of what the user picks. This would end up being quite the stupid decision. Eighty-four times the rows, ten times the time, same answer. Why is this so much worse? I'm not entirely sure. My best guess is that the guard keeps primary-key pruning from engaging, so you get a full partition scan with post-filtering. I clearly measured the effect but I'm not exactly sure how to prove the mechanism . was my only savior here. The fix was to stop putting in the WHERE-clause filtering by hand and use Grafana ad-hoc filtering. The key learning here is how the plugin does that and why it makes a difference. It does not rewrite your WHERE clause. It appends the filter as a ClickHouse query setting, , which means it is applied at planning time and prunes exactly like a hand-written query. When I first set up the main table, I did the following: The logic made sense at the time, even if it didn't end up holding up. We're sorting by network device first since of all of these, there are going to be the least number of network devices. But what I didn't really think about is that every panel is time-bound because of course they are, that's how Grafana works. In ClickHouse only the front of the sort key gets the fast lookup path. Obvious in retrospect, but stumped me for a hot minute. I thought I was structuring this by lowest cardinality (device) and then out to most cardinality, which logically made sense. If all you want is information on the right way to do this, you can find that here: https://clickhouse.com/blog/working-with-time-series-data-and-functions-ClickHouse and https://oneuptime.com/blog/post/2026-03-31-clickhouse-choose-order-by-key/view What's important to understand about this index is that it's not the same as like MySQL necessarily. It's the physical layout, the rows are written to disk in sorted order like a phone book sorted by lastname, firstname. ClickHouse by design doesn't keep a very large "bookmark list", every 8192 rows (called a granule), it writes down the sort-key values at that spot. That bookmark list is the primary key. So the Fast lookup only works for a prefix of the sort columns: Everyone asked me why I did that, let me explain. The two entries are not the same thing. Within any fixed (observer, action, application) group, sorting by (hour, exact-time) produces exactly the same order as sorting by (exact-time) alone — the hour is just a rounded copy of the same value, so it can never re-order two rows. So my first design was basically this . So time sat at position four, and the most common predicate in the dashboard was aimed at the worst-positioned column. The tail-logs panel with ORDER BY Timestamp DESC LIMIT 1000, a multi-hour window, measured before the guardrails went up, read 291.8M rows, 158 GiB, in 18.9 seconds; with one observer filter, 23.1M rows, 936 ms. The newest rows sit at the end of each device’s block so a newest-N query has to walk every device's block to find the tail. Here's what makes this decision consequential: can't reorder, prepend, or absorb existing columns. You can only append columns added in the same . Get the key wrong on day one and the fix is not an ALTER; it's a new table, ~520 GiB of backfill per shard under live ingest, and an atomic swap. Ask me how I know. MODIFY ORDER BY turns out to be less 'modify' and more 'build a second house next door, carry everything over by hand, then swap the mailboxes. After all those lessons, I ended up running a benchmark that was the kind of beautiful baby benchmark you dream of where it confirms you solved every problem. An identical aggregate query, 156 ms vs 1,690 ms. 10.8x faster! I felt great about that for most of an afternoon, which is how long it took to notice that 88% of the old table's partition had tiered to S3 while the new table sat entirely on local NVMe. We had benchmarked a Ferrari against a tractor and concluded the tractor was slow. Which is true. It is also not the thing we measured. Excellent work. The same-tier control with both on NVMe produced no noticeable query differences. So was it worth reorganizing? I'm not sure! I think? This took a long time to do, so triple check this part. So when you first run across projections in ClickHouse, they seem pretty great. A projection is not a separate table you manage. It's an alternative physical ordering of the same data, stored inside the same table. Think of it this way: your table is a book sorted by chapter number. A projection is an index at the back, sorted by topic. When a query matches your projection, it should just automagically read from the projection instead of the base table. You don't need to change the query, which is great. New data populates the projection on INSERT, which is what you want for this sort of stuff. Think of it like this: Projections are great for network firewall data because the data doesn't need to change shape. I just need to optimize my querying of the data for specific usecases. The biggest misunderstanding about how projections worked is that I made a lot of assumptions about how expression matching worked. I assumed (for no good reason) that expression matching was loose and that CH would do a lot of work to ensure my query was routed to the projection. and won't be used by a query asking for , which didn't occur to me until I read this: https://obsessiondb.com/blog/clickhouse-projections-complete-guide Also every filter column must be in the projection's GROUP BY. This means a filter on an absent column → full fallback to raw. This was a fucking nightmare to debug until I figured this out. Here are the basic lessons you must understand to do a large scale project. All of this was figured out when I started just obsessively checking . In order to use projections in ClickHouse you have to verify the routing every single time. Here are my mistakes in no particular order with projections: I like Grafana, I love the ClickHouse Plugin group of maintainers, all good vibes over here. But I misunderstood how much this plugin was going to do for me vs what I had to do myself. This is the part of the review where I say nice things and then hand you a first-aid kit. You will not have any idea that the thing you wrote is wrong until you check it by hand or it blows up. My advice is to configure your dashboards to blow up early and often so you don't find out months later that you messed up the query. ClickHouse is a good tool, but the more complex the thing you try to do with it, the more sharp edges you'll find. I'm also surprised at the ecosystem lacking stuff I expected. I wanted a linter. I got a side quest to make my own. I had trouble getting a full-fledged real migration utility for rolling out changes with advice about impact and cost and then confirming I can always roll back. There are still real gaps in this ecosystem people should be aware of. As always if I got it wrong (or you know something I don't), email me. ClickHouse is a 2 shard x 3 replicas setup underneath the Altinity operator. Every table is a wrapped in tables and Grafana only ever queries the wrapper. Grafana is the official plugin, 25 panels and then ad-hoc filters up at the top that apply to every panel at once. Hard guardrails we cannot break. = 1 billion, = 100 GB, = 60s, all in mode. I can raise these but the entire point of this process was to operate inside of the CH model for data and try to leave those alone. "All the Smiths" → jump straight to the S section. Fast. "Everyone named John" → Johns are scattered through every last-name section → you read the whole book. Slow. Their table also splits data into daily partitions (separate files per day). That's a coarse filter that works no matter what the sort key is (remember this, it becomes important) — the hour bucket . It rounds 14:37:22 down to 14:00:00. This is the "cutoff" — the exact time. — puts the bucket first, so an hour range is one contiguous block. All the Smiths together. Every table stores data in parts, which are directories on disk. When you add a projection, each part directory gets a subdirectory for the projection data with an anonymous table with its own columns, index, granultes. Projection parts merge at the same time as main table parts, so there's no lag like what you can experience with materialized views. Projections don't backfill by default but you can backfill them, which is a mistake that I made initially where I added one, nothing was better and I assumed I had messed up. The query’s aggregates must be a subset of the projection’s — count() and sum(duration) won’t serve avg(duration). Every filter column must be in the projection’s GROUP BY — otherwise, silent fallback to raw. The time predicate must use the projection’s own expression: toStartOfHour(Timestamp) >= X routes; toStartOfInterval(Timestamp, INTERVAL 1 HOUR) >= X — identical values — does not. Same query: 2,022 rows versus 12,373,248. It doesn’t reason about equivalence. It’s a bouncer with a laminated list, and toStartOfInterval is toStartOfHour’s identical twin in a different jacket. Projections don't automatically backfill which is fine but be aware when you run your benchmark and nothing has changed. You gotta think really hard before you add a projection. It's effectively a copy of the data, so make sure your ORDER BY is correct before you try to layer projections on top of it. GROUP BY partition tells you the coverage of your projection. You need this in order to know whether the query has access to enough historical projection data for you to actually materialize the thing you want to show. A wrong table name in the ad-hoc filter config is silently ignored. An unqualified table key instead of the schema-qualified one returned 79,350,116 rows where the correct key returned 2,314,678 . A typo produces a confidently wrong dashboard. The only reliable test: change a filter, confirm the number actually changes, per panel. Grafana's ad-hoc filter UI offers one operator and one value per key. There is no way to express through the filter box — a long-standing upstream limitation ( grafana/grafana#10918 , still open for cross-column boolean OR). Grafana's time handling and panel model will surprise you in every possible way. Hour-bucket edges are a correctness defect factory.

0 views
Simon Willison 1 months ago

Hello Python 3.15, goodbye 3.10, and now I can use 3.11

Python 3.15 is here ! Here's the essential What's new in Python 3.15 guide, this year curated by 3.14 and 3.15 release manager Hugo van Kemenade. As someone who builds open source software that works in every supported Python version, I get to say goodbye to Python 3.10 and start embracing features from Python 3.11. First, some highlights from Python 3.15. I'm most excited about Explicit lazy imports from PEP 810 . These let you import modules like so: The module will then be imported the first time those symbols are used in the code that follows. My LLM CLI tool implements plugins. I had to recommend manual lazy imports in the plugin author documentation, because simply installing plugins that used heavy dependencies such as could add multiple seconds to every invocation of the command. Lazy imports directly address this startup cost. The new unpacking syntax is neat. Here's an illustration lifted from the documentation : PEP 798: Unpacking in Comprehensions by Adam Hartz and Erik Demaine proposed this as an extension of the unpacking syntax from PEP 448 , which landed in Python 3.5 back in 2015. Python's existing tracing profiler has been renamed from to profiling.tracing , and has been joined by an exciting new sampling profiler called Tachyon, aka profiling.sampling . Sampling profilers work by periodically capturing stack traces from running processes, then using those to calculate which functions have the most time spent inside of them. They're very low overhead and can provide a deep insight into running systems, without needing to instrument those systems first. I've previously used tools like py-spy for this. Now this capability is baked into the standard library. If you have the PID of an existing Python 3.15 process you can start gathering samples by running this (I had to use on macOS): Interact with the main program a bit, then hit Ctrl+C to stop gathering and see a report, which looks something like this: It can also produce flame graphs like this and heat maps like this , and can even output a format that can be viewed in the Firefox profiler UI. Miguel Grinberg investigated How fast is Python 3.15? the other day against the 3.15.0rc3 release and found performance to be mostly the same as 3.14, with the exception of the experimental JIT. The JIT gave a 1.2x boost over the standard 3.15 interpreter (on a Fibonacci test), the first time Miguel observed a meaningful JIT performance increase. Miguel's explorations of 3.14 last year found significant boosts against Python 3.13 across many more categories. I used the following sequence of commands to get to use Python 3.15 on my Mac. First I had to upgrade itself to ensure it knew about the new release: Then I installed the stable 3.15 release: And told that I wanted that to be the default for every command, unless the project itself pinned a different version: Now the following reports 3.15 as the version: I already have a bunch of tools installed using . I reinstalled them to use Python 3.15 like this: Then I can confirm the version like this: The release of 3.15 means that Python 3.10 (released in 2021) is officially no longer supported . My policy for my main open source projects is to keep them working in every supported Python version. I can now drop 3.10 and start using features that were introduced in 3.11. Here's the What's new in Python 3.11 document. I finally get to rely on tomllib being available! Datasette makes extensive use of . 3.11 adds TaskGroup and timeout() , and several more new features. is a new in 3.11 alias for . Python's SQLite support got some significant upgrades in 3.11, including: Plus a whole bunch of improvements to Python's typing system . I plan to land Python 3.11 improvements to a bunch of my projects over the next few weeks. Here's my first Datasette PR removing that workaround. I've also updated my click-app , datasette-plugin , python-lib , and llm-plugin cookiecutter templates to test against Python 3.11 through Python 3.15. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . setlimit() and getlimit() expose a mechanism for setting SQLite runtime limits from Python. create_window_function() lets us create custom window functions in Python, as a class with , , , and methods. blobopen() provides an efficient mechanism for partial reads and writes of large binary values stored in a SQLite row. You can now call to remove a SQLite authorizer. Datasette had a workaround for this!

0 views
Jeff Geerling 1 months ago

Radxa's Q8B has 2x the performance and expansion of the Pi 5

There was a time I'd look at a board like the Radxa Dragon Q8B (at left, above) and be like, "there's no way I'd spend $209 on an SBC with 8 gigs of RAM". But we're in 2026, and seeing the 8 gig Raspberry Pi 5 going for almost the same amount, I figured I'd give it a shot. On paper , the Q8B beats the Pi 5 in pretty much every way. A lot of that is thanks to this Snapdragon 8cx Gen 3 chip, which is the same chip I tested on Microsoft's Windows Dev Kit 2023 .

0 views
Jim Nielsen 1 months ago

“Getting off the Modernization Treadmill”

My notes from this talk by Alexander Petros at Big Sky DevCon 2026 . Alex starts by noting how “modernize” used to mean something along the lines of “update this thing that was made before I was born”. But now “modernize” means something more like “update this thing from 5-10 years ago” (hence the framing of the talk, the “modernization treadmill”). Using a real-world example of an incredibly slow website that was required to access state-sponsored programs for welfare, Alex points out the disparity in conditions between those of us who make software and those who have to use them: The people who develop these websites are usually doing so on high-powered internet connections and high-powered devices, but they're not using them in the conditions that the people who most need those benefits are going to be. Then he shows a Reddit thread where somebody essentially posted, “I’m having problems with this website. I’ve been waiting for months for my application to go through. Any suggestions?” And one Reddit user responded, “The best thing you can do is go into the physical office, get a case worker, and your problems will be solved within the hour.” I guess we've come full circle now. It used to be: “Don’t talk to anybody. It’s faster and more convenient to use the website!” But now it’s: “Don’t use the website. It’s faster and more convenient to talk to somebody!” Have we failed at making websites? And is our failure, at least in part, rooted in the fact that we don’t leverage the basic tools for making websites: HTML, CSS, and (in a distant third) JavaScript? Alex goes on to argue that the technologies of the web have an ideological bent and, if used as designed, can solve so many of the performance, accessibility, and usability issues that plague so many websites. The grain of the web’s technologies are rooted in these values: Which means if you use them as intended, they are optimized to deliver outcomes rooted in those same values. So if you like those values and you want those outcomes, use the platform. Take HTML, for example. Here’s Alex: HTML does [performance improvements] for you for free. If you've coded your website in a proper, semantical, structure HTML style, it will just get better over time at zero cost to the people who built that website HTML is your friend. HTML won’t give you up or let you down . HTML will make it difficult for you to make a bad website. Write it in to the requirements of the project you’re doing that it work without JavaScript. Not necessarily that it doesn’t have any JavaScript, but just that the core functionality of the website can happen without JavaScript. If you do this […] you will find that it’s very hard to deliver a bad web page because the structure that HTML requires is one that fundamentally is good for the user, performant, and cost effective. Technologies are imbued with culture, which influences what you do and how you do it. If you can align your ideological beliefs with your technological choices, you might end up with an outcome that aligns with your values — who would’ve thought, eh? A lot of modern software developers come from websites like Facebook, they come from big tech companies [who] fundamentally have a different set of priorities. Their job is to keep you on the website as long as possible so that you consume more ads. But that’s the opposite set of requirements and priorities that the government needs to be doing, which is to build something that is clean, quick, efficient, and gets you in and out as fast as possible. So [I tell people] that the technology they use comes from [an] ideological place. But there are different ideological places that produce different technological results and if we start from those through lines then we can produce services that help people who need them. The ideological principles of the web are well established : users over everything else. If you use HTML as much as possible, you’ll make something that’s as user-friendly as possible on the web. Reply via: Email · Mastodon · Bluesky User-friendly Backwards- and forwards-compatibility Long-term viability Universal accessibility

0 views
Langur Monkey 1 months ago

JPEG XL finally lands in Chrome!

Today, I was browsing Hacker News , when a particular item caught my eye: Hacker News post with the Chrome JPEG XL announcement . I opened it promptly and, indeed, it was not a prank. Finally, after years and years of bull**** from the Chrome team, they are shipping it. They are shipping a JPEG XL decoder. For those of you who don’t know, JPEG XL is a new image codec that includes most of, if not all, the features we’d want in the image codec of the future: This is awesome news for the codec . Whether we like it or not, Chrome is the dominant web browser, with ~66.5% of the market share , very far from its competitors (Safari, Edge, and Firefox, in that order). Now, I don’t use Chrome at all, but I still think that this was just the final piece of the puzzle needed to help the format go mainstream. So, what finally pushed them over the edge? If you read their announcement , they cite “consistent feedback and requests from web developers” and its massive popularity in the Interop 2026 project. Translated from corporate PR-speak: the community simply refused to let it go. Web developers, photographers, and open-source advocates kept pushing, complaining on bug trackers, and demanding a truly superior image format instead of settling for Google’s own WebP or the heavily pushed AVIF. The technical excuse the Chrome team used for years was that a C++ decoder was too much of an attack surface to run safely in the browser. Well, it has now been elegantly sidestepped. To bring JPEG XL to Chrome 155, they have integrated , a pure Rust reimplementation of the decoder. This tackles the memory safety issues head-on, eliminating the risk of out-of-bounds reads and use-after-free bugs of other decoders. And to make sure it doesn’t run sluggishly, they built a SIMD abstraction layer ( ) to maximize hardware performance without compromising Rust’s safety guarantees. This is a massive victory. Not just for JPEG XL, but for the open web. As I demonstrated in my comparison back in 2023 ,JXL has significant advantages over older formats and, in many cases, over AVIF as well, especially for high-fidelity photography, lossless compression, and progressive decoding. The fact that it allows lossless transcoding of existing JPEG files is of special importance. It means that we can finally upgrade our backlog of old images without losing a single pixel of quality, while saving a big chunk of disk space and/or bandwidth in the process. With Firefox adding it to Labs earlier this year, and Chrome finally capitulating, Safari is left as the major outlier with only partial support. The dark days of “Google kills JPEG XL” are officially over. It’s time to start converting your image libraries, updating your tags, and serving . The future of (high-quality) web imagery is finally here, and I’ll be ready for it. File size reduction by \(~20-60\%\) w.r.t JPEG. Lossless JPEG transcoding. Progressive decoding. Wide gamut, HDR and 32-bit support. Animations and transparency (alpha channel). Shines with high-fidelity photographic images. Fast-ish encoding and decoding. Royalty-free and FOSS. Support for super high-resolution images, of up to 1 terapixels ( \(2^{30}-1\) pixels per side).

0 views

How fast is Python 3.15?

It's October once again, and that means it is time to take the new release of Python for a spin (technically, it is the 3.15.0rc3 release that I'm using, the official 3.15 release is still a few days out). As I did with my Python 3.14 performance article of a year ago, today I'm sharing a new run of my informal Python benchmark, comparing Python 3.15 against previous interpreters all the way back to 3.10. If you are not interested in the charts and the tables and just want to read my analysis, feel free to jump to the conclusions section at the end.

0 views
matklad 1 months ago

Benchmark In Milliseconds

How long should a micro benchmark run? My rule of thumb is to tweak the input size until the benchmark takes about 300ms, for the following reasons: The imminently-to-be-stated assumption here is that the purpose of benchmarking isn’t so much a precise measurement of performance, but rather providing the author with enough intuition to make a correct decision. Milliseconds are integers ranging from 1 to 999. Enough precision to notice even a small improvement, and easy to scan visually. No need for different units or floating points (compare with ). Anything faster than, say, risks being skewed by fixed costs (e.g, interpreter startup). Hundreds of milliseconds is an eternity for a computer, usually enough to make one-off overheads irrelevant without using fancier (= less robust) techniques to explicitly account for them. For a human, hundreds of milliseconds is fast, but noticeable. Pushing numbers into human-perceptible range allows me to use my intuitive sense of time and speed, it doesn’t rely exclusively on numeracy. It’s plain fun to see, as a result of optimization work, how a previously lagging CLI command becomes “instant”. But anything longer than a second makes iterating on the benchmark slower than it needs to be. Running a benchmark 10 times in a row to eyeball variance should be fast!

0 views
Unsung 1 months ago

A loading state within a loading state

One of the classic parables about loading goes as follows. A building manager had a problem: people were complaining about the elevator being too slow. But upgrading the elevator was an enormous expense, so the manager came with an alternative solution: they put a mirror next to the elevator call buttons. Even though the elevator didn’t become any faster, complaints trickled to a halt, as people were checking themselves out in the mirror and that changed their perception of how long the wait really was. In the late 1970s and early 1980s, it wasn’t uncommon to equip early home computers with audio tape players, using the same cassettes designed to record and play music. Software was encoded as sound. A lot of redundancy was needed to account for many low quality cassettes and misaligned playheads, so loading times were really long. There were other disadvantages – no random access but definitely random errors, tapes wearing out and being “eaten” by the player – but tapes were so much cheaper than floppy disks or cartridges that they endured. = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/1.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/1.1600w.avif" type="image/avif"> On one of the computers, the British ZX Spectrum from 1982, an interesting convention emerged. As the game was arriving from the cassette, what was loaded before the code itself was its “title card.” The loading process of just this one image took about 35 seconds, and watching the graphic emerge was mesmerizing, almost like deciphering a puzzle – especially when, only toward the end, the color attributes appeared and the whole thing clicked into place: Some of these were beautifully done ( here’s a gallery of… all of them ?), created by talented artists working in a very unforgiving medium, a pleasure to watch materialize on the screen… …or, at least, it felt so for the first few encounters. Upon loading the game for the nth time, you grew more and more aware of the fact that you are spending precious time waiting to load a screen you’ve already seen. This was different than the first example; the mirror existing never slowed down the elevator. Sure, it was just 35 seconds out of a 5–20 minute load time – I told you cassettes were slow! – but still. Some people loved them, others built truncated versions that skipped the title screen altogether (as much as you could imagine just fast forwarding through it like you would through a song, it didn’t work that way). Figma is a web design app, and its editor arrives in a big, everchanging blob of JavaScript, in addition to having to load and decode the contents of the file itself. This isn’t 5–20 minutes, but it’s long enough that it necessitated a loading screen of the heavy variety: one with a progress bar. During my time at Figma, an idea would reappear with surprising regularity: what if we put a little hint next to the loading bar? Something that could teach you about a useful keyboard shortcut, or a trick? You’ve seen that before, often in games : = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/4.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/4.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/5.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/5.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/6.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/6.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/7.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/7.1600w.avif" type="image/avif"> It seemed like a thoughtful gesture for the user, a mirror of the elevator mirror idea. But I fought it, tooth and nail, for one very specific reason: this would remove pressure to make this screen fast. Had we given the screen another purpose, subconsciously or not, we might start caring less about improving it – as a matter of fact, you could make an argument to introduce artificial minimum loading time just so that the user could finish reading the tip! I had a similar feeling when, in 2018, Gmail replaced its simple progress bar loading state with this: It was huge, corporate, intense. It felt like an admission of defeat: our app is slow, and I guess we’re surrendering ourselves to it. But recently, I’ve noticed not just that Gmail’s loading state has been simplified, but also that it doesn’t appear nearly as often: I don’t know the technical details: is it caching? a more intense refactor? Either way, as much as I absolutely dislike what Gmail has become – there might be no more harrowing place in the web app world than Gmail’s settings, for example – kudos to the team for making it better. Gmail seems to be loading a lot faster now. (There is one small exception: it would appear that the loading state image itself doesn’t have a proper loading state – a rather strange omission.) There is, I believe, a more universal lesson in here. Loading is a complex space, where sometimes the most natural solution is a bad one, sometimes making things faster is making things slower , and sometimes new thoughtful design for whatever delay there is can be a better use of engineering effort than focusing on shaving off more milliseconds. (It’s both speed and the perception of speed that matter.) Also, this is exactly what Y2K was : We remember and react to loading states that are elaborate, cute, memorable – but we never see or link to loading states avoided . And while you see occasional stories told by engineers making things faster, you don’t often hear accounts of people within companies, somewhere at the intersection of design and engineering, who work hard designing thoughtful loading states, avoiding loading state bloat, or figuring out some clever hybrid approach. But, to be fair, there is also no parable of a building manager who just forked over some cash and installed a better elevator.

0 views