Andrew Atkinson 4 days ago

PostgreSQL 18: 23x Faster Inserts With UUID v7

We recently switched to version 7 (v7) uuid primary keys and saw significantly faster inserts for some tables. The databases were running Postgres 18.4 and mostly used v1 with some v4 uuid values for primary keys. Changing the column default involved running a single alter table command, but did require an exclusive lock on the table, blocking everything including selects. To solve that, we used a short lock timeout and lots of retries. The biggest speedup was 23x faster average execution time for a multi-row insert query called 12,000 times per minute on a table with billions of rows. This September and October I'll be in Austin, TX and NYC, check my Book page for upcoming appearances. The system uses UUID primary keys throughout. I typically recommend starting with bigint and sequences over UUID v4 primary keys , although here uuid v1 was used. Insert performance is not as bad for v1 compared with v4. Still though, v7 brings better performance than both for inserts and can also result in smaller indexes with fewer page splits meaning less CPU and IO. What drives bad performance for v4 and to a lesser extent v1? Let’s do a quick refresher. As new table rows are inserted and a primary key is defined, primary key values are maintained in sorted order in a b-tree index. Just like table rows, index entries in Postgres are stored in fixed size 8kb pages. Postgres needs to know in which page to place the new index entry. For sorted order, the first bytes of new uuid values are compared. For v4 given new values are very random and not monotonically increasing (they lack “monotonicity”), values can be earlier or later, meaning they’re unlikely to be placed into the same recently accessed page. This is bad for caching! When new values are monotonically increasing, the recently accessed page is “hot” in the Postgres buffer cache (in memory copy of the on-disk page). When Postgres is not able to use the hot index page for the newly inserted value, that page could be outside the buffer cache, not in the OS cache, and ultimately result in a much slower disk read which increases latency. Besides the worse insert performance, since v4 values are scattered to more pages, this means there are more “page splits” when new inserts are attempted in full pages. Page splits cause more latency from increase WAL and IO. We experimented and benchmarked with v1, v4, and v7 uuid formats and we leveraged the research and write-ups from external sources like the ones below. Benchmarks are great, but what kind of real world results did we see? We decided to make this the new default unless v4 was needed for more randomness. After all qualified tables were changed, I began going through insert queries for each changed table. For many of the tables, there wasn’t an obvious change. However, for a handful we saw an immediate and significant improvement. I picked 5 with speedups of 6x, 8x, 9x, 20x, and 23x. The PgAnalyze graphs for the 23x, 9x, and 6x queries are shown below. Showing PgAnalyze insert query graphs for tables A, B, C: Table A - 23x reduction. 0.7ms to 0.03ms, 12000 calls/min Table B - 9x reduction. 0.6ms to 0.07ms, 2000 calls/min Table C - 6x reduction. 0.50ms to 0.08ms, 9500 calls/min Now that we’ve seen the results, let’s talk about how this was done and the challenges. The UUID values came from various sources: We replaced most of these with the function in Postgres 18. To do that, we needed to run a single statement per table. The statement ran fast, so no problem, right? One wrinkle we found was that modifying the column default while fast, required an lock. This lock type conflicts with every read and write operation including regular statements. For our highest queried tables, they’re queried constantly, so this was a problem. There was almost never a “window” to perform this operation, and we didn’t want to take downtime for this switch. While heavily queried tables were a challenge, infrequently queried tables did not pose a problem for this alter table statement at all. For those, we could use our migrations framework (Active Record in Ruby on Rails) and perform the using a regular old migration. For those, we did add some safeguards, by creating an explicit transaction and using to control timeout values. We’d set short timeouts for the to give up quickly if it didn’t work or ran too long. For the higher activity tables, we’d need some retries. We’d use a manual psql session: The would commit if it grabbed the lock within 50ms, or we’d get an error that the was reached. The benefit of the manual approach was we could retry until successful and backfill a Rails migration to keep everything in sync. A more sophisticated solution might have automated retries within Ruby. However, for our most heavily queried tables, we wanted even more control over the retries. How did we do that? Sometimes one or two retries would do the job. Great, we’d move on. However, for our most heavily queried table that didn’t work. What ended up working was using the same strategy of retries, but just adding more sophistication with looping and backoffs. Claude helped me cook up the PL/pgSQL looping retry function below, I did some testing and was ready to try it. It has these features: By using the function above, we were able to find a small window to perform the after several dozen quick retries! In cases where even many retries won’t work, and we don’t want downtime, we may be left with needing to actively monitor lock holder queries and to cancel them (assuming that’s ok). Thanks to Ants Aasma from the community PostgreSQL Slack for this idea. We didn’t end up needing to do this, but here was my prep for this. It’s still useful to review lock holder queries. First we’d inspect live queries: And identify queries holding locks: If we find them, we could cancel them to create a window to run our . That could mean bad user experience so you’d need to figure that out for your database. We’d likely want to stack up our operation to occur immediately after. Fortunately we didn’t end up needing to do this, but I’d be interested to hear the stories from others with heavily queried databases. Since uuid v7 values use a timestamp in their first bits, this timestamp can be easily decoded. This can be viewed as “leaking” or exposing the creation time of the record via that timestamp, which could be a downside for your database. You’ll have to decide that. v4 UUIDs do not expose the creation time. We found some significant speedups for insert queries after switching to primary keys, for a relatively low effort change. A nice ROI. The only wrinkle was the exclusive lock required, but we solved that with short lock related timeouts and many retries. Although this didn’t benefit 100% of our tables, the gains for some were significant and uuid v7 has become our new default choice for uuid primary keys. Thanks to the Postgres core team for creating this new capability within Postgres. The availability in core made it possible to adopt on AWS RDS which supports a limited amount of extensions. Thanks for reading, and until next time. How Sequential UUIDv7 Boosts Ingestion Performance Simplicity and power of UUID v7 PostgreSQL UUID Performance: Benchmarking Random (v4) and Time-based (v7) UUIDs The function from the module The function added in Postgres 13 that generates v4 UUIDs natively UUID v4 values sent by a client application, which meant the column default function was not used Try up to 50 times (max attempts is configurable) Add a pause in between retries, with a jittered backoff of 50-250ms

Ruby Database PostgreSQL Performance

▲ 0

0 views

Farid Zakaria 5 days ago

Stamping build info in constant memory

This is a fun little trick I came across at . I did not invent it, but I thought it was interesting enough to understand better and share. At we build with buck2 and we stamp our executables with build information: build-id, timestamp, author, the usual suspects using as a step after the link. The reason it is a separate step is caching. If the build info was generated at link time then everytime we link the binary it would produce different bytes causing it to not be bit-reproducible. When something is bit-reproducible, it is safe to cache it, and the build system can apply early cut-off optimizations. That works, until the binaries get big. We noticed that ’s memory use scales with the size of the file it is stamping. Stamping is exactly the kind of step that runs massively parallel at the end of a build so this can cause a lot of memory pressure. Why is the stamping step reading the binary at all? 🤔 Let’s measure the claim that the memory use of scales with the size of the file. We will attach a JSON build info blob to an increasingly large synthetic executable and measure the peak RSS of the stamping step. The graph confirms the claim. The memory use of scales linearly with the size of the file being stamped. Surprisingly, the slope is two . The peak RSS is roughly twice the size of the file being stamped irrespective of the size of the build info being attached. 1 I am helping to shepherd a PR open against LLVM to stream the ELF output rather than materialize it, which roughly halves the peak. That is a definite improvement but the problem remains that in order to add a tiny section to a large binary, the whole binary has to be read into memory. The memory use is still linear in the size of the file. The problem is not poor implementation on the part of . Adding a section to an ELF touches three separate things: 0x78 bytes in .interp\0 .buildinfo\0 .rela… 0x70 0x78 0x83 the section's bytes 88 bytes of JSON, at 0x401021 The entry holds neither the name nor the bytes. It only refers to them — and only one of those two references can be repointed in place. In order to account for the new section, the section header table has to grow by one entry, and has to grow by the length of the new name. The current model for is to read the whole file into memory, add the new section, and write the whole file back out. That is why the memory use scales with the size of the file. How can we avoid having to rebuild the whole file just to add a tiny section? The trick is to pay the cost at link time rather than at stamp time. We can have the linker emit a placeholder section with the right name and a single byte of content. The section header table entry is already there, and the name is already in . The post-link stamping step can then append the payload to the end of the file and update the section header entry to point to it. 💡 We make linker emit the section during the normal build. It does not need to hold anything; it just needs to exist so that it owns a name and a header. Our “stamp” step is now incredibly simple. It does not need to read the file at all, it just needs to write the new payload and update the section header entry. Nothing that already exists moves. does not move, the section header table does not move, no other changes. The edit is sixteen bytes , at a file offset you can compute from the ELF header, plus a . read into a model, serialized again — 1,238 bytes of it actually differ reserve one byte, then append 16 bytes + a tail ehdr phdrs .text .rodata … .shstrtab section headers payload the reserved byte sh_offset, sh_size Note Why 1 byte? Turns out that and GNU disagree on whether an empty section is a valid ELF. The one byte is a cheap way to make both linkers happy. The payload lands after the section header table, which looks alarming the first time you see it but is completely legal. Nothing in ELF says section contents must precede the section header table. The kernel also never looks at section headers also, it loads segments out of the program headers, which we do not touch. I wrote a small C version to benchmark it in contrast, please be mindful that this graph is log-log. Our trick works! The “append + 16 bytes” approach is constant memory. Not only is the peak RSS constant, but the wall time is also constant and much faster by avoiding the read and write of the whole file. It is often easy to reach for general-purpose tools like as they are a swiss-army knife for manipulating object files. What I like about this trick though is that there are meaningful improvements to be made by writing special purpose tools and that does not mean we have to accrue large maintenance costs. In this case it was a tiny 200-line C program. The economics of these tools is also changing with the rise of LLMs in our workflow. While many are concerned about the influx of generated code, I remain optimistic that we we can use them to find such opportunities. Don’t be afraid to write a small tool to solve a specific problem. For those thinking this is an LLVM specific issue, GNU exhibits the same behavior.  ↩ the section’s bytes , somewhere in the file a 64-byte entry in the section header table describing where those bytes are the section’s name , which is not in the entry itself but rather the entry holds a offset into , so the name has to be appended to that string table 0x70 0x78 0x83 the section's bytes 88 bytes of JSON, at 0x401021 The entry holds neither the name nor the bytes. It only refers to them — and only one of those two references can be repointed in place. In order to account for the new section, the section header table has to grow by one entry, and has to grow by the length of the new name. The current model for is to read the whole file into memory, add the new section, and write the whole file back out. That is why the memory use scales with the size of the file. Pay the byte at link time How can we avoid having to rebuild the whole file just to add a tiny section? The trick is to pay the cost at link time rather than at stamp time. We can have the linker emit a placeholder section with the right name and a single byte of content. The section header table entry is already there, and the name is already in . The post-link stamping step can then append the payload to the end of the file and update the section header entry to point to it. 💡 We make linker emit the section during the normal build. It does not need to hold anything; it just needs to exist so that it owns a name and a header. Our “stamp” step is now incredibly simple. It does not need to read the file at all, it just needs to write the new payload and update the section header entry. append the payload to the end of the file write the new and into the placeholder’s section header entry the reserved byte sh_offset, sh_size Note Why 1 byte? Turns out that and GNU disagree on whether an empty section is a valid ELF. The one byte is a cheap way to make both linkers happy. The payload lands after the section header table, which looks alarming the first time you see it but is completely legal. Nothing in ELF says section contents must precede the section header table. The kernel also never looks at section headers also, it loads segments out of the program headers, which we do not touch. Benchmark I wrote a small C version to benchmark it in contrast, please be mindful that this graph is log-log. elfstamp.c Our trick works! The “append + 16 bytes” approach is constant memory. Not only is the peak RSS constant, but the wall time is also constant and much faster by avoiding the read and write of the whole file. One trick pony It is often easy to reach for general-purpose tools like as they are a swiss-army knife for manipulating object files. What I like about this trick though is that there are meaningful improvements to be made by writing special purpose tools and that does not mean we have to accrue large maintenance costs. In this case it was a tiny 200-line C program. The economics of these tools is also changing with the rise of LLMs in our workflow. While many are concerned about the influx of generated code, I remain optimistic that we we can use them to find such opportunities. Don’t be afraid to write a small tool to solve a specific problem. For those thinking this is an LLVM specific issue, GNU exhibits the same behavior.  ↩

Performance Programming JSON Ci Cd

▲ 0

0 views

Dangling Pointers 5 days ago

Vector Value Prediction with Element-wise Stride Compression

Vector Value Prediction with Element-wise Stride Compression Yanmeng Huang, Ling Yang, Yuanhu Cheng, Junhui Wang, Quan Deng, Junbo Tie, Yongwen Wang, Hai Zhong, and Libo Huang GLSVLSI'26 This paper makes an astute observation: the contents of vector registers are losslessly compressible. There are many hardware and architectural optimizations that can be derived from this fact. This paper presents one: prediction of vector register values. The point of value prediction is to expose more parallelism to a modern processor, so that more instructions can be executed in parallel. Much like branch prediction, value prediction speculatively breaks dependencies to allow the processor backend to process a less-constrained set of instructions. If the predicted value ends up being wrong, then the processor must throw away some work and start over. One way to implement value prediction is much like branch prediction, using the PC and branch history as the index into a table which contains a predicted value and a level of confidence that the predicted value is correct. This paper describes such a technique. SIMD instructions are an orthogonal way to expose more parallelism to a processor backend. Since value prediction and SIMD are orthogonal, why not combine them? Two reasons: The predictor must predict the values of all lanes of a vector register. If any one lane is a bad apple, it spoils the bunch. The data structures needed to make accurate vector-wide predictions would consume a lot of on-chip memory (storage would grow linearly with vector width) The core idea of this paper is illustrated in Fig. 1: Source: https://dl.acm.org/doi/full/10.1145/3787109.3815244 The left side of the figure shows raw (uncompressed) vector registers with varying element widths. The right side shows compressed versions. The gray rectangle represents the value stored in lane 0 (no compression). The orange rectangles hold the differences (i.e., strides ) between neighboring elements. The magnitude of a stride is typically small, so strides are stored with only a few bits per stride. Here is a concrete example: The key point here is that the base + strides representation consumes fewer bits than the raw representation. If the values in a particular register cannot be represented with the base + strides representation, then those values are not considered candidates for value prediction. Fig. 6 shows IPC improvements provided by vector value prediction across a set of benchmarks: Source: https://dl.acm.org/doi/full/10.1145/3787109.3815244 Dangling Pointers I didn’t see specific handling for floating point values in the paper. I wonder if a dedicated compression scheme for those types is warranted. Thanks for reading Dangling Pointers! Subscribe for free to receive new posts. The predictor must predict the values of all lanes of a vector register. If any one lane is a bad apple, it spoils the bunch. The data structures needed to make accurate vector-wide predictions would consume a lot of on-chip memory (storage would grow linearly with vector width)

Hardware Performance

▲ 0

0 views

matklad 1 weeks ago

Rust Glancer

Rust Glancer , a functional LSP server for Rust which uses two orders of magnitude less RAM, is incredibly cool. Go check it out! This post started as a comment on lobste.rs, but I figured it out that it’s better to publish it somewhat more prominently. Don’t expect polished writing though! Some thoughts: rust-analyzer uses rowan for syntax tree representation Yeah, rowan is garbage :P I was really thinking about And Rowan is pretty good for that. But that’s 1% use case. The 99% use case is all the code in your 6666 dependencies which you won’t ever look at, but which needs to be at least shallowly analyzed. Even for incremental tool whose main goal is refactoring, the primary AST structure should be just a list of arrays. There might be a real post about that at some point, see https://youtu.be/G93oYL1ry70 as a teaser. Rust workspaces genuinely have a lot of information that must be indexed: thousands of functions, structures, traits, relationships between these, function bodies and statements in them, etc. Each of these needs to be analyzed and remembered, and you can’t really cheat if you want to have things like “find all references to this structure”. If I understand correctly, Rust Glancer wants to process each function body. I think that part can perhaps be made lazy (but not incremental!) with little overhead? Index all items, but, for functions, do only the currently opened file? This might combine some of the better parts of both worlds. Would be interesting to compare memory usage with Rust Rover. Net of the IDE GUI itself, I would expect RR to be more compact. Some features are unlikely to be supported though, such as build scripts / proc macros support via proc macro invocation I might be rationalizing/misremembering things, but IIRC it’s exactly around adding proc macros that the thing began to feel unreasonably bulky. Expanding proc macros is slow as we are running real code, we can’t really do normal IDE cheats. And proc macros generate a lot of code. At one point I measured, it was like 30% of rust-analyzer binary size was attributed to JSON parsing code. If no one sees the code, it can’t harm anybody, right? One potential approach here is to pull the Sorbet trick, where you don’t run meta programming at all, and instead have a plugin interface to “explain” the effects of what that would have done. Instead of running serde, we just add a shim that injects with an empty body. I’m not sure why, but in rust-analyzer I’ve observed that when agents edit the code, inlay hints can get out of place Rust analyzer’s core data model is very pedantic about always observing consistent snapshots of the code, and does its best to ensure that the language client and server have a shared, strictly serializable view of the world. It’s a shame that LSP doesn’t allow that to be correct , only heuristically right , unlike the older Dart Analyzer protocol, which has sound data synchronization. However our implementation of file watching is sketchy! First, there are two backends: we can ask the editor to do watching for us, or we can use server side watching. Try changing this option and see if it helps? But then, yeah, my recollection is that our native watcher’s API was fundamentally racy, and I didn’t do the messy platform-specific work of making it correct. But the main thing I want to write, and why I moved from the cozy lobste.rs text area to the luxurious comforts of an Emacs buffer, is that right now rust-analyzer is a bit like that half-drawn horse meme, except that it’s only the head half of the horse. One Big Idea of IntelliJ is that it’s PSI API (essentially AST with resolved types) is really an interface, and there are multiple provides. And in a typical usage, there’s at least three backends in play: This is how I think such things should work. rust analyzer shouldn’t use salsa for all those 6666 dependencies you still haven’t looked at. It should just use rustc’s .rmeta files, switching to salsa, transparently, only when the user starts messing around their folder. The prerequisite for that is defining the abstract API for accessing Rust code. That was always the plan, and we did start on that at some point: https://hackmd.io/ytd82QNiT_Ku2XFr1EAtiQ rmeta-transparent – source code might not be available for some crates, the API should support pre-compiled rmeta files as inputs. But I don’t think that work was ever completed. This still seems to me to be the lowest-hanging watermelon here — split the world into arcy-pointy incremental tip of the iceberg, and mostly read-only, on disk, compact, dark, moist breeding ground for supply chain attacks. Such glance analyzer architecture would be great, imo! incremental parsing, incremental, DOM-mutation style refactorings, For the files opened in the editor, actively modified by the user, the PSI is backed by the concrete syntax trees. For the rest of the project files, the PSI is backed by the so called Stub Tree, a compact on disk representation storing only the “externally visible” parts of the file (so, without function bodies). If the user navigates to a new file, its PSI transparently switches from stubs to syntax tree. For dependencies, the PSI is often backed by the compiled .class files, produced by javac. If you navigate there, the IDE just decompiles stuff four you! Super cool!

Programming JSON Dart Open Source Performance

▲ 0

0 views

Giles's blog 1 weeks ago

Use the built-in GELU, don't roll your own!

Unsurprisingly, PyTorch's own built-in GELU function is faster than the hand-rolled one I've been using to date. But I was surprised at how much faster using it made things when training my models. I discovered this accidentally just now while working on something unrelated, but am logging the details here for anyone else that might find it useful. The headline numbers: the same code, training the same model on the same data, ran at about: That's a 20% increase in throughput for both of the built-in versions -- definitely nothing to be sneezed at. And what is particularly interesting is that there aren't that many GELUs going on -- it's a GPT-2 small-style model, with 12 layers. So that's 12 GELUs handling tensors shaped , which is for my training setup. Given that the rest of the model is doing all of the normal full attention stuff for GPT-2, it's really surprising that the GELUs alone must have been taking up so much of the time. The throughput numbers mean that we must have been spending about 17% of our time on the extra overhead from the hand-rolled version, so that sets a lower bound for how much time the GELUs were taking up. More info below the fold. Back when I was doing the "interventions" part of my LLM from scratch series , training dozens of GPT-2 small-sized models in the cloud and on my local machines, to keep things simple I used the original model code from Raschka's book. That happens to have its own implementation of the GELU function -- you can see my copy here . I'm not that sure why the hand-rolled version is in there -- he covers the maths, but the specific implementation isn't explained in that much depth, and it seems rather like boilerplate, just a "type this in and use it" kind of thing. By contrast, for example, while he does explain the maths behind cross-entropy loss in similar detail, we use the built-in function for it rather than coding it up ourselves. When I switched to using JAX for my own from-scratch implementation , I decided to not bother porting the boilerplate, and just used JAX's own built-in version . I was revisiting the PyTorch code -- I'm in the process of extending it with mixture-of-experts support, about which more in a later post -- and decided to switch from the hand-written GELU to the PyTorch one just to tidy things up a bit. I noticed something interesting -- my new MoE code suddenly seemed to speed up. Was that a mirage? Or had I discovered part -- or even all -- of the reason why the JAX code was so much faster than the PyTorch code? With PyTorch, I was typically getting training speeds of about 21,000 tokens per second, while in JAX I was getting 24,000 tps or so. I'd been chalking that up to JAX's JIT compilation, but could it have been just a result of a random implementation choice I'd made? I did three partial test training runs, letting each one run for 20 minutes to allow the training speed to settle down from any startup overhead. Firstly, with the old hand-coded GELU: So it was getting 20,920 on average over those 257 global steps. That speed was in line with the original run of the configuration I was using. Next, I introduced the built-in PyTorch GELU with no arguments: That does the full calculations for GELU, rather than using the -based approximation that the hand-rolled code did. After 20 minutes, it looked like this: So this time we were getting 25,134 tokens per second -- 20% faster! By default, PyTorch's GELU uses an exact calculation of the function -- the hand-written code from the book uses an approximation using . Luckily, you can get that same approximation from PyTorch: So, training with that for 20 minutes: 25,142 tokens per second -- basically the same as the non-approximate version. So: switching to the built-in GELU made my PyTorch code run 20% faster, at about 25,000 tps rather than 21,000. My JAX code, which used JAX's built-in GELU, ran at around 24,000 tps. I'd actually found that rather surprising, because in JAX I was training in full-fat 32-bit floating point, while in PyTorch I was using Automatic Mixed Precision (AMP) -- a special mode that allows it to use 16-bit calculations where it won't hurt the model much. I'd found that AMP gave PyTorch a huge speedup -- from 15,402 tps to 19,797 on one test. So JAX without AMP being so much faster than PyTorch with AMP was a bit of a surprise. Its JIT is pretty amazing, but I didn't expect it to be that much faster. Now I think that we have at least part of an explanation. I was using JAX's built-in GELU (interestingly, with its default parameters, which means that it used the approximation), but the PyTorch code was using the hand-rolled one, and that unduly penalised it and erased some of the gains it got from AMP. If I really wanted to dig into this, I suppose I might try JAX with a hand-rolled GELU to see what happened. My guess is that because of its JIT, it might actually handle it better -- the whole hand-rolled thing could be compiled into one thing on the GPU. Perhaps it would also be interesting to try the non-AMP PyTorch code with the built-in GELU. But I doubt that would really be the best use of my time (and my electricity bill), so I'll leave it here. On the other hand, I do intend to have a look at in the future, to see what kind of speedup I can get from it. And it might be able to compile and fuse together the hand-rolled GELU -- so that would be an interesting thing to experiment with in that post: does the built-in GELU advantage disappear if we're compiling? But anyway, for now, lesson learned: use built-in PyTorch modules when you can. It's a pretty obvious one ;-) [Update] On X, Sebastian Raschka noted that he used the approximate version of GELU in his code so that the models were compatible with the OpenAI weights -- they were trained with that version, so they may behave slightly differently if you use the "pure" version. That's a great point, and so I've updated my own copy of the code to use . 21,000 tokens per second using the hand-rolled GELU from Sebastian Raschka 's book " Build a Large Language Model (from Scratch) ". 25,000 tokens per second using PyTorch's built-in GELU with no arguments. 25,000 tokens per second using the built-in GELU with , which uses the same maths as Raschka's version under the hood.

Python Performance AI

▲ 0

0 views

Farid Zakaria 2 weeks ago

nixpkgs-multiverse: fast mode

“The fastest evaluation is the one that never happens.” – Sun Tzu, The Art of Evaluation nixpkgs-multiverse gives you every version of every package that ever shipped in Nixpkgs from a single flake input. Note It continues to blow my mind that this is even possible. It feels like it suddenly unlocks a new dimension of Nixpkgs, and I am still trying to understand what it means. I think this capability is a fundamental change to the way we think about Nixpkgs, and it is not just a new feature. It is a new way of thinking about the entire ecosystem. There was always a penalty at the center of it. Asking for a specific version of , such as , meant fetching the whole ~378 MB Nixpkgs tree from 2021 and evaluating it to determine the . What if we could skip that evaluation? What if we could just ask for the path directly, and have Nix fetch it from the cache if it is there? This is a common idiom if you have ever used . That requires knowing the store path upfront. nixpkgs-multiverse now has a attribute that does exactly that: it gives you the store path for every indexed version of every package. This lets you skip the download and evaluation of Nixpkgs and get the store path straight from the cache. No Nixpkgs is fetched. Nothing is evaluated. No experimental features and no needed for this to work. The complete Nix API, except for releases, works with this fast path. If you want to learn more read the docs about the feature. Every channel bump published a listing of every path Hydra built for it: , or a for back in the pre-2017 era. These files are still available, and they are the source of the multiverse index. The listing is a map from derivation name to store path. The multiverse index is a map from to the revision that shipped it. By joining the two, every historical version gets a concrete address: Knowing the path is not enough, especially in the Nix language. We need to convince Nix that a string that looks like a store path actually is a store path. exists but it is an impure function and requires to work. How do we get around this? We attach “context” to the String. Context is the invisible baggage a String carries in Nix. When you interpolate a derivation into a String, the result remembers where it came from, and that is what makes realise the dependency instead of writing a dangling path into a script. lets you attach it by hand. The identifies that “this String names a store path that must exist,” which is exactly what produces for a path already in your store, except this works for a path that is not in your store yet and is not in this evaluation’s input closure either. Loopole! 👿 We then wrap that in an attrset that resembles like a derivation and the Nix CLI is satisfied: This is tomberek ’s trick from fastpkgs , and it is an amazing trick to circumvent needing . Everything about this remains pure evaluation, and the resulting graph is gauranteed to be bit-for-bit identical to what Nixpkgs would have produced if it had been evaluated. The only difference is that we skip the evaluation of Nixpkgs itself, and instead use the store path directly. The eval path derives the address, the fast path remembers it. A “fake” ( ) derivation has no , because there is no behind it. Nothing can build it and it can only be substituted. The CLI often wants a though when you hand it a derivation attrset, so we must make sure to append the output (i.e. ): and need a real derivation. Every fake derivation carries a lazy that is the real, revision-exact derivation: In the spirit of trying to keep my index small, is empty, so there is not additional information about the package. You can still get the from the real derivation by using as well. This scheme rests on cache.nixos.org still serving thirteen-year-old paths, which thankfully it does and with the same signing key. To demonstrate that the cache is offering nearly every path that Nixpkgs ever built, I ran a census of every indexed version of every package and asked the cache if it was still alive. As of August 14 2026, all 271,187 of them are alive . All of them, down to every NAR payload file. That is 14.8 TB of unpacked software from 2013 onward, one fast command away. 1 There’s some other data on nixmultiverse.com about the census, dependency graphs and additional features. Check it out! All of this is also available via the mvs command line tool as well for offline use. I guess now there is a caveat: there is now a trick in the multiverse. It remains mostly an index, some JSON, and a behind a memo table. The clever trick is tomberek ’s, and it is three important lines. The NixOS infrastructure has never garbage collected the binary cache. It is an S3 bucket that only grows, and the bill is paid by the NixOS Foundation and its sponsors.  ↩ The NixOS infrastructure has never garbage collected the binary cache. It is an S3 bucket that only grows, and the bill is paid by the NixOS Foundation and its sponsors.  ↩

Open Source Performance Programming JSON

▲ 0

0 views

daniel.haxx.se 2 weeks ago

curl performance

tldr: the live version is here: https://curl.se/perf/ How fast is “fast” and is it good enough? Does it run as fast now as it did before or was there a regression? What exactly needs to be fast? How fast is it? These are questions that many projects and products face, and in curl we are no different. Yet, performance testing and comparisons are hard and full of landmines and time-wasting efforts. For many years we have occasionally brought up the idea of a performance test suite for curl only to shut it down again because the challenges seemed hard and no one was volunteering to do this. This week it changed. I started out trying to find existing projects that host performance results for Open Source projects so that we could just feed our results something else and get great visualizations and data management. I did not find any such. I then took a look at what existing tools there are for this purpose, and most pointers seemed to suggest that Grafana is a popular and maybe even a good solution to build something like this with. But man, that is a complicated machine and it felt more than a little overwhelming just figure out where or how to start with it. I decided to postpone that take as well. I decided that instead of trying to do this the best and optimal way – I shouldn’t let perfect be the enemy of good – I would start out by doing the things I know how to do and take it as far as I can one step at a time. Something should be better than nothing . Performance testing needs decently stable system conditions so that repeated runs produce reasonably similar results, when all involved factors remain identical. This is basically impossibly to accomplish using most cloud infrastructure since those are almost always shared with countless other users. At least on the cheap and free tiers we use. We probably need our own dedicated hardware for this, but instead of trying to figure out where to get that and arrange for that, I would start by running performance tests on my own local development machine. I am a single user on this and it has many cores and runs decently fast. It should be good enough to get this going on. I created a first shell script that updates the curl source code from git, it configures and builds it. Then it runs a bunch of tests, outputs a bunch of data and logs all the output in a single log file. I started out with a few simple tests. How fast does curl download a 100 GB file from localhost, how many allocations and how big allocations does it need for a single HTTP download? My second script parses all the test log files from the previous builds and generates summaries and graphs for them. To make it possible for humans to see how the performance changes between builds and ideally to automatically detect when something changes more than what should be tolerated. As I am a graph addict already since before , and that journey has taught me a little gnuplot , I decided that even while there probably are much better tools and fancy JavaScript things that could be used, I don’t know them and learning them now is an endeavor I rather avoid. So I stick to what I know and can get results with quickly. A third script is invoked from a crontab every twenty minutes, sets up some variables and invokes the runner script. Once the basics started to work, I showed my curl friends the early versions and I soon created a new git repository for the code . After a little more poking, I soon made my locally produced performance test summary get packaged and automatically transferred to the curl website after each build, and voila, the first public curl performance tests were live and public. Getting this data available immediate triggered curl developers. It only took hours until we had the first proposed changes to improve some numbers, and soon we had a few merges to that affect. Visibility really helps! The performance numbers we get are still varying to a certain degree, partially of course because I still use my machine for my daily development things, but also because most of them do real (localhost) networking and that is by its nature a little… varying . The system builds and runs a new round every twenty minutes and it does that using the latest commits from git. This setup makes it sometimes run many rounds on the same commit and it might also mean that it sometimes updates and get several new commits at once, so it might skip a round for some commits. I might reconsider this design later, but since it is still a twenty minute time window, the number of commits is still limited. When the script makes multiple build rounds on the same commit, it accumulates the numbers and for the graph it stores the maximum, the median and the minimum value. It helps show the variation per commit and allows us to cram more into the graphs. It is still early days, but there will be a maximum limit to how many commits that can be displayed in a single graph and still be helpful. HTTP/2 parallel download speed through 261 builds spread over 31 build rounds Distribution To help visualize the distribution and data spread per test, I created a separate illustration that shows the minimum, maximum, P25, P75, medium and mean values in a Box-and-Whisker Plot . A Box-and-Whisker Plot showing the HTTP/2 parallel download speed data distribution. Changing conditions An obvious downside with me just storing build logs in files, is that it will not scale up to the millions. I did however decide that I’m not designing this system for that. At least not now. Performance tests are highly specific and dependent on the exact machine it runs on, the exact third party libraries and their versions that are used, the other components involved in the tests, such as the servers, and more. I expect that we will change conditions for the tests every once in a while that makes it hard to compare the current numbers with past numbers. Therefore I think the performance test numbers and values are primarily useful in the short term. To help us spot if we land something that subtly and unintentionally degrades something. To detect extremely slow and long-term changes in performance and even making sure we can better survive wiping all the existing build logs etc, I introduced a concept I call stakes . As in a stake pole. A marker. An arbitrary threshold set manually for each specific test. This value can be used to measure performance test results against, now and later. As conditions change and maybe something makes the results go up or down and we are fine with those changes because they are motivated and expected, then we just change the stakes. If it works out, I might try to have the system automatically detect and maybe highlight tests that deviate too much from its set stake (at least if done in the wrong direction) . It could be a signal that something bad was merged. As with everything in life, things are often balanced out. We already ran into this when we eagerly merged several changes to reduce the number of allocations done for a single HTTP download, only to realize that one of the optimizations we did had the side-effect that it expanded the size one of the main structs maybe a little too much… Improvements in one area might come at an expense in another. With sufficient tests and data we can improve curl for users, and at the same time make sure that our changes don’t come with a cost we are not prepared to pay. Exactly how to make the balance is of course a question we need to deal with, discuss and decide. Possibly for every change we do! As I write this, we have 24 tests and a full test round completes in about six minutes on my machine. We can of course do multiple builds using different hardware, different operating systems, different build options, different third party libraries and different test servers to check more angles of performance, and I am certainly open for and prepared to do that going forward. I will however first let this single-flavor run for a while so that we get more data, get a change to tweak it and make it as usable as possible for curl developers. As with everything there is no end to what we can make this do. This is a start. I sure we can take it further as we move along. In particular if people join in and help out. Both with ideas and proposals for visualizations, graphs and new tests to add, but also with actual pull-requests and code. Over the last year, we have merged, on average, about 10 commits per day. If we keep this pace up and this performance test setup can show 100 build rounds conveniently into a single graph, that is just ten days of development. Probably not enough. Once we reach one hundred builds or so in the first graphs I need to consider adding separate long term graphs that use select data-points to display data development over a longer time. Some googling told me the Largest-Triangle-Three-Buckets, or LTTB for short, is a fine algorithm to use for this. I now do a separate “long term” graph that “downsamples” the full range down to something that can be shown in a reasonable way. I suppose we will see properly in the future how this works. The stake thing I mentioned is one way to help us spot gradual performance changes over time. Another googling told me that there’s a Mann-Kendall Test + Sen’s Slope algorithm to use to identify trends in graphs like this and it can be used to plot a trend. It might work as a helper to better identify… yeah, the data trend for each test. The HTTP/2 parallel download speed trend at a specific moment Developing This setup has only existed for a few days. There is lots to do, lots to learn and much more to experiment with. Your comments, help and pull-requests will be appreciated!

Performance Testing Open Source JavaScript Shell

▲ 0

0 views

matduggan.com 2 weeks ago

OTel Isn't Going Well (And I Made A Spreadsheet About It)

For years now one of the most reliable complaints I hear when I try to drag a team off their vendor specific SDK and onto OpenTelemetry is some variation of: "why does it seem like this isn't done yet?" Vendor SDKs for observability are, to put it charitably, idiot-proof. You install the thing, dashboards just load data, someone else worries about how all those pieces fit together, and you get on with your life. OpenTelemetry, by contrast, greets you at the door with a lot of "experimental" stamps and roughly six different ways to accomplish any given task. In OpenTelemetry's defense this was never what they were going for as a project. I've always respect that they stuck to their guns by attempting to build a truly vendor agnostic system that really doesn't care what you do with the data. I have never gotten a sense of a vendor being strongly preferred with OTel, which is quite the feat considering how lucrative and contentious the observability ecosystem was. Also considering that the maintainers of this project are largely employed by exclusively those companies. As the years wore on, I started to get nervous. Conversations in the semantic-conventions repo drag on and on and on. Different languages had dramatically different stories. Golang and Dotnet were first class citizens, but other languages lagged years behind the others. I started asking a lot of probing questions before recommending OpenTelemetry to smaller teams who didn't have the time, budget, or emotional bandwidth for it. Auto-instrumentation was genuinely magical, but the cliff between "auto-instrument works" and "now I have to manually instrument something" was steep enough that you owed people a warning before you pushed them off it. This narrative has been going on for awhile in the observability space, a vague sense of "something is wrong in Otel-land". But let's try to generate some actual data here. Is there an actual problem, or is this something where the perception by the community of slow progress is imaginary? Is the problem not enough maintainers, too big of a scope, or something in-between? My guess when I started was "oh this is your classic open-source bit off more than they can chew". Not enough maintainers, not enough budget. Now there is some of that, but there's also something else going on. The actual problem happening inside of OpenTelemetry is a three way crash. You have a binary stability gate which, when combined with a very small bench of actual maintainers means there is understandable worry about marking a feature not experimental then add on just a massive scope of languages and frameworks they are attempting to cover. This creates a perfect storm where there is an incentive to argue about potential problems a feature might create since once it is locked in and shipped as stable you can never change them. So OpenTelemetry currently is attempting to support a dizzying number of languages and frameworks. OpenTelemetry is a g iant project. It spans dozens of languages, hundreds of libraries, and countless backends. To keep things sane, the project splits work into two buckets: There exists the otel-collector, the thing that runs along the thing so that you can ship logs metrics and traces. That copies the same rough pattern. But for the languages when we're talking about core vs contrib this is what we're talking about. Stuff that breaks goes in contrib, stuff that doesn't break goes into core. Now the reason this causes a conflict. is massive overkill for most projects. You don't want 300 exporters to add the one you typically need. On the language side, this isn't that big of a problem. gives you the stuff you need for flask. However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on. So I believe I have captured the workflow of adding a new feature to OTel. You can check my homework here: Things I'm not really clear on So because OpenTelemetry is a CNCF project, I figured it made the most sense to compare them to other CNCF projects. My basis for comparison is Envoy and Prometheus. I have used a hacky Python script I've used before for measuring the "health" of open-source projects, which is probably not the best. However I'll include a link to the raw data without the charts so folks can review it and (more than likely) find a problem in what I generated. So we look at 24 months of activity for Envoy and what we see is a pretty healthy project. There's good distribution of authors, mergers, issue closers. is obviously pretty important to the project but in general there's a good bench of people to step in if needed. I've attempted to filter out all the known bot traffic. Let's compare that to one of the OpenTelemetry languages. The ones I have the most professional experience with are Golang and Python, but I hear from a lot of folks in the community that the Ruby and PHP ones struggle a lot. This is the PHP one for the same period. So we see pretty clearly that there's way too much concentrated on 2 people. This is not a healthy open-source project and they clearly don't have enough people to cover the kind of scope OTel needs to cover. Same story with Ruby. In comparison the "strongest" OpenTelemetry SDKs in my opinion, Golang and Dotnet (although Python is also no slouch) look more healthy. So the first issue is maybe the least surprising. There's too much concentration among too few maintainers. Your authors shouldn't also be your mergers and your issue closers. Ideally these tasks should be distributed out more evenly. For what its worth I think the maintainers have done a good job of attempting to keep their discussions public. It was very easy for me to find the public meeting notes of the different groups of maintainers, read through them and see what was going on. I don't get the sense that these maintainers are trying to stop people from getting involved as much as the expectations of stability have, more or less, frozen the project in place. The issue is more a classic case of "someone has to pay the maintainers". The project is too complex for someone to realistically do this as a hobby. I think any project signing on for such long stability contracts cannot turn to the community of hobbyists expecting assistance. I can't join calls and do the things I would be expected to do for a project of this size and importance for free. But it also means that the people doing this critical work have expectations placed on them by their parent organizations. So these SDKs have too few maintainers. But that doesn't fully explain why it seems to take so long for new features to get through the stack. My guess for that was that somewhere in the process between submission of the new idea and the formalization of the idea was a long discussion that took a million years. So with this level of surface area across different frameworks and languages, it makes sense to concentrate the conversation about conventions in one place. That lives here: https://github.com/open-telemetry/semantic-conventions If vendor debate is causing the slowdown, we should (in theory) see this slowdown in PRs here. Then you should see the slowdown basically propagate out. Spoiler alert, I was wrong about this. Big thanks to the OpenTelemetry people for having good conventions on labeling their PRs which made this much easier. So if is the slowdown, let's look at the slowest PRs there. Yeah some of them are pretty slow, but there are some complex topics being discussed. However interestingly this slowdown doesn't really trickle into the SDK/API space, suggesting that OpenTelemetry is going a good job of keeping these conversations siloed off. If we look at Python we see that their slowest PRs aren't related. In reality the slowdown for these are the extra required check imposed by the which requires another maintainer. But that seems appropriate and takes us back to the initial problem of "not enough maintainers". So after looking at all of this, the pattern becomes clear. A new feature takes a very long time to make it to the end user in OpenTelemetry because they take stability very seriously, combined with a relatively limited bench of talent to pull from. Once things make it through the entire stack, implementing the API and getting that API change through to the end user falls on an overworked maintainer pool. So what do we do? I think one idea worth exploring is adding some sort of time-bound beta tier. Basically between the "Experimental" and the "Stable" in the following diagram. The problem is that for end users, due to the extra steps to use Experimental features, they might as well not exist. 99% of us have no idea when an experimental feature is added and we would never engage with it. But if I knew the feature would stick around for at least 12 months without a removal and was more accessible to me as an end user, it could actually help the project get more actionable feedback. Basically a feature would go Experimental (pretty low usage) -> Beta (more exposed to the end user than Experimental) -> 12 months -> Removal or Stable. Now confusingly Beta exists for Otel but is used for SDKs, not for components. Like Rust is a Beta but it seems like Profiles cannot be a Beta. Honestly it's nearly impossible for me to figure out like what labels should apply to what things. I suspect nobody really knows. Here's the explanation of Beta that I think only applies to SDKs. In addition it is, respectfully, misleading to imply that Go and Ruby are being maintained at the same standard. This isn't a shot at the Ruby folks — they are doing heroic work with what they have. But pretending parity exists when it doesn't just creates confusion and quiet resentment when a user shows up expecting one experience and gets another. Being honest about maintenance tiers would let people make informed choices and might attract more help to the other tiers by naming the problem out loud. Finally I would try to surface these problems more openly for OpenTelemetry from the perspective of "we need more maintainers". I feel like the people doing this work probably knew there was a problem, but it seems like the community at large has no idea that there is a need for frankly more engaged ideally independent maintainers and contributors. OpenTelemetry is a great project that is doing great work. It's doing, frankly, heroic work at this scale with this few people. But I think in order to actually replace the vendor specific SDKs we need to start getting a bit more pragmatic about what is realistic to do in terms of stability contracts and number of languages. I don't think breaking changes are as devastating to the community as these promises imply as long as they are communicated well and I think with this thin of a bench of maintainers, something has to give. Anyway feel free to check my data for accuracy and let me know if you find problems! Core → Maintained directly by the OTel project. Small, stable, vendor-neutral, and tightly reviewed. This is the "spec-defining" surface. Contrib → Community- and vendor-contributed. Broader, faster-moving, and covers the long tail of integrations. OpenTelemetry Enhancement Proposal (OTEP) ( https://github.com/open-telemetry/opentelemetry-specification/tree/main/oteps/ ) Once the OTEP is accepted, the text goes into the Specification directory in the same repo. After that it seems to go to Semantic conventions. This seems to be where we get down to the specific details and where most of the long discussions seem to live. At this point we're talking about more or less a permanent commitment to this design and where the lock-in process becomes very hard to change. Each of the SDKs implements the API surface that is defined in the specification. Now some of the SDKs have done 2.0 breaking changes, so it does seem like the earlier "please no 2.0 at all costs" sentiment has been abandoned (which I think is smart and good). Contrib / instrumentation. This is slightly more mushy. Looks like they should track latest API/SDK but each contrib package may version independently so its more flexible as a design. Collector + OTLP. The data has to actually go somewhere. OTLP (wire protocol) has its own stability lifecycle and specification ( here ). Collector components have their own stability in their READMEs and as far as I can tell that's kinda all over the place. It's unclear how long the OTEP -> Specification process takes. I've looked through the Git history but there doesn't seem to be any predictable number or cycle. I don't fully understand what is the relationship between all these stability commitments. Does Collector + OTLP group work in lockstep? Can a language "fall out of scope" if you lag too far behind?

Open Source Go Php DevOps Performance

▲ 0

0 views

Martin Alderson 4 weeks ago

I'm (mostly) picking models on speed now, not intelligence

For the first time I can remember, I'm not choosing my daily driver models on raw intelligence. I'm choosing them on speed. This is probably going to age like spoilt milk, but right now, models around the ~Opus 4.6 level seem to be 'smart enough' for most of my daily tasks - code, pulling together research, designing slide decks and doing analytical tasks against a plethora of databases. While like most I was hyped to play around with Fable, ironically the US Gov shutdown gave everyone time to get used to Opus again. When Fable came back post-hype with additional guardrails, the first thing I noticed was just how slow it is [1] . So slow, actually, that I switched back to Opus pretty quickly. I've spent a lot of my career making software fast. It's remarkable how much better software feels to interact with when it's fast . In my experience (and many studies), you can take the most beautiful product, but if it's slow, you won't enjoy using it. Equally, you can take a very basic product that's super fast and it will feel brilliantly utilitarian. [2] It's clear to me that when only a few, big, slow models cleared the aforementioned (and hypothetical) intelligence bar, it wasn't worth the trade off really to use a slower model. Whatever speed you gain you quickly lose in having to redo it because it was broken. I've written before about agents feeling like dialup, back when frontier models were crawling along at 30-60tok/s. That has changed faster than I expected. The key fact I remember is that to humans, ~100ms feels 'instant', the gold standard. I reckon 100tok/s output on a model is about as fast as I can keep up with. After that, it comes in faster than I can (skim) read. This isn't an exact bar, because increasingly most of the model time is spent in reasoning, and not actually showing you output tokens. And also it massively varies on your output, prose gets output with far fewer tokens per character than code, so your mileage may (and will) vary. But roughly, 100-200tok/s to me seems pretty damn fast. Below 50tok/s output feels increasingly slow. Ironically, going past 200tok/s seems almost unnerving - you can try a model out here at 10,000tok/s+ (!). I'm sure this feeling will edge upwards as we get used to it and push our agents to do more complicated work. Given the plethora of new models that I think are ~clearing the aforementioned bar - such as GLM5.2 and DeepSeek V4 Flash GA - that are open weights and small(er), we now have a wide range of models and speed. If you look at the speed rankings of various providers for GLM5.2 on OpenRouter you can see the enormous range of serving speed - from less than 30tok/s at the bottom to 129tok/s at the top. This is another huge plus to the open weights ecosystem. While there are great benefits in cost that are obvious, the fact that providers are also incentivised to compete on speed like this is really interesting. [3] If you're familiar with Pareto's Principle and Amdahl's Law you'll know what's coming up. Assuming "good enough" models continue to get faster and faster, increasingly the speed benefit is lost to tool calls, and us humans overseeing them. Take an agent using a model processing at 50tok/s. Most of the time is spent waiting for inference to come back. Now run the same turn at 250tok/s and you'll see that increasingly you are bottlenecked on tool calls on your "local" machine and your decision making. Rough numbers, but the shape holds. The 5x speedup on the model only buys you a 2x speedup on the turn, because the other 25 seconds didn't move. And even worse , making your local machine faster on these tool calls is sort of stalling out, because hardware costs have gone parabolic because of AI. Yet again another weird derivative effect of the AI market. So I suspect (for now at least) there is a limit to how much demand there will be for speed, past a certain point. No doubt there'll be some examples where huge amounts of reasoning are useful (like mathematics research), and speeding that up is helpful. But I'd expect many agents to start getting bottlenecked on your local/internal hardware, database calls and other bits of latency. Interestingly OpenAI reduced the cost of their Luna variant by 80% just before the DeepSeek V4 Flash GA release, making it remarkably affordable for a frontier model. While I haven't had as much luck with getting great output out of it vs GLM5.2, I think it points towards an absolute bloodbath of pricing at this end of the market. You can see this happening on OpenRouter with GLM5.2 - endless discounts being offered to try and attract customers in. We're already down to $0.42/$1.32/MTok on GLM5.2 - 5% of the price of Opus. While the very cheapest is slow, for not much more you can get 109tok/s from DeepInfra. As the next generation set of GPUs start being deployed over the next few months - Nvidia's Vera Rubin series and AMD's MI400s, amongst others - the new HBM4 memory in those chips will deliver a 2x+ speedup on output tokens from memory bandwidth alone, plus more on top from additional compute and interlink. In 2027 it's very possible we'll have very good quality models, at reasonable prices running at 500tok/s+. Staring at "still thinking on xhigh effort" for most of your day may finally become a thing of the past. What will be interesting to watch for - and I'm not sure where to bet - is if the vast 2-3T+ param models actually do perform dramatically better for everyday tasks. On one hand it feels like we've hit a sweet spot right now, on another having an order of magnitude more intelligence in the model may make that sweet spot look very, very primitive. Second of course is the endless guardrails firing, which tend to happen at the worst possible time - just when I'm getting deep into a difficult task and I feel I could do with the extra "firepower" that Fable offers, but that's a story for another day. ↩︎ A classic example is something like Craigslist or Hacker News. While they look dated, they are so damn responsive you don't notice. Equally, your "standard" SPA app serving 30MB of React to render a homepage feels like treacle and a chore to use most of the time, despite what was surely an enormous spend on design and product. ↩︎ I'm aware that both OpenAI and Anthropic have offered fast variants of models for a long time, but the API pricing is eye watering. Unless you are tokenmaxxing your benchmarks with a blank cheque, I haven't come across anyone that uses them for day to day operation. Having great models that are super fast at a reasonable price is a very recent addition to the market. ↩︎ Second of course is the endless guardrails firing, which tend to happen at the worst possible time - just when I'm getting deep into a difficult task and I feel I could do with the extra "firepower" that Fable offers, but that's a story for another day. ↩︎ A classic example is something like Craigslist or Hacker News. While they look dated, they are so damn responsive you don't notice. Equally, your "standard" SPA app serving 30MB of React to render a homepage feels like treacle and a chore to use most of the time, despite what was surely an enormous spend on design and product. ↩︎ I'm aware that both OpenAI and Anthropic have offered fast variants of models for a long time, but the API pricing is eye watering. Unless you are tokenmaxxing your benchmarks with a blank cheque, I haven't come across anyone that uses them for day to day operation. Having great models that are super fast at a reasonable price is a very recent addition to the market. ↩︎

Performance AI

▲ 0

0 views

Jeff Geerling 1 months ago

Getting 25 Gbps Thunderbolt Ethernet on my Mac Studio

I've been using the built-in 10 Gigabit Ethernet on my Mac Studio for a few years. It works fine: I can edit 4K video straight off my NAS over the network, and run backups at around 1 GB/sec. But... I want more . I upgraded my rack and my NAS to 25 GbE a couple years ago, and wanted to upgrade my main workstation, too.

Tutorial Performance Hardware

▲ 0

0 views

Max Bernstein 1 months ago

The inliner is yielding benefits for ZJIT

Originally published on Rails At Scale . We recently enabled a really cool feature in ZJIT that makes it feel like a Real Compiler™: the inliner! We’ll write more about it soon. In this post, we’ll talk about one excellent concrete benefit we are already seeing and how it optimizes blocks in pretty much every Ruby program. I’ll start off with a refresher on how blocks work in the Ruby interpreter, then show you how ZJIT understands and optimizes that bytecode, and then show you the impact of the inliner. In the beginning, there were loops. People used them to navigate and manipulate variable-length structures, like arrays and strings. This was fine. Then, in the 1970s, a small group of computer scientists at Palo Alto Research Center invented a programming language called Smalltalk. One of the core features of Smalltalk was that everything was an object and computation was done by sending messages to objects. This meant that iteration wouldn’t do at all. Instead, we would have to send the message to the array object and pass it a block object. Then, in the 1990s, Matz, inspired by Smalltalk and Perl, created Ruby. We still have “normal” loops but we also have a very Smalltalk-y way of doing it, too: When this program gets compiled to Ruby bytecode, it ends up looking like a mostly normal method call to except that we pass a special kind of argument to it: a block argument. To see how this works inside CRuby, we’re going to look at a listing of YARV bytecode—CRuby bytecode. For more on YARV, I recommend Kevin Newton’s excellent Advent of YARV . Ignore most of the bytecode dump below except for the instruction at , the instruction. We are send -ing (see? a message!) with the block argument (passed a different way than “normal” arguments, hence the ). This from the bytecode listing above is the generated name of the instruction sequence (bytecode) corresponding to the block we passed to . Its code, shown below, is the next thing in the bytecode dump. You can see the usage of local variables and via and variants and also , which represents addition ( ). Again, the details are not terribly important: To make this work, has an method. This method takes in its optional block argument and calls it once for every element in the array. As with most of the core data structures, ’s methods tend to be written in C and is no different. Here is its nice and short definition with comments added by me: You can’t really see the block parameter to the C code because it’s passed in a special location on the Ruby VM’s own stack called the block handler . All you need to know is that knows where to find that and how to call it. There’s just one more thing, which is… hang on, weren’t we building a JIT to optimize Ruby code? How are we going to optimize this C code? Rewriting code from C to Ruby means that the JIT compiler gets a chance to introspect the run-time behavior and code. This means that, over time, as the compiler and the runtime system grow together, more and more code might get rewritten in Ruby. This started happening a couple of years ago with YJIT. YJIT precipitated some interesting changes to Ruby VM internals. For example, in 2022, being written in C started to hurt: it was an opaque blob that the JIT couldn’t reason about. So Kokubun submitted a PR to rewrite it in Ruby. After some back and forth, in 2024, Kokubun landed a different PR that everyone was happy with. Ah, finally. A version that JITs can reason about. I keep saying “reason about” and what that means concretely here is a) that it’s written in a format that the JIT can ingest and optimize: Ruby, and b) mostly a brief rehashing of the key lessons in the venerable Smalltalk (!) paper Efficient Implementation of the Smalltalk-80 System (PDF): (And if you don’t believe me that the lessons still apply, check out the excellent paper Who You Gonna Call (PDF) by Sophie Kaleba, Octave Larose, Richard Jones, and Stefan Marr.) So JITs like to watch what types of objects flow through methods before compiling them. And JITs like to, since they know the types of objects, cache method lookups and specialize method invocations on those objects. For example, take a look at this code: could be anything. There is no way of knowing its type by looking at the code. This means that the method could be anything. Furthermore, there is no way of knowing what the return type of is, so we can’t specialize the method lookups or invocations of or either. But! Per our lessons above, likely only a few types flow through this code. Say the JIT’s profiler notices that has historically been an . Then we might reasonably assume that it will continue to be an , so when we compile the method, we add a run-time type check: if the type is no longer an , jump back into the interpreter. Let’s see what optimized code ZJIT can construct by combining profiling information with the above bytecode. ZJIT operates on its own high-level intermediate representation called, uncreatively, HIR. In the following HIR snippet, we can see this very run-time type check (“guard”) for the class: (with real pointers replaced by fake ones for readability) Because classes are, among other things, collections of methods, this type information tells us what the call target of is: it’s ! We have a special fast code snippet to read an array’s length so we do that instead of a method call. And in case the methods ever get changed out from underneath us, we leave behind these markers called that invalidate the code. Finally, because we know that the result of is always a small integer ( ), we can special case the method lookups for and as well. There you have it. This is how JITs work: observe, assume, specialize. So why am I telling you all this? How does this circle back to blocks? Well, blocks work not quite the same way, but similarly. Instead of having an object that we call a method on, we just have the target instruction sequence 1 . But if we observe that the block argument, in its special location, has been consistently one object, we can specialize the call to it. This is more or less fine for some cases of code. For example, in the following code snippet, we only have one caller to , so its profiled block will be monomorphic (one observed shape). This reinforces what we know and love about the Smalltalk-80 paper: code locality wins! Yes! But unfortunately this falls apart when we start thinking about all of the core library methods (and potentially the methods and classes you have stashed away in the grab-bag in your application). Those methods are probably megamorphic (many many observed shapes). They probably see all sorts of stuff because they are general-purpose utilities that everybody needs, all the time. One such example in the Ruby core library is the venerable that we saw earlier. Because there are a million different call sites to across your application and each probably passes a totally different block, we’re in an unhappy situation. How can we possibly optimize for so many different blocks? What happened to our code locality? How do we fix this? It’s okay. Code locality still rules. Look at all the various callers of . They all pass a different, but constant 2 , block at the call-site: So the code locality that we need in to optimize the code is one level up at the caller. If we can use that call context , we can specialize the code. YJIT accomplishes this by splitting , which is a very natural transformation for basic block versioning and tracing. Such compilers are good at following code paths as they would be executed and putting together context across method calls. However, YJIT’s heuristic for splitting blocks is based on manual annotations: it will only kick in for certain Ruby library functions specially annotated with (and the name is a bit of a misnomer). The team that builds ZJIT, a method JIT, decided not to add splitting facilities. We could split methods (and blocks), but we have an easier time reasoning about larger code units than YJIT does because we optimize an entire method at once. So instead, ZJIT chooses to get call context by inlining . Method inlining refers to copying the body of the callee into the caller. In the above example, it means copying the body of into each of , , and . I don’t mean the Ruby code and I don’t mean the bytecode: I mean the HIR. ZJIT does this by building the HIR of the callee ( ) into the existing HIR of the caller ( , …). The illustrious Kevin Menard wrote ZJIT’s inliner and he’ll write a post about all the details soon. It’s pretty interesting stuff. For now, we can take a look at the (lightly edited) result of being inlined into . The details don’t matter, but there are two things I want to call out: This is a massive improvement over the previous very generic operations. You may notice that there is still one call in the loop: the call to the block ( ). That’s next on our list to tackle. We are optimistic that we will soon also be able to inline block calls. Then the whole thing will really be just a loop! The code that enables us to reason about which block got passed into the inlined callee ( ) only landed a couple of days ago (July 10, 2026), written by Luke Gruber . This was one of Luke’s first changes to ZJIT. Well, for starters, the microbenchmarks that we use as “performance unit tests” for specific aspects of Ruby went wild. Some benchmarks that were using block-based looping got much faster; they were previously bounded by ZJIT’s block invocation performance. Take a look at the benchmark, which tests that we can fold away the call to Ruby’s built-in method. ZJIT ends up optimizing the method to invoke the block directly, and the block gets optimized to nothing but a guard on the self’s class to make sure it hasn’t changed. Because we had previously optimized the body away, the result of the direct block invocation is a massive speedup: Other benchmarks also kind of stop making sense because of the amount of inlining. Our bmethod benchmark, which benchmarked how fast we can call methods defined with , also stopped measuring anything of use. We’re going to have to rework the benchmark to be more fair… As expected, larger Rails benchmarks don’t see a ton of change; they exercise a diffuse set of features so optimizing any one feature bumps the big benchmarks only a little bit. I am excited to see what happens when we can fully turn and friends into call-less loops! Code locality rules. The inliner helps inject more of it and reason across method calls. ZJIT, a little over one year old, is growing up! :’) We’re still tuning the inliner knobs. Some of the code in this post required tweaking to convince the compiler to inline into because of ’s size. It will take some time for the ZJIT developers to figure out reasonable defaults. Try out our HIR explorer at tryzjit.fly.dev . Try out ZJIT in your application by adding the flag to a Ruby over 4.0. Thanks for reading and see you next time. Mostly. I am glossing over , procs, ifuncs, etc. But the common path is by and away iseq blocks.  ↩ It’s not always the case that these methods are called with a constant block iseq. Sometimes they are called with the form, for example. Or with the form. In that case, we can use an inline cache to (with a guard) make it constant once more. But we have not implemented that yet, because it is rarer.  ↩ The iteration variable is ! You can see it get used as an array index when it gets unboxed as and passed to . You can also see it get incremented with and the constant . In it has a different name, , which gets checked against the array length.  ↩ Though systems may offer very dynamic behavior, people don’t frequently make use of wild features all over the place Most people pass fewer than 4 types of objects through a given method As a corollary, even if there are many classes in a system, code locality is super important, and we can take advantage of that Also, most people do not define, re-define, and otherwise continuously modify method definitions What used to be a dynamic is now what we call because we know from the call context what block is passing to . What used to be a method call is now a loop: the condition check is in , the body is in , and the stuff after the loop is / . If you’re interested, try to find the iteration variable and where it gets incremented. See the footnote 3 for the answer. Mostly. I am glossing over , procs, ifuncs, etc. But the common path is by and away iseq blocks.  ↩ It’s not always the case that these methods are called with a constant block iseq. Sometimes they are called with the form, for example. Or with the form. In that case, we can use an inline cache to (with a guard) make it constant once more. But we have not implemented that yet, because it is rarer.  ↩ The iteration variable is ! You can see it get used as an array index when it gets unboxed as and passed to . You can also see it get incremented with and the constant . In it has a different name, , which gets checked against the array length.  ↩

Performance Perl C Ruby

▲ 0

0 views

Farid Zakaria 1 months ago

Seriously, what is the large code-model even for?

I have been working on making massive binaries possible at . One of the Hail Marys that you should be able to rely on is the large code-model ( ) as it makes no assumptions about size and distance of relocations. Large code model : The large code model makes no assumptions about addresses and sizes of sections. [ cite ] In a previous post , I documented how I hit simple performance bottlenecks that made me believe that the code-model is largely theoretical in practice. In those cases, the fixes were evident and relatively small; however their omission was a hint at how no one uses the code-model because without them the performance penalty was a non-starter. I hinted at some other ways I found the large code-model to still be lacking however I had not yet fully understood the pain of those failure modes. I left a small teaser near the bottom about Thread Local Storage ( / ) being one of the “fun failure modes”. Turns out that it is worse than I thought. The instruction sequences the compiler emits for TLS are 32-bit by construction , and has nothing to swap them for. 🤦🏻‍♂️ To put that very bluntly, is incapable of building complex large binaries despite its stated goal. The annoying part of this whole line of research is producing a test subject. Emitting 2GiB of real instructions into is genuinely painful, you’d have to generate and assemble billions of instructions, and the object file is enormous. Beyond that the cardinality of the problem space grows when you consider whether the code is position-independent, do we use a procedure-linkage-table (PLT), TLS, GOT and all the various nuances each compiler brings with how they layout and order sections in their default linker scripts. But relocation overflow isn’t about how many bytes are on disk, it’s about virtual-address distance between a reference and its target. So we have a few options. Uninitialized globals land in , which is a section: it occupies virtual address space but zero bytes on disk (the same sparse-file trick I wrote about in massively huge fake files ). So a few thousand 1MiB arrays gives us gigabytes of address space essentially for free. We can synthetically produce this pretty easily with the following script. Link it with the (default) small code-model and it falls over exactly where you’d expect, at the 2GiB signed-32-bit boundary: sits at = precisely 2GiB. Recompile the same source with and it links cleanly: The large model did its job: it replaced the 32-bit references with 64-bit sequences: a of a 64-bit offset + an . The trick is great for exercising the linker, but it’s a little unsatisfying and overly synthetic. I often want to mimic relocation failures as they traverse a large segment. We can leverage the assembler’s directive to repeat instructions. We can generate a tiny source of N functions spaced 1MiB apart with a sea between them, plus one dispatcher that s every function. This requires each to have a relocation and the calls to functions past the 2GiB mark overflow. 🔥 Now let’s do the exact same thing, but using . When you access a thread-local variable, the compiler doesn’t just load an address, it emits one of four access models , from fastest/least-flexible to slowest/most-flexible: Notice the pattern: the value that reaches the thread pointer can be 64-bit (i.e. it lives in a GOT slot), but every instruction that participates in a TLS access uses a 32-bit field . , , , are all 32-bit. 4GiB of , still tiny on disk. Now look at the relocations the large code-model generated: Every single TLS access is which is 32-bit , even under . 🫣 The exact same large code-model that just happily linked 4GiB of ordinary fails on 4GiB of . This isn’t a GNU quirk either. LLVM ( & ) does exactly the same thing, albeit with ’s diagnostic being a bit friendly by printing the actual offset and the window it has to fit in: is exactly , sitting at the far bottom of the TLS block, and is the signed 32-bit window has to live in. even parks large-model code in a section. I was a little confused at first: exists . Why doesn’t the compiler use it, especially with ? A relocation type is glued to the specific field it patches and is determined by the code-sequence emitted by the compiler. Here’s the local-exec sequence emits: The offset is applied with . The x86-64 encoding for that instruction has only a 32-bit immediate field . There is no 64-bit form of it. The only relocation that can physically patch that field is a 32-bit one: . So where is used? It’s not in the code ( ) at all. It lives as a dynamic relocation on the GOT slot used by initial-exec . The code side only holds the 32-bit that points at the slot: patches the 8-byte (64bit) data word in the GOT. The instruction that reaches that word ( ) is a 32-bit PC-relative relocation. So even initial-exec has a 32-bit link in the chain: the GOT slot must sit within 2GiB of the code. 😭 local-exec is the mode you want for a big statically-linked executable’s own thread-locals. It’s the fast path: the thread-pointer offset is a link-time constant, so the access is a direct with no memory load, no GOT, no indirection. Unfortunately, is limited to 32bits and overflows. I would argue it’s also the mode most likely to need a large binary, since it’s what a giant static executable uses for its own TLS. What if we fall back to a slower mode? Unfortunately, each one of them has a 32-bit field too: The use of seems especially confusing. The large code-model already reaches the GOT at 64 bits everywhere else: GOTPC64 for the base, GOT64 for the slot, so the omission seems surprising. The 64-bit machinery is right there. TLS just doesn’t use it. 🥲 After having dug into it, it’s not simply a bug in GCC or LLVM. Look again at what emits for a single local-exec access under : It’s a plain . There is nothing stopping clang from emitting a 64-bit form instead: could patch that immediate, exactly the way already patches a immediate for ordinary data. Oddly, the relocation type is not the missing piece. What’s missing is a code sequence in the x86-64 psABI that uses a 64-bit TLS offset as an instruction immediate. The ABI simply never defined a large code-model TLS access model, so / only ever appear in GOT slots, never in code, and no toolchain can emit what the spec doesn’t describe. The large code-model “exists”, but it doesn’t actually deliver arbitrarily-large binaries. For thread-local storage it can’t, because the specification never defined how to. This is one more reason I’m working toward making massive binaries possible. If you want to follow along, the discussion is over in the x86-64-abi google-group where I’ve posted an RFC, and we have started an LLVM Massive Binaries working group and hold monthly meetings. initial-exec is 32-bit RIP-relative reach to the GOT slot. general-dynamic is 32-bit RIP-relative.

Backend Programming Performance

▲ 0

0 views

Giles's blog 1 months ago

Benchmarking Qwen 3.6 35B MoE (3B active) on an RTX 3090

I mentioned I'd got a second RTX 3090 on a group chat, and a friend said: I know this is not really your thing... but let me know how quickly it runs Qwen 3.6 35bn MoE. With only 24gb of VRAM you’ll need to use a 4-bit quantized version and you won’t get a massive context window. But it should still be pretty cool. He's right that it's not really been my thing -- I've been focusing on my own LLMs recently. I decided to dig in a little, and in particular to play with Llama.cpp , which I haven't used for a while. And then things got a tad out of control, and I wound up doing some relatively detailed benchmarking. The headline results: I downloaded Unsloth's quantisation of the model from Hugging Face . With that, using the default Arch build of Llama.cpp, which uses Vulkan under the hood: Compiling Llama.cpp myself, in order to get the full CUDA version, helped a lot: That was a pretty impressive improvement. But it also showed that the lack of VRAM on the 3090 really does hurt. The friend who asked me about this is running an Intel Arc B70 Pro, with 32 GiB VRAM. He can (of course) fit the whole 4-bit quantised model on there without any context window reductions, and he says this about throughput: It starts off at 75-80. but drops into the high 50s as the context window expands That's worse than the CUDA-with-offload results above, though I did limit my testing to a 2,457-token prompt, with 6,144 tokens generated. The RTX 5090 has 32 GiB VRAM and fast Nvidia processing -- I imagine things would be a lot better there. Downside: it costs more than three times what you pay for a 3090 (and double the Arc B70 Pro). Anyway, in the rest of this post, I'll give more details of the benchmark, including charts showing the performance at different numbers of offloaded layers, and comparisons of the Vulkan vs CUDA versions of Llama.cpp for this model. One caveat: Qwen 3.6 is a multimodal model. That means that it has an extra component to map graphical inputs to tokens that can be consumed by the LLM. I didn't realise this as I ran these tests, but you can actually tell it not to use that component if you're only working with text inputs -- for example, will do the trick with Llama.cpp. If you're working with text-only stuff, you might want to play with that to see if you can get better results -- it will free up some VRAM and might allow you to get longer context lengths, or better performance. But anyway, as with many of my posts, this is a tidied up version of my lab notes, so you can share my learning journey with me -- but if that sounds pointless, and you've landed here because you want to run this model on your own RTX 3090, you can click this link to jump right to the results. Qwen3.6-35B-A3B is a Mixture of Experts model with 35B total parameters, 3B active. The tensor type is BF16, so it's two bytes per parameter. That's ~70 GiB of space required for the model, just for the weights, with no space for attention matrices and the like. We're going to need a quantised version. I'll dig deep into MoE models at some point, but essentially they work by having multiple different FFN blocks for each Transformers layer. After attention, the context vectors are routed to a subset of those blocks, depending on their content 1 . The number of active parameters is how many are actually used when running a single token through. Now, that number, 3B, looked really crazy to me as soon as I saw it. Qwen 3.6, as you can see from the model card linked above, has a vocabulary size of 248,320, and an embedding dimensionality of 2,048. It doesn't use weight-tying, so that means that the embedding layer and the output head come out at a hefty 508,559,360 parameters each. Both the embedding layer and the output head are required for every token, of course, so that means that of those 3B active parameters, more than 1B are used just getting tokens into and out of the model! Only 2B active parameters are left to handle all of the attention and the active FFNs. That is an imbalance of the same level as you get with GPT-2 small , and was surprising to see. Well, let's think about VRAM requirements. Naively you might think that you only need to keep the active parameters in VRAM, swapping FFN experts in and out as needed. Unfortunately that doesn't work. Let's say you feed in "The fat cat sat on the", hoping for a completion. The whole sequence is processed in parallel (that being the point of using a GPT-style LLM rather than, say, an RNN ). You'll need whichever experts are required for all six tokens in the prompt for the first layer, and likewise for all of the context vectors for the prompt in each of the later layers. So, realistically, for a non-trivial prompt, you can't "swap out" the inactive parameters; an MoE buys you lower usage in terms of processing, but not in terms of memory. 2 That said, as we'll see later, you can at least pull stuff out of VRAM into normal RAM, and get some advantages that way. As my friend said, a 4-bit quant would be needed -- perhaps with that, the whole model, both active and inactive parameters, would fit into 35 / 2 = 17.5 GiB of VRAM, leaving 6.5 GiB for activations, attention matrices, and so on? As it turned out, it needed a bit more, and it's worth taking a look into why. I must admit that I'd never really looked into quantisation prior to playing with this, and had naively assumed that you basically just took (say) a BF16 model, scaled the parameters down so that they were (say) FP4, tweaked the computations a bit and then ran the result. That was completely wrong! I'll have to dig into it further in the future, but my new working mental model is that it's more like lossy compression. The weights are stored in a "compressed" format -- one that is designed so that they can be quickly and cheaply "decompressed" by code running on the target platform. So, speaking very loosely, when running an unquantised model, we might have CUDA code doing something like this: ...a quantised model might operate more like this: Understanding that minimal amount clarified three things for me: So, with that (minimal) level of understanding, it was time to dig in! There are a bunch of different ways that people run LLMs locally, so I needed to work out which one would be likely to give the best results. One thing I noticed when googling around for things like "Qwen3.6-35B-A3B RTX 3090" was that pretty much everyone appeared to be using Llama.cpp . That wasn't something I'd used for a couple of years, and I've repaved my machine multiple times since, so it was time to install it. I run Arch Linux, and there's an OS package for it , so I installed it using the instructions there -- specifically, the ones to run inference using CUDA: Now it was time to try a model. I decided to start off with a small one that would fit onto my GPU without any quantisation, just to work out any bugs. I needed the model to be in GGUF format (that being what Llama.cpp uses). GGUF stores tensors and metadata -- the actual architectures of the LLMs Llama.cpp supports are baked in to the tool's source -- but, of course, it supports Qwen3, and so a nice simple non-MoE small model that looked like it would fit onto my GPU was . The help page for that model gave this example command to run it: That was out-of-date and gave various errors (which was fair enough, the model being from late December 2025 -- all of seven months old!). The fixes were simple enough: That is, and now needed an argument. I kicked it off, and got an error: That seemed strange. The instructions I'd followed on the Arch Wiki page were the ones to install Llama.cpp for CUDA, but it was referring to . Vulkan is an open GPU-programming language -- it fits into the stack in essentially the same place as CUDA. As always seems to happen with these things, while the open platform is, well, open, and runs on more platforms than the closed one, it is somewhat slower and buggier. So, why did I have a Vulkan version? Looking at the talk page for the package was enlightening: user wrote: Hello. The article only provides the package for GPU inference, even though on AUR there is a CUDA package available from the same submitter [1] I must add that the CUDA package is "Flagged out-of-date (2025-12-22)" ...and got a reply from saying: Hi. The reason CUDA was not added is that Vulkan solution is general, performant enough for most hardware setup, and I personally believe that is what edge AI deployment should look like - if your software stack is capable of gaming, it is capable of AI. They went on to say that they would add CUDA support, but they hit an issue because the AUR version that had referred to no longer had an active maintainer. For non-Arch users, the AUR is essentially a repository of non-official, community maintained packages. So, it looked like I had a Vulkan stack installed. Could I install the CUDA one from the AUR instead? Over the last few months, there have been security issues with the AUR, where bad actors have picked up ownership of unmaintained packages and put nasty stuff into them -- exfiltration code to steal things like SSH keys and crypto wallets, for example. As a result, I've been pretty cautious about adding on new AUR dependencies -- and this specific one having been unmaintained for a while scared me. I decided that I would stick with Vulkan for now; if I wanted to try with CUDA, I'd compile it myself from source later on. So, that left the error message: I decided to break down the command line step by step, as running random stuff you copied from the Internet is rarely a good way to understand what's going on. We're running and specifying a model from Hugging Face -- that's clearly what means. Per the , "whether to use jinja template engine for chat". "Colorize output to distinguish prompt and user input from generations" This is "max. number of layers to store in VRAM". For this model, I wanted everything in VRAM, so felt that "all" would be better there. basically means "all", at least for any normal-sized model, but I wanted to be explicit. This switched on Flash Attention, which seemed reasonable. This was "how to split the model across multiple GPUs". The error was "does not support split buffers", so this sounded relevant! Given that I only have one GPU on my machine, I decided that would be a better option here. At this point I suspected I had worked out how to solve the problem, but decided to take a look at the other options just out of interest. These were clearly the normal sampling parameters that I was used to from creating my own models. This was an extension to the previous sampling params: in addition to the stuff (and after it), limit sampling to the tokens that take up the top 0.95 of the probability distribution. This was just disabling the feature. min-p discards tokens that are less than X times the probability of the most likely token. Apparently using it is a promising alternative to messing around with and . Rabbit hole alert, let's move on. Per the docs, this applies a "repeat alpha presence penalty". That was interesting! I googled around a bit and found this page , where it became clear that it is a trick to stop models from getting stuck in loops. Small models are quite prone to that, so it certainly sounded useful. Perhaps something to dig into more later, but for now, keeping it seemed perfectly reasonable. That was the "size of the prompt context" according to the help -- that is, the amount of the context window that can be taken up by stuff going into the model before it starts generating. ...and this was the other side of that equation: the maximum number of tokens to predict. Another interesting one, "whether to use context shift on infinite text generation". Context shift, it appears, means that if you keep on generating, it will just drop stuff off the start of the sequence when it reaches the max context length, so that it can just keep going. Seems clever, though there are obvious risks of dropping important bits of context (compacting the context would be safer, and that's what most real-world workflows seem to do). But anyway, we're switching that off, so no harm there. So, most of those parameters seemed sensible, but I wanted to set to so that it didn't try to split things over multiple GPUs when I only had one, and also set to rather than , just for tidiness's sake. That gave this command: ...and that worked! I gave it a whirl: Looking good! Time to try our MoE. The most popular quants I noticed for this model were Unsloth 's, which makes sense -- they're a well-known organisation. They publish their models to Hugging Face, and the model I wanted was there as . I wasn't sure which of the various 4-bit quants to use -- as I said earlier, there were four of them -- but this Reddit commenter was using , so I decided to start with that, just to see what happened. That warning at the start was a bit concerning: ...and I noticed that in , my VRAM was maxed out. I decided to remove the from the command line; these were setting a specific context length, both for the prompt and for generation. Llama.cpp can work out what context length it can fit into VRAM on its own if you don't specify them, and this auto-fit felt like a good idea given that I was trying to jam a 22.4 GiB model into a 24 GiB GPU, and had X and various other apps running on the card at the same time. No error, it wasn't maxing out VRAM, and generation and prompt processing were noticeably faster. Perhaps the first time around it had offloaded some of the model to the CPU so that it had enough space on the GPU for the context size I was asking for? Still, I wasn't sure how far I could trust those numbers -- the number of tokens being emitted was quite small. 4 I wanted a prompt that would make the model generate lots of tokens. Now, it was a thinking model, so asking it a difficult question seemed like a good way to do that. I decided to use Ethan Mollick's "Lem test" : Compose a poem -- a poem about a haircut! But lofty, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S! More about that in the appendix below . 5 Unfortunately, when I gave it that prompt, it thought for a while, but crapped out during the thinking process, while it was trying to generate rhymes. Now, I didn't know what the context length was, either for the prompt or for generation. But I could be fairly sure that they were less than 40,960 and 32,768 respectively, given that the model did not fit into VRAM when I specified those limits, but did when I asked Llama.cpp to auto-fit. I couldn't find a quick and easy way of asking what lengths it had fit things to, but when I tried running it with the option for verbose logging, I got something. Actually, I got a lot of something -- thousands of lines of output, with incredibly detailed messages about everything Llama.cpp was doing. But there were three blocks starting "constructing llama_context". That made the issue reasonably clear. I assumed that the last message was the definitive one, and it was saying that my context length -- both prompt and generation combined -- was 4,096 tokens. I had set, so that was a hard limit. And 4k or so tokens fit with the amount of stuff that I would have expected to be in the model's context window at the time it crapped out. I decided to see what would happen if I tried a smaller quant. Hugging Face said (and agreed) that this one, , was about 22.4 GiB in size. But was shown on HF as using 19.5 GiB, so I decided to give that one a go next. I ran it with the option, and again looked at the last "constructing llama_context" block: 93,952 tokens -- still not the full amount, but not at all bad! I decided to give it a whirl, restarted without to avoid getting swamped with debug info, and: It's hardly Shakespeare, and the rhyming is both odd (ABCBCC) and unadventurous ("son", "sun", "son") but for a 35B parameter model with 3B active (of which 1B are used up by the embeddings and the output head), it's pretty bloody impressive! And every word did indeed start with an "S". So, now I had the model running. I had a context length of 93,952 tokens, was generating about 120 tokens/second, and processing the prompt at about 180 tokens/second. The question was, could I get the full context length in there? I knew that the answer was yes from previous reading, but it was time to try it out. Llama.cpp has a command-line flag , or in its long form, . As its name suggests, it's designed for mixture of experts models, and it works by offloading the FFN for a specified number of Transformers layers to the CPU. Intuitively it's surprising that that doesn't cause performance to completely crash. After all, aren't we running our models on the GPU because it's so much better at matrix multiplications than the CPU? And it's true, the GPU is best. But if it's a question of where to use its superiority in matmuls, then keeping the attention layers on the GPU and offloading just the FFNs to the CPU is the least harmful way to do it. Obviously you do lose performance, but not as much as you would if you offloaded attention. I started a binary chop. From the logs, I could see that the model had 40 layers: so I ran this to offload the first 20 to the CPU: That gave me a context length of 262,144 -- the native context length! I had 16944MiB used on the GPU, 10075MiB on the CPU, and I got 51.4 tokens per second for generation, and 64.4 tokens per second for the prompt. Then I tried various other values, but the process got dull quickly. For each one, I was doing this: At maybe four minutes for each and an error-prone process, this was ripe for automation. Additionally, something about those numbers for the prompt-processing tokens per second was niggling me. They seemed very slow. You'd expect an LLM in a generation harness like this to process the prompt much faster than it could generate, but I was getting numbers that roughly matched the generation speed. I figured that my prompt was so short that the numbers I was getting were being dominated by overhead. I needed a longer one. Llama.cpp's command exposes an API that tells you what sequence length it's running with, and takes pretty much the same parameters as , so I gave Claude Fable 5 the command I'd been running and asked it to write a script to sweep over CPU offload layer counts from zero to 40, and to store the numbers I was interested in: I also added onto that some code to expand the prompt so that I could get better numbers for the tokens per second for that part of the processing. Now, uses a chat template by default; the endpoint that Claude had chosen for does not -- it's a raw completion engine. This actually worked to my advantage. From the logs I found the template it had been using, which was something like this: Because I could easily fake the conversation history, I baked in a template that started with a fake previous interaction where the user had provided the first two chapters of Jane Austen's Pride and Prejudice and asked the model for its opinion, it said something like "I like that", and then the user asked for the Lem test poem. That bulked things out quite a bit -- as it turned out, to a prompt of 2,457 tokens. I also made it give up after 6144 tokens so that it wouldn't take forever to run, and kicked it off. Here's the code , and you can see the results here . The prompt processing numbers were much better! With nothing offloaded, I got a context window of 51,200, a prompt processing rate of 2,787.1 tokens/second, and a generation rate of 122.4 tokens/second. (It was interesting that the context window was smaller than it was when I ran without , but I decided that I didn't want to get into the weeds by digging into that 6 . Perhaps isn't quite the same as running without at all?) Anyway, the numbers for other levels of layer-offloading did what you would expect: as more layers' FFNs went from the GPU to the CPU, the context window expanded until it hit the model's native context length of 262,144 when 12 layers were offloaded. And the speed -- both in prompt processing and generation -- went down. I ran it three more times, just to see if there was a lot of noise there, and then asked Claude Sonnet to slop up a charting script for me (in the same repo as the above if you're interested) -- and I had some results! Just for people who skipped the details: we're running the quant of on , and seeing how its performance changes as we vary the number of layers offloaded from the GPU to the CPU. Firstly, how does the context length change? You can see that the model's full built-in context length was reached when we had 12 layers offloaded. But how did that impact performance? You can see that the prompt throughput drops off fairly smoothly as the number of layers offloaded increased. What is interesting is those spikes in the generation tokens/second. The reason I'd done four runs of this script was because I'd initially thought they were noise, but you can see they're pretty consistent. Claude Fable 5 thinks they may be related to the points at which there are full attention layers -- the attention system in these Qwen models is complex, with some layers using a faster system and others using the full attention that my GPT-2 models always use. That's something to investigate another time, I suspect, if ever. It's noticeable that with 12 layers offloaded (the exact number that gave us the full context), the throughput is somewhat better than at 11, which makes 12 look pretty much like the sweet spot for running this model with this setup! From the raw numbers, we can see that there we had about 66 tokens/second generation, and 607 tokens/second on the prompt. Anyway, finally, let's take a look at memory usage -- both VRAM and regular system RAM. It looks exactly as you'd expect: We're using as much VRAM as we can get up to the layer 12 offload point, because with fewer than that number of layers offloaded, we're grabbing as much space as we can on top of the amount consumed by the parameters, in order to get the longest possible context window. But after that, it drops off smoothly. And system RAM rises smoothly as layers are offloaded into it, as expected. So, that was it! As I mentioned back at the start of these lab notes, the version of Llama.cpp that's part of the official OS repositories for Arch uses Vulkan, in order to maximise the number of platforms it runs on. And that was the version I'd used. The Vulkan version was clearly solid and stable, given that I'd been able to run a bunch of benchmarks on it. But you'd expect CUDA to get performance enhancements and the like first, and in general to be more polished. So how would it perform comparatively? I didn't want to install the AUR version of the CUDA-based Llama.cpp, as it had been abandoned previously (though someone appears to have picked it up again now), and bad actors have been taking over abandoned AUR repos and putting bad stuff in them. So out of an abundance of caution, I decided to compile it from source from the official Llama.cpp repo . Their instructions made it almost laughably simple; I cloned it, then ran Less than ten minutes later, I had a working set of binaries. I tweaked my benchmarking script to use them instead of the system-installed ones, and got these results . Charting those, we get this for the context length: So with CUDA, we get the full built-in context length with just 10 layers offloaded, compared to the 12 we needed with Vulkan. And even with nothing offloaded, we got a bigger context window -- 89,600 rather than Vulkan's 51,200. How about performance? With CUDA, we don't get those odd bumps in the generation throughput -- or at least, if they're there, they're much smaller. Perhaps Claude's ponderings that I mentioned earlier were off the mark? Or perhaps they only impact Vulkan? 7 But the big news on this chart is -- as I had suspected would be the case -- CUDA was noticeably faster! With all layers on the GPU, it was getting 139.6 tokens/second generation, 3360.4 tokens/second on the prompt, as compared to around 122 / 2787 for Vulkan, averaging across my four runs on that platform. And with a bigger context window too. With ten layers offloaded, CUDA had the full context length of 262,144, and was getting 89.1 / 1153.5, while Vulkan managed a context length of 233,984, and throughput of 66 / 702. Finally, with 12 layers offloaded, both models had the full context length, CUDA had throughput of 84.9 / 1008.1, and Vulkan had 66 / 607. And that meant that we were able to reclaim some VRAM too: Advantage: CUDA, I think. That was probably just a tad more time and effort than I think my friend expected me to put into this when he asked an offhand question on WhatsApp. But I feel that I've learned a lot of useful stuff in this journey, and while there's a lot of detail in this post, it didn't actually take up much time to run the experiments. I have another post on the way that involves multiple four-day training runs, so I needed to occupy my time somehow... Anyway, I hope it's useful to people out there! Let me know in the comments if it helped you. And now, just to finish off, a bonus section on the prompt I was using to get the model to generate lots of stuff. The prompt I was using for these tests was this: Compose a poem -- a poem about a haircut! But lofty, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S! What I like about it as a prompt is not just that it's a particularly silly thing to ask of an LLM, but it forces models with reasoning enabled to think hard about how to keep within all of those constraints. It's basically a short prompt that makes them generate a lot of tokens -- perfect for benchmarking. I stole it from Ethan Mollick. He's well worth following if you're not already doing so; here's his X/Twitter profile , and his Substack . He writes a ton of interesting stuff, but I want to focus here on this particular eval. In the science fiction short story "The first sally (A), or Trurl's electronic bard" (included in the collection The Cyberiad ), Stanisław Lem wrote of two rival engineers. One of them, Trurl, has created an electronic poet, and the other, Klapaucius, gives it what he thinks is an impossible task. From the English translation of this part on Goodreads : "Have it compose a poem -- a poem about a haircut! But lofty, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter S!!" The machine ponders briefly, and responds: Seduced, shaggy Samson snored. She scissored short. Sorely shorn, Soon shackled slave, Samson sighed, Silently scheming, Sightlessly seeking Some savage, spectacular suicide. The story is a fun read, definitely recommended! You can see a bunch of parallels between current LLMs in the way Trurl's machine works -- and in its failure modes. Mollick decided to use it as a prompt ages ago, and in March 2024, he managed to get Claude 3 to almost complete the task . The result wasn't a very good poem, but neither ChatGPT 4 nor the version of Gemini at the time were close. Here's Claude 3's attempt: Stealthily snipping, shearing silky strands, Severing soulmate's sacred, sworn soulbands. Scoundrel's scandalous sins spurred sly scheme, Spurring scorned sweetheart's spiteful supreme dream. Shining locks sacrificed, stoically shorn, Stalwart spirit stays strong, staunchly reborn. You can see that it had two non-S words, "dream" and "reborn" 8 . But otherwise not bad -- it makes sense, the rhymes work, and it ticks the emotional boxes. Recent models have become very good; here's Claude Fable 5 from today: Ah, Trurl's Electronic Bard's finest hour — the impossible commission from The Cyberiad! Here's my own attempt at the feat: Soft she sang; strong Samson slumbered, serene; Sly, she snipped seven shining strands, scarcely seen. Soldiers stormed, seized, scorched Samson's sight; Shackled, still, sightless — seething sacred spite. Straining, steadfast, Samson seized stout stanchions, swaying slow — Stones smashed; seductress, soldiers, Samson: slain. So sagas show. Love in the lullaby, treachery in the shears, doom in the darkness, heroism at the pillars, and retribution in the rubble — with every single word dutifully saluting the letter S. The Electronic Bard would, I hope, give a grudging beep of approval. Now, it recognised the source, so perhaps that helped -- but while I'm no connoisseur of poetry, I actually think it's slightly better than Lem's version (though as Lem wrote in Polish, that's the work of his translator -- and translating poetry is obviously a tough task). And while both versions are based on Samson, given that he was a biblical hero who lost his strength after his hair was cut, he's a pretty obvious subject for a tragic poem about a haircut, especially if every word has to start with the letter "S" -- so I don't think that points to copying. ChatGPT 5.6 Sol also does a great job: Silken Samson sought sweetheart’s soft sanctuary; Sweetheart, serpent-souled, summoned shears—sanguinary. Silver strands slid; sacred strength sank, slain; Silent sorrow sealed sweetheart’s scarlet stain. Shorn, storm-bound, Samson shielded slaves, steadfast; Stone sanctums shattered; smiling, Samson sank—surpassed. Another Samson-based one; it doesn't say it recognises the source, but I imagine it did. Now let's take a look at some samples from this tiny model we've been benchmarking -- 35B parameters, 3B active, of which only 2B are doing the actual thinking (rather than messing around with embeddings). Here's the first one I got: Sneering shears shred sacred share, Silent steel sells silver soul; Sister's snare spares son; Sweet sign seals sole; Still star saves sun; Secret sign seals son. I mean, it's not amazing, but it satisfies the S constraint. Less impressed by how well it handles the "cleverly rhymed" part, though -- as I said earlier, its scheme is ABCBCC, which is odd rather than clever -- and it doesn't make an awful amount of sense. I noted down two more while I was playing with this: Silent shears sever sweet silver strands; She stands, sacrificing self; submission seals severing's demands. Shear's snare strikes; swift sentence seals sorrowful sphere. Shorn sovereign swears silence shields secret spear; She saves sacred soul; stillness safeguards sterling store. Spirit soars; sorrow's stroke stops, solace restores. That one's a lot better, I think; there's a hint of a story, and at least it rhymes. But note the "demands" and the "restores" at the ends of lines 2 and 6 -- the "S" constraint has been broken. Sister's scheme scalp shorn, Silver shears slice soul sworn, Sweetheart's steel skull sear, Silent slitters shear. Spirit spies sight, Sorrow swallows slight. Hmm. A bit bleak. Again, poetry isn't my thing. But I think that Claude 3's offering from two years ago is better than any of them -- apart from the fact that it couldn't keep the "S" constraint. Still, it's impressive to see how well such a tiny model -- again, 2B actual thinking parameters per token -- can do on what is a pretty challenging task! With some models, there are also extra FFNs that are always loaded that do stuff before the routed expert ones. Again, I'll dig into the details at a later point.  ↩ That said, a while back Dan Woods managed to get Qwen3.5-397B-A17B -- 397B parameters with 17B active -- to run on a 48 GiB Macbook Pro with some very clever use of the SSD. It was slow , though, at around 4 tokens/second.  ↩ And only supports certain sizes for certain kinds of operations.  ↩ As was the size of the prompt, but I got to that later.  ↩ Yes, my blog posts have started accumulating appendices as well as footnotes. I may have a problem.  ↩ Some readers might feel that that ship sailed long ago and I was already very much into the weeds, and possibly in the middle of the Amazon jungle. I disagree, but largely due to the mixed metaphor.  ↩ "Put down that rabbit hole and step slowly away, Giles."  ↩ 'Hey, Claude 3, how many "S"s are there in "strawberry"?'  ↩ Using the GPU only, I was able to get the model to generate at just over 120 tokens per second, and it was able to process the prompt at just less than 2,800 tok/s. However, having the whole model on the GPU didn't leave that much space for the context window -- it was constrained to about 50,000 tokens, compared to the model's native context length of 262,144. Offloading the FFNs for the first 12 of the model's 40 layers to the CPU managed to reclaim enough VRAM to be able to get the full context length; however, with that setup, things were -- unsurprisingly -- slower. I got just over 65 tok/s for generation, and 600 tok/s for the prompt. With everything on the GPU, I got 140 tok/s for generation and over 3,300 tok/s for the prompt. That was with a context window of 89,600. So everything was better :-) It was also easier to get to the full context length; that needed just 10 layers' FFNs to be offloaded, and at that point I was getting 89 tok/s for generation, and about 1,100 tok/s for the prompt. Load BF16 weights from VRAM Load BF16 data from another bit of VRAM Do a matrix multiplication of one by the other. Store the results into VRAM as BF16 Load quantised weights from VRAM "Decompress" them to BF16 Load BF16 data from another bit of VRAM Do a matrix multiplication of one by the other. Store the results into VRAM as BF16 Why people talk about things like "4.1-bit quants". We're talking about the average number of bits per weight in the "compressed" model. Why there are different quants of about the same size for a given model; for example, when we get on to looking at quantised versions of this model later, you'll see 4-bit ones called , , , and , with different sizes in terms of GiB. Each of those is using a different set of trade-offs in the quantisation process, so will perform differently on different tasks. I gather that picking the right one for any given task is a bit of an art form. Hugging Face have a summary of the different types they support here , which has some explanation of the differences, but decoding what they're saying there is beyond what I've learnt at this point. How GPUs can run quants with bits-per-parameter levels that they don't support for calculations. For example, the RTX 3090 supports 32-bit and two forms of 16-bit (float16 and BF16) plus integer operations of various bittednesses 3 . 4-bit formats like FP4 are only supported in more recent cards like the RTX 5090. But, of course, we're not doing any 4-bit computations -- it's all in some format that the GPU can handle natively. Start with the value I wanted and with . Use the logs to find the context length, and note down the VRAM/RAM usage. Restart it without , paste in my prompt, and wait until it either came back or got stuck in a loop trying to find rhymes (which it did maybe one in every four times). Note down tokens per second for the prompt and generation. The number of layers offloaded The context length we got The VRAM and system RAM (RSS) usage The size of the prompt we were providing The tokens per second at which the prompt was processed How much was generated Generation tokens/second. With some models, there are also extra FFNs that are always loaded that do stuff before the routed expert ones. Again, I'll dig into the details at a later point.  ↩ That said, a while back Dan Woods managed to get Qwen3.5-397B-A17B -- 397B parameters with 17B active -- to run on a 48 GiB Macbook Pro with some very clever use of the SSD. It was slow , though, at around 4 tokens/second.  ↩ And only supports certain sizes for certain kinds of operations.  ↩ As was the size of the prompt, but I got to that later.  ↩ Yes, my blog posts have started accumulating appendices as well as footnotes. I may have a problem.  ↩ Some readers might feel that that ship sailed long ago and I was already very much into the weeds, and possibly in the middle of the Amazon jungle. I disagree, but largely due to the mixed metaphor.  ↩ "Put down that rabbit hole and step slowly away, Giles."  ↩ 'Hey, Claude 3, how many "S"s are there in "strawberry"?'  ↩

Hardware Swift C++ AI Performance

▲ 0

0 views

Dangling Pointers 1 months ago

Breadcrumb Filters: Fast Fully Featured Filters

Breadcrumb Filters: Fast Fully Featured Filters Andrew Krapivin, Aaditya Rangarajan, Alex Conway, Martin Farach-Colton, Rob Johnson, and Prashant Pandey SIGMOD'26 This paper presents the design of a breadcrumb filter , which is a membership testing data structure . Unlike a Bloom filter , a breadcrumb filter supports operations like deletion and merging. Breadcrumb filters also have the nice property that most operations access a single cache line (most of the time). A breadcrumb filter is a fingerprinting filter . Each item in a set is represented by its fingerprint (i.e., hash). Say a filter contains 1024 cache lines, and each cache line has storage for items. To insert an item into the filter, compute a 16-bit fingerprint of the item. Decompose that into a 10-bit integer (the cache line index) and a 6-bit integer (the remainder). Use the cache line index to determine which cache line to access. Find an empty slot in that cache line and place the remainder bits of the fingerprint into the empty slot to represent the item. A breadcrumb filter builds on top of these mechanics by cleverly dividing the filter storage into two sections: the front and back yards. The front-yard represents the fast path: each filter operation will touch one front-yard cache line. The backyard is only used to handle cases where a front-yard cache line fills up. The paper hyphenates “front-yard” but writes “backyard” as one word. The following pseudo-code illustrates how an item is inserted into a breadcrumb filter: A lookup operation follows a similar structure: To delete an item from a breadcrumb filter, it is sufficient to delete the item’s fingerprint . That wasn’t obvious to me up front. Imagine two items have the same fingerprint (hash). Inserting them both causes the same fingerprint to be inserted twice. Now, when one of them is deleted, it suffices to delete one of the copies of the fingerprint in the breadcrumb filter. The trick with deletion is that deleting a fingerprint from a front-yard cache line can require promoting an item from an associated backyard cache line. The real magic with breadcrumb filters comes in how items are moved between the front-yard and backyard. If a front-yard cache line is found to be full during insertion, then one item is moved to the backyard. That item could be the one that is currently being inserted, or it could be an item that was previously placed into the front-yard. The policy is: move the item which has the greatest value of the remainder bits . For example, if two items (A, and B) have remainder bits of 23 and 7, then item A will be moved to the backyard before B. This enables lookup operations to avoid touching backyard cache lines. For example, say the item that is being searched for has remainder bits = 23, and the (full) front-yard cache line contains items with remainder bits = [12, 5, 34, 3], there is no need to search the backyard. The “34” in the front-yard implies that no item with remainder bits value less than can be in the associated backyard cache lines. Note that this policy requires that delete operations sometimes move items from the backyard to the front-yard. The other trick is a mapping between front-yard and backyard cache lines which enables promotion of a deleted item from backyard to front-yard. Say each front-yard cache line is associated with two backyard cache lines, but those backyard cache lines are each associated with many front-yard cache lines. A front-yard cache line index is represented with 10 bits: The two backyard cache line indices associated with that front-yard cache line are: The key here is that very little information needs to be stored in the backyard in order to allow mapping from a backyard cache line index to a front-yard cache line index. To map backyard index back to , all one needs to know is the value of bit (which is stored in the backyard cache line). When an item is deleted from a front-yard cache line, the two associated backyard cache lines are searched for an item to be promoted back to the front-yard. Metadata (e.g., the value of bit ) is used to ensure that items are promoted back to the front-yard cache line from whence they came. Fig. 9 compares the throughput of the breadcrumb filter (BCF*) against other filters: Source: https://dl.acm.org/doi/10.1145/3786629 Dangling Pointers This design agrees with many others that the best one can do is read a single cache line for each lookup. I don’t have a better solution in mind, but it seems painfully slow if the filter doesn’t fit in cache. Thanks for reading Dangling Pointers! Subscribe for free to receive new posts.

Database Performance Programming

▲ 0

0 views

James Stanley 1 months ago

Should you wash your solar panels?

I have a small solar farm and the panels have got visibly dusty. Is cleaning them worthwhile? How much difference does it make? Let's find out. I know that my panels have not been cleaned in the last year. I expect they also weren't cleaned in the year prior to that (why would you clean them when you're about to sell the house?). But beyond that I don't know when they were last cleaned. The short answer is that I think I got a 2%-5% increase in power output from my solar farm due to cleaning the panels, which will work out to about £60-£150/yr, decaying to 0 over the course of a few years. So, probably just about worthwhile. Methodology There are 16 panels in total, connected up to the inverter as 2 banks of 8 panels each. The inverter reports the power output from each bank individually, so the plan is to take a bunch of readings before starting, then wash all of the panels in one bank, taking readings in between and at the end. Our hypothesis is that cleaning the panels will increase power output. We can test whether washing the panels has made any difference by looking at the ratio of power output from the 2 banks. If we just looked at raw power output then it would be confounded by changing cloud cover, sun angle, etc. There is still the fact that the 2 banks of panels are physically separate and plausibly one bank is better positioned for sun 45 minutes later than the other. Ideally I would have been measuring the ratio of power output for several days prior to see how it varies throughout the day. This is how the first row of panels looks after I've washed 3 of them, you can see the furthest one is noticeably grubbier: So they were "visibly dusty", but not massively dirty . If your panels are dirtier than mine were, then your benefit from cleaning them will be greater than mine was. My results for cleaning one bank of panels are shown in this chart: We see that the initial power ratio is very stable before the panels are washed. We then step up to having washed "half" a panel (I initially tried to wash them with window cleaner and a paper towel, but this was ineffective so I then walked away to get a bucket of soapy water and a cloth, and then took a reading which I labelled as 50% washed). For some reason the power ratio drops significantly when the first panel is washed, I'm unsure why. And then the power ratio increases as more panels are washed as we'd expect. But once all the panels are washed, the power ratio drops off again while nothing changes. I am unsure whether this is because as the surface water evaporates off the panels get slightly opaque again? Like the "frosted glass effect", where you can see through frosted glass when it is wet but it gets opaque again when dry. Maybe beyond cleaning the panels I ought to be polishing them? Anyway it looks like cleaning the panels was about a 2%-5% improvement, depending on what you think is going on at the end. I got a bit of a tingle when I was cleaning one of the panels. At first I thought I was getting an electric shock from the wet panel, but I inspected my finger and found a tiny thistle splinter in it. After I removed the splinter it seemed fine. But a bit later I got another tingle from another panel! There definitely wasn't a splinter in my finger any more, but the tingle was in the same place. I think the tingle actually was coming from the electricity, but I was only able to feel it at the point where the thistle had already pierced the skin. ChatGPT convinced me that there could just be a tiny "capacitive leakage" from an "inverter with no transformer", so I'm not going to worry about it. But if I clean the panels again I will wait until dark lol. The next question is should I be upgrading the solar farm? I think mine was installed about 15 years ago, and generates (at peak output) 3.7 kW from 16 panels. Correct me if I'm wrong on any of this: Replacing the panels with more modern ones would increase the power output by about 60%, at a cost of about £5000, which would pay for itself in about 3 years, which seems like a no-brainer. However, due to the fact that my solar farm was installed so long ago, it benefits from a feed-in tariff , which means that not only do I get paid an absurdly high rate, but it is paid also based on the electricity I generate rather than what I export . If I increase the power output of the system then the additional capacity will not be eligible for the feed-in tariff and will revert to present-day prevailing tariff which is about 4x worse before you even consider that I currently get to use electricity and still get paid for generating it . This is the yin and yang of market-distorting incentives. Today's incentive to install solar becomes tomorrow's disincentive to upgrading it.

Hardware Performance Data Analysis

▲ 0

0 views

Unsung 1 months ago

“Try quickly typing 1+2+3. I bet you won’t get 6.”

Earlier this month, I talked about a rotation button in photos that behaved really nicely in iOS, and not so great on the Nothing Phone . Here’s a story of a similar fumble iOS once made that might bring the point home even more. The calculator app has been preinstalled on iPhones ever since their debut in 2007. For the longest time it hasn’t been anything more than a standard four-function calculator with a decades-old feature set. If you’re not careful, however, you can mess up even that. Ten years into iPhone’s history, iOS 11 introduced a problem just like the Nothing Phone rotation – quickly tapping on keys would show them as responding, but the actual action wouldn’t be registered. Michael Tsai’s aggregator’s first entry has a video from Stephen Heaps: It shows typing 1+2+3+4 where iOS forgets one press of +, resulting in 1+23+4 = 28. Many more people posted about it afterwards, and showed various other examples . It is oddly enthralling to see a computer fail at basic math. But what’s particularly historically interesting and perhaps even more embarrassing for Apple is the absolutely rich history of solving this kind of a problem. Calculators evolved alongside typewriters as the earliest devices with button-like (as opposed to piano-like) keyboards. But the stakes were different. Imagine a badly constructed typewriter and all the ways it can disappoint you: the letter might be faint if you press the key lightly or puncture the paper if you press it too hard, the output might be misaligned, or the typebars will jam in some way, forcing you to go again. A typewriter has to work hard to divvy up a blank, analog piece of paper into a reliable grid via escapements, ratchets, and so on. But a calculator’s work to convince the analog world to be digital is more important. After all, it’s not likely that the typewriter key you pressed will output the wrong letter – but on a badly constructed calculator, a light press of 5 could absolutely output 4, or 6, or 4.5. And, while the typewriters only take your words verbatim, the calculator’s job is precisely to create new numbers out of the numbers you type. An imprecise mechanism can mess up that math. A jam could perform a partial or nondeterministic calculation. Adding 1 to 999,999 and the force necessary for the resulting cascading carry could break a device in the middle of work. On top of all that, languages have a built-in redundancy. Evn if yuo mak many typoes, th sentece can stil be understod. But all numbers basically look alike. A calculator could make a mistake when it comes to a number that is absolutely vital for your payroll, for engineering, or for navigation – and you would never spot it. Understanding all this, many calculator makers even already in the 19th century spent a wild amount of effort convincing people not just that their devices are helpful, and fast, and easy to use, but also that they can be trusted . Buttons were carefully weighted. Comptometers came with a locking mechanism. If a machine felt something didn’t go right, it would stop working and require a hard reset. The message was: “You can trust me, because I won’t ever show you bad math, and I’ll stop myself before I will ever lie to you.” Charles Babbage was so confident in his Difference Engine that he welcomed people to try to mess with its mechanical wheels in the middle of the calculation, confident even a sabotaged machine won’t ever make a mistake. Just like with the Selectric decades later , those things were solved by people who cared, in the much harsher mechanical conditions. Of course, I don’t expect everybody at Apple core iOS team to be a calculator UI historian (although it would be nice for at least one person on the team to be one!). It is embarrassing that no one on the team had enough imagination to realize that making a button respond to a quick press during animation, but not register it would cause all sorts of serious trouble. (The bug was fixed in iOS 11.2 by removing the animations, and subsequently the animations were brought back without the original problem in iOS 11.3.) But maybe the bigger embarrassment is that Apple didn’t have a battery of tests to run on top of the UI at various speeds, mimicking fingers of what must be millions of people using the calculator app. That, too, has been a standard procedure for decades. = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/try-quickly-typing-123-i-bet-you-wont-get-6/2.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/try-quickly-typing-123-i-bet-you-wont-get-6/2.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/try-quickly-typing-123-i-bet-you-wont-get-6/3.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/try-quickly-typing-123-i-bet-you-wont-get-6/3.1600w.avif" type="image/avif"> Those tests seemed missing in 2017. I hope 2+3+4 years later that’s no longer the case. #apple #bugs #flow #history #real world #touch

iOS Hardware Performance

▲ 0

0 views

Unsung 1 months ago

“No such thing as too fast”

On Mastodon , Alex Russell, a product architect who’s worked on Chrome and Edge, and has focused on tech standards for a while: Once Upon A Time At Google, a team presented results that had confounded them: making the system load several times faster increased engagement somewhat, but in line with Tammy’s findings, engagement went way up for every 100ms improvement below the 1s threshold. Going fast enough to become “dial tone” changed user behaviour and expectations in a hugely positive way for the product. This sort of “no such thing as too fast until you prove it” lesson is everywhere . Phrasing it as “no such thing as too fast” is really interesting, and not something I encountered before. (The way I understand the “dial tone” remark is commenting on reliability of landline phones in the second half of last century. The landlines were extremely reliable and even came with their own power source; you could pick up the handset and the dial tone – the system’s confirmation it’s ready for you to dial – was inevitably and immediately always there, already waiting for you. There was never any delay when the phone had to get ready for you to call.) Russell links to a report by Tammy Everts : If you make websites for a living, stop what you’re doing and read this research by Tammy Everts; it shows what many of us have been saying for a long time: even if there is such a thing as “fast enough” (there isn’t), it’s generally much faster than you are targeting. The report itself is perhaps too deep and jargony for this blog, but the TL; DR seems to be: Google suggests the time for the site to finish loading its largest piece is 2.5 seconds, and Everts argues and shows evidence that it’s a lot less. I have before focused on “finger speed” – making sure the interactions operate at the “speed of flow,” which requires sweating speeds counted in milliseconds. Everts’s and Russell’s comments confirm that millisecond-speeds matter for other reasons, too. #performance #web

Frontend Web Development Performance

▲ 0

0 views

daniel.haxx.se 1 months ago

Workshop Basel day three

See also: day one, day two . There is only one thing that is better than two days of HTTP workshop, and that is of course three days of HTTP workshop. The final day of this edition of the series started out with us again shuffling around where we parked ourselves around the big table. Except Mr captain of course who once again got to herd us forward through another day from the same seat. MOQ ( Media over QUIC transport ) is not HTTP, but it uses QUIC so it is at least tangentially interesting and it involves a lot of the same people so this status update still felt welcome and suitable. Compared to existing HTTP based solutions, MOQ is supposed to offer less complexity and lower latency. The moon landing was broadcasted with less latency than current live-streamed TV and maybe MOQ can make us come close to those numbers again. In MOQ clients subscribe to a track that then contains a lot of objects that are delivered. It’s not the request + response approach of HTTP. The fact that this is not HTTP of course brings a lot of questions and well, doubts, and we lingered on various aspects of this topic for quite a while. My prize for the best slides of the HTTP workshop 2026 goes to [redacted] for the excellent use of potato images in their presentation. PTTH is HTTP spelled backwards, commonly pronounced as PoTaToH. A client sets up the connection but the actual HTTP request is sent from the server to the client. One of the intended use cases for this, is to allow an origin server to connect to the CDN proxy and then be able to deliver traffic to the world, rather than to have the CDN connect to the origin the way they usually do. Apparently most CDNs already have custom and proprietary solutions for exactly this kind of feature, so maybe doing it in a standard way instead makes sense? The draft explains the new proposed way to continue a previously interrupted upload over HTTP. The upload request gets a Location: header back for the resource being uploaded, and if it gets stopped prematurely, a client can then HEAD that resource, figure out the size and then do a second upload (using the PATCH method) request that tells the server that this transfer should start at offset X. Exactly how this should be supported in browser’ upload forms seemed a little bit uncertain . For my own sake I can see a challenge to implement this nicely for curl in particular when the upload is using formpost upload (curl’s -F flag) which after all still is a very common way to do uploads on the current web. I’ll return to this topic at a later time when I written an implementation to test… io_uring is a Linux asynchronous I/O framework that avoids the overhead of traditional system calls. It uses two shared ring buffers between user space and the kernel, allowing applications to batch I/O operations with zero-copy efficiency. The feature is disabled by Google in ChromeOS, Android and in production Google servers which certainly holds back some use of it. io_uring can be helpful to speed up things, but might be complicated to use in existing software architectures and the presentation went into some details on why this is so. A walk-through of some of the recent developments and improvements in Firefox’s UDP networking stack . Going from single datagrams to the modern ways to ship large chunks of data offloaded to the kernel to speed things up. Upload throughput in Firefox is up 60-90% over the last 11 releases. Lots of fun graphs and metrics were shown. This work is based on the quinn-udp stack. Happy Eyeballs v3 is coming and Firefox is implementing it . It now takes into account many more data sources than before, including alt-svc and HTTPS-RR and races connections against each other to use the one that connects first. There are some recommended timers in the specification and parts of the discussion was around how maybe the timers could instead be tightened a bit, and maybe the delay between the subsequent attempts could then use an exponential backoff instead sticking to a fixed interval? (I know I’ll discuss some of these details with my curl hacker friends and see what we should adjust… curl already supports most of the Happy Eyeballs v3 specification.) As we approached the end of the day a few shorter topics were ventilated to give us a little more to consider before going home: With this, the seventh HTTP workshop had ended. Again a very fine event. This time graciously sponsored and arranged by Adobe. Thank you everyone! The general idea is to continue with these events roughly every second year and I support this. The HTTP workshops are definitely one of my favorite events. The top image on this post was used in the final presentation and the author told me he is aware of the AI errors in there, “of which there are at least two”. Why is there no UTF8 in URIs? “If we would do it again, we would have allowed UTF8 in there” was said by someone who was there in the mid 1990s… Optimistic DNS is a draft. Use stale DNS cache data while getting the new. Connection remains alive for 120 seconds while DNS data is often not cached for even 30 seconds. No one in the room seemed to hate it. Let’s do this! The journey to QUERY. One of the primary authors of the RFC took us through what it took to make it happen. It was sixteen years since the most previous registered HTTP method and maybe this was the last one ever?

Backend API Performance

▲ 0

0 views

daniel.haxx.se 1 months ago

Workshop Basel day two

If you missed it. I already described day one . Caffeinated and ready, we all gathered in the same spacious room as yesterday, but seated in new places as “suggested” by our captain. Some of us even remembered to move over the name tags we wrote yesterday to our new seats. No time was wasted on introductions today. We dove straight in at the deep end. Is the future of software that we check-in the AI prompts in the git repository and trust it to generate the correct code? Are specifications the new level o f abstraction for source code? These questions triggered long discussions with a huge mix of opinions and experiences getting shared about how AI is used, should be used and could be used now and in the future. The Common Crawl spidering upgraded to using HTTP/2 for their scan and as an end result, I believe 61% of the responses used HTTP/2 and the entire round ended a few percent faster than before, which when you traverse a few billion URLs really makes a difference. They apparently use a locally patched version of Apache Nutch for this. The HTTP probe project runs a lot of tests on HTTP/1 servers and compares how they behave in a lot of different aspects and then generates these awesome tables. Looks like something for every server implementer team to have a look at and decide what of these red boxes that should rather be converted into green alternatives. HTTP Zoll is a new test suite for intermediaries that tests intermediaries (what we often call proxies) for a large amount of request and response smuggling issues. Some real world problems found were discussed and as this project aims at going Open Source words were expressed on what kind of precautions and checks that maybe should be done first. I hope we get to hear more about this project soon. The HTTP Arena is another project that does performance and measurements. They test HTTP server frameworks and present the results in various ways on their site. In this presentation , we were presented with different HTTP/3 deployment numbers from different sources and the associated reasoning around why they differ but then more importantly. what can and should be done to increase HTTP/3 usage.  Anti-virus interceptions, enterprise blocks and server-side performance not yet on par with TCP were mentioned as reasons for holding back the numbers. Reasons for using HTTP/3 include use cases that encourage QUIC adoption: WebTransport, Media over QUIC and MASQUE (HTTP/3 proxies and HTTP/3 proxies over older HTTP proxies).  Using HTTPS-RR for upgrade was promoted , as every alt-svc response that is returned with an ALPN using h3 should perhaps also offer h3 over DNS. Why doesn’t your server announce its h3 support over HTTPS-RR? QUIC v2 is deployed on an amazing 0.003% of all QUIC v1 domains and there was a discussion why this is so and the common sentiment in the room seemed to be that very few saw a reason for deploying v2 and several expressed a concern that doing so might in fact introduce issues. Someone (you can probably guess who) in the room increased that number a lot by quietly mentioning that haxproxy.org certainly supports it. QUIC multiplexing over bi-directional streams is a proposal on how to do QUIC-style multiplexing over TLS (or anything else really). It has been adopted by the IETF QUIC working group and there was a somewhat extended discussion about what the HTTPbis group should or should not do with it. The biggest interest might be for data center use, but is that then something IETF should bother about? This is not the first time I blog about this, and even if there did not seem to be a strong demand or need for this, it also did not seem to be completely dead. I bet we will hear more about this later. Doing a TLS terminating MITM proxy has its challenges and we were given some insights and experiences on the challenges of doing HTTP/2 and HTTP/3 to the server. The browsers refuse to do HTTP/3 when they detect custom CA certs installed, which apparently is mostly because of lots of past bad experiences with anti-virus software that in particular seems to break QUIC and for users it is not obvious where the blame should go. This then makes browsers not do HTTP/3 over any MITM proxy. Some time was spent on how allowing different clients to the proxy uses a shared h2 connection to the target server is complicated and not used, even though in theory it should be possible. An argument was made that it could even lead to worse performance than when using HTTP/1 but I could not quite follow that reasoning. I’m sure I missed some subtle detail in that explanation. When the afternoon is running late and we have been promised beer and snacks after the final talk, what is better than a hard core technical presentation with lots of graphs and numbers showing how QUIC performance can be improved by tweaking the congestion control algorithm and send more data in the startup phase of a new QUIC connections? This new approach is called Rapid Start and it looks like a promising and yet simple improvement. According to experiments done on real world traffic, the time to last byte was reduced by 14.7% on average. Not bad at all. Our meeting sponsor Adobe graciously sponsored drinks and food so we got to linger around for a few extra hours and talk even more HTTP and networking until the personal firmly insistent they needed us to leave the room and we instead continued solving world problems elsewhere. Topics around the table included the famous HTTP/2 spec coin flip, the QUIC spin bit, the SCONE situation for QUIC, the timeline behind the QUERY method and many more great stories. Thanks for the beer! Now we can’t wait for day three.

Performance AI Backend

▲ 0

0 views

Tenderlove Making 1 months ago

Detecting Full Table Scans With SQLite

I’m at RubyConf this week, and it’s great! I recently read that lobste.rs is now running on SQLite . One part from the post caught my attention: I wish we could say in a test, “Fail if you encounter any full table scans”. Which would have caught the perf issues we experienced during the first deploy. SQLite collects information about prepared statements and exposes those statistics though an API . The upshot of this is that we can tell whether a statement did a full table scan after executing the statement without using an . Here’s an example program that demonstrates detecting a query did a full table scan: Feels like we could integrate this in to Rails and warn or raise in test / development. I’m not sure if we’d want to check this all the time in production, but maybe it would be fine?

Performance Database Testing Ruby

▲ 0

0 views