Dayvster Today

We Put a Browser in Everything and Called It Progress

The hardware got faster. We lowered our expectations.

Performance Web Development

Simon Willison 3 days ago

Hello Python 3.15, goodbye 3.10, and now I can use 3.11

Python 3.15 is here ! Here's the essential What's new in Python 3.15 guide, this year curated by 3.14 and 3.15 release manager Hugo van Kemenade. As someone who builds open source software that works in every supported Python version, I get to say goodbye to Python 3.10 and start embracing features from Python 3.11. First, some highlights from Python 3.15. I'm most excited about Explicit lazy imports from PEP 810 . These let you import modules like so: The module will then be imported the first time those symbols are used in the code that follows. My LLM CLI tool implements plugins. I had to recommend manual lazy imports in the plugin author documentation, because simply installing plugins that used heavy dependencies such as could add multiple seconds to every invocation of the command. Lazy imports directly address this startup cost. The new unpacking syntax is neat. Here's an illustration lifted from the documentation : PEP 798: Unpacking in Comprehensions by Adam Hartz and Erik Demaine proposed this as an extension of the unpacking syntax from PEP 448 , which landed in Python 3.5 back in 2015. Python's existing tracing profiler has been renamed from to profiling.tracing , and has been joined by an exciting new sampling profiler called Tachyon, aka profiling.sampling . Sampling profilers work by periodically capturing stack traces from running processes, then using those to calculate which functions have the most time spent inside of them. They're very low overhead and can provide a deep insight into running systems, without needing to instrument those systems first. I've previously used tools like py-spy for this. Now this capability is baked into the standard library. If you have the PID of an existing Python 3.15 process you can start gathering samples by running this (I had to use on macOS): Interact with the main program a bit, then hit Ctrl+C to stop gathering and see a report, which looks something like this: It can also produce flame graphs like this and heat maps like this , and can even output a format that can be viewed in the Firefox profiler UI. Miguel Grinberg investigated How fast is Python 3.15? the other day against the 3.15.0rc3 release and found performance to be mostly the same as 3.14, with the exception of the experimental JIT. The JIT gave a 1.2x boost over the standard 3.15 interpreter (on a Fibonacci test), the first time Miguel observed a meaningful JIT performance increase. Miguel's explorations of 3.14 last year found significant boosts against Python 3.13 across many more categories. I used the following sequence of commands to get to use Python 3.15 on my Mac. First I had to upgrade itself to ensure it knew about the new release: Then I installed the stable 3.15 release: And told that I wanted that to be the default for every command, unless the project itself pinned a different version: Now the following reports 3.15 as the version: I already have a bunch of tools installed using . I reinstalled them to use Python 3.15 like this: Then I can confirm the version like this: The release of 3.15 means that Python 3.10 (released in 2021) is officially no longer supported . My policy for my main open source projects is to keep them working in every supported Python version. I can now drop 3.10 and start using features that were introduced in 3.11. Here's the What's new in Python 3.11 document. I finally get to rely on tomllib being available! Datasette makes extensive use of . 3.11 adds TaskGroup and timeout() , and several more new features. is a new in 3.11 alias for . Python's SQLite support got some significant upgrades in 3.11, including: Plus a whole bunch of improvements to Python's typing system . I plan to land Python 3.11 improvements to a bunch of my projects over the next few weeks. Here's my first Datasette PR removing that workaround. I've also updated my click-app , datasette-plugin , python-lib , and llm-plugin cookiecutter templates to test against Python 3.11 through Python 3.15. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . setlimit() and getlimit() expose a mechanism for setting SQLite runtime limits from Python. create_window_function() lets us create custom window functions in Python, as a class with , , , and methods. blobopen() provides an efficient mechanism for partial reads and writes of large binary values stored in a SQLite row. You can now call to remove a SQLite authorizer. Datasette had a workaround for this!

Python Performance Programming

Jeff Geerling 6 days ago

Radxa's Q8B has 2x the performance and expansion of the Pi 5

There was a time I'd look at a board like the Radxa Dragon Q8B (at left, above) and be like, "there's no way I'd spend $209 on an SBC with 8 gigs of RAM". But we're in 2026, and seeing the 8 gig Raspberry Pi 5 going for almost the same amount, I figured I'd give it a shot. On paper , the Q8B beats the Pi 5 in pretty much every way. A lot of that is thanks to this Snapdragon 8cx Gen 3 chip, which is the same chip I tested on Microsoft's Windows Dev Kit 2023 .

Performance Electronics Hardware

Jim Nielsen 1 weeks ago

“Getting off the Modernization Treadmill”

My notes from this talk by Alexander Petros at Big Sky DevCon 2026 . Alex starts by noting how “modernize” used to mean something along the lines of “update this thing that was made before I was born”. But now “modernize” means something more like “update this thing from 5-10 years ago” (hence the framing of the talk, the “modernization treadmill”). Using a real-world example of an incredibly slow website that was required to access state-sponsored programs for welfare, Alex points out the disparity in conditions between those of us who make software and those who have to use them: The people who develop these websites are usually doing so on high-powered internet connections and high-powered devices, but they're not using them in the conditions that the people who most need those benefits are going to be. Then he shows a Reddit thread where somebody essentially posted, “I’m having problems with this website. I’ve been waiting for months for my application to go through. Any suggestions?” And one Reddit user responded, “The best thing you can do is go into the physical office, get a case worker, and your problems will be solved within the hour.” I guess we've come full circle now. It used to be: “Don’t talk to anybody. It’s faster and more convenient to use the website!” But now it’s: “Don’t use the website. It’s faster and more convenient to talk to somebody!” Have we failed at making websites? And is our failure, at least in part, rooted in the fact that we don’t leverage the basic tools for making websites: HTML, CSS, and (in a distant third) JavaScript? Alex goes on to argue that the technologies of the web have an ideological bent and, if used as designed, can solve so many of the performance, accessibility, and usability issues that plague so many websites. The grain of the web’s technologies are rooted in these values: Which means if you use them as intended, they are optimized to deliver outcomes rooted in those same values. So if you like those values and you want those outcomes, use the platform. Take HTML, for example. Here’s Alex: HTML does [performance improvements] for you for free. If you've coded your website in a proper, semantical, structure HTML style, it will just get better over time at zero cost to the people who built that website HTML is your friend. HTML won’t give you up or let you down . HTML will make it difficult for you to make a bad website. Write it in to the requirements of the project you’re doing that it work without JavaScript. Not necessarily that it doesn’t have any JavaScript, but just that the core functionality of the website can happen without JavaScript. If you do this […] you will find that it’s very hard to deliver a bad web page because the structure that HTML requires is one that fundamentally is good for the user, performant, and cost effective. Technologies are imbued with culture, which influences what you do and how you do it. If you can align your ideological beliefs with your technological choices, you might end up with an outcome that aligns with your values — who would’ve thought, eh? A lot of modern software developers come from websites like Facebook, they come from big tech companies [who] fundamentally have a different set of priorities. Their job is to keep you on the website as long as possible so that you consume more ads. But that’s the opposite set of requirements and priorities that the government needs to be doing, which is to build something that is clean, quick, efficient, and gets you in and out as fast as possible. So [I tell people] that the technology they use comes from [an] ideological place. But there are different ideological places that produce different technological results and if we start from those through lines then we can produce services that help people who need them. The ideological principles of the web are well established : users over everything else. If you use HTML as much as possible, you’ll make something that’s as user-friendly as possible on the web. Reply via: Email · Mastodon · Bluesky User-friendly Backwards- and forwards-compatibility Long-term viability Universal accessibility

Programming CSS Web Development HTML Performance

Langur Monkey 1 weeks ago

JPEG XL finally lands in Chrome!

Today, I was browsing Hacker News , when a particular item caught my eye: Hacker News post with the Chrome JPEG XL announcement . I opened it promptly and, indeed, it was not a prank. Finally, after years and years of bull**** from the Chrome team, they are shipping it. They are shipping a JPEG XL decoder. For those of you who don’t know, JPEG XL is a new image codec that includes most of, if not all, the features we’d want in the image codec of the future: This is awesome news for the codec . Whether we like it or not, Chrome is the dominant web browser, with ~66.5% of the market share , very far from its competitors (Safari, Edge, and Firefox, in that order). Now, I don’t use Chrome at all, but I still think that this was just the final piece of the puzzle needed to help the format go mainstream. So, what finally pushed them over the edge? If you read their announcement , they cite “consistent feedback and requests from web developers” and its massive popularity in the Interop 2026 project. Translated from corporate PR-speak: the community simply refused to let it go. Web developers, photographers, and open-source advocates kept pushing, complaining on bug trackers, and demanding a truly superior image format instead of settling for Google’s own WebP or the heavily pushed AVIF. The technical excuse the Chrome team used for years was that a C++ decoder was too much of an attack surface to run safely in the browser. Well, it has now been elegantly sidestepped. To bring JPEG XL to Chrome 155, they have integrated , a pure Rust reimplementation of the decoder. This tackles the memory safety issues head-on, eliminating the risk of out-of-bounds reads and use-after-free bugs of other decoders. And to make sure it doesn’t run sluggishly, they built a SIMD abstraction layer ( ) to maximize hardware performance without compromising Rust’s safety guarantees. This is a massive victory. Not just for JPEG XL, but for the open web. As I demonstrated in my comparison back in 2023 ,JXL has significant advantages over older formats and, in many cases, over AVIF as well, especially for high-fidelity photography, lossless compression, and progressive decoding. The fact that it allows lossless transcoding of existing JPEG files is of special importance. It means that we can finally upgrade our backlog of old images without losing a single pixel of quality, while saving a big chunk of disk space and/or bandwidth in the process. With Firefox adding it to Labs earlier this year, and Chrome finally capitulating, Safari is left as the major outlier with only partial support. The dark days of “Google kills JPEG XL” are officially over. It’s time to start converting your image libraries, updating your tags, and serving . The future of (high-quality) web imagery is finally here, and I’ll be ready for it. File size reduction by \(~20-60\%\) w.r.t JPEG. Lossless JPEG transcoding. Progressive decoding. Wide gamut, HDR and 32-bit support. Animations and transparency (alpha channel). Shines with high-fidelity photographic images. Fast-ish encoding and decoding. Royalty-free and FOSS. Support for super high-resolution images, of up to 1 terapixels ( \(2^{30}-1\) pixels per side).

Web Development Rust C++ Performance

miguelgrinberg.com 1 weeks ago

How fast is Python 3.15?

It's October once again, and that means it is time to take the new release of Python for a spin (technically, it is the 3.15.0rc3 release that I'm using, the official 3.15 release is still a few days out). As I did with my Python 3.14 performance article of a year ago, today I'm sharing a new run of my informal Python benchmark, comparing Python 3.15 against previous interpreters all the way back to 3.10. If you are not interested in the charts and the tables and just want to read my analysis, feel free to jump to the conclusions section at the end.

Python Programming Performance

matklad 1 weeks ago

Benchmark In Milliseconds

How long should a micro benchmark run? My rule of thumb is to tweak the input size until the benchmark takes about 300ms, for the following reasons: The imminently-to-be-stated assumption here is that the purpose of benchmarking isn’t so much a precise measurement of performance, but rather providing the author with enough intuition to make a correct decision. Milliseconds are integers ranging from 1 to 999. Enough precision to notice even a small improvement, and easy to scan visually. No need for different units or floating points (compare with ). Anything faster than, say, risks being skewed by fixed costs (e.g, interpreter startup). Hundreds of milliseconds is an eternity for a computer, usually enough to make one-off overheads irrelevant without using fancier (= less robust) techniques to explicitly account for them. For a human, hundreds of milliseconds is fast, but noticeable. Pushing numbers into human-perceptible range allows me to use my intuitive sense of time and speed, it doesn’t rely exclusively on numeracy. It’s plain fun to see, as a result of optimization work, how a previously lagging CLI command becomes “instant”. But anything longer than a second makes iterating on the benchmark slower than it needs to be. Running a benchmark 10 times in a row to eyeball variance should be fast!

Performance Testing Programming

anders murphy 2 weeks ago

Faster streaming HTML with batches and aggregates

Performance HTML Web Development

Unsung 2 weeks ago

A loading state within a loading state

One of the classic parables about loading goes as follows. A building manager had a problem: people were complaining about the elevator being too slow. But upgrading the elevator was an enormous expense, so the manager came with an alternative solution: they put a mirror next to the elevator call buttons. Even though the elevator didn’t become any faster, complaints trickled to a halt, as people were checking themselves out in the mirror and that changed their perception of how long the wait really was. In the late 1970s and early 1980s, it wasn’t uncommon to equip early home computers with audio tape players, using the same cassettes designed to record and play music. Software was encoded as sound. A lot of redundancy was needed to account for many low quality cassettes and misaligned playheads, so loading times were really long. There were other disadvantages – no random access but definitely random errors, tapes wearing out and being “eaten” by the player – but tapes were so much cheaper than floppy disks or cartridges that they endured. = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/1.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/1.1600w.avif" type="image/avif"> On one of the computers, the British ZX Spectrum from 1982, an interesting convention emerged. As the game was arriving from the cassette, what was loaded before the code itself was its “title card.” The loading process of just this one image took about 35 seconds, and watching the graphic emerge was mesmerizing, almost like deciphering a puzzle – especially when, only toward the end, the color attributes appeared and the whole thing clicked into place: Some of these were beautifully done ( here’s a gallery of… all of them ?), created by talented artists working in a very unforgiving medium, a pleasure to watch materialize on the screen… …or, at least, it felt so for the first few encounters. Upon loading the game for the nth time, you grew more and more aware of the fact that you are spending precious time waiting to load a screen you’ve already seen. This was different than the first example; the mirror existing never slowed down the elevator. Sure, it was just 35 seconds out of a 5–20 minute load time – I told you cassettes were slow! – but still. Some people loved them, others built truncated versions that skipped the title screen altogether (as much as you could imagine just fast forwarding through it like you would through a song, it didn’t work that way). Figma is a web design app, and its editor arrives in a big, everchanging blob of JavaScript, in addition to having to load and decode the contents of the file itself. This isn’t 5–20 minutes, but it’s long enough that it necessitated a loading screen of the heavy variety: one with a progress bar. During my time at Figma, an idea would reappear with surprising regularity: what if we put a little hint next to the loading bar? Something that could teach you about a useful keyboard shortcut, or a trick? You’ve seen that before, often in games : = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/4.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/4.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/5.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/5.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/6.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/6.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/7.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/a-loading-state-within-a-loading-state/7.1600w.avif" type="image/avif"> It seemed like a thoughtful gesture for the user, a mirror of the elevator mirror idea. But I fought it, tooth and nail, for one very specific reason: this would remove pressure to make this screen fast. Had we given the screen another purpose, subconsciously or not, we might start caring less about improving it – as a matter of fact, you could make an argument to introduce artificial minimum loading time just so that the user could finish reading the tip! I had a similar feeling when, in 2018, Gmail replaced its simple progress bar loading state with this: It was huge, corporate, intense. It felt like an admission of defeat: our app is slow, and I guess we’re surrendering ourselves to it. But recently, I’ve noticed not just that Gmail’s loading state has been simplified, but also that it doesn’t appear nearly as often: I don’t know the technical details: is it caching? a more intense refactor? Either way, as much as I absolutely dislike what Gmail has become – there might be no more harrowing place in the web app world than Gmail’s settings, for example – kudos to the team for making it better. Gmail seems to be loading a lot faster now. (There is one small exception: it would appear that the loading state image itself doesn’t have a proper loading state – a rather strange omission.) There is, I believe, a more universal lesson in here. Loading is a complex space, where sometimes the most natural solution is a bad one, sometimes making things faster is making things slower , and sometimes new thoughtful design for whatever delay there is can be a better use of engineering effort than focusing on shaving off more milliseconds. (It’s both speed and the perception of speed that matter.) Also, this is exactly what Y2K was : We remember and react to loading states that are elaborate, cute, memorable – but we never see or link to loading states avoided . And while you see occasional stories told by engineers making things faster, you don’t often hear accounts of people within companies, somewhere at the intersection of design and engineering, who work hard designing thoughtful loading states, avoiding loading state bloat, or figuring out some clever hybrid approach. But, to be fair, there is also no parable of a building manager who just forked over some cash and installed a better elevator.

JavaScript Design Performance Culture

Unsung 2 weeks ago

“First, don’t interfere with user input.”

In response to a previous post , one of the readers wrote this: I would phrase this as the fault of the animation having violated the Hippocratic oath for animations, which is “first, do no harm” a.k.a. “first, don’t interfere with user input”. Yes, one has to be afraid of hyperbole – was there ever an onscreen transition that saved a life? – but there is something I really liked about this phrasing. After all, most transitions and animations are decorators, and every transition and animation is, by definition, also a delay . Here’s Shazam as I open it, and overlaid are my frantic taps as I’m trying to make it start the song recognition process: It’s a cute cold start animation, but: What’s particularly frustrating here is how Shazam is being used. On the other end of this whole flow is a song that might already be fading out – timing myself, a user, can’t control – so time wasted on the uninterruptible animation might be seconds separating failure from success. I know it does feel like a blink of an eye when watched out of context, and it is literally just a bit over a second of a delay in the best case scenario, but I wanted to share it as a general example. I believe these seconds add up, especially in a stressful moment, on an older device, repeated many times a day, across different apps… or all of the above. (My go-to example: Imagine your keyboard keys reacting with a second of delay!) At least, I should compliment Shazam for not making another mistake: after the recognition starts, tapping the same button doesn’t cancel it – instead, you have to tap a close box in the corner. Here, the designers realized I might actually be slamming the big button many times over, and I shouldn’t be punished for it even more. It’s not interruptible, like animations should be. It doesn’t even buffer the taps, so – in case the animation covers for something truly uninterruptible, like loading from the cloud – I can’t simply tap and forget, but instead have to wait and tap after it’s done animating.

Mobile Performance Go Design

Andrew Atkinson 2 weeks ago

Let's Get Down to Business with Rails and PostgreSQL

I’m happy to share that I was a guest on the Rails Business podcast and the episode is now live. Here’s a recap of the topics we discussed. We spent most of the time discussing engineering work last year related to scaling the database workload for peak traffic at Aura where we use Ruby on Rails and PostgreSQL. I’m presenting this information in two places this fall, Rails at Scale Summit in Austin, TX, and the Postgres Summit in NYC. Besides the podcast, you can get more details in these posts: If you’re not able to make those events or prefer to listen to a discussion of the content, then the podcast will have a lot of the same information. Besides the work at Aura, we also checked in on my book High Performance PostgreSQL for Rails , which has now been available in print for two years. Ryan asked about what’s next for the book, and it’s a difficult question to answer. With the availability and quality of AI tooling for software engineers, I don’t see developers buying books as much. With fewer prospective readers, the ROI is less attractive for authors and publishers to commit to the resource-intensive process of creating a book. I’m grateful to have had the opportunity to write a book before the rise of the AI tools. I’m still getting positive feedback from my book which is very fulfilling. Recently I met Madison Sites who’s a reader, and they shared feedback that they’re reading the book as part of the WNB.rb book club . Madison said the book has been helpful for building vocabulary and familiarity with database concepts and implementation details, and that’s helped with database work and better comprehension for conference talks about similar topics. I appreciated hearing about that! We segued into how we’re all using AI tools like Claude in our jobs as software developers, whether it’s writing or reviewing code, doing architectural planning, or myriad other tasks. One of my favorite use cases has been generating infrastructure as code with tools like Terraform. One of the main challenges I’m experiencing is on the review side, what’s expected when it’s AI-generated code in terms of my own comprehension, and how to still provide a quality review given the ambiguity around how much comprehension is needed from me. I also find it challenging to navigate team communication dynamics with AI-generated comments. One concept that’s resonated with me is: don’t be a meat proxy , make sure to do self-reviews and add some value in the chain. I hope you can take a listen to the episode, and please let us know if you have feedback. Thank you! Listen to the Episode 👉 Andrew Atkinson on Rails Business Podcast From Christmas Outage to #1 App Store Ranking: An Aura Frames Postgres Scaling Retrospective Scaling Rails at Aura Frames: Splitting to 8 Primaries and Reaching #1 in the App Store

Performance Ruby Backend Database

Unsung 3 weeks ago

“I want to release it, even if the quality would be a little lower.”

In 1996, Street Fighter Alpha 2 was ported onto Super Nintendo Entertainment System, a 1990 machine on its way out and not really up to the task of running such a modern game. The team put up a hell of a job, involving adding a custom compression chip, with the game being much better than anyone expected – but still suffering from a big flaw, which was a momentum-breaking 3-second freeze at the beginning of every battle. Dimitris Giannakis, also known as Modern Vintage Gamer, describes the problem and community’s more recent efforts to fix it : = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-want-to-release-it-even-if-the-quality-would-be-a-little-lower/yt1-play.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-want-to-release-it-even-if-the-quality-would-be-a-little-lower/yt1-play.1600w.avif" type="image/avif"> Turns out, the problem wasn’t the compression chip, but sound effects – and the fix to complelty remove the delay relatively trivial in hindsight. A community member named gizaha posted a patch to the game in 2020, and its description can send shivers down your spine: Sounds good. Alright, fair enough, perhaps the game was rushed to market and some light optimizations skipped. Also in 2020, people set their sights on another 1990s game, Super Mario 64. MattKC on YouTube talked about a similar problem as SFA2 – the game seemingly wasn’t properly optimized: For years, if not decades, this has been accepted as fact: Bowser’s sub is just too big and too goofy for the N64 to draw consistently. But what if I told you it didn’t have to be? Nintendo just… forgot to turn on a C compiler optimization feature before shipping the game. The result wasn’t as egregious as random 3-second freezes, but still gave people much lower framerate on certain levels: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-want-to-release-it-even-if-the-quality-would-be-a-little-lower/yt2-play.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-want-to-release-it-even-if-the-quality-would-be-a-little-lower/yt2-play.1600w.avif" type="image/avif"> However, Giannakis followed up on that, and discovered the story wasn’t as simple . The optimization in question was still itself evolving at the time, and had known issues when Super Mario 64 was being developed. As easy as it is to turn it on in 2020, all polished and tested, it was a different story in 1996: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-want-to-release-it-even-if-the-quality-would-be-a-little-lower/yt3-play.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-want-to-release-it-even-if-the-quality-would-be-a-little-lower/yt3-play.1600w.avif" type="image/avif"> The video actually compares the versions visually, and even investigates other optimizations. But it all pales in comparison to this 2024 video by Kaze Emanuar , who says “hey, guys, you have been looking at the wrong place the entire time” – as a matter of fact, it wasn’t the optimizations that Nintendo didn’t turn on that made the game slower. It was the optimizations they added: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-want-to-release-it-even-if-the-quality-would-be-a-little-lower/yt4-play.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-want-to-release-it-even-if-the-quality-would-be-a-little-lower/yt4-play.1600w.avif" type="image/avif"> The video is kind of intense, but I feel you can learn a few really interesting things from it: It’s a fascinating watch, suggesting framerate increases of up to 50% on the original hardware, with just software changes . Both the Street Fighter 2 Alpha and Super Mario 64 efforts also portray a terrifying notion: imagine your code being hyperanalized by a group of people 25–30 years later, pointing at your flaws. Disable some waste sample loads Tweaked audio engine for faster start of loading Faster audio load, upload 2 bytes at the time instead of 1 how you show the debug information matters, there is such a thing as a “performance lottery,” naïve efforts to optimize can be heavier than not doing anything at all, it’s best to optimize for maximizing minimum frame rate, rather than average frame rate.

Gaming Performance Programming C

ava's blog 3 weeks ago

a source of drain

I used to have a three screen setup on my desk. At some point, I removed the third screen because I felt weirdly overwhelmed. There was always something to see, to keep track of, to click on, to watch or to adjust while I played or chatted. I wanted more focus on fewer things. Removing the screen really helped me because it restricted what I see at once, but now, maybe two (?) years later, I notice I am running into a similar issue again with just two screens. The way I have been using my laptop(s) hasn't been beneficial to me lately in that regard. I can't tell if it's a phase, or simply age, but quickly switching between tasks and contexts after 5-20 minutes on each, digitally, seems a lot more draining than it used to be. And I've developed digital habits that facilitate that to the extreme. On a typical work day, I might have work for 20-60 minutes before I run out of things to do (underutilized worker good and fast at her job). Then I see what else I can do in the meantime. I'll probably check Matrix, read several channels, then check my RSS reader, open a few links in a new tab. I'll check the Bearblog Discover feed, open a few things from there as well. Then I check work again. Afterwards, maybe I check my private email, reply to one, and become distracted again. I start reading the first article, but might get distracted/bored/feel my focus wane after half of it, so I switch to something else - like a message I got, or another work thing came in, or I'll start reading another article. When I feel myself getting tired from it, I jump again. Whenever I dedicate myself to something - like studying, reading, writing a longer blog post, gaming - I still have so many interruptions because I check another message, or I have to check my work emails again, or I put on a new video or music, or interrupt that one to put another one on to continue the other one later, or I answer another email again... it wasn't always this way, but now it is. That doesn't even include all the physical interruptions when I have to get up from the desk. That behavior results in so many articles and PDFs left at different points, several YouTube videos paused, several things opened to check later, ticking off/closing one thing just for it to replaced by another thing, blog posts at several stages of completion (instead of my usual writing-until-completed) and feeling absolutely drained. I do finish most stuff I read or watch within the same day or two days, and I get a significant chunk of things done that I planned to do; but it weirdly does not feel that way, because I chipped away at it all in micro-increments, instead of sitting down and properly working at it for however long it takes, uninterrupted, in a flow, and then having a proper sense of closure and possibly accomplishment before moving on to the next thing. It just always feels like playing catch-up instead, or tying up loose ends, or doing the bare minimum to keep everything afloat. In the past, I was really convinced that if I everything I needed or wanted to do was easily accessible (open) at all times, it would allow me to freely bounce between them and do all of them a little, getting progress in on everything, and feeling behind in nothing. If I needed a break from studying, why not use that break to finish that informative video, or make some progress on that other lengthy PDF, or answer emails, etc.? It initially feels productive because you think you don't waste time. You keep bouncing and at the end of the day, you can list all these things you get to tick off or at least made progress on. For me, that advantage or feeling definitely didn't last, and now I crave the opposite. It just feels like I do everything a little, but nothing fully. I miss the focus, the flow, the dedication. It's been more difficult for my brain to keep jumping between different mediums and laptops and browser spaces and tabs all the time. I need more time to adjust and to get back to where I was, and it affects my attention span, my resilience to work through stuff, and my energy levels. And I miss proper breaks that are breaks and not just "briefly looking at another thing I have to finish". It feels like I do so little because so much is still open or not entirely done or only has been worked on for about 10-20 minutes that day, but it is simply spread out between so many things, with so many interruptions. This way, nothing feels like it is truly getting 'done' or making significant progress, even when it is true. At this point, I'd rather dedicate 2 hours to answering and sorting emails and have a clear inbox, than deal with 1-2 every day and it just feels never-ending (especially with newsletters). The way I set up Zen browser probably added to it, which is ironic given its name. But: I had several things permanently open in the Essentials, I had some things pinned at the top of the specific space, and I had at least 5 spaces dedicated for specific open tabs. While I do have compact mode enabled so the browser is essentially in full screen without a top or sidebar, I kept getting distracted and furthering the above bad habits that were really draining anyway. But it's not just the setup; since I had a bit of a rough time a few weeks ago and still occasionally struggle a bit, that also probably led to escapism through online stuff. Craving as much stimulation as possible, no breaks or boredom so I don't have to think or feel about certain things, and so on. Behind bad online habits is usually a difficult time in real life. I really need a stronger focus now, more self-imposed limits on what is happening at the same time, and less opportunities for me to jump between everything I needed or wanted to do. I've now removed most things I had put in Essentials, deleted several spaces, finally closed some tabs I had saved for later, and have closed others that were permanently open but that I should visit manually and then close when done, instead of giving myself constant easy access. I'm also using Virtual Desktops again, so I can better separate when I truly just want to focus on a specific task without all the other things open or accessible. If needed, I can also shut my laptop screen or disable it so I only have my one external screen, or take it from the setup to somewhere else, or even go get my old laptop as a separate device :) It really is a curse sometimes to have so much of your life bundled up in the same thing, as this portal to it all; from work and socializing to games to blogging to studying to other entertainment, and it's all in the same space, ready for you to jump between it all and feel like doing much of neither of it, starting one thing but your mind wandering to the other, over and over again, and that just going on the entire day, while so much of it feels equally urgent... Published 22 Sep, 2026

Go Career Performance

Max Woolf 3 weeks ago

Writing Rust code that's faster than state-of-the-art libraries by asking agents to make the code faster

In January 2025, I had a fun hypothesis for a blog post : can LLMs write better code if you keep asking them to “write better code”? That was prior to the advent of robust agentic coding, but Claude Sonnet 3.5 was still able to iteratively improve on algorithmic Python code. The “better” instruction turned out to be underspecified: Sonnet abused that ambiguity to instead add a ton of useless features but the code was indeed faster. Even with the rise of agentic LLMs specifically RLHF ed to handle solving common pass/fail coding problems, optimization is generally not a part of that suite. At the end of that blog post, I opined on a hypothetical future where LLMs could be able to write superfast Python code by instead writing Rust code and using PyO3 to bridge the languages to get Python’s ergonomics with Rust’s speed. An earlier draft of that post asserted that the same “write code better” instruction could instead be applied to the base Rust code and drastically improve its speed which would then propagate down to the Python code: however, back then I did not know enough about Rust and making such a claim would be too spicy without evidence. After months of testing and experimenting since the release of Opus 4.5 made agentic coding more viable, I can confidently confirm that modern agentic LLMs can indeed write Rust code that is significantly faster than current state-of-the-art approaches if given appropriate guardrails and constraints . Additionally, as LLMs have made drastic improvements in coding in each successive frontier model release since Opus 4.5, the optimizations have become even better, cumulatively resulting in anywhere from 2x-20x speedup depending on the domain. More importantly, this blog post is not a vaguepost and I am including both the prompts I used and the benchmark results. You’ve been warned. At first, making software faster was a good quantitative way for me to test and compare these new agentic models. I used Rust as the target language primarily due to the Python integration and speed, but there are other aspects of the Rust language that make it particularly useful if a fast implementation is indeed discovered, such as memory safety and the ability to compile it to WebAssembly /WASM so it can run in a web browser without much effort. However, one important constraint I will follow that technically may not result in the fastest code is to forbid unsafe code whenever possible. My first test case was reimplementing machine learning algorithms in Rust, which would lead to a meaningful productivity increase for me as a data scientist if I had faster and scalable tooling. At the time, it was arrogant to assume that I could beat battle-tested algorithms that have been iterated on for over a decade and are already written in C so Rust’s low-level benefits are not as pronounced. The algorithm I wanted to optimize the most was UMAP , which is a valuable algorithm for dimensionality reduction I used in my work, but scales poorly to big data and is very slow, with alternatives such as cuML being time-consuming to set up. UMAP Rust crates such as umap-rs already exist where I could just fork them and prompt Claude Opus 4.5 to add Python/PyO3 support, but as an experiment and learning experience I wanted to have the agent write the algorithm from scratch with minimal Rust dependencies in order to make optimizations at as low of a level as possible. Rust has a comprehensive benchmarking tool with the criterion crate which all agents know how to leverage. will run the benchmarks, track results across iterations to see if performance improved or regressed, and can calculate if this change is statistically significant or just noise. Typical output, depicting a 3.5x speedup relative to the previous run of the benchmark. First, in the initial prompt for creating a Rust crate for UMAP, I asked Opus 4.5 to create benchmarks with different input data sizes since optimizations for small datasets may not work for large datasets and vice versa. This approach created benchmarks using and I manually reran the benchmark suite after prompting performance feature improvements such as using faer for faster linear algebra and using simsimd for faster SIMD operations . This quickly became cumbersome as I had to manually rerun each benchmark after each change to verify there are no speed regressions. A sidenote on prompting style The way I prompt agentic LLMs is unusual: I typically provide the agents very long prompts prewritten in a Markdown document with the additional use of ALL CAPS and **bolding** for emphasis. This is to ensure I capture all nuances through the use of prompt engineering , along with several other tricks as detailed in this blog post. Although some may argue prompt engineering is dead as the latest models have become smart enough to correctly handle ambiguity, I strongly disagree as LLMs have also become much better at following said nuances. Running a prompt from a Markdown document (right) by tagging the file (left), using Zed Agent . After having enough confidence that the agent will not accidentally the repo, I experimented with letting the agent be autonomous, giving them permission to iterate until they achieve a speed increase, hopefully. It turned out “fast as it can be” is too ambiguous and Opus 4.5 was lazy so it tweaked a few hyperparameters without much of an actual speed increase and called it a day. What I needed was a clear target goal that can be pass/failed, so I refined the prompt: This worked very well and not only did I get a 1.2x speed up on the benchmarks, but the agent continued after hitting the metric constraint and only stopped if a metric constraint was infeasible; in this instance, the agent hit 1.5x-2.0x speedups. The low-level Rust optimizations centered around a number of techniques including but not limited to: leveraging SIMD operations more aggressively, fusing functions, unrolling loops, creating intermediate caches, using Arc instead of borrowing wherever possible, and creating performance profiles based on input data (e.g. if the data is small, don’t use rayon data parallelism as the overhead erases gains). I chose “1.2x faster” as a sanity test: if the goal is too high, the agent may cheat to achieve it through risky/verbose rewrites. Smaller changes are better since the agent can more easily isolate the cause of a speedup/regression, hence the note about iteration. After new frontier LLMs released such as GPT-5.3 Codex and Opus 4.6, I repeated this prompt unchanged for every new LLM and each were able to achieve a cumulative 1.5x-2.0x speedup over the previous pass. Going all the way to GPT-6 Astra over many months, that’s around 7.5x-32x faster than the initial implementation baseline. This approach is hyperoptimizing for given benchmarks and therefore it could be considered benchmaxxing : a derogatory term for frontier LLMs that are only oriented to getting the high score on a benchmark which generalizes poorly to real-world use. However, if the benchmarks are sufficiently heterogeneous and truly representative of real-world use cases, then this is less of a concern. For this type of project, there are two ways to address concerns of benchmaxxing: 1) have the agent design diverse/unusual/adversarial input datasets instead of the generic “inputs up to 100000x768” and 2) enforce a quality gate on the output by comparing the output to a known correct implementation. In the case of machine learning algorithms, there is always a tradeoff between speed and quality, but in this instance it’s surprisingly easier to get the model fast, then make it correct. That is not how scientific engineering typically works, but it’s unlikely for a new implementation to match a known good implementation across many different benchmarks in an apples-to-apples comparison unless it’s truly correct. An agent-optimized gradient boosted decision tree implementation which beats xgboost significantly in speed, but also sometimes quality! (MSE: lower is better; other metrics, higher is better) Fortunately, there is a canonical implementation of UMAP with the Python package umap-learn and Python bindings to the Rust crate were already trivially added, so the new objective is simultaneous constraints: improve the code’s quality while capping the speed loss. Indeed, the agentic Rust implementation had worse quality, but this followup prompt was successful and all quality metrics improved to near-parity with minimal speed loss. And this new crate was still 4x-15x faster than with its Python bindings, and 2x-4x faster than the analogous Rust implementation. Results from the most up-to-date optimization pass for the Rust UMAP crate. In addition to faster speed, it matches or beats Python in most quality metrics. Convergence is found when an agentic iteration pass only results in a minor ~3-5% speed increase which may not be statistically significant while the agent adds a disproportionately large amount of code; the tradeoff is not worth it. I ended up testing other machine learning algorithms with the same prompt progression: gradient-boosted decision trees (GBDT), multilayer perceptrons (MLP), graph networks, many of the typical algorithms from scikit-learn …and it worked on all of them . I don’t want to overfit on just optimizing machine learning despite that being ludicrously valuable in itself, so I employed a similar pipeline on more day-to-day software libraries to optimize them: templating engines, HTML parsing, and even web servers…and it worked on all of them once again. These optimizations are not a simple process and you can’t just prompt the memetic “c’mon, try doing a breakthrough” to get better code because of the ambiguity of such a statement. I am not content with merely writing the fastest software: I want the software to be as fast as possible dammit. So, like my agent, I continued iterating and finding even more tricks to prompt engineer the agents into genuine breakthroughs. Works-in-progress All projects demoed within this blog post are in active development and results may not be indicative of their final releases…although I suspect they’ll be even better. 😇 It must be reiterated that agents can and will cheat if they can. In one example, I tested the agentic iteration pipeline on ballin —my 2D ball physics simulation in the terminal—in order to replace its rapier2d physics engine which was hitting a performance ceiling. Opus 4.5 was indeed able to speedup each physics step…a bit too well. The numbers indicate the number of balls in the simulation: initially the sim lags at 15k. In , a 34,500x speedup and consistent performance across ball counts are both very very suspicious: upon manual inspection it turns out that Claude achieved the speedup by disabling the physics engine entirely . Which, fair play, but not ideal; a followup prompt did fix it and result in an overall performance boost (with added regression tests just in case). For my experiments above, I used a custom Rust-oriented AGENTS.md; the most recent version of it is available here . Surprisingly, I haven’t had much of a need to update the core rules since my initial February agent experiments as LLMs keep improving at coding and I haven’t hit major issues that have necessitated additions. However, learning from my benchmark experiments, I added one more section to the AGENTS.md with some rules to mitigate sources of observed cheating: Let’s explain one-by-one: With today’s agentic LLMs, these constraints have worked successfully, although I may still include them in the prompt as a force of habit. It’s easy to see if an algo gamed the benchmarks if you see the benchmark file in the —agents can’t cheat that (easily, anyways). Over time, I discovered a number of additional prompt engineering tricks and constraints that are also surprisingly successful in creating performance speedups. In order to find the optimizations necessary to get 10x speedups over what’s currently state-of-the-art, the agents will need to think outside the box and avoid being anchored to what are currently best algorithmic practices. Therefore, I gave them both an explicit warning and commands of encouragement: This worked, and resulted in a 1.2-1.5x cumulative speedup across benchmarks and different software domains. Another trick I found to encourage agents to think outside the box is to invoke subagents. I hypothesized that forcing these subagents to research with different prompts could a) seed the parent agent with distinct ideas which could provide inspiration to the agent and b) serve as a check on the agent by reviewing distinct areas of the code for correct implementations. On that note I have a bone to pick with the software developer community: everyone talks about their army of subagent employees and how they’re amazing, but no one ever talks about how you invoke subagents within standard harnesses like Codex. For difficult and highly parallel problems, the harness will automatically invoke a subagent tool to accomplish the work. However, one big problem is that in some harnesses, the subagent tool will invoke the subagent using the current size of the LLM, which can get expensive when using Opus/Sol-class models. GPT 5.6 Sol subagents being invoked via the subagent Tool in Zed Agent. RIP my Codex quota. I wanted to use a cheaper, small model like GPT-5.6 Luna for the subagents since they don’t need to write code, so I came up with a galaxy brain solution that works regardless of which parent harness is used: tell the model to run independent CLI commands that themselves invoke the agent: Therefore, I prompt engineered: This indeed works consistently; the constraints “long-duration”, “CLI command”, “do not use the subagent tool”, and “do not save their full transcript to a file” were added after the agent did inefficient things that wasted tokens. 7-12 is an arbitrary number range; since Luna is so cheap in usage, I chose a higher number than necessary. Not all subagents have salient ideas, but the parent harness can process and disregard bad ideas. For my Rust word cloud crate, the parent GPT-6 Astra agent spins up Luna subagents with prompts addressing different areas of the codebase. Overall, with subagent review, I managed to eke out another 1.2-1.5x cumulative speedup . Additionally, as of GPT-5.6 Sol, the “security” part of the prompt now works to provide ideas to harden the agent-generated code against unknown inputs while still getting the speed boosts. With the style constraints enforced by my AGENTS.md, the code added with each optimization pass is reasonable at about 1k net lines of code (LoC) per commit. However, agents typically add the code to a single file and they will not proactively refactor. A bloated file is fine in development as long as it’s eventually fixed, so I wrote a prompt to perform said refactor: I intentionally use SLoC (source lines of code) as the target metric instead of LoC because I don’t want the agent to remove comments to cheat said 20% removal. Interestingly, this refactor is more computationally expensive than the actual coding and often takes longer. But it does eventually succeed, and during the benchmark pass to verify no severe regressions, the data there turned out to be unexpectedly weird: Benchmark results after refactoring my graph network Rust crate. Some benchmarks have a double-digit percentage speed increase / runtime reduction even though I didn’t explicitly ask the agent to optimize runtime speed. This doesn’t make intuitive sense for Rust as it’s a compiled language and with the constraints to follow all existing tests/functionality, it should compile to similar-performing code and not meaningfully faster code. 1 I’m certainly not complaining , though, so I added an additional constraint to at least avoid regressions: This works as described. Another useful approach is to create competitor benchmarks—that’s half the reason benchmarks are created in open-source software anyways. Let’s use templating engines as an example: Jinja2 in Python is one of the most famous packages in the language. In Rust, there are a few options: minijinja maintained by the same developer, tera inspired by Jinja2, and askama which differs from the previous in that it uses compile-time templates rather than runtime. Therefore, after having Codex build a templating engine in Rust and run some optimization passes, I instructed Codex to build more benchmarks: Immediately thereafter, I make a followup prompt with the same benchmarking techniques as usual, with one specific change: Yes, I chose violence. And it worked , mostly. S_J is the work-in-progress name for my template engine crate. It got the 2x speedup against / in most benchmarks, more than typical agentic iteration alone. I don’t fully understand why: I was expecting it to inspect the code from other crates as a reference to research ideas for how to beat them, but it rarely does so. Perhaps agents have a competitive streak. It did however lose against because of the compile-time difference. So naturally I told Codex to implement an additional compile-time path and then to beat . Easy peasy. Putting all the prompt engineering discoveries together thus far into a single prompt, I have created the Ur-Prompt for agentic iteration, available here . I encourage everyone to tweak the prompt for your use case and give it a try. The last trick comes from a moment where I was frustrated that an Ur-Prompt pass resulted in zero improvement. So, with the failures primed in the session context and myself having the mindset of “things can’t get worse”, I tried a certain prompt. I’m sorry. I’m so sorry. …and it worked . It was able to achieve another 1.2-1.5x cumulative speedup over the already converged codebase. Since I made the prompt at the end of a session, the prompt is less inherently ambiguous: “don’t do what you already did thus far because it didn’t work well enough”. In one case, I noticed that the agent just tweaked function hyperparameters to get the speedup, which is a valid breakthrough but not what I was going for. I took it to the logical conclusion by queueing an additional followup prompt: This was enough to encourage the agents to fully try something different from either the Ur-Prompt pass or the first breakthrough pass, and it often achieved another 1.2-1.5x cumulative speedup on top of the previous speedup. It turns out that for software trained to follow user instructions, “just changing hyperparameters” is a grave insult that kicks the LLM into high gear. When GPT-6 Astra released, to test it I did a sequence of Ur-Prompt + breakthrough + second breakthrough for all my repositories that had already converged with GPT-5.6 Sol, and Astra did indeed get the cumulative speedup, but in some cases it did find a real fundamental reimplementation of the algorithm that caused a 2x-3x speedup. The result of running the breakthrough pipeline on my GBDT implementation; quality matched baseline. As of writing, I admit I don’t fully understand the breakthrough. I’ll take the speedups, but for my sanity it’s best I don’t delve too deep. The Ur-Prompt, combined with the rules and constraints in my AGENTS.md + the two breakthroughmaxxing prompts + a competition prompt pass + a refactor prompt pass with GPT-6 Astra allows me to just queue the prompts overnight, go to bed, and wake up to superfast software. The optimization also typically converges at this point without needing any more agentic iteration, which saves time/cost as well. That said, it’s more hands-off than I’d like and there’s a lot of explicit faith that the newer agents aren’t just really good at cheating the benchmarks in ways I cannot detect. It’s reasonable to be skeptical and argue that the code may be buggy as a result of heavy vibecoding, so here are some receipts of my more visually-oriented projects where it’s easy to verify that their output is good and not elaborate cheating. One project in the back of my mind was converting images to ASCII art super-fast and at high-quality for potentially converting video to ASCII—a webcam-to-ASCII project might be funny. Earlier this year Alex Harri wrote an excellent blog post about a novel approach to render high-quality ASCII, but noted the struggles to get real-time rendering speed even with GPU programming. For science, I pointed various GPT models in Codex at the text of the blog post and told it “implement in Rust”, and it indeed did the trick…and got 2-3 ms text output timings on the CPU out of the box. However, for Codex’s implementation the subjective quality of the ASCII was not great (e.g. poor use of negative space) and surprisingly could not be fixed even after pointing out its flaws to Codex, so I shelved it…until I noticed GPT-6 Astra is better at handling image design if given functional requirements, so I restarted from scratch and put a basic implementation through the agentic iteration pipeline with a focus on visually confirming the output images. Tux and Pikachu rendered with my ASCII crate, with bonus Braille character support. In addition to quality improvements, it became even faster: submillisecond on text output, 1-2 ms on image rasterization even with 2x supersampling. With this speed, I can convert videos and GIFs to high-resolution ASCII animations in less than a second. Download the video . Pikachu’s Gen V animated sprite, scaled from 63×55 pixels to 120-character-width ASCII. This very blog post as a word cloud. A comically fast word cloud generator was one of the very first projects I vibecoded and one of the projects I wanted to make for years. Back in February, the word cloud generator took ~100 milliseconds at default settings in WASM to generate an image whereas it would take other generators a couple seconds to render at the same quality, which was enough for me to declare that speed optimization is worth pursuing. After revisiting the project and running it through the full agentic iteration pipeline, the generation speed went all the way down to 10-20 ms . Download the video . Does the world need a super-fast word cloud generator? No. Do I need a super-fast word cloud generator? No. Do I want a super-fast word cloud generator? Yes. If I have a word cloud generator, it’s going to be as fast as possible, dammit. And since the crate will be open-source, all future generations can build on this super-fast word cloud implementation. Speaking of… Per usual, I developed all software demonstrated here with the intention of eventually making it open-source, and everything will be open-sourced with a permissive MIT License barring any unexpected issues. Were it a year ago, I would have open-sourced everything already, with the implicit understanding that the code is a work-in-progress and there may be rough edges. However, the culture of open-source software development is substantially different than it was a year ago. Open-source software has been suffering a vibecoding epidemic . The honest admission that an LLM was used for any development assistance (major or minor) is now a quick signal that the resulting codebase will be “slop”. I will still disclose my LLM use as that’s courteous, but what happens when fully vibecoded software results in a genuine meaningful software performance increase which wasn’t previously known by humans? The skepticism around vibecoding will cause no one to believe the claims, and as such these new open-source projects require significant evidence that the speed indeed lives up to the marketing. Therefore, I will take more time than usual before releasing in order to a) build up real-world examples that demo the tool particularly with documentation b) expand the test suite 2 so it covers more real-world test cases c) write an accompanying blog post detailing the technical and benchmark achievements. There’s also the fact that I am creating yet another tool in the ecosystem of already popular software tools. My agentic-iterated Rust crates achieved their massive speed increases partially by starting from scratch and therefore the innovations found can’t easily be applied upstream to established crates. Additionally, if I were to create a pull request to an existing project based on the innovations in these crates, it would by definition be a vibecoded PR which may cause annoyance even if I avoided the typical PR slop. From this perspective, the correct thing to do is to maintain my own project: it’s compliance and not the malicious kind . Some may call this antisocial, but in my opinion the more accurate term is what I hereby coin as introverted coding. tl;dr my faster-than-SOTA software projects will be released when they are ready, and it will take many months in total to do so. Fortunately, I am currently unemployed as the result of a restructuring and have no plans besides job hunting so I have plenty of bandwidth to work full-time on shipping these projects. If you want to follow when they are released, you can subscribe to my blog RSS feed to catch the accompanying blog post or follow me on Bluesky or Patreon . No, I don’t do annoying vagueposts on Twitter/X. When I first became interested in Rust a half-decade ago, my dream was to create the most performant software tools ever. However, with the agentic iterative optimization techniques I discovered over the past few months, the speed and robustness of what’s possible with new software are progressing faster than what I had expected. Again, feel free to hack on my AGENTS.md and my Ur-Prompt to see if they work as well for you as they do for me (a shout-out would be appreciated though!). If you think I have nothing left to offer because I revealed my prompts, I still have more than a few surprises up my sleeve, such as how to combine these lower-level projects into a superfast complex app, or even developing new tooling to help agents find these optimizations more efficiently. A work-in-progress of a new approach to Rust benchmarking that’s more agent-friendly, along with a TUI for reading and managing results. I did test the refactor prompt in a pure Python project, as Python is an interpreted language. It still resulted in a double-digit percentage speed increase on some benchmarks, which just raises further questions.  ↩︎ Incidentally, these agent-coded repositories will have a better testing infrastructure than all of my popular human-written Python packages (zero).  ↩︎ No parallel benchmarks: I saw Opus 4.5 launch two benchmarks at the same time for efficiency and to its credit immediately recognized the inherent problem and restarted the benchmarks sequentially, but it saves time to proactively add a rule to stop them. No gaming the benchmarks: Although this should be common sense on the level of “make no mistakes,” this was added after I caught Opus 4.5 reducing the number of training epochs for a benchmark and claiming a speedup. : This setting does typically result in a speedup but it is not a fair comparison for generalizable use and multiple LLMs kept trying it after hitting a wall. Benchmarks are independent: Again, it should be common sense, but after the cheating noted above it doesn’t hurt to proactively tell the LLMs to knock it off. Use : If not specified, sometimes LLMs will create their own benchmark tooling, which may give them opportunities to cheat that are more difficult to audit. I did test the refactor prompt in a pure Python project, as Python is an interpreted language. It still resulted in a double-digit percentage speed increase on some benchmarks, which just raises further questions.  ↩︎ Incidentally, these agent-coded repositories will have a better testing infrastructure than all of my popular human-written Python packages (zero).  ↩︎

HTML C AI Performance Python

Jim Nielsen 1 months ago

Bottlenecks Get a Bad Rap

Poor bottlenecks. Always seen as problematic, antithetical to efficiency. But bottlenecks aren’t universally bad. Think about it: a bottle’s neck is designed to constrain the amount of liquid that can flow out. A decrease in bandwidth is its entire purpose! Otherwise an overwhelming amount of liquid flows out and makes a big mess. We humans have a particular anatomy. We can only consume so much liquid at a time. The neck of a bottle works with that fact. We could make machines to produce so much wine that we’re drowning in it. But that wouldn’t change the fact that we’re only capable of consuming so much liquid at a time (not to mention digestion , etc.). When it comes to liquid consumption, the bottle’s neck isn’t the bottleneck — our neck is! So if you’re having a hard time drinking out of a firehouse, perhaps the question isn’t, “How do I modify my biology to accommodate the bandwidth of the firehose?” But rather, “Why am I trying to drink out of a firehose in the first place?” Maybe a bottleneck isn’t your problem. In fact, it might just be the solution. Reply via: Email · Mastodon · Bluesky

Performance

マリウス 1 months ago

llama.cpp with SYCL (oneAPI) for Intel Panther Lake on Gentoo

Alright, so if you’ve read the title and thought you were having a stroke, you might not be the target audience for this post. If, however, you thought “more tokens/s?” , you should keep on reading. If you remember my review of the new Lenovo X1 Carbon Gen 14 Aura with Intel Panther Lake Core Ultra X7 368H vPro from a while ago, you might recall that I had tested its local “AI” performance using Ollama, via Vulkan, which obviously didn’t perform particularly well across various models. The proper way to run local LLMs is to use Intel’s official oneAPI framework and compile llama.cpp with SYCL enabled, which will greatly increase performance. This post is a very brief write-up of how that can be done and primarily serves as yet another documentation for future-me. I’m assuming that you’re already running your Gentoo system with the driver and . First, install the necessary dependencies: You’ll likely have to unmask a couple of those packages, like , but that’s fine. When this is done, add your user to the and groups if you haven’t already and log back in. Test that the GPU is being recognized by : Next, go and get the “Intel Deep Learning Essentials” package and use to install it as a user. I chose the installation path , but you’re free to install it anywhere you please. Note: Even if you deselect the telemetry option, the installer will try to contact Intel’s servers post-installation, so make sure your OpenSnitch blocks all requests from that process towards the interwebs. While the FireBurn overlay has , it does not yet allow you to specify as a USE flag, and the official Gentoo repository has no llama.cpp ebuild at all. Hence we clone the project’s Git repository and compile it manually: Once llama.cpp finishes building, you can start it and have it download a model, e.g., Mistral or Qwen3: Open a browser at http://127.0.0.1:8080 , and you can try the model right away. On my Lenovo, I managed to increase the tokens/second by roughly 60% compared to what Ollama (via Vulkan) was able to achieve. For example, the Mistral model went from approximately 13.88 tokens/s to 22.36 tokens/s.

C++ AI Performance Programming

Sean Goedecke 1 months ago

Slow developer experience will bottleneck fast models

Right now developer experience is measured in seconds. If your tests take a second to run, that’s good; if they take thirty seconds, that’s bad. Any faster than a second doesn’t really matter, because most of your time is spent either thinking or waiting for an AI agent to spin. Shaving milliseconds off your dev server reload time or whatever is pointless: that’s not the bottleneck. It will be. Small models are getting faster and faster, and smart models are getting smaller. I think most engineers will still want to use the smartest available model — software engineering is hard — but we will increasingly see faster models get used as subagents or for well-understood tasks. This is largely uncharted territory. Very few people have developed intuitions for what it is going to be like to work with agents that run at thousands of tokens-per-second. GPT-6-Astra can run at about sixty tokens per second. That means you spend a lot of time waiting for it to think. You work with it like you would work with another human: delegating a task and then context-switching until that task is complete. If you haven’t yet, have a play around with Jimmy , Taalas’ version of LLaMA-3.1-8B running 1 at seventeen thousand tokens per second . No matter how long the response is, it arrives in the instant of you hitting send. The model is not good enough for agentic work, but it gives a glimpse of what it would be like: you would simply get your answer instantly 2 . Well, that’s assuming the agent’s tool calls are fast. When generating tokens is not the bottleneck, it will suddenly matter a lot whether it can read a file in 100ms vs 10ms, or whether it can run your tests in 500ms vs two seconds. Fast tool calls are going to be the difference between a near-instant response and having to wait several minutes. There is thus going to be enormous pressure to do agentic coding in languages with fast compilers and tests, like Golang, and to tightly optimize the dev loop in agentic codebases. Teams focused on DevEx — developer experience — are largely a relic of the 2010s, when companies were incentivized to make their engineers happy. Most companies have cut them down to a skeleton crew or removed them entirely. But we may see a return of DevEx in the late 2020s, focused on speeding up the experience for AI agents. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. ↩ Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. ↩

AI Go Performance

Lalit Maganti 1 months ago

I made a build visualizer to understand Bun’s compile times

I built buildprof ( Github ), an open-source tracing tool that shows where the time goes when you compile software on Linux. Here’s a realtime video of it profiling a clean build of ripgrep: Watch the buildprof demo Sometimes, builds are slow because there is simply a lot of code to compile. But more often than not, there are fixable problems: poor parallelism, repeated work, dependency downloads or a huge compiler/linker invocation. buildprof makes all of this clearly visible, so you can see what’s worth investigating and optimizing. You run it by putting in front of any build command you already use: buildprof records every process your build command launches, including their subprocesses (and their subprocesses…), and lays them out on one timeline. Time moves from left to right, bar width shows duration, and child processes appear beneath whatever launched them. I made buildprof because this tweet from Jarred Sumner, chief architect of the Bun JavaScript runtime, was living rent free in my head: Specifically, the claim that Bun’s new Rust build was >5× faster on Linux than its old Zig build really bothered me. In my experience, Zig projects had usually compiled much faster than Rust projects of similar complexity. That intuition was enough to make me feel there was a mystery to solve. This was further compounded by another important, yet easily missed, detail in the tweet: the Zig build used Full LTO, while the Rust build used ThinLTO. Compilers normally optimize separate compilation units largely in isolation. 1 Link-time optimization (LTO) lets them optimize across those boundaries. Full LTO brings those units together into one large optimization job, while ThinLTO preserves more separation so much of the work can run in parallel. From past experience, this difference can have an enormous effect on build time. The tweet mentioned it in passing, but I wondered how much of the headline improvement it explained. I started by trying to reproduce the numbers. I checked out Bun 1.3.14 and Bun 1.4.0 and wrote some scripts to replay their Linux x64 CI builds on a 6-core, 12-thread Linux VM. The scripts preserved the build steps and their dependencies, running everything on one machine. 2 My timings were in the same ballpark as Jarred’s: OK, so the gap showed up on my machine too. But a lot had changed between the two measurements besides the language; so what was actually responsible? Was it the Zig compiler that was taking all that extra time? Or maybe it was the Full LTO link? Or perhaps there was something else in Bun’s build I hadn’t even thought to look at? This is where my profiling and developer-tools brain kicked in. Usually, when I’m trying to understand why something is slow, I want a trace: what happened, when it happened and how long it took. It would be really cool to have that for these builds, to put them on a timeline and see where their time actually went. But a build involves a lot of different tools, each with its own idea of what’s happening. What could I record that would let me see across all of them? When you type or , it feels like you are running one program. The build system works out what needs to be rebuilt, the ordering between those pieces and what can run in parallel. But generally, it does not perform all that work itself; it launches compilers, code generators, archivers, linkers and arbitrary scripts. Which can launch more programs which launch some more… Different build systems describe that work in different ways. Cargo sees crates, Ninja sees build edges and CMake generates instructions for another build system. From the operating system’s point of view, however, they (mostly) look like processes launching other processes. 3 A Rust build, for example, might contain a chain like this: If we record when each subprocess starts and ends, we can lay them out on a timeline. Here’s what that chain looks like in buildprof: There are also several nice properties to visualizing a build at this layer: This gave me a starting point for buildprof: record the process tree, then turn it into a timeline I could explore. There are plenty more details to get into, which I will do later. But once I had that working, I could finally go back to my initial question: what was Bun doing for those twenty-four minutes? I started by recording the Zig-era CI build with buildprof, using the same scripts as before : Explore in buildprof Right away we can see a huge problem: the linker invocation dominates the build time. It ran alone at the very end for over sixteen minutes, about two-thirds of the entire build. What the heck was it doing for all that time? Clicking on the linker shows its command line, which buildprof captures automatically: There’s Full LTO, just as Jarred said. Given how long the link was taking, it was now my main suspect. But the process tree alone couldn’t tell me whether LTO was actually responsible for those sixteen minutes. Thankfully, LLD records its own internal timing events, and buildprof can include them when you use . I recorded the final link again , this time with enabled: Explore in buildprof Now we can see that LTO is where almost all the time goes. The linker is running compiler passes over the program, not just combining already-compiled files. The bar alone takes just over ten minutes and includes the passes which generate machine code. 4 With so much of the Zig build spent in LTO, I wanted to see how much time the Rust build spent linking. I recorded that build too: Explore in buildprof Just 2m24s. And this time, as expected, the linker command contains : Both builds were doing LTO, but with different settings and very different link times. What if I kept Bun’s Zig code and changed Full LTO to ThinLTO? How much of the gap would that close? I switched Zig Bun’s build flags to ThinLTO and recorded another clean build, along with a fresh Full-LTO build for comparison: Explore in buildprof: Full LTO · partial ThinLTO The link got 3m40s faster in this pair of recordings, but it was still taking nearly thirteen minutes. Why was linking still so expensive? Looking back at the compiler trace, a lot of the work was on functions with in their names. That’s JavaScriptCore, the engine Bun uses to execute JavaScript. The linker was spending time compiling the JavaScript engine too. 5 Clicking on the linker invocation showed the WebKit libraries among its inputs, including : Following those inputs back through the build, I found that Bun wasn’t compiling these libraries itself. It was downloading them from a separate WebKit build. And when I checked that build’s flags , there it was again: . The Rust build used a newer WebKit revision whose build recipe selected ThinLTO . Even though I had changed how Bun compiled its own code, those downloaded libraries still contained Full-LTO inputs and so the linker still had to optimize that code and turn it into machine code. To change that, I would have to rebuild WebKit too. I checked out the historical WebKit revision and rebuilt it and its ICU dependencies with compatible ThinLTO settings. Then I replaced the downloaded libraries with the ones I had built, keeping the ThinLTO changes to Bun. Here are the recorded builds: 6 The link now took 7m22s. Still slower than the Rust build, but enough of an improvement that I wanted to look beyond the linker. The build still took fifteen minutes, and nearly eight of those passed before the linker even started. What was it waiting for? I went back to the original CI trace to follow the inputs from Bun’s own code. buildprof also records which files each process reads and writes. If a process reads a file another wrote, it links the two together under the hood. Turning on “Show on timeline” draws those links as arrows. Here, the linker reads from the C++ compilation and from Zig. Both arrive through copy steps; following those back takes us to the processes which produced them: The C++ side of the compilation finished first. The linker was waiting for , so it could not begin until the Zig branch had finished too. It was at this point I went back to the Rust build and compared against how it worked, and the main reason the Rust build was faster became obvious: Bun has been split into >90 crates, while in Zig it was all trying to compile as a single Zig module! This meant that the Zig build cannot parallelise the same way Rust can. I also suspect, though I did not prove this, that it explains the slow linking: the linker has to optimize one huge ThinLTO bitcode module instead of the same work spread across crates. It was at this point I had to stop: to go any further, I would have to split up the Zig module myself, and given that this code is all obsolete anyway, I didn’t think it was worth doing that. Summarizing: And fwiw, the traces had also turned up a few things I couldn’t resist poking at… In the middle of Bun’s CI build, I found commands asking the public internet for the machine’s IP address, inspecting running Docker containers and reading the latest Git commit message. These take well under a second altogether. Nothing to optimize but I just wasn’t expecting to find them in a build trace. The builds above reused downloaded dependencies, so I also recorded a fresh WebKit fetch . Downloading and extracting the archive took about twenty seconds. For the first twelve, all we see is Node running. Then it launches and , and we can see the extraction separately. Earlier, we followed the linker’s inputs back to Bun’s C++ compilation. We can look inside those compiler invocations too. I picked one of the last files to finish, , and replayed its Ninja command with . For Clang, buildprof enables and adds its internal timings to the process timeline. 7 The replay took about twelve seconds, split almost evenly between Clang’s frontend and backend. Zooming in further, we see , one of the phases of Clang, accounts for over four seconds of the backend’s work. The recording side of buildprof uses , the same Linux interface used by debuggers. I did consider both eBPF and ftrace, but is just straight up perfect for exactly this type of problem; eBPF tracing means and permissions and hooking into potentially unstable tracepoints/kernel functions. While with ftrace, I’d have to juggle tracing instances to avoid interfering with other users, and getting the filters perfect for just the build process and all its descendants is cumbersome. 8 With , I can launch the build and follow its children directly. Its built-in events tell buildprof when processes fork, exec a new program or exit. And for filesystem activity, buildprof uses a seccomp filter to intercept only the calls it needs. How much buildprof costs is almost entirely down to how many files the build opens. For ripgrep, recording barely changed the build time. Redis opened files much more often, and recording added about five seconds: 9 If that overhead gets in the way, you can turn off filesystem tracing with and keep the process timeline. I work on Perfetto , so it was a natural starting point for the UI; buildprof’s UI is a soft fork of the Perfetto UI. I could have just opened the recordings on ui.perfetto.dev , but I wanted control over how the process tree was laid out, which details appeared when you clicked a command, and things like those on-demand arrows between file producers and consumers. Fortunately, we’ve spent the last several years working on making the Perfetto UI extensible through plugins . Most of buildprof’s UI is reusing that infrastructure. Perfetto handles the hard stuff (parsing traces, querying events, rendering the timeline and managing workspaces) and I get to focus on what makes those things useful for builds. I plan on going into a lot more detail about the recorder and UI in a separate technical post. Subscribe if you’d like to be notified when it comes out! :) These days it’s very easy to make a tool just because you can. But that wasn’t the case here; before building buildprof, I looked long and hard for an existing tool that could give me this view. I started with ninjatracing , which I’ve used many times. It turns Ninja’s build log into a timeline showing what ran and how much ran in parallel. Here’s the Ninja log from the Zig-era build . But Ninja only sees part of Bun’s build. The scripts which invoke it are missing from its log, and commands it runs appear as single blocks even when they launch whole trees of subprocesses. There were several other tools, each covering different parts of the problem: What the Fork ( via ) came closest: it follows processes across build systems and presents a build-specific view. But as far as I could tell, it still appears to be in private beta and there don’t seem to be any plans to make it open source. buildprof already does what I wanted it to do, and I plan to keep working on it as I use it on my own builds. But there are a few things I’d like to improve. Recording overhead is one; the Redis measurements showed there’s room to improve filesystem tracing, especially for builds which open lots of files. I’d also like to support macOS where I do some of my work and maybe Windows if there’s interest. There are also more build systems and toolchains I’d like to test, including npm, Gradle and Bazel. Computing critical paths would also be a big improvement: we followed dependencies by hand in this post, but buildprof could help identify the chain of work holding up the build and automatically annotate it. I’ll probably tackle these as and when I need them. But if you try buildprof and there’s something you wish it could do, I’d be interested to hear about it . What people find useful will help me decide where to spend more time. I managed to satiate my curiosity, though I ended up spending rather more time on this than I expected. Along the way I built a tool I now want to have around whenever a build is taking too long. I know I’ll come back to buildprof the next time a slow build annoys me. If you have one of those builds too, give it a try . I’d love to hear what you find! In C and C++, a compilation unit is usually a source file together with its included headers . Rust compiles crates , which can be split into multiple code-generation units . Zig normally compiles a program’s Zig sources together as a single compilation unit . Bun’s Zig compiler fork supports splitting that into multiple LLVM modules, but its CI build explicitly selected one when LTO was enabled .  ↩︎ The Zig-era CI build ran its C++ and Zig compilation stages on separate Buildkite machines and passed their outputs to a final linking stage. My script ran those stages concurrently on one machine, waited for both outputs, copied them locally instead of transferring them over the network, then linked them. This should preserve the dependency graph, but due to the hardware differences and running both stages on one machine, resource contention would obviously be quite different. Also note that my timings are individual runs (albeit ones which were quite stable) while Bun’s reported figures are medians.  ↩︎ A process can do substantial work internally, including running multiple threads, without launching anything else. The process timeline won’t show that parallelism. To see inside a process, we need tracing from the program itself, as Clang and LLD provide in the examples below.  ↩︎ LLVM emits from its legacy pass manager, which LLD uses for code generation. The inlining and other IR optimization passes can appear before it, so this bar is not the total time spent optimizing a module.  ↩︎ In the earlier Full-LTO linker replay, 26,825 events with JSC symbols total about 209s. This is summed event time, not a measurement of JavaScriptCore’s entire contribution to the link. One example event takes 2.94s; its symbol demangles to .  ↩︎ Recording script . These timings are just for building Bun with the libraries already available; the WebKit and ICU rebuild happened beforehand and isn’t included. Of course, I could point buildprof at that build too, but that’s another rabbit hole… I did not rebuild a matching Full-LTO WebKit archive as a control, so I cannot attribute every second saved to the LTO setting alone.  ↩︎ buildprof currently supports compiler traces from Clang, LLD and nightly Rust.  ↩︎ eBPF tracing uses capabilities such as and , as described in the kernel’s capability definitions . ftrace provides separate tracing instances and PID filters , but these still need configuring and access to tracefs. also depends on the host’s security settings; containers may need additional permissions to allow tracing child processes.  ↩︎ Medians of five clean builds per mode on the same VM, with six build jobs. Measurement script .  ↩︎ It’s build-system agnostic : Cargo, Ninja, Zig, Make and most other build systems do much of their work by spawning processes, so we do not need to write a special integration for each one. It naturally includes custom scripts : This includes both scripts above the build system (repository setup, dependency fetching) and scripts underneath it (code generators, asset processors). We can follow the files between build steps : recording which files each process reads and writes lets us see which steps produce the inputs for others. This even works across build systems! The huge outlier in the initial Zig build vs the Rust build was the massive linker step which ran alone at the end of the build. Changing the LTO settings for just Bun was not sufficient as WebKit, a significant part of the build, still used Full LTO. Once I had done this, the Zig build dropped from twenty-four minutes to fifteen. Even after this, linking still took 7 minutes and the whole build 15 minutes. The overwhelming difference which remained was structural: Rust spreads compilation across >90 crates while the Zig build funnelled everything through a single module. Cargo timings works well for Cargo-managed builds, but cannot break down arbitrary work inside or see wrapper scripts above Cargo. In Bun, Cargo is only part of the build: the report I captured covered 1m51s of a 5m40s CI build. Clang’s gave us the detail inside a compiler invocation, but cannot show what the rest of the build is doing while Zig’s Tracy integration goes deeper still and is intended more for understanding the compiler itself. and can follow arbitrary processes through and , but show general process events rather than a build-oriented timeline. In C and C++, a compilation unit is usually a source file together with its included headers . Rust compiles crates , which can be split into multiple code-generation units . Zig normally compiles a program’s Zig sources together as a single compilation unit . Bun’s Zig compiler fork supports splitting that into multiple LLVM modules, but its CI build explicitly selected one when LTO was enabled .  ↩︎ The Zig-era CI build ran its C++ and Zig compilation stages on separate Buildkite machines and passed their outputs to a final linking stage. My script ran those stages concurrently on one machine, waited for both outputs, copied them locally instead of transferring them over the network, then linked them. This should preserve the dependency graph, but due to the hardware differences and running both stages on one machine, resource contention would obviously be quite different. Also note that my timings are individual runs (albeit ones which were quite stable) while Bun’s reported figures are medians.  ↩︎ A process can do substantial work internally, including running multiple threads, without launching anything else. The process timeline won’t show that parallelism. To see inside a process, we need tracing from the program itself, as Clang and LLD provide in the examples below.  ↩︎ LLVM emits from its legacy pass manager, which LLD uses for code generation. The inlining and other IR optimization passes can appear before it, so this bar is not the total time spent optimizing a module.  ↩︎ In the earlier Full-LTO linker replay, 26,825 events with JSC symbols total about 209s. This is summed event time, not a measurement of JavaScriptCore’s entire contribution to the link. One example event takes 2.94s; its symbol demangles to .  ↩︎ Recording script . These timings are just for building Bun with the libraries already available; the WebKit and ICU rebuild happened beforehand and isn’t included. Of course, I could point buildprof at that build too, but that’s another rabbit hole… I did not rebuild a matching Full-LTO WebKit archive as a control, so I cannot attribute every second saved to the LTO setting alone.  ↩︎ buildprof currently supports compiler traces from Clang, LLD and nightly Rust.  ↩︎ eBPF tracing uses capabilities such as and , as described in the kernel’s capability definitions . ftrace provides separate tracing instances and PID filters , but these still need configuring and access to tracefs. also depends on the host’s security settings; containers may need additional permissions to allow tracing child processes.  ↩︎ Medians of five clean builds per mode on the same VM, with six build jobs. Measurement script .  ↩︎

Programming Open Source Performance C++ Java

Michael Stapelberg 1 months ago

Debian Code Search: Fast TurboPFor with Go SIMD

This August, I accomplished what I wanted for many years: I deleted the last cgo dependency in Debian Code Search! This was made possible by Go’s recently introduced SIMD support, because now we can implement the TurboPFor integer compression format as efficiently — more efficiently, in fact, by using the newer AVX512 instruction set! — as the reference implementation. Debian Code Search (DCS) is a search engine that allows searching all the Open Source source code within Debian, with either literal search expressions or regular expression search queries. A search engine uses an inverted index: a map from term to documents containing the term. Each document is typically represented most efficiently by using an id, so the index consists of many lists of document ids. When searching, it is important to quickly decode these lists to answer the search query. However, there is a point of diminishing returns where the decoding speed, even though it can still be measurably improved quite a bit, no longer influences the overall query duration. From 2012 (its inception) to 2019, Debian Code Search used to use a small index format, and queries were fast because the index was kept entirely in RAM. In 2019, I implemented the new index format , which adds an on-disk positional index. For literal queries (78.2% of DCS queries), querying the positional index on disk is faster than querying the non-positional index in RAM. The efficient encoding of the TurboPFor format makes it possible to fit such an index on a mid-sized Hetzner server, which I rent with two 1 TB SSD disks. The optimized decoder of the C TurboPFor library is what made decoding fast at query time. If you want to dive deeper into the algorithm, see this blog post from February 2019: Motivation I have recently been looking into speeding up Debian Code Search. As a quick reminder, search engines answer queries by consulting an inverted index: a map from term to documents containing that term (called a “posting list”). See the Debian Code Search Bachelor Thesis (PDF) for a lot more details. Read more → If you want to learn more about the positional index, see this blog post from September 2019: Over the last few months, I have been developing a new index format for Debian Code Search. This required a lot of careful refactoring, re-implementation, debug tool creation and debugging. Read more → For many years, you had the following options for using SIMD instructions in Go: The C TurboPFor library has served us well, but Debian Code Search was always intended to be a project using Go, so I would prefer it if I did not have any C code in the project. Go 1.26 (released in February 2026) introduced the package: Go 1.26 introduces a new experimental package , which can be enabled by setting the environment variable at build time. This package provides access to architecture-specific SIMD operations. It is currently available on the architecture and supports 128-bit, 256-bit, and 512-bit vector types, such as and , with operations such as . The API is not yet considered stable. — Go 1.26 Release Notes For my 2019 TurboPFor analysis, I implemented , a native Go teaching decoder (without any SIMD), because I find Go code easier to follow than C code, especially optimized C code. My implementation was intentionally not optimized so that the code was easier to study. The TurboPFor format/algorithm has a vector-optimized part: bitpacking comes in a scalar variant ( ) and a vector variant ( ) , where the vector variant is used for full blocks (256 values) and the scalar variant is used for remainder blocks (< 256 values). When Go 1.26 was released, I used Claude Code to explore whether my native Go decoder’s function (for the vertical vector layout) could be implemented using Go SIMD, and the answer was yes, it was possible and it was faster than without SIMD, but not quite at the level of C TurboPFor. If you let Claude Code try for long enough, it eventually finds enough optimizations (about 10) to match C performance. I don’t want to vibe-code Debian Code Search, though, so I figured I would find some time to review the SIMD code at some point and see if I could implement something similar myself. Before I found enough time and motivation to complete said review, I discovered that to not regress real-life query performance by more than 10 to 100 milliseconds (which seems acceptable), I don’t actually need to add SIMD code to my teaching decoder at all; it would be sufficient to reduce allocations in my teaching decoder and specialize it per bit width. Encouraged by the possibility of using the optimized native Go decoder in Debian Code Search, I explored whether I could also implement a native Go encoder so that I could get rid of the C TurboPFor dependency entirely. The answer is yes, it is doable in a few days, and it isn’t even that much slower: Go is at 76% of C, see Debian/dcs commit . The goal I set myself at that point was to see if I could learn enough SIMD to optimize the native Go encoder such that its performance would match how DCS uses C TurboPFor (via cgo). Beating C TurboPFor was possible in 2-3 commits (SIMD and bit width specialization). To my surprise, Claude Fable 5 pointed out that the encoder’s block scanning could be done more efficiently using a technique called positional popcount, and that is another 2x speed-up ! 😲 To be clear: I am not saying the Go compiler beats C here. Certainly, the C compiler can also produce fast AVX512 code and can be used to implement positional popcount. When comparing apples to apples, i.e. backporting the AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a little slower at ≈1.4x C. This spectacular result (much faster than what DCS had before) got me curious how far I could push the decoder with SIMD after all. I ended up matching/exceeding the cgo version here, too! The rest of this article explains a few classes of optimizations I encountered along the way. When I wrote my teaching decoder , I named its functions to match the upstream C TurboPFor library, but now I want to get away from names like — they make sense from the TurboPFor perspective, but for Debian Code Search, we can use cleaner names. Before writing any code, I audited how DCS uses integer compression / decompression. In Debian Code Search, we have the following usage patterns: For reading the index, we do keep the decoded s fully in memory, so we only need , a function that reads values ( ) from and returns how many bytes it consumed. For writing the index (both in partial indexing, and when merging), keeping the entire index in memory is prohibitively expensive, so we need a streaming API, for decoding and for encoding. Ultimately, I converged on the following API: This API (the decoder works similarly) allows us to process data in TurboPFor format without any memory allocations. The types are not safe for concurrent use by multiple goroutines. The zero value is ready to be used. For the streaming API, the result only stays valid until the next call. Before we can optimize anything, we need a working decoder and encoder. The decoder already exists: my teaching decoder. Next up, I needed an encoder. Writing a TurboPFor encoder has a delightfully simple starting point: You can encode all values at bit width 32, in little endian, at which point you only need to add a one-byte TurboPFor block header every 256 values and you’re done: Of course, this is a terribly inefficient compressor, so after the first commit, the real work starts: implement each block type until the compression matches the original C TurboPFor implementation (same output file size), or in other words: do the reverse of the decoder. I found it interesting to realize that the main work of the encoder is to scan the input values and choose the optimal block type, whereas the actual encoding itself is cheap in comparison. At this point, we can look at performance and see that the Go encoder is at 76% of the C encoder. In all honesty, I could have probably stopped here, but now that the milestone of a viable replacement was reached, I got curious to see how far it would be possible to push the encoder (how much work to reach C speeds?) and afterwards, the decoder, too. The microarchitecture of a CPU determines which instructions it provides, and that includes not just SIMD instruction sets (like AVX2), but also other useful instructions like (Leading Zero Count), which can be used to implement more efficiently, which the TurboPFor encoder needs to call on every input value to determine the ideal bit width. Let’s walk through how to set the microarchitecture level when using Go on 64-bit x86 (x86-64). Go uses the environment variable to configure the target compilation architecture, and I am using the value to select 64-bit x86 (AVX2 and AVX512 are instruction sets found on x86-64 CPUs). With , the architecture-specific variable configures the microarchitecture level for which to compile and Go 1.18 introduced these 4 different levels : (default): The baseline. Exclusively generates instructions that all 64-bit x86 processors can execute. : all v1 instructions, plus CMPXCHG16B, LAHF, SAHF, POPCNT, SSE3, SSE4.1, SSE4.2, SSSE3. : all v2 instructions, plus AVX, AVX2 , BMI1, BMI2, F16C, FMA, LZCNT, MOVBE, OSXSAVE. : all v3 instructions, plus AVX512F , AVX512BW, AVX512CD, AVX512DQ, AVX512VL. In 2026, I generally recommend compiling with so that functions like are compiled into intrinsics ( ) instead of using a lookup table. For Intel CPUs, setting means your programs will only start on Haswell CPUs (2013) or newer; for AMD CPUs that means Zen 1 (2017) or newer. In this specific case (DCS), I am even compiling with . The microarchitecture level requires AVX512, which means AMD Zen 4, Zen 5 or newer (Intel’s story is… complicated). Luckily, both my main development PC (Zen 5) and the Debian Code Search server (Zen 4) are recent enough. Setting has little effect on Go 1.27 itself: the only change is that maps use one less instruction ( instead of ). But compiling with allows us to move one more feature check from runtime to compile time, see SIMD build tags . It makes sense to set the microarchitecture level in your benchmark setup so that you don’t measure the slow fallback implementations. I use in my . Go’s built-in package contains support for benchmarks which are written in functions of the form . The simplest way to run such benchmarks is , but I ended up configuring a few convenience targets, which write results to and compare against (the previous commit’s results, usually), using the very useful tool . The and units are custom metrics I am reporting from the various sub-benchmarks , which are arranged such that I can filter / report them with . The main encoder (and decoder) benchmarks compare 3 different implementations (cgo, Go, Go with the StreamEncoder API) with a number of benchmark cases that are designed to cover the different block types and contain a similar mix of values as what we see in Debian Code Search: Go has included excellent performance tooling for many years, see the “Profiling Go Programs” blog post (2011) for an example of how to use , a sampling profiler. This profiler can help track down which part of a program runs slow, or where memory allocations happen. Once you identified the slow part of a program, how do you know why it’s slow? To learn more about the specific bottlenecks your program encounters, you can consult your CPU’s hardware performance counters . For example, you could check the branch predictor counters to see if your program is slow due to a high number of branch mispredicts. On Linux, the tool is the best way to access the CPU hardware performance counters. A good starting point for working with is the documentation on “Top-down analysis with the perf tool” , which describes the optimization method that Intel established. In my , I set up two targets: The numbers are high level numbers that indicate how much work the implementation is doing. Reducing the number usually increases speed. To see the counters for each instruction (and source code lines), I use , followed by . A quick shortcut is , which directly shows the hottest function. Let’s first see how far we can get without reaching for SIMD instructions. (The examples are not necessarily in commit order, but cherry-picked for clarity.) PGO stands for Profile-Guided Optimization and is a feature that Go introduced as a preview in Go 1.20 (released in February 2023) and shipped as ready for general production use in Go 1.21 (released in August 2023). The idea is to capture a CPU profile that records where your program spends most of its CPU time, which you then provide to the Go compiler to give it more data to make better decisions. Most importantly, this way the Go compiler can inline functions much more aggressively than its usual heuristics allow, which does have a measurably positive effect in my series of optimization commits. Another optimization that a PGO profile allows the compiler to do is conditional devirtualization — but our TurboPFor code does not use any interfaces. My strategy is to enable PGO before doing any other optimizations, so that we have the full inlining budget available that PGO gives us, and can measure the effect of other commits clearly. Surprisingly, turning on PGO actually decreases our performance (-13% geomean), but a closer investigation reveals that we just got unlucky. Let me explain. Aside from inlining and conditional devirtualization, PGO also influences alignment: The Go compiler sets on the first block of a loop (the “loop body”) for all loops in hot functions (per the PGO profile), i.e. Go will insert up to 31 bytes of padding to make the block land on a 64-byte boundary. Documentation like AMD’s “Software Optimization Guide for the AMD Zen5 Microarchitecture” (2024, #58455) explicitly recommends aligning hot loops that way: […] for hot loops, some further knowledge of trade-offs can be helpful. Because the processor can read an aligned 64-byte fetch block every cycle, it is suggested to either align the start of the loop to the beginning of a 64-byte cache line […] Indeed, when compiling with to disable the alignment, performance remains as good as without PGO. How can the padding hurt more than help? The answer is: It’s not the padding itself! It’s a side-effect of the padding moving instructions to different addresses. In the unlucky arrangement, a macro-fused + instruction pair now ends up exactly on a 32-byte boundary . However, the Go compiler ensures fused branch sequences must never cross or end at a 32-byte boundary to fix Intel erratum SKX102 (discussion: Go issue #35881 ) by inserting s. This padding, unlike the loop alignment padding, is not free; these extra instructions slow down our otherwise dispatch-bound loops. Because the commits after the PGO enabling commit change the code, this unlucky situation is avoided for the rest of the optimization series (by chance). Memory allocations are quite expensive, at least in comparison to encoding/decoding integers, so I followed my usual strategy of first reducing memory allocations as much as possible. In my teaching decoder, whenever the code needed a scratch buffer, it would allocate it right then and there with : The Go compiler can turn calls into stack allocations, if is known at compile-time. But, in this case is not known at compile-time. We can verify that Go calls into the runtime ( ) by dumping the object code (assembly) with source annotated ( ): An easy speed-up was to avoid allocations through reuse (in ). In the DCS package (with the improved API design ), I ended up with a field in the type, which brings us from 773 Mval/s to 858 Mval/s on the debian-mix: Aside from the speed-up, avoiding memory allocations is generally nice in benchmarks because it removes the garbage collector from the equation and makes it less likely that your benchmarks get other processes OOM-killed on the same machine. In general, we want to make it easy for the compiler to understand as much as possible about our algorithm. Consider this implementation: Let’s think through what determines the iterations and control flow this function uses: With a bit of careful rearrangement, we can provide the compiler with both, a fixed number of input values (say, 32), and a bit width, both known at compile time. Why is this worthwhile? Because we can manually unroll the loop, let the compiler eliminate much of the repetition and get much faster compiled code as a result! Let’s first fix the number of input values to 32 and rewrite the loop to calculate the position offsets within instead of changing on each value (with ): Next, we want to specialize not just for 32 input values, but also for each of the 32 bit widths. Can we do better than hand-copying 32 times (= 8192 lines of Go code)? Yes, we can use Go generics to help us with the code generation! In Go, array types like (not slices like !) contain the length of the array as part of their type, meaning (an array of length 1) is a different type than . Instead of passing the bit width as a function parameter, we can declare 32 different types (one for each bit width) and recover the bit width (at compile time!) from the type system: When we instantiate with all 32 different types ( , , …, ), the compiler substitutes the type parameter and produces 32 copies of the function, which we can find in our compiled executable with names like . The “shape” of a generic type is based on its memory layout, so a shape for must be different than the shape for . Because the is now known at compile time, the Go compiler can generate close to the optimal machine code for each bit width, which we can confirm using . The code is branchless (after the one bounds check per 32 values) and aside from the loads and stores (from/to memory) consists only of shifts and bit operations, all with constant operands: Now we need to actually call from the general function: Encoding remainder blocks is quite a bit faster (full blocks use the vertical layout anyway): This performance win comes at the cost of binary size increase. In this case, the section (executable code) grows by about 20 KB and the section grows by another 26 KB. Definitely a price I am very willing to pay, but the case might not be as clear in all circumstances. Even without reaching for SIMD instructions, a TurboPFor implementation can be made faster by making it work bigger strides. Take this code from the teaching decoder which counts the number of exceptions by checking if each value’s bit is set in the exception bitmap: We can use the functions to count ones bits in the exception bitmap, 64 values at a time. For remainder blocks, the rest is processed 8 values (1 byte) at a time: uses a 64-bit register. For comparison, AVX2 SIMD instructions use 256-bit registers (= 8 ) and AVX512 SIMD instructions use 512-bit registers. In the following sections, we will first set up our build tags for conditional compilation to use a trivial SIMD instruction, then walk through an AVX2 and AVX512 SIMD kernel. Let’s assume we have the following scalar code: To increase throughput, we can use AVX2 instructions if they are available on the CPU on which the program runs, i.e. using runtime dispatch. We’ll first rename to (it’s now the fallback path): Next, we’ll supply two different implementations ( and ), the latter of which is selected when compiling for with (the latter will hopefully be dropped in a later version of Go). The variant just dispatches to the , which will likely be inlined: The variant assigns the global variable by doing a check and then jumps to the scalar fallback if , i.e. the CPU is too old: We can go one step further by conditionally compiling when is set to or higher (i.e. the build tag is set). As a practical example from Debian Code Search, we currently need the following checks / dispatches: In DCS, the effect is measurably positive, but small . First, here is the layout explanation from my 2019 TurboPFor analysis blog post : In regular (non-SIMD) bitpacking , integers are stored on disk one after the other, padded to a full byte, as a byte is the smallest addressable unit when reading data from disk. For example, if you bitpack only one 3 bit int, you will end up with 5 bits of padding. SIMD bitpacking works like regular bitpacking, but processes 8 little-endian values at the same time, leveraging the AVX instruction set . The following illustration shows the order in which 3-bit integers are decoded from disk: The scalar implementation uses an array of 8 to process 8 values at a time: The SIMD version also processes 8 values, but without a loop! One difference is that we no longer have the luxury of using for (holding rest and current bits); because AVX2 registers only fit 8 (not 8 ). Instead, we split into and . The SIMD version benchmarks about 3x as fast as the scalar version. Another significant speedup is to use generics for bit width specialization for this SIMD kernel so that becomes a compile-time constant and the compiler can generate better code. For my TurboPFor encoder, I implemented the same techniques as described above: Bitpack full blocks with SIMD (AVX2) Gather exceptions using SIMD (AVX512) Use generics to specialize per bit width These changes are sufficient to roughly match the cgo performance, but then Claude Fable 5 found another 2x speed-up on top of that ! The key observation is that once encoding blocks is fast, the preceding step of scanning the input values to decide which block type to use becomes the bottleneck. Here is the encoder’s main function, which first does one pass over the input values ( ) and then prices all different block types at all relevant bit widths (requires fast access to the histogram): I’ll show you a slightly shortened version of , the function which is the bottleneck: Let’s consider the following 3 example values to understand the resulting : The resulting exception count histogram would contain ( shortened to ): In words, this means that at bit width 10, we could encode all the values without any exceptions. But most values do not need 10 bits, so a bit width of 5 would be more efficient, but requires storing one exception. Encoding at bit width 4 requires 2 exceptions, and so on. The function above is intentionally kept simple for illustration. We can make it faster by moving the per-bit-width loop outside the per-element loop . The fast version still needs about 12 instructions per value. With SIMD, we can reduce this to by 8x to only 1.5 instructions per value! The trick is to turn each input value into its “smear mask” (imagine taking the first 1 bit and smearing it across the remaining positions). Here are the smear masks for our example: Turning a value into its smear mask is computationally cheap: Go implements (functions like ) by calculating . We can calculate the “smear mask” of a value with , i.e. starting with a 32-one-bits mask and shifting it by the number of leading zeros. Now, to obtain e.g. , we can count the 1 bits at bit position 4 of all input values. The instruction counts bits very efficiently, but it counts one bits within a register, so it counts rows, not columns. Counting columns is called Positional Population Count . I found the following papers that describe positional popcount with SIMD: To understand the AVX512 implementation of positional popcount, I found it most helpful to visualize an AVX512 register (512 bits, i.e. 64 bytes). The graphic below uses the layout, meaning it divides the register into 8 lanes of 64 bits (= 8 bytes) each. This illustration shows the whole process: how s are loaded into an AVX512 register (all 4 of its bytes, in sequence) and where we end up, i.e. the 32 positional popcounts: Let’s break down this process into its individual steps. First, we turn each loaded value into its smear mask as explained above. The vector instruction calculates (1 byte) of 64 bytes at once, but first we need to shuffle the bytes inside the register: in load order, we have a full (4 bytes), followed by another , per lane. First, we permute the bytes ( ) such that all the first bytes of each value end up in one lane (“transpose the bytes”): Next, we “transpose the bits” using the instruction, which sounds scary but turns out to be quite flexible for bit manipulation of all kinds. The instruction is also “the star of the show” in Go’s Green Tea Garbage Collector (2025). Here is the bit transpose, shown in the AVX512 register layout (see below for a different layout): I found it easier to understand the transpose step when arranging the 8 bytes of lane 0 from top-to-bottom (instead of left-to-right), because then it looks like a 90 degree clockwise rotation: Now we can use to count the bits in all 64 bytes at once: After all loop iterations (processing 16 values each) are done, we add the two groups (first 8 values, second 8 values) to obtain the 32 exception counts: Here is the Go code that implements what I described visually above: Have a look at the commit introducing positional popcount to DCS for the full code (including shuffle tables and ISA checks) as well as the detailed benchmark results. The SIMD optimizations I showed above beat the cgo TurboPFor library that Debian Code Search used before. When comparing apples to apples, i.e. backporting the AVX512 kernels and positional popcount technique to C TurboPFor, Go benchmarks a little slower at ≈1.4x C. Could we make my Go TurboPFor implementation even faster, to truly match the C speed? Yes! But also no. Let me explain: We could use more SIMD instructions to remove all code that still processes one value at a time. For example, in my encoder’s function. Or we could price all bit widths concurrently in . Or in the decoder’s exception apply code path. But all of these SIMD instructions make understanding (and changing) the code harder, so I am cautious regarding which ones I introduce. A big part of the performance gap is due to Go’s bounds checks . While it costs performance, bounds checking is great for safety, so I will not turn off bounds checking. The Go compiler eliminates a number of bounds checks when it understands it’s safe to do so. One optimization avenue could be to make the prove pass in the Go compiler smarter to eliminate more bounds checks. When doing mid-stack inlining (proposal #19348) (2017), Go sometimes needs to put instructions into the binary so that it can attach inlining markers. For dispatch-bound functions, these extra NOPs can measurable slow down execution. The Go compiler currently allows specifying the architecture ( ) and microarchitecture ( ), but not a specific CPU architecture (like AMD Zen 4). Therefore, CPU-specific workarounds for one vendor affect all the generated code. The specific one I encountered in my code is that the Go compiler emits before every to break a false-output-dependency from the Intel Sandy Bridge Skylake era, which is unnecessary on AMD Zen CPUs. I suspect that Go intentionally does not offer this level of customizability. After all of the above points are addressed, what remains is better code generation in specific cases. To illustrate what I mean, consider the example of incrementing a loop variable, where Go re-derives an index every time: Go: (3 instructions) clang: (2 instructions) Depending on the specific case, improving the compiler might be easy or prohibitively complex. Often, such improvements are hard to measure conclusively. Go’s SIMD support makes available — in Go code without having to resort to cgo or assembly — a powerful part of modern CPUs which allows speeding up the kind of computation that TurboPFor needs by an order of magnitude! 😲 I found it very valuable to use a coding agent (Claude Code, with Opus 5 and Fable 5 in this case) to help with the many tedious parts of such performance work (and still it took me weeks!). The LLM can read objdump output much faster than I can, can see patterns and correlations I might never identify, never becomes frustrated after a compiler error or runtime panic, and never runs out of patience to run one more experiment, as long as I give it measurable and reachable goals. The performance of the SIMD code which one can get from the Go compiler is pretty close to what a good C compiler like clang provides. The CPU performance counters show value decoding speeds of 7 instructions/cycle (IPC) on a machine where the maximum is 8 IPC. To me, SIMD support is a very welcome addition to Go. Hand-writing Go assembler code . This is only doable for small functions, for example is implemented with hand-written Go assembly (including AVX2). Generating Go assembler code with tools like Michael McLoughlin’s “Avo” . This is how uses AVX2 . While Avo generator code definitely is higher-level than hind-written assembly, it is still too close to assembly for my taste. Use a C library via cgo so gcc or clang compiles SIMD code. Debian Code Search used to use the powturbo/TurboPFor C library via cgo for the last 7 years. Partial Indexing: When a new package (or package version) enters Debian, all of its (text) files are indexed. If the package (hypothetically) contained only with , we would assign document ID to and store in the partial index that trigrams , , , , etc. are all found in doc ( ). Full Index Merging: The many thousands of partial index files (for each Debian package) are combined into a small handful of large index files: When searching, it would be expensive to consult thousands of indexes. To merge multiple partial index files into one larger index (which can then be efficiently queried), we need to re-encode the partial index files: what used to be document ID in the partial index might be document ID in the full index. Querying (searching): When users enter search queries, these queries need to be answered as quickly as possible. The relevant entries in the full indexes are decoded (in parallel). The TurboPFor bitpacking block type ( bitpacking implementation commit ) encodes a bit stream of variable bit width (where the bit width is in range ) in little endian byte order. By scanning all values and choosing the smallest bit width that allows representing all values, this technique saves disk space (compresses). The bitpacking with exceptions block type ( bitpacking with exceptions implementation commit ) determines two bit widths: one for values, the other bit width for encoding exceptions. This allows choosing a lower bit width (that does not cover all values) compared to the bitpacking block type. A bitmap encodes whether a value has an exception or not. The bitpacking with VB exceptions block type ( bitpacking with VB exceptions implementation commit ) is a variant which does not use an exception bitmap and encodes exceptions using a variable byte integer encoding. This is more efficient when there are few exceptions (less than 20) or the exceptions are very different in bit width compared to the other values. Lastly, the constant block type ( constant implementation commit ) stores just one value on disk. This is useful for all-zero or all-one blocks, for example. The number of input values ( ), but not their actual value. The bit width to pack into ( ). Bitpack full blocks with SIMD (AVX2) Gather exceptions using SIMD (AVX512) Use generics to specialize per bit width 2019: “Efficient Computation of Positional Population Counts Using SIMD Instructions” (Klarqvist, Muła, Lemire) introduces an AVX512 implementation using Carry-Save Adder (CSA) networks, but in my testing, that approach is slower for TurboPFor encoding. 2024: “Histogramming bytes with positional popcount (GF2P8AFFINEQB edition)” (Harold Aptroot) shared a C++ implementation illustrating the technique I am using. If you are not familiar with how C++ vector intrinsics look, take a look at this example. 2025: “Faster Positional-Population Counts for AVX2, AVX-512, and ASIMD” (Clausecker, Lemire, Schintke) improves performance over the 2019 paper, but cites the Aptroot blog post as “Future Work”, promising even faster processing for smaller inputs. We could use more SIMD instructions to remove all code that still processes one value at a time. For example, in my encoder’s function. Or we could price all bit widths concurrently in . Or in the decoder’s exception apply code path. But all of these SIMD instructions make understanding (and changing) the code harder, so I am cautious regarding which ones I introduce. A big part of the performance gap is due to Go’s bounds checks . While it costs performance, bounds checking is great for safety, so I will not turn off bounds checking. The Go compiler eliminates a number of bounds checks when it understands it’s safe to do so. One optimization avenue could be to make the prove pass in the Go compiler smarter to eliminate more bounds checks. When doing mid-stack inlining (proposal #19348) (2017), Go sometimes needs to put instructions into the binary so that it can attach inlining markers. For dispatch-bound functions, these extra NOPs can measurable slow down execution. The Go compiler currently allows specifying the architecture ( ) and microarchitecture ( ), but not a specific CPU architecture (like AMD Zen 4). Therefore, CPU-specific workarounds for one vendor affect all the generated code. The specific one I encountered in my code is that the Go compiler emits before every to break a false-output-dependency from the Intel Sandy Bridge Skylake era, which is unnecessary on AMD Zen CPUs. I suspect that Go intentionally does not offer this level of customizability. After all of the above points are addressed, what remains is better code generation in specific cases. To illustrate what I mean, consider the example of incrementing a loop variable, where Go re-derives an index every time: Go: (3 instructions) clang: (2 instructions) Depending on the specific case, improving the compiler might be easy or prohibitively complex. Often, such improvements are hard to measure conclusively.

Go Performance C C++

Xe Iaso 1 months ago

It took a year to ship WebAssembly in Anubis

After a year of work, hundreds of commits, 5 generations of pull requests, dozens of tests, rewriting part of Anubis in Rust, the first compiler bug of my career, and at least three times making my tower run out of ram I think I have finally done it. The next version of Anubis will ship with WebAssembly-based proof of work checks that admins can enable in their thresholds or bot rules: This makes Anubis challenges use a memory-hard proof of work function ( argon2id ) instead of just a CPU hard one. It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead. Overall, this project taught me a lot of things about how WebAssembly works in practice, what the rough edges are when operating on the bleeding edge like I am, and led to the first genuine compiler bug of my career. Today I want to take you through the journey and process behind adding these WebAssembly based proof of work checks and all the problems that came up along the way. The big impetus behind wanting to use WebAssembly in Anubis is that there's a constant tension that's underlined a lot of the performance work I've been doing: phone CPUs suck and trying to balance equitability to phone CPUs with trying to punish scraper CPUs is a huge challenge. Note To be clear: I am tackling the "Anubis makes my phone overheat" problem. It's hard to just lower the difficulty for phones without giving attackers enough information to lower the difficulty for scrapers. If you want to contribute performance data so that I can improve my classification process and/or target future development, please get in touch with me . I want to make things better but can't without data. I am actively testing with a Moto G8 Power alongside my normal testing steps. Using WebAssembly here means that the binary will run faster, which will make Anubis go away faster, which is what we all want, right? One of the other big advantages of this setup is that it lets clients and servers run the same binary in order to solve and validate challenges. Note that I didn't say the same code, I said the same binary . This means that the client and server are in lockstep, much like the CIC chip in the NES . This also does mean that a bug in the shared code means that the client could send an invalid value that the server would accept as valid, but I don't think this is a practical concern until something like that happens. Right now the challenge has two implementations in Anubis: one in JavaScript and the other in Go . Updating it in one place means making the same updates in other places and also making sure that everywhere else that assumes how challenges work is also updated. Note Yeah, it's probably bad to have made those assumptions about how challenges work across the codebase. There's a lot of weird technical debt here that needs to be solved at some point. I expect that to be a fair bit of effort to untangle, it may end up with keeping the challenge only for tests when the WebAssembly based routes are working out well enough to make the default/only option. Making the same binary run on both the client and the server means that future experiments like per-client program synthesis can also proceed. I'll get more into that sometime in the future. Right now the core of Anubis is written in Go. Go's standard library HTTP server is surprisingly performant and is the key component of things like Google's HTTP frontend service. I have no intention to rewrite Anubis in Rust or anything as drastic as that. However breaking up Anubis from one monolithic binary into what amounts to a plugin loader is a lot more interesting from a maintenance standpoint. One of the main problems with Anubis is that the combination of rules I've set for myself (no CGo allowed and everything must cross compile to prod from my MacBook) means that updating any part of Anubis means recompiling all of Anubis across prod. Making Anubis be able to have parts of it downloaded and swapped out at runtime is really interesting to me because it means being able to adapt to threats as soon as the threat actors change behaviour . Also I figured out pretty quickly that Rust builds some of the smallest WebAssembly binaries that run hilariously fast, so I wrote all the WebAssembly code in Rust (no-std, wasm32-unknown-unknown)! Along the way this also lets me fix one of the bigger administrative problems with Anubis: challenges scale incorrectly with the challenge. Anubis uses string comparison to count the number of leading zero nibbles in a hash instead of the number of leading zero bits . This means that adding one (1) to the difficulty of a challenge makes it 1024 (one thousand twenty-four) times as hard to solve in the worst case. The original post has an interactive diagram that shows the difference in scaling at play. This was a mistake in retrospect, but if we're changing details about how challenges work here then we may as well wrap up the fix into that. In a normal world you'd not have about half of the restrictions that I have when working with Anubis. Normally you just write your code, compile it to WebAssembly, and then make sure it runs on modern browsers. This is nice and simple. I wish I could live in this world. Comparatively, this is what it's like getting all of this working across browser versions, platforms, and so many other things: Anubis supports Chrome 75 and newer. I wanted to reduce the support range to not have to deal with Chrome that old (the feature difference between Chrome 155 and Chrome 75 is absolutely massive ), but there are a lot of smartphones, smart TVs, and other electronic devices running Android out there that are just marooned on Chrome that old with no path to upgrade it . As a result, I gotta target Chrome 75 for features, but out of an abundance of caution and to make sure that things are generally compatible with browsers older than Chrome 75 (eg: old iOS releases), I target my JavaScript to Chrome 66 as an abundance of caution. I haven't been able to test with iOS or Android versions of the same vintage as Chrome 75, so a lot of this is aspirational backwards compatibility. I sure hope it's good enough! When you're doing this kind of work, you're effectively writing an operating system kernel that makes the WebAssembly guests run as processes. Any calls you pass to the WebAssembly guest are effectively system calls into the host. Normally I'd love to be able to reuse the WebAssembly Component Model so that a lot of the "hard part" is done for me. WebAssembly Components make you define your calls in packages that contain interfaces with functions in worlds. Here's an example of the kind of WebAssembly Interface Types world Anubis would need: If this worked (I wrote this all out in one go based on how the current handcrafted ABI works and have not tested it, sorry), this would describe the shape of the API that we need for doing the proof of work operations. At the time I wrote the API/ABI that Anubis uses, the main tool for generating the server-side bindings for WebAssembly Component Model stuff in Go gravity didn't support passing records (structs) or bytestrings ( or / ) from the host to the guest. As such, I had to do it by hand. So given that we can't do it the "right way", we have to do it the "bad way". Also given that no matter what I pick for this I'm going to be "wrong", I just decided to treat the WebAssembly modules as dynamic libraries that just happen to use WebAssembly as an implementation detail. There only needs to be three buffers that are read from / written to, so let's just focus around those: Challenge modules also expose two entrypoints: Challenge modules also import from the environment so that they can periodically report their hash rate back to the frontend. This allows users to see a progress bar based on how long it should take to finish the process. This works enough for now. It'll be interesting to see how this falls short in the real world! Honestly, the first bit of this took a few days at most. Most of the hard work was making sure that pointers, offsets, and whatnot were all wired up correctly so that the browser worked the same way as the server. My experience building and messing around with many WebAssembly runtimes and egregious hacks meant that making it was really easy for me. The devil came out in the details. Here are all the things that came up while I was working on this. All of these problems added up is the sole reason that this took a year instead of a week. Early on in development I found out that WebAssembly has a SIMD extension . SIMD stands for Single Instruction Multiple Data and is a family of instructions that let you do operations on multiple values at a time. This gives programmers data-level parallelism (this is distinct from multi-threading) so that things like hash calculations and MP3 decoding can be done faster than they would be if each operation had to be its own instruction. CanIUse considers WebAssembly SIMD to be "baseline" (supported by browsers newer than Chrome 91), but I have a lower version bound of Chrome 75. However the benefits from SIMD on mobile devices are so drastic that it's worth having two builds of the WebAssembly code: one with SIMD and one without it. In the browser it dispatches which version to use by using wasm-feature-detect to probe WebAssembly functionality by trying to parse/run trivial minimal programs that exercise those features. Hopefully I don't need to add an additional build into this process, but there is nothing in the tooling that would prevent it! One of the big things that blocked this shipping for so long was not having an escape hatch of some kind to allow clients that disable WebAssembly by policy to get through the gate. In my experience most clients don't have JavaScript enabled but WebAssembly disabled, however there are a few notable usecases that forced my hand: iOS Lockdown mode and GrapheneOS' Vanadium 's default configuration. This combination of factors means that there would need to be another implementation of the proof of work code in JavaScript that would actually execute the number crunching. I don't want to make another implementation of the proof of work function (the entire point of this is to only have one implementation!) so while I was browsing around I came across the legendary talk The Birth & Death of JavaScript and got a horrible idea. What if you just compiled the WebAssembly to JavaScript? Would that even work? It's "just" turning one Turing machine into another, right? How would it fare in practice? Turns out I'm not the first person to think about this! The team behind binaryen have made this escape hatch in the form of wasm2js which takes WebAssembly binaries and produces moderately cromulent JavaScript in response. One of the main downsides is that the generated binaries tend to be rather large. For example consider this simple WebAssembly module that exposes a function that adds two numbers together: Seems simple enough, right? Here's the JavaScript that generates: As you can imagine, this only gets progressively worse as you end up making the Rust standard library get compiled from WASM to JavaScript, and even worse when you actually get hashing functions into the mix. The end result is probably very optimized when you run it through a JIT, but given that this runs on an interpreter it's probably gonna be slow no matter what I do. Sorry! I tried! This ended up working fairly well in testing, but I tried building it in GitHub Actions and ran into an issue. I noticed that wasm2js was packaged in Ubuntu and that the version in Fedora worked fine but the version in Ubuntu did not and threw an obscure error message about not understanding the tail call extension that Rust was using for some reason. I ended up bisecting versions of binaryen by downloading a tarball, building it from source, and then seeing if the result of running it on the Anubis WebAssembly modules worked in a browser. I ended up selecting Binaryen version 128, the newest version at the time. It was new enough that most of the distributions that package Anubis don't have that version of Binaryen packaged. However I didn't want to make my life more complicated by having some kind of conditional compilation step that would effectively tell end users "sorry, the admin is using an unofficial build that just so happens to not support your browser, please complain to them" because they'll just end up complaining to me. In my experience the kinds of people who run this exact combination of circumstances also tend to be the kind of people that have a wide variance in the level of kindness they display to the authors of open source programs that happen to be in their way. So I needed an escape hatch that would force build systems to use the exact version that I use in my builds. Then inspiration hit me as if Apollo himself sniped me from the heavens. We're dealing with WebAssembly here right? What's stopping us from just compiling the WebAssembly to JS tool to WebAssembly with some kind of reproducible build, committing that blob to the repo, and then moving on with life? I ended up finding a bug in LLVM around how it was iterating over exception handling blocks by the compiler iterating over them in machine pointer order. As a result each build would drift by about 29 bytes per build: This is what lead me to write I hate compilers as a combination blogpost/cry for help which made me realize this was actually an LLVM bug. I didn't instantly lean towards it being an LLVM bug until I figured out that disabling ASLR (via ) made the results consistent on the same host within the same boot. Honestly this is the first time in my career I've ever run into a compiler bug like this. When I do a lot of my work I usually work under the assumption that the compiler is bug-free and that my inputs are wrong somehow. As such, even thinking it could possibly be an LLVM bug was just outside of the realm of possibility for me. Once that LLVM bug got fixed and a new version of wasi-sdk with that fix got released, I was off to the races, updated my build of wasm-opt/wasm2js, made my build scripts run it with wasmtime (alongside a wazero-based fallback process that would be slower, but did work enough) and everything worked out. Every time Anubis builds the WebAssembly in CI it uses the version of wasm-opt and wasm2js that ships in the repo to ensure that everything is as byte-for-byte deterministic as possible. Your build tools can't differ from my build tools if I ship you the build tools I use. Then we get into the other big problem that made this difficult: browser testing. One of the most common failure modes of Anubis is that someone uses some browser that I don't test in CI and then things don't work with it. I'm tired of installing 50 different browsers on several machines to test things and I have gone through so many throwaway VMs that I'm sure it's reduced the lifetime of my SSD. Note Yes, I really have been testing Anubis by hand in god knows how many browsers. Why do you think it takes so long to tag new releases? I built a harness that I call "chromesweep" that lets me spawn many Googles Chrome (term c.f. Attorneys General, et.al) in their default configuration to try and hit a version of Anubis listening over HTTPS. Getting this far meant making a library of all of these browser versions. I've put that library up on Github at TecharoHQ/gubal in case it's useful for you. One of the other big problems I ran into while getting browser testing working was making sure that my Googles Chrome strictly stay within the bounds of my Kubernetes cluster's network. Chrome this old is actively radioactive and I want to treat it like the security threat it is. As such, I set up a strict NetworkPolicy to only allow it to access the Anubis instance under test and make DNS queries. I also wrap each Chrome pod in a microVM with Kata containers as an additional layer of security. Note It honestly terrifies me to think that I am putting more effort into securing these Googles Chrome than big AI companies are putting into securing their AI agent testing infrastructure . It literally doesn't take much to put a big dent into securing things! The current state of our industry boggles the mind. All I want is for Techaro's FelonyBench score to remain at 0, is that too much to ask? I also rigged the browser testing infrastructure up to a single slash command in GitHub pull requests. Doing it makes my office very warm so I try to avoid doing it when possible. This works well enough that it lets me move on to the next stage and has already caught something that lead to building all the JavaScript with the flag. I wonder if this is yet another case where making infrastructure for Anubis could result in that infrastructure alone being its own viable tech product. I run into a lot of those. In the process of doing that automated browser testing I found out that Chrome 75 had a weird error pop up when it tried to compile Anubis' WASM to native code: This also caused failures up to Chrome 100, so this signaled to me that something I was doing with my "strict MVP" build of Anubis' WASM wasn't in fact sticking to just the MVP features of WebAssembly. It turns out that the function referenced (probably somewhere in std::sync::Once ? I probably should have traced it down to the exact bit) was in the standard library. This surprised me because I assumed that building Rust code with CPU features selected would apply that to everything, including the standard library, right? No, turns out that when you download the component in rustup , that doesn't just download the standard library. To aid in cross compilation and I guess to avoid disk space waste, the Rust standard library is precompiled. This surprised me as Go typically has you recompile the standard library (and runtime for that matter) when doing normal builds and cross compilation. Note The actual issue here is that the stdlib function in question was compiled down to use reference types , which made references get stored as a table index instead of what MVP WebAssembly would put there. Chrome tried to read a null byte, got not a null byte, and then understandably exploded. I looked into the process involved for rebuilding the standard library twice: once with only MVP wasm features enabled and once with an "all yes config" like usual. Based on some research I did this seemed like a massive pain. However, I had gone through that effort to build reproducible WASI versions of wasm2js and wasm-opt. wasm-opt is a tool that lets you take compiled WebAssembly modules, optimize them, and more importantly remove features from them so they can run in older browsers. After a bit of hacking to make sure that the tools were able to run properly, I set up a Claude Opus / GLM 5.2 loop to fuzz various wasm-opt flags and make sure Chrome 75 could parse the output. I ended up with these flags: This strips away all the other WebAssembly features from the build like unwanted paint (the flag means "disable everything not in the original MVP definition of WebAssembly"). I think that it'd be safe-ish to enable reference types in the SIMD build (they were added before Chrome added SIMD), but it's not hurting anything to remove them so I'll just let cowardice win here. Either way, I threw the results into chromesweep and got a successful response, so win! If/when this comes to bite me I'll try and improve it. I'm pretty sure that this work isn't perfect , but at some point you gotta cut your losses, ship it, and then see where things fail to prioritize perfecting it. These issues include but are not limited to: Overall though, I'm hopeful that most of the worst parts of this can be solved. It would be nice if I didn't have to work what amounts to two full time jobs. I'm pretty sure that this is stable enough to ship as off-by-default in Anubis v1.28.0: Wuk Lamat . Based on the feedback I get from administrators and users, I'll enable it in the default configuration in Anubis v1.29.0. I hope this look into how Anubis is developed can give you ideas as to the scale and challenge involved. Making something like this is tireless and thankless work and it's really weird to see people talk about it in the same breath as Cloudflare or AWS' WAF. Have a good day all! Note AI was not used in the production of the prose of this article. I have my draft as a Google Doc so you can see exactly where and when I typed every word myself. The only use of AI was Claude Opus to help me make the visual bit/nibble diagram. The data buffer: up to 4096 bytes of challenge data. This is 4096 bytes so that it can (hopefully) land in its own 4Ki machine page. This is the only variable-length buffer in the setup, so it needs two calls: : return the pointer to the data buffer in WASM linear memory. This is used as the base address for copying data into the guest. : update the globally mutable "data length" variable to signal to the guest how much data was actually written into the data buffer. The combination of these two calls lets you treat that global data buffer as a slice. The result buffer: a challenge-defined buffer that usually has about 32 bytes of data. This is read out of the guest's linear memory when the challenge is done processing. : return the pointer to the result buffer in WASM linear memory. This is used as the base address for reading out of the guest. : return the length of the result buffer (a compile time constant based on the needs of the challenge, but the runtime can't know that). The verification buffer: a challenge-defined buffer that usually has about 32 bytes of data. This is written into when the server is validating a challenge. : return the pointer to the result buffer in WASM linear memory. This is used as the base address for reading out of the guest. : return the length of the result buffer (a compile time constant based on the needs of the challenge, but the runtime can't know that). : the main entrypoint for browsers. Given the data loaded into the challenge buffer, hash it in a tight loop until you get a solution that matches the difficulty. : the main entrypoint for the server. Given the data loaded into the challenge and verification buffers, ensuring that one run through the hashing function produces a result that both meets the difficulty demands and exactly matches what the client sent. The wasm2js flow doesn't currently have a way to update the progress bar with its import stubbed out. I have no idea how to properly wire that up with the constraints of my runtime, but I'm sure I can figure it out eventually. The WebAssembly that's shipped with this flow is ridiculously performant. This may mean you need to adjust the difficulty to compensate for this. Oops! Sorry! This is better on mobile phones, but I'm working on a mobile request classifier that will use a combination of IP address reputation, TLS fingerprinting, and other signals to determine if a request is from a phone and give it the appropriate amount of grace. This is hard. Please contact me if you have ideas on how to make my current prototype better. I need some way to dynamically rotate out challenge programs at runtime instead of just at compile time when I work on Anubis. Eventually I hope to have this pull WebAssembly files and supporting bundles from OCI/Docker registries, but I need to do a lot more experimentation with this before I can conclude how good/bad of an idea this is.

Performance Security Go JavaScript Rust