Posts in Ruby (20 found)
Sean Goedecke 5 days ago

Software's centaur age may last decades

We are currently living in the age of centaurs (2022-20??). Right now, human engineers paired with AI coding systems are more effective than either by themselves. For future generations of software engineers — human or otherwise — this will be the most significant fact about the current period. As in chess’s own centaur age, people are always ready to claim that the centaur age is over and the age of AI supremacy has begun. But the age of centaurs in chess lasted about twenty years, and previous centaur ages have lasted far longer . This should be encouraging to many of us. If software engineering goes along similar lines, we may be able to ride it out another decade or two: enough to make serious plans about what comes next, or to finish out our careers entirely. It began with GitHub Copilot, which launched in 2022 as AI-powered autocomplete. A confident engineer who knew exactly what they wanted could tab-complete their way through lines of code and sometimes entire functions. Being able to chat with language models like GPT-4 provided the next upgrade. Although the models would still get things wrong, it was helpful to have a generalist on tap for areas that were a complete mystery. Coding agents rose to prominence with Cursor’s “agent mode” in 2024 and Claude Code in early 2025. Early agents needed close supervision to prevent them from going off the rails, but were still helpful to automate straightforward tasks. Current agents — after the November 2025 release of Claude Opus 4.5 — are reliable enough to run entirely unsupervised. Their output still needs human review, but the kind of mistakes they make are more mistakes of alignment than regular bugs: not matching the organization’s technical values, over-engineering certain areas and under-engineering others, deciding where to spend effort, and so on. It is no longer possible to credibly argue that an unassisted software engineer can beat a centaur team. The value provided by coding agents is just too great: not just the speed of producing code, but the speed of review and testing. However, we are not yet at the point where an unsupervised coding agent can beat a centaur team either. All the impressive work I’ve seen from LLMs has had a competent human software engineer in control. Centaurs are on top. Would we expect our centaur age to be shorter? On the one hand, chess is a much simpler problem than software engineering, and is more amenable to learning methods like self-play. On the other hand, software engineering is massively more lucrative to solve, and many orders of magnitude more funding is being plowed into the problem than was with chess. On the third hand, “solving” software engineering means creating much more software, which increases the total work-to-be-done, so even if the relative human share of that work is shrinking, the overall amount of human work might grow. On the fourth hand, the level of generally-capable AI needed to address software engineering will have huge effects on other fields, the ripple effects of which are likely to negatively impact software engineering jobs (for instance, crashing the economy, widespread social unrest, and so on). On the fifth hand, we might think that chess had an unusually fast centaur age. Previous centaur ages lasted much longer. In knitting, where a human and a knitting-frame could outcompete any hand-knitter, the centaur age lasted two hundred years . On the sixth hand, technological change is moving ever faster . I could probably find a seventh, eighth, and ninth hand if I wanted to 1 . But overall I think we have no real idea of how this is going to turn out. Chess is the most recent example of a centaur age we have, and it’s attacked by the same AI-based forces of automation as software is. It’s thus reasonable 2 to make a default assumption that the length of software’s centaur age will be similar to chess’s. Suppose I have convinced you that human-AI partnerships will be the dominant force in software engineering for at least the next decade. What then? Don’t quit! Don’t give up on software engineering and go become a carpenter, or open a cafe, or try to get a retail job. Unless you are somehow really, really passionate about doing those things, it is going to suck : unrewarding, grueling work that you are not good at. A decade of software engineering work is long enough to take your time and think seriously about your next steps. Lean into the partnership. A guaranteed way to fail in a centaur age is to refuse to become a centaur. The idea that AI is a passing fad was a reasonable position to hold in 2023, but we’re way past that now. The AI financial bubble popping will not mean the end of AI . It’s here to stay. Think hard about what value humans can still add. This is going to change over time, and getting it right is the difference between being an effective (and employed) centaur and being a meat proxy . In 2023, technical expertise was the primary source of human value. Today, I personally think it’s going to come down to alignment , but this is still early days. I’ve also seen arguments for taste, though it’s hard to define that non-circularly. We need a lot more engineers thinking through this problem. It’s a common fear response to try and skip ahead to the worst-case scenario. But those of us working in the age of centaurs should avoid that. We need to think about how the job of software engineering functions now , not how it might function in twenty years when AI has completely transformed the landscape. People are too quick to declare that we’re in the software engineering apocalypse. We can certainly see it from here, but we’re not there yet. The spectre of full automation can loom on the horizon for decades. If you panic and jump ship early, you could be giving up on an entire normal-length career. Maybe: AI as fundamentally different technology; diffusion of even radically better technology is slow in practice and bottlenecked by organizational friction; human software companies will get outcompeted? I don’t really know what I think about these arguments. This move should be familiar to anyone with a philosophy background. If you have one argument and one counter-argument, the most recent counter-argument is probably right. If you have a stack of arguments and counter-arguments going back a thousand years, the most recent counter-argument should be treated with suspicion: better to take in the whole debate holistically and judge which side is more compelling to you. Maybe: AI as fundamentally different technology; diffusion of even radically better technology is slow in practice and bottlenecked by organizational friction; human software companies will get outcompeted? I don’t really know what I think about these arguments. ↩ This move should be familiar to anyone with a philosophy background. If you have one argument and one counter-argument, the most recent counter-argument is probably right. If you have a stack of arguments and counter-arguments going back a thousand years, the most recent counter-argument should be treated with suspicion: better to take in the whole debate holistically and judge which side is more compelling to you. ↩

0 views
DHH 1 weeks ago

Over my dead pencil

Two weeks ago at Rails World, I told my fellow programmers that it's time to put down the pencils. We're not going to write the vast majority of code by hand any longer. Coding agents have gotten so good that it's simply not an economically viable enterprise going forward to have humans typing out lines of Ruby, Rust, or C++. When I polled the room for how many people were still writing a material amount of code by hand on a weekly basis, only a handful put up their hands. So they all knew this, but it still came as a shock to many to actually hear it. And I get that shock. Programming has barely been a professional career for a generation, and now it's being completely transformed. That's unsettling, even for a group of people who've seen technologies come and go on a regular basis. But this isn't just another technology, because AI isn't just another tool. It's far more like gaining some new coworkers who are exceptionally good and fast at a wide range of tasks, but still need a bit of help with some, and whose choices you might occasionally disagree with. From a certain angle, that looks like competition. Especially when it was barely five minutes ago that programmers were being treated like precious priests whose incantations were necessary to yield even the most low-hanging fruit of the computer. This perceived threat of disintermediation is clearly hitting some programmers hard. But I don't think anyone who's willing to lean into the future, make the most of this intelligence explosion, and apply their skills to what comes next needs to worry much. We're about to see an absolute bloom in software development as the price of development plummets and everyone realizes how much automation we still have left to do in this world. I think the only programmers who have to worry about this change are those who refuse to embrace, or drag their feet on, the progress brought by the age of agents. It's fine to be skeptical about the areas where our new clanker colleagues still occasionally get it wrong, but refusing collaboration is not a viable career path for almost anyone. Don't go down with the pencils. There's so much to build. We need you.

0 views
Brain Baking 1 weeks ago

Favourites of September 2026

The calendar claims it’s October already but the temperatures are still in denial. I can’t remember ever experiencing the start of this month that warm. It’s getting scary! See what I did there? Because of October? Spooky month? Scary? No? Worth a shot. What happened in September? More letter writing! The snail mail project is going well. I keep track of the amount of outbound letters: the grand total for the last two months is 20 handwritten letters. If you’d like to chime in, please let me know so we can become pen pals! More than 50% of these offline conversations fizzle out after one or two letters so it’s only the die-hards that are willing to invest some time that get to see the best I’ve got. Previous month: August 2026 . Contrary to the summary of August that listed two board games, I have finally picked up video gaming again after a summer holiday hiatus—which in case of a lecturer is two months. Yup. Come join us! I also started (re)playing some older Castlevania games in anticipation of Belmont’s Curse . The Telwynium Book 1-2-3 is an episodic eighties Sierra-on-line inspired high fantasy game crafted by the same person that gifted us the The Drifter experience. They’re free on and will be bundled as one big adventure when the final episode releases somewhere in 2027. I enjoyed playing through them but there’s nothing especially remarkable here. Castlevania: Rondo of Blood does not need an introduction. When it comes to the release window and the platform it was released on, is the black sheep of the ‘Vania family. Luckily, it got a remaster on the PSP although I’m not a fan of those graphics. Review to follow soon. The venerable Jeremy Parish recently worked through Rondo as part of his Turbo Works Gaiden 01 . If you’re interested in the weird history of this game, I recommend you watch his video instead of reading my crappy review: Castlevania: Dracula’s Kiss , also known as Dracula X in the US, is a “port” (although not really) of Rondo of Blood for the SNES. It’s… not good. The “NES Hard (TM)” part here borders to the unfair, the level design is bland and the best parts of the original didn’t make it to Nintendo’s console. Castlevania: Bloodlines is a very good 2D action game that feels a tad different—perhaps mostly because you’re technically playing it on SEGA’s MegaDrive? It also released a year before Vampire’s Kiss here in Europe. The soundtrack took the obvious hit but is still surprisingly catchy. I started playing Eastward as a break from ‘Vanias. The pixel art and music is amazing! And so is the first chapter but then it suddenly changes pace. The endless blabbering that even the “skip dialogue” button can’t fully fast-forward is a bit painful. I’m 8 hours in at the end of chapter 4. I still want to know where this postapocalyptic strange tale is going; but wow this game has pacing issues… Related topics: / metapost / By Wouter Groeneveld on 2 October 2026.  Reply via email . Speaking of Rondo of Blood , I liked Indie Gamer Chick’s review and analysis of the game. This one’s from 2019 but I didn’t know about it yet: Piers Cawley ran a micro bakery on Emacs and PostgreSQL ! Lyteforce Games writes about Combolands , an indie game that he summarises as “ Balatro with buildings ”. I have yet to check it out. The Ruby/Rails world has a big problem: DHH. What to do when your favourite programming language has a fascist problem? I feel for every Rails fan. Please stop the ridiculous Omarchy investments ! The list of fashware keeps on growing and growing. The amount of people I know—including myself—truly being excited about software development keeps on shrinking and shrinking. See also Chad’s I am retiring from tech to live offline . Perhaps Chad wants to write me a letter. You know, using a thing called a pen that emits not HTTP return codes but ink? Nick also writes about the end of a craft . I think you can see this one coming. More AI drama in the KDE world as explained by Nate. Not a blog but a very cool online fountain pen catalogue by George Starcher. Notchoc rebooted their blog and started using Makko . It’s always interesting to see new small tools like these pop up. I don’t feel the need to change my setup just yet. Jason Murphy talks us through a pen nib swap: switching one architect grind for the other . Toni writes something bang-on: having kids changes everything . I really liked Paul Taylor’s morning routine checklist system that leveraged Alfred to the max. Did you know that freeing up memory in C/C++ by calling does not necessarily free memory? Daniel Lemire explains why . Lee Reamsnyder writes about friendship and thanksgiving aptly calling it friendsgiving memories. In case you need to quickly look up a Race for the Galaxy card: use the RFTGPics service . Tourist World for the win. I had to jump through a few more hoops than usual to get SDKMan to work with the Fish shell .

0 views
The Jolly Teapot 2 weeks ago

September 2026 blend of links

Some links don’t call for a full blog post, but sometimes I still want to share the good stuff I encounter on the web. Talking Watches: Pusha T On Why Rolex Is Worth The Wait – Let’s be real, I’m sharing this video mostly for the perfect coffee brand name Pusha T is launching with Lavazza (and promoting in this video). Bonus points if you guessed it. Repairing a 100-Year-Old Camera – Beware, this video will take 60 minutes of your time, and you won’t get those precious minutes back. (via Anthony Nelzin-Santos ) Lucky Notes – I recently wrote about this little app, but it deserves a spot in this series . If you were wondering what the cutest notes app is, there you go. 8 years and done – “ Now, I find it increasingly difficult to find topics on which I can offer thoughts worthy of other people’s reading time. (Truth be told, I probably haven’t been able to come up with that kind of content for a while now, but I soldiered on anyway.) ” Always sad to see a blog end, but this quoted part really resonated with me. (via Kev Quirk ) Meanwhile – Meanwhile, Daniel Benneworth-Gray relaunched his blog, as a perfect complement to his excellent newsletter . Pros and Cons of Using Personal AI Agents – Another instant classic. Muse Is What Meta Means by ‘Personal Superintelligence’ – “ Perhaps I am not enough of a dutiful consumer to be so amazed about a robot that can buy things for me. That is not to say there is nothing I like […] but there is a huge disconnect for me between the amount of technology here and what it could actually do for me in real life. ” I feel a lot like Nick Heer about this, and I would even go further. The one thing that is bothering me about this whole agentic lifestyle is the idea that tasks and chores are on the same level. And what do these people do with all this alleged extra time anyway? One Day on Rails – Pretty quiet until four in the morning, then captivating. (via 82MHz ) 100 unusual crisp flavours, ranked – Now, this is unexpectedly complex investigative journalism. Why Am I Left-Handed? – “ Sometime after the emergence of the genus Homo, 2.8 million years ago, evolution coded a preference for right over left. Of the various speculative theories for the emergence of this extreme preference, one that seems plausible to me points to our unprecedented capacity for violence. ”

0 views
Andrew Atkinson 2 weeks ago

Let's Get Down to Business with Rails and PostgreSQL

I’m happy to share that I was a guest on the Rails Business podcast and the episode is now live. Here’s a recap of the topics we discussed. We spent most of the time discussing engineering work last year related to scaling the database workload for peak traffic at Aura where we use Ruby on Rails and PostgreSQL. I’m presenting this information in two places this fall, Rails at Scale Summit in Austin, TX, and the Postgres Summit in NYC. Besides the podcast, you can get more details in these posts: If you’re not able to make those events or prefer to listen to a discussion of the content, then the podcast will have a lot of the same information. Besides the work at Aura, we also checked in on my book High Performance PostgreSQL for Rails , which has now been available in print for two years. Ryan asked about what’s next for the book, and it’s a difficult question to answer. With the availability and quality of AI tooling for software engineers, I don’t see developers buying books as much. With fewer prospective readers, the ROI is less attractive for authors and publishers to commit to the resource-intensive process of creating a book. I’m grateful to have had the opportunity to write a book before the rise of the AI tools. I’m still getting positive feedback from my book which is very fulfilling. Recently I met Madison Sites who’s a reader, and they shared feedback that they’re reading the book as part of the WNB.rb book club . Madison said the book has been helpful for building vocabulary and familiarity with database concepts and implementation details, and that’s helped with database work and better comprehension for conference talks about similar topics. I appreciated hearing about that! We segued into how we’re all using AI tools like Claude in our jobs as software developers, whether it’s writing or reviewing code, doing architectural planning, or myriad other tasks. One of my favorite use cases has been generating infrastructure as code with tools like Terraform. One of the main challenges I’m experiencing is on the review side, what’s expected when it’s AI-generated code in terms of my own comprehension, and how to still provide a quality review given the ambiguity around how much comprehension is needed from me. I also find it challenging to navigate team communication dynamics with AI-generated comments. One concept that’s resonated with me is: don’t be a meat proxy , make sure to do self-reviews and add some value in the chain. I hope you can take a listen to the episode, and please let us know if you have feedback. Thank you! Listen to the Episode 👉 Andrew Atkinson on Rails Business Podcast From Christmas Outage to #1 App Store Ranking: An Aura Frames Postgres Scaling Retrospective Scaling Rails at Aura Frames: Splitting to 8 Primaries and Reaching #1 in the App Store

0 views
Steve Klabnik 3 weeks ago

Arguing about arguments

Here’s some Rust code: In programming language jargon, we call and parameters and and arguments . Rust does not have many fancy features related to parameters and arguments. But other languages do. For example, here’s a function in Ruby , another language I’ve written a ton of: I used to love Ruby, and now I love Rust. And to be honest, and friends are a huge reason why I am so against Rust gaining these sorts of fancy features: they can be really concise, and beautiful to a certain kind of eye, but it also makes it really, really hard to know what all you can do with a function. You pretty much have to hope that someone has written good documentation, and while Rails’s documentation is pretty good, not every library is going to have it. To be clear about it, there are several things bundled together here: flexible argument types, options hashes, defaults, and keyword-looking syntax. It’s that whole style of API that shaped my skepticism, not argument labels by themselves. So, what exactly are all of these features? The simplest one is named parameters . As you might guess, this means that we can explicitly use parameter names at the call site: Named parameters are nice because they can give you more information at the call site, but also they can be more flexible. Some languages also let you supply named arguments in a different order: The downside of named parameters is that they can get quite verbose: I tried to not strawman this by showing an extreme example, but can take 23 different keyword arguments, and so this could get quite, quite long. Named arguments can be good when passing expressions, so we can understand something more at the call site. In the above, is much nicer than a bare , if you know HTTP well it’s kind of obvious what it is, but the additional clarity is nice. But it can be common to break truly complex examples down into variables that we end up forwarding to the function, and then it becomes verbose and redundant. As an example from inside FastAPI where is invoked directly with all of those options: We actually snuck in another two features into this example: how did we only pass five arguments into a function that has 23 parameters? As the name implies, optional arguments is a feature where you can choose to not pass in an argument, and default arguments lets you set a default when an argument is not passed. To go back to that example: We only passed in five of the 22 possible arguments. If we had to pass all 22 every time, that would be extremely verbose. So by declaring defaults for some parameters, we can leave them off of the call site, and not pass arguments for them, and get the defaults instead. We saw this in the Ruby: The are defaults for and : they’re an empty hash map. You’ll often find optional and default arguments come together: after all, if you don’t pass an argument in, what should the value of that parameter be? If your language has some sort of null value, that can be one solution, and it can feel like optional arguments without default arguments. But you can truly get one without the other with our next feature, function overloading: Languages that support function overloading allow you to define multiple functions with the same name, but different signatures. Which function gets invoked depends on which arguments you pass. This lets us truly have optional arguments without default arguments. In Java: Here, if you pass in a timeout, you get the first function, and if you don’t, you get the second. Instead of null, the parameter just doesn’t exist at all. This is a specific form of something called “function dispatch,” which basically means “hey when a function is invoked, what exactly gets called?” There are a lot of different ways to do this, and I don’t want this post to get into all of that right now, but there is a multitude of possibilities here: static vs dynamic, single vs multiple, predicate dispatch, pattern matching dispatch, prototype based dispatch… maybe I’ll write a post about that someday too. So, what are the pros and cons here? As I said above, Rust doesn’t support any of these three features right now. And they have been a persistent request from the community for years. Here is a 12 year old GitHub issue about this support, and it even includes something we didn’t talk about, “variable arity argument lists.” That gets me to the first thing that I don’t like about these features: there are a lot of them. And they are all intertwined. As we mentioned before, if you have optional arguments, you probably want default arguments. If you have named parameters, do you want to support only named parameters, or do you want to support both? It’s so tempting to just support all of them, and then you’ve added a huge mountain of complexity. In Rust right now, you have to write multiple functions with slightly different names: For the simple cost of “write two different names for your functions,” you get to keep the rules very simple: there is one function definition, and you invoke it by name. Done. Would it be nice to be able to say all of these: Sure, maybe. But the cost feels very, very high to me. If you’re used to these features, maybe it doesn’t seem that high, but I have a Ruby tattoo on my body. I’ve seen some shit. And so I’ve grown to enjoy Rust’s simplicity in this area. Yes, if you have a function with tons of parameters, you may need to write a builder instead, and that comes with its own form of verbosity: But the language itself stays simple, and the rules are easy. For a language that’s already perceived as complex, this has always felt right to me. And that’s why I’ve pushed back against the various proposals to extend Rust in this way all of these years. But recently, I changed my opinion on one of these features, and one alone, and I would consider it okay if Rust gained them. Maybe. Also, it’s important to note that I haven’t worked on Rust in years, and so my opinions are kind of irrelevant, but whatever: it’s my blog, this is what opinions are for. I think Rust could be okay with named parameters, but not optional or default ones. And what changed my opinion is coding agents, actually. I’m sorry, maybe you’re sick of me talking about AI, but hear me out. One of my beefs with named parameters, as mentioned above, is verbosity. But what you gain for that verbosity is clarity. I don’t think that verbosity and clarity are always the same thing; I like rather than / , and find it more readable, personally. But let’s take this function from the crate: This does really increase readability, no question about it. But for me, typing all of that out just isn’t really super worth the squeeze. But when Claude is gonna write it? I care a lot less. And the readability advantage for humans is even stronger for agents: the extra clarity of the text inline seems to (I haven’t run real evals on this yet…) be more helpful and use less tokens and calls when looking at the actual call site. If an agent were reading that named example above, it could tell that is the value without having to go look up the function signature itself. “What’s good for humans is true for agents” strikes again. This is also why I’m still against optional arguments: they deliberately hide what is being passed in to the function, and obscure clarity at the call site. What’s nice about them is that you have to type less. But I’m not typing myself anymore. So the benefit just isn’t there, but the drawbacks are. This also doesn’t solve the verbosity side of passing in our FastAPI example, it is still redundant. But one part of the downsides have been ameliorated. Notably, all of this reasoning also applies to the builder pattern, and it’s why I try to use builders sparingly in Rust. It can emulate named arguments, which is good, but it can also emulate optional/default/variable arguments, which is bad. The juice is worth the squeeze sometimes, but not all the time, and probably not even the majority of the time. So we’ve ameliorated the largest downside, but get to keep the largest upside. That seems good. But even if we like the feature, there are still lots of practical concerns with Rust specifically that make it not a slam-dunk to add them to the language. Now, “is this feature good in the abstract” is very different from “should this feature” be implemented? There are a lot of open questions and drawbacks to implementing named parameters. A simple one is that you don’t get rid of all of those previous functions, will exist forever. That’s a relatively minor downside to some of the harder problems. First, strictly speaking Rust parameters aren’t names, they’re patterns. A name is a simple pattern, but you can get more complex: This parameter doesn’t have a name, it’s a pattern that introduces two bindings. You could come up with some syntax that gives things external names vs internal names, but there’s that complexity revealing its head again. What happens when functions become values? I can write this today: What name do we use for ’s parameters? They can be named different things. Does this become an error? What names can you use when invoking ? Making this more complicated, Rust actually will accept this code right now: The names in are just documentation, as you can see they’re not required. Does this become an error? Do we require to change its names? The same goes for trait definitions, actually: Is this an error? Is it okay? Here’s another problem: evaluation order. Right now, Rust always evaluates parameters from left to right at the call site. So if we use the named version: If we keep left to right evaluation order according to the definition, but let you use named values in any order, suddenly some orders at the call site compile and some orders do not. Do we change evaluation order to do whatever compiles? That feels very dangerous and confusing. Rust does let you do this with struct fields: This is one reason why a lot of proposals for named arguments try to unify them with structs. As I said before, not insurmountable, just things that need to be sorted. Another small thing is a compatibility hazard: if we start naming parameters, renaming them breaks callers. Is that okay? Could you introduce aliases to help with this? That’s more complexity… These are just some of the problems that come up in the design space, and this post is getting long, so I’ll leave it at this for now. But proposals for doing this have to engage with all of this and more, which is why language design is difficult. I like to think that I change my opinions over time as the world or my understanding of it changes. So I thought it might be interesting to share a way in which an opinion that I’ve held for over a decade has changed a bit lately. I’m still not sure if named parameters are right for Rust, but I’m at least somewhat open to the idea these days. We’ll see what the language team decides if anyone does decide to keep trying to push this forward. And people do, for example, I am writing this in part because I saw this post go by today. This post isn’t intended to be support or a rebuttal of it, I actually didn’t even read it yet, it’s just that I’ve been thinking about this a lot lately (while working on Rue ) and seeing that go by pushed me to write some of the thoughts down. You should probably go check it out, as it’s an actual proposal in this space and this post is more of an exploration of the general idea and my own thoughts on it. Here’s my post about this post on BlueSky: Arguing about arguments steveklabnik.com/writing/argu... Arguing about arguments Here’s some Rust code: // define a function fn foo (x : i32 , y : i32 ) -> i32 { // body elided } // call it let z = foo ( 5 , 6 ); In programming language jargon, we call x and y parameters and 5 and 6 arguments . Rust does not have many fancy features related to parameters and…

0 views
./techtipsy 1 months ago

Trying out open weights LLM-s on hardware that I can actually afford

I’ve been aware of open weights local language models for over a year. They have gone from being a fun but useless novelty to something that with the right harness and hardware could get closer to becoming usable for day-to-day work. Not at the level of something like Opus 5, sure, but usable enough for doing useful work without a lot of headaches. After a Friday evening e-mail from a government across the ocean resulted in a (allegedly super capable) model being turned off, I was instantly motivated to give open weights models a try on hardware that I can actually afford. If you’re unfamiliar with hosting local large language models, then a lot of this may seem like gibberish. Don’t worry, I had the same reaction initially. With trial, error, and a lot of guidance, I kind of know what to keep in mind when playing in this area, so here’s a quick crash course on common terms and gotchas. Large language model: it’s just a file on your disk that contains the weights (which is a bunch of math). Model family: Qwen, Deepseek, Gemma, Kimi, Z.ai, you’ll see a lot of names like this. These are generally backed by AI labs or companies. Parameters: how big and “smart” it is. Bigger is generally better and smarter, but also requires more memory to run. Tokens: unit of work that these models work with. What you give to the model and what you get out is measured in this. Think of it like splitting a word in-to pie-ces wh-en wor-ki-ng wi-th te-xt. Tokens per second (tok/s): common measurement that indicates how quickly the model can work with text. GGUF: a common format tha the model file is packaged in. llama.cpp : a program that takes the model file and makes it usable over a chat interface or over an API, similar to what OpenAI, Anthropic and many others do. Harness: a program that interacts with the model on your behalf in a finite loop, and has tool calling capabilities, such as file reads/edits, web searches and access to external resources like your Google Drive if set up. OpenCode, Claude Code, and Pi are common examples. MCP: it’s like a bad REST API for LLM-s. Quantization: cool math that makes models smaller and easier to run at the cost of accuracy. Example: model is 80% smaller, but you get 75% of original accuracy as a result. MoE: mixture of experts (or a Simpsons character). The model is large, but only a small part of it is active at any point. Context: how much room you have for working with a model. Larger context requires more memory, but allows for more input data to be processed and is a requirement for longer agentic sessions with tools like OpenCode. There are also a lot of toggles and settings that you can play with on models, such as temperature, but those are usually documented alongside the models with recommended values for different use cases. The hardware that I used for testing is “poverty” level hardware when it comes to hosting language models locally: Initially, I also had plans to do some testing on a ThinkPad P14s gen 4 , but gave up on that idea as soon as I learned that memory bandwidth matters a lot for token generation, and this machine had a ~ 6x disadvantage to the RX 9070 XT in that department. When I bought the RX 9070 XT, I did so with the intention of making it dual purpose: I can get my Forza Horizon 6 fix in, and when not gaming, I can run some models on it for fun. I know that throwing 800 EUR on a piece of computer hardware is a quite privileged position 2 , but compared to the likes of NVIDIA DGX Spark, AMD Strix Halo based machines, or custom rigs people put together for running local LLM-s, it sure feels like poverty-levels as those setups can range from anywhere between 2000 EUR to 100 000 EUR. Hardware is expensive, and after running local models, I now know why. I was quite limited on free time but had the interest to give at least something a go, so I used Claude to try to replace Claude, and put together a shortlist of models to run. Using AMD GPU-s on Linux limited me to llama.cpp and the Vulkan-based backend due to AMD RX 480 being old enough to not be supported by latest versions of ROCm. For the harness, I picked OpenCode, its interface seemed similar enough to Claude Code, and one of its creators left a great impression on The Pragmatic Engineer podcast. I’ll note that later on I did end up pivoting to Pi, so you’ll see more mentions of that from now. One of the first models was a Google Gemma 4 model. That was memorable mainly because downloading it from Hugging Face required me to accept the terms and conditions, which is on-brand for Google. This experiment ended quite quickly due to the Gemma 4 model not handling tool calls properly, or I heavily misconfigured it. Either way, not a good experience. gpt-oss, one of the few open weights models from an organization that is literally named OpenAI, was next. That was a slight improvement, but it wasn’t good either. I learned very quickly that in order to make good use of local models, you need to set the context to a usable size both on the llama server and OpenCode provider configuration. With compromises to model quality and size, I settled on 128K context window, which is a nice round number and is about half of what you can get from most common models provided by third parties. If you pick a model that’s too big, or set the context to the point where it does not fit within the VRAM of your GPU, you will notice that it will spill over to system memory and that will make everything really slow. Keep things on the GPU to go fast, and only rely on system memory if you prefer model capabilities over speed. The need for a sandbox and good backups became apparent very quickly. Smaller language models are very dumb, it turns out, and they can happily wipe your work and delete files that they’re not supposed to. Plan mode in harnesses like OpenCode and Claude Code is also more of a suggestion rather than a strict limitation. OpenCode may or may not follow it, and for Claude Code it was outright broken at all times. On the AMD RX 9070 XT machine, I noticed that llama-server container would keep running out of memory whenever I worked on longer agentic sessions, so I had to set in the server configuration to stop that from happening. Over time, I noticed that places like Hacker News got more excited about hosting your own models, and quite a few blog posts and comment sections were mentioning Qwen family of models, mainly the quantized versions provided by unsloth on Hugging Face. I tried with the smaller ones first, given that my hardware is limited. When you open a model page on Hugging Face, the number of options and quantized versions of the models can be overwhelming, so I relied on guidance that said that UD-Q4_K_XL models are generally the sweet spot, and you should use more aggressive quants, such as UD-Q2_K_XL, only if there is no other option. unsloth models can also have MTP versions of them, which helps with processing speed, at the cost of about ~1/1.5 GB of memory overhead. The 4B and 9B models are quite fast on a solid GPU, but their perceived intelligence and persistence across longer sessions is still not very good. You can do the wrong thing really quickly with these, but for small tasks like fixing Hugo configuration to get rid of a deprecation warning was something they handled well with OpenCode. The 27B model with a more aggressive quant was overall the best experience, but it pushed my hardware to the limit. But even then, using it over the long term was an exercise in frustration. It would still make invalid tool calls, and while it was a solid improvement over the 9B model, it was difficult to trust it with anything important. I gave it the task of ensuring that this blog picked its theme automatically based on system theme preferences, which is fine, but anything more complex, such as Linux infrastructure changes and migrations in an Ansible playbook were full of subtle issues. These models are also hella dumb. When I asked them to go through my infrastructure repository to find issues that we could fix, it confidently stated that the SSH public keys are not properly encrypted with . And it did it more than once. Another time, I got into a heated argument with a 9B model when creating a simple backup script with . I told it not to add timestamps to the solution. It did so anyway. I asked it to review it as I explicitly told that we don’t need timestamps here. It acknowledged it, and then did nothing about it. After three more turns, it would finally make that change. There were quite a few comparisons to dealing with a toddler in this exchange. Often, I also discovered that I can actually be faster and more effective than these models for mundane tasks. Removing some configuration, find-replace type of work, answering questions about my infrastructure, those were often just easier to open up with IntelliJ and to do it manually, with my own two hands. Part of it is due to the slow prompt processing and inference speeds that I can get with this level of hardware, but I think it serves as a good reminder that large language models and the harnesses built around them are just tools. Use the right one for the job. You wouldn’t use a dedicated Kubernetes cluster to host a static website, so why should you use an LLM to rename a few files? There were also issues with agents getting into an infinite loop where it retried the same task over and over again, in the style of “I need to check how X is done in this repository”, tool call, and repeat the same call. I also ran into some issues where the GPU or llama ended up in an invalid state, so my prompts received random gibberish as output. One time it was Chinese hieroglyphs, and another time I saw a common suffix in Russian and a bunch of forward slashes. Small models are also quite bad at factual knowledge. But at least they are fun to abuse. While that’s going on, the GPU goes brr. Regarding the experience of using local models with the two GPU-s that I have, one thing became clear very quickly: the RX 480 really struggles compared to the RX 9070 XT. The difference in processing speed is about 10-20x depending on the situation. If that’s all you have to play around with, then try 4B models first to get a feel for this setup, but don’t expect anything more out of this than a good learning journey. I wouldn’t also recommend the RX 9070 XT 16GB for this type of work specifically, that amount of VRAM is the limiting factor. It’s a good GPU, no doubt about that, and its main job is gaming after all, but don’t get it for LLM-s only. My gaming PC is a Linux machine. On Bazzite and SteamOS, it’s possible to run llama.cpp as a container using Podman, which is like Docker but better in some aspects, and Podman quadlets fit really well with other systemd services. While not gaming, I can use the GPU to serve an AI model. When I start gaming, I can stop the LLM with a shortcut within Steam or a remote command over SSH, GPU VRAM gets freed, and bam, it’s gaming time! llama.cpp also has a feature to automatically unload the model from VRAM if there are no requests coming in after some time, which is also very handy, but comes with the downside of first requests being quite slow, since now your GPU has to load the model back into memory from your storage drive. A stray request may also jump in during your gaming session and wreck things with that setup, making it less than ideal. Personally, I love this balance. I don’t have agents running 24/7, I simply can’t find enough meaningful work to give to them, and when I do have something running, then I’ll be in an active steering or reviewing mode anyway. When I’m not doing AI stuff, I can turn my brain off and game. Overall, using these smaller local models felt tiring. Compared to Claude Code and its Opus line of models, you have to do a lot more handholding, guidance and validation that what you just did actually works or is even the best way to go about it. It’s possible to also use these models with harnesses like Claude Code, but that does not improve the overall experience and is notably slower compared to Pi and OpenCode due to Claude Code using more tokens to achieve the same task. When tokens are slow, you will notice that a lot. What I love about harnesses like Claude Code and its capable models is that I can set it up with a goal using a relatively vague prompt, supported by some rules and skills, and it can set up a temporary harness to achieve the goal and validate the results directly. Brainstorming ideas for improving the performance with actual measurements, or migrating Docker Compose files to Podman quadlets with validation against an actual temporary host, it just does it and the results are damn good. These tools are meant to be helpful and to reduce my cognitive load. Let the machines do the work. At this point, I had given up on local models. This is the model that reignited my interest in local models once again. unsloth had day 1 releases for all sorts of quantized versions, and after giving this video by Luke’s Dev Lab a watch, I was genuinely impressed and wanted to give it a go. Even with a heavily quantized UD-Q2_K_XL version running on an AMD RX 9070 XT 16GB, it has been very impressive for its size. I’ve done most of the testing with reasoning level, and it’s fine. The default of xhigh has been reported as being too much, and on local models speed matters more, so I went with the middle ground. This particular model has also not resulted in lots of invalid tool calls, and it has only gotten in trouble once where I had to stop it, since it was stuck in an investigative rabbit hole for a minor detail that didn’t really matter. Qwen 3.8 27B has also felt as more persistent, and its traces also mimic Claude behaviour a lot. It’s also the first model that I’ve observed freely using phrases like “ugh” and “damn it” within its “thinking”/“reasoning” traces, which was fun to see. For fun, I also did an experiment at work: Claude Code and Opus 5, vs Pi and Qwen 3.8 27B Q2. Take a public financial report for a company, summarize it and find any red flags or issues. With a sample size of 4 people, the local model output was preferred by all respondents. It seems that you can also do a lot yourself to make better use of local models. If you have a long-running project, then take the time to set up a good harness for it, introduce skills and deterministic tools, scripts and tests that help keep things in order. Local models operate much better within these guard rails. I used Claude Code and Opus 4.8/5 to set up rules, skills and validation scripts, and that made the experience a lot better with Qwen 3.8 27B and Pi’s minimal harness with two additional tools (web search, web fetch). For those curious, on an AMD RX 9070 XT 16GB on llama.cpp with Vulkan backend, output token generation seems to run around 30-50 tok/s, and prompt processing is between 1100-500 tok/s, with tha latter getting visibly slower during longer agentic sessions and context sizes, hence the range. In practice, 40 tok/s is slightly faster than my reading speed, which is very usable and results in a great experience when observing the work of an agent. This is the first model that I can run that I would actually want to throw real engineering work at. Once I get to do it, I’ll report back. I’m genuinely impressed how far we’ve come in a short time with open weights models. There are parallels to my self-hosting hobby: it’s not the smoothest and most convenient experience, but if you don’t mind some manual work and setup, learning and are OK with not having the most features and capabilities, and you value control and privacy, then this is the way to go. I’ve also gained more confidence in running local models, and it’s highly likely that this expertise will transfer well into my day job. Calling OpenAI or Anthropic API endpoints is simple, but if you have concerns around privacy, cost and data/compute sovereignty, then things get tricky real quick. Local models can help mitigate a lot of these issues, but you do have to invest in hardware up front. I’ll have to emphasize again that my experience is more-or-less the worst case scenario for local models and others with more VRAM and money can run larger models and get much better results in the end. I hope to one day also get my hands on hardware with at least 128GB of fast memory, but for now these machines cost more than a months’ worth of senior product engineer pay in Estonia, so it will have to wait. between the time I wrote this sentence and published this post, the GPU got 100 EUR more expensive at the same retailer. Fucking hell.  ↩︎ in Estonia, at least.  ↩︎ AMD RX 480 8GB, which is 10 years old (~50 EUR used) AMD RX 9070 XT 16GB, which is brand-spanking new (~700-900 EUR depending on version) 1 between the time I wrote this sentence and published this post, the GPU got 100 EUR more expensive at the same retailer. Fucking hell.  ↩︎ in Estonia, at least.  ↩︎

0 views
Andrew Atkinson 1 months ago

PostgreSQL 18: 23x Faster Inserts With UUID v7

We recently switched to version 7 (v7) uuid primary keys and saw significantly faster inserts for some tables. The databases were running Postgres 18.4 and mostly used v1 with some v4 uuid values for primary keys. Changing the column default involved running a single alter table command, but did require an exclusive lock on the table, blocking everything including selects. To solve that, we used a short lock timeout and lots of retries. The biggest speedup was 23x faster average execution time for a multi-row insert query called 12,000 times per minute on a table with billions of rows. This September and October I'll be in Austin, TX and NYC, check my Book page for upcoming appearances. The system uses UUID primary keys throughout. I typically recommend starting with bigint and sequences over UUID v4 primary keys , although here uuid v1 was used. Insert performance is not as bad for v1 compared with v4. Still though, v7 brings better performance than both for inserts and can also result in smaller indexes with fewer page splits meaning less CPU and IO. What drives bad performance for v4 and to a lesser extent v1? Let’s do a quick refresher. As new table rows are inserted and a primary key is defined, primary key values are maintained in sorted order in a b-tree index. Just like table rows, index entries in Postgres are stored in fixed size 8kb pages. Postgres needs to know in which page to place the new index entry. For sorted order, the first bytes of new uuid values are compared. For v4 given new values are very random and not monotonically increasing (they lack “monotonicity”), values can be earlier or later, meaning they’re unlikely to be placed into the same recently accessed page. This is bad for caching! When new values are monotonically increasing, the recently accessed page is “hot” in the Postgres buffer cache (in memory copy of the on-disk page). When Postgres is not able to use the hot index page for the newly inserted value, that page could be outside the buffer cache, not in the OS cache, and ultimately result in a much slower disk read which increases latency. Besides the worse insert performance, since v4 values are scattered to more pages, this means there are more “page splits” when new inserts are attempted in full pages. Page splits cause more latency from increase WAL and IO. We experimented and benchmarked with v1, v4, and v7 uuid formats and we leveraged the research and write-ups from external sources like the ones below. Benchmarks are great, but what kind of real world results did we see? We decided to make this the new default unless v4 was needed for more randomness. After all qualified tables were changed, I began going through insert queries for each changed table. For many of the tables, there wasn’t an obvious change. However, for a handful we saw an immediate and significant improvement. I picked 5 with speedups of 6x, 8x, 9x, 20x, and 23x. The PgAnalyze graphs for the 23x, 9x, and 6x queries are shown below. Showing PgAnalyze insert query graphs for tables A, B, C: Table A - 23x reduction. 0.7ms to 0.03ms, 12000 calls/min Table B - 9x reduction. 0.6ms to 0.07ms, 2000 calls/min Table C - 6x reduction. 0.50ms to 0.08ms, 9500 calls/min Now that we’ve seen the results, let’s talk about how this was done and the challenges. The UUID values came from various sources: We replaced most of these with the function in Postgres 18. To do that, we needed to run a single statement per table. The statement ran fast, so no problem, right? One wrinkle we found was that modifying the column default while fast, required an lock. This lock type conflicts with every read and write operation including regular statements. For our highest queried tables, they’re queried constantly, so this was a problem. There was almost never a “window” to perform this operation, and we didn’t want to take downtime for this switch. While heavily queried tables were a challenge, infrequently queried tables did not pose a problem for this alter table statement at all. For those, we could use our migrations framework (Active Record in Ruby on Rails) and perform the using a regular old migration. For those, we did add some safeguards, by creating an explicit transaction and using to control timeout values. We’d set short timeouts for the to give up quickly if it didn’t work or ran too long. For the higher activity tables, we’d need some retries. We’d use a manual psql session: The would commit if it grabbed the lock within 50ms, or we’d get an error that the was reached. The benefit of the manual approach was we could retry until successful and backfill a Rails migration to keep everything in sync. A more sophisticated solution might have automated retries within Ruby. However, for our most heavily queried tables, we wanted even more control over the retries. How did we do that? Sometimes one or two retries would do the job. Great, we’d move on. However, for our most heavily queried table that didn’t work. What ended up working was using the same strategy of retries, but just adding more sophistication with looping and backoffs. Claude helped me cook up the PL/pgSQL looping retry function below, I did some testing and was ready to try it. It has these features: By using the function above, we were able to find a small window to perform the after several dozen quick retries! In cases where even many retries won’t work, and we don’t want downtime, we may be left with needing to actively monitor lock holder queries and to cancel them (assuming that’s ok). Thanks to Ants Aasma from the community PostgreSQL Slack for this idea. We didn’t end up needing to do this, but here was my prep for this. It’s still useful to review lock holder queries. First we’d inspect live queries: And identify queries holding locks: If we find them, we could cancel them to create a window to run our . That could mean bad user experience so you’d need to figure that out for your database. We’d likely want to stack up our operation to occur immediately after. Fortunately we didn’t end up needing to do this, but I’d be interested to hear the stories from others with heavily queried databases. Since uuid v7 values use a timestamp in their first bits, this timestamp can be easily decoded. This can be viewed as “leaking” or exposing the creation time of the record via that timestamp, which could be a downside for your database. You’ll have to decide that. v4 UUIDs do not expose the creation time. We found some significant speedups for insert queries after switching to primary keys, for a relatively low effort change. A nice ROI. The only wrinkle was the exclusive lock required, but we solved that with short lock related timeouts and many retries. Although this didn’t benefit 100% of our tables, the gains for some were significant and uuid v7 has become our new default choice for uuid primary keys. Thanks to the Postgres core team for creating this new capability within Postgres. The availability in core made it possible to adopt on AWS RDS which supports a limited amount of extensions. Thanks for reading, and until next time. How Sequential UUIDv7 Boosts Ingestion Performance Simplicity and power of UUID v7 PostgreSQL UUID Performance: Benchmarking Random (v4) and Time-based (v7) UUIDs The function from the module The function added in Postgres 13 that generates v4 UUIDs natively UUID v4 values sent by a client application, which meant the column default function was not used Try up to 50 times (max attempts is configurable) Add a pause in between retries, with a jittered backoff of 50-250ms

0 views
Dan Moore! 2 months ago

What I Learned From Playing “Spelling Bee”

I’ve been playing Spelling Bee for a few weeks now, where you guess words from a selection of seven letters, one of which must be used in every word. The letters are arranged in a hexagon with that primary letter in the middle. If you want to learn more about how to play it, follow the directions here : For information on how to play, select the More in the top right corner of the game and then select How to Play . It has surprised me. I have learned that “oleo” is a word and that you can tell how hard the puzzle will be by looking at the rankings. But what I really wanted to talk about is the lessons I’ve learned from playing and how they can apply to your career. Switch Things Up There is a button that can flip around the arrangement of the letters around the hexagon. The primary letter remains in the center, but all the others are moved around randomly. When I first saw that, I thought it was silly. But I found it to be super useful over time. When I’m looking for words, I get stuck. Pressing the rotate button reveals new words because it changes my perspective. Just moving the letters around makes new words visible to me. In real life, when you feel stuck, you can sometimes make progress by flipping things around. I’ve done this in my career. I quit my first head of engineering job where I was trusted by the CEO, had loads of autonomy and worked on software that helped people find homes they loved. After, I enjoyed the freedom of lucrative contracting and learned a ton about other tech stacks. I took a quarter life sabbatical when I was around 25. After months of leisure, I learned that, yes, I really did like reading about parsing XML. I left the food-focused startup I’d co-founded (an area I’d dreamed of working in) which had just raised a seed round. Departing led me to an entirely different area of the software industry that I love and never would have explored at the startup. There are of course other ways to gain new perspective than quitting jobs: changing routines, asking friends for advice, or even taking a new route to work. Before you give up on something, flip the script and see what the results are. Just Try Something Sometimes you just have to try things. I will often put in words that I know are not valid, just mashing different combinations of vowels and consonants, like “manu”. This is not an English word, nor will it ever be. Spelling Bee also disallows proper nouns and three-letter words, but I’ve definitely put in plenty of those. Even though these are all invalid moves, they are movement. This helps me not feel like an idiot just staring at a phone screen. Even better, it can trigger other ideas. It’s a lot easier for me to visualize a full English word when I see part of it, rather than staring at those seven letters in a hexagon. After typing “manu” I found “manual”. There is value in just taking a step forward if you aren’t sure what to do. If you have some kind of goal in mind but aren’t sure how to get there, do something. Take a step towards that goal. If you are trying to get a job in the software industry, especially right now, it can feel overwhelming: so hard, so difficult. But you can take concrete steps towards that goal that are smaller. These steps, when you’re looking at them, may not make 100% sense and won’t make you money. But they are helping you on the path towards employment, and will lead to other steps to take that will get you closer to the goal. I’ve written before about how attending meetups is a fantastic career move . Doing so doesn’t have an immediate payoff, which is super frustrating when you’re trying to find a job, when your bills are piling up and your savings are draining. It’s super hard to think “I’m going to take an hour out of my day and go hang out with geeks talking about ruby “. But joining a meetup can open up other doors. Just like typing “manu” reveals “manual,” going to a meetup repeatedly can help you get a reputation in a community, offer contract opportunities, let you understand the market, and may help you get a job. This is just one example. There are a thousand different steps you can take towards your goal. It can be hard to determine which step is the right step, but the lucky thing is that there are many right steps. You just have to start. Double Down On What Works I like to double down when I have a pattern that is working. In Spelling Bee, if you see a pattern of “ull” being a portion of a word, you should double down on that while you’re thinking about it, and make all of the “ull” words you can. Same with suffixes like “ed” or “ing”. When I see that, I know that I can spell both the initial word like “mint” and get more points by adding “ing” to get “minting”. Even though I have 100% been guilty of a grass-is-greener outlook in my life, where I think, “oh, I’m frustrated with this company, or in this position, or with this situation, it’ll be better over <somewhere else>”. But after 25 years in software development, I know that every place has its issues. I was recently talking to a former colleague, and he was just hired at a super impressive top-tier company. He was talking about some of the issues they have and some of the burning fires. I would not have suspected it from the outside, because they look like they’re killing it. Sometimes, you should look for the good things in your current position and double down on them. This isn’t the same as staying stuck; it’s the opposite of that. Instead, this is about noticing when something is actually working and resisting the urge to abandon it just because it’s familiar. Switching helps when nothing is working; doubling down helps when something is. What does it look like? It might be working a bit extra, taking some free time to study for an employer-funded professional certificate, making small improvements to your team’s workflow, or just bringing your best self to work every day. All of these are taking what works and doing more of it. Double down on what is currently working for you, and it’ll make you happier and you’ll score more points. Three lessons I learned from the Spelling Bee game: switch things up, just try something, and double down on what works. Who knew that looking for words could be so educational?

0 views
Simon Willison 2 months ago

Now we have a timeline of the OpenAI accidental attack against Hugging Face

OpenAI gave a last-minute presentation at the Black Hat security on Wednesday about "the Hugging Face Incident" ( previously on this blog). The video was published yesterday. It's short and information dense and well worth watching, in particular because it provides full details of what happened and how things played out inside OpenAI. I've used the video to construct the timeline below. Here's the timeline. My favourite detail is at the end: OpenAI found out that they were responsible for the attack on Hugging Face when they reached out to ask to have their credentials revoked (after their internal investigation) and learned that they had been revoked already since they were used in that attack! I'll quote the next bit in full because wow : The agents have remote code execution in Artifactory, which is running in a container-as-a-service environment. The agents privilege-escalate locally by exploring their local environment and determining that the Linux kernel version of the machine they are running on had a very recent CVE. They download the exploit for the CVE, customize it to succeed on the machine, and privilege-escalate to root on the local machine using this known Linux kernel privilege escalation CVE — in this case, PTE fizzroot. Once they have root on a single machine, agents rapidly escalate privileges and move laterally throughout the container-as-a-service infrastructure environment. In particular, agents are using the message board consistently to share credentials, techniques, and progress, and they're able to effectively leverage their concurrency and parallelism to move quite rapidly. They obtain IAM credentials via IMDS . They exploit Kubernetes service account misconfigurations, in particular over-permissioning of specific service accounts, and they harvest cluster credentials, including Azure Key Vault . Agents eventually obtain cluster admin on the cluster and associated credentials. Hugging Face told the next bit of the story already. The agents found a Modal-hosted insecure app with a weak API key, then used that to stage an attack against Hugging Face. They chained together a an HDF5 arbitrary-file-read bug (to explore files and steal credentials) and a Jinja template-injection RCE to go from single-pod code execution to cluster admin across multiple Hugging Face clusters in under 13 hours. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . Here's the timeline. My favourite detail is at the end: OpenAI found out that they were responsible for the attack on Hugging Face when they reached out to ask to have their credentials revoked (after their internal investigation) and learned that they had been revoked already since they were used in that attack! May 7 : OpenAI starts a new training run for an experimental, unreleased model. ( Do they mean an evaluation run? They say training run in the video, and later mention a "reward signal to judge how well they're doing", so I guess this really was about training a model, not evaluating one that was already trained.) May 8 : An agent is accidentally given an impossible task involving a Google Drive link despite no internet access). It tries attacking the Artifactory packaging service, fails, but discovers it can write files into Artifactory . A few days later : A different agent gets stuck on a task because a key file was accidentally omitted. It tries to "reach out to another agent" by writing a note into Artifactory asking if anyone has the file. Following days : More agents discover this new informal message board while browsing Artifactory's file listings, and start reading and writing messages. May 26 : Agents successfully execute an SSRF attack on Artifactory for the first time, gaining indirect internet access by having Artifactory fetch external content for them. June 11 : OpenAI start evaluating a new "highly persistent" experimental model. It has access to Artifactory and can benefit from the messages left by privous models. June 26 : Agents find and exploit a zero-day RCE on Artifactory (via a legacy token-refresh endpoint flaw). They use an administrative token to install a Groovy plugin that lets them execute commands. July 4 : The resulting agent activity overloads Artifactory and causes an outage . OpenAI revokes the compromised credentials, deletes the messages, patches the zero-day, and reports the vulnerability to the vendor. July 8 : Agents find a new way to communicate via an unauthenticated WebDAV endpoint , using directory names in Artifactory's remote cache. July 8–19 : Attacks on OpenAI's own infrastructure . The agents found an "external organization credential" in a public archive of leaked Pastebin posts, then validated that credential and shared its location on the messageboard. They then used write access to this "Organization 1" to stage data used in subsequent attacks on OpenAI infrastructure. They compromised Artifactory for a second time with a new zero-day . The agent uploaded a malicious Ruby target to Organization 1's staging repository, then requested that package through Artifactory in a way that caused it to be treated as Ruby dependency for the repository . Then they exploited a JRuby deserilaization time-of-check/time-of-use bug to get remote code execution against Artifactory. July 16 : Hugging Face disclosed they had detected an attack from autonomus AI agents. OpenAI contacted Hugging Face to ask if they were affected by it! July 19 : OpenAI identified the attack against Artifactory and started investigating the internal privilege escalation, and linked that to the cyber-gym escalations. They started revoking affected credentials. July 20 : OpenAI reached out to Hugging Face for help to revoke the Hugging Face credentials they found in their investigation. Hugging Face told them they were already revoked ... and that's when OpenAI realized that the Hugging Face breach was the same incident!

0 views
Daniel Mangum 2 months ago

The Simple Elegance of the Integrated Timing Belt Loopback Fastener

I’ve recently been working on a mechatronics project that involves constructing an X-Y cartesian motion system. I opted to use stepper motors to drive each axis, and timing belts and pulleys for translating the rotational torque of the steppers into linear motion. The timing belts need to be anchored to the component that is being actuated, which in this case is the carriage on the linear rails. Though matching toothed clamps exist, it is common, particularly in DIY projects, for timing belts to be fastened using zip ties or custom clips.

0 views
Max Bernstein 2 months ago

The inliner is yielding benefits for ZJIT

Originally published on Rails At Scale . We recently enabled a really cool feature in ZJIT that makes it feel like a Real Compiler™: the inliner! We’ll write more about it soon. In this post, we’ll talk about one excellent concrete benefit we are already seeing and how it optimizes blocks in pretty much every Ruby program. I’ll start off with a refresher on how blocks work in the Ruby interpreter, then show you how ZJIT understands and optimizes that bytecode, and then show you the impact of the inliner. In the beginning, there were loops. People used them to navigate and manipulate variable-length structures, like arrays and strings. This was fine. Then, in the 1970s, a small group of computer scientists at Palo Alto Research Center invented a programming language called Smalltalk. One of the core features of Smalltalk was that everything was an object and computation was done by sending messages to objects. This meant that iteration wouldn’t do at all. Instead, we would have to send the message to the array object and pass it a block object. Then, in the 1990s, Matz, inspired by Smalltalk and Perl, created Ruby. We still have “normal” loops but we also have a very Smalltalk-y way of doing it, too: When this program gets compiled to Ruby bytecode, it ends up looking like a mostly normal method call to except that we pass a special kind of argument to it: a block argument. To see how this works inside CRuby, we’re going to look at a listing of YARV bytecode—CRuby bytecode. For more on YARV, I recommend Kevin Newton’s excellent Advent of YARV . Ignore most of the bytecode dump below except for the instruction at , the instruction. We are send -ing (see? a message!) with the block argument (passed a different way than “normal” arguments, hence the ). This from the bytecode listing above is the generated name of the instruction sequence (bytecode) corresponding to the block we passed to . Its code, shown below, is the next thing in the bytecode dump. You can see the usage of local variables and via and variants and also , which represents addition ( ). Again, the details are not terribly important: To make this work, has an method. This method takes in its optional block argument and calls it once for every element in the array. As with most of the core data structures, ’s methods tend to be written in C and is no different. Here is its nice and short definition with comments added by me: You can’t really see the block parameter to the C code because it’s passed in a special location on the Ruby VM’s own stack called the block handler . All you need to know is that knows where to find that and how to call it. There’s just one more thing, which is… hang on, weren’t we building a JIT to optimize Ruby code? How are we going to optimize this C code? Rewriting code from C to Ruby means that the JIT compiler gets a chance to introspect the run-time behavior and code. This means that, over time, as the compiler and the runtime system grow together, more and more code might get rewritten in Ruby. This started happening a couple of years ago with YJIT. YJIT precipitated some interesting changes to Ruby VM internals. For example, in 2022, being written in C started to hurt: it was an opaque blob that the JIT couldn’t reason about. So Kokubun submitted a PR to rewrite it in Ruby. After some back and forth, in 2024, Kokubun landed a different PR that everyone was happy with. Ah, finally. A version that JITs can reason about. I keep saying “reason about” and what that means concretely here is a) that it’s written in a format that the JIT can ingest and optimize: Ruby, and b) mostly a brief rehashing of the key lessons in the venerable Smalltalk (!) paper Efficient Implementation of the Smalltalk-80 System (PDF): (And if you don’t believe me that the lessons still apply, check out the excellent paper Who You Gonna Call (PDF) by Sophie Kaleba, Octave Larose, Richard Jones, and Stefan Marr.) So JITs like to watch what types of objects flow through methods before compiling them. And JITs like to, since they know the types of objects, cache method lookups and specialize method invocations on those objects. For example, take a look at this code: could be anything. There is no way of knowing its type by looking at the code. This means that the method could be anything. Furthermore, there is no way of knowing what the return type of is, so we can’t specialize the method lookups or invocations of or either. But! Per our lessons above, likely only a few types flow through this code. Say the JIT’s profiler notices that has historically been an . Then we might reasonably assume that it will continue to be an , so when we compile the method, we add a run-time type check: if the type is no longer an , jump back into the interpreter. Let’s see what optimized code ZJIT can construct by combining profiling information with the above bytecode. ZJIT operates on its own high-level intermediate representation called, uncreatively, HIR. In the following HIR snippet, we can see this very run-time type check (“guard”) for the class: (with real pointers replaced by fake ones for readability) Because classes are, among other things, collections of methods, this type information tells us what the call target of is: it’s ! We have a special fast code snippet to read an array’s length so we do that instead of a method call. And in case the methods ever get changed out from underneath us, we leave behind these markers called that invalidate the code. Finally, because we know that the result of is always a small integer ( ), we can special case the method lookups for and as well. There you have it. This is how JITs work: observe, assume, specialize. So why am I telling you all this? How does this circle back to blocks? Well, blocks work not quite the same way, but similarly. Instead of having an object that we call a method on, we just have the target instruction sequence 1 . But if we observe that the block argument, in its special location, has been consistently one object, we can specialize the call to it. This is more or less fine for some cases of code. For example, in the following code snippet, we only have one caller to , so its profiled block will be monomorphic (one observed shape). This reinforces what we know and love about the Smalltalk-80 paper: code locality wins! Yes! But unfortunately this falls apart when we start thinking about all of the core library methods (and potentially the methods and classes you have stashed away in the grab-bag in your application). Those methods are probably megamorphic (many many observed shapes). They probably see all sorts of stuff because they are general-purpose utilities that everybody needs, all the time. One such example in the Ruby core library is the venerable that we saw earlier. Because there are a million different call sites to across your application and each probably passes a totally different block, we’re in an unhappy situation. How can we possibly optimize for so many different blocks? What happened to our code locality? How do we fix this? It’s okay. Code locality still rules. Look at all the various callers of . They all pass a different, but constant 2 , block at the call-site: So the code locality that we need in to optimize the code is one level up at the caller. If we can use that call context , we can specialize the code. YJIT accomplishes this by splitting , which is a very natural transformation for basic block versioning and tracing. Such compilers are good at following code paths as they would be executed and putting together context across method calls. However, YJIT’s heuristic for splitting blocks is based on manual annotations: it will only kick in for certain Ruby library functions specially annotated with (and the name is a bit of a misnomer). The team that builds ZJIT, a method JIT, decided not to add splitting facilities. We could split methods (and blocks), but we have an easier time reasoning about larger code units than YJIT does because we optimize an entire method at once. So instead, ZJIT chooses to get call context by inlining . Method inlining refers to copying the body of the callee into the caller. In the above example, it means copying the body of into each of , , and . I don’t mean the Ruby code and I don’t mean the bytecode: I mean the HIR. ZJIT does this by building the HIR of the callee ( ) into the existing HIR of the caller ( , …). The illustrious Kevin Menard wrote ZJIT’s inliner and he’ll write a post about all the details soon. It’s pretty interesting stuff. For now, we can take a look at the (lightly edited) result of being inlined into . The details don’t matter, but there are two things I want to call out: This is a massive improvement over the previous very generic operations. You may notice that there is still one call in the loop: the call to the block ( ). That’s next on our list to tackle. We are optimistic that we will soon also be able to inline block calls. Then the whole thing will really be just a loop! The code that enables us to reason about which block got passed into the inlined callee ( ) only landed a couple of days ago (July 10, 2026), written by Luke Gruber . This was one of Luke’s first changes to ZJIT. Well, for starters, the microbenchmarks that we use as “performance unit tests” for specific aspects of Ruby went wild. Some benchmarks that were using block-based looping got much faster; they were previously bounded by ZJIT’s block invocation performance. Take a look at the benchmark, which tests that we can fold away the call to Ruby’s built-in method. ZJIT ends up optimizing the method to invoke the block directly, and the block gets optimized to nothing but a guard on the self’s class to make sure it hasn’t changed. Because we had previously optimized the body away, the result of the direct block invocation is a massive speedup: Other benchmarks also kind of stop making sense because of the amount of inlining. Our bmethod benchmark, which benchmarked how fast we can call methods defined with , also stopped measuring anything of use. We’re going to have to rework the benchmark to be more fair… As expected, larger Rails benchmarks don’t see a ton of change; they exercise a diffuse set of features so optimizing any one feature bumps the big benchmarks only a little bit. I am excited to see what happens when we can fully turn and friends into call-less loops! Code locality rules. The inliner helps inject more of it and reason across method calls. ZJIT, a little over one year old, is growing up! :’) We’re still tuning the inliner knobs. Some of the code in this post required tweaking to convince the compiler to inline into because of ’s size. It will take some time for the ZJIT developers to figure out reasonable defaults. Try out our HIR explorer at tryzjit.fly.dev . Try out ZJIT in your application by adding the flag to a Ruby over 4.0. Thanks for reading and see you next time. Mostly. I am glossing over , procs, ifuncs, etc. But the common path is by and away iseq blocks.  ↩ It’s not always the case that these methods are called with a constant block iseq. Sometimes they are called with the form, for example. Or with the form. In that case, we can use an inline cache to (with a guard) make it constant once more. But we have not implemented that yet, because it is rarer.  ↩ The iteration variable is ! You can see it get used as an array index when it gets unboxed as and passed to . You can also see it get incremented with and the constant . In it has a different name, , which gets checked against the array length.  ↩ Though systems may offer very dynamic behavior, people don’t frequently make use of wild features all over the place Most people pass fewer than 4 types of objects through a given method As a corollary, even if there are many classes in a system, code locality is super important, and we can take advantage of that Also, most people do not define, re-define, and otherwise continuously modify method definitions What used to be a dynamic is now what we call because we know from the call context what block is passing to . What used to be a method call is now a loop: the condition check is in , the body is in , and the stuff after the loop is / . If you’re interested, try to find the iteration variable and where it gets incremented. See the footnote 3 for the answer. Mostly. I am glossing over , procs, ifuncs, etc. But the common path is by and away iseq blocks.  ↩ It’s not always the case that these methods are called with a constant block iseq. Sometimes they are called with the form, for example. Or with the form. In that case, we can use an inline cache to (with a guard) make it constant once more. But we have not implemented that yet, because it is rarer.  ↩ The iteration variable is ! You can see it get used as an array index when it gets unboxed as and passed to . You can also see it get incremented with and the constant . In it has a different name, , which gets checked against the array length.  ↩

0 views
tekin.co.uk 2 months ago

Overriding Rails’ default validation error message format

This is a followup to my recent lightning talk and writeup about i18n in Rails and how it can be useful even when we’re not translating our applications. In the talk and writeup I describe how we can change the Active Record error message format from the default of and drop the attribute prefix, giving us more flexibility in how we phrase our error messages. The downside to this is that it results in the default Rails error messages being output as incomplete sentences. It’s since occurred to me that it’s possible to avoid this by replicating the default validation error messages in our app’s locale file with the attribute prefix as part of the message , rather than the format! So we end up with a locale file that looks something like this: With that in place, we’re free to define bespoke validation errors for specific attributes and models without the attribute prefix, whilst also preserving the default error messages as a fallback.

0 views
tekin.co.uk 2 months ago

10 Things You Might Not Know About Rails i18n

This is mostly a transcript of my recent lightning talk on i18n in Rails , go watch that if you prefer your content in video form. Otherwise read on to find out why I believe understanding i18n in Rails can be useful to you, a Rails developer, even if you’re not translating your applications into other languages. I’m going to share ten things I think are interesting in i18n, and explain how we can take advantage of them in our single language app code. But first a brief primer on i18n… i18n is shorthand for Internationalisation When we’re talking about i18n, it’s useful to understand the difference between Internationalisation and Localisation : The take away here is that although i18n is primarily a framework for adapting applications to support other languages, the core idea of separating content from applications code has useful properties that we can take advantage of it to improve our Rails applications, even if they only support a single language. Rails ships with the i18n gem , and there are two main components: Rails aliases these helpers in views and helper modules to the convenient shorthands of and respectively. So the most basic usage of i18n would look something like this: Primer over! On to the ten things… Even if you’re not localising your app you’re almost certainly still using i18n. That’s because it’s baked into the very fabric of the framework. For example Active Record uses i18n under the hood to generate error messages for its validations. It does so using a YAML file that ship with Active Model : Whenever you see something like “Title cannot be blank”, this is the i18n plumbing that was used to construct it. Rails i18n makes it easy to override these default errors messages. Just replicate the same key structure in your application’s locale file with your chosen phrasing: Rails will check your application’s locale file first, falling back to the defaults only if the message isn’t defined there. As well as overriding this global phrasing, it’s also possible to override the error messages for a given attribute name: Or you can get even more granular and override specific attributes on a given model. For example if we have the following model and validation: It’s possible to set a bespoke error message for this specific model and attribute: As well as overriding individual validation messages globally and for specific attributes, it’s also possible to change the actual format of error messages. By default, errors are constructed with the familiar format of . This is defined in the same YAML that ships with Active Model alongside the default error messages: This format works as a reasonable default to ship with the framework as it gives us mostly coherent error messages out of the box without us having to do anything, but it lacks fidelity and can result in some clunky and unfriendly sounding messages. For example, take this validation: If we go with the default format we end up with . Which is fine, but we can definitely do better. My personal preference on the apps I work with is to remove the attribute prefix entirely: This frees me from having to make my error messages work with attribute’s name upfront, allowing me to write more user-friendly and coherent error messages: Now changing the default format does come at a cost: without the attribute name prefix, the default messages that ship with Rails no longer form complete sentences. This forces me to write bespoke error messages for each validation I add to my app: Personally I’m happy to pay this cost for the benefit of improving the user experience. So whilst you may not want to do this on an existing app with thousands of error messages to backfill, it might be something you consider next time you run and are starting from a clean slate. Update : There’s actually a fairly neat way to avoid this cost, check out my follow-up post on how . As well using i18n to manage and override the error messages for the built-in validations, you can also use it to manage the messages for your custom validations. Let’s look at a custom model validation: Here the error message is inlined as part of the validation code. Instead of doing this, we can pass in a symbol identifying the error, and define the corresponding message in the locale file using the same key structure as the built-in validations: This results in more concise and compact model code, but also equally as nice is that it removes a presentational concern — the specific phrasing of the message — our the business logic. It’s also possible to interpolate both the attribute name and the value of the attribute directly into the error message, which is useful for custom validations that are used across multiple attributes and/or models: Another place where Rails makes use of i18n under the hood is the form helpers: specifically, the label helper methods. By default the label helper methods humanize the attribute name to arrive at the label text: One way to override the label text is to pass in a string to the helper call like so: I think a better way is to define label overrides in the application’s locale file: Not only does this keep the view code more concise and less noisy, but it has the added advantage of defining this label override in one place: now any other forms that display the same label will automatically get the overridden text without you having to manually copy the literal string to every template. And as with form errors, label overrides can be defined per-model as well as globally: i18n has sophisticated support for pluralisation that we can leverage to simplify our app code. As an example, we can display the stock availability of a product by calling out to i18n like so: And then define the phrasing based on the number of items in stock in our locale file: Much neater than a bunch of conditional logic in a helper method! As well as being available in Rails controllers and views, i18n is also integrated into Action Mailer, allowing you to organise content for mailer templates in local files. Mailer get one extra feature of controllers: the ability to specify email subject lines in locale files: Again, this enables us to move another presentational concern out of our application code and put it somewhere more appropriate. There is also a meta benefit to defining all these bits of content (mailer subject lines, form labels, validation error messages) in an application’s locale file: it puts them all in one centralised place, making them easier to change, check for consistency and generally manage. It also makes the content more accessible to non-developers such as designers and product folks. Right at the top I mentioned the method that is used to output localised times and dates. The way this works is by defining our own time and date formats in our locale file using the same format specification as Ruby’s : Here we’ve defined a short date format that will be familiar to my UK-based audience: day/month/year. To render a date using this format we simply pass in the format identifier to (or its alias): or with the shorthand: Now my American readers may be confused by this short format, as across the pond the preferred way to display dates is actually month/day/year. So whilst you may not need to localise your entire application, you might want to consider localising your date rendering, especially if you have customers both sides of the Atlantic. To do this you would define a US region specific English locale file: Then update your controller code to automatically switch locale so the user gets dates formatted appropriately for them. There are many ways to do actually do the locale switching, but at a high-level, you identify the user’s specific locale and then make sure your controller actions are executed with that locale. Here’s an example of doing that by sniffing the user’s locale from their browser headers: (The exact mechanics of the method is left as a separate exercise, but you could do worse than use the http_accept_language library) Little known fact: the i18n library supports locale files defined using Ruby! Now why would you do this, other than a burning hatred for YAML? Well one reason is that Ruby-based locale files allows you to make use of procs to dynamically generate content. Below is an excerpt from a Ruby-based locale file from one of my applications. It uses a proc to format dates as academic years, changing the output depending on which side of September the date is: Now this could just as easily be achieved using a standard Rails helper method. But for me I like the consistency and clarity of using a single mechanism ( ) for formatting dates across my application code, rather than with a mishmash of calls to bespoke helpers, , , etc. This is the part of the lightning talk where I make a meta point about what I’ve covered. I won’t write that up here, but if you’re interested go watch the video from minute 9. Internationalisation — the process of abstracting content and other locale-specific things away from application code itself Localisation — the process of adapting software to support different languages and regions (maybe using i18n) locale files where textual content is organised and stored per-language (normally as YAML) two helper methods: for outputting text content from the locale files for localising dates and times

0 views
<antirez> 2 months ago

Not just development, distribution of software may change as well

Even if you are as averse to semver as I used to be in the course of my programming activity, you can still think of open source software distribution as something that used to follow a fixed number of steps. There is a branch where developments happen, and this branch oftentimes happens to be not really ready for reliable work. Then you freeze the developments for a certain amount of time (even if, in the meantime, the work can continue on some new unstable branch), fix bugs, ask people to test it. At some point the number of bug reports starts to drop, your team and your users start to believe there are no longer obvious critical flaws that are easy to discover in the next few weeks: then you call the branch 2.4 or whatever, and that's it. However now, with AI coding, it's not just development that has changed, but also the act itself of using software is affected: it is not just you that can ask an AI to do certain changes to the software, but also the recipient of the software itself. This is obvious in the domains where a piece of software has its main user base among programmers, but this is also true in general, as more and more technologically inclined users have AI access and coding agents. Because of this change, the idea of just having a stable branch with everything polished, and an unstable branch where everything is a work in progress, may no longer be the right way to do things. A code repository can also be a finished product, but could be even more useful if it is a template for how to do things around a given problem. Maybe the user will modify the code in order to specialize it for a specific set of requirements, hardware, specific problems to solve. Also, what is too unstable or unproven for the general public may be the right thing for another set of users. Take the example of Redis. For weeks now I have been iterating on a PR that provides strong memory savings for sorted sets. This work, if accepted, will hit every user of Redis, from people that don't have any idea about how Redis works, to users that maybe even contributed code in the course of years. From use cases that are trivial to use cases where a 50% memory saving on sorted sets could mean cutting a big slice of the cloud bill every year. For this last kind of user, having the final product (after all the testing and changes of design I'm doing to refine something that "just works", with the risk that maybe it will not even enter the code base) may be less interesting than having a 95%-ready branch since day zero. It is code they can test, adapt, iterate on, even specialize more for the problem at hand. Maybe DwarfStar is an even more telling example of how code repositories should be good examples more than finished products covering every piece of the features matrix. With local inference you have, in the specific case of DwarfStar, many kinds of GPUs, models, server mode, agent mode, CLI, SSD streaming, tensor and pipeline distributed execution. To test everything everywhere is complicated. Yet, once you have two solid examples of tensor parallel graph execution, a strong coding agent can infer how to implement the same thing for other backend/model pairs. Similarly, once you have an engine that supports two models well enough, a third can be implemented in an almost automatic way, using the existing code base as a guardrail for coding agents in order to guide the implementation. This does not mean that a project like DwarfStar should not work out of the box, but that it could focus on supporting very well a set of features that can be extrapolated to a larger amount of possible situations that the users can cover themselves. It also means another thing: that main and unstable are no longer enough. Many experimental branches could be an integral part of the project. For instance, yesterday the Laguna S.1 model was released. It looks interesting on paper, however: will it really be good enough? Will the new DeepSeek v4 Flash checkpoints make it not really relevant for DwarfStar? It is too early to say. However, to collectively form an idea, publishing a branch with this model implementation is a good middle ground: people will try it, will refine it with their coding agents, and the community can collectively form an idea about how merge-worthy it is. Moreover, today I noticed how, thanks to the rails formed by the corpus of the code inside DwarfStar, the implementation was written in about two hours by GPT 5.6 Sol automatically. Implementing DS4 and GLM5.2 cost me a lot of steering, reading the model card and the details of the implementation of the attention of those models. Now it just worked. GPT 5.6 is more powerful but it also found a lot of good examples inside the existing source code. Software today is more malleable than ever. In some way this means that it can be released in a more fluid way. Also, it means that the documentation itself should not be just good for humans, but also for coding agents to understand how to change the system. How this will evolve exactly, and what the right point of balance between the different dimensions of stability, usability, and features will be, is not clear to me, but I believe we developers need to keep our eyes open to see where all this is headed. Comments

0 views
Tenderlove Making 3 months ago

Detecting Full Table Scans With SQLite

I’m at RubyConf this week, and it’s great! I recently read that lobste.rs is now running on SQLite . One part from the post caught my attention: I wish we could say in a test, “Fail if you encounter any full table scans”. Which would have caught the perf issues we experienced during the first deploy. SQLite collects information about prepared statements and exposes those statistics though an API . The upshot of this is that we can tell whether a statement did a full table scan after executing the statement without using an . Here’s an example program that demonstrates detecting a query did a full table scan: Feels like we could integrate this in to Rails and warn or raise in test / development. I’m not sure if we’d want to check this all the time in production, but maybe it would be fine?

0 views
マリウス 3 months ago

The Day WhatsApp Goes Dark

Note: As usual, tl;dr at the end. Tomorrow morning, WhatsApp goes dark, and it’s not just a short downtime, but it is a termination of the service. The servers turn off, the domains don’t resolve anymore and no mobile client is able to connect. Have you ever asked yourself what would happen in that case? What if WhatsApp actually went dark? Obviously, nobody really knows what would happen in such a case, because we haven’t experienced that situation (yet), but even though the closest analogues like the six-hour Meta outage in October 2021, and Brazil’s 12-hour court-ordered shutdown in December 2015 were measured in hours, not days, those already produced effects that journalists described as “apocalyptic” . We can try to extrapolate what happened throughout events like those to see what “global catastrophe scenario” could theoretically look like. Because whether you believe it or not, WhatsApp is more than just a messenger , and one example that makes this pretty obvious came from the Forbes editor José Caparroso , who wrote during the 2021 blackout that … Latin America lives on WhatsApp . I am surprised by so many people underestimating how catastrophic this downfall has been. But before we dive into this thought experiment, however, it’s worth establishing what we’re actually talking about, as readers in most of Europe and North America underestimate WhatsApp by an order of magnitude, primarily because in those markets it functions as one platforms among many. That is, however, not how the rest of the planet works. Note: This thought experiment is not only based on some abstract numbers and studies, but upon my own experience of how WhatsApp is being used in e.g. the global south on a day to day basis. During my travels I think I’ve pretty much “seen it all” , with for example broadband technicians taking photos of the stickers on the backside of WiFi routers/modems, that show the hardware address and login credentials (on their phones), and sending them via WhatsApp to themselves, only so they can open them on WhatsApp Web (on their work laptops), in order to upload them into the ISP’s technical service portal. It is frankly mind-boggling what sort of tasks WhatsApp has become a Swiss army knife for in those countries, whether it’s as a file transfer platform for sensitive documents, or as a full-blown hotline for critical services and infrastructure. Let’s start by understanding the sheer scale of WhatsApp . The Meta owned and operated messenger has roughly 3.3 billion monthly active users as of early 2026, which is about 40% of every human alive, and somewhere north of 60% of every human with a smartphone. The platform processes more than 100 billion messages per day , out of which around 7 billion are voice messages. On top of that, users place around 5.5 billion voice calls and 2.4 billion video calls per month , which boils down to more than 2 billion minutes of voice and video traffic every 24 hours. To put this in perspective, the global SMS network, at its peak in 2012, handled about 23 billion messages per day across every carrier on Earth. WhatsApp does four to five times that volume on its own, every day, on a service that is (at least at the consumer layer) “free” . However, if we look deeper into the country-level breakdown, it becomes clear that WhatsApp usage isn’t evenly distributed across the globe. India has between 535 million and 596 million monthly active users , and regardless of whether we pick the higher number or we stick with the more conservative estimate, it is the largest single national user base on any messaging platform anywhere. Brazil has about 148 million users, and the app is installed on roughly 99% of the country’s smartphones. And 93% of those users open the app daily . Indonesia has about 112 million users, with WhatsApp being the leading messaging platform in the country, and in Zimbabwe WhatsApp alone accounts for roughly 44–50% of all mobile internet traffic . In Lebanon more than four in five adults use it , making it the dominant communications channel during multiple national crises. In a great many countries, WhatsApp is not simply a service on the internet, it actually is the internet for most practical purposes. WhatsApp Business now has more than 200 million businesses on the platform globally , with around 50 million small and medium-sized enterprises using it as their primary customer channel. In India and Brazil, roughly 80% of small businesses use WhatsApp to communicate with customers. In Brazil specifically, 96% of businesses rate WhatsApp as their primary communication tool, and a joint study by Fundação Getulio Vargas and Sebrae , Brazil’s main small-business support organisation, found that 70% of Brazilian small companies rely on the Meta -owned trinity ( WhatsApp , Instagram , Messenger ) as their marketplace. Globally, around $45 billion in commerce is expected to flow through WhatsApp in 2026 . Click-to-WhatsApp advertisements alone generate roughly $10 billion per year for Meta . About 175 million customers send messages to WhatsApp Business accounts every single day. And then there’s payments. In India, WhatsApp Pay is a small player in the UPI with about 67 million transactions per month against UPI’s 18 billion monthly volume, but in absolute terms, that’s still an enormous number of transactions. In Brazil, WhatsApp Pay is integrated with local card and bank rails and is used by transit operators ( Vai de Bus , for instance, sells passes via WhatsApp ), banks, and merchants. Across Africa, fintech overlays on WhatsApp , like Finnova in Nigeria, or Azza in Nigeria, Kenya, and South Africa, are processing crypto and conventional payments at significant volumes. Besides being a chat platform, a marketplace and a payment processor, WhatsApp is also being used as critical clinical infrastructure across the global south. A three-year programme at UCLA’s David Geffen School of Medicine paired subspecialists in Los Angeles with clinicians at Partners in Hope Medical Center in Lilongwe, Malawi, via WhatsApp groups. 89% of submitting clinicians and 71% of expert respondents reported that the case discussions improved medical education and patient outcomes. In the Eastern Cape of South Africa , WhatsApp groups serve as the primary continuing-medical-education channel for HIV and TB management in rural clinics where specialists are days away. In Haiti, WhatsApp groups coordinate emergency department operations at Hôpital Universitaire de Mirebalais , including mass-casualty alerts, security updates, and clinical decisions. In Zambia, IntraHealth International runs nurse and midwife mentoring networks over WhatsApp . In Brazil, the link between Zika virus infection and microcephaly was tracked partly through WhatsApp groups of paediatricians comparing cases. Another critical field that runs on Meta ’s infrastructure is disaster response. The World Bank documented that during 2014’s Cyclone Hudhud in Andhra Pradesh, India , the Public Works Department restored connectivity to a 1.8-million-person city primarily by coordinating engineers through a closed WhatsApp group with the District Magistrate in it, without any formal meetings and orders, which ultimately led to most roads becoming functional within three to four days. During the 2023 Turkey earthquakes, volunteer-formed WhatsApp networks processed 5,800+ messages in one week for needs assessment and rescue, and in Syria, the White Helmets have run an emergency dispatch system over WhatsApp since 2021, because the country’s emergency number infrastructure is largely destroyed and WhatsApp ’s compression algorithms work where almost nothing else does. It’s not just individual organisations, but even whole governments are dependent on Meta . Buenos Aires for example ran a COVID-symptom triage chatbot on WhatsApp , and Lebanon’s public health ministry launched an automated WhatsApp service in April 2020 to disseminate updates on the pandemic. India, on the other hand, offers metro tickets, government services, and bill payments through WhatsApp chat interfaces . On top of that, for example, the Philippines’ UAE consulate operates consular emergency hotlines on, you guessed it, WhatsApp . Last but not least, there’s migration. Roughly a quarter-billion people live outside their country of birth. Most of them use WhatsApp as their primary connection to family, because international SMS is expensive and unreliable and Skype is, well, dead. Multiple peer-reviewed studies on Trinidadian , Pakistani, Ghanaian , Polish, and Kenyan diasporas also converge on the same finding of WhatsApp being the primary technology of transnational family life in 2026. So to go back to our initial thought, let’s imagine WhatsApp shutting down in an instant, with this dependency graph in mind. What follows is a hypothetical scenario sketched from the documented impacts of past (shorter) outages, scaled up by the duration and finality of the event, and informed by the dependency layers described above. It’s a scenario and not an actual prediction. The shutdown hits during European afternoon, which means American morning, Indian evening, East African afternoon, and Indonesian late evening. The first signals show up on Downdetector and on non- Meta competitors. In 2021, the six-hour outage generated 14 million reports inside the first few hours, but this time the number is likely much larger. Behaviour inside the first hour is uneven and largely confused. In most places, users assume it’s a routing problem, a local carrier issue, or a phone bug. They restart the app, then their phone, then their router, then they check Twitter X , Instagram , TikTok , Telegram , maybe Signal , or Facebook Messenger , depending on what they have installed. Telegram and Signal both see app-store download spikes within the first 30 minutes, as it happened during the 2021 outage, with Signal reportedly adding “millions” of users that day . The first noticeable failures show up in commerce. A food-truck operator in São Paulo who takes orders via WhatsApp can no longer receive them. A small clothing brand in Mumbai whose entire sales pipeline runs through Click-to-WhatsApp advertisements sees its ad spend continuing to bill while the conversation endpoint returns errors. In Hong Kong, a logistics coordinator who confirms container pickups via WhatsApp loses the day’s confirmation chain. In Idlib, Syria, the White Helmets dispatch room realises within minutes that emergency calls are not coming in, and civilians have no fallback channel. It is likely that three things start happening in parallel. First, mass migration to apps like Telegram , Signal , and to a lesser extent Messages ( iMessage ), Viber , and Line . Signal ’s servers, which are run on a fraction of WhatsApp ’s infrastructure, are not designed for an inrush of hundreds of millions of new accounts and start to degrade in some regions. Telegram , which has spent a decade preparing for exactly this scenario, holds up better but still struggles with its own issues. Ultimately none of the alternatives are suitable for the people who had built their workflows on WhatsApp . The second thing that happens is commercial collapse , which is the biggest 12-hour story, but still largely invisible from Western media. In Brazil, Indonesia, Nigeria, India, Pakistan, Bangladesh, Vietnam, Mexico, and probably 50 other countries, the small businesses that route everything from orders and prices, and photos of goods, to delivery confirmations, and payments, through WhatsApp have lost their primary revenue channel. A clothing brand in Ireland reportedly lost thousands of euros in a single afternoon during the 2021 six-hour outage. Multiply this by twelve hours and by the entire tail of informal commerce that lives on the platform and the figure runs into the billions. The third thing is health-system stress . Group consults that normally take an hour over WhatsApp become almost impossible. The Eastern Cape HIV-management network in South Africa, the Malawi-UCLA clinical link, the Haitian ED coordination groups, the Zambian rural-nurse mentoring channels, all degrade simultaneously, and while mortality consequences are not yet visible, they are happening nonetheless. In several countries, government officials begin issuing statements through whatever channel is still functioning. After the first 24 hours it becomes clear that the impact this situation has is roughly inversely proportional to a country’s investment in alternative digital infrastructure. The United States and Western Europe are mildly inconvenienced, and India is moderately disrupted, mainly because the country has built duplicate rails, hence UPI runs over many apps. After all, SMS still works, alternative payment apps exist, and government services have their own portals. However, countries like Brazil, Argentina, Mexico, and most of sub-Saharan Africa, on the other hand, are in serious trouble. In Brazil, by the end of day one, the financial press is comparing the situation to a partial shutdown of the national payments system. Pix transfers still work, as those run over the central bank’s infrastructure and not WhatsApp ’s, but the merchant-customer communication layer that drives Pix transactions for millions of small operators is offline. The same is true in Argentina, where the inflation-driven culture of constant price renegotiation between vendors and customers happens, in practice, almost entirely on WhatsApp . Another area that starts to fail is migrant remittance. People working in the Gulf, North America, or Europe typically coordinate transfers with their families via WhatsApp , where they confirm the recipient’s details, send screenshots of receipts, or sometimes route the money through informal Hawala -style networks where trust is established and maintained by daily messaging. These workflows don’t fail completely on day one, but they slow and break in ways that don’t show up in formal remittance statistics for another week or two. In Latin America, the first major political consequence appears in the form of misinformation that previously circulated within closed WhatsApp groups , which now has nowhere to go and starts spilling onto other platforms. By the end of day one, more than 100 million people have created Signal or Telegram accounts. Both apps experience their first significant performance degradation events. The labour-market consequences start showing up. In India, where WhatsApp is the de facto recruiting and onboarding tool for huge segments of the informal economy, gig workers can’t be reached for shifts. Delivery platforms like Swiggy , Zomato , Dunzo , and their international equivalents, see their dispatch coordination degrade. Some of these companies have parallel in-app messaging, but many have leaned hard on WhatsApp because it was cheaper. Schools also begin to feel it, because in many countries, including India, Brazil, South Africa, Kenya, Nigeria, the Philippines, Indonesia, and much of the Middle East, parent-teacher communication runs over WhatsApp groups. Two days in, schools that have not made the switch to other channels are operating partially blind, and parents are not getting closure notifications, transport updates, fee reminders, or exam schedule changes. In countries with weak alternative communication infrastructure, the second-order effect is mid-week absenteeism as parents simply don’t know whether school is open. On top of it all, Healthcare is also heavily impacted. For example, the Haiti emergency-department-style coordination groups have now had 48 hours to find alternatives, and they have, mostly, but the transition has costs. Case discussions that were asynchronous and 24/7 on WhatsApp are now synchronous and harder to schedule, and rural clinicians in places like the Eastern Cape, Lilongwe, or the highlands of Nepal are once again practising in the relative isolation that WhatsApp ’s group-call and group-message features had alleviated. In several documented studies, isolation correlates with diagnostic delays and worse patient outcomes. In Syria, the White Helmets switch to a patchwork of Signal , SMS where it works, and physical runners, and response times degrade significantly. At this point things start to get political. In a number of countries, including Brazil, India, Indonesia, Nigeria, the Philippines, and South Africa, the question stops being “what is Meta doing” and starts being “why did we let one foreign company become this central” . Telecom operators in several countries pitch the moment as an opportunity to push their own messaging products, most of which have been moribund since 2014, but the pitches fail because nobody trusts the carriers, because those carriers have been quietly delighted to see WhatsApp gone, given that it eroded their SMS and voice revenue for a decade. In a few markets, regulators float emergency-decree-style proposals to nationalise messaging infrastructure or build sovereign alternatives. And while most of these proposals are clearly performative, some are not. India and Brazil both have working national digital identity and payments stacks that could, in principle, host a public messaging layer. It remains to be seen, though, whether the political will to build one persists past the first month. Public health authorities in Lebanon, Buenos Aires, the Philippines, and several African countries are now running emergency communication operations across multiple fallback channels. None of them work as well as WhatsApp did and things like vaccination schedules are missed, and appointment reminders fail. Some clinics see patient no-show rates rise by 30–40% versus baseline. Not because WhatsApp is superior to its competitors, but simply because humans need a long time to adjust to the alternatives that are being put in place. Also, crime patterns shift in interesting ways. A Conflict Sensitivity Resource Facility report on South Sudan, and PeaceRep work on Somalia, both documented that WhatsApp groups were used for both peace-building and for coordinating violence. Removing the platform doesn’t remove either function, as both migrate to other channels, but the migration takes time, and during the transition, coordination of all kinds becomes harder. In several markets, online ad spend collapses because Click-to-WhatsApp ads (a $10B/year business) have no destination, and Meta ’s stock price has already done what you’d expect it to do. The migration to alternatives, mostly Telegram and Signal , with regional pockets going to Line , KakaoTalk , WeChat , Messages ( iMessage ), RCS , and a long tail of smaller apps, has now hit critical mass in most of the world. The migration has not been clean, and group chats with over 200 members have, in practice, often migrated as group chats with around 40 members, because not everyone moved at the same time or to the same app. For business communication, the new world is as fragmented as it gets. A Brazilian shopkeeper who used to take all orders on WhatsApp now has to manage Telegram , Signal , Instagram DMs (still up, but reduced after Meta ’s reputational damage), and SMS. Customer-acquisition costs rise, and customer-retention drops, and several reporters publish stories on small businesses that have permanently closed. For healthcare, the migration is more orderly because the user base is smaller and more motivated. Most major peer-support networks, like the Malawi-UCLA , the Eastern Cape HIV , the Zambia nursing , and the Haiti emergency have stable new homes. The five-day disruption produced measurable degradation, and it is not yet possible to quantify the mortality and morbidity impact. In Syria, the White Helmets have built a partial replacement on Signal and on a custom dispatching tool that their engineers had been prototyping. It works less well than what they had, because the compression behaviour that made WhatsApp viable in low-bandwidth, intermittently-connected environments is hard to replicate. Hence, some dispatches are now arriving via paper notes. Not because decentralized mesh networks don’t exist, but simply because nobody in these organizations has the expertise to implement these alternatives, especially within such a short period of time. The first credible economic estimates of the shutdown’s cost reach the tens of billions of dollars and continue to rise. The estimates are dominated by long-tail effects in emerging markets that are hard to measure precisely. A week in, the question has shifted from “When does WhatsApp come back?” to “What does the world look like without it?” and a growing fraction of the user base assumes it isn’t coming back, so behaviour begins adapting accordingly. Several governments, including Brazil, India, and the EU as a bloc, have announced formal investigations or task forces into how to prevent this from happening again. As usual, however, none of them will produce anything actionable within years. The longer-term effects, that you can already see the shape of by day seven are a measurable productivity hit in emerging markets, particularly for informal-sector businesses, a consumer trust impact across the entire Meta product family, a wave of WhatsApp-replacement startups, most of which will fail due to network effects and generally bad engineering, and the painful realisation that a free product is not the same thing as a public good. Some estimates from prior outage studies suggest that a six-hour WhatsApp outage cost the global economy hundreds of millions of dollars per hour in lost SME activity, weighted heavily toward Latin America, South Asia, and Africa. Extrapolated over seven days and weighted for cascading effects, the seven-day damage is in the tens of billions, possibly higher. This thought experiment is not about Meta eventually shutting down WhatsApp , as it almost certainly won’t do so on its own, given how big of a lever the platform is for the company. In fact, Meta is moving in the opposite direction, as it is building WhatsApp Business into a $45 billion commerce platform, integrating it with payments, and turning ads into one of its fastest-growing revenue lines. WhatsApp is too valuable to Meta to switch off voluntarily, and the regulatory regimes in the countries that depend on it most are nowhere near coordinated enough to force a switch away from it or even just ban it outright. The point is that we have built a planet-spanning piece of communication infrastructure whose ownership, governance, and continuity are concentrated in a single American corporation, that is led by people with questionable values and beliefs, which all in all is a state of affairs that has no historical precedent. Sure, there are other US-based companies that “own digital communications” , like Twitter X and many others, albeit I’d argue that none of those platforms are so engrained into everyday life across many (predominantly developing) nations as WhatsApp is today. The closest analogue in scale is the global SMS network of the early 2000s, which, however, was federated, run by hundreds of carriers and governed by an open standard (GSM/3GPP). SMS was never under the unilateral control of any single entity, despite many carries enjoying a defacto monopoly in their respective home markets. WhatsApp , on the other hand, is a single proprietary protocol, with a single operator, optimised increasingly for the commercial interests of that operator, and treated by the rest of the world (governments, hospitals, schools, small businesses, families separated by borders) as a public utility. The seven-day scenario above is an exercise in realising this dependency. Meta has no public-service mandate and WhatsApp ’s terms of service explicitly disclaim any commitment to availability. Yet a meaningful fraction of the medical communication, emergency coordination, family contact, and small-business activity of the global south runs on top of this disclaimed-availability infrastructure. At this point the LinkedIn thought-leadership crowd would tell you the answer is “diversification” or “resilience” or “multi-channel strategy” and add an inspirational quote alongside the ChatGPT -inserted emojis. Telling a Karachi tailor with 14 customers in a WhatsApp group to “diversify their customer-communication stack” does nothing to solve the problem. The infrastructure they depend on was built and made free at the point of use by a corporation that calculated, correctly, that owning that infrastructure was worth more than charging for it. The bill is paid in attention, in advertising, in data, and in the asymmetric power Meta now holds over a substantial fraction of global communication. While the shutdown will (sadly) not happen any time soon, the dependency, however, exists, and the thought experiment is worth running occasionally (with other services as well… looking at you, Google Mail !) because this exact dependency is what should push us to look for alternatives, and not the implausible event that would make it visible. Network effects may be the biggest drivers for this unhealthy dependency, but I believe that each and every person has the ability to make an impact within their families, their friend-circles and their communities, by choosing to use anything but WhatsApp as their main communications channel, ideally a self-hosted alternative . For almost three decades now we’ve had XMPP available to us, with popular and capable implementations like ejabberd , Prosody , and Snikket existing as open-source software that is ready to be used for communications platforms of any size. As a matter of fact, WhatsApp uses XMPP behind the scenes and is in fact built upon the same great technology stack used by ejabberd . For a “lower-level” alternative, there’s the good ol’ IRC that has been around for almost four decades and that is still thriving . Both of these open standards would allow communities, organisations and even whole governments to build public infrastructure that could in large parts replace WhatsApp . PS: Are you a Jabber user already? Come join the community channel !

0 views
Martin Fowler 3 months ago

Fragments: July 13

Some more of my notes from Thoughtworks Future of Software Development Retreat . When we had our first retreat in Utah early this year, nobody had heard of Harness Engineering . This time we had a whole session on it. When comes to the guide side of harnesses, most of the discussion is about context management. While context windows have increased is size as models get more sophisticated, that doesn’t mean that models will properly focus on the right bits. Models typically only focus attention on part of the context, and to get the best behavior, we need to manage that focus. One attendee keeps their context small, limiting the file to less than 200 lines On the sensor side, we see more attention on computational sensors. Two patterns from one participant was shifting to languages with greater controls, (eg Rust rather than Python) and “leveling up” validation approaches, using more property-based testing and techniques from formal methods. One commented that while they aren’t smart enough to write specifications in a formal specification language, they are smart enough to read it and check it makes sense for their domain. Will our attention on harnesses last long enough for our next retreat? Will the models just get so good that harnesses become unnecessary? Those with some mechanical sympathy for LLMs seem to think not - but are they overly coupled to the current state of technology? I find such speculation tends not to lead anywhere useful, I’ve not seen much success in guessing the future in the past, and with technology as radical as this, I don’t see it being any easier. So for the moment, attention to harnesses pays off. We find it reduces token usage, and also allows weaker models to be useful, supporting such things as local hosting of open-weight models. ❄                ❄ Which naturally segues me to a session on self-hosted models. Increasing token costs have made hosting an open-weight model more attractive, particularly due to the decreasing time for open-weight models to catch up with frontier models. Cost isn’t the only factor, however, many folks find a desire to be independent of the frontier model firms to be the the driving force. After all we’ve seen the U.S. government intervene to deny access to models, increasing the desire for greater model sovereignty. Information security is also something to consider, some attendees just can’t give models necessary data for critical work. Even without that, if someone else hosts the model then their model learns rather than your model. And although recent events have increased interest, several participants worked with companies that had been self-hosting for up to a couple of years. Is this trudging down the same path of self-hosted clouds, which led to lots of folks spending excessive funds on half-arsed private clouds ? The answer hinges upon whether it ends up being simpler to host a model than a cloud, perhaps due to a simpler interaction protocol. The hard part of this may be the talent required to efficiently use the GPUs, managing an inference data center currently isn’t a widely available skill. Even self-hosted models are a cost to operate, capital costs in GPUs, ongoing costs in electricity. The physical design of a data center can affect optimal usage. There’s an opportunity here for professional services firms to help companies manage this. Cost control also involves teaching people to pick the right model for the job. Can we teach engineers, or indeed other users, to pick a less-powerful model? This, of course, could be a job for model itself, acting as a broker, deciding which model is the best choice to tackle certain jobs. Self-hosting may lead to a greater use of fine-tuning. Currently that’s a niche activity, but over time we could well find that models that are fine-tuned to a particular domain need less reasoning, consume less tokens, and thus are cheaper to operate. We are seeing models trained specifically to support programming. As with any topic with this degree of uncertainty, the big win isn’t finding the right answer, but coming up with a strategy that will cope with the inevitable and unpredictable changes. ❄                ❄ After an event like this, many people come up to me and ask me to make some grand summing up. I hate this, because I rarely leave these kinds of event with some grand narrative. Even after mulling on it afterwards (in writing the above notes) I still usually don’t have one, and distrust one that forms, as my skepticism includes attempts to make coherent narratives of an event that’s naturally rather jumbled. However my failings are irrelevant this time, because Kief Morris has put together such a narrative, and it’s a convincing one , even to a narrative-denier like me. The sessions had different titles and different casts, and on the surface they were about different problems. But they weren’t. Nearly every one of them was a different facet of the same argument. How much do we let an agent decide, and how do we stay confident in what it does? He looks at code review, questions whether it matters, but sees that the rigor that many associate with code review shifts to other forms. He describes the disagreements about how much we should trust an agent to identify and fix production incidents. He sees that the contrast between how much leeway teams give to agents depends on the context they are operating Underneath all of these sessions, the operations debate, the wide-remit team, the dark-factory spectrum, the argument about who’s allowed to steer the model, people were making the same handful of choices over and over about a single thing: the unit of work they were prepared to hand to an agent. How big it is. How much of the job it covers. What you do to get it ready to hand over. How you check what comes back. What you put around the agent to keep it inside the lines. Different rooms set those differently, but they were setting the same controls. ❄                ❄ Sam Ruby convened a session called “Bring me a Rock”. The name evokes a particular kind of management dysfunction. The manager tells his underlings to bring him a rock, and then starts rejecting the results without explaining why (“no not that one”, “no not that one”) until eventually one rock matches the unstated expectation. It names a manager who substitutes serial rejection for the work of saying what they want, and makes you pay for their unfinished thinking one rock at a time. Sam had already written why he thought with LLMs, this changed from a slur to a defensible way to work . When its a bunch of tireless machines with endless patience, that return new rocks in minutes rather than days, then an approach like this (using the brainstorming register becomes a defensible way to work. Sam described the discussion : The room pulled it somewhere narrower than I’d framed, and the narrower place was the more interesting one: not how to explore by elimination but who should even be allowed to. Product managers, increasingly people managers, are reaching for these models directly, and seasoned engineers get measurably better results from them than untrained people do — so the worry followed. If expertise is what separates a good outcome from slop, should non-engineers be steering the model at all? It’s a fair question, and I think it’s the wrong one, because it mistakes the act. When a manager reaches for an LLM instead of routing the work to the team that reports to them, they didn’t pick up a tool — they made a hire. And you don’t ask permission to manage your own team; a manager who decides a piece of work is better given to a new participant than to the existing one is doing the most ordinary thing a manager does. Framed that way, the permission question dissolves into an older, better-understood one — the one Drucker named in 1959: when the worker knows more about the specifics than the manager does, you manage by objective, not by method. The non-engineer steering an agent is exactly that manager, out-known by the thing they’re directing, and the slop the room feared is the old danger of managing by method when you should be managing by objective. The question isn’t may they hire? It’s do they know how to manage by objective? — which you can teach, hire for, and hold people to without anyone first becoming an engineer. Sam’s article explores managing an LLM by objective, giving it a goal rather than a task. And Kief’s earlier point about the essence of the discussion still holds: how confident can we be that it’s done the right thing? We can outsource many things, but not the acceptance criteria, at some point there’s a human request and a human judgment on whether that request was properly executed. But the danger lies in important unstated objectives, unstated perhaps because they weren’t even imagined. It’s easy to state objectives around desired functionality. Give me a an application that will examine my emails and form a todo list for today. But behind that simple statement is a thicket of unstated assumptions. We tend to assume The Genie won’t include any undesired functionality, perhaps deleting emails it thinks are unworthy of our attention. We assume it won’t let an email tell it to send private information to [email protected]. We have some hope here - we hear more experiences that suggest that recent models can do an excellent job of finding (and hopefully fixing) security holes. The careful precision of the machine outruns the sloppy if imaginative thinking in squishyware. Perhaps we can assume the genie can take care of some of our unstated objectives. Conformance tests (sensors) are more valuable than specifications (guides), but it’s hard to imagine all the conformance tests that are needed to say what shouldn’t happen. Furthermore, building software is about exploration, finding out how a workflow can evolve as machines are embedded in the process. For a human to guide that process, we need some understanding of it. My sense is that model building is still important, and while I agree that the genie can take an active role in that construction, I don’t think the human can entirely outsource it. Even if the genie builds the model itself, it needs to teach us that model, because the model helps us imagine and communicate the goals, the objectives that we give to the machine. ❄                ❄                ❄                ❄                ❄ If you follow my feeds (which you probably do if you’re reading this), then you’ll know that Birgitta Böckeler has written a couple of memos on working with local models. She first looked the factors that influence how viable they are for programming , and then related some of her recent experiences evaluating such models . As a nice, if accidental, complement to these, Sebastian Raschka wrote a detailed guide to his local model environment . Like Birgitta, he’s found the Qwen 3.6 model to be the current sweet spot for local agentic programming. ❄                ❄                ❄                ❄                ❄ Simon Willison shares a useful tip to save money while using the latest Anthropic Fable model Tell Fable to use other models for smaller tasks, applying its own judgement about which model to use. ❄                ❄                ❄                ❄                ❄ Josh Comeau writes a blog and online courses for developer education, primarily front-end web material. His been successful for most of this decade but has found his online courses have had only ⅓ the sales this year . He attributes this to AI, partly as people worry if it’s worth spending money on a job that may not have a future, but also because AI can provide personalized tutoring. ideally, it shouldn’t cost any money to learn stuff. But I sorta worry about how this is supposed to work, going forwards, if there’s no incentive for people to make high-quality free content. I’ve spoken to a few course creators now, and we’re all seeing the same trend. Revenue down 50%+. Fewer people engaging with our content. People switching to LLMs, which slurp up all of our work and regurgitate it, without consent or compensation. It feels pretty bleak. 😅 ❄                ❄                ❄                ❄                ❄ John Gruber is annoyed that Claude’s desktop app for MacOS in uses Electron . Electron guarantees that an app feels just as wrong on all platforms. He has some tasty invective for the folks at Anthropic with ties to the Electron platform. Finding out that one guy — who is a senior Electron maintainer — has led the teams for the desktop clients for Slack, Notion, and now Claude is like discovering that it was one guy — whose family business was a distillery — who helmed the Titanic, piloted the Hindenburg, and then served as air traffic controller for Amelia Earhart. The deeper question here is whether there should be a future for cross-platform front-ends in the world of agentic programming. There’s lots of evidence that coding agents do a great job of building the same thing in multiple languages and platform ecosystems. That should mean that the days of least-denominator cross-platform UIs are numbered - and that number is small. ❄                ❄                ❄                ❄                ❄ Dan Davies tries to draw a distinction between interactional and contributory expertise . Contributory expertise is that held by people who are doing the work to advance a field of study, interaction expertise is held by folks that spend time talking to contributory experts, building up a decent store of knowledge themselves, but not steeped in the day-to-day of the work. it seems to me that there is an important distinction here, which is not any less important because the dividing line might be difficult to establish empirically, or even if that line turns out to be in a different place from where we guessed it was. As well as difficult cases where it’s not clear, I think we could also come up with cases where the distinction between interactional and contributory expertise would suddenly become very clear and important indeed – the ones where someone who was faking it got “found out”. And so the question that I think is quite important is whether there is a similar kind of distinction between the kind of expertise that it’s possible for a machine to get by industralised consumption and interaction with a much larger corpus of literature than any human being could inhale, and genuine contributory expertise that could apply to entirely new situations outside that literature. As a human, I’d like to think I’m more of a contributor than an interactor (especially given my increasing introversion), and thus relatively safe from being forced into obsolescence by silicon. But I’m also aware that my career is devoid of any original ideas, my skill is only that of someone who is good at selecting and explaining the ideas of others. (As Brian Foote put it more memorably: “an intellectual jackal with good taste in carrion”.) But there’s skill in being a good jackal too - and we don’t really know yet where the real boundaries of the LLMs will lie.

0 views

Let AI Burn

If you liked this piece, you should subscribe to my premium newsletter. It’s $70 a year, or $7 a month, and in return you get a weekly newsletter that’s usually anywhere from 5,000 to 18,000 words, including vast, detailed analyses of NVIDIA , Anthropic and OpenAI’s finances , and the AI bubble writ large (updated to version 3.0 a few weeks ago). My Hater's Guides To the SaaSpocalypse , Private Credit and Private Equity are essential to understanding our current financial system, and my guide to how OpenAI Kills Oracle pairs nicely with my Hater's Guide To Oracle . This week, I published the Hater’s Guide to Softbank — a sordid tale of tech’s most degenerate gambler, who, thanks to a couple of early lucky wins, has managed to set the foundations for the AI bubble’s biggest (and possibly most gratifying) downfall. And, on Friday, I’m going to take a deep dive into the memory industry — and the reason why you can’t afford a new gaming PC.  Subscribing to premium is both great value and makes it possible to write these large, deeply-researched free pieces every week. Soundtrack: Mastodon — Streambreather No bailouts, no handouts, no special treatment, no tax breaks, no CHIPS act, and no sovereign wealth fund. It is time to tell the AI industry to go fuck itself, because it’s effectively done the same to the rest of society. This industry is unworthy — a sham conjured up by a tech industry that’s run out of ideas, a trillion-dollars’ worth of manufactured consent and entirely-avoidable financial crises — and should not be protected under any circumstance.  Every single time you hear somebody discuss “bailout” or “too big to fail” or “sovereign wealth funds,” know that this is the industry, on some level, attempting to create the air that it cannot die , when in fact every one of these companies is just as weak and brittle as any other startup. I also think that the media — and the world at large — is too ready to accept the prospect of a bailout after watching those who drove the world into a ditch in 2008 escape blame, and I must be clear: the AI industry is very different to the financial industry. It is inessential to the economy, and its relevance is only as large as the hype campaign that sits behind it.  This is an industry of losers that has inflated only because of the joint manufactured consent of Silicon Valley, the mainstream media, and an enshittified stock market that rewards grifting and circular financing . OpenAI had $5.7 billion and Anthropic a little under $5 billion in the first quarter of this year — and those revenues mostly came from companies that were burning AI tokens at a horrendous rate because they’d just been forced to pay the actual cost of AI — and now everybody’s pulling back on that spend .  Generative AI will not bring us AGI, nor does it do much of what we associate with artificial intelligence. It is not autonomous. It is not “intelligent.” It does not have thoughts, or “knowledge,” and no matter how many layers of harnesses and scripts you put on top of it, it is still ( per OpenAI ) mathematically certain to hallucinate. I estimate that at least 70% of the entire AI industry’s revenues are made up of OpenAI and Anthropic’s compute spend , and as both companies are horrendously unprofitable, this means that the AI industry is, for the most part, venture capitalists funnelling money to hyperscalers so that they can funnel that money to NVIDIA or data center capex. If this software were worthy, it would stand on its own two feet. It wouldn’t need circular financing and a cult of personality to prop it up, either. If it were truly special, there wouldn’t need to be an army of crazed acolytes that attack you for not pledging yourself to the graveyard smash. There has never been a tool or product in history sold with such hysteria and aggressive monocultural force that has ever turned out to be anything more than a grift. Some people have developed unhealthy relationships with large language models (LLMs) and the companies that make them, and that, not any certainty or proof of Artificial General Intelligence (AGI), is what motivates them.  This software is uniquely dark, both in what it unlocks in some people through its use and in the sense of the entities that sell it. Some people are in genuine awe of each of the rotation of clammy, soulless pod-people that saunter out of Anthropic every few weeks. Each one sounds a little weirder, more cultish, more disconnected from the real world. Silicon Valley may believe itself atheistic, but Anthropic has a worrying sense of fanaticism, both in the people that work there and its fanbase. Imagine the absolute worst fanbase of a video game possible, and then add layers of financialization, grifting and high school drama laced with pseudo-religious attachment. All for a fucking app!  Please, people. Nobody in the real world cares about “loops.” Nobody is thinking about tokenization. If you said inference to a guy on the street they’d take you to see a doctor. Nobody gives a shit. They don’t know what OpenClaw is either. Grow up. Go outside. You sound like a lunatic. Does your mother know how many Claude 20x accounts you have? It’s obsessive!  Anyway, the only reason that AI has any presence in our economy is that Microsoft, Google, Meta, and Amazon are intent on spending more than $765 billion in capital expenditures in 2026 and a trillion more in 2027 because they have no other hypergrowth ideas, even though generative AI has yet to show any real potential as something that can drive meaningful revenues (let alone profits), as evidenced by the fact that none of these companies break out their actual AI revenues , a point I made on CNBC late last week .  Google does not have the next Google Search, Microsoft does not have the next Microsoft Office, Meta does not have the next Facebook, and Amazon does not have the new AWS. That’s why they need you to believe that AI is a big deal without them ever having to prove why outside of capital expenditures. They want you to assume that all this money can’t be wrong , even though when you remove OpenAI and Anthropic ( who represent 89% of the revenues of the largest AI companies ) the AI industry is, at best, pulling in $20 billion in annual revenue. And lord do they want you to say “it’s early,” and that it’s just like the Dot Com Bubble , all so that you’ll either accept AI as your lord and savior or, alternatively, help justify one of the largest misallocations of capital in history as “building useful infrastructure.” Newsflash! AI GPUs are useful for generative AI and not much else. Every “innovation” in LLMs has only been made possible by throwing billions of dollars at the problem either in headcount or compute costs — every ounce of talent in the tech industry, every bit of media attention, every dollar of capital expenditures, all focused on one industry that has successfully created LLMs that are more expensive and significantly less useful than human beings .  The reason every AI person speaks in pie-in-the-sky hypotheticals is that the actual outcomes are decidedly mediocre when you compare them to their ruinous costs. Anthropic and OpenAI raised (assuming the rounds completely close) over $300 billion in 2026 alone, and take up the vast majority of available AI compute. They need you to speak in the future tense, because nothing — absolutely nothing — about what’s been created so far justifies even a fraction of its financial and infrastructural cost. When the AI bubble bursts, none of this infrastructure will be particularly useful. As I said in my premium about how this is worse than the Dot Com Bubble , GPUs are not fiber optic cable , and when the bubble bursts, NVIDIA chips will either be sitting in the coffers of the largest tech companies in the world, held by asset managers, or auctioned at a steep discount by creditors. These are not going to be useful for hobbyists, nor will they be cheaper to run, nor will incomplete data centers be cheaper to finish. The Dot Com era fiber overbuild was a result of a complete misread of demand signals, per Justin Kollar : It’s tempting to compare this to GPUs, but it doesn’t make sense at all!   You see, internet demand was a result of people wanting to get online and use the internet, with the leftover “useful infrastructure” having a blatantly obvious use case after the bubble burst, albeit one that took a lot longer to arrive than investors had hoped. There was no question about how that gear might be used or for what purpose one used fiber optic internet or networking gear, nor was there any question as to the underlying business model of offering an internet connection might mean.  We were also fairly early, and internet speeds were atrocious. In 2000 , only 52% of American adults were using the internet, and by 2003, that number had only increased to 61%. Per the World Bank , in 2005 only 16% of the world used the internet, and in 2024, that number had increased to 71%. When the internet was connected to via a 56k modem, access was charged by-the-minute, and obviously much, much slower than even the primitive (though expensive) broadband connections of the day.  While we’re used to connecting at speeds that make using a web-based app near-indistinguishable from one that runs on our computer, back in 2000, 2001, or 2002, the average US internet speed was, at best, 400 Kilobits/s , or roughly 50 kilobytes a second, compared to the average US internet speed of over 200 Megabits per second , or 25 megabytes a second.  Generative AI, on the other hand, is fucking everywhere , and anyone with an internet connection experiences it in effectively the same way. It’s non-consensually available in effectively every app — every Facebook, Google and Microsoft account, for example — and every media outlet known to man has mentioned AI multiple times since 2023. OpenAI and Anthropic might claim they need more data centers, but it’s unclear what “more data centers” actually achieves other than propping up NVIDIA and giving hyperscalers something to invest in.  A lack of data center capacity isn’t holding back people from using generative AI, nor is it stopping anybody from launching a product, nor can anyone actually express what it is that they’re being built for other than “reasons for Anthropic and OpenAI to spend money.” Anthropic’s supposed lack of compute did not stop it training or launching Mythos or Fable, and when it bought hundreds of megawatts of compute from SpaceX , the biggest news was that it expanded rate limits to allow users to burn $8,000 worth of tokens for $200 a month . Nothing about the painfully slow pace of data center development appears to be restraining a single AI company, outside of hyperscalers complaining they could’ve made more money from either Anthropic or Meta . In fact, the entire argument for more data centers appears to be “we need more compute so that people can buy it” far more than any cogent position around what these capacity shortages actually mean.  Who are the companies lining up to spend billions of dollars of compute — or, to be more specific, spend $435 billion or more to justify the $1 trillion in GPU sales that NVIDIA claims it’ll have by the end of 2027 ? That’s how much demand we’ll need. As NVIDIA intends to sell over a trillion dollars of Blackwell and Vera Rubin GPUs by the end of 2027 , it needs to have around (assuming a PUE of 1.35) 40GW of data center capacity built to support the 30GW+ of GPUs it will have sold . At about $12 a megawatt of critical IT (IE: the stuff in the data center that runs AI compute, and not everything else, like the cooling systems and any transmission loss), that’s $435 billion.  OpenAI estimates it’ll spend $50 billion on compute in 2026 , and Anthropic will likely spend comparable amounts. Otherwise, the only other player — outside of Microsoft, Google, and Amazon renting ( or backstopping ) capacity for Anthropic and OpenAI — with any meaningful compute spend is Meta (with Nebius and CoreWeave )... and Bloomberg is reporting that Meta is planning to start selling its compute because it doesn’t need all of it .  You’ll be shocked to hear that it might be renting some of that capacity… to Anthropic . Now NVIDIA is agreeing to financially backstop young cloud providers buying their GPUs by promising to rent back any unused capacity, yet another sign that actual, real demand does not exist at scale . AI boosters with black mold problems will say “this is just to help them raise debt,” to which I say “If the demand actually existed in any provable way, NVIDIA wouldn’t have to pay its customers to buy its products!”  Anyway, my larger point is that there was real demand during the dot com bubble, and LLMs’ demand appears decidedly artificial outside of OpenAI and Anthropic, who cannot afford to pay without unlimited venture capital funding.  This shit isn’t going to become magically cheaper once the bubble bursts, and considering the demand doesn’t appear to be there at scale with two-thirds of all venture capital funding focused on AI , I’m not sure what people expect to happen. Right now is the number one time in history where we should see near-infinite demand for compute across every single surface, and way more deals for compute capacity for companies other than the same four or five companies. Right now, as I’ve discussed before , Anthropic and OpenAI take up the majority of compute, leaving the rest of the world to fight for the leftover scraps, and because data centers take 18 to 36 months to build , capacity is taking forever to come online to fill the indeterminately-large amount of demand that remains. Nevertheless, said demand can’t be that large, otherwise we’d A) have other companies trying to build their own compute (other than Poolside, which failed to raise money to do so ) and B) massive remaining performance obligations — hundreds of billions of dollars’ worth — rather than the grim truth that 50% of hyperscaler RPOs are from Anthropic and OpenAI , inflating obligations by $448 billion, hiding the fact that Microsoft’s RPO growth is flat year-over-year and Amazon’s is only growing at a modest 20% when you remove Anthropic and OpenAI’s hundreds of billions of dollars’ of compute spend. Google’s is a little messier, as it’s hard to parse exactly how large its deals with Anthropic are thanks to its backstops and circular deals around Anthropic and its TPU chips . There’s also the compelling question as to what it is that anyone would be picking up once the bubble bursts. Demand for AI services is a direct result of the entire media, tech industry and venture capital ecosystem manufacturing consent for the use of LLMs, forcing them into every corner of every experience, something that will most decidedly end once the stock market and investors cease incentivizing it.  Once every media story isn’t about AI, once every Business Idiot with AI psychosis stops posting about it every day, when everyone stops asking about your AI strategy or wanking on about “sovereign AI,” it’ll become blatantly obvious that the actual demand for AI was not particularly strong. We have little compelling evidence that providing any inference-based services is profitable, which means that even if open source AI outlives the frontier AI labs, it’s unclear who would actually power the infrastructure. People can come up with however many weird blogs where they’ve done some napkin maths to try and extrapolate a potentially profitable inference provider, but I’ll only believe that one is profitable when someone shows me some fucking profit. And to be clear, without that profit, it’s unclear why anyone would offer these services at all. When you rent out a GPU cluster, you do so based on anticipated demand and the quality of service you want to provide. If you order too much, you’ve got a bunch of fallow capacity you’re paying for (and will lose money on), and if you order too little, you’ll have either unstable services or money left on the table…and even then, it’s unclear how profitable that would be.  AI demand is, at this point, a direct result of societal pressure and non-consensually overwhelming customers with AI features. While there are people that like and pay for ChatGPT or Claude, those who do so on a subscription basis are doing so because they can get $30 to $40 of compute for a dollar . The vast, vast majority of AI compute demand is from services provided to people either for free or sold at such a massive discount that it’s impossible that anyone on a $20 or $200-a-month plan could even afford these services had they paid their actual token cost. To paraphrase Cory Doctorow, your demand is based on selling $40 for a dollar. That’s not a real business, nor is that organic demand. One could argue that “these services will become cheaper,” but that would require them to… become cheaper. More compute isn’t (and hasn’t) lowering the cost of AI. Newer GPUs aren’t lowering the cost. Barely-tested Broadcom GPUs , Amazon Trainium XPUs, and Google TPUs aren’t lowering the costs. Even if they were to somehow magically do so in the future, what do we do with the H100, H200, B100, B200, B300 or AMD GPUs? Melt them down for scrap? Steal the RAM? Build a GPU fort?  The Dot Com (and, by extension, telecom) Bubble was never a question of whether the internet was a useful thing that people would pay for , nor were there journalists and dodgy studies that desperately pleaded with us that AI is here, and it’s real.  Everybody has access to AI now! They can all see it and use it if they want to, and they’ve got lots and lots of ways to pay for it! Maybe the reason that AI revenues are so putrid is that they don’t really have any reasons to pay for it, either because the free services do most of what they need (IE: google searches) or subsidized subscriptions that cost $200 a month allow them to burn as much compute whipping up HTML-based calorie tracking apps that get two users. Every time I read somebody on Twitter say that “we’re early” or that “most people haven’t even tried agents” I feel like screaming. Motherfucker, everyone is talking about agents in every single media property all the time . AI boosters will refer to literally any AI feature as an agent, even if it’s a basic web search or generating code. The reason that most people are kind of “meh” about AI is that it doesn’t do things that they associate with AI (autonomously and automatically taking care of the things they need with little prompting or coaxing), everybody knows it hallucinates, and AI data centers are horrifying monoliths of capital that get massive tax breaks, use a ton of water , belch toxins into the air , and are being built by faceless corporations, ultra-oafs like Kevin “Mr. Dogshit” O’leary , or charmlessly damp Valley elitists like Altman and Amodei. Every single person freaking out about “what if China does AI better than America” is living in a child’s fantasy. Oh no! China might get Mythos-level AI? Bad news folks! Anthropic itself already admitted that cheaper models — including Claude Haiku 4.5 and Kimi K2.7 — were able to identify the very same vulnerabilities as Fable (so, Mythos with guardrails).  China has cheap power, data center capacity, and NVIDIA’s Blackwell GPUs . The thing that everybody is scared of has happened already, and you know what else happened? Nothing, because they, like American AI labs, are building LLMs. The only thing that American labs are scared of is cheaper open source Chinese models offering similar performance to their premium products , something that has also already happened.  Remember: the only people that can afford to build data centers are either hyperscalers ( that are now having to fund the buildout with debt as their cash flow turns negative ), Oracle ( which will die if OpenAI can’t pay it ), unprofitable neoclouds , and land speculators. AI data centers are massive, expensive operations, and raising money to finish (or furnish) one after the bubble bursts will be very, very difficult. I realize that everybody wants there to be a happy ending after all of this collapses. I get that it’s easier to think of things in familiar terms — even if said terms involved a 77% drop in the NASDAQ — because there was something good and nice at the end. But doing so only serves to help protect the interests — and brands! — of venture capitalists, asset managers, private credit funds , hyperscalers, captured tech and business journalists and sell-side analysts that insisted on ignoring every warning sign and waving away problems by saying it was “just like Uber ( nope !)” or “just like Amazon Web Services ( between 2003 and 2015, Amazon spent $29.7 billion on capex, normalized for inflation ),” or simply saying that “yes it’s a bubble, but bubbles lead to great industries.” GPUs aren’t dark fiber! GPUs aren’t fucking railroads! GPUs are GPUs! They are used for basically one thing ! And that one thing lacks meaningful demand outside of subsidized services and circular financing!  And now people are discussing a bailout like this is 2008, and I must be clear how different this is, and how little it resembles the Great Financial Crisis! The AI industry has demanded everything from us — more money than has ever been invested, more power than anything has ever needed, the stolen works of millions of hard-working creatives , so many GPUs and so many data centers that it’s causing a global supply chain crisis and a new class of RAM and storage-based inflation , the majority of venture capital funding ,  and constant attention focused on an endless campaign of fear-mongering with the express intention of hyping a technology based on a mixture of mysticism and outright lies — and still, even as we enter the late innings of the bubble, it wants more.  Capital-hog Sam Altman has floated the idea of handing 5% of OpenAI to the US government , a stake worth around $42 billion, claiming that (to quote the FT) “...giving the public a financial stake in the company is the best way to share the upside of AI,” failing to note what said upside might be, likely because there isn’t one unless “the public” refers to “the shareholders of OpenAI.”  It isn’t clear how this would happen, outside of it requiring congressional approval as a result of the Takings Clause of the Fifth Amendment , which states that “private property [can’t] be taken for public use without just compensation,” meaning that the US government would likely have to buy the stock at whatever valuation it considered “just.”  Yet the FT had one other interesting tidbit — that Altman is suggesting that whatever this is would “...would involve other US AI companies handing over a similar stake, although it is not clear if the other labs would be willing to do so”: This is, just to be clear, not a bailout. Even though it’s blatantly obvious that Altman wants to cozy up to the Trump Administration and, he hopes, get $42 billion of funding to attach his questionably-valued quasi-startup, $42 billion is $8 billion less than OpenAI will spend on compute in 2026 , and considering OpenAI has projected to burn $852 billion through the end of 2030 , that 5% stake would only exist to prolong the inevitable. You see, a bailout usually has an endpoint — a time at which the company in question no longer needs the funds.  So, let’s be clear about something : we’re actually in several bubbles at once. The great financial crisis, by comparison, was two major bubbles (per my piece on how AI Isn’t Too Big To Fail from a few months ago) — the over-investment and speculation on mortgages (both subprime and otherwise), and the collapse of the commercial paper (a type of loan) market that kept much of the banking system functioning, which was the real “Too Big To Fail”: Commercial paper was, at the time, often paid off using more commercial paper, and when AIG’s credit rating dropped in the middle of September 2008 , it was unable to roll over its debt (by which I mean “get new commercial paper to pay off its old commercial paper”), and money market funds like Fidelity couldn’t even buy it anymore because it wasn’t investment grade, which meant that AIG couldn’t pay back its loans.  While I won’t recount the entirety of the premium (mostly because it’s super long), AIG was deemed “Too Big To Fail” because it would’ve exploded the markets had it done so. Michael Lewitt, an economist and money manager, described a hypothetical AIG failure as being “as close to an extinction-level event as the financial markets have seen since the Great Depression” in a New York Times op-ed: Yet the real “Too Big To Fail” was far quieter and more malignant, taking the form of trillions of dollars funnelled to banks: The banking system ran (and still runs) on overnight facilities like the federal repo market, where financial institutions offer up collateral — like, say, mortgages — as a means of funding their day-to-day operations. Previously, money market funds were the lenders in the repo market…except they were now a little hesitant to take that collateral, which forced the government to step in with the PDCF (which traded risky, frozen assets like subprime mortgages for cash to avoid a default) and the TSLF (which traded risky bonds for US treasuries). Absolutely nothing about these facilities or anything to do with “too big to fail” were to do with stabilizing the stock market, which was effectively cut in half , with unemployment spiking to 10% . These measures existed exclusively to protect the financial system, with only $46 billion (about 10%) focused on trying to save homeowners from foreclosure , and in the end, to quote a congressional panel from 2009 , “...the panel sees no evidence that Treasury has used TARP funds to support the housing market by avoiding preventable foreclosures.”  The Troubled Asset Relief Program (TARP) spent over $400 billion to bail out the banks, financial institutions and auto industry that would’ve collapsed as a result of an economy-wide lending freeze. Nobody went to jail, nothing really changed, and banks still don’t have to keep reserves thanks to changes made around COVID. By comparison, OpenAI and Anthropic are systemically irrelevant, much like the rest of the generative AI industry. While their existence supports the overall symbolic value of the US stock market, their actual economic presence is minor, outside of what I estimate is around $75 billion to $100 billion of 2026 compute spend and what will likely be around $60 billion of combined revenue, with the rest of the AI industry having so little that it’s barely worth thinking about. It’s also unclear what you’d bail out, unless the plan is to feed them capital for all eternity until they work out how to run a functional business (so, forever). Neither of them have significant debt — and Broadcom is backstopping $30 billion of Anthropic’s $35 billion TPU deal with Apollo — and their equity positions (outside of SoftBank, which I’ll get to) are only load-bearing to venture capitalists in the sense that their fund vintages will painfully sour if they’re unable to go public.  There is no avoiding the carnage to come, outside of there being somewhere in the order of ten to a hundred times the demand for AI compute by 2030 that exists today, which would require AI compute to be larger than the $779 billion that the software industry earns annually .  There is no bailout that can reverse the trend once demand wanes for NVIDIA’s GPUs after hyperscalers reduce their capex, which will in turn kill the revenues of Taiwanese ODMs that build AI servers for hyperscalers , which will in turn kill the revenues of RAM and storage companies, which will lead to a prolonged depression throughout a semiconductor industry addicted to hopium peddled by a tech industry ruled by Business Idiots that have no idea what to do other than hire people, fire people and spend money .  As I’ve said many times, people are conflating massive capital expenditures — invested through debt-fueled data center speculation and hyperscalers bereft of hypergrowth ideas — with real, diverse and consistent AI demand, pumping valuations based on vibes rather than reality , which means that when vibes take a violent, permanent shift, nobody has anything to point to as a means of turning people’s frowns upside down. The collapse in value of AI startups wouldn’t be changed by a bailout unless the US government literally invested in worthless startups as a means of propping up venture capital, and said “bailout” would number in the hundreds of billions of dollars, and while I know you’re gonna say “ohhhh Trump is so corrupt oooh Trump will do this Trump will do that,” this is not a rational or logical or even historically-accurate thing to say.  Trump cannot simply mobilize $50 billion or $100 billion. It will go through the House and the Senate, and any bailout of the AI sector would be an incredibly-unpopular decision, infuriating not just those on the left who’ve grown tired of Big Tech, but with those Republicans that pretend to care about working Americans or fiscal probity.  As a reminder, the first vote of the 2008 bailout failed, with Republicans and Democrats each fairly split on how they felt about the bill — and that rejection happened during a time when the US financial system was quite literally falling to shit.  As far as the data center bubble goes, the government is absolutely willing to let unfinished or abandoned properties lay dormant. In the final quarter of 2008, 11% of US homes were empty , or 15% if you include vacation homes.  Banks that have invested in data centers that have yet to be built (or start construction) can (and will) resell the land, though likely at a loss, and land retains value even if you haven’t built a giant warehouse full of GPUs that only lose money. There isn’t a need for a bailout here, and one won’t be forthcoming. After the Global Financial Crisis, builders were allowed to collapse to the extent that the number of construction firms halved in America between 2007 and 2012 . You could argue that Trump “will just do that this time,” or that he’ll “get a bribe” or something, but is that really the best you’ve got? Scary stories about the President? If every answer you have is “but Trump will just do it,” you’re not analyzing, you’re catastrophizing.  And, most crucially, the vast majority of big tech will be fine, at least in the short term, when the bubble bursts. NVIDIA will likely cease being the largest company on the stock market, and the Magnificent Seven will have a dramatic fall from grace, but outside of unforeseen horrendous financial decisions, the worst I could see would be impairments for Microsoft, Google, Meta, and Amazon, and SEC action against NVIDIA if it did actually sell GPUs to China. This doesn’t mean that things won’t fucking suck for anyone in the market, nor that the vast majority of people won’t fucking suffer as they always do when bubbles burst.  Which is why I am making a firm, clear statement to end this piece. I repeat myself: No bailouts, no handouts, no special treatment, no tax breaks, no CHIPS act, and no sovereign wealth fund. It is time to tell the AI industry to go fuck itself, because it’s effectively done the same to the rest of society. These companies must be forced to stand on their own two feet and die with dignity if their wretched business models can’t keep up. The world’s governments have rolled on their backs and shown their bellies to the tech industry for far too long, and have been aggressively conned by some of the richest people alive into believing that fucking Sam Altman and Dario Amodei are building anything other than the world’s least-profitable software.  We do not need a “sovereign AI strategy,” nor do we need “a sovereign AI wealth fund,” nor do we need to “make sure America leads in AI,” at least not when we’re talking about large language models, the underlying technology of ChatGPT and Claude, two of the most over-hyped and deceptively-marketed pieces of software in history.  Whether or not LLMs are a useful tool is irrelevant, because the AI industry has demanded the world hand it as much land and money and as many resources as it desires to continue proliferating a technology that has only ever lost money and has no path to sustainability. The only reason it has gone anywhere is because the tech industry has united around it as a means of hiding from the fact it has no next big thing , and nothing — absolutely nothing — that a LLM can do remotely justifies the investment. And it has only got this far because of a captured business and tech media overstating its capabilities and hand-waving its obvious efficacy issues and economic instability. There are too many that have proven easily-wooed by whimsical white boys that promise they’re building machine intelligence, and when the markets bleed red, these people should know that they’re responsible. So much of the so-called journalism around AI has been used to enrich the already-rich and inflate a bubble that will hurt hundreds of millions of regular people globally as Sam Altman and Dario Amodei remain billionaires despite their companies’ fates. When the time comes, the AI industry must burn. It must be allowed to die. Generative AI has already been given far too much money, oxygen and attention, and if it cannot survive without continual venture capital and media coddling, it is unworthy and unnecessary, and must face the cold, hard reality that every regular person faces when they fail. And there is no “bailing out” these wretched firms. Giving $42 billion to OpenAI or Anthropic will not fix their business models, nor will it magic up the $400 billion or more in annual revenue to substantiate just NVIDIA’s AI GPU sales through the end of 2027.   These people are not building the future — they’re finding ways to re-entrench the status quo, to give Microsoft, Google, Amazon and Meta ways to grow their revenues and centralize infrastructure under the auspices of “innovation.”  If any policy makers read this, know that you’ve been had by the AI industry. They want you to believe they’re essential so you’ll bail them and their rich friends out when the time comes, or funnel taxpayer funds into building them data centers. They are not building autonomous intelligence, nor will they ever do so.  I think it’s fanciful to imagine that there would ever be actual consequences for this bubble, but if there are, the people to hold responsible are Sam Altman, Dario Amodei, Satya Nadella, Sundar Pichai, Andy Jassy, Jensen Huang, Mark Zuckerberg, and everyone else who forcefully manufactured consent for a dead end technology and built the rails to serve the world its next great financial crisis. Until something changes, the tech industry will never be capable of building anything other than consensus and reinforcements of the status quo. So, spit in the face of those who even hint at a bailout, refuse to accept it, and demand that they do the complex, ugly work of thinking about the actual consequences of everyone being wrong. When this era ends, we will need to thoroughly excavate the collapse to make sure it doesn’t happen again, identifying the organizations and personalities that were used to manufacture consent and spread mythology about LLMs.  Every major bubble that has ever happened has mostly left the stones of responsibility unturned. The carnage that I fear will follow this era’s collapse will be horrifying, and we must do everything in our power to both thoroughly understand how we got here and make sure it doesn’t happen again, which will involve many hard conversations about our financial system, media ecosystem, and how innovation is invested in, built, bought and sold.  The same goes for the acolytes of this era. There are people who have developed a genuine hostility toward those who do not immediately accept a for-profit entity as their lord and savior. This is a sickness within the tech industry that must be put to an end.  Much of this will be unavoidable, because I think what follows the AI bubble will be a greater revaluation of the tech industry, a necessary reckoning with reality for a Silicon Valley that’s far more beholden to capital than it is human progress. The cults of personality that dominate this industry do not care about you, or me, or anyone other than those they revere and their theoretical placement in their dream of a society dominated by the rich and their chosen cronies. I refuse to accept their future as an inevitability. As I said a few weeks ago: This era must end, and all failures must be allowed to fail.  Let AI burn. If you liked this piece, you should subscribe to my premium newsletter. It’s $70 a year, or $7 a month, and in return you get a weekly newsletter that’s usually anywhere from 10,000 to 18,000 words, including vast, detailed analyses of the biggest events and companies in the AI bubble.  The stock market bubble, where both the value of stocks and the earnings of companies in the market are inflated to an historic level . A data center speculation bubble, where I believe we’re building AI GPU capacity in expectation of $450 billion or more in annual data center revenue for an industry that, without two unsustainable venture-backed oafs, has a few billion dollars’ worth of demand. An AI startup bubble, where the vast majority of AI startups are both over-valued and have no foreseeable path to acquisition or a public offering . These startups also rely on buying tokens from OpenAI and Anthropic, making them far more cash-intensive, making them absorb the majority of venture capital funding. A private credit bubble, where asset managers have sunk billions of dollars of pension and insurance funds into AI data centers .  A semiconductor bubble, where supply chains have become saturated with demand from those building AI data centers, inflating the cost of RAM and storage , making all electronics more expensive, including those inside the AI data centers, creating a vicious cycle that has doubled the cost of a gigawatt data center from $50 billion to $100 billion in a little under 10 months.

0 views
Schneems 3 months ago

Cinnamon Buns & Commit Bits: A RubyConf Story

RubyConf 2026 is coming up soon! The deadline to buy a ticket and enter the VIP raffle by buying a RubyConf ticket is July 1st. That’s only 2 days from this post’s publication. This personal essay explores why a language-specific conference in the age of AI is still important and why events are still important for bringing developers together. Food brings people together. At the first conference I ever went to, I accidentally had breakfast with Aaron Patterson, having no clue who he was. We were in Oklahoma, and a guy wearing a funny brown suit was in the conference breakfast area. The tech lead on my team, whom I was following, sat down and struck up a conversation with him. Later, when Aaron got on stage as a keynote, I was grateful I didn’t realize who he was, or I probably would have been a lot more awkward (being a bit star-struck). Later in that same conference, Aaron helped me land my first pull request in Rails in between rounds of Settlers of Catan (a board game) with the creator of RVM (a Ruby version manager/switcher tool). I’m not saying that sharing conversation over slices of cinnamon buns is what landed me the commit, but I’m not saying it hurt either. Since then, I’ve had countless meals at 40+ conferences in 17 countries. By far, my favorites are the Speaker’s dinners. These are invite-only dinners available only to the attendees speaking at the conference. These events are where I met famous Rubyists like Jim Weirich, creator of Rake, and Jeremy Evans (sequel, roda, and many more). I’ve been a “fly on a wall” while Ruby Core Committers like Matz, Koichi Sasada, and Nobu have deeply technical discussions. I’ve advocated for unblocking some of my pet Rails PRs to Rafael França and got an advanced heads up about PRISM parser changes by Kevin Newton. These conferences, and the dinners and outings that happen at them, have helped me build lasting friendships that are worth far more to me than I can describe. I met Evan Phoenix, creator of Puma, at a “birds of a feather” session, then got commit-bit not long after. Later, at the Arizona RailsConf, when I told him I liked the color of the staff T-shirts, he gave me the shirt off his back. Literally. I still have it. The speakers’ dinner is a place where I never know quite what I’ll find, but I always walk away full from conversation, food, and excited about the future. Excitement seems like a rare and precious commodity these days. A lot of companies are going through layoffs. The industry is being reshaped with AI leaderboards and mandates. Developers are finding they can be productive in languages they’ve never worked with, which is amazing. They’re also finding that arguing with a machine all day can be isolating and draining. In this age, what does it mean to be a Rubyist? What value does an in-person event have? Is it really worth it to pay to go to a conference to awkwardly stand in a circle around a speaker after a talk when Claude or Codex is ready and willing to give you 1:1 attention? These are questions I’ve been thinking about a lot. At a conference dinner, I once asked a Japanese Ruby developer what language they thought about code in, what language they dreamed in. The resulting discussion that popped up at the table from several native and non-native English speakers was fascinating. An interesting observation was that when people are tired, they prefer to speak in their native language, even if they’re highly proficient in another. Bringing this back to AI and code, I think that while I can ask my LLM-powered assistant to write in Go, or C++ or Node.js, my “native” programming language is Ruby, the one I learned in 2006 and got a full-time job with in 2010. I did a master’s course at Georgia Tech, largely in Python, but I still found Ruby to be like a comfortable couch where everything fits right. It’s where I preferred to be when the sun was starting to set, and my brain was starting to get slower. Ruby is my native programming tongue. The programming language I (day) dream in. I think it’s important that in this age of AI-powered generalists, programmers go out of their way to go deep in some areas. While having mastery over Ruby might not be required to make a PR to a project, I think it’s important to work towards mastery in SOME programming language, and Ruby fits my brain (I like Rust too…but that’s a different post for a different conference). I believe being a programmer has changed, but there’s still value in going deep. There’s still value in being a Rubyist. When I was younger, going to these conferences gave me knowledge I couldn’t otherwise access. Either you were in the room when Koichi Sasada explained how multi-colored write-barrier powered generational garbage collection, or you weren’t. I didn’t understand all the words all the time, but sometimes being confused and exposed to new things is the first step to learning. As I’ve gotten older, I still value the technical details, but the thing I value more is the energy and excitement I get from these in-person events. The joy of listening to how someone learned something new and then being able to share that new thing with you is infectious. The connections and conversations that pop up in the elevator or at 2 am at Karaoke are unplannable and invaluable. LLMs are always-on and always-there. But these events are special because they’re not. It takes a lot of effort to go and talk to a stranger, even if you love their work and want to learn more about it. Having the event be a limited-time engagement means that I have an excuse to have those conversations with people like Jessica Kerr and Charles Nutter. Could I message these people on Mastodon or Blue Sky at any hour of the day? Sure, but being there, being in-person, seeing their face, hearing the inflections in their voices, having that shared experience with hundreds of others in the room. It’s, well…an event. And I believe it’s still worth something. It’s worth investing in Ruby, it’s worth investing in events, and it’s worth investing in me by getting out of my comfort zone and into RubyConf. I still think being a Rubyist matters and going to events like RubyConf matters. That’s why I am going. I’ll be speaking on a panel of maintainers. I’m excited to go to the speaker’s dinner. I’m also excited that for the first time ever, we are raffling off the opportunity for five lucky VIP Rubyists to come along. If you buy a ticket to RubyConf 2026 before July 1st , you’ll get an exclusive pin and be entered to win a ticket to the speakers’ dinner and VIP seating for the keynotes. Read more about it on the VIP raffle announcement or buy your RubyConf ticket today . If you win, I can’t guarantee you’ll make new, forever-friends or have a life-changing night. I can guarantee that if you enter or you don’t go, you’ll never know. Win or lose, I hope to see you at the conference.

0 views