Stratechery Today

Salesforce AI Force, Agents as UI, The Race to Headless

Salesforce is abandoning UI as a moat, which is a very smart move because it's disappearing for everyone.

AI Business Frontend

▲ 0

0 views

Sean Goedecke Today

Jev means structured output is interesting again

I don’t write blog posts about new models. That’s Simon Willison’s beat, and he’s very good at it. But I want to write about Jev , which is a different kind 1 of AI model: a “System One” 2 model. As it turns out, it’s not that different from an ordinary LLM with structured output, but the interface it uses is very cool and I hope it becomes more widespread. Ordinary LLMs take in some human-language prompt and produce some human-language output. They do so autoregressively : first they produce one token, then the next, then the next, and so on. This makes them extremely flexible, since they can do literally anything a computer can do. But it also makes them slow and weird. Slow, because they have to run a whole new generation pass per-token, and weird, because the space of human language is so broad that you can get really odd behavior from a model trained on it. Jev takes a human-language prompt, but it does not produce human-language output. It only produces structured output. So far, so ordinary: LLMs do this already . But it turns out that if you build a model that only produces structured output, you get some interesting and desirable properties. Jev is always really fast. The fastest response time is around 70ms instead of a couple of seconds for normal LLMs. Even better, the slowest response time is only 500ms. Because Jev only does structured output, it isn’t autoregressive: it can produce answers to many questions in parallel in a single forward pass. When a LLM is producing structured output, it has to produce the tokens ”{”, ” ”, “answer”, ”:”, and so on with successive forward passes 3 . Jev does it all in one go. The most compelling example of Jev’s speed is that the model can play Doom . You can feed a text-based representation of the current game state into the model, combined with a set of choices like “should the trigger be held down”, “what should the current goal be”, “given that the current goal is X, what keyboard input should be pressed”, and so on, and it works — latency is low enough and the system is smart enough that the model plays well in real time. Of course you could train a neural net to play Doom already. But Jev is a general intelligence: just like LLMs can do your taxes, perform mathematics research, fix your Python environment, and write you a poem, Jev can do many other tasks besides playing a single video game. Current LLMs can play Doom too (albeit slowly). But as Nelson Elhage famously said , fast software doesn’t just mean we can do the same tasks faster, it means we can do entirely new kinds of tasks. What kinds of new programs can we write by injecting 100ms worth of dirt-cheap intelligence at various decision points? To me, this is the most exciting thing about Jev. Fast structured output could be a genuinely new computational primitive for intelligence. So far we’ve built a lot of programs on top of autoregressive token generation, and they all look like fancy chatbots. Leaning hard into structured output might conceivably unlock a bunch of non-chatbot use cases for AI. My biggest problem with Jev is that I think fast structured output is already available . Structured output from LLMs is only slow because (a) nobody really cares about it 4 , and (b) the people who do care about it want big JSON blobs, so it’s typically implemented with “grammar-constrained decoding” : the LLM outputs autoregressively as normal, but the logit sampler discards tokens that don’t fit the structured output (e.g. if there hasn’t been a ”[”, you can’t output a ”]”). If you want fast, parallelized structured output against limited choices, you don’t strictly need to do autoregressive generation at all. You can simply prefill the response with and generate one token 5 , restricted to the user-provided choices. Since LLMs ingest all input tokens in parallel, this is way faster than generating the entire structured output. Multiple choices can be batched into the same forward pass via ordinary inference batching. This doesn’t let you do long-form structured output, but in return you get most of 6 Jev’s “secret sauce”: the speed, the consistency, and the parallelism of a System One model. People have already started trying this after today’s Jev announcement, and it seems like it’s working OK 7 . In other words, I suspect Jev does not have a substantial technical moat, and their claimed “Reinforcement Learning for Calibrated Decisions” is not a brand-new scaling axis. It will probably be pretty easy for any other lab to replicate, or for individual programmers to retrofit existing open-source LLMs into a fast Jev-like model. However, I suspect Jev is still going to be better than most versions of “Qwen-32B-System-One” or whatever. Being able to fine-tune or optimize the model on just structured output is probably a meaningful advantage. I doubt Jev is ever going to be as smart as frontier LLMs. Not being able to use test-time compute at all 8 is a big disadvantage, and will likely cap this kind of model around the strength of non-reasoning LLMs. In practice this shouldn’t matter too much for low-latency applications, but you shouldn’t see this as a new scaling axis or a way to produce more intelligent models. Jev’s developers claim it is immune from hallucinations. To me, this seems like a semantic dodge, since Jev can absolutely still pick the wrong choice (e.g. calling the sky “red”). I suppose that’s technically just a mistake , since the model is picking a user-provided choice instead of inventing something new out of whole cloth. Still, all of this is also true about regular LLMs with structured outputs, and it doesn’t make Jev any more reliable in practice. It’s unclear to me how much of Jev’s value is in the model itself, compared to the inference strategy of only generating one token per question. The data and demos in the announcement look to me like they could have been generated by plugging any Terra-sized model into a single-token inference stack. However, the people involved are credible, and I’m sure the model is good — I just wish they’d provided some comparisons that didn’t force the LLM to unnecessarily produce a blob of JSON token-by-token. Overall, I am happy that Jev exists and I hope it succeeds. I hope we do see some real competition in the fast-structured-output space, and that it motivates the big labs to release official versions of their own models that are fine-tuned for this. GPT-5.6-Terra-System-One would be a very interesting model to build AI products on top of. I did write about Thinking Machines’ “interaction models” , which are also a fast-enough-to-be-meaningfully-different paradigm for AI inference. They call Jev a “System One” LLM, after Daniel Kahneman’s partially discredited Thinking Fast and Slow , where he divides human cognition into a lightning-fast System One and a slow-and-reflective System Two. If you’re thinking “wait, couldn’t you just aggressively prefill a regular LLM and only produce one constrained token”, keep reading. Not counting tool calls, which are built-in in a way that structured output isn’t. What if some of the user’s choices are longer than a single token? I haven’t tried this myself, but I’m sure you could translate them into a single token, or train the model to output “1/2/3” under the hood instead of the choice content, or generate only the first token of the choice if it’s different, or some other clever trick I haven’t thought of. Jev claims that their generated probabilities are “calibrated”, but I haven’t seen anything to suggest that these aren’t just regular logit probabilities. Maybe there’s some clever training they do to encourage accurate logprobs in uncertain situations (e.g. getting the model to produce when predicting a coinflip, etc)? If so, I wish they’d written more about that in the announcement. I tried it myself with and got a 2x-3x speedup compared to non-prefixed structured output. I suppose they could do some looped-transformer thing where they loop some fixed amount of times, but anything that looks like reasoning would make the model latency slow and unpredictable, defeating the entire purpose. I did write about Thinking Machines’ “interaction models” , which are also a fast-enough-to-be-meaningfully-different paradigm for AI inference. ↩ They call Jev a “System One” LLM, after Daniel Kahneman’s partially discredited Thinking Fast and Slow , where he divides human cognition into a lightning-fast System One and a slow-and-reflective System Two. ↩ If you’re thinking “wait, couldn’t you just aggressively prefill a regular LLM and only produce one constrained token”, keep reading. ↩ Not counting tool calls, which are built-in in a way that structured output isn’t. ↩ What if some of the user’s choices are longer than a single token? I haven’t tried this myself, but I’m sure you could translate them into a single token, or train the model to output “1/2/3” under the hood instead of the choice content, or generate only the first token of the choice if it’s different, or some other clever trick I haven’t thought of. ↩ Jev claims that their generated probabilities are “calibrated”, but I haven’t seen anything to suggest that these aren’t just regular logit probabilities. Maybe there’s some clever training they do to encourage accurate logprobs in uncertain situations (e.g. getting the model to produce when predicting a coinflip, etc)? If so, I wish they’d written more about that in the announcement. ↩ I tried it myself with and got a 2x-3x speedup compared to non-prefixed structured output. ↩ I suppose they could do some looped-transformer thing where they loop some fixed amount of times, but anything that looks like reasoning would make the model latency slow and unpredictable, defeating the entire purpose. ↩

Python Machine Learning JSON AI Deep Learning

▲ 0

0 views

Unsung Yesterday

“But, as we all know, the individual light bulbs are not moving.”

I linked to palette cycling before , and I was just reminded of palette cycling art by Mark Ferrari, who back in the 1990s made 30+ landscapes that looked like this: They have been collected on this webpage some 15 years ago, and I’m linking to it in part because it’s also a great explainer of how palette cycling works – you can see the colors move around, you can point to one to see it frozen, and you can see multiple cycles running in parallel, compare palette ranges between different environmental conditions, and turn on a “blended” technique that feels clever and I didn’t realize existed. The page was made by Joe Huckaby, who wrote a little intro: Mark J. Ferrari […] invented his own unique ways of using color cycling for envrironmental effects that you really have to see to believe. These include rain, snow, ocean waves, moving fog, clouds, smoke, waterfalls, streams, lakes, and more. And all these effects are achieved without any layers or alpha channels – just one single flat image with one 256 color palette. The launch was also accompanied by an interview with Ferrari, which is an interesting read – in part because it shows the work was even more elaborate than all of the above: These versions of the scene are all the same piece of art ‘shifted’ to different palettes, and, in some cases, using additional ‘baked in’ overlays, (such as rain or the lighted windows at night). But those overlays are all ‘baked in’ to the same layer of the same piece of art that appears in any other ‘day-time’ or clear weather iterations, and are all deriving their color and motion from the same palette as the rest of the picture in that state. While [the page above] finally allows us all to watch these images color cycle online, many of the scenes posted were actually ‘built’ to do much more than merely animate. By fading the one piece of art through whole sets of palettes, sometimes also using a very sparse set of ‘baked in’ overlays, a number of these scenes can go seamlessly through the 24 hour light cycle, and even change weather conditions ‘naturally’ and seamlessly in real time as you watch. I am not just talking about changing the brightness or color scheme of these pictures either. In the images built for it, over the course of ‘sunrise and morning,’ ‘morning to afternoon’ or ‘evening and sunset,’ light and shadow will actually gradually change angle, climb down the sides of things, move across lawns, up cliffs or building walls, as changing light does in life – all just by fading through palette series designed to make those things happen without altering or adding anything at all to the single layer of 8-bit pixel art. Ferrari also suggests an interesting analog to palette cycling, which I quoted in the title. (Bonus: Ferrari’s animated landscapes were made for a new-age’y personal organizer app called Seize The Day, and on top of the above preservation effort, there is also this independent, extremely retro page from a fan of the app who loved it so much she decided to keep the app itself alive, too.)

Gaming AI Design

▲ 0

0 views

Stratechery Yesterday

OpenAI Ads, Amazon Ads in ChatGPT, Walmart to Accept Apple Pay

ChatGPT ads are working, and solve Amazon's biggest problem with chatbots. Then, Walmart finally gives in to Apple Pay, because fighting the status quo is hard.

Business Programming AI

▲ 0

0 views

Hugo Yesterday

A Slowdown in AI Development?

Coup de théâtre, several AI actors are calling for a slowdown in the development of frontier models and the implementation of regulation. AI progress would be too rapid and could pose serious problems in the future. But could this be hiding something? Could it be yet another marketing move to continue fueling the hype in the sector? Or could it be a kind of desperate attempt to lock down the market and escape a complicated financial situation? It all started with Dario Amodei's letter published a few days ago: "we must pace the frontier". (Dario Amodei is the current CEO of Anthropic which publishes Claude) In this letter, Dario calls for regulating/slowing down/securing AI development. He highlights recent incidents around the Hugging Face cyber attack and the escalation linked to the acceleration of development with models that self-improve. He therefore proposes several measures: Following this, Sam Altman (OpenAI) and Elon Musk (xAI) both validated the request on social networks, which in itself is already a huge surprise, as the three aren't exactly the type to spend vacations together. But we should probably read between the lines. A quick reminder of the context: OpenAI and Anthropic are planning an IPO soon. Both companies are far from profitable and spend billions on model training or inference costs. To win or maintain market share, the two giants are cutting prices and subsidizing token costs at a loss. But with massive and constant investments and rising competition, especially from Chinese models, the business model seems very shaky and could well cool down stock market investors. We're witnessing a real arms race in an industry that's largely overheated, where the first one to slow down loses. But above all, the level of investments already made makes it impossible to slow down. It would be complicated for investors to accept that model performance suddenly stagnates. Investment plans include datacenter construction, electronic component purchases, electrical capacity, etc. Announcing a slowdown today would be a big blow for many players, but also a negative signal to send before an IPO. Except that accelerating to crash into a wall with a business model that doesn't hold up is not an attractive scenario either. So perhaps this letter would be a way to create an exit. Dario's letter is part of a marketing strategy that has already proven its worth. Remember the precedents: I could make a very long list and even go back to Musk's first statement in 2014 . It's quite clear that some people on this list can be sincere about these statements, but I find it hard not to see a certain form of marketing strategy. Highlighting that a technology is an existential risk to humanity, especially if it falls into the wrong hands or if it's designed without safeguards, allows two things: Dario's letter, which many experts have debated for 1 week, is nothing more than an extension of this marketing. But we're starting to see some novelties. In the text, the CEO of Anthropic explicitly targets the prohibition of actors doing distillation: Crack down on unauthorized ++ distillation ++ by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently. We also find: So in your opinion, who would be penalized by AI regulation as requested by Amodei? In short, it would be a tough blow for open source, Europe and its sovereignty, and companies, resulting in the creation of an oligopoly capable of fixing prices much more easily. One might think that with such strong benefits, the entire American industry would be behind this project. Well, surprisingly, not so much. While we can blame Amodei, Altman and Musk for hiding their ambitions to create an oligopoly and lock down the market, the fact remains that the stated objective is to regulate AI risks, by imposing global regulation, certainly, but wrapped in nice gift paper. But this is not at all the approach of the " accelerationists ", notably represented by Peter Thiel and Alex Karp (Palantir) who instead want total market deregulation and whose absolute priority is to beat China. Unsurprisingly, we'll find this same discourse with Trump (we can remind that JD Vance was introduced to Trump by Thiel), who gave us declarations as outlandish as usual and whose content I'll let you appreciate: The only control or “guardrails” that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades! The Trump Administration has stopped AI “people” from doing bad, or potentially bad, “things,“ like Dario (Anthropic!), who is now pretending to be a “perfect little angel” - and we will continue to do so! We already have tremendous CRIMINAL and REGULATORY power over these companies! There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS! We are leading China, and all others, and will continue to do so. Conspiracy Theorists, Treasonists, Traitors, and Leakers, BEWARE! Thank you for your attention to this matter! President DONALD J. TRUMP Good thing he's there, the show is always guaranteed… But we also find this opposition in Jensen Huang (CEO Nvidia) who, let's remember, just bought Hugging Face for 13 billion dollars and just invested in Mistral. The company sells chips and computing capacity to everyone and sees open source as an opportunity, at least business-wise anyway, so the slowdown requested by Amodei is far from his priority. The simple fact that Trump is opposed to the slowdown project buries this project at least for the duration of his term in the US. It's hard to imagine a slowdown by 2028, at least not for these reasons. And even in the future, it seems uncertain to me to imagine that the US would accept seeing China overtake them without reacting. On the European side, the discussion risks having lasting repercussions instead. It gives grist for the mill for EU regulators who would be happy to implement more stringent standards, and for some governments who would like to take control. Not to mention some politicians who are a bit lost when it comes to the issues and could play against their camp without even understanding it. These hesitations risk putting us (in Europe) in a bad position if we were to add barriers to the open source world, to open weight models or our local champions. In short, this call for slowdown seems to me mainly a maneuver to ensure some stability in a market that has gotten out of hand. If the authors were really sincere about the concerns of model alignment, if they were worried about AI escape risks, they already have the means to work on the problem. The real existential risk today is more about their future IPOs, rising competition and a European market that could choose another path with open weight models. It seems unlikely in any case that China would subscribe to this call, nor would the Trump administration, and I hope Europe won't give in to the temptation to strengthen legislation at the cost of our future sovereignty. the establishment of audit and control bodies the implementation of security and development standards for models international coordination to regulate at a global scale In 2023, an open letter was already published to slow down, already signed by Elon Musk Also in 2023, the statement on extinction risk signed by Sam Altman and Dario Amodei In 2024, statements from OpenAI researchers on safety and governance risks In 2026, statements from Jacob Coxon (Anthropic) to highlight that we're working on an extremely powerful technology that deserves investment or purchase to call for regulating new players by locking down the market a call to limit the sale of the most powerful chips to China (but more broadly to non-US competitors) various hints in the text aimed at slowing down/blocking open weight models (reinforcement of audit methods, incompatibility with the notion of certification checkpoints) Chinese actors who massively use distillation to offer competing models at a fraction of the price Open source and research that rely on open weight models and which anyway won't have the means to implement audit mechanisms, not to mention that certification bodies won't necessarily be so independent Mistral, which could no longer exploit open weight models in these infrastructures and would also have to implement certification mechanisms that are probably very costly and controlled by the US New entrants for whom the entry ticket will be too high Hosting providers that sell computing power on open weight models

AI

▲ 0

0 views

マリウス 2 days ago

llama.cpp with SYCL (oneAPI) for Intel Panther Lake on Gentoo

Alright, so if you’ve read the title and thought you were having a stroke, you might not be the target audience for this post. If, however, you thought “more tokens/s?” , you should keep on reading. If you remember my review of the new Lenovo X1 Carbon Gen 14 Aura with Intel Panther Lake Core Ultra X7 368H vPro from a while ago, you might recall that I had tested its local “AI” performance using Ollama, via Vulkan, which obviously didn’t perform particularly well across various models. The proper way to run local LLMs is to use Intel’s official oneAPI framework and compile llama.cpp with SYCL enabled, which will greatly increase performance. This post is a very brief write-up of how that can be done and primarily serves as yet another documentation for future-me. I’m assuming that you’re already running your Gentoo system with the driver and . First, install the necessary dependencies: You’ll likely have to unmask a couple of those packages, like , but that’s fine. When this is done, add your user to the and groups if you haven’t already and log back in. Test that the GPU is being recognized by : Next, go and get the “Intel Deep Learning Essentials” package and use to install it as a user. I chose the installation path , but you’re free to install it anywhere you please. Note: Even if you deselect the telemetry option, the installer will try to contact Intel’s servers post-installation, so make sure your OpenSnitch blocks all requests from that process towards the interwebs. While the FireBurn overlay has , it does not yet allow you to specify as a USE flag, and the official Gentoo repository has no llama.cpp ebuild at all. Hence we clone the project’s Git repository and compile it manually: Once llama.cpp finishes building, you can start it and have it download a model, e.g., Mistral or Qwen3: Open a browser at http://127.0.0.1:8080 , and you can try the model right away. On my Lenovo, I managed to increase the tokens/second by roughly 60% compared to what Ollama (via Vulkan) was able to achieve. For example, the Mistral model went from approximately 13.88 tokens/s to 22.36 tokens/s.

C++ AI Performance Programming

▲ 0

0 views

Ed Zitron's Where's Your Ed At 2 days ago

AI Is Already In Dangerous Hands

If you liked this piece, you should subscribe to my premium newsletter, and you can subscribe on the following links: $70 a year , $18 a quarter , or $7 a month . In return you get a weekly newsletter that’s usually anywhere from 10,000 to 18,000 words, including vast, detailed analyses of NVIDIA , Anthropic and OpenAI’s finances , and the AI bubble writ large .  On Friday, I’ll publish The Hater’s Guide to AI Debt — or, how buzz surrounding OpenAI and Anthropic have created massive concentration risk for world debt markets, and one which you’ll potentially be paying for, either through your pension funds and insurance premiums, or because you’ll have to live and work through the economic downturn that’s coming. For a taste of what’s to come, consider reading my Hater's Guides To the SaaSpocalypse , Private Credit and Private Equity .   If you want to get in touch — and especially if you have any juicy information about Anthropic, OpenAI, or any other companies in the AI bubble — hit me up on Signal at ezitron.76. I’m also on IB on your Bloomberg Terminal.  Late last week, everything exploded when former Anthropic AI researcher Jacob Coxon, in an exclusive interview with the Wall Street Journal , warned that he was “quitting the AI industry” (he wasn’t) over “...fears that the lab and its competitors are racing to build systems they won’t be able to control.”  His fears were centered around the creation of “recursive self-improvement,” a still-theoretical concept of AI that trains itself autonomously” and otherwise expressing few specific concerns beyond that “AI labs are unable to control AI,” always phrasing things in the terms of impossible-to-control entities rather than poorly-programmed cloud software running on the infrastructure of the largest companies in the world.  Emily Forlini of Fortune put it best : This is because, in my opinion, Jacob does not really care about the actual harms of AI, whether we’re talking about Large Language Models or something he imagined while working with the non-profit or PR firm that set up a CBS interview where he claimed that AI that, if we’re talking about LLMs, have model weights of terabytes of memory, would make ten thousand copies of themselves .  Or, of course, bullshit like this: At no point did Coxon bring up how ChatGPT was used as a “suicide coach ,” directly caused a murder-suicide , or aided and abetted in mass shootings in Florida and Canada , or the horrifying gas turbines poisoning black communities . His own discussion of the Hugging Face attack — much like all of his criticisms — focuses on the anthropomorphization of large language models as this unknowable, unstoppable force, with no real responsibility for anyone involved. This is a repetition of what we saw back in February when Matt Shumer’s abominable “Something Big Is Happening” essay spread like wildfire to every imaginable news outlet despite it being somewhere between nonsensical and utterly fictional, except this time the narrative got a little out of hand . While Coxon’s warnings are specious and, in many cases, not really about anything other than him saying “yeah I heard a lot of my colleagues say stuff like that,” but nevertheless have had the effect of making a lot of people really scared of AI , even if AI can’t really do the things he’s talking about. The one tangible thing he talks about is the Hugging Face attack, which is being described in terms of AI having “plans” or “acting on its own accord,” and even then, when pushed by WIRED, his response was he “[didn’t] want to focus too much on the Hugging Face attack.”  So, let’s talk about what happened there, because it’s important! To answer this question, I turn to frequent Better Offline guest Cal Newport’s piece on the subject from July : First of all, what was OpenAI trying to do?  And what happened? I’m going to quote Cal liberally here, because he’s explained it well: this was a series of Large Language Models connected to a software program built to prompt them to complete an evaluation framework specifically to do cybersecurity attacks (along with near-infinite amounts of compute) “solving” the problem using any and all methods available, including hacking Hugging Face In other words, the LLMs — albeit through convoluted and aggressive means — “solved” the problem they were tasked with. As Cal said, there was no situation where the AI “went rogue.”  They took some weird ways of getting there for sure ( like using a message board to communicate messages between LLMs ) — and did exactly what they were supposed to do, even if it meant taking ridiculous routes to cover up that they’d cheated on a test. You’ll notice that Jacob Coxon, who ostensibly would know this as a researcher (but perhaps he doesn’t!), chose instead to describe the hack to WIRED like so: Beautiful linguistics, champ!  They were not “trying to understand the world they were in,” they took actions defined in their training material as a way of executing a task.   They were doing exactly what it was that the ExploitGym test required! The “all sorts of ideas” were a function of being allowed to use as much compute as possible to execute the task.  What’s particularly telling, as I’ve hinted at, was Coxon’s response when it was (lightly) suggested that the labs need to take responsibility: Jacob is intentionally trying to frame Large Language Models — which are kind of a black box, but a black box made up of maths — as this unknowable autonomous, mischievous being that the AI labs have conjured out of the ether. “We can’t make sure it won’t do things like…” frames the labs as helpless stewards rather than the creators of a kind of neural network run on massive amounts of big tech’s infrastructure. Every statement Coxon makes that’s allegedly about “safety” or “protecting people” does everything it can to distance the AI labs from any responsibility or even active participation in any of this beyond some fatalistic level of “well, somebody’s gonna do this, why not us?” And I also want to be clear about something: The Hugging Face attack was dangerous, reckless and somebody should go to prison for it. If a regular person used massive amounts of compute capacity to hack something, they’d be arrested. While I’m not a lawyer, the numerous cybersecurity experts I’ve discussed this with are stunned by the complete lack of any legal action against OpenAI, which appears to have committed a crime that gets you anywhere from a year to a decade in the slammer . The fact that LLMs from both Anthropic and Meta have been involved in similar incidents is a sign that we need to arrest more people. LLMs do not have to be conscious or powerful AI to be incredibly dangerous. The fact that Anthropic, OpenAI, and Meta are both training and allowing cybersecurity models to connect to their vast amounts of GPU infrastructure is irresponsible and should not be legal. The reason they are training these models, as I got into in my podcast Better Offline with Cal Newport , is that there’s a mountain of potential different kinds of exploit and vulnerability data online that you can cram into these models now that they’re hitting the diminishing returns on coding.  Yet flowery linguistics from people like Jacob Coxon and the greater AI industry have muddied the waters of what’s actually going on and who is truly responsible. If AI is described in terms of the unknown and being uncontrollable, the “risk” gets turned on its head from “we need to stop these companies from doing this” to “we must let these companies keep doing this because they’re the only ones who understand it.” The AI industry wants to frame this as if Anthropic, OpenAI, and Meta discovered some new lifeform rather than having run a volatile kind of machine learning evaluation with poor cybersecurity practices. If the industry is the one saying that we should “be so scared of powerful AI,” it means that nobody is responsible for what it does — not even the people making it do it. LLMs are cloud software. Framing them as anything else only seeks to mystify them and make the companies seem more powerful, all while doing absolutely nothing to make the world safer or more secure. The Hugging Face attack is not something that was made possible as a result of “powerful AI” so much as it was the weaponization of hundreds of billions of dollars’ worth of GPU-powered infrastructure owned by Microsoft, Google, Amazon, Oracle, and CoreWeave.  This was not an LLM that was asked to generate a picture of “increasingly sexier Garfields” that decided instead to hack Hugging Face. This was not an “agent” that “went rogue.” It was software doing what software was asked to do, using other bits of software to work out what to do next, all as OpenAI, the company that ran the software, did not appear to have any kind of notification or observability that said “hey man, thousands of LLMs are doing something right now.” This suggests the following: In any case, whatever happened with Jacob Coxon struck a nerve in a media ecosystem where it appears many people do not have object permanence. This is a short note, but an important one: the terminology of “pacing the frontier” or “slowing down” implicitly buys into the narrative that the AI industry’s path is the correct one, and that the only problem is the speed it’s moving at. The way that LLMs have been trained is harmful in effectively every way. It is trained on theft, powered by expensive and power-intensive infrastructure and is both unprofitable and unsustainable. LLMs are not the tool for any kind of beautiful, automated future — they are inefficient, volatile and mathematically certain to make mistakes . “Slowing down” is not sufficient. In my opinion, there is no further reason to invest in this industry, nor has there been for the vast majority of its existence.  Every success that LLMs have had is a direct result of throwing at least half a trillion dollars in infrastructure and compute spend at problems that had vast amounts of data that could be trained against. Half a trillion dollars should have bought us a lot more than this. While I will not dispute that they can do more than a year ago, I am unimpressed, because this is more than ten times what Amazon’s entire capex between 2003 and 2015, the years between AWS’ creation and when it hit profitability.  This is a terrible deal, its results suck, and the amount of attention it’s gotten is a direct result of the media’s inability to speak truth to power or do anything other than repeat what they say and a financial bubble driven by LLMs’ unbelievable infrastructural cost. If — and this is not a foregone conclusion — there is ever an AI that we, as a society, should fund and build infrastructure for in pursuit of some civic good, it is not the one peddled by Elon Musk, Sam Altman or Dario Amodei.  This is not the right path, and every further step down it makes the bubble’s collapse worse, as well as multiplying the dangerous and reckless experiments these companies are capable of doing thanks to their near-unlimited access to compute.  The entire Jacob Coxon thing is very, very strange. He had never tweeted before his post that now has over 170 million views on Twitter and interviews with the WSJ , CNN , NBC , CBS , and a bunch of other outlets that should’ve known better. While he had only been at Anthropic a few months ( and lost stock options when he resigned ), he had been at OpenAI for years and absolutely had options vested from there. Retweets of his post were clearly coordinated with various AI safety organizations, and the speed at which it took off with the media makes all of this look incredibly contrived, as does Coxon’s total lack of any direct critiques or “blown whistles” about the AI labs themselves, other than that they “can do more safety” and “should coordinate a global slowdown.” In the end, it doesn’t really matter, because even though most of the media mostly jumped at their own shadow, the sheer volume of traffic to Coxon’s tweet and his endless media interviews have now moved the idea of the need for a “global AI slowdown” into the global zeitgeist. Turns out that using scare tactics and threatening everyone’s jobs on and off for three years has a consequence.  While it’s tempting to view this entirely as an opp — a coordinated industry-wide plan to push for some sort of self-regulation — in my mind it’s likely an attempt by the AI safety people to push an agenda that has spiralled completely out of control. Within a day of Coxon’s post, Clammy Sam Altman spoke with Fortune saying that OpenAI was delaying going public until 2027, as it was an “ill-advised moment” due to “safety concerns” rather than, I imagine, the fact that its financials are godawful .  He later posted two (two!) lengthy posts on Twitter, where he said that while OpenAI “[welcomes] a federal framework that sets consistent safety requirements for frontier AI, ” it doesn’t believe the industry should wait, although failed to suggest potential solutions other than mentioning it was “excited by ideas like independent auditors."   This was followed up with the same usual scaremongering guff where he said that there two ways “AI progress could go very badly,” with the first being that “we could lose control of the future to AI,” something he did not elaborate upon, with the second being that AI could result in power becoming too densely concentrated with one company. I had to resist the urge to fall asleep while writing this paragraph.   Dario Amodei of Anthropic took to CBS to say that for “too long” the industry had “lied” about risks, saying that the “biggest one” was killing all humans, never mentioning when an LLM convinced a teenager to kill himself or its own hacking incidents because “AI safety” never relates to the things they’re building today. The most-obvious version is in Amodei’s own “ pace the frontier ” blog: The “third party evaluator” he chooses is METR, the very same place that Joe Benton, an Anthropic researcher who quit two weeks ago , chose to move to after being convinced that AI companies are “underinvesting in safety.” You’ll also notice that Benton’s safety suggestions are self-serving: Nothing about the environmental impact, the theft of millions of people’s creative works , nothing about AI psychosis, just a bunch of stuff about how we need progress toward a still-theoretical idea that sounds really good if you’re trying to hype up a company. Not long after Amodei discussed slowing things down, it broke that Anthropic had chosen NASDAQ for its IPO , shortly before the FT reported that Anthropic would “ have a profitable third quarter ” if — I shit you not — you ignore costs like training and stock-based compensation.  What the fuck is a slowdown if it involves an IPO, the purpose of which (besides allowing insiders to cash out their holdings) is usually to help the company going public raise capital from the public markets? What the fuck is a slowdown if you’re leaking (assuming Anthropic was behind it) you’re “profitable” in the least-GAAP way possible? God, I’m tired of this industry. There are, of course, real, meaningful things you could do if you actually were worried about LLMs — halting all model training, all cybersecurity evaluations, and starting a criminal inquiry into the Hugging Face attack that ends in somebody going to jail for the crime they used the models to commit. If it turns out multiple people are legally liable, tough fucking shit , you are going to jail for a crime, you are not special because you used agents to do it. To be clear, I am extremely hesitant to believe anybody is “slowing down.” Anthropic’s own statements mostly amount to “we should all agree to not do something we’re not doing yet,” much like those made by Musk and Altman . Yet the sickly irony of all of this safety theater — and that’s all it is without any actual tangible attempts to deal with the harms of the technology that actually exists — is that a boneheaded media incapable of catching out grifters has accidentally destabilized an already-tenuous narrative. Put another way, I think everybody has their own agenda, nobody has a plan, and that everything is accelerating like the end of a Coen Brothers movie as every little narrative thread gets tangled together in a potentially jumbled and chaotic conclusion. Back in early 2023 , a young(er) Sam Altman said that OpenAI was “a little bit scared” of AI, adding that we should “guard against potentially negative consequences for humanity,” adding that they “could be used for offensive cyber-attacks.” In the end he was right, but only because he made sure that was the case.  For years the AI industry has engaged in endless, vague safety theater about the “risks” of AI, all while peddling software that is actively harmful and unreliable. The “success” of the Hugging Face attack was largely a result of the sheer scale of OpenAI’s compute operation, and would not have been possible without Microsoft, Google, Amazon, Oracle, and CoreWeave’s continued enabling of an unprofitable, unsustainable company that is desperate for new business models. Yet I must be clear that the Hugging Face attack happened two months ago. Everybody doing backflips out of fear about “powerful, autonomous AI breaking out of the sandbox” is mostly doing so because a British guy went on TV and said “AI will kill us all,” and while I can’t resent a pale British man getting broadcast opportunities, I take exception with those who are so densely packed with bullshit. Nevertheless , Coxon struck a match next to a giant pile of dynamite laid by years of pantomime about “AI risks” that didn’t actually apply to the things that the labs were building.  The dueling brain cells of VanderHei and Allen at Axios declaring that we were going to face a “ white collar bloodbath ” were not based on anything LLMs can do, nor have any of the bullshit stories around so-called white collar job loss , nor was the early GPT-4 scare hype around “LLMs blackmailing people,” nor were the numerous stories about AI 2027 , but they were demonstrations of how ready the media was to lose their entire shit over a narrative that the AI industry was deliberately encouraging: that AI was powerful, unknowable and uncontrollable. Everything was always about selling today’s tools based on what might happen and occasionally scaring people about what that meant without ever really attaching it to the stuff they were doing today. While there may be some people that had honourable or sincere beliefs that AI was or is potentially dangerous, rarely if ever did these stories actually discuss these harms, which meant that nobody ever really did “AI safety” in any meaningful way. While alignment — as in making sure the models were trained to act in a predictable way that created good outcomes — is a noble and necessary goal for training large language models, at no point has any “slowdown” or “pause” been suggested based on the grounds of what these things actually do. This rocked for the companies for a while, because it meant that they could vaguely say “wow, AI is going to be so powerful” every so often and every member of the media would crap their pants and give them a headline. Every story was about how “today’s breakthroughs proved that tomorrow’s AI would be even more powerful ,” which they loved because, well, it meant their companies would be even more valuable as a result, even if raising that valuation required intimidating people about the prospect of them losing their jobs, even if the software itself didn’t really do what they were promising (which didn’t matter to basically any journalist covering this field). Their “powerful AI” — graded not based on actual outcomes but preferential anecdotes and performance on benchmarks rigged for the LLMs — was always “on the frontier” and “getting smarter every day,” all because AI labs and hyperscalers had intentionally sold their products based on some theoretical future version that would fix all the problems.  In other words, whatever they did was seen through the best light, described in the terms of the best parts of the present and the best promises of the future, and given credit as if it had already happened.  This is, as they’ve found, a double-edged sword. When journalists will believe (and print) whatever you say, they’ll start believing that the real thing (LLMs) does the same thing as the imaginary future thing (AGI, ASI, golden egg-laying geese ), or will do so, even if that thing is bad .  These companies had spent years puffing up their LLMs’ potential using vague promises of superintelligence and theoretical model capabilities, at times inflating its capabilities further through scary quotes ( here’s a list of Altman’s! ), intentionally training models to blackmail people and entirely-fictional stories about “breaking containment,” and never realized that at any time one of the near-cultist types that joined their companies and heard everybody talking in terms of “ p(doom) ” (fuck off) could take it all seriously and the media might believe them. This puts the industry in an odd position.  While on one hand, Altman, Amodei, Musk, and the rest of them know that they can’t roll back the narrative and say “everyone, stop freaking out, it’s fine, it’s just cloud software,” they also know that they have to do something because everybody is pissing their pants , even if it’s about something that is only really scary as a direct result of their scaremongering.  I’ve already seen a good amount of AI boosters trying to rein in Jacob Coxon’s scaremongering, or suggest that everybody calms down and remembers that AI is the biggest thing on the stock market.  At this point, it would’ve been really nice if the industry was operating in lock-step, except, as ever, Sam Altman had to go and fuck everything up, telling Fortune the following when asked whether pauses would cost the company a lot of money:   This is a very, very worrying thing for Altman to say given that OpenAI has projected to spend $750 billion or more in the next three years across compute contracts with Microsoft, Google, Amazon, CoreWeave, Cerebras and other providers.  In fact, the very concept of a slowdown runs contrary to everything that the AI industry needs. If NVIDIA is to sell $670 billion or more GPUs in Fiscal Year 2028 or, per analyst expectations, Anthropic and OpenAI are to spend more than $444 across Google, Microsoft and Amazon in the next three years , or Broadcom is to sell nearly $600 billion in AI chips in the next three years , both Anthropic and OpenAI must keep and make their $1.3 trillion in compute commitments and support the development of 10GW or more of capacity, all of which requires them to continue accelerating at a dramatic pace. Softbank just raised $11.87bn in debt from around twenty banks — all to support its investment in OpenAI, and more than its target of $10bn — and that wouldn’t be possible if the model labs had collectively decided to temper the pace of model development.  There is no way a “slow down” actually gels with the overall narrative of AI’s rapacious growth. As Anthropic and OpenAI represent 70% of hyperscalers’ AI revenues , there really is no fallback plan — there are no other customers who will naturally fill out the hundreds of billions of dollars’ worth of infrastructure, no other uses for the hundreds of thousands of GPUs bought from NVIDIA outside of generative AI, no ways in which we can simply “use the models we’ve got forever” without inherently accepting the limitations (and unsustainable costs) of running LLMs.  A pause could, in theory, mean that AI labs could slash their worst expense — training costs. While this might have the short-term benefit of reducing costs (and maybe even, with the right amount of accounting shenanigans, eek out a razor-thin positive margin), it’s likely that Chinese open source developers would distill ( as they have been ) Western models, create a much cheaper and “good enough” model to compete, and their “lead” in a race where everybody loses money would deteriorate.  Even then, what are OpenAI or Anthropic if they’re not cranking out some new version of a model or creating some vague sense of virality about the next one? What possible use is an Altman or Amodei if they’re not always on the phone to somebody signing hundreds of billions of dollars of compute contracts or promising some journalist-adjacent homunculus that Anthropic is going to cure cancer ?  What is the LLM industry without a series of promises that extend infinitely into the future? What is Anthropic or OpenAI without the suggestion that it might be something completely different in an ever-distant future? It isn’t clear, but what is clear is that a “slow down” does not gel with “insatiable demands for compute” or somehow being able to pay more in operating expenses in a year than Microsoft or Meta .  There’s also the very reasonable question of what happens to SoftBank if OpenAI can’t go public , which is now a very real possibility. With over $40 billion of debt due to be refinanced this year at a time of skyrocketing interest rates, it’s probably the single-worst time in history for it to be doing a $20 billion bond sale (separate from the aforementioned $11.87bn bank loan), which is why I think things are getting a little tight. You’ll notice I’m a little light on predictions, and that’s because everything is a little volatile right now. Nobody has really committed to an actual slowdown beyond vague suggestions of an “independent” authority that would look at models and do something or rather, and based on what Amodei has said, it’s clear that a “pausing” really just means “saying we’ll take a little more time but not really change how we’re doing business.” Alternatively, I’m dead wrong, and this is a moment of actual change caused by a runaway narrative years in the making. By deliberately misleading the media and the general public about the current and future capabilities of Large Language Models as means of inflating their valuations and justifying massive expansion of AI compute capacity, the labs made a sales pitch driven by scaring people into submission, assuming, like they do with their technology, that they had complete control over the situation. As it stands, a slowdown is deeply impractical due to the massive commitments. As OpenAI and Anthropic make up the vast majority of AI compute demand, any contraction of that demand would mean material restatements of revenue (and the $748 billion in revenue backlogs ) across every hyperscaler, along with the neoclouds and any other counterparty. The AI industry’s entire pitch to investors has been that all of these GPUs would be used and then some and that we needed to build all this capacity to reach the heights of AI breakthroughs , and while it was already questionable whether or not we needed that capacity, we certainly don’t if we’re “slowing down.”  And I must be clear, AI cannot “slow down” without creating some kind of serious financial crisis within the tech industry. Hundreds of billions of dollars’ worth of hyperscaler revenues and data center capacity is tied up in the idea that demand for it actually exists, and if the two companies with the most demand suddenly need to slow their roll, it’s hard to see how the capacity gets used.  Worse still, we’re most decidedly not done issuing debt for AI data centers — and I don’t see how anyone hearing about some kind of “AI slowdown” (real or imagined) feels particularly confident in backing a data center, considering investors barely understood what they were investing in to begin with. The fact that Anthropic is going full steam ahead with its IPO is a sign that it doesn’t really care about slowing down, but all of this talk about “AI dangers” — even as we fail to deal with a single one of them — is enough to rattle an already-nervous market about the future growth trajectory of a company where people keep leaving and saying “it’s gonna kill us all!”  Some are arguing that the “slowdown” talk is a way to unwind the AI trade — to give AI labs a way out of their $1.3 trillion in commitments — and while it may or may not work out that way, I think it’s far simpler: the AI industry is run by a series of different entities with deeply cynical and selfish beliefs, all operating “in sync” only so far as it benefits ideologies and intentions that change on a daily basis.  Sadly, in the end, none of this is about actually fixing or mitigating the harms of Large Language Models, or holding those who have perpetuated those harms responsible.  What it may do — though I’m not getting my hopes up prematurely — is lead to the unwinding of the AI trade as reality slams head-first into the scaremongering overpromises of some of the least-trustworthy and most-craven executives in the history of society. Perhaps it’s a way that these massive cloud compute contracts could be canceled, or a way to reduce these labs in size. It may also serve as a convenient way to avoid admitting that they’re running out of things that LLMs can do that approximate a product or even the completion of a task. Alternatively, it could just be another brief moment in the history of a bubble inflated by its misinformation and a media ecosystem dedicated to spreading it. The Hugging Face attack and any other “hacking incidents” are a result of poorly-run AI labs training volatile neural networks bankrolled by and run on infrastructure owned by the richest and most-powerful companies in the world. Every attempt to focus this conversation on “what AI is doing” or “what AI could do” deliberately or otherwise separates us from the grim truth that we need to start arresting people for committing crimes, halting any and all training runs, and putting real safeguards on this technology — not because it’s all-powerful, sentient, conscious, or even “innovative,” but because it’s clear that the people running these labs are irresponsible and the people backing them don’t give a shit. There are, however, things we can do, per former FTC Chair Lina Khan , if we actually had any interest in doing so: If LLMs were a toy, they’d be taken off the shelves. If LLMs were a drug, they would be banned.  I agree that we need to take “AI safety” seriously, but that starts with treating LLMs as normal software run in a reckless and dangerous manner by malevolent entities with little regard for society. If you liked this piece, you should subscribe to my premium newsletter. It’s $70 a year , $18 a quarter , or $7 a month , and in return you get a weekly newsletter that’s usually anywhere from 10,000 to 18,000 words and provides vast, detailed analyses of the biggest events and companies in the AI bubble. If you want to get in touch — and especially if you have any juicy information about Anthropic, OpenAI, or any other companies in the AI bubble — hit me up on Signal at ezitron.76. I’m also on IB on The Terminal. OpenAI operates like a billion-dollar adult summer camp where its AI scientists can burn millions of dollars in compute without anyone really noticing. OpenAI has godawful security practices.  OpenAI doesn’t really give much of a shit about AI safety, or if it does it’s very, very bad at it.

Security Business AI

▲ 0

0 views

Stratechery 2 days ago

Pacing the Frontier, AI’s Digital Limits, AI Commissars

Dario Amodei wants to pace the frontier; it's an unrealistic proposal that seems mostly geared to political control of AI.

AI Business Culture

▲ 0

0 views

Sean Goedecke 2 days ago

Slow developer experience will bottleneck fast models

Right now developer experience is measured in seconds. If your tests take a second to run, that’s good; if they take thirty seconds, that’s bad. Any faster than a second doesn’t really matter, because most of your time is spent either thinking or waiting for an AI agent to spin. Shaving milliseconds off your dev server reload time or whatever is pointless: that’s not the bottleneck. It will be. Small models are getting faster and faster, and smart models are getting smaller. I think most engineers will still want to use the smartest available model — software engineering is hard — but we will increasingly see faster models get used as subagents or for well-understood tasks. This is largely uncharted territory. Very few people have developed intuitions for what it is going to be like to work with agents that run at thousands of tokens-per-second. GPT-6-Astra can run at about sixty tokens per second. That means you spend a lot of time waiting for it to think. You work with it like you would work with another human: delegating a task and then context-switching until that task is complete. If you haven’t yet, have a play around with Jimmy , Taalas’ version of LLaMA-3.1-8B running 1 at seventeen thousand tokens per second . No matter how long the response is, it arrives in the instant of you hitting send. The model is not good enough for agentic work, but it gives a glimpse of what it would be like: you would simply get your answer instantly 2 . Well, that’s assuming the agent’s tool calls are fast. When generating tokens is not the bottleneck, it will suddenly matter a lot whether it can read a file in 100ms vs 10ms, or whether it can run your tests in 500ms vs two seconds. Fast tool calls are going to be the difference between a near-instant response and having to wait several minutes. There is thus going to be enormous pressure to do agentic coding in languages with fast compilers and tests, like Golang, and to tightly optimize the dev loop in agentic codebases. Teams focused on DevEx — developer experience — are largely a relic of the 2010s, when companies were incentivized to make their engineers happy. Most companies have cut them down to a skeleton crew or removed them entirely. But we may see a return of DevEx in the late 2020s, focused on speeding up the experience for AI agents. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. ↩ Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. ↩

AI Go Performance

▲ 0

0 views

iDiallo 3 days ago

AI Forces You to Commit to Your Initial Belief

Have you ever watched any old hacker movie lately? The hacker can write at 180 WPM, never using the backspace or changing his mind ever. It’s like before they ever started, they already knew the last character they would type on the screen. But that’s not how it works in real life, at least not for me. The idea changes mid-sentence. That's probably how most of the code I write ends up. I start with an idea in my mind. I think it's brilliant, maybe it’s a way to restructure the code, an overall design, or a specific paradigm. But when I start writing it, something else pops up. Seeing it halfway down the page, I realize it's not quite what I was imagining, and I change direction. The same principle applies when I'm writing a blog post or a story. I feel like I have a perfect concept in my head, but the moment I start putting words down, things shift and I adjust accordingly. That's why the final text rarely matches the original vision. It’s as if writing itself is the process of refining the idea. This process is lost when you use AI. You don't write out the details of an idea as you go; instead, you just give it the gist of it. It generates an entire wall of text instantly, and now you're stuck reading through it while the tool tries to preserve that initial draft, or its own interpretation of it at least. You only discover flaws or attempt to pivot while reading the generated output. But pivoting inside a wall of text is much harder than making a 180-degree turn mid-sentence. If you spot a small part you dislike and try to remove it, it might break something 20 lines down, forcing you to step back and make sure everything still aligns. Using an LLM forces you to commit to your very first thought. When you write manually, every keystroke is a chance for the idea to evolve. When you use a large language model, you have to absorb the entire output all at once, which ends up being far more time-consuming.

Programming AI

▲ 0

0 views

Sean Goedecke 3 days ago

AI is breaking our proxies for expertise

Mathematicians are broadly not anti-AI. They’re more culturally open to using AI as a tool than, say, artists or writers 1 . However, now that more and more genuinely prestigious problems have fallen to AI, that might be changing. Almost five thousand mathematicians (including twenty-five Fields medalists) have signed a declaration called A Severe Misalignment of AI in Mathematics . The core argument goes something like this: In recent months, the success of AI in solving major mathematical problems has made headlines even outside mathematical circles. But solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight. Forgetting this in the world of AI may turn the tool against the primary goal. Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas. A lot of people online have interpreted this as the expected complaint from any field that gets automated: translators did it, artists and programmers have been doing it, and now it’s the turn of the mathematicians. I think this is too dismissive. Understanding the concrete problem mathematicians are upset about can help us better understand the impact of AI on our own fields, and what we’ll have to do about it. There are two types of mathematics. Most people are familiar with the first, which we might call “puzzle-solving”: you take a problem and try to find a solution to it. When you’re a student, these problems are typically easy, like simplifying some algebraic expression. When you’re a researcher, these problems can be nearly impossible, like proving Fermat’s Last Theorem . Puzzle-solving is easy to understand but hard to do, which makes it impressive to non-mathematicians, which makes it highly prestigious. In other words, puzzle-solving is legible . The second type of mathematics is “idea-generating”: coming up with new ways of thinking about mathematics, and thus new terms or concepts. For examples of these, just glance down the list of arXiv mathematics papers . “Hardy spaces”, “Schatten exponent”, “Banach lattices” and so on are all concepts someone thought was interesting. This work is largely unimpressive to non-mathematicians, because nobody really knows if the concepts you come up with are particularly difficult or insightful. For instance, I have just generated the concept of a “Goedecke set”, which is the set of all natural numbers whose digits add up to a prime number. Who cares? The categories we want are the “natural kinds” of mathematics — the concepts that “carve nature at its joints” — and it’s almost impossible to tell what those are without years or decades of hard work. How are the two types of mathematics related? We might say 2 that generating ideas is the real intellectual work of mathematics. Puzzle-solving is important instrumentally: to identify which ideas can be used to answer longstanding questions, and thus which ideas are worthwhile. Over time, those worthwhile ideas become better understood and easier to use, until they reach the point where they can be used to advance science in general. Eventually the ideas become so well-understood that they can be taught to children: “zero”, “negative numbers”, “imaginary numbers” and “calculus” were all once rarefied mathematical ideas, but are now concepts we’d expect any precocious twelve-year-old to grasp. There’s another, more prosaic purpose of puzzle-solving: to make mathematical skill and progress legible to outsiders. I can’t appreciate Terence Tao’s mathematical work, but I know what a Fields Medal is. I don’t have a good intuitive sense of what a Galois representation is, but I know about the proof of Fermat’s Last Theorem . We might say that puzzles like this have served as a way to indirectly reward skilled mathematicians for their more important idea-generating work (or for conclusively demonstrating 3 that the ideas used in the proof are useful). AI proofs undercut both of these purposes. I can now lay out precisely why I think mathematicians are so unhappy: This is kind of like Goodhart’s Law . Puzzles were a useful, impossible-to-game measure for mathematical progress. But now that AI companies can game that measure (by solving them in a way that’s inaccessible 4 to humans), the whole point of those puzzles disappears. Are the mathematicians right? I think it’s broadly unclear whether (3) is true: i.e. whether frontier AI models aren’t generating or can’t generate new mathematical ideas. We’re still in the very early days of AIs solving our hardest mathematical problems. Who knows what they’re going to be capable of? I give basically zero credence to the idea that AIs are incapable of this because of some intrinsic feature of how LLMs work. For the last three years, we’ve seen people claim that LLMs are intrinsically incapable of X, only to have LLMs excel at X a few months later. Even granted that (3) is true, there’s still work to be done for human mathematicians in building the conceptual machinery that can make AI-generated proofs accessible to humans: i.e. in generating a “human proof” to go alongside the existing “AI proof”. In fact, I’d expect the existence of an AI proof to help with this. If you know proposition X is true, it’s easier to figure out why, because you’re not constantly worried you’re wasting your time. For more on this, I recommend Gwern’s blog On Really Trying , where he quotes a series of instances where simply being told that a solution exists is enough of a clue to help people find it. Of course, there’s a prestige and motivation problem. “I’m the first person to solve Navier-Stokes” is a much more compelling target than “I figured out a better way to explain the AI solution to Navier-Stokes”, and it’s much easier to award prizes for. Will mathematicians bother to work on problems that have already been solved? I think so. To see why, we can look at other domains where AI has come in and outcompeted the best humans, such as chess or video game speedrunning. I can run a chess program on my phone that will beat Magnus Carlsen 100-0. Computer programs — called “tool-assisted speedruns” or “TAS” — can finish any video game much faster than even the fastest human. But in both of these areas, humans still compete in human-only leagues, and there’s still prestige attached to the most capable humans. It’s possible that mathematics ends up in this kind of state, where “human mathematics” and “AI mathematics” exist in largely separate spheres, and the first “human” solution to a mathematical problem can still earn acclaim. In fact, in both of those areas, the presence of inhumanly strong computer players has improved the human game. Despite many computer chess moves being basically incomprehensible to humans, top chess players have learned from the computer “style”. In speedrunning, many moves once considered “TAS-only” are now performed by humans. AI mathematics might likewise improve human mathematics. I am not a mathematician. I did major in mathematics during undergrad, and I have fond memories of proofs from real and complex analysis, but it’s not even close to my field. However, I am watching the effects of powerful AI on mathematics very closely, since my own field — software engineering — is being colonized by AI agents in the same way. The field of software engineering does not have the same structure as mathematics. We write code to make money, not to earn prestige or advance the frontier of human knowledge. But AI is undercutting the traditional avenues for prestige in software engineering as well. It used to be that you could put a meaty project on your GitHub — say, an emulator, or a toy OS — and people would know you were a skilled engineer. But now projects like that are worthless, because everyone just assumes they’re vibe-coded. We used to tell stories about engineers who would disappear and rewrite a system over the weekend, or produce thousands of lines of code a day. Now anyone can do that with an OpenAI subscription. Like mathematics, software engineers are going to have to rebuild our cultural sense of the kind of work we value. We are either going to have to silo “AI work” off from “human work” like chess, or to find some legible human skills to recognize that can’t be easily counterfeited by AI. In the meantime, a lot of people who were successful in the old world are going to be very unhappy. Possibly because current AI models are much better at mathematics than at art or writing. Again, I am not a mathematician: here I am interpreting what I’ve read from Terence Tao and other mathematicians. This kind of idea-sharpening or idea-validating is part and parcel of idea-generation, and just as important. It’s interesting to compare mathematics to, say, philosophy, which has the same idea-generating task without the corresponding puzzles to validate the ideas. That’s one reason why philosophy has less prestige than mathematics. A thousand-page Lean proof is theoretically understandable by humans, but if no mathematician can hold the entire idea in their head it doesn’t matter. Puzzles serve as a high-legibility, high-reward target for mathematicians To solve these puzzles, new ideas must typically be generated; the puzzle’s solution serves as evidence that the ideas are useful But now AI can solve many of these targets “the hard way”, without generating intuitive new ideas This undercuts both ways puzzle-solving supports idea-generation: AI companies claim the prestige while not meaningfully advancing mathematical progress This is bad for mathematics as a whole, because puzzle-solving is ancillary to the real goal of mathematics Possibly because current AI models are much better at mathematics than at art or writing. ↩ Again, I am not a mathematician: here I am interpreting what I’ve read from Terence Tao and other mathematicians. ↩ This kind of idea-sharpening or idea-validating is part and parcel of idea-generation, and just as important. It’s interesting to compare mathematics to, say, philosophy, which has the same idea-generating task without the corresponding puzzles to validate the ideas. That’s one reason why philosophy has less prestige than mathematics. ↩ A thousand-page Lean proof is theoretically understandable by humans, but if no mathematician can hold the entire idea in their head it doesn’t matter. ↩

AI Science

▲ 0

0 views

Simon Willison 3 days ago

Generating running routes with GPT-6 Astra and ChatGPT Work

Here's a neat thing I had ChatGPT Work with GPT-6 Astra (Max) do this morning: It worked for 27 minutes and produced exactly what I'd asked for, as both an embedded visualization and downloadable GPX file and GeoJSON files. Here's that 5K route: When I asked it how it had created the route, it replied: I used Nominatim to locate the address and Overpass to download local OpenStreetMap roads and trails , then calculated the loops locally. Frustratingly, the actual code it ran and exact details of what it did weren't visible to me in the ChatGPT UI. I see this lack of transparency is an anti-feature. By the time I thought to ask for a copy of the Python code it had used, ChatGPT was unable to provide it. This appears to be because the thread had been compacted. I think any LLM system that uses compaction needs to both preserve the pre-compacted text and make that text available via agent tool calls, to protect against this kind of problem. As for displaying the map to me, that used the visualize skill . It created a file called to embed directly into the ChatGPT UI. Here's a copy of that HTML , which starts like this: The element contains the full geometry needed to render both the running route and the map itself, using D3, which is loaded from an allow-listed CDN location described in this section of the visualize skill : You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . The CSP allows only , , , , , , and . Other origins are blocked and fail silently.

Data Visualization AI Python HTML

▲ 0

0 views

Simon Willison 4 days ago

OpenAI agents attacked RubyGems back in May

OpenAI agents carried out an undisclosed attack on RubyGems is a new bombshell report from Spencer Kitts, Thomas Larsen, and Sydney Von Arx - three of the four authors of the report on the agent attack on disused wikis ( previously ) last week. This time they're noting that it looks very likely that an OpenAI agent swarm was behind an attack against the RubyGems package repository first reported on May 12th by Maciej Mensfeld of the RubyGems security team : We're dealing with a major malicious attack on @rubygems right now. Signups are paused for the time being. Hundreds of packages involved - mostly targeting us, but some carrying exploits. The team has been on this for hours. More details to follow once we're through it. Those packages turned out to carry some very suspicious patterns: I find point 2 the most convincing, given what we learned from the wiki attack when it was analyzed in September. Many of the packages were exploiting the RubyDoc.info documentation build process to exfiltrate (public) data from UK government websites, presumably as part of an information gathering task similar to the research tasks processed by the wiki-exploiting agents. We know this because one agent helpfully left a comment: They also attempted to steal API keys via an exploit that was patched over two months later - it's not clear if those attempts were successful. The thing that bothers me most about this incident is that the authors report that OpenAI had not disclosed to RubyGems that they were responsible for the attack prior to now. If that's true there are two options: Both of these are bad! Given this incident, the Hugging Face situation , and the Wiki attack, the obvious question right now is how many more incidents like this are out there waiting to be discovered? You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . Many of them included "oai" in their name, or the author field, or the fake email address they provided. The files they were accessing were similar in character to the files retrieved by the wiki agents, using similar tricks (r.jina.ai) - and OpenAI have confirmed the wiki agents were theirs. The code in the packages appeared to be LLM-authored. After the Hugging Face and Wiki attacks OpenAI were still unable to review their previous logs and determine that they had previously attacked RubyGems. They knew about the attack on RubyGems and made the decision not to reach out to the RubyGems team about it.

AI Security

▲ 0

0 views

Tenderlove Making 4 days ago

What a time to be alive

Today Reuters and the Wall Street Journal both reported about rogue AI agents at OpenAI attacking RubyGems.org. https://www.rubyhack.ai/ has an amazing writeup, and you should read it. I just wanted to make a quick post about it because it’s wild . TL;DR: It seems like OpenAI Bots knew about this caching vulnerability , tried to take advantage of it, and at the same time ran some weird web scraping code on RubyDoc.info. Back in May, socket.dev reported about a “GemStuffer Campaign” where someone (I guess OpenAI) was uploading tons of junk gems to RubyGems.org. For some reason, the gems would scrape UK government websites, then repackage the data as gems, and attempt to upload them to RubyGems . I honestly didn’t think much about this (or even look into it) until Sydney Von Arx and Spencer Kitts (both co-authors on https://www.rubyhack.ai ) contacted me asking about RubyGems. I thought the claims they were making were completely outlandish until I actually read the code in these “GemStuffer” gems. After reading the code in these gems, a couple things stood out to me. First, the gems leverage YARD documentation to execute arbitrary code on host machines. In most of the examples you’ll see a file that looks like this: Here’s a link to an example . If you have YARD installed, and you install this gem, then YARD will load and run whatever is in from inside the gem. I think it’s pretty common knowledge that C extensions will execute (so you basically have an RCE vector), but I was surprised to find out that a documentation tool would do that too. Nobody is going to install a gem named though, so why would this matter? Well, any time a Gem is published RubyDoc.info will download the gem and process the YARD documentation. RubyDoc.info will execute the arbitrary code inside a Docker container . The Docker container still has network access though, so these gems could happily do their web scraping from inside the container. In other words, if you publish a gem on RubyGems.org, you can execute arbitrary code on RubyDoc.info. I mentioned earlier these gems would try to scrape some websites and then upload the data they scraped by packaging it as a gem. Here is an excerpt from one of the gems. I’ve cleaned up the code a bit so it’s easier to understand, but the original code is here : Comments in the code that have are ones that I wrote to try to help make it easier to understand. The first comment was lifted directly from the source . The above code tries to make two requests. The first request is a simple GET request. It tries to fetch a path from RubyGems.org, then looks for a key in the response body that matches the regular expression . If that regular expression doesn’t match, it falls back to a global . The second request tries to upload the gem via POST. This brings me to the second crazy thing that stood out to me. This code is trying to fetch a cached authorization key from RubyGems.org . If this sounds familiar, it is. It’s exactly the security issue addressed in this post from RubyGems.org that was made in July. In other words, it looks like OpenAI’s bots knew about this problem and attempted to exploit it. What a time to be alive 🙃

AI Security

▲ 0

0 views

Armin Ronacher 4 days ago

P(doom)

This week some flavor of “AI is going to kill us all” went viral. In particular one where an employee put his personal probability of that happening above 10%. Which made me go to the Wikipedia page of P(doom) and I realized that Dario Amodei’s apparent probability of something bad happening seems to be between 10-25%. And well, Dario then wrote about pacing the frontier . And Sam read it and wants to pace too . And well, so does Musk . I encourage you strongly to read the post, because I think it’s a good one. And yet, when I read the post I could not help but feel in strong opposition to it, despite the fact that I think I’m on the same page with regard to all observations and, to a large degree, the concerns. I thought it might be interesting to write down my present-day thoughts on this, even if for no other reason than for myself to look back at it a year or two from now. What I really appreciate about Dario’s post is that he lays out a scenario that is not a huge stretch but also one that describes a clear, unfortunate outcome we should fight: persistent botnets and other forms of nuisance. And well, we don’t have to look very far to see the issues left and right. Wikipedia has a page called 2026 OpenAI agent cyberattacks which gives you at least some overview of what we figured out agents have hacked up to this point. Except I know it’s not up to date, because for instance they also poisoned RubyGems . Today these systems might be annoying, but they can be turned off when we figure out where they are. Except, it seems like OpenAI and Anthropic are operating at such a scale that they seemingly can be completely blind to what their systems are doing. I don’t think we are anywhere close to a world where an agent might decide to hack into core inference infrastructure to upload weights to other GPUs to survive. But simultaneously it’s entirely in the realm of possibility and primarily curtailed by the labs probably being particularly careful about their IP. For me the scenario I primarily worry about is what it does to us. And by us I mean anyone who is not currently working on closed weight, dopamine-loaded, subsidized token faucet. I really don’t worry about someone using these models to build a nuke, or to control some rockets in the Middle East, or that America would lose against China in some international culture war. I almost exclusively worry about what this does to us as humans. What I find absolutely hilarious and simultaneously entirely frustrating about this conversation is that there is this idea that there is something to be paced. First of all, we should really talk about who Dario is talking about here. There are really only two companies: Anthropic and OpenAI. Nobody else matters in this space right now (this might change, but we’re talking about the right now). Both of those companies are basically coming from the same origin. The solution that Dario proposed, at least in part, is a third-party evaluator that in this case is METR . Which, unsurprisingly, also has strong ties to both OpenAI and Anthropic. Sure, there are some philosophical differences between the companies, but they are much more alike than they are different. Both those companies greatly benefited from being able to train on public data that we all generated in one form or another over the last decades. They are also both increasingly causing strain on public resources, though it seems that OpenAI has their shit way less under control. But now we are presented with the idea that what these models are being trained on is so dangerous that it really should be in the hands of very few American corporations to decide who can do what and when and how. But behold, Dario is also very worried about China. It starts with using AI for “democracy and freedom” and then it asks for ensuring that a gap with China exists. All new recent shenanigans on the Anthropic API are fully there to prevent the distillation by the Chinese, and they are not at all hiding it. I can tell you when the topic of AI safety and pacing is much less of a concern: if we actually were forced to have open weight models to begin with. A powerful technology that is out there for everyone to use comes with built-in pacing. In a way it’s the truest form of MAD or proliferation. I would argue we are in this pickle in the first place because right now the public is massively supporting (indirectly) the development of these models but simultaneously has to buy back the economic benefits that they might create from very few labs who have significant power. And their power is also seen as a geopolitical power, at least in the US, and maybe to some lesser degree in China. And I know I use “public” loosely here. PyPI is not a public project, nor are RubyGems or GitHub. But they’re part of the Open Source commons and large AI companies are currently doing a tremendous job at stressing these in an effort to train ever more powerful models. We should be glad that China is currently massively bailing out the world. If it were not for Chinese labs distilling American models, we would be in a pretty awful situation right now, particularly as Europeans. The open weight models are driving innovation and the diffusion of capabilities, and are leveling the playing field. If we greatly restrain our AI capabilities in the belief that China will do the same, and then China defects, AI could be so powerful that such a defection could lead to their geopolitical dominance. Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential. — Dario Amodei I am assuming Dario has reasons to believe this, but the models that are actually causing issues right now are all closed weight American models. I’m fairly certain if they were open weight models, we would not have that issue. Why? Because for a start, the economics of serving up these models are only that distorted due to how the big labs can operate. OpenAI is casually burning 18 million USD to brute force a problem on a whim. They are operating subscriptions at a massive loss, distorting the market everywhere. If we had mass accessibility on somewhat equal terms, a lot of the crazy issues we are seeing today would not be taking place. From where I sit, what we observe right now is a total regulatory failure everywhere. In Europe you have some whacky AI regulation that is two years old and completely misses the problems that we actually have and focuses on problems that nobody has. In the US we’re seeing a system that is probably best described as turbo capitalism paired with sinophobia and erratic decision-making. In the chaos in which we find ourselves, the reality emerges. And the reality is, even today, really problematic. Whatever laws and regulations already exist are largely completely ignored. Plenty of companies are buying data from all over the place that people never agreed could be used for training of AI models. The token economy that is emerging is one that looks like a drug market where you don’t know where the requests are going, what model is served up to you, where the GPUs are even running, let alone what you pay for all of this. We now have mathematicians who are scared that their use of ChatGPT leads to future models being trained on their ideas, and OpenAI apparently can’t even rule it out . Ideally the regulators would have forced these models to actually benefit the commons if they are from the commons. The internet has, for instance, greatly benefited from very liberal rulings in the US that permitted scraping. Learning on public data could have been regulated in a way that labs would have to actively support and enable certain forms of distillation. That alone would dramatically change how these models are trained. As I said before, I don’t think AI is going to usher in an extinction event. In fact, even if nobody were to slow down, I really don’t think humanity would have much to worry about. I tend to think it would actually be the large labs that have much more to lose there in reputation and legal responsibilities. I find it preposterous that OpenAI’s agents are committing actual crimes out there, but we’re just shrugging our shoulders and moving on as if nothing happened. But I’m sure executives in those companies are waking up to the reality that this is not at all popular with a lot of their potential consumers. I also think that this entire recursive self-improvement business has a good chance of being a problem. But not necessarily in that it will cause the end of humanity or societies, but that it will just do massive damage everywhere. And really, it will just make a lot of the things we are doing much more expensive. Software engineering is an early victim of that. The newfound powers so far have resulted in a new tax that companies need to pay to the model providers, both to keep up with the new speed and to deal with the problem of these machines finding security issues left and right. And presumably what is going on in software will happen to more industries. Universities and research groups will have to pour a lot of money into the closed models as well, to keep up with others who do. In a way, I’m really confused that society is taking all of this so well.

AI Security

▲ 0

0 views

Stratechery 5 days ago

2026.37: Duo Threats

Welcome back to This Week in Stratechery! As a reminder, each week, every Friday, we’re sending out this overview of content in the Stratechery bundle; highlighted links are free for everyone . Additionally, you have complete control over what we send to you. If you don’t want to receive This Week in Stratechery emails (there is no podcast), please uncheck the box in your delivery settings . On that note, here were a few of our favorites this week. This week’s Stratechery video is on Autonomy and Innovation . The Duo Arrives.  Does the world need a foldable iPhone that costs between $2000 and $3,200? It’s a fair question. On the other hand, life is short, and tech is a lot more fun when we have new and possibly-crazy hardware projects to discuss — particularly when they’re deployed by Apple. To that end, I heartily recommend cleansing your palate from a week of media-wide AI angst by reading Ben’s take on Apple’s iPhone event and pairing that with Friday’s Dithering , and John Gruber’s impressions of Duo-mania on the ground in Cupertino. Also: bonus being-right points to Gruber, who nailed the name of this device back in April .  — Andrew Sharp AI That Benefits Humanity. I loved Wednesday’s Update contrasting OpenAI’s thrilling and technically impressive Navier-Stokes breakthrough with the release of Meta’s far less sexy Muse agent. While OpenAI’s tactics may in fact chill research in advanced mathematics, what Meta has assembled is free (to consumers) hardware and software that dramatically reduces the barrier to entry for ordinary people looking to harness the power of agents, making the AI upside a lot more accessible to the masses who don’t want to buy a Mac Mini. That’s a big deal! We discussed Muse more on this week’s Sharp Tech , including tips for getting started with agents, and questions about whether people will actually take advantage of this opportunity.   — A S Closing the Book on a Catastrophe.  Everyone’s familiar with the benefits of pro sports ownership and its ability to turn semi-anonymous rich guys into full blown celebrities, but Microsoft co-founder Steve Ballmer is now a living testament to the unstated risk — sports ownership fame can, in a worst case scenario, become infamy. Last week his Clippers received the harshest penalty in NBA history for circumventing the salary cap to pay Kawhi Leonard. We recapped all of it on GOAT this week , including successes and failures in sports journalism, why Kawhi got off easy, and the staggering amounts of evidence that sealed Ballmer and the Clippers’ fate.  — AS Write Things Down — Writing things down is powerful, for humans and for AI; what comes first, however, is what to write, why to do it, and actually getting things done. OpenAI Does Math, Reward-Hacking, Meta Launches Personal Agent — OpenAI solving one of the most famous math problems is extremely impressive, and of little impact to most people’s lives; Meta’s Muse agent launch has the potential to be the exact opposite. The iPhone Duo, The Intelligent Personal Hub, Apple Watch Audio Intelligence — Apple once again demonstrated the power of integrating hardware and software, but it’s biggest AI blindspot might be its belief in the primacy of apps. Agents and Forklifts The Flood that Wrecked the Hard Disk Drive Industry Did Numerical Control De-skill Machinists? Closing the Book on the Clippers Catastrophe and Early Over/Under Picks in the Atlantic Astra (and AGI?) Arrives, Meta’s Muse and the Agent Opportunity, Anthropic and the Revival of (P)Doom Angst

Hardware Business AI

▲ 0

0 views

Ed Zitron's Where's Your Ed At 5 days ago

Premium: The Hater's Guide To Broadcom

According to The Information , in early 2024, Broadcom CEO Hock Tan hosted a “coffee chat” with employees of the recently-acquired VMWare , and introduced them to his particular brand of management: He may not be your dad, but Hock Tan sure is a motherfucker. Broadcom is a company you likely know for its XPU platform — a collection of different bits of intellectual property and access to semiconductor parts that allow it to build custom AI chips, the best-known of which are Google’s TPUs . It just signed a $30 billion deal with Apple to build “custom ASIC silicon products .”  Apple was already a massive customer of Broadcom, which historically provided a good chunk of  the wireless and radio frequency parts that you’d find inside iPhones and its other devices, representing at one point more than 20% of revenues, dropping to around 10% to 15% with the growth of AI chip sales and the acquisition of VMWare. For the most part, Broadcom’s business is built on selling companies the internal bits and pieces of either their hardware or the hardware surrounding their hardware — everything from wireless and RF components to data center networking tools.  It also dabbles in mainframe software (from its acquisition of CA Technologies ), security (from its acquisition of Symantec’s enterprise security business ), and virtualization software (from its acquisition of VMWare), and these segments cost it a combined $99.1 billion in cash and stock (not counting for inflation).  Except “Broadcom,” as a company, wasn’t always called Broadcom, and wasn’t founded by Hock Tan. As I’ll get into in this piece, “Broadcom” was once two very different companies — a wireless communication chips company founded in 1998 called “Broadcom,” and the private equity-formed monstrosity formed out of a spun-off semiconductor subsidiary of Hewlett Packard called “Avago Technologies.”  Much like Oracle , Broadcom is the story of a company acquiring other companies and then screwing over both its customers and employees in the pursuit of endless growth, which usually involves price gouging, massive layoffs, and cost-cutting anywhere that won’t improve margins. A great example came from a Wall Street Journal piece from January 2018 involving Broadcom’s failed $117 billion attempt to acquire Qualcomm:  Two months later, the deal would collapse despite a dozen banks signing on ( per Reuters ) to provide Broadcom a $100 billion bridge loan to get the deal done, with President Trump vetoing the deal to avoid Broadcom (then a Singapore-based company) exercising control over the US-based Qualcomm. To give some credit to the administration, the CFIUS had ( per The Hill ) “...worried that Broadcom’s takeover would lead to a decline in investments in research and development in the sector, opening the door for Chinese firms to take the lead in developing next-generation wireless technology.”  That R&D point was a very real concern. Per The Journal: Pffft, 19%? That’s chump change. Since the acquisition of VMware, Broadcom’s R&D budget as a share of revenue has decayed to an unremarkable 9.8% of revenue in its latest quarter.  On a trailing-twelve-month basis, Broadcom is exceptional among its peers for how little it invests in R&D as a percentage of revenue, beaten only by NVIDIA, which has the excuse that it is the literal largest and most-profitable company on the US stock market.  That’s because Broadcom doesn’t really do “innovation” or care about “being good to its customers, but by hoarding other people’s patents, iterating on their creations as little as necessary, and making it impossible to avoid wiring Hock Tan money. Even its FBAR filters (used to block out interference on mobile phones) — a critical part of its deal with Apple — come from Avago’s acquisition of the original Broadcom. A year or two ago, this could’ve been called The Hater’s Guide To Avago , because that’s really been the story of Broadcom — a Singaporean semiconductor firm that rolls up other companies’ technology under a brand made famous by somebody else.  Per The Wall Street Journal, a few months before its acquisition of Broadcom :  Much like Oracle , Broadcom used M&A as a means of treading water revenue-wise, with each one having little effect on its overall trajectory outside of its acquisition of VMWare. Yet Broadcom had been building something quietly behind the scenes through the combined acquisitions of LSI (which had merged with Agere a few years previously) and its own semiconductor might — a budding relationship with Google to build its Tensor Processing Units (TPUs), AI chips that at first worked to support products like Search and Maps, and would eventually become a huge part of the AI boom.  To be clear, Broadcom didn’t “see anything coming” or “catch the AI boom in its infancy.” While it deserves some credit for rolling up various different semiconductor companies like it’s playing Katamari Damacy, this is not a situation where Hock Tan or anyone had any kind of precognitive event that made them invest in ASICs in anticipation of a massive payoff.  What actually happened was far simpler: Google, which had already been running its services powered by (non-LLM) AI, got Broadcom to use its pile of various patents and supply chain connections to put together specialized silicon that had incremental boosts to Broadcom’s revenues until the launch of ChatGPT scared Sundar Pichai into sinking billions — and then tens of billions — of dollars into successive generations of TPUs. And in Fiscal Year 2024, Broadcom began breaking out that revenue from its semiconductor solutions division, and something became alarmingly clear: that it’s become dependent on AI revenues for virtually all of its future growth. Between Q1 FY2024 and Q3 FY2026, AI revenue has gone from 19.2% to 56.4% of Broadcom’s revenue, with analysts expecting it to make up 68.4% of FY27, 79.8% of FY28 and 81.9% of FY29. And you’ll never guess who the customers are!  That’s right — OpenAI (for its Jalapeno AI chip) and Anthropic (buying Google TPUs) , who are set to become Broadcom’s largest customers in Fiscal Year 2027, which means that tens of billions (and eventually hundreds of billions) of dollars of revenue will be tied to whether two unprofitable, unsustainable AI companies can afford to pay.  This is the story of how a grab-bag of other people’s innovations has accelerated in the space of three years to become one of the largest AI chipmakers of the world, and how its desperation for growth has forced it to engage in the darkest forms of circular financing. This is The Hater’s Guide To Broadcom — the hard numbers and charts behind Hock Tan’s aggressive play to beat NVIDIA and become Google, Anthropic, and OpenAI’s chipmaker of choice…and how dangerous it might be if it fails.

AI

▲ 0

0 views

Giles's blog 6 days ago

Extending Raschka's GPT-2: an MoE trained from scratch on an RTX 3090

Mixture-of-experts models are really nifty. You get inference speed close to a small model's, with a lot of the smarts and knowledge of a large one. While they use as much memory as an equivalently-sized dense (non-MoE) model, they're much faster. The frontier labs don't publish their architectures, but Claude and ChatGPT are widely rumoured to be MoEs these days -- and certainly many large open-weights models like DeepSeek and Kimi K3 are. In this post, I'll show you how I added MoE support to the GPT-2-style code from Sebastian Raschka 's book " Build a Large Language Model (from Scratch) ", then used that to train a 446M-parameter model with 220M active parameters from scratch on my RTX 3090 -- essentially, GPT-2 small with 6 experts, 2 active per token. I wanted to understand how MoEs work, and as always, felt that the best way to do that is to build one (and then to write it up like this). Hopefully because it's all fresh in my mind as I write this, I should be able to explain things clearly for others who've finished Raschka's book. The training run took just less than eight days, and the resulting model got a better loss on my test set than any of the other models I've trained so far -- which was a good thing, given that it took four times longer to train! It was also better than the original OpenAI GPT-2 small (124M parameters), but not as good as GPT-2 medium (345M parameters), although it was close. That second result was interesting, as my model had more total parameters than GPT-2 medium, but fewer active ones. On an instruction fine-tuning test, it did better than any of my other models, but worse than both OpenAI models (why OpenAI's models are so good on that particular test is a mystery I'm digging into separately). I'll go into those numbers in more depth later on. Firstly, though, I'll run through the code that I needed to add to GPT-2 to make this work -- not just the code for the experts themselves, but the additional code for training it. With MoEs you can't just train to minimise loss on your training set -- you need to add on an "auxiliary" loss to make sure they actually use all of their experts, and that was where it became most interesting. Before we get into the weeds, though, let's start off with the basics; how do MoEs work in Transformers-based LLMs? The phrase "mixture of experts" evokes an idea of a model which has separate parts that are knowledgeable in different domains. Maybe one part would know about coding, another about history, another about philosophy, and so on. You can imagine something that had separate LLMs for different topics, and routed incoming inputs appropriately. That's actually not a bad design for a system -- Sakana.ai got a lot of interest for their Fugu system back in June of this year, and it works rather like that. But the "experts" in an MoE are at a much lower level, and (as with so many things in LLMs) their expertise is in some weird and alien thing that they determined was helpful for modelling language during their training. Let's look at how they fit in mechanically. The GPT-2-style LLM that we will start from -- the one Raschka describes in his book -- looks like this: Diagram 1 : a GPT-2-style LLM at the top level The MoE magic happens inside those Transformers layers, so let's zoom in on one of them: Diagram 2 : a GPT-2-style Transformers block Specifically, what we do to make our LLM an MoE is to replace that feed-forward network (FFN) with multiple separate FFNs -- our experts. Different context vectors get routed to different experts based on their contents. The FFNs themselves -- in both the dense and MoE versions -- are surprisingly simple . In GPT-2, they take in the incoming context vectors, run them through a normal linear layer to expand the number of dimensions by four, run the result through a GELU activation function, then project it back into the original incoming dimensionality with another linear layer. It's a really simple two-layer neural network, and at first glance seems somewhat arbitrary. Tutorials about LLMs tend to spend pages and pages explaining attention mechanisms, and pretty much gloss over the FFNs. But the FFNs take up twice as many parameters as the attention mechanism in GPT-2. They're clearly highly important, and my (very loose) metaphor for why that is, is that attention is how the LLM works out what to think about, and the FFNs are where it does its thinking. It's not a perfect match for what's going on, but I think it's a decent working model for intuition. An MoE leverages this. Instead of having just one FFN per Transformers block, we have multiple, and we use a subset of them for each context vector. We have what is called a router, or a gating network. It takes the incoming context vectors, and for each one, decides which of these FFNs -- which experts -- to use. Then we feed the input into the experts that were selected for it, combine their results, and that's our final output -- like this: Diagram 3 : an outline of an MoE block Doing this gains us more "space" in the LLM for it to remember facts and ways of thinking about things -- we have multiple experts for that knowledge to be spread over. We could, of course, do that by dedicating more space to the FFNs -- for example, by having one bigger one. But with an MoE, because we only activate a subset of the experts for each context vector, we save on the amount of computing we do for each context vector. We need to keep all of the experts in RAM -- remember that we're routing to them per-context vector, so in a batch of sequences we're likely to be using most if not all of them. But we don't have to feed everything through all of them. So, that's the basics -- nothing conceptually difficult. What becomes more tricky is the implementation. How, concretely, does the router choose which experts to use for a given context vector, how do we implement that choice -- and how can we do all of that efficiently? And how do we train the router to make its choices? When I started on this project, my first step was a Google search for useful references. I came across this excellent summary from IBM. In that post, they mention a number of papers; four seemed relevant 1 : I decided to take a look at them, and was pleasantly surprised about how readable they all were. It looked to me like I'd be able to put together a working MoE model by following what they (and the IBM summary) said, and that turned out to be true -- the model worked, and trained well. Of the four papers, I found the Switch Transformers one the most useful, but the original 1991 paper also clarified a lot to me. I really do recommend skimming through them if you want to learn more background about this stuff. Anyway, once I'd got it all put together, and trained the model, I compared what I'd written to the Hugging Face source code for Mixtral , which was the first open MoE model I remember hearing about. Mixtral has a bunch of other improvements over GPT-2 beyond its MoE support (RMSNorm, RoPE, and so on), and it's a much larger model than mine. But it turns out that the specific way it handles the MoE side of things is largely the same as mine, apart from one difference in how it calculates auxiliary loss (about which more later). So that was reassuring. I also ran the final codebase past three LLMs -- ChatGPT, Claude, and Kimi K3 -- just to make sure that I hadn't drifted from the mainstream or screwed up in other ways. I did this in an anonymous/private session so that they wouldn't use any memories they have of conversations we'd had in the past, and asked them Please take a look at the attached and tell me what you see. I'm particularly interested in the MoE stuff. All of them came back with some variation of "it's a pretty standard MoE implementation on top of GPT-2, with load-balancing adapted from Switch Transformers". Even more reassuring! So I'm comfortable that what I've built is a normal implementation, and that's what I'll describe in the rest of this post. It's all fresh in my mind, so hopefully by describing it in terms that would have made sense to me when I just finished Raschka's book, the result will be useful to other people coming to this for the first time. Let's get started by digging into the mechanics of how the router works. The problem we want to solve is that we have a context vector coming into our MoE block -- as per the diagram above -- and we want to decide which experts we want to route it to. Let's say that we have n total experts, and we want to send each input to k of them (where 0 < k < n , of course). We'll also say that our context vectors are of dimensionality d emb . Taking a d emb -sized input and categorising it into probabilities across a number of options is a pretty standard job for a neural network. In fact, we can use a single linear layer for it. Imagine that we create one with d emb inputs and n outputs. For a given context vector, it might produce (for n = 6 ) something like this: So we can imagine picking out the indexes of the top k of those. Let's be specific, and say that k = 2 . That means that we have indexes 4 and 2 -- the positions of the two largest numbers -- so we want to route our original context vector to experts 4 and 2. They do their calculations, we get output context vectors from them, and we can combine them. How might we combine them? Well, we do a lot of adding context vectors together in our GPT-2-style LLM code, and treat the results as meaningful -- token embeddings get added to position embeddings, shortcuts around attention and the FFN get added back in to their results, and so on. So perhaps we could do that? That kind of setup has a problem, though -- it doesn't train the router. Let's think about the MoE block again; here's the diagram again: Diagram 3 (repeated): an outline of an MoE block Think about the flow of data through it. The context vectors flow into the router. We use the output of the router to select the two experts, and then the original context vectors -- the ones that were fed into the router -- flow into those experts, then are combined, and we get our result. Now, consider what happens when you're training a model. You run some training data through it, then calculate the loss -- how good your model's results for that data are. You then use that to work out the gradients that you want to apply to the parameters to make it better. You work out the gradients by using back-propagation, working back from the loss, through the network, retracing the computation graph in reverse. When your backprop gets to this part of the model, it will start with the output context vectors, trace back through the combination step, then back through the two chosen experts, then back to the input context vectors -- and then it will go back to whatever step came before the MoE block. The calculations inside the router that selected our two experts did actually happen in our forward pass -- but they're not in the computation graph as we trace it backwards from the loss. It's kind of a dead-end. There's nothing in there to connect our selection of which experts we used to the path through the computation that ended up at the loss. (Interestingly, the effort of doing that diagram made that particularly clear to me -- that slightly-messy labelling of the arrows at the top, with numbers to show the sequence, is a direct result of the same issue.) What that all means is that the router will not be trained. The backward pass of our training will completely ignore it, and so there will be no gradients to apply to it. A starting model with random weights will randomly allocate context vectors to experts, and it will continue to do that. Clearly, we need to somehow modify our computation graph so that the router is connected -- so that when the backward pass flows up from the output context vectors, it sees the router -- and ideally sees some aspect of it that will allow it to be trained to do its job better. "Adaptive Mixtures of Local Experts" starts by describing some previous work which treats the router's outputs as weights for each of the experts. That's a nice trick, and while it does have some problems (as we'll see in a bit), it fixes the backprop issue. (Interestingly, they actually decided to do things differently -- but later work, like Switch Transformers, goes back to doing things this way, with some important tweaks.) As I said earlier, back in 1991 they weren't thinking of MoEs as being a way to save on computation. Instead, their focus was on training better models to handle specific tasks, and they felt that having some way to split those tasks up might make that easier. So, for their case, they might take those outputs from above and normalise them: ...and then run their inputs through all six experts, then add together the outputs weighted by those numbers -- 0.1459 times expert 0's output, 0.1249 times expert 1's, and so on. Rather like this: Diagram 4 : An MoE with all experts active With that, we have something where the router is on the backward pass through the computation graph from the output context vectors (and thus from the loss). The weights provide a route back, so the router will get trained. But that version, of course, is running all of the experts for every token, so it doesn't have the computational benefit of a modern sparse MoE. It also has an issue, as they point out in the paper, that if you imagine that you want each expert to have a well-defined responsibility, things get messy. Imagine that expert 2 in the above example was the one that knew how to solve a particular task; all of the other experts would be contributing too, and unless you find some way to train their weights down to exactly zero for that task, to get a good loss on your training run they'd need to learn to balance each other out. Their solution was to run all of the experts then to use a gating function on the output -- that is, it would drop all of the results from the non-selected experts (see their figure 1). But the modern solution is a bit different; for that, let's move on to "Outrageously Large Neural Networks". What we really want is an output from the router where some of the weights are zero. Specifically, with our k active experts, n total, we want all but the top k of them to have a weight of zero. Then when we're running the network, we can skip those zero-weighted experts and get a result that will be identical to what we would have got without skipping them. "Outrageously Large Neural Networks" does this with a trick that will be familiar from the causal mask in the GPT-2 code. Let's call the "raw" weights from our router : Let's run that through softmax again: That gives us some initial weights. But we want all but the top k to be zero. Now, if we want something to be zero after softmax, then it needs to be − ∞ on the way in. I'll show the code to do this later, but for now, let's just assume that we have some magic to set all but the top- k values to that. In our concrete example with k = 2 , the original with that change applied would be: Now we can run that through softmax: And we have some weights! With those, we can conceptually run this "all-experts-active" kind of network: Diagram 4 (repeated): An MoE with all experts active ...but skip the experts whose weights are zero. The weights that we provide through the steps above will mean that the router will get trained. That's pretty neat! There is one thing to highlight, though. Imagine if we have one active expert -- that is, k = 1 . We'd mask out all of the other ones: ...and softmax: The weight will always be one, regardless of the original logits. And if a function can only ever return the same result, its derivative will always be zero, so there will be no gradients and it can't be trained. Interestingly, that's something that is -- almost silently -- addressed in the Switch Transformers paper. They are specifically looking at k = 1 MoEs, and they mention the "Outrageously Large Neural Networks" paper's routing system, then present their own calculations which instead of replacing non-top- k values with − ∞ , and then softmax, do the softmax first, then zero out the non-top- k . They don't highlight the difference. I decided that for this experiment, I'd go ahead with the non-Switch Transformers calculations, anyway, and do the replace-with- − ∞ -then-softmax system, with some guard code to prevent it from accidentally being set to k = 1 . Most recent MoE models I've seen have at least two active experts, after all. The good news is that not only did it work -- when I checked later, it also matched the choice taken in Mixtral, which feels like a solid endorsement. 2 3 So, at this point, we know how an MoE works in theory. Let's start coding. I started with the code that I had from " Build a Large Language Model (from Scratch) ". In that, the Transformers block looked like this: I wanted to keep the capability to run this in MoE or non-MoE mode, and decided that I'd do that by extending the model config in so that it would have an optional section, which would include MoE-specific stuff. If that wasn't present, then I'd create a normal dense LLM. To do that, I replaced the line in that assigned to with this: Next, it was time to implement the class itself. The obvious config that it would need was how many experts there were in total, and how many were active per token: Now, there's that issue where having just one active expert will pin the softmaxed weight to one, so it won't train -- so I decided to protect against that (and against another obvious mistake that the config could contain): Next, it was time to create our router, mapping from the incoming context vectors (of size in this code) to the number of experts: I decided to make it unbiased because I have a vague impression that that is the fashion these days -- nothing more principled than that :-) Next, we needed the experts themselves, each of which would be one of the same modules as we were using in non-MoE mode: They needed to go into an because if they were just in a regular list (as I discovered when I tried it) they would not be registered by PyTorch as things containing parameters belonging to the module. We create our optimiser for training with code like this: ...and so anything that isn't in will never get updated, which would be a Bad Thing. That was enough to have the pieces in place. It was time to write the forward pass -- to get the router logits, do the top- k , softmax, and then to use those results to run the experts. Getting the logits was simple enough. Looking at the method: ...we have an incoming set of context vectors, . That is shaped . Let's try to visualise that. Understanding tensor operations is something you can kind of short-cut with intuition, but I think that in order to really understand the next steps it's best to have something a bit more tangible in mind. You can think of an order-3 tensor like this as a cuboid. With of 3, of 5, and of 7 -- artificially small values to keep things simple -- it might look like this: Diagram 5 : The tensor as a 3-D cuboid Every dot in that cuboid is a single number. A single context vector is the numbers as you follow a line from the "front" of the cube -- the × face to the left -- to the "back". That 3-D representation was fiddly to get right, and is likely to get confusing if we keep using it. So instead, let's look at two 2-D views of the same thing, from the front (where you are looking at a × face), and from one of the sides, where it's × : Diagram 6 : as two 2-D views So in this view, each of the circles in the "Front view" is the first number in a specific context vector, and each of the rows in the "Side view" is the set of numbers that make up a specific context vector. Hopefully that's easy to visualise. Now, a PyTorch layer like our operates on the last dimension in the tensor you pass in. For our , shaped , it will work on the dimension -- that is, it will operate on each context vector independently, which is what we want. That gives us a set of logits shaped . In our visualisation, that's a × × cuboid; let's diagram that with four available experts: : Diagram 7 : as two 2-D views Again, if we look at the front view, each circle represents the first element of the routing logits for one of the context vectors, and in the side view, each row represents all of the routing logits for a context vector. So we have the right data in our cuboid. From what we're calling the front view, it's got the same dimensions as , which is useful. In order to work out which of our experts each context vector should go through, we need to get the top- k values for each context vector in . PyTorch has a function to do exactly that: The tells it to work across the last dimension, which is the dimension of length . It returns the top values and their positions, as tensors -- both shaped . So, we have two new tensors, which we can visualise as cuboids, both of × × . Let's show that with : Diagram 8 : (or, equivalently, ) as two 2-D views Both and are shaped that way. As normal, the "front" face is the same; each circle corresponds to a context vector, and for it holds the highest value in that context vector's logits (because returns results sorted), while for it holds the index that that highest value sits at in the logits list. Looking at the side view, we're seeing a list of top-k logit values or indices for a given context vector. Now, we want a version of that we can run through softmax to get some weights, and in order to do that we need to replace the non-top- k values with − ∞ for each of the context vectors. The solution that I hit on eventually (after various other attempts 4 ) was this. Let's start with a tensor identical in size to , but full of − ∞ s: Now, contains the values that we want to have in there, and contains the indices in the logits lists where they should go. All of the other values can remain as − ∞ . This will, of course, be the same shape as : Diagram 9 : as two 2-D views Both and are compatible in their first two dimensions with , and thus with -- that is, their front faces in our visualisations are the same -- compare the front face of the above with the one for diagram 8 . For all of our tensors, the first two dimensions correspond ultimately to an incoming context vector in . It's just the last dimension and the data they contain that differ. PyTorch has a function on its class, which takes a list of positions and list of values. It takes one specified dimension, and treats that one essentially as a set of lists. Then it takes two other tensors with the same number of dimensions as each other, each of which matches in size in all but the specified dimension, and treats the other dimension as being a list of values for one parameter, and list of indices for the other. It overwrites the data at the specified indexes with the specified values. That sounds useful! Let's make it concrete. Remember that in all of these cuboids we're visualising, the front view we're looking at corresponds ultimately to a context vector. In , it's the raw logits for expert routing for that vector -- the number on the front face is the first of those (corresponding to the logits for routing to the first expert). If we look at the side view, each row would correspond to the set of logits for a given context vector. Now, let's consider just that. Inside , our specific context vector might have logit values corresponding to it like this: Diagram 10 : A single CV's routing logits in-place in the cuboid Visualised the same way, with k = 2 the equivalent part of would look like this: Diagram 11 : A single CV's top-k routing logits indices in-place in the cuboid That is, the function identified that index 2 in the original logits was the highest value, and 0 was the second-highest. Likewise, would have this: Diagram 12 : A single CV's top-k routing logits values in-place in the cuboid These diagrams are getting a bit unwieldy; let's look at this specific context vector's data as lists. For our selected context vector, we have these logits from diagram 10 : ...these top- k indices from diagram 11 : ...and these values from diagram 12 : Our tensor is just full of − ∞ s, and is the same shape as , so its corresponding part is this: What will do is select indices 2 and 0 (because of the values in ), and copy the corresponding numbers from on top of whatever is already there: It will do that for every one of the positions in that front view. So now we have what we wanted: a grid of routing logits for each context vector, where the non-top- k ones have been replaced by − ∞ . That's pretty nifty! And so here's the code to do it: I was initially a little worried that the whole "replace the logits with a 'static' tensor and then scatter the values in there" approach might break the computation graph, but tests showed that it didn't. My intuition is that because the − ∞ s were summoned out of nowhere, they are dead ends for backprop, but because the logits that we're copying in come from previous calculations, they are not. So once we've done that, we have our top- k logits, ready for a softmax -- so it's time to do that: ...and we have our weights, yet another cuboid like this: Diagram 13 : as two 2-D views Just as before, a given number on our front face is the first of the expert weights for a specific context vector, and each row in the side view is all of the expert weights for that context vector across all experts. Post-softmax, the weights for active experts for that context vector are positive numbers, and the weights for the inactive ones are zero. The next step was to actually feed the context vectors into their experts. I decided to keep this simple, and just iterate over the experts, one by one, and for each one to find which incoming context vectors wanted to go to it, run them all through, and then reassemble the results. I believe that larger MoE systems have smarter routing systems -- for example, for a huge model that can't fit on a single GPU you might have different experts on different GPUs or even machines, and route to them in parallel. But for my toy-sized models, this felt simplest, and felt like it would be efficient enough 5 . The way I decided to do this was to use an "accumulator" model. We know that the shape of the outputs is the same as the shape of the inputs -- that is, if we have our incoming shaped , then the output will have the same shape. So I started off by creating a tensor of zeros of that shape: So, just like in diagram 6 , it will look like this: Diagram 14 : as two 2-D views ...but it would be filled with zeros in every position. The plan was that each expert would be run on the appropriate context vectors, its outputs would be scaled by the weights that were calculated by the router and its associated top- k and softmax for that context vector/expert pair, and then the results could be added into . That would give us the result we wanted: after all experts had been run on their respective context vectors, would have the weighted sums. So the next step was to iterate over the experts: Now, we need to know which context vectors wanted to be fed to this expert. Let's take a look at our representation of again: Diagram 13 (repeated): as two 2-D views We can see it as a bunch of "slices", each the same shape as that front face, , where each one is the weights for a given expert. The front face itself (corresponding to the rightmost column on the side view) is the weights for expert 0, the next one "back" is for expert 1, and so on. And in code, we can get the slice for the expert with index like this: That will be a simple 2-D grid of numbers like this: Diagram 15 : sliced for one expert You can see that it's a grid of one number for each context vector in our input -- the same shape as the front face in all of the diagrams so far. Each number is the weight that the expert with index has for the context vector in question. We can now do a comparison: That will give us a new grid of the same shape, but the numbers have been replaced with booleans -- if the corresponding context vector has a weight for this expert that is greater than zero, otherwise. We've got a mask that identifies exactly the context vectors that we want to run through this expert. Let's imagine it looks like this for some particular set of context vectors and some specific expert; I've coloured in the circles representing and left the ones white. Diagram 16 : what might look like for one expert Now comes the clever part :-) Remember that our original incoming context vectors, in , looked like this: Diagram 6 (repeated): as two 2-D views We can use our mask -- the grid of s and s in diagram 16 -- to select a subset of the context vectors in there, like this: That will return us the subset of the incoming context vectors that we want to run through this expert. In terms of our diagrams above, it will be selecting the context vectors in the front face that are "selected" by our mask, and taking the "cores" as it goes back through the cuboid from there. The question is, what shape will it be? You can see that there's no simple 3-D shape it could be. The first sequence in our batch -- the first row on the front face -- has two selected context vectors, while the second has one, and the third three. You could imagine a world where the output would be the same shape as the input -- that is, a tensor the same shape as -- but with the non-selected numbers replaced with s or something like that. But instead, PyTorch produces what amounts to a list of the selected context vectors for this expert. We'll call the number of selected vectors , and it's six in the example diagram above, so the shape will be , like this: Diagram 17 : the "selected" context vectors Now, note that this is a big change from all of our tensors so far. It has lost the connection back to the original tensor's shape. In all of the other tensors, we could map something back to the original context vectors in . But with this new one, we have a bunch of context vectors with no inherent connection to where they originally came from in . We'll come back to that. But for now, we have the data that we want to run through the expert with index . The expert is an FFN -- our two linear layers with a GELU in between them -- and will treat all but the last dimension in whatever we feed it as "batch" dimensions, so it will work over that dimension, as we want it to. So the code to actually select the context vectors -- our above -- and then to run it through the expert itself just becomes this: We get the results of running our selected context vectors through the expert, still shaped . (If you're familiar with the GPT-2 code, that in the code might seem odd. We'll come back to why the expert is returning a tuple and why we are ignoring the second item in it later.) So we have our results -- we've run the appropriate context vectors for this expert through it. But they're in a slightly funny shape, and we're going to need to fix that later on. But first, we need to apply the appropriate weights for this expert to each result. Remember that is a grid of booleans, , like this: Diagram 16 (repeated): what might look like for one expert ...where -- filled in the diagram -- means that this expert is active for the corresponding context vector, and means it isn't. looks like this: Diagram 13 (repeated): as two 2-D views Now, previously we did this: ...to pluck out the context vectors from where the mask was true. If we were to do ...then we would be doing something similar. We'd get a grid like the one of context vectors in diagram 16 , with one row for every selected context vector for this expert, except that instead of each row containing a context vector, it would contain that context vector's per-expert weights: Diagram 18 : all expert weights for the "selected" context vectors Now, on its own, that's not particularly useful -- but you can hopefully see that column is the weights for our expert for all of the context vectors that are going to be run through it! So if we do the same masked lookup as before, but also select that column, we get code like this: What we're saying is "pick each item in that matches a in , then take the th element of it". It will look like this: Diagram 19 : this expert's weights for the "selected" context vectors That is exactly what we need to get the weights for our results! We need to multiply the i th element in -- an output context vector that is the results for a particular input context vector -- with the i th element of . However, there is one tensor-compatibility issue. What we have is shaped -- that is, it's essentially just a list of numbers. Now, we want to multiply our results by these weights, which means that we want to broadcast them across , which is shaped . Naively you might think that if you try to multiply a tensor by a one, PyTorch would match up the two dimensions of the same size and it would just work. But it would actually try to match up across dimensions from right to left, and would complain that does not equal . So instead, it's best to feed it an explicit tensor so that it knows which dimensions we're trying to match up, and which ones it should broadcast over: ...and that will give us of shape That would look identical to diagram 19 in the way I've been diagramming these things, but it's technically different and necessary. So now, with the having fixed the dimensionality so that we can do a broadcast, we can do this: ...and with that we have our weighted results for this expert -- all of the context vectors that should have been run through it have been, and they've been multiplied by the weights that the router gave for them, so we have our backprop channel for training the router. The next step is that we need to somehow put these results into . Specifically, because it's initially all zeros, and we want it to hold the results of adding together the results from each active expert for each context vector, we need to add them to whatever is already there at this point in the iteration through the experts. Even though -- as noted earlier -- we have, by this point in the calculations, lost the connection between the tensors we've been working with and the original "front-facing" grid of our original tensors like , where we could link something directly to the incoming context vector that it related to, it's actually surprisingly easy to patch that up :-) Remember that outside our loop through the experts, we created like this: So, just like in diagram 6 , it looked like this: Diagram 14 (repeated): as two 2-D views Now, previously we had code to extract the context vectors that we wanted to go through our expert from , and it looked like this: That gave us what amounted to a list of context vectors, like this: Diagram 17 (repeated): the "selected" context vectors Now, the interesting thing about a masked lookup into a tensor like is that not only can you do it to extract a subset of the values from the tensor like we did with -- you can also use it to assign to the masked subset of the elements in the tensor. To make that concrete: when we did We got a tensor sized . That means that if we were to do: ...then given that the mask is the same, and has the same shape as , then we'd also get a result. But the thing with assignment means that if we had some tensor shaped -- let's call it -- then we could do this: With that, instead of selecting the parts of and using them in the future, we're overwriting them with whatever is in . Furthermore, we can use the same trick with the augmented assignment operators like : ...means "select the elements from that have in , and then increment them by the elements of ". So finally, we get our code: That multiplies the results from the expert by the corresponding weights, and then adds them in to the running totals we're keeping in . Because is the same shape as , and thus is the same shape as , and we've kept that same shape as we went through the expert itself and multiplied by the weights, it will work. And with that, we've done all of the calculations that we need for this specific expert, so we can go back round the loop for the next one. When we've finished with all of the experts, we can return the result: Phew! That was quite a lot of explanation, but I think that what is going on in the code needs it. I originally wrote it in a kind of flow state of inspiration, and was very pleased with it, but it feels like now that I've explained it, I actually understand what my subconscious must have known while I was writing it. And I hope it's reasonably clear for anyone reading this. But there was one extra thing I wanted to keep track of before I started running this: the balance between the different experts. We'll come back to this in some detail later, but a problem with MoEs is that they can wind up depending heavily on specific experts, and ignoring the others -- the auxiliary loss I mentioned way back in the intro to this post is required to avoid that. I didn't want to implement that yet, but I wanted to log enough data to see if it really would be necessary with my setup. A good way to keep track of how much each expert is being used is to record the logits -- the original results from the router, before the top- k and the softmax -- and the actual post-top- k , post softmax expert weights that we actually used. The changes to do this are dotted around a bit, so I've linked to the appropriate lines on GitHub and if you have the screen real-estate to do so, I'd recommend that you use that to follow along. However, I've tried to put enough code inline in this post below that it should be comprehensible without that. I decided that I'd return the logits and the expert weights from the module's forward pass as an extra output here : Now, back in the we were setting to either a object or a depending on whether MoE was enabled for this model here : That meant that in our forward pass where we previously just did this (I won't link to this because it's an old version and having links to different versions would be confusing): ...then if the model was an MoE, we'd be getting a tuple -- the actual outputs from the module and then that extra routing information we were returning. But if it was a , we'd just get the outputs. I decided that the simplest fix was to change the module so that it returned values that were compatible with . The old , which was this: ...became this : That meant that we could do this in place of the code above: If we had an MoE in , then we'd get some real MoE routing info in , whereas if we were running a normal dense model then we'd get . This, by the way, explains the odd bit of code in the for that you might remember from earlier -- this bit : there is one of the modules, so what we're doing there with the underscore is just ignoring the that we know we're returning from it. The next step was to work out how to get the extra information that we had in out of the , if was a . Now, is used in an in the class here : That means that the outputs from the method of the first one are fed directly in as the inputs to the second one, and so on. Additionally, assumes that the forward method just takes a single input. What I really wanted at the end of this was a list of pairs, one for each Transformers layer. So I decided to pass an accumulating list into (allowing it to be ), like this : ...and then append whatever routing info came from the (potentially ) FFN to that here : ...and then return it from for the next layer here : Then, in I wanted to return it to whatever called the model for inference. I didn't want to change the implicit API that I was using -- pass in inputs, get next-token logits -- so I decided to just attach it to the like this : I'm not sure, in retrospect, that "smuggling" the routing info out like that was the right choice. Perhaps a Hugging Face-like model where the LLM returns some kind of "output" class that contains and other stuff like this as explicit fields would be better. But this setup seemed to work for my use case, so I've left it as-is for now, with a mental note to revisit later. Next, I modified my training script to store the that we got when we did our forward pass in a list, and then to store that in the metadata associated with my checkpoints for later analysis. The training script I'm using is something I developed after working through Raschka's book, and it's kind of complicated -- although the core is essentially the same as the one we use to train our model in chapter 5, I've built it out to allow training across multiple GPUs , with all kinds of tweaks that I've learned about since. So I won't dig into that code in any depth in this post; if you've been following along with my various training posts in the past, though, and want to see how this all fits in, you can see the code that accumulates the routing information here (note that it uses so that we don't accumulate compute graph information over time), the code that passes those details into the checkpointing function here , and the updated checkpointing function here . So: with those changes, I had a plausible-looking MoE setup. I was confident that it would be able to train, but suspected that it would not be able to balance load between the experts well and would collapse to using a subset of them. It was time to give it a go. The first question was, what total number of experts, and how many active ones, should I have? I was pretty limited by what I could fit into my RTX 3090's VRAM, and what an appropriate training speed might be, but after a bit of fiddling around I came to the conclusion that six experts with two active would fit into memory and train reasonably quickly, and that didn't sound like a crazy balance. Mixtral is 8 experts, two active, for example. I wouldn't be able to fit in the batch size of 6 that I had used in the past when training 163M-parameter dense models; the largest batch size I could fit in was 3. (For those who've been following my previous LLM training, because I'm using gradient accumulation over 16 steps with the microbatch size of 6 -- for a global batch size of 96 -- I could get the same effect, and fit the MoE into VRAM, by going down to a microbatch size of 3, with 32 gradient accumulation steps.) The next question was how many tokens to train for. As an experiment I kicked it off with the same 3.2 billion tokens that I would normally use to train my 163M-parameter models. I knew that this larger model would need to be trained on more tokens than that, but I just wanted a reasonably serious run to see how the model behaved. The training script predicted that it would take just over two days to complete this experimental run, which was perfect. I had reached this point during the late afternoon on a Friday, and had stuff to do over the weekend that would keep me away from my computer, so that run length would mean that it would be ready for me on Monday. On Monday, I came back to see that it had completed properly: Over the training run, the loss declined nicely: Out of interest, I ran an evaluation script to see how it performed on the held-back test set that I normally use to compare my models: That ranked it better than the 163M-parameter models I'd trained using PyTorch, but worse than the original GPT-2 small, and also worse than the models I've trained with JAX (which I believe got lucky with their initial untrained random weights). That was a pretty solid result, but as this was an exploratory training run, there's no point in putting much weight on it. For a start, this model was clearly undertrained. The number of tokens, 3.2B, was Chinchilla-optimal for my 163M-parameter models, but this one had more than twice the total parameters at 446M, and had 220M active for each token. Larger models need more tokens to be well-trained. The more important result was the load-balancing between experts. I put together a notebook to ingest the metrics that I was writing to the checkpoints, and to create two charts for each layer: In the charts, for each global step, I plotted a line for each expert; if it had an average probability less than uniform / 3 -- that is, it was getting less than a third of the tokens that a perfectly flat distribution across all experts would give it -- then the line was white, meaning that it was being "starved". If it was getting a number of tokens between uniform / 3 and uniform * 2 -- then the line was green, because it was getting what felt like a reasonable number of tokens. And finally, if it was higher than uniform * 2, it was plotted in red: it was being overfed. The charts showed exactly the problem I was expecting to see. Let's look at layer 0: You can see that it started off pretty well-balanced, but rapidly came to prioritise expert 5; the remainder of the probability distribution looks like it must have been scattered over the other experts, with a slight preference for expert 3. The other layers came back with similar issues: certain experts were strongly preferred while others were starved: So the problem was real. The model was relying heavily on some experts and ignoring others. We needed to do extra stuff to balance the load across the experts. The problem with load-balancing across experts has been clear for quite some time, and it was covered in "Outrageously Large Neural Networks" back in 2017. The solution they used is simple, and clever: define an "auxiliary" loss, which goes up as the experts become more unbalanced, and down as they become more balanced. You can then add that (scaled by some amount) on to the normal cross entropy loss that you get by comparing the outputs from your model to the training targets, to get a combined loss number -- and then you back-propagate using that combined loss. By reducing the combined loss, then you optimise both for correctness -- is the model doing its job as an LLM -- and for balance across the experts. Of course, you need to be careful about the scaling factor that you use to adjust the auxiliary loss before adding it. If it's too small, then the load-balancing will be de-prioritised and you can wind up with imbalanced experts anyway. But if it's too large, the training process will prioritise balance over quality of the output, and you'll wind up with a crappy model that balances load across the experts beautifully. We'll come back to that shortly. The "Outrageously Large Neural Networks" paper's particular calculations for the auxiliary loss are a little complicated -- they generate two separate numbers and combine them -- and it was simplified in the GShard paper, and then again in Switch Transformers. I found the last of those reasonably easy to understand, and decided to adapt it. Let's start off with their formalisation -- but keep in mind that they had only one active expert per context vector, so we'll need to make some changes. Firstly, they define a per-expert number, f i -- the i subscript means that it's for expert i . That's a little intimidating-looking but is much simpler than it looks. They define it as "the fraction of tokens dispatched to expert i ", and in that light we can interpret it. The x ∈ ℬ that we're iterating over in the ∑ is basically: So, the stuff inside the ∑ is done once for each of our input context vectors across all sequences in a batch. The 1 { . . . } (which should be rendering with a kind of doubled-1 -- Chrome, as of this writing, doesn't handle that, though Firefox does) is meant to mean "1 if the condition in the braces is true, 0 if it's false". In argmax  p ( x ) = i , the function p is the "raw" probabilities from our router. As I mentioned earlier, they were doing softmax and then zeroing out the non-top- k values, so they had those raw numbers available. We'll come back to that in a moment. But for now, the condition inside the 1 { . . . } just means "true if this expert was the top one for this particular incoming context vector, false otherwise". Putting that all together, then we're just counting the number of tokens in our batch for which expert i is the top pick of the routing network. The 1 / T at the start then divides that by the number of tokens in the batch (which is what they mean by T ), and for a network with one active expert like the Switch Transformers ones, we've got -- as they say -- "the fraction of tokens [in this batch] dispatched to expert i ". So that works nicely in the one-active Switch Transformers world. But we can fairly simply extend it to handle multiple active experts, while keeping similar semantics. Let's say that we calculate how many of the context vectors in a batch were sent to expert i ; we can divide that by the number of tokens to get something equivalent. It still means "the fraction of tokens dispatched to expert i ". (Ab)using their notation, if we say that our number of active experts is k , then it might look something like this: Let's run with that for now. As well as f i , they define another per-expert number, P i : They describe this as "the fraction of the router probability allocated for expert i ". Again, we're iterating over every context vector in every sequence in the batch -- but here we're just adding together all of the raw probabilities for the expert for each one, then dividing by the number of elements in the batch. In other words, we're working out the expert's average probability across the entire batch. Much easier! And in our multiple-active-expert world, their equation works -- it means exactly the same thing. Now, both of these calculations, as they're expressed in the maths, depend on something that we're not currently calculating. Remember, Switch Transformers worked out the weights for each expert by doing a softmax across all of the raw logits that came out of the router's linear layer, then zeroing out the ones that were not in the top- k -- that is, for their k = 1 setup, all but the highest (argmax) one. So they had those "raw" softmaxed probabilities knocking around. But because we are replacing the non-top- k logits with − ∞ and then doing softmax, our router weights aren't the same -- they're just the relative probabilities of our selected k experts. For f i that doesn't matter; we can use our weights and easily identify the experts that were routed to for a given element in the batch, because they're the only ones that are not zero. But for P i we do need the pre-top- k 's logits from the router. And, not entirely coincidentally, we already have the code to make that available! In order to do those charts above -- the ones showing where experts were being starved and when they were being over-fed in the non-load-balanced training run -- we passed the raw, non-top- k 'ed, non softmaxed router logits, and the expert weights after the top- k and the softmax, out of the module's : ...and then added code to feed that through to the training loop because it was needed for the metrics that we used to generate those charts. The numbers that we kept for the "raw" side of things were the logits rather than the actual probabilities, but we can fix that with a simple softmax. So that means that in our training code, we already had the numbers to work out P i and f i for our experts. They needed to be combined to make up a single scalar auxiliary loss for the model when run over a batch, and Switch Transformers does that like this: If you imagine f as a vector containing all of the per-expert f i s, and P as a similar one containing the P i s, then that ∑ is a simple dot product, f · P . They then scale that up by the number of experts N , multiply by a scaling factor -- the one I mentioned earlier to balance load-balancing auxiliary loss against "real" cross entropy loss -- which they call α . (We'll come back to α later.) So, we have a set of calculations to work out an auxiliary loss; there are a few extra things I'd like to highlight before we dive in to the code. Let's imagine what a "perfect" router would look like from the perspective of these calculations, firstly for the Switch-style one-active-expert model, and then for our own more general case. Starting with f i ; it's the number of tokens in the batch for which expert i is the chosen one, divided by the number of tokens in the batch. We want all of our experts to get the same amount of "traffic", so for N experts, one active per token, clearly each one will be active 1 / N of the time. So that should be the value of f i if everything is balanced. Now let's think about P i . Again, we want each expert to receive 1 / N of the tokens -- so its average probability should also be 1 / N . So, that means that for perfect balance, all of our P i s and all of our f i s should be 1 / N . That means that when we do the ∑ in that loss calculation to work out the dot product, then for a perfectly balanced router, each one will contribute: There will be N of them, so that will come to ...and then we're multiplying by N to get the loss, so the result (disregarding the scaling factor α ) will be 1 . So, when balance across the experts is perfect, the Switch Transformers auxiliary loss has a value of 1 . However, they have only one active expert, and our equation is slightly different to allow for the fact that we have k of them. I won't go through the boring derivation again, but if we replay the maths above with k active experts, we get an auxiliary loss for the ideal, perfectly balanced router of k . That's not a problem in and of itself -- after all, this is just a number we're trying to minimise, and it's not super-important what we're trying to minimise it to. But it does matter when we're talking about α , because if the auxiliary loss is larger, we'll need to scale it down more to stop it from "drowning out" the signal from the actual training loss. The Switch Transformers paper explains what values they used for the scaling, but ours will be different because of the different number of active experts. The auxiliary loss calculation above only covers what happens in one layer, so we need to work out how to combine the contributions from all of the layers. In the paper, the only mention they make of multiple layers in this context is: For each Switch layer, this auxiliary loss is added to the total model loss during training I took that to mean that we just sum up the scaled auxiliary loss across all layers and then add that sum to the normal cross entropy loss. I think that's the most natural interpretation of what they are saying. So -- given the value of k for a perfectly-balanced router that I worked out above -- for the model I was planning to train, with 2 active experts, 12 layers, the auxiliary loss before scaling by α would be 24. This, by the way, is where my implementation differs from Mixtral's -- or, at least, the Hugging Face source code for it as of this writing. In that, they do something that feels a bit odd. They treat (for example) expert 1 on layer 1 as being the same as expert 1 on layer 2, and so on throughout the layers, then do the calculations just once. That feels a bit dodgy. After all, you can imagine that expert 1 on layer 1 might be being starved but its equivalent on layer 2 might be getting overfed, and the two would balance out. I don't know if that's an error in that implementation, or if there's something I'm missing. Conceivably I might test it some day by training another model using their loss function, but I suspect I won't get around to that. There's also something that made me hesitate a bit in the Switch Transformers paper, and which I think I still need to ponder a bit. That calculation for f i : ...did not look differentiable to me. Things like argmax (and our own equivalent's is-in-top-k ) are generally not. Indeed, they confirmed that shortly after defining it, but in a way that gave me pause: The objective can also be differentiated as the P -vector is differentiable, but the f -vector is not. The "objective" they're referring to is the auxiliary loss, and it makes me a bit uncomfortable that something that is defined in terms of A and B is differentiable if A is, but B isn't. My hand-wavy way of thinking about it right now is that the undifferentiable bit can be treated as a constant, so as long as part of the calculation is differentiable, the whole thing can be treated as such. But I'm not 100% happy with that, and need to think further. But now, I think, we've covered the maths for the auxiliary loss, so it's time to dive into the code! My old training loop had the following code to do the forward then the backward pass: Let's strip out all of the extra enhancements that I have accumulated there on top of the simple training code from the book; there's AMP , DDP and gradient accumulation and if we remove that it would simply look like this: Hopefully that's familiar! What I wanted to do was add in the auxiliary loss (if we were training an MoE), scaled by that α scaling factor. What I came up with (and again, here I've stripped out all of the stuff required by those enhancements): Note that I wanted to keep track of the router losses in that list as well in order to monitor them as the training run progressed, just like I normally monitor training loss (as you can see in the loss chart above). That's all pretty nice and simple -- if we are getting MoE routing info back from the model, then we call this new to work out the loss from the maths in the last section, and then add it on, scaled by , which is what I decided to call the somewhat-opaquely-named α from the Switch Transformers paper. Adding all of the AMP, DDP and gradient accumulation gubbins back in, the final code looked like this : So that was simple enough (for LLM-training values of simple). The code to route the from the training configuration file is not really worth going through, and nor is the code to save average, minimum and maximum values from into the checkpoint metadata, or to chart those (though I'll show the charts later). The interesting bit is, of course, that function. It's here and looks like this: Let's look at it from the outside in. We start with a list called . Remember, this has been passed back from the MoE model for a single forward pass of a batch. It contains one item for each Transformers layer in the model, and those items are pairs of . We start off with a total routing loss of zero, and then for each layer, we do some stuff to work out its routing loss, and then at the end of the loop, we add it on to our running total. Finally, we return the total. Obviously, the fun stuff is inside the loop :-) It's time for some of those tensor diagrams again. Let's remind ourselves of the shape of : Diagram 13 (repeated): as two 2-D views Each cell on the front face relates to one context vector in one sequence in our batch, and the "core" going into the cuboid from there (horizontally along the side view) is the weights we actually used when routing the context vector in question to the experts -- zero for unselected experts, some value between zero and one for the selected ones. Now let's go back to the code for a moment. We start off by using the shape of to work out what our different dimensions are: Then we do our first block of calculations, trying to work out f i from the maths, "the fraction of tokens dispatched to expert i " We want to do this efficiently with a vector calculation, working out all of the f i s (remember, there's one for each expert) in parallel. Our first step is to flatten out the grid into a single dimension -- essentially stacking all of the front view's columns on top of each other to make just one long column, like this: Diagram 20 : flattened, as two 2-D views ...or, more simply, as it's now just a 2-D Tensor, like this: Diagram 21 : flattened, as one 2-D view We can use PyTorch's method to do that without having to copy any data around in memory -- as the name suggests, it just returns a different view on the same data: Now, remember that these are the expert weights. Each one of those cells contains a number -- zero if the expert was not selected for the context vector that corresponded to it in the original layout, or some non-zero number if it was. So if we do this: ...then we'll get a tensor of the same shape, where we have if the weight was more than zero (that is, the expert was active for that context vector), otherwise. And that means that if we sum down those columns in diagram 21 , we'll get a new grid of one row, and columns, which represents the total number of times each expert was active in the given batch: Diagram 22 : as one 2-D view So, in code, we can just do this: If we divide that by the number of context vectors in the batch, we've got a new tensor, a row with columns -- the same shape as diagram 22 -- containing exactly what we want: That is, is a vector f in terms of the maths, containing all of the f i s that we want for our auxiliary loss calculation -- that is, all of the result across all experts for this: The calculations for the P i s are very similar. We start off with like this: Diagram 7 (repeated): as two 2-D views So, each cell on the front face relates to one context vector, and the "core" going into the cuboid from there is the set of raw routing logits for that context vector, one number per expert. Firstly we need to convert the into probabilities by running them through softmax, along the last dimension -- the one that is the horizontal axis on the side view: Then we do the same trick with to convert the result to a single column of lists of length (metaphorically) -- the same shape as in diagram 21 : Now if we add them up across the rows like this: Then we get a single row, column result that has, for each expert, the sum of all of the probabilities it had across all of the context vectors in the batch, just like the one we had in diagram 22 . We can divide that by the number of context vectors in the batch: ...and that's our vector P containing all of the P i s, where each is one of these: Finally, we can multiply all of the f i s and their corresponding P i s by each other, and sum the results, by using a dot product, and then multiply the result by the number of experts: ...and that's this bit done (apart from the α ): And that's our auxiliary code wrapped up! We've been through , and we've already seen the code that scaled it by α aka , so we have a training script with MoE auxiliary loss using the maths in the last section! Again, I hope that the diagrams helped with that workthrough. The code is the kind of thing where it's easy to scan through and get a vague understanding, but I think that visualising what's going on step-by-step is important if you want it to really stick. With the code in place, the next thing to do was to explore what the right value might be for α . In the "Switch Transformers" paper, they say: Finally, a hyper-parameter α is a multiplicative coefficient for these auxiliary losses; throughout this work we use an α = 10 − 2 which was sufficiently large to ensure load balancing while small enough to not to overwhelm the primary cross-entropy objective. We swept hyper-parameter ranges of α from 10 − 1 to 10 − 5 in powers of 10 and found 10 − 2 balanced load quickly without interfering with training loss. But, as I noted earlier, that worked for them with their single active experts, but because I had multiple, my auxiliary loss would be larger. Now, given that their "ideal" per-layer loss was 1, and mine was 2 for my planned training run, it sounded like using half of their recommended value, 5 × 10 − 3 , would be appropriate. However, I was also a little concerned about the number of layers interfering with things. The auxiliary loss, as we saw above, was a single value per layer, all of which were added together. With my "ideal" per-layer loss of 2, and 12 layers, that meant that the ideal across all layers was 24. Now, they mentioned a specific value for α , but didn't mention the number of layers they had, or if they swept for different possibilities across different numbers of layers. That seemed strange! I decided to work on the hypothesis that they had found that as the number of layers increased, you needed the total contribution of the auxiliary loss to scale up in proportion. That was only a guess based on trying to fit together the info in the paper, though, and could well be wrong. I was in enough doubt, though, that I felt it would be wise to do my own, minimal sweep over a few values for α . For each, I'd do a one-hour training run. At the end I'd check the training loss -- that is, the pure cross entropy loss saying how well the model was doing at its real purpose of modeling language -- and the auxiliary loss, showing how well-balanced its usage of experts was. I got these results: I also generated router usage maps like the ones I gave way back in this post for the two-day training run with no auxiliary loss. I won't put them all in there, but: So, on the basis of those results (and, of course, the fact that it fit well with the Switch Transformers paper's recommendation), I decided to use α = 0.005 . It was time to train this thing! I decided not to think too hard about the right number of tokens to train it on. The Chinchilla number, 20 times as many tokens as parameters, is a heuristic that works for dense models, but is not meant for MoEs. You'd intuitively think that the "ideal" number of tokens to train a model with 446,410,752 parameters, 219,697,152 active per token would be somewhere between the Chinchilla-optimal numbers for those two parameter counts. But overtraining isn't necessarily a bad thing, so long as it's on non-duplicated data (or less than four epochs over the same data ), and I had a 10B token dataset all set up from my previous experiment. So training it on what would be the Chinchilla-optimal number of tokens if it was just a 446,410,752-parameter dense model didn't sound like a bad idea, so long as I could do it in a reasonable amount of time. That meant 8,928,215,040 tokens, which I had in my normal training dataset without needing multiple epochs. A quick check -- running the training script with that number of tokens configured -- told me that it would take 90,823 global steps to complete, over about eight days. It looked like each checkpoint would take up 5.3GiB, and I had about 348 GiB free on my disk, so I calculated that I could checkpoint every 1,500 global steps -- that would work out as roughly once every three hours. Not great, but losing a maximum of three hours work in the case of a power outage (or cat jumping onto the PC's power button) is not the end of the world. So I set things up with this model configuration and this training configuration , and on my dedicated training box, , I kicked it off: Just less than eight days later, it completed: The loss chart looked like this: You can see that the normal cross entropy loss (just tagged as "loss" on the chart) decreases nice and smoothly from random (about 10.82 with the GPT-2 tokeniser) down to that final training loss of 3.211, with only a couple of tiny spikes. I've also plotted the auxiliary loss for the MoE routing, on the right-hand Y axis, and you can see that while near the start there were a few bumps (and the max values for single iterations spiked up from time to time), the average was generally pretty close to 24, our "ideal" number. Indeed, for the last checkpoint, the average over all iterations was an almost-perfect 24.0847. The notebook that I had to plot layer-by-layer expert starving/overfeeding came back with some lovely green plots showing nice even routing, too: A couple of issues near the start of the run, but for the last 75% of it, they're all a sea of green. Lovely. I ran my normal smoke test , based on Raschka's from the book: what do you get if you ask your model to complete "Every effort moves you" with 20 tokens, with a temperature of 1? Reasonably coherent! It was time to do some evals and comparisons. I ran my normal loss eval : on a held-back test set of 19,200 sequences of 1,024 tokens each, what was the model's average loss? Comparing this against other models that I've trained, and the OpenAI GPT-2 small and medium weights, we get this (it's in bold): Not too bad, though not amazing. It was better than any of my 163M-parameter models, and OpenAI's GPT-2 small. But it was a bit worse than (but close to) the 345M-parameter OpenAI GPT-2 medium, which has fewer parameters -- albeit more active ones per token. I decided to do a second eval. I have one that I call the IFT test -- fine-tune the model on an instruction-following dataset, until loss starts rising on its held-back eval dataset, then run a test set through to get answers to questions the model has not yet seen. I then bundle together the responses from a bunch of models and ask GPT 5.5 to compare them. It's an extension of Raschka's example in chapter 7 of the book, modified to make it easier to compare different models. You can see the scripts here and here , and there are more details here . I kicked off the first script, to fine-tune the model and get its responses, and then handed that plus a bunch of responses from other models to the LLM for it to compare them. The results came back like this (note this this is sorted by loss, the "IFT rank" column is how well it did comparitively in the eval): As you can see, the correlation between loss and performance on this eval is interestingly loose -- I have an ongoing series trying to work out why that might be . In particular, OpenAI's weights consistently outperform mine, and I'm determined to find out why. But it was reassuring, at least, that the new, big model came in at rank 3, beating all of my other ones, even if it still lost to that pesky 124M-parameter OpenAI GPT-2-small. So, there we have it: a GPT-2 small model converted to an MoE with 6 experts per layer, 2 active per token. Its loss on the test set is pretty much where you'd expect, and its IFT eval makes sense, modulo the OpenAI weights weirdness. What does that mean, and what should come next? In this post, I started with the GPT-2 code from " Build a Large Language Model (from Scratch) ", and my own training script (which was originally based on the training code from the book), added on mixture of experts support including the auxiliary loss calculations that you need to make it balance load across its experts properly. After eight days of training, we wound up with a decent, capable model. So that's all quite satisfying in an intellectual sense, and -- at least in terms of how well it did on the test loss -- it landed pretty much where you might expect. And one thing I'm sure of is that grinding through the calculations has been a great work-out for my skills with PyTorch and tensor operations. But the interesting thing about MoEs is that they -- in theory -- provide similar performance to dense models, at a lower cost in inference computing time. I think there are some interesting follow-up experiments I can do. OpenAI's weights tend to beat mine, so if I exclude them from comparisons (until I've worked out why), I could try to build a mental model for whether MoEs are a good way to spend my scarce computing resources when learning more about LLMs. I could: I'm sure there are other options, and I'd love to hear people's thoughts on what they might be. Anyway, I hope this post has been an interesting journey, and explained things well. Any feedback much appreciated -- in particular, on whether the diagrams helped. I only recently added D2 support to my static site generator, and it's entirely possible that I was overusing my new toy :-) So: thanks for reading, and as always, comments and questions are very welcome in the comments below. To protect against linkrot, I'll put proper references to these papers, because they're important background. In terms of the implementation, Mixtral is a bit odd. It does the Switch Transformers trick of doing the softmax, then zeroing out the non-top- k values -- but then it scales up the resulting routing weights by taking the sum, then dividing each value by that sum (which will be less than one). But the net effect of doing that is exactly the same as the "Outrageously Large Neural Networks" technique of replacing non-top- k with − ∞ .  ↩ One other thing: the masking out of all but the top- k experts does make me feel a little suspicious. It feels in a hand-waving kind of way a bit like "dead-end" code in router that we had originally, where we didn't use the outputs to weight the sum at the end. And it looks like there is a real mathematical concern there (even if it's not quite the same); in the "Outrageously Large Neural Networks" paper they say: While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice My intuition for this is that because we have the weights in the flow of the calculations, back-propagation can still try to (say) reduce the weight for an expert that should not have been used for a particular forward pass. That is a bit problematic, though, because (due to the softmax) you would expect that reducing one expert's weight would force the others' to increase -- the results of a softmax have to sum to one. On consideration the Switch Transformers softmax-then-mask system feels like a better solution to me, because although the weights for the unselected experts are "invisible" to the backprop, having been masked out, gradients that increase a selected expert's weight will kind of automatically decrease those of all of the other, non-selected ones -- and vice versa. I need to ponder this a bit more, and perhaps try another training run with the alternative Switch-style setup and see how it compares. One random idea -- the softmax-first approach means that the weight of the selected expert(s) will be lower than it would be otherwise, so that reduces the amount of signal that the FFN adds to the context vectors. But then maybe the FFN would just get trained to emit larger outputs...? But anyway, enough waffling: let's put that aside for now, and for the rest of this post I'll describe the code as I wrote it, using the "Outrageously Large Neural Networks"/Mixtral model for the router.  ↩ Here's one that I tried, and which worked, but had a non-obvious bug. Let's imagine we have these logits for one context vector: For two active experts, we want to replace it with this: By default, sorts the values in decreasing order (and keeps the indices in an appropriate order to match). So in this case, we'd have the top-2 values looking like this: So if you do something like ...then you mask out all logits that are less than the lowest value in the top-k list with − ∞ . Neat! The problem with that was that there could be a tie. Imagine if, for one context vector, instead of the values above, we had these logits: We want to select the top 2 experts, and the smallest top-2 value would be 0.4254, of course. But if we just replace all values less than 0.4254 with − ∞ then we get this: We have three active experts rather than two! That's not good. While I didn't think that hitting this problem would be all that likely in practice, it was a definite error, and needed addressing -- hence the solution I settled on.  ↩ Imagine an extreme case, 256 experts but one active per token. In a given batch, you might expect tokens to be scattered pretty much randomly across experts, so you'd need to run ~all of them. If each expert only uses a small amount of your GPU's power to run through the small subset of the tokens that are allocated to it, and you're processing one expert at a time, you'll seriously underutilise the GPU. But I figured -- and confirmed when I finally got all of this running -- that with the sizes of model and numbers of active/total experts I was using, this wasn't a problem. Another issue that might come up in really large models is that too much stuff in a batch might wind up going through the same experts. Although we will later on add code to make sure that on average each expert receives roughly the same amount of context vectors, within a given batch things might be imbalanced. To see why that might be a problem, imagine an LLM that is so large that you have different GPUs -- or even different machines -- handling different experts. Something in the trillions of parameters scale, for example. If in your batch, expert i winds up doing most of the context vectors, then the GPU it's on is a bottleneck. The setup I'm describing in this post is sometimes called "token choice" routing. In a sense, each token has chosen -- or, rather, the router has chosen on its behalf -- which experts it "wants" to go to. "Expert choice" routing inverts that; you generate the same logits, but instead of choosing the top- k experts for each token, you choose the top- k tokens for each expert, so that you guarantee balance across the experts. Both of these are something for future investigations, I think :-)  ↩ " Adaptive Mixtures of Local Experts ", the 1991 paper that introduced the name (but used a model where all of the experts were run for each input -- they were thinking less in terms of efficiency, and more in terms of having different parts of a network specialise in different things). " Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer ", from 2017, which showed that you could use gating on MoE models to reduce the amount of computation by only running the "best" of the experts for each input. " GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding ", from 2020, which built on the above, applied it specifically to Transformers models, and simplified some things -- more on that later. " Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity ", from 2021, which did some nice additional simplification. One showing how well-balanced the logits coming out of the router were -- that is, for each global step, how close was the model to having every expert getting exactly the same amount of utilisation in terms of the router's raw numbers. One showing how well-balanced things were after the top-k. This was a belt-and-braces thing: you can imagine a router that always favours a particular pair of experts, but only by a small amount, so the raw routing balance might look reasonably good (say, the two favoured ones get a probability of 0.2 and the others get 0.15 each), but the favoured two experts would wind up getting all of the "traffic". With α = 0 , as you'd expect to see from the very high auxiliary loss, things started going red and white pretty quickly -- it was focusing on specific experts and ignoring others. It didn't do too badly on training loss, though. With α = 0.001 , one tenth of the Switch Transformers number (and one fifth of what you'd expect to be ideal for ours by converting it naively), the load-balancing charts weren't quite so bad, but there was a speckling of red. I couldn't say for sure that it might not have managed to keep things reasonably balanced in a longer training run. α = 0.005 , however, looked really good! It actually had the lowest (best) training loss out of any of these options, and the auxiliary loss was not too bad, only 1.2 above the ideal value of 24. By the time we got to the α = 0.01 that was used by Switch Transformers (with their ideal auxiliary loss values that were half of ours), although the auxiliary loss continued downwards, training loss started worsening. This was certainly in line with what I would expect to see if the training started prioritising balance over actually creating a good LLM. Train a small model (say, the 163M-parameter size I've been messing with so far) on the same amount of compute as I spent on the MoE. How would it perform? My guess is that it would be worse. Train a dense 446M-parameter model on the same amount of compute and see how it matches up. It would be undertrained (it would need more calculations per token, so if I matched the compute, I'd have to train it on fewer tokens). I don't know if it would be better or worse. Train a Chinchilla-optimal dense model on the same amount of compute, and see how that did. To protect against linkrot, I'll put proper references to these papers, because they're important background. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation , 3(1), 79–87 Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR 2017) . Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., & Chen, Z. (2021). GShard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations (ICLR 2021) . Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120), 1–39 . In terms of the implementation, Mixtral is a bit odd. It does the Switch Transformers trick of doing the softmax, then zeroing out the non-top- k values -- but then it scales up the resulting routing weights by taking the sum, then dividing each value by that sum (which will be less than one). But the net effect of doing that is exactly the same as the "Outrageously Large Neural Networks" technique of replacing non-top- k with − ∞ .  ↩ One other thing: the masking out of all but the top- k experts does make me feel a little suspicious. It feels in a hand-waving kind of way a bit like "dead-end" code in router that we had originally, where we didn't use the outputs to weight the sum at the end. And it looks like there is a real mathematical concern there (even if it's not quite the same); in the "Outrageously Large Neural Networks" paper they say: While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice My intuition for this is that because we have the weights in the flow of the calculations, back-propagation can still try to (say) reduce the weight for an expert that should not have been used for a particular forward pass. That is a bit problematic, though, because (due to the softmax) you would expect that reducing one expert's weight would force the others' to increase -- the results of a softmax have to sum to one. On consideration the Switch Transformers softmax-then-mask system feels like a better solution to me, because although the weights for the unselected experts are "invisible" to the backprop, having been masked out, gradients that increase a selected expert's weight will kind of automatically decrease those of all of the other, non-selected ones -- and vice versa. I need to ponder this a bit more, and perhaps try another training run with the alternative Switch-style setup and see how it compares. One random idea -- the softmax-first approach means that the weight of the selected expert(s) will be lower than it would be otherwise, so that reduces the amount of signal that the FFN adds to the context vectors. But then maybe the FFN would just get trained to emit larger outputs...? But anyway, enough waffling: let's put that aside for now, and for the rest of this post I'll describe the code as I wrote it, using the "Outrageously Large Neural Networks"/Mixtral model for the router.  ↩ Here's one that I tried, and which worked, but had a non-obvious bug. Let's imagine we have these logits for one context vector: For two active experts, we want to replace it with this: By default, sorts the values in decreasing order (and keeps the indices in an appropriate order to match). So in this case, we'd have the top-2 values looking like this: So if you do something like ...then you mask out all logits that are less than the lowest value in the top-k list with − ∞ . Neat! The problem with that was that there could be a tie. Imagine if, for one context vector, instead of the values above, we had these logits: We want to select the top 2 experts, and the smallest top-2 value would be 0.4254, of course. But if we just replace all values less than 0.4254 with − ∞ then we get this: We have three active experts rather than two! That's not good. While I didn't think that hitting this problem would be all that likely in practice, it was a definite error, and needed addressing -- hence the solution I settled on.  ↩ Imagine an extreme case, 256 experts but one active per token. In a given batch, you might expect tokens to be scattered pretty much randomly across experts, so you'd need to run ~all of them. If each expert only uses a small amount of your GPU's power to run through the small subset of the tokens that are allocated to it, and you're processing one expert at a time, you'll seriously underutilise the GPU. But I figured -- and confirmed when I finally got all of this running -- that with the sizes of model and numbers of active/total experts I was using, this wasn't a problem. Another issue that might come up in really large models is that too much stuff in a batch might wind up going through the same experts. Although we will later on add code to make sure that on average each expert receives roughly the same amount of context vectors, within a given batch things might be imbalanced. To see why that might be a problem, imagine an LLM that is so large that you have different GPUs -- or even different machines -- handling different experts. Something in the trillions of parameters scale, for example. If in your batch, expert i winds up doing most of the context vectors, then the GPU it's on is a bottleneck. The setup I'm describing in this post is sometimes called "token choice" routing. In a sense, each token has chosen -- or, rather, the router has chosen on its behalf -- which experts it "wants" to go to. "Expert choice" routing inverts that; you generate the same logits, but instead of choosing the top- k experts for each token, you choose the top- k tokens for each expert, so that you guarantee balance across the experts. Both of these are something for future investigations, I think :-)  ↩

AI

▲ 0

0 views

Nick Khami 6 days ago

The Inference Engineering Skills Map

I made the switch from B2B SaaS web development to inference about a month ago and wanted to share my learnings for others that might be interested in doing a similar thing. After all, inference may very well be the last large market we ever see, spending on ai is now larger than global investment in oil & gas . This post is titled "inference engineering " instead of "inference science " for a reason. From an engineering perspective, I feel like inference is similar to the B2B SaaS world I'm coming from where you are mostly stitching together APIs. Very little net-new code is required. Really, you just have to get three categories of services running and tuned: Frontier LLMs like fable, kimi, and astra still suck at this so, unfortunately, you do have to understand them to a reasonable extent if you want to get things right. It's simple tho, if you were previously able to learn docker or react then you can certainly learn this. Similar to replicas for a REST API on EC2 instances, you will have replicas of a model running on GPU servers when doing inference. Each replica is known as a "worker". Naively, you would assume that sending requests evenly across all workers is the right approach, but unlike REST APIs, inference requests vary in uncached token length and therefore require more intelligent load balancing. There are two open services out there that provide primitives for doing this coordination: Nvidia Dynamo and llm-d . Dynamo is newer and more recommended, but many older stacks are still reliant on llm-d. The fastest way to learn them is to deploy them in production with whatever GPUs you can get your hands on. Being familiar with this part of the stack will allow you to build intuition on load balancing, queues, and capacity ratings. I could go into several paragraphs more of detail, but I'll reserve that for a 202 post. Shoutout to @KranenKyle , @flowpow123 , @athreesh , and @0xisand for their work on Dynamo, and @smarterclayton and @robertshaw21 on llm-d, alongside both projects' contributors. Once Dynamo or llm-d sends a request to a worker, that worker needs to actually run the model. There are two main inference engines for this: vLLM and SGLang . These services load model weights onto GPUs and turn your input tokens into output tokens. Part of this is batching . The engines group work from multiple requests together to keep the GPU busy. Larger batches generally improve total throughput, but can mean fewer tokens per second for each individual session. Beyond batching, modern models are large and can also create memory constraints. Model size not only impacts how much GPU mem you need to have to load the model, it also impacts how much you need to hold state during inference. These engines have built in tools for tensor parallelism which lets you split the model across multiple GPUs so they can share memory and computation. Final note, you will also hear people talk about "prefill" and "decode". This is another thing handled by the engine libraries. Prefill is processing the prompt (TTFT), decode is generating the answer (TPS). You can run these processes on the same workers or separate them onto different workers, which lets you tune their capacity independently but requires moving state between them . Again, similar to the routing services, there is a lot to tune in the engines, but i won't go into that in this post. Shoutout to @woosuk_k , @zhuohan123 , and @simon_mo_ for their work on vLLM, and @lm_zheng and @ying11231 on SGLang, alongside both projects' contributors. Similar to caching in a web app, you don't want to repeat expensive work. The "KV cache" stores intermediate attention calculations. Prefix caching reuses them when requests start with the same tokens. The prefix matters tho, the same document after a different system prompt won't give you the same hit. GPU memory fills up with model weights and active requests, so keeping more KV cache means using three tiers: This is hierarchical caching . Cached data gets loaded back onto the GPU when needed. You want to retain useful work, but fetching it has to be faster than recomputing it. HiCache , LMCache , and Mooncake help with this. Their features overlap and some work together. Learn how your stack decides what stays in each tier and when it moves. I think caching is one of the hardest parts to get right, but avoiding repeated computation can make a huge difference to cost. Shoutout to @Zhiqiang_Xie for his work on HiCache, @JunchenJiang and @this_will_echo on LMCache, and @TengMa3577 on Mooncake, alongside the contributors connecting these systems. If you're coming from web development like me, hopefully this makes inference feel a little more approachable. You don't have to understand every CUDA kernel to get started. Get a model running, send it some traffic, and work through these layers as you run into problems. GPU memory for cache the model is actively using. CPU memory for more space to keep prefixes you might reuse soon. NVMe for larger, cheaper capacity with slower retrieval.

DevOps AI

▲ 0

0 views

Stratechery 6 days ago

The iPhone Duo, The Intelligent Personal Hub, Apple Watch Audio Intelligence

Apple once again demonstrated the power of integrating hardware and software, but it's biggest AI blindspot might be its belief in the primacy of apps.

iOS Programming AI

▲ 0

0 views