Latest Posts (20 found)

The NVIDIA Bubble

If you liked this piece, you should subscribe to my premium newsletter, and you can subscribe on the following links: $70 a year , $18 a quarter , or $7 a month . In return, you get a weekly premium newsletter that dives deep into the finances of the AI bubble. Over the past few months, I’ve published a two part guide to the world of AI debt, an analysis of the (dubious) impact of generative AI on the global economy , deep analyses of the biggest players in the AI infrastructure space like NVIDIA (part one and two ) and Oracle (part one and two ), and the role private credit is playing in funding AI data centers , and how private equity, reeling from years of successive failed software acquisitions, is doubling down on the AI bubble .  Last week’s piece was a 13,000-word deep dive into Megaspeed — the shady firm at the heart of Jensen’s downfall, which somehow bought billions of GPUs despite being, just a few years ago, an unsuccessful developer of free-to-play mobile games. It’s all very weird, and I spent a lot of time making sense of it all.  On Friday, I’m going to share how we fix the tech industry, and how we stop the next bubble before it even starts.   If you want to get in touch — and especially if you have any juicy information about Anthropic, OpenAI, or any other companies in the AI bubble — hit me up on Signal at ezitron.76. I’m also on IB on your Bloomberg Terminal.  This weekend, thousands of horrible little men cried out for the return of their pervert emperor, demanding that NVIDIA restore disgraced CEO Jensen Huang, who resigned after it leaked that he’d had multiple affairs , one of which (as described in my free newsletter last week) was with a woman called Alice Huang who may or may not have been a Chinese spy, but was most certainly selling restricted AI chips to Chinese companies through a company called Megaspeed ( covered in depth in my premium last week ).  Within the space of a week, illegal chip sales routed through Alice Huang’s empire went from $4.8 billion to “at least” $7.5 billion , to $12 billion between March 2024 and June 2025 according to the Wall Street Journal.  The Information also reported that companies connected to Alice Huang invested in DeepSeek, the famed Chinese AI lab that spooked the world back in Q1 2025 for, ironically, suggesting that we didn’t need to buy as many NVIDIA GPUs. Let’s be clear about something: Jensen Huang resigned and is not coming back. To paraphrase Semafor’s Rohan Goswami , NVIDIA’s board was packed with allies , deliberated at length about what to do (and didn’t really want to do it) , and has both let him walk and done nothing to retain him, I’d personally guess because the probe the board is running already has some idea of wrongdoing such as, I dunno, the giant illegal smuggling ring that has been covered at length by basically every media outlet. NVIDIA has also said it was not considering bringing him back . Even now, a week into this bizarre crisis, few are truly reconciling with the allegations raised in May 2026 by Culper Research’s short report , namely that 20% or more ($43bn+) of NVIDIA’s FY 2026 sales came from selling through China. Even two profiles of Culper in the New York Times and Wall Street Journal fail to substantively discuss the material of the guy they’re god damn profiling. That’s because, to paraphrase Culper’s own work, Megaspeed is just the tip of the iceberg. As I covered last week , Culper directly connects Megaspeed to popular AI server manufacturer Supermicro , whose co-founder and board member Wally Liaw was arrested with several co-conspirators back in March for trying to sell over $2.5 billion in chips to a Chinese passthrough entity Culper believes is a Megaspeed affiliate. Liaw had previously resigned in 2018 due to an accounting scandal ( per short seller Hindenburg ), only to be rehired in 2021 as a consultant and reinstated to the board in 2023. Let me spell it out for you: a co-founder and member of the board of one of the largest manufacturers of AI servers was arrested for selling chips to China, specifically to a company affiliated with the very same smuggling operation that Alice Huang ran.  Supermicro is a huge, publicly-listed company with a history of accounting scandals. 70% or more of its revenues come from selling AI servers powered by NVIDIA GPUs, and back in August 2024 — months into Megaspeed’s $12 billion purchases of eventually-smuggled GPUs — Supermicro was NVIDIA’s third-largest customer, around that it was facing allegations of accounting manipulation and sanction evasion by Hindenburg that never went anywhere because our markets are broken and our SEC is trash. Anyway, my point is that this story is just getting started, and that those in the Copium dens are trying to reframe this as a “guy who made a mistake” rather than a philanderer that appears to have overseen the GPU equivalent of Iran Contra. Whatever rumors you may have read about a potential Jensen comeback appear to be peddled by the most desperate of AI boosters who, based on reports on CNBC , do not appear to be aligned with what Huang himself has said.  And really, why do they want Jensen back so bad?  It’s simple: Jensen Huang’s value has nothing to do with GPUs or AI, and everything to do with keeping the AI bubble inflated. Perhaps it’s more appropriate to call it the NVIDIA Bubble , because without it, the AI industry doesn’t really make much sense.  As I’ll make clear later in this piece, NVIDIA played an active role in keeping the wheels of the AI bubble turning — funding model labs, buying startups, and backstopping compute for Neoclouds, thereby allowing them to raise debt. Its interventions didn’t merely sustain the industry-wide spending spree, but amplified it. In the process, NVIDIA transformed the AI bubble to something that attached itself to the personal wealth of ordinary people — often without them realizing it.   As the stock prices of NVIDIA (and the rest of the Magnificent Seven) became engorged on AI-related hysteria, the exposure of ordinary (and unwitting) savers to the AI bubble grew. Investors who followed the mantra of “don’t buy individual stocks, just buy index funds” ended up in a position where, due to the top-heavy nature of the S&P 500 , and the predominance of mega-cap tech stocks, a significant chunk of their wealth is gambled on the question of whether AI is a bubble or not. And that’s without mentioning other factors — like how insurers and pension funds are exposed to the AI bubble (and, in particular, the AI infrastructure buildout), either directly or indirectly through their investments in private credit.  Or, put succinctly, if NVIDIA sneezes, the retirement savings and personal investments of ordinary retail investors will catch a cold. And if NVIDIA catches the flu, we’ll see the wealth of retail investors contract the fiscal equivalent of pneumonic plague — the type that Soviet scientists cooked up in a Siberian lab during the Cold War, and that gives Annie Jacobsen nightmares .  Despite many saying “NVIDIA isn’t NVIDIA without Jensen,” in truth it’s still selling the same chips, has the same fab space and supplier agreements, and outside of his horrible personality (and cult of personality ) seems virtually identical. Back in August, NVIDIA said that it expected 70% year-over-year growth in Fiscal Year 2028 (starting February 2027) , which suggests — unless something has materially changed! — that NVIDIA had clear expectations and agreements to hit those targets. CEO Colette Kress should, if you believe everything that’s been said over the last few years about “insatiable demand,” have absolutely no problem selling just as many GPUs at just as fast a rate… …unless, of course, there’s another reason that “NVIDIA isn’t NVIDIA without Jensen.” The truth is that what people actually mean is that “NVIDIA isn’t NVIDIA without Jensen using its balance sheet to grow at a high pace involving constant, quasi-legal-to-illegal dealmaking involving channel stuffing and illegal smuggling.” The Wall Street Journal reports that Jensen Huang had told his finance team ( in August, the very same month it raised guidance) that the company was facing a glut of GPUs and had over-ordered capacity at TSMC and with memory manufacturers, necessitating further circular financing: I dunno man, I’m no securities lawyer, but I feel like worrying that you have an oversupply of chips and then telling investors that you actually expect to sell more chips than ever feels like some sort of stock manipulation, and it wouldn’t be the first time NVIDIA’s done it in the last year. Back in September 2025 , it announced it would invest $100 billion in OpenAI to build 10GW of data center capacity, with Jensen Huang saying live on CNBC that there had “never been an engineering project of this complexity and this scale ever,” referring to a deal that he would dismiss as “ never a commitment ” five months later, because it was only ever a “ letter of intent .” The truth is that Jensen Huang lied. You can dress it up in terms of “memoranda of understanding” or “letters of intent” not actually meaning “a contract has been signed,” but Jensen Huang went on CNBC and did everything he could to convince you that NVIDIA was investing $100 billion in OpenAI to build 10GW of data centers, much like he went on CNBC a few months ago to convince you that (and I quote the headline) “NVIDIA [had lined] up $500 billion in financing” despite these being A) memoranda of understanding and B) no money ever changing hands or being raised by anybody to do anything and, to quote Bloomberg , said $500 billion figure having “no obvious provenance.” There is no other way to interpret this other than intentional manipulation of the markets perpetuated by the mainstream business and technology press, all because nobody would ever push back on Jensen Huang.  And that’s because everybody knows, deep down, that we’re all living in an NVIDIA Bubble — a financial mania manufactured, maintained and sustained using NVIDIA’s credit rating and cash to brute-force the markets and the media into believing in anything. While the AI bubble is relatively young, the NVIDIA bubble has been inflating for over a decade. I don’t even mean that derisively! Jensen Huang and NVIDIA were formative to the entire AI bubble, both in creating the infrastructure, demand, and even the underlying research that eventually led to the LLM boom.  Way back in August 2016, Jensen Huang delivered “the world’s first AI supercomputer” — the DGX-1 — to a small AI lab called OpenAI that had recently raised money from yCombinator’s Sam Altman and Tesla/SpaceX’s Elon Musk . Three months later, OpenAI announced it was moving to Microsoft Azure , training on NVIDIA Pascal GPUs in the earliest days of AI compute. A few years later, NVIDIA found itself in flux, with a beat on earnings in February 2019 clouded by a decline in both hyperscale and cloud sales , but Jensen Huang was already working on an acquisition that would change everything, agreeing to buy Israeli high-speed networking firm Mellanox that March for $6.9 billion in cash , outbidding both Microsoft ( which was already using its tech at Azure ) and Intel for the InfiniBand technology that would eventually allow NVIDIA to network together clusters of AI servers.  Four months later in July 2019 , Microsoft would invest a billion dollars into OpenAI “...to build “new Azure AI supercomputing technologies.” Four months later in November 2019 , NVIDIA and Microsoft would announce that Azure would now offer a “a new kind of GPU-accelerated supercomputer in the cloud on Microsoft Azure,” networking together up to 800 NVIDIA V100 GPUs using Mellanox’s InfiniBand tech, and the first time I can find where you could rent out massive clusters of AI GPUs. Microsoft and NVIDIA engineers even trained their own conversational AI model called BERT . The deal itself wouldn’t close until April 2020 , and a month later in May, Microsoft and OpenAI would announce that they had built a 10,000 GPU “supercomputer” together, with sources telling The Register that it was likely made up of AMD CPUs and NVIDIA V100 GPUs.  The same month, NVIDIA would release its A100 GPUs , which remain somewhat of a workhorse for AI Compute, and were a huge commercial success. Another significant milestone came two months later in July 2020 , when NVIDIA would publish a blog about how OpenAI researchers had released a paper describing how they’d trained GPT-3 using its Volta-era V100 GPUs ( released in 2017 ).  In August 2020 , NVIDIA’s data center revenue would outpace its gaming revenue for the first time in the company’s history, “aided largely by the Mellanox acquisition” per Holger Mueller of Constellation Research.  In other words, Jensen Huang and NVIDIA were far more foundational to the AI bubble than Sam Altman or Elon Musk, quite literally buying and building the company and the infrastructure necessary to build large language models.  While June 2017’s Attention Is All You Need paper absolutely kicked off the transformer-based model boom, NVIDIA and Baidu’s late 2017 research into mixed precision training that allowed model developers to use nearly half as much memory during training without losing accuracy, and also train faster on NVIDIA’s newer chips, a technique that became the standard in the early days of LLMs, shouldn’t be overlooked. In August 2021, OpenAI would launch Triton , a Python-esque coding language specifically built to make it easy to write code that ran on GPUs that would become foundational to the entire AI boom. Three months later, Microsoft announced that OpenAI’s GPT-3 was now available on an invite-only basis to customers via its new Azure OpenAI Service . Then came 2022 — a rough year for everybody, per my second Hater’s Guide To NVIDIA :  NVIDIA’s Q3 FY23 (November 2022) earnings were rough , missing on analyst expectations , though not as rough as Q2 FY23, when it had to pre-announce that it had missed on guidance by a little under $2 billion .  While NVIDIA had started shipping its H100 GPUs with data center sales eclipsing its gaming revenue by about $2.2 billion, it had yet to fill the hole caused by the post-pandemic dropoff of PC and graphics card sales or, more aptly, the hole left by the collapse of cryptocurrency mining.  And then a mere two weeks later, Sam Altman announced the launch of ChatGPT , creating the greatest sense of FOMO in the tech industry’s history. Per the New York Times , “...the entire agenda of [Google] had changed” as a result of ChatGPT, and Satya Nadella was “gleefully” ordering billions of dollars of NVIDIA GPUs.  By March 2023, every major hyperscaler was now offering NVIDIA H100 chips , and a month later in April 2023, NVIDIA would invest in the Series B round of CoreWeave , a former Ethereum miner backed by Magnetar Capital, which was famous for backing dodgy CDOs during the Great Financial Crisis .  Somewhere in early 2023, CoreWeave got a new customer — NVIDIA itself, according to The Information . This deal was part of a wider strategy dubbed “Project Osprey,” wherein NVIDIA would spend $1.3 billion over four years to rent back its own GPUs for whatever reason, making it the second largest customer behind Microsoft, who was renting capacity for OpenAI. That Microsoft deal would eventually lead to the creation, per The Information , of the first GPU-backed financing led by private equity firm Blackstone, emphasis mine: This deal would, as The Information notes, become the template for basically all data center financing, rationalizing the notion of lending unprofitable, debt-filled neoclouds like CoreWeave billions of dollars to build data centers as long as some sort of other counterparty agreed to rent capacity in the future. And when lenders needed further justification — by which I mean proof of cashflows — they’d happily tick the box because NVIDIA had agreed to pay its customers to rent back the GPUs that all that debt was being raised to buy. In September 2023, Bernstein analyst and alleged professional Stacy Rasgon attacked “Twitter randos” for suggesting, and I quote , that it was a conspiracy theory to believe the following: Three years later, Rasgon has now downgraded NVIDIA , and worries about the following: Stacy was, of course, talking about Jack Gamble (AKA JG_Nuke) who had repeatedly critiqued CoreWeave as a pump and dump inflated by NVIDIA in its earliest days in 2023.  I mention this because I believe analysts and the mainstream media have been critical to inflating the NVIDIA Bubble, and in laundering the reputations and rationalizing the inherent lunacy of the AI bubble. As long as NVIDIA “beat and raised” — as in beat analyst expectations and raised guidance — everybody was happy and looked away from the many, many things wrong with this picture. And 2023 was the year that NVIDIA’s circular financing took off, with the company investing in Cohere , Synthesia , Runway , Hugging Face (which it would eventually acquire), Perplexity , Together (an inference company that also buys GPUs), and Mistral . At this point, NVIDIA’s bubble had begun to inflate, with a landgrab to invest in anything AI related and to buy as many GPUs as possible, per me in Part 2 of my Hater’s Guide To NVIDIA: What was missing, of course, were revenues. But none of that mattered! A captured media ecosystem and a brainless market credited AI with basically any related revenue growth for anybody who bought NVIDIA GPUs.  Behind the scenes, Jensen was doing everything he could to keep growth going, making sure CoreWeave was able to raise more than $10 billion in debt from a handful of very specific bankers that he’d conned into believing the idea that building endless AI data centers would create infinite cash.  By February 2024, the sudden influx of demand led to Jensen ruefully discussing the “fairest” way to decide who gets its GPUs , and a month later he’d announce Blackwell , the next-generation NVIDIA GPUs that cost tens of thousands of dollars more depending on the configuration , specialist cooling and entirely new racks, massively increasing the cost of data centers and, in turn, increasing NVIDIA’s revenues.  In June 2024 , Jensen would claim that “the more you buy, the more you save” under what he called “CEO Math,” but I would call “complete bullshit”, per Business Insider: That, by the way, is about as much “skepticism” as you’ll find in this era, and why the bubble was allowed to inflate. At no point can I find any substantive conversation about the actual benefits of buying NVIDIA GPUs , or any of the financial returns , or really anything other than…well…fluff.  You see, everybody loved the AI bubble. Liking AI allowed you to talk about something new and exciting , and nobody would argue with you if you said you thought it was going to make everybody a bazillion dollars. They would, however, attack you viciously if you dared to suggest anything was wrong — something they still do to this day! — because, as Jensen realized, AI had become load-bearing for the entire stock market, even though the only people making money on AI were NVIDIA and other semiconductor companies. And somewhere between 2023 and 2024, Jensen met Alice Huang (no relation) , and between March 2024 and June 2025, per the Wall Street Journal , Megaspeed would buy “nearly $12 billion worth of top-of-the-line NVIDIA chips” and send them to Chinese firms, including Blackwell chips that it claimed had demand that was “well above supply” in August 2024 and by October 2024 was “ sold out for the next 12 months ,” partially as a result of Jensen Huang creating that demand via CoreWeave and in his dealings with Megaspeed. In 2024 alone, NVIDIA would invest in data center firm Applied Digital (CoreWeave’s landlord), as well as neocloud Nebius and Stargate Abilene data center builder Crusoe . In 2025, it would anchor CoreWeave’s IPO ( which couldn’t have proceeded without NVIDIA’s help ), agree to pay neocloud Lambda $1.3 billion over four years to rent back its GPUs , invested twice in British neocloud Nscale, invested in Crusoe again , and extended its deal with CoreWeave to pay it $6.3 billion . In August 2025 , CFO Colette Kress would sing the praises of “extraordinary” and “soaring global demand,” and in October 2025, Jensen would claim ( and Reuters would report ) that there were “$500 billion in bookings for its AI chips.” What did that mean? Who knows. Nobody asked. Nobody dared harsh the flow. Would NVIDIA make $500 billion in that period? No. Huang would claim in NVIDIA’s Q3 Fiscal Year 2026 earnings ( November 2025 ) that “compute demand keeps accelerating and compounding across training and inference — each growing exponentially.”  All of this was part of the kabuki theater of NVIDIA — claims that there was “astonishing demand” for AI GPUs at a time when it was doing everything it could to inflate that demand by funding companies explicitly to raise debt and buy GPUs. The first sign of desperation began in September 2025, when Jensen Huang would go on CNBC and announce that non-existent deal with OpenAI . Per the Wall Street Journal , talks never progressed past their earliest stages. This did not stop anyone from reporting the deal as done for literally months — its intention was entirely to create the imaginary sense of “demand” that would mean you have to do whatever Jensen needs to get his GPUs.  And, of course, to continue fooling the media into believing that there was “incredible demand for AI.” In a December 2025 piece, TIME would mention that Jensen Huang was “unhappy,” because NVIDIA “had to publicly declare that its China revenue had shrunk to $0,” leading to him lobbying the Trump administration to change chip export rules that NVIDIA was at the time vigorously violating. By the end of 2025, NVIDIA had also agreed to around $27 billion worth of cloud compute agreements in total — literally agreeing to rent back its GPUs from its customers — and as of its last earnings report, these agreements had grown to a remarkable $36 billion. Not once during these three years did we ever learn what revenues Microsoft, Google, Amazon, or Meta made from renting out NVIDIA GPUs, but we didn’t need to, because the media was all too happy to claim that Microsoft and Google’s bets on AI had paid off as early as 2024. It didn’t matter that OpenAI or Anthropic lost billions, it didn’t matter that CoreWeave was a time bomb , it didn’t matter that NVIDIA was literally paying its customers to buy its GPUs , it didn’t matter that NVIDIA was effectively stepping in whenever anyone had any doubts about the AI bubble , as long as hyperscalers — and most importantly NVIDIA — kept beating on estimates and raising guidance. Even though, as I’ve mentioned, the actual revenues from AI are nowhere to be seen. Thanks to Bloomberg , we eventually found out that Microsoft’s Fiscal Year 2026 AI revenues were around $34 billion, with $24.1 billion coming from OpenAI, meaning that $265 billion in capital expenditures since 2022 (at the time) had bought Microsoft a single-digit AI business outside of feeding itself money by investing in OpenAI. As we can see in hindsight, it’s clear that there never was “insatiable demand” for AI chips outside of Jensen Huang’s pressure tactics, some of which involved claiming there was “limited availability” of chips that he himself made sure were in limited supply, mostly by making sure that NVIDIA-incubated customers bought them using either NVIDIA’s investment or debt raised using that investment, like when NVIDIA put in at least a billion dollars into an SPV that raised money for Elon Musk to buy NVIDIA GPUs in January 2026 . And the most-damning proof we’ve had of there being no real “demand shortage” is that NVIDIA has been forcing customers like Amazon to take billions of dollars of GPUs months in advance of when they’d need them as part of an “early rack receipt program,” with Business Insider revealing one crucial piece of information: In other words, Jensen Huang said that if you don’t buy the chips now — even though you have “no clear forecasted path” to install them — you won’t get them when you “need” them. As I’ve said recently, based on analyses of earnings statements and documents viewed by this publication, I believe there are over $200 billion worth of NVIDIA GPUs that are sitting in warehouses waiting to be installed as a result of artificial demand pressures created by Jensen Huang and NVIDIA.  This is how NVIDIA has beaten expectations and raised guidance basically every fucking quarter for the past two years — circular financing, illegal chip smuggling and channel-stuffing.  The most important rule of living in the NVIDIA Bubble was that everybody kept in line, saying there was always more demand for GPUs, always more demand for AI, and that everybody was making lots of money… even though none of that was actually true. Back in August, I used analyst notes to prove that the vast majority of AI demand — 70%+ of revenue and 80%+ of compute demand — was from Anthropic and OpenAI buying compute, and the sheer power of the NVIDIA Bubble was such that people simply ignored it because it was uncomfortable to think about. Last week, Sam Altman said the quiet part out loud at a Samsung conference , emphasis mine: One particularly-childish view that I’ve heard is that if NVIDIA falters, AMD or Broadcom could potentially “pick up the slack” and “capture the demand,” as if said demand was not entirely a result of Huang’s dealmaking and his aggressive use of NVIDIA’s balance sheet. Your open source model company fails to close a financing round? NVIDIA will pay $6 billion to “license” its tech and take some of its employees . A shitty AI search company that nevertheless buys hundreds of millions of dollars’ worth of compute starts running out of money? No problem, NVIDIA’s got billions to make sure Perplexity keeps chugging . CoreWeave’s landlords get worried about it not making its lease payments? No problem, NVIDIA has backstopped the leases . Anthropic needs another $10 billion to make prepayments on its compute? No problem, NVIDIA is kicking $10 billion into its IPO .  There’re also very few other companies out there like NVIDIA, which had twelve-month trailing free cash flows of $127 billion through its latest quarter, second only to Apple, which had $136 billion.  As of its latest quarter, NVIDIA also had about $99.4 billion in total liquidity — more than AMD ($15.8 billion) and Broadcom ($24 billion) combined, with better credit (AA versus A and A- respectively), lower net debt, and an interest coverage ratio (IE: how many times it could pay its interest expenses with its operating income) of 435x compared to Broadcom’s 17x and AMD’s 65x. Additionally, NVIDIA has significantly more capacity booked with TSMC and the memory cartel, largely as a result of being able to afford it and justify it with upcoming revenues, even if said revenues were artificially inflated with circular deals. In other words, NVIDIA has more cash, better credit, a better balance sheet, and spent three fucking years building hype, FOMO and funding its own customers so that by 2025 there were entire companies that existed functionally to either buy or rent NVIDIA GPUs.  This is a plan that took years — or a decade, depending on how you think about it — to execute, and its collapse was inevitable, because it’s clear that despite his bravado and confidence, Jensen Huang was desperate and would do literally anything to keep the beat-and-raise machine alive.  The Chinese smuggling was and is one of the single-most brazen acts I’ve seen in the history of capitalism — a flagrant, well-documented campaign of export control violations perpetuated at times in broad daylight, ignored by much of the press, to this day referred to by some as a “conspiracy theory” despite GamersNexus’ smuggling report from August 2025 , the New York Times reporting in October 2025 , Bloomberg’s reporting in December 2025 , Wally Liaw of Supermicro’s arrest in March 2026 , and Culper’s massive short report in May 2026 proving in vast detail that NVIDIA is absolutely involved in illegal smuggling of GPUs to China. Yet the most-damning part of all of this is how willing everybody was to ignore the circular financing, and no, I’m afraid that all the grab-bag articles saying “wow, NVIDIA sure has invested in a lot of its partners!” don’t count.  The media rubber-stamped every single circular deal throughout the AI bubble because there were hundreds of millions or billions of dollars attached — even if they weren’t real! — from NVIDIA or one of NVIDIA’s customers, all without any consideration or discussion of how unstable it was or, more importantly, how NVIDIA was inflating its revenues by creating these circular relationships. While the cries to restore Jensen Huang have fallen on deaf ears, even if he were to return, it’s clear that the circular financing era was coming to an end regardless of who was wearing the leather jacket.  And it’s important to measure how incredibly large a loss that is. Per The Wall Street Journal : That means 27% of all venture capital funding in 2026 flowed from NVIDIA, and those that didn’t lock down their funding are incredibly worried, including Nscale, which, per the FT , needs more money to complete its massive contract with Anthropic: Bloomberg also reports that NVIDIA invested $2 billion in Nscale in September — a billion more than we knew — but apparently said billions weren’t sufficient, and it’s unclear what’ll happen next, much like it’s unclear what happened to NVIDIA’s supposed offer to buy Reflection AI, per the WSJ, or its investment in Thinking Machines: Huh, that’s weird. Why would NVIDIA invest in or buy AI labs?  Ohhh. Right. The NVIDIA Bubble might get pricked if anyone buying AI compute — even those who quite literally can’t afford it! — blinks for even a second about the prospect of buying more.  And that’s been the core of the Jensen Huang hustle — NVIDIA invests, NVIDIA rents, NVIDIA backstops, or NVIDIA introduces you to Microsoft, Anthropic, OpenAI, or Meta, one of whom is a customer of basically every single AI lab, which should tell you about how much demand for AI there really is. The Journal’s piece ends with a gutpunch that tells the entire story of the AI bubble:  Without Jensen, there is no AI bubble, because if anybody else acted like Jensen they’d look like a fucking lunatic, mostly because the NVIDIA Bubble’s purpose was to rationalize the irrational and provide justifications for basically anything. If you worried that your customer couldn’t pay you, Jensen would invest. If you worried that you couldn’t pay your vendors, Jensen would backstop your leases.  And despite the constant threat of demand outstripping supply, Jensen could always get you more GPUs — and, per Elon Musk , many of his deals were “a handshake” rather than a “written contract,” with a “purchase order coming together at some point later.” One has to wonder how many of the “deals” announced by NVIDIA in the last few months — or any of that egregious forward guidance — was based on Jensen’s “handshakes,” or his ability to mobilize suppliers and vendors and financing under the banner of “insatiable demand for AI.” Except…it’s very clear that era is over. Per the WSJ , Jensen was warning as much at a conference in October, emphasis mine: This is three very specific warnings baked into one: While the third point is obviously bollocks, the first two suggest both tightening financing conditions and NVIDIA’s unwillingness to bankroll its own bubble, and a suggestion that with or without Huang, the end was already approaching at high speed. And without Jensen — and the central bank of AI — there’s very little keeping this entire thing afloat. While hyperscalers might have been afraid of pissing off Jensen and getting put in Jensen Jail , I don’t see Amy Hood (CFO of Microsoft) being intimidated by the meager presence of Colette Kress demanding that she (and other hyperscalers) take billions of GPUs in advance.  Without Jensen, NVIDIA loses its don (credit to Michael Burry for that one) and its ability to use mob-like tactics to push around its customers or constantly provide, per the New York Times , questionable deals that nevertheless made bankers billions of fees: Huang’s cachet was so significant that he was able to bully the CEOs of effectively every major private equity firm into lying on television about a non-existent $500 billion deal to fund data centers, emphasis mine: Jensen’s presence was not just finding the deals , but calling up the right bankers at the right time and giving them explicit or implicit guarantees that NVIDIA’s balance sheet would hold up the other end of the deal. And without Jensen, the NVIDIA Bubble bursts, leaving behind an industry that must reconcile with the consequences of its destructive mania. Whatever value you may or may not think AI has, it is not paying for itself. It is a continual drain on the debt markets and the venture capital industry.  Anthropic has $141 billion in cash, wants to raise another $70 to $90 billion, and will not be cash-flow positive before 2030 according to underwriter Goldman Sachs . It also has $895 billion in compute commitments, which is the real reason NVIDIA is investing $10 billion at its IPO — Jensen doesn’t care if the chips ever get rented, he cares that somebody has a reason to buy them, or issue debt to the company doing so. In reality, the “demand” for AI GPUs was manufactured by Jensen Huang and the hype cycle he created through exploiting the desperation of a flailing tech industry. The industry’s lead cheerleader and House Whip was also its central bank, an evil Michael Clayton figure that loved profiting off of bullying, monopolization, and legal cons that only made sense because he was going to do anything he could to keep you from not paying for AI, which required keeping as many companies alive as possible.  Anyone else doing this in literally any context would be considered a lunatic. Perhaps Jensen was one too.  In any case, nobody else has the balance sheet, credit rating, assets, cashflow and psychopathy to do the same. Jensen spent 33 years inflating the NVIDIA Bubble, and it took less than a week for it to collapse.  I want you to think deeply about why you think there’s “demand” for AI, and how much of that belief is built upon a fantasy concocted by Jensen Huang: Even Anthropic’s demand appears to be slowing, with growth ( per Ramp ) trending to around 15% to 25% quarter-over-quarter in Q4 2026, down from 47% in the quarter before and over 140% in the quarter before that.  One has to wonder how Anthropic — or OpenAI’s — customers will be able to afford those tokens if there’s a generalized sense that the money has run out. Remember: it’s not just that Jensen invested , but that everybody ultimately relied on the idea that Jensen would always be there if somebody else couldn’t invest. There was always the sense that NVIDIA had everybody’s back, mostly because, well, NVIDIA has, up until recently, had everybody’s backs as part of a giant, quasi-legal circular financing operation that the media applauded right up until it sputtered. As I’ve explained for years, the entire tech industry has become obsessed with growth-at-all-costs thinking — or what I call The Rot Economy — which inevitably dragged it into an era that was defined not by delivered value but by speculation on what might happen if we all put another few trillion dollars into something.  In reality, buying NVIDIA’s GPUs was always a shitty deal, though nowhere near as bad a deal as renting them. Hyperscalers won’t share their AI revenues because their AI revenues suck, and they’ve spent $1.4 trillion or more on what appears to be less than $100 billion worth of revenues, with the future theoretical hundreds of billions contingent on hyperscalers spending hundreds of billions more. OpenAI made up 11% of Microsoft’s total revenues in Q1 Fiscal Year 2026 , which suggests that Microsoft’s overall revenues are in decline and that its AI revenues aren’t filling the gap outside of an AI lab that lost $10.9 billion in Q3 2026 and raised only $11 billion of the $30 billion it wanted in its latest funding round at $1.25 trillion , which was also lower than expected. CoreWeave, Nebius, IREN, and every neocloud is woefully unprofitable, with the justification being that if they just stopped raising money and finished all of these data centers they might one day have positive margins, somehow. Braindead analysts are now suggesting that “customer prepayments will help instead of debt,” which is contingent on companies that rent AI compute making prepayments, most of whom are unprofitable.  While hyperscalers could fill the gap, hyperscalers are already in an overbuild situation, mostly driven by Jensen Huang’s pressure campaign.  Demand for AI services has never, ever made sense, with Anthropic and OpenAI making up 89% of all AI startup revenues and both of them being both unprofitable and agreeing to spend over $1.3 trillion in the next seven years , most of which is contingent on continual funding and debt. We do not know the revenues of effectively any other AI startup, only annualized run rates that tell us very little about the actual financial conditions of the company and the occasional reporting that tell us a dog like Cognition would burn $800 million in 2026 alone .  Even with the best possible read of those annualized run rates, these startups raise hundreds of millions or billions of dollars’ worth of funding to make tens or hundreds of millions of dollars in revenue, and need to do so every few months due to the remarkable cost of renting NVIDIA GPUs, which despite being multiple generations in do not seem to be getting more efficient for the end customer. In many ways Jensen Huang created the perfected form of Jack Welch’s GE Capital , manipulating the markets, the media and the banking system into affirming his every decision by making sure that everybody had somebody else to pay or get paid by, even if the entire system was terrifyingly brittle.  And the worst part is that Jensen never really cared about whether there was demand for AI outside of its ability to justify further debt or cash for buying AI GPUs. The majority of the backstops that NVIDIA has provided — like its leases for Oracle, Anthropic, and OpenAI — require the data center capacity to be built before NVIDIA pays a dime, punting any theoretical time at which he or his heir would have to “backstop” anything into a time so far in the future that it doesn’t really matter.  Every CoreWeave or neocloud debt deal follows the same path — a customer agrees to buy unbuilt capacity, the debt is raised, the GPUs are bought, and then nothing happens until the data center is actually finished, which may take years, but none of that matters as long as NVIDIA got paid. Who knows what happens in NVIDIA’s earnings today. Perhaps they beat and raise again, with Colette Kress coasting off of the previous three months’ worth of circular financing deals that kept the machine alive.  Then again, the amount of stories in the past week that have said something to the effect of “Colette Kress and NVIDIA do not want to do circular financing anymore” suggest that the company may be getting ahead of something — such as pulling or lowering revenue guidance, perhaps blaming everything on the “challenging transition” rather than “the collapse of the circular financing economy.” In any case, the NVIDIA Bubble has popped. Jensen Huang is done. NVIDIA is no longer the circular financing titan, and anybody depending on its theoretical backing for future financing or funding is shit out of luck. There is no longer a well-funded boob-signing billionaire playboy left to beguile the press, con financiers, and pump the most wretched of bags in an endless conga-line of dodgy deals with goal of goosing NVIDIA’s revenues by any means necessary, even if the eventual data centers never actually get bought. The NVIDIA Bubble inflated at the perfect time, with Jensen capitalizing on the industry-wide knowledge ( as I wrote back in the Rot-Com Bubble ) that tech was flat out of hypergrowth ideas. NVIDIA offered hyperscalers and their brainless executives a panacea — a magical chip that allowed them to monopolize both the software layer and the infrastructure underpinning it, if only they’d just buy enough GPUs and build enough data centers to house them. All that mattered to Jensen was keeping them believing in the dream. Unfortunately, the vast majority of people do not want to pay for AI, and the vast majority of those who pay for AI compute can’t afford to do so. Those buying NVIDIA GPUs refuse to explain how much money they’re making from renting them out, or what it costs to do so. We do not know how many active AI GPUs there are in the world, and hyperscalers like Microsoft have taken great pains to mislead us into believing they have gigawatts more in capacity than they actually have .  There’s a reason that NVIDIA has $36 billion in compute commitments — because the actual demand for AI compute is much, much smaller than the demand he created for his GPUs. None of this mattered because Jensen Huang always made sure that nobody thought about it too much. The moment NVIDIA’s growth slows, it’ll be all that anybody can think about.  If you liked this piece, you should subscribe to my premium newsletter, and you can subscribe on the following links: $70 a year , $18 a quarter , or $7 a month . If you want to get in touch — and especially if you have any juicy information about Anthropic, OpenAI, or any other companies in the AI bubble — hit me up on Signal at ezitron.76. I’m also on IB on your Bloomberg Terminal.  You’re gonna need customer prepayments to buy GPUs, which will require cash up front from your customers, who are for the most part functionally in default. You’re not going to be able to raise debt without being able to do so, if at all. Bring me cash or I’m not going to guarantee I can sell your GPUs, which are in tight supply, honest. There’s massive demand for AI GPUs . Is there? Jensen Huang pushed customers to take billions of dollars’ worth of chips months in advance, with $200 billion or more of them sitting in warehouses thanks to a lack of capacity or power to install them. That “demand” appears to be a result of channel-stuffing, and if it’s not, why the fuck did they have to pay for the chips and take them earlier than they need them? There’s massive demand for AI compute . Is there? The vast majority of AI compute is sold to either Anthropic or OpenAI. The vast majority of Microsoft, Google, Oracle, CoreWeave, Nscale, Lambda, Amazon and Firmus’ backlogs come from OpenAI, Anthropic, Meta, or a hyperscaler buying capacity for OpenAI and Anthropic. There’s massive demand for AI tokens. Is there? 80% of Anthropic and OpenAI’s revenues come from 1% of their customers, the vast majority of which are unprofitable AI startups , with the largest purchasers of these tokens — like Perplexity, Cursor, and Reflection AI — all funded by NVIDIA.

0 views

Neat stats are neat

Adam has officially released NeatStats , and I have to say, it looks, well, kinda neat! I’m not a big stats person; I do think knowing too much about your personal site is not all that healthy and can push you into weird directions, but for people who do want to get at least some basic information about the traffic they get, this looks promising. I also quite like the terminal view on the landing page; it reminds me of the default Goaccess view. It’s interesting how for years there was pretty much no action in the “analytics for websites” space, but now—and I suspect this was partly because of GDPR—we have loads of them: from Plausible , to Fathom , to Umami , to Simple Analytics , to Pirsch , to Tinylytics , and the list goes on and on and on. All that said, I’ll stick with parsing my logs with Goaccess those rare times when I want to see what’s happening on my site, and I’ll also keep assuming that nobody reads these posts, other than AI scrapers. Thank you for using and believing in RSS. You're awesome. Send me an email :: Sign my guestbook :: Become a supporter

0 views

Apple Permissions Update, The New App Paradigm, My iPhone Duo Review

Apple doesn't ruin the Mac, but they don't fix the agent problem, raising questions if they can think outside the app paradigm. Then, my iPhone Duo review.

0 views

2026-11-16 19:11: Had the BearBlog API existed 18 months ago, I don't think I'd have created Pure...

Had the BearBlog API existed 18 months ago, I don't think I'd have created Pure Blog. Great to see Herman finally implement this and I'm excited to see what people do with it. I'm still really enjoying Pure Blog though, and I'm glad it didn't exist back then. 🙃 https://docs.bearblog.dev/apihttps://docs.bearblog.dev/api/ Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment .

0 views
Unsung Today

Guardians of the gestures, pt. 1

When you use Figma, you can press ⌘R to rename any object: Even if you hide the left pane interface, ⌘R still works – now, the label itself becomes the input field (it even shows up if not normally visible for secondary objects): And, when you select multiple objects and press ⌘R, you get this dialog, allowing you to write down more explicit rename rules for the entire group – maybe number them sequentially, maybe just remove one word while keeping the other ones intact: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/guardians-of-the-gestures-pt-1/3.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/guardians-of-the-gestures-pt-1/3.1600w.avif" type="image/avif"> I don’t love this dialog and I think it could use a revision, but I didn’t want to talk about its details here. What I wanted to mention is that even here, you can still simply and mindlessly type a new name and press Enter – and all the frames will be renamed to this name: This was a deliberate design decision. Either one of these three modes supports the most barebones version of the interaction: new name first, Enter to confirm second. As a user, you should not care whether the UI is visible or not, and if the gesture to rename one object works, your fingers will expect it to work for many objects, too. Motor memory just doesn’t want to think too hard . Motor memory loves things to be deterministic . It was nice to see this principle employed by the recently improved Apple Notes, as well. In TextEdit and a lot of other software, you can use ⌘K to linkify a piece of text. The interface is often very, very simple, often showing you just a field to input the URL: The troika of keyboard actions – ⌘K to add a link, ⌘V to paste a URL, Enter to confirm – has been reliably working like this since the 1990s, when the freshly blue and underlined links were the talk of every town. I bet many people linkify using this gesture not thinking of the three keystrokes they’re making in precise sequence, and as a matter of fact, often not thinking about it at all . Apple Notes added a few more link options recently: linking to other notes, linking deeper inside a specific note, changing the name of the link. The window reflects the extra functionality (and complexity), but the rudimentary, almost atavistic gesture – ⌘K, ⌘V, Enter – has been protected… …and I’m grateful to the guardian who made sure it was.

0 views

A very lucky husband

I was chatting with a friend in the fediverse today, and I realised that I have been married for 14 years. Sandra and I started dating 18ish years ago, which is - give or take a few years - roughly half my life so far. And while that makes sense, chronologically, it absolutely amazes me. When we started our business, working from home, we talked about what it would be like to work together, and live together, and basically be around each other pretty much all of the time. I am not a very sociable person (although I can turn that on, for limited periods, when I need to), and honestly that idea was slightly frightening to me. Would it ruin our relationship? Would we struggle for lack of personal space? I think that my concerns were valid, but it turns out that they were misguided: more than 10 years later, and we’re still together. Really, it is more than just “still together”. Much more. I get to spend an awful lot of time, every day, with someone who is not just my best friend, but also the person that I love. In a way, it is daft: despite being around each other all the time, I occasionally catch a glimpse of Sandra - not doing anything special, or dressed up, just something normal and mundane in the house - and I see the most beautiful, clever, loving woman that I’ve ever met, and I smile. I wake up each morning, feeling both lucky and grateful. And boy do I make sure that I tell her this.

0 views

Crocosmia

Maya sits beneath an enormous red petaled flower, 700 meters tall, impossible and behemoth. She thinks of her mother, Jane, whose disappearance presaged the arrival of the skyscraper-sized crocosmia, a moment known as the “great turning.” Jane’s art is mysteriously connected to that huge shift in global politics, the surprising and simultaneous decapitation of seven world leaders. The book is a kind of parable of revolution, told through the lives of mother and daughter, as they move from commune to isolation, always carrying both experiences with them. For to be with others is also to be alone, and to be alone is not to be apart. View this post on the web , reply via email , or become a supporter .

0 views
Blog System/5 Yesterday

Remote EngFlow caching meets an old G4

During BazelCon 2026 in early October, the fine folks at EngFlow gave me a very cool piece of swag: a Cache & Carry box or, in other words, a portable remote cache for Bazel builds. The only ask was for me to post a photo on social media, which I’d have gladly done, but… that seemed like too low a price for a nice piece of hardware. So I promised I would write an article instead. The article is late, though, as I have been busy getting ZKeys out the door and trying to find a good idea for what to talk about. I finally got the right inspiration after taking an old PowerBook G4 laptop from 2004 out of the closet to play with Tabby-PPC . That experiment failed, but as the laptop sat on my desk, I thought: could I make it leverage the Cache & Carry and demonstrate the clear benefits of remote caching? The answer was “of course yes” before even trying, but it was not going to be easy: this machine is severely underpowered by today’s standards and won’t run Bazel, so showing remote caching via a modern build system wasn’t going to be possible. I had to find an alternative demonstration path, and find it I did: the recc compiler wrapper . Even then, the benefits weren’t great on the first try, so I had to debug and iterate. In the end, the compilation time for itself dropped from a little over 1 hour to just 3 minutes. Let’s see how. Typing articles like this one, with manually-taken pictures and hand-edited videos, requires a lot of work. Subscribe to Blog System/5 to show your support. EngFlow is a relatively new company that offers a hosted remote caching and execution service for Bazel builds. More specifically, EngFlow is an implementation of the Remote Execution (RE) API . And fun fact: the protocol name is RE, not RBE as many people call it, because RBE was Google’s own RE service that never really saw the light of day. The EngFlow Cache & Carry in its box before unpacking it. Cache & Carry is a little toy that packages the EngFlow remote cache (not execution) software into a Raspberry Pi 4 preconfigured to work as an appliance, enclosed into a branded 3D-printed case and packaged into a fancy cardboard box. Mind you: this is what I wanted to achieve with my own EndBOX last year but never went the last mile to distribute it as a truly finished toy. The EngFlow Cache & Carry Raspberry Pi 4, with its custom branded case. Getting started I’m quite used to EngFlow’s product by now because we use it at Snowflake to power our in-house builds and tests, so using the Cache & Carry was no difficulty. And it shouldn’t be for anyone given that configuration and management are pretty easy. Here is the cluster status page right after plugging in the device and connecting to it, showing how this “cluster” is really just a cache and has no remote execution features: EngFlow's standard Cluster status page in the live dashboard. Using it to run Bazel builds is trivial, really. All it took was to stick these few lines in my file, as the instructions say: And just like that, builds started showing up in the UI: EngFlow's standard Invocations browser in the live dashboard. This is one of the weakest parts of the UI, to be honest, because of its poor search capabilities, but the team is hard at work redesigning it. We can even peek into one of the builds to observe cache statistics: EngFlow's Invocation Details drill-down view in the dashboard. Here we can see the cache usage status and that the whole build was cached and took only 17 seconds to run (compared to the 13 minutes it takes otherwise). One thing that did surprise me about this setup is the ability to carry power and Ethernet over a single USB-C cable between the Raspberry Pi 4 and the host. I didn’t know this was possible, and it’s a great feature: if you just want to use the Cache & Carry with your laptop, all you have to do is plug it in with one cable and it just works. Getting Bazel to work against EngFlow was easy. But… as I told you earlier, EngFlow is not just a remote caching and execution service for Bazel: it is an implementation of the RE API. With some imagination, you would think that there might be clients other than Bazel for this API, and you would be right. Buck 2—Meta’s build system—is an obvious example, but there are more niche clients. And the one I discovered while searching whether supported EngFlow and finding that it didn’t was : a compiler wrapper like and that uses RE for caching and remote execution. Building and installing is easy: But note I installed the binary as , not . The rationale for this is that we have to configure . This is done via environment variables, and as I don’t like to pollute my regular environment with program-specific noise, I created a launcher like the following: With this in place, we can run a single compilation command and verify that the wrapper runs: And, with a repeated invocation, we can ensure that caching is in effect: OK, time to pick up the PowerBook G4 from 2004. This little 12” laptop sports a PowerPC G4 (7447A) running at a whopping 1.5 GHz, with 768 MB of RAM, a super-duper-slow 80 GB spinning drive, and a 100 Mbit Ethernet link. Not a lot, you’d say… but this computer was frigging amazing back in the day. I had the iBook G4 back in the day and was always jealous of the PowerBooks I saw in the wild, unaffordable to me, so when I could grab this one for $40 a few years ago, I just had to. EngFlow Cache & Carry in its box next to the PowerBook G4 12” from 2004 we’ll use as a client. Right now, this machine is running Debian forky/sid, one of the only Linux distros out there that still works out of the box on 32-bit PowerPC. As of 2026-11-09, it reports: With the specs out of the way, let’s go back to the premise of the article: showing you what a massive and unsurprising difference (remote) caching can make during compilations, and how, because RE is “just a protocol”, we can leverage it even from old machines. For this, we’ll measure ’s own build from a clean slate, cached and uncached, because it’s a large enough C++ project that takes about an hour to build out of the box—not too little, but also not so much as to be discouraging. Here are the final numbers of the comparison: Let’s break them down. My first attempt at using remote caching was to build with itself in the “default” configuration for and the toolchain installed by Debian. This meant GCC 16.2 and no custom flags to CMake. The clean, fully-cached build took about 8 minutes to finish. This showed a nice ~7x speedup compared to the clean unwrapped build, but somehow it felt like not enough. Why did we still spend 8 minutes to compile something when everything should be cached? I had two theories, but neither of them was valid: The machine doesn’t have a lot of RAM for C++ compiles. Was causing it to thrash, maybe because it buffers files in memory instead of transferring them in chunks? Looking at the code, the answer was no: implements chunked uploads and downloads. The machine’s Ethernet port is slow. Were network transfers slowing things down? Not really either: the output tree is only ~150MB after a build. (Mind you, with Bazel and remote execution, you’d typically use to avoid ferrying large intermediate object files, but that’s not an option with a compiler wrapper that knows nothing about the end-to-end build graph.) With both theories debunked, I had to look elsewhere: CPU consumption. It is no surprise that compiler wrappers are compute-heavy. One specific CPU-bound operation for C++ wrappers is dependency scanning: these wrappers have to (re)discover on their own what a specific compilation action depends on, because nothing tells them upfront. When the wrapper sees something like , it can tell that depends on … but what about the thousands of other implied dependencies like header files and toolchain binaries involved in that innocent invocation? You must know what those are in order to generate a cache key that can disambiguate different environments correctly. Taking some measurements, dependency scanning was indeed the problem as more than 5 out of the total 8 minutes of the cached build were spent on it. Looking through the codebase, I found that implements support for : a helper tool that ships with Clang and is specifically designed to speed up dependency discovery. Would it help here? I switched to Clang, reran the build and… whoops, still no bueno. The unwrapped build was, interestingly, much faster than the GCC one, but did not help: the logs claimed that it succeeded only for 23 out of 124 scanner runs, and so it didn’t materially change the overall result. Looking further, I found that was not being effective: ’s scanner adds to the compile commands, and because precompiled headers (PCHs) had been built without that option, Clang rejected using it. So… what if we disable PCHs by adding to the CMake invocation? Well, it turns out this made no difference either because CMake emits in this case, and apparently probes for these flags wrong, which in turn causes to fall back to slow dependency scanning. I applied a very hacky, targeted fix to to ensure was effective, reran the measurements, and got these numbers. Focus on the PCH? and Dep scanning columns: Yay! End-to-end clean, fully-cached build times fell from 8 minutes to 3 minutes, and those 3 minutes are pretty much the cost of the entire build and account for local non-compiler work such as protobuf generation and linking. Still not zero, but much better than what we started with. Here is the 1-minute video comparing build times. The difference is pretty massive: Recording of recc's own compilation on a PowerBook G4 without and with remote caching. As you can see, there is nothing magical about remote caching, and EngFlow’s Cache & Carry is a little toy to play with this technology. You can, of course, download an open source alternative such as Builbarn , but playing with a preconfigured appliance was fun and removed a lot of the complexities of setting such a system up. When paired with a modern build system such as Bazel, remote caching (and execution) can be trivially enabled and can achieve near 0-second build times when the whole build is cached: these build systems have precise knowledge of the full build graph and know which paths in it are or are not invalidated for a given build. And because they have this knowledge, they can apply optimizations to the whole build such as not downloading intermediate artifacts. But as we have seen, RE is just a protocol, and if you can speak it, you can leverage it from pretty much any machine (even if ancient and “worthless”) and any build system. You’ll hit some inherent limitations, like a compiler wrapper having only a very narrow view of the build, or you’ll have to deal with strange corner cases that don’t allow maximum efficiency, but you can make it work. Keep caching and carrying! The EngFlow Cache & Carry in its box before unpacking it. Cache & Carry is a little toy that packages the EngFlow remote cache (not execution) software into a Raspberry Pi 4 preconfigured to work as an appliance, enclosed into a branded 3D-printed case and packaged into a fancy cardboard box. Mind you: this is what I wanted to achieve with my own EndBOX last year but never went the last mile to distribute it as a truly finished toy. The EngFlow Cache & Carry Raspberry Pi 4, with its custom branded case. Getting started I’m quite used to EngFlow’s product by now because we use it at Snowflake to power our in-house builds and tests, so using the Cache & Carry was no difficulty. And it shouldn’t be for anyone given that configuration and management are pretty easy. Here is the cluster status page right after plugging in the device and connecting to it, showing how this “cluster” is really just a cache and has no remote execution features: EngFlow's standard Cluster status page in the live dashboard. Using it to run Bazel builds is trivial, really. All it took was to stick these few lines in my file, as the instructions say: And just like that, builds started showing up in the UI: EngFlow's standard Invocations browser in the live dashboard. This is one of the weakest parts of the UI, to be honest, because of its poor search capabilities, but the team is hard at work redesigning it. We can even peek into one of the builds to observe cache statistics: EngFlow's Invocation Details drill-down view in the dashboard. Here we can see the cache usage status and that the whole build was cached and took only 17 seconds to run (compared to the 13 minutes it takes otherwise). One thing that did surprise me about this setup is the ability to carry power and Ethernet over a single USB-C cable between the Raspberry Pi 4 and the host. I didn’t know this was possible, and it’s a great feature: if you just want to use the Cache & Carry with your laptop, all you have to do is plug it in with one cable and it just works. Bypassing Bazel Getting Bazel to work against EngFlow was easy. But… as I told you earlier, EngFlow is not just a remote caching and execution service for Bazel: it is an implementation of the RE API. With some imagination, you would think that there might be clients other than Bazel for this API, and you would be right. Buck 2—Meta’s build system—is an obvious example, but there are more niche clients. And the one I discovered while searching whether supported EngFlow and finding that it didn’t was : a compiler wrapper like and that uses RE for caching and remote execution. Building and installing is easy: But note I installed the binary as , not . The rationale for this is that we have to configure . This is done via environment variables, and as I don’t like to pollute my regular environment with program-specific noise, I created a launcher like the following: With this in place, we can run a single compilation command and verify that the wrapper runs: And, with a repeated invocation, we can ensure that caching is in effect: Let’s go retro OK, time to pick up the PowerBook G4 from 2004. This little 12” laptop sports a PowerPC G4 (7447A) running at a whopping 1.5 GHz, with 768 MB of RAM, a super-duper-slow 80 GB spinning drive, and a 100 Mbit Ethernet link. Not a lot, you’d say… but this computer was frigging amazing back in the day. I had the iBook G4 back in the day and was always jealous of the PowerBooks I saw in the wild, unaffordable to me, so when I could grab this one for $40 a few years ago, I just had to. EngFlow Cache & Carry in its box next to the PowerBook G4 12” from 2004 we’ll use as a client. Right now, this machine is running Debian forky/sid, one of the only Linux distros out there that still works out of the box on 32-bit PowerPC. As of 2026-11-09, it reports: With the specs out of the way, let’s go back to the premise of the article: showing you what a massive and unsurprising difference (remote) caching can make during compilations, and how, because RE is “just a protocol”, we can leverage it even from old machines. For this, we’ll measure ’s own build from a clean slate, cached and uncached, because it’s a large enough C++ project that takes about an hour to build out of the box—not too little, but also not so much as to be discouraging. Here are the final numbers of the comparison: Let’s break them down. First build, slow My first attempt at using remote caching was to build with itself in the “default” configuration for and the toolchain installed by Debian. This meant GCC 16.2 and no custom flags to CMake. The clean, fully-cached build took about 8 minutes to finish. This showed a nice ~7x speedup compared to the clean unwrapped build, but somehow it felt like not enough. Why did we still spend 8 minutes to compile something when everything should be cached? I had two theories, but neither of them was valid: The machine doesn’t have a lot of RAM for C++ compiles. Was causing it to thrash, maybe because it buffers files in memory instead of transferring them in chunks? Looking at the code, the answer was no: implements chunked uploads and downloads. The machine’s Ethernet port is slow. Were network transfers slowing things down? Not really either: the output tree is only ~150MB after a build. (Mind you, with Bazel and remote execution, you’d typically use to avoid ferrying large intermediate object files, but that’s not an option with a compiler wrapper that knows nothing about the end-to-end build graph.)

0 views
David Bushell Yesterday

Visited links are purple

A few weeks ago Kev Quirk asked: Do you use on links? — and I said to myself: No! I don’t use visited links… but why don’t I use visited links? TL;DR: I’ve gone and styled visited links, and they’re purple. I did a cursory five minutes of research and found Wikipedia still styles visited links (blue to purple). MDN does too. As does the GOV.UK design system — they have a special class to opt-out of the visited style. BBC News turns black links to grey, how emo. I decided to style visited links within my body copy. Which to be fair is most links but not the big headings on my blog index for example. I was too intimidated to turn those purple. Maybe you can give me the courage to style those too? It looked a bit weird at first but I like the colourful additions to my blog. WCAG 2.2 guidelines have no requirement to differentiate visited from unvisited links. Regardless, visited links should meet all the same criteria applied to any link. There is one specific exception ( following this discussion ). Because web browsers have strict style limitations and technical constraints due to privacy concerns : Where color alone distinguishes between visited and unvisited links, it does not result in a failure of this Success Criterion, even where the contrast between the two link colors is below 3:1. SC 1.4.1 Use of Color (Level A) This means that a subtle contrast different in blue vs purple is not considered an accessibility failure under that rule. Web accessibility is all about ensuring a comparable experience for all users. Nothing can be perfectly accessible though. I think this small enhancement is fine, even if not everyone can see the difference. It’s a rare exception though. I’m conscious that too often excuses are made to ignore UX parity. I think any higher contrast or another visual indicator would be off-putting and impede readability. Forgoing any style doesn’t help anyone either. If blue and purple is good enough for those examples mentioned above, I’m on board. Bryce Wray commented on Kev Quirk’s blog to note that Chromium browsers introduced link partitioning in 2025. I had no idea this was a thing! What do you think, do you care about visited link styles? I found Kev Quirk’s blog on bubbles.town . Bubbles is a charming social RSS aggregator. It’s doing a great job at supporting small personal blogs and promoting non-toxic topics. Bubbles is far from perfect though. It probably needs a tech filter. It certainly needs more participation in comments and voting. The developer is actively working to improve the site and a community is growing, so that’s cool to see. I hope Bubbles can thrive and remain human. This is the web we need to bypass the dead web Google et. al. are gatekeeping. Let them wall themselves in with their slop! Thanks for reading! Follow me on Mastodon and Bluesky . Subscribe to my Blog and Notes or Combined feeds.

0 views
Brain Baking Yesterday

In Defence Of Housekeeping

When live gives you lemons, make lemonade. Life never gives me lemons: I live in Belgium, not in Spain or Italy. Live gives me chores. Endless chores that keep on piling up the typical way they typically pile up in households with two small kids. There’s multiple heaps of laundry loitering around (ha!) the house, a suspicious amount of missing or non-matching socks, shoes that never make it into the cabinet in the hallway, toys getting sucked up by the black hole also known as the dusty areas under the cupboard and sofa, forever-dirty windows thanks to grubby little fingers at a height of about ten to twenty centimetres, and dog hair—so much dog hair. No lemon squeezing here but rather housekeeping, and a lot of it. Two minutes after I helped the youngest out of his chair this morning he was walking around with one sock. The black hole must have claimed the other as I couldn’t find it and was in a rush to get to work. Sometimes the black hole is merciful and spits out something we’ve been looking for. Sometimes it’s not. And yet I like the act of housekeeping, just as I like the act of cleaning up a messy piece of code to turn it into a neatly ordered, sense-making block that no longer requires a second look. Throwing clothes into the washing machine, the tumble dryer, folding and sorting them into piles, and finally putting them back where they belong in our respective bedrooms is a part of life that is severely underappreciated and perhaps also misunderstood. Life is often mundane. Doing mundane chores only adds to that desperate feeling of uselessness. Shouldn’t I be doing something exciting, grand, and life-altering right now instead of vacuuming and cleaning up the kids’ mess? Thinking like this often leaves me frustrated as I can’t escape the current situation we are in. So instead of approaching chores as a chore—even that word feels painful to pronounce—I try to approach them as a housekeeping task. There is no difference, except that airing, dusting, vacuuming, adjusting, tidying, and cleaning sound more positive than doing a chore . Let’s celebrate mundaneness. Housekeeping yields two things. First, the work in and around the house is done: you are indeed entitled to the sweat of your brow. Is that BioShock reference too obtuse? I think not. It’s always a pleasure to pause and look around: things are clean! Revel in that moment because the dog is about to shake his tail spreading more gunk and the kids are about to storm up to these boxes of neatly stored toys to empty them after than you can blink. Is that one of the cats I hear making puke noises again? Second, doing the work invites a few mindful moments 1 where your undivided attention to these tasks might help to calm your mind, leaving you with less instead of more frustration. Remember, life is mundane. You might feel upset reading that sentence, claiming I generalised and your life is special. Fine: my life is currently mundane. Does that make you feel better? How to deal with mundaneness is one of life’s greatest challenges. Simply enjoying both the process and the result of housekeeping well get you a long way. Why does mundane sound so negative? Its meaning, according to various dictionaries, is belonging to or characteristic of the earth —as opposed to spiritual things. Biking to work is mundane. Eating is mundane. Reading a book is mundane. Typing these words in a text editor to upload them onto this website is mundane. All of these can also be called chores if the interpretation is a routine task or job . When looking for other blog posts on housekeeping, I can only find technical blog housekeeping announcements: small tweaks made to the theme, the engine, the publication process. Tiny fixes for the RSS feed, a slight repositioning of the header image. Doing all these things—also known as fiddling with the blog—is fun and yields a more pleasant blog. Of course, the big difference between that housekeeping and real housekeeping is that real housekeeping can overwhelm you (especially if you have two young kids) and real housekeeping keeps on coming back again and again in exactly the same form. Once that header image on your blog has been repositioned, it’s not likely to fall down again, but the kitchen floor will need vacuuming again and again. The biggest difference between virtual “blog” housekeeping and real “house” housekeeping is that you share that house and expect housemates to take part in the cleaning. The young kids just don’t because they can’t. Also, my interpretation of housekeeping isn’t the same as my wife’s. She often accuses me of Monica-behaviour (and I don’t think I need to explain this one). This is often the cause of squabbles that undermine my entire point, but still. Don’t forget to dust the duster. I didn’t realise there’s an entire Mindful Cleaning movement that works “by placing attention on present-moment sensations instead of thoughts about finishing, fixing, or judging the mess”. Sounds exciting, where’s that mop?  ↩︎ Related topics: / housekeeping / parenting / By Wouter Groeneveld on 16 November 2026.  Reply via email . I didn’t realise there’s an entire Mindful Cleaning movement that works “by placing attention on present-moment sensations instead of thoughts about finishing, fixing, or judging the mess”. Sounds exciting, where’s that mop?  ↩︎

0 views
Stratechery Yesterday

Obituaries

Listen to this post: Dearly beloved, we are gathered here today to get through this thing called life. ―  Prince Jensen Huang resigned as CEO of Nvidia on Thursday, November 12, 2026. He had been CEO of the company ever since he founded it on April 5, 1993. Nvidia started out as a graphics card company; Huang’s exit — nominally because of affairs with a buyer who may or may not have been a Chinese spy, as well as at least two company employees — was ultimately because Nvidia was, by the powers that be, considered a defense contractor, and Huang never adjusted. In this, Huang was the ultimate victim of his own success. Nvidia’s first graphics cards simply translated triangles into pixels. The 2001 GeForce 3 introduced a programmable pixel shader, which meant a video card’s functionality was no longer pre-defined. The killer move, however — which almost killed the company — was the 2006 introduction of CUDA, which was an API for programming those shaders. For much of the next 16 years Huang sold the gospel of CUDA , convinced it was the platform for a future of accelerated computing; then, in 2022, came ChatGPT. ChatGPT made clear what GPUs were most useful for: large language models, and Nvidia made the best GPUs. More than that, Huang had realized that the future of accelerated computing was not just chips, but rather systems; that drove the purchase of Mellanox in 2019, which provided the technology for Nvidia to scale out from chips, setting it down the road to being the best option for the sort of large scale training that made large language models possible. For this Nvidia charged very high prices, with margins exceeding 70%, because they could: they had the platform, the technology, and, over the past few years as others sought to make their own chips, the supply chain. Huang’s impact on the Asian supply chain in particular cannot be overstated. Byrne Hobart, in his book Boom: Bubbles and the End of Stagnation , notes that one of the benefits of bubbles is that they provide a coordination function, as the fear of missing out incentivizes multiple entities with interlocking dependencies to all commit to building at the same time; Huang was, in many respects, a one-person coordination function. Huang gave multiple keynotes a year, which he explained as how he communicated with the ecosystem broadly, and was famous for eating and drinking his way across Asia in particular, convincing reluctant suppliers that the AI boom was real, and locking up the lion’s share of their output for Nvidia and its stunning commitment to a yearly release cycle. Huang, who had toiled for decades in relative obscurity, known primarily to gamers, and only barely to the tech industry, became a rock star, swarmed wherever he went, always in his black leather jacket. He also, to his ultimate detriment, became a political figure. Nvidia had long had no presence in either Washington D.C. or Beijing , and was blindsided by the Biden administration’s imposition of chip controls in 2022. Nvidia’s immediate reaction to those initial controls, which restricted exports based on interconnect speed, was to immediately release the A800 in China, which was an A100 with a third of its interconnects disabled. That set the tone for the conflict that I believe was the true cause of Huang’s downfall: Huang strenuously disagreed with the various iterations of the chip ban, and Nvidia only bothered to do the bare minimum to abide by the letter of the law, never the spirit. That approach, in all likelihood, extended to Nvidia’s oversight (or lack thereof) of Singaporean entities that were alleged to be fronts for chips for China. Some of these were legal: Chinese companies are allowed to access cloud computing facilities outside of China that house leading edge Nvidia chips, although chip control advocates are seeking to close that loophole; some were not legal, in that the chips were ultimately ferried to China. Megaspeed, whose executive, Alice Huang, was the first person Huang was alleged to have an affair with, was officially in the former business, and alleged to actually be in the latter. There are whispers that Megaspeed’s ultimate customer was not, or not just, Alibaba, but also the Chinese military. These are whispers, to be clear; it was the affairs that were enough, at least publicly, to lead to Huang’s resignation. Nvidia’s durability — at least that of their margins — was already an open question. Start with CUDA: when the market was software written for GPUs, a platform — including not just an API but a huge set of libraries targeting every conceivable market for accelerated computing — was a nearly impenetrable moat. That is why LLMs like GPT were written using CUDA. LLMs, though, both brought Nvidia its incredible success, and also seeded its greatest risks. The first is that the vast majority of developer energy moved up the stack, using the LLM’s API as a platform; for most companies, CUDA didn’t matter. The second is that Anthropic, which found funding from Amazon and Google, originally also-rans to the Microsoft-OpenAI partnership, had to learn how to use Trainium chips from Amazon and TPUs from Google; this was difficult, but ultimately meant that the leading LLM did not have any Nvidia dependency at all (Anthropic runs on Nvidia chips, because they learned how to run on anything). The third is the most devastating: LLMs are so effective at writing software that any software-based moat will inevitably dry up. Nvidia’s second moat, its networking and systems approach, is more durable, although challenged by Google (and by extension, Anthropic) in particular. Nvidia’s systems approach is also the easy answer for enterprises and governments seeking to own their own hardware, and neoclouds seeking to build data centers that are independent of the hyperscalers. The latter market rents chips to both hyperscalers and frontier labs, as well as individual companies, but is challenged as obvious sources of power and political permission disappear. They also need money to afford Nvidia’s systems, which has increasingly come from Nvidia itself. Nvidia’s third moat is its mastery of the supply chain, and this is the moat that suddenly looks much more tenuous. Huang has bragged about the fact that he has never signed a contract with TSMC over their three decades of partnership; it’s a relationship built on a handshake and mutual trust with Huang specifically. This is, per the point above, a very Asian way of doing business, and Huang was the master of that. It’s questionable whether or not that trust and willingness to commit will persist when the handshake is no longer there. Nvidia’s most important moat, however, was the degree to which it was not a normal company. Huang organized Nvidia as an exoskeleton around one man: himself. There is very little management at Nvidia — Huang famously has 60 direct reports — for the same reason that there was no succession plan: Nvidia could move so quickly and so aggressively because the company still behaved as if it were a startup. Indeed, lots of established companies — including those still led by founders — claim to still be a startup, but only one, Nvidia, really was organized that way. That paid off with all of those successful bets; it also created an environment where there were no checks on Huang. And, now that he is gone, the exoskeleton is much more poised to collapse than you might expect. The first part of the AI buildout was funded by free cash flow, particularly from the hyperscalers. Then, during the second half of last year, nearly everyone started turning to debt. Now — starting with Google this past summer — the primary funding mechanism is equity. In the long run, this should come full circle, and future build-outs will once again be funded by free cash flow, generated from the infrastructure that is being built now. The problem for Nvidia is that the companies most able to navigate this progression are the hyperscalers, which have the resources to challenge Nvidia’s hold on the chip market. To that end, Nvidia has been very eager to ensure that it is selling not only to the hyperscalers, but to other would-be competitors, like the neoclouds; there will likely, at some point, be serious questions raised about exactly how Nvidia went about allocating compute in pursuit of this goal. The neoclouds, however, are at a meaningful disadvantage when it comes to funding. First, their debt is significantly more expensive, given that their only business is renting GPUs (as opposed to, say, running the most valuable advertising businesses in the world, or the world’s largest e-commerce platform, or dominating enterprise software). Second, their equity is increasingly worthless, given the significant risks attached to that credit. If a neocloud can’t pay, it’s the equity holders that get wiped out first. This led to what I termed Nvidia’s Risky Business : Nvidia increasingly used its cash flow and credit rating to backstop and fund the companies who were buying its chips. Bears called this circular financing; bulls called it a necessary bridge to a future where AI revenue paid down these companies’ debt and drove massive upside. What is undoubtedly true is that this was the sort of risky business that is uniquely possible for a multi-trillion dollar company that is completely controlled by one man, who has made an entire career of pushing the limits and making risky bets — and seeing them pay off. It’s also a series of bets that are unlikely to continue. The neoclouds should be very nervous about losing their patron, along with their creditors and equity holders. The other problem is that the revenue necessary to make all of this build-out work is something on the order of a trillion dollars industry-wide, and it needs to materialize sooner rather than later; if it falls short, it’s the hyperscalers who will prioritize their own compute first, and they will likely be seen as a safe haven for everyone committing to spending. In that world, Nvidia’s neocloud hedge — absent Huang’s willingness to shore the ecosystem up — is liable to turn into a millstone, with their assets flooding the market even as the supply chain ramp-up brings ever more supply to market. Perhaps because he had been preaching the gospel of accelerated computing for so long, Huang was the leading figure pushing back against the doomer narrative surrounding AI. Yes, AI was a massive business opportunity, but it was still normal technology; the safety issues presented were engineering challenges to be solved. What required consideration was not just the next five years, but the next fifty. It’s a view that I am, in full disclosure, generally aligned with. Many others are not. On one hand there are the aforementioned doomers, certain that fast takeoff is imminent — or has already started — as AI increasingly builds better AI in a recursive loop. The end state is an AI that is massively more capable than humans, and uncontrollable by them; the only hope is to ensure that AI is aligned with humanity, by which adherents to this viewpoint mean aligned with them and their view of the world. This contingent is, broadly speaking, represented by Anthropic. And, in this view, actively working to enable and accelerate uncontrollable open weights models is a threat to humanity. On the other hand there are the China hawks, who view AI as a national security issue, and the key to U.S. military dominance in a world where China owns the means of production of most militarily relevant hardware. In this world giving China access to chips is effectively treason against the United States. For both of these groups, Huang was public enemy number one, because of his determination to do what was best for Nvidia’s business, which was to sell chips to China. This made sense for Nvidia for multiple reasons: This last point is worth dwelling on. The chief objection that China hawks in particular make to the idea that it is a good idea to sell China chips in order to prevent them from surpassing the U.S. is that China is already heavily invested in building their own ecosystem. This is certainly true, but I think it underplays the complexity of chips and what is necessary to actually compete with the West. Specifically, the Chinese government can only control the supply side, by spending money and making top-down imperatives; building an ecosystem, however, whether that be for AI specifically, or semiconductors broadly, requires demand as well. Progress in models and systems and chips depends on a feedback loop between the user and the developer, or the customer and the manufacturer. The demand side, however, is much more short-term oriented than the Chinese government might prefer. The model companies’ concern is the other model companies: that means if they are distilling, we must distill too. If they are using Nvidia chips, then we must too. This matters up and down the stack, just as it has always mattered in technology, which really is different in this regard from almost every other industry. A useful analogy is to consider the Internet. Why is it that country after country can profess a desire to have an indigenous alternative to the American tech giants, yet miserably fail in every attempt to generate it? It’s because the tech giants are only a click away! In this regard, China’s Great Firewall was one of the most brilliant geostrategic moves ever: by virtue of cutting off access to the American tech companies, China created the demand that pulled China’s own tech giants into existence. You need demand, not just desire. This is why China itself is currently banning the import of Nvidia’s two-generations-behind H20 chips, after the Trump administration foolishly banned their import, only to reverse themselves while bragging about the reason to do so — i.e. keep China addicted to Nvidia. China is replaying the Great Firewall playbook: force its internal demand to use its much worse alternatives, because that is the only way those alternatives become better. And, in the long run, those alternatives, and the entire ecosystem that supports them, will flood the world. Huang was, to be perfectly clear, talking his book when it came to his insistence on selling chips to China, or, in the case of the current controversy, allegedly turning a blind eye towards nominally legal buyers who were intent on violating the chip ban; that doesn’t mean, however, that he was wrong about the long-term effects, at least if AI is a normal technology (and I haven’t gotten to the risks this imposes on the Taiwan Strait ). What is clear now, however, is that the doomers and the hawks have won: AI, at least in terms of policy, will be treated as anything but normal, and in the fullness of time, it is the U.S. chip ecosystem that will pay the ultimate price. The birth date for the last obituary was the invention of the transistor; the birth date for this one is the founding of Intel. Robert Noyce and co-founders had spent the last decade at Fairchild Semiconductor, but were constrained by their parent company’s fundamental conservatism and chafed at their lack of upside. Arthur Rock effectively created the venture capital industry in Silicon Valley through his financing of Intel; Noyce created the concept of the commercial-market-led technology company that made it up in volume. At that time the largest single purchaser of chips was the U.S. government; Noyce realized that the fastest way to progress was to make chips for everyone, including the government, and on his terms, not theirs. From that point on, tech as an industry operated with the assumption that the world was its customer, and that their own executives defined the route to progress. Government was free to buy their products, but not to define them, and if they wanted the best products, then they best not restrict who they could sell them to either. That’s how important volume was, and, over time, it became clear how powerful and essential ecosystems were. Other industries operate differently; I thought this article in the Financial Times was striking, and not just because of the headline: Bessent says Nvidia needs ‘shake-up’ with external CEO : Bessent said that it is important “to examine whether Nvidia had a culture of compliance around the US export restrictions” and whether its top executives knew they were being skirted. “Was it a low-level fail? Was it in the middle, or was it up in the C-suite? And who knew? These shell companies were obvious, they were being investigated, and did Nvidia use them as a workaround to sell chips to the Chinese?” he said. “Were they pressing these seemingly illicit sales to goose their revenues and their earnings?” The Treasury secretary added that Nvidia needs to provide answers within a matter of days: “I think we need answers in the next 10 days from the board and from the outside counsel”. He added that “we will see if Nvidia intentionally ignored the warning signs of these shell companies, many of them in Singapore”. Bessent has pointed the spotlight on Howard Lutnick’s commerce department for failing to ban exports of Nvidia’s chips to Megaspeed, the Singaporean company linked to the alleged Chinese spy who had an affair with Huang. US authorities have been investigating whether Megaspeed, which builds AI data centres in Asia, has been involved in smuggling high-end Nvidia chips, which are powering the AI boom, into China. Bessent has blamed the commerce department more than once this week for failing to ban exports of Nvidia’s chips to Megaspeed. Commerce delayed efforts to add Megaspeed to Washington’s blacklist of companies the US considers a risk to national security. “Many other departments were pushing for this, and commerce had the pen,” Bessent repeated on Sunday. “We have all the rules we need. We just need them enforced.” Lutnick is under mounting pressure from within the administration and from Congress, which has demanded that he explain how he allowed chips to be exported to a company linked to an alleged Chinese spy. US lawmakers are also accelerating efforts to crack down on advanced AI chips reaching China. If Nvidia “were a bank and they hadn’t complied like this, the [Treasury’s Office of the Comptroller of the Currency] would be in there right now, and a lot of the executives would be removed”, Bessent said. Scott Bessent is the Treasury Secretary, which is to say that he operates an apparatus that is arguably just as, if not more, powerful than the U.S. military. Treasury controls the flow of money around the world, and its soldiers are the banks; if Treasury has sanctioned an entity, then it is up to banks to ensure those sanctions are enforced. They need to know their customers, and they need to analyze every dollar flowing through their systems; if they do not, they are breaking the law. Ignorance is not a defense. Ignorance, on the other hand, has been tech’s defense basically forever. Yes, there are bad people in the world, which means that if the whole world is using your products, they will be used to do bad things. That was considered an acceptable trade-off, however, because most people are good and the benefits that come from scale — and an avoidance of the friction that comes from trying to track the activities of billions of people — were worth the risk. This was certainly what Huang believed, and, to his detriment, he aggressively belittled anyone who dared disagree (the extent to which Huang is despised in Washington D.C., as evidenced by the giddiness and bloodlust online last week, is extraordinary). In Huang’s view, as long as the Commerce Department said that it was ok to sell to Megaspeed, then it was ok to sell to Megaspeed — and to sleep with one of its executives. Bessent views the matter entirely differently: he thinks that Nvidia should operate like a bank. It’s not just their job to abide by the letter of the law, but the spirit of it. It’s not the government’s job to enforce the chip ban, but the company’s. An attestation from a Singaporean customer that chips aren’t going to China isn’t worth anything; actually putting boots on the ground and ensuring compliance is the only way Nvidia cannot be in violation of the U.S. government’s dictates. Again, for tech, this is extraordinary, even if it is normal in banking. Extraordinary enough, in fact, that Huang is being labeled by many — not just in Washington, but also in the finance world — as a traitor. Huang, in my estimation, is just a businessman who cares about nothing else other than Nvidia; that worked out great when he made GPUs for gaming that no one in government cared about or understood. That’s just a normal technology; when AI was deemed to be anything but normal, however, business as usual wasn’t just over: it was a crime, and it cost Huang his job — and maybe more. It’s worth pointing out that while this episode was shocking, I should not, in retrospect, have been surprised. Earlier this year, when Anthropic faced off against the U.S. government, I wrote in Anthropic and Alignment : At the same time, what is the standard by which it should be decided what is allowed and not allowed if not laws, which are passed by an elected Congress? Anthropic’s position is that Amodei — who I am using as a stand-in for Anthropic’s management and its board — ought to decide what its models are used for, despite the fact that Amodei is not elected and not accountable to the public. And, on the second point, who decides when and in what way American military capabilities are used? That is the responsibility of the Department of War, which ultimately answers to the President, who also is elected. Once again, however, Anthropic’s position is that an unaccountable Amodei can unilaterally restrict what its models are used for. My point in that Article was that if AI is as exceptional as claimed, then the government will demand a say in how it is used, whether or not private company executives agree; moreover, that was a good thing, or at least a least-bad thing. The same analysis applies to Nvidia and Huang: Huang may have fundamentally disagreed with the chip ban, but at the end of the day, it was his duty to accept that it was the law of the land. To the extent he did not do so is ultimately his responsibility. Perhaps the most tragic aspect of the Huang affair is the extent to which he was the personification of The American Dream. Huang was sent to America as a nine year old who didn’t speak a lick of English, and somehow ended up at a boarding school in rural Kentucky for misbehaving youth, where he would trade help on homework for protection from bullies. From there he went to a state school in Oregon, became an engineer, before famously ending up at a Denny’s diner where he and his co-founders decided to start Nvidia. Then, over the following 33 years, he took risk after risk, created innovation after innovation, until he became a hero not just in the United States, but across the world. And then, you might say, he flew too close to the sun. Certainly he did so in his private life: it’s not just the affairs, which may be par for the course for rich and powerful men; rather, it’s notable just how reckless the affairs were. The ones that were revealed last week were with a customer and potential Chinese spy, and two of his own employees; both are fireable offenses. Meanwhile, the way in which Huang treated his opponents — including actively seeking to get people fired who he felt stood in the way of getting his way — bore the marks of a man who has constructed a life, and a company, where no one can or will stand up to him. That is a recipe for disaster, and here we are. At the same time, it really does bum me out that what ultimately doomed Huang was a set of rules that really are a big departure from the status quo — and, in my opinion, misguided. Huang’s actions — or lack of actions, in terms of enforcing compliance with the chip ban — are treated as if they are an obvious crime, when in fact this is a dramatic shift in expectations from a short time ago (although there is certainly a callback to previous controversies around selling advanced computing to adversaries). Huang should have adjusted and accepted the new reality, make no mistake; I’m sympathetic to the fact he couldn’t, and recognize the failure in not having structures in place that could have warned him about the dangerous road he was embarking on, even as I realize Huang’s willingness to navigate dangerous roads is precisely why he was an American success story. Beyond Huang, however, there is the question of how we compete with China, and I am disappointed, although not surprised, that we are pursuing the route of bans and forced compliance, instead of innovation and openness. Make no mistake, I am no China dove, and I have been very early to the argument that we erred greatly in willingly exporting entire industries to the communist state. That, however, makes it even more imperative that we sprint in the opposite direction of China, towards more freedom in a bet on inventing new industries, not more control and a focus on gatekeeping what we have. I get the logic, particularly in the case of AI, which is extraordinary; I lament the implications of what it says about our belief in ourselves and our ability to stay ahead. Presented by As AI models improve, you might expect the market will consolidate around a handful of providers. But user data tells a different story. Among OpenRouter customers spending at least $100 a month, the median number of AI model families in use has doubled over the past year , from three to six. Even as AI models become more capable, businesses are using more models, not fewer. This makes sense when you consider that intelligence isn’t one-dimensional. A model that’s exceptional at writing code may not be the fastest at answering customer questions, or the most economical at processing millions of documents. The question is shifting from “which AI model should we build on?” to “which model should handle each task?” The answer, increasingly, is ‘it depends.’ Which is great for flexibility, less great for simplicity. OpenRouter, which recently joined Stripe , allows businesses to access and govern hundreds of models with the flexibility to optimize for cost, performance, and reliability as the market evolves. Learn more at openrouter.ai . The first and most obvious reason is that it is good for a company to be able to sell to everyone, including the second largest market in the world (China has traditionally made up about a quarter of Nvidia’s sales, long before AI was even a thing). The second is that China is a large enough market, with sufficiently motivated and capable developers, that it can actually create a meaningful CUDA alternative; furthermore, such an alternative would, in the long run, not be constrained to China, but rather compete with Nvidia all over the world. The third is that China’s AI ecosystem, unconstrained as it is by such niceties as terms of service, distills frontier lab models and releases them as open weights models; those models, meanwhile, are tuned to run on Nvidia chips. These models give Nvidia leverage versus the frontier labs all over the world, including in the West. The fourth is that China is uniquely positioned to create a fundamentally different approach to systems design using optical interconnects to scale out training clusters. This is much more inefficient in terms of power usage than Nvidia’s more elegant designs, but China is also much more capable of manufacturing excess amounts of power to make it work. The fifth is that, in the very long run, China can create an entire alternative supply chain for semiconductors broadly, rendering Nvidia’s mastery of the Western-aligned supply chain much less important.

0 views
iDiallo Yesterday

People will lose their jobs over GTA 6

GTA 6 is about to come out, and I've seen many people online joking that they'll take the week off to play it. I'm not a big fan of GTA, but the series did leave an impression on me. The last one I played was Vice City. It was an amazing game, and I still listen to its songs. It's where I first heard "Your Love" by The Outfield. I love the game, but I don't have the luxury of playing video games the way I used to. All this to say that I understand people taking time off. When Metal Gear Solid 5 came out, I didn't just take the day off. I slowly prioritized the game over the work I was supposed to be doing. I had a contract with a client who couldn't make up his mind about a project. I delivered the project, a flower subscription service, in the span of a month. He had paid a team in Ukraine $25k to hand him a broken Drupal website. I rewrote it for half the price, and he finally had the ability to manage the website on his own. But a website isn't complete once it's delivered. There are bugs and feature requests, or the client might just change his mind about things. So he retained me for maintenance. We launched in the summer, and he couldn't have been happier. Customers flocked in, and all was well. Now it was time for maintenance. The problem was that the client called me every day. He wanted to change everything, all the time. My expert opinion had no effect on him, he just wanted things to be different. I was getting paid for it, so I made the changes. I was constantly on the phone with him that summer. It was annoying, but that's the life of a contractor. I could have kept going, but September 1st was the release date for MGS 5. I could have managed if I had put my mind to it. But I had already pre-ordered the game, and I had played Ground Zeroes, the prelude that came out earlier, at least a thousand times. Nothing was going to get between me and my game. When the game came out, I stopped answering calls. Instead, I replied to emails. Whenever the title "A Hideo Kojima Game by Hideo Kojima Produced by Hideo Himself" appeared on the screen, I used the time to respond to email. Some of the work he requested was never completed, because I knew he would change his mind. The more I ignored the client's calls, the more frustrated he became, and the more of the story I missed. "Who is Skull Face again?" I didn't want to miss a second of the dialogue. At one point, the client called and I didn't pick up, as usual. He sent a text message: "PICK UP NOW!" I knew it was serious. So I didn't pick up right away, because doing so would have made it look like I had been intentionally ignoring him. I called him back some minutes later, only to find out it was nothing urgent. The call turned into a therapy session. He was frustrated that people had flocked to the website during the grand opening, yet it had translated into few subscriptions. The website was only a few months old, mind you. I could have given better advice, but Quiet was shooting at me from a distance, and I hadn't figured out how to win that battle yet. Besides, I was there to do technical work, not to solve his business problems. He sensed it. When he found a bug on the website, I was eager to fix it. When he wanted to discuss ways to make the subscriptions more appealing to customers, I checked out. Eventually, he confronted me and asked whether I was all in on this business. I wasn't. I was busy thinking that the Huey character design was lazy. Why did they recycle Otacon? Couldn't they come up with something new instead of playing on our nostalgia? Anyway, I knew a couple of friends who had just started a web consulting business. They could solve business problems and do the development work, so they would be more helpful than I was. Plus, it wouldn't make sense for him to retain me while working with them. I found them their very first client, and the client found a better team. My consulting pattern used to be to deliver a job and then, during the maintenance phase, look for other clients. That way, I never had downtime. But this time, I used the downtime to play my game. Remember, this was 2015. $13k went so much further then than it does now. You can imagine my disappointment when the game ended abruptly, as if the producer had been kicked out mid-development. Kojima had been ousted from Konami. Still, it was a good game. But I needed money to survive, so for my next contract, I ended up charging $18,000 for a single static HTML page . What I mean is, as an MGS fan, I understand the GTA fans.

0 views
Andrew Healey Yesterday

Introducing the World's Strongest Sokoban Solver

Using agents to do state-of-the-art solver research.

0 views
Unsung Yesterday

“Could a modern chess engine on a Pentium have defeated Kasparov in 1996?”

On February 10th, 1996, a computer program (IBM’s Deep Blue) defeated the reigning chess champion (Garry Kasparov) in a game of chess for the first time. Deep Blue would go on to lose the match, but a year later, would get its revenge in a 3.5-2.5 rematch. In order to achieve this feat, IBM spent several million dollars on research and development building a special-purpose computer specifically for this purpose. Deep Blue itself (the victorious, upgraded version) was powered by 30 P2SC processors each running at roughly 120 MHz, and (most importantly) 480 custom VLSI chess co-processors used to perform chess-specific operations like move generation and position evaluation. When it was built, Deep Blue was the 259th most powerful supercomputer in the world, and could analyze over 200 million chess positions per second. […] Skill in chess is measured by an Elo score. […] The best chess engines that you could run on commodity 1996 hardware were rated roughly 2300 Elo. Deep Blue was rated 2700. In contrast, modern Stockfish is rated closer to 3700. This raises the question: how much of the advantage of a modern chess engine comes down to the fact that a modern desktop computer is extremely fast, and how much of the advantage comes down to the fact that we’ve just gotten better at programming chess engines? This post by Nicholas Carlini is a more technical post than I usually share in this neck of the woods, but I wanted to to this for this ending: The first time we humans do anything, we rarely do it efficiently. Once we see a path that would clearly work if only we scaled it up, we do just that. Only later, after we’ve had the time to better understand the problem at hand, do we come to recognize that we could have achieved the same feat much more efficiently by being clever. I mostly did this for fun, because whether I could have beaten Kasparov with commodity hardware sounded like a fun question to answer, but I also hoped it would provide a tiny bit of insight into the rate of algorithmic progress in “AI”. Chess engines in 1997 were already quite advanced; after all, Alan Turing helped write the first chess program nearly fifty years earlier. So I suppose at the time it seemed plausible that maybe there weren’t that many more ideas left to find. And with hindsight it’s easy to see that’s wrong. This is interesting to me less in the AI context, but more in the “people getting better at understanding and programming things” sense. It reminds me of Racing the Beam , a fun 2009 book by Ian Bogost and Nick Montfort, chronicling the life of a certain Atari 2600 videogame console, released in 1977. The console predates me, although it has been on the market for many years. Toward the end of its life, the programmers understood it so well and were so much in sync – literally and figuratively – with its antiquated hardware, they made games with graphics so great they seemed to belong to the next generation of hardware. I am pretty sure if you were to grab some mid-1980s Atari 2600 games and go back in time to 1977, the creators of the console itself would tell you these were not possible to make. This is an 1977 launch title Combat, followed by the famous Pitfall! from 1982. I know they will both look primitive from the flattened perspective of today, but believe me – Pitfall! was a masterpiece that inspired a generation:

0 views

Now · November 2026

I’m long overdue in writing one of these updates. There’s a lot on my mind at present. I keep meaning to sit down and write about it, but have been finding it difficult to put words on the page. The elephant in the room, of course, is LLMs: specifically, what it means to be a “software engineer”—if that is even still the right term—in this LLM age. Right now my main focus is trying to figure out how to achieve more balance in my life.

0 views
Manuel Moreale 2 days ago

Friends with new sites

Always makes me happy to blog about friends with new sites, and let more people know about their existence and drive extra traffic in their direction. Carl has pushed online his new digital publication Mouthprint and has written about it on his blog . Mattia has refreshed his site to better reflect his always-changing life. I’m sure he’s gonna tweak it 45 more times in the next 12 months, while he tries to figure out what to do with his life between taking photos, making apps, writing newsletters, and a million other things. And lastly, Kevin ’s new version of Overkill is finally live. I say finally because I worked on this one for what feels like 12 years and I’m glad it’s now online. He’s constantly asking his AI slave to make random changes to the codebase, so don’t blam Thank you for using and believing in RSS. You're awesome. Send me an email :: Sign my guestbook :: Become a supporter

0 views
David Bushell 2 days ago

RSS Club #010: Chatbots

This is an RSS-only post, thank you for subscribing :) My RSS-only posts have a permalink for those who prefer reading from my website. That link is hidden and unindexed. URLs can be linked to but in the spirit of “RSS Club” I’d prefer if my RSS-only posts were only linked to from other RSS-only content. Warning: this post contains heavy slop-talk! I’m trying my best to leave them behind but everywhere I go is inundated with chatbot bullshit. Below are some scattered thoughts on the topic that I’m keen to leave behind. So I’m dumping them on you! Aha! I’ve noticed a change in how people comment on my blog posts. I think the reason is they are only reading a chatbot summary . They question stuff that was meticulously answered in the post. They misinterpret quite obvious jokes as serious statements. They jump to extreme conclusions. They latch on to small tangential details that bear little consequence to the main content. They fail to understand the mood and tone of my post. I don’t always know if a commenter read and understood what I wrote. It’s quite apparent though when they’ve read something that I never wrote. I’ve been blogging long enough to know when I failed to make myself clear. Nope, this is a new phenomenon. There is a distinct change in how a few people comment. Far from everyone, and it’s those “normal” comments that convince me there is something off with the others. A comment based on a wrong chatbot summary is far more discouraging than any negative or insulting criticism. I have a thick skin for the latter. I’m seriously considering getting my website banned from Lobste.rs for this reason. On sites like Hacker News the dumb comments get lost in the noise. Lobsters is small enough that the bullshit sticks. Suspected comments based on a chatbot summary have a habit of arriving earlier than others. It’s exhausting to deal with. Maybe I just have to ignore it? One of life’s small joys is referring to all chatbot output as “slop” and “vibe-code”. It really riles people up because their chatbot usage is different . You must understand this. They can get their chatbot to sing, it’s others who are failing to finesse. Many in the web industry have spent the last year (or more) trying to predict the output of their chatbot. The fact that non-determinism † is fundamental to chatbots working at all doesn’t register with slopaholics. “You’re prompting it wrong” is the entire conversation . † Technically PRNG but it’s sufficiently random to mangle human knowledge. Everyone likes the taste of their own slop. No one seems to grasp that perhaps they too have stopped learning and started renting a sorry excuse for brain activity from a bunch of fash-tech broligarch grifters. These billionaire psychopaths invented “AI” by beating all meaning out of the term. If you watch Ghost in the Machine you’ll learn that the term “Artificial Intelligence” has deeply dehumanising origins. However, most of us think of the cool future promised by sci-fi books and movies. Regardless of what “AI” means, it’s quite the indictment that people have settled for the saddest most pathetic definition. Once distinguished web developers are now advocating for cognitive surrender to Google and their “Modern Web Guidance” . These people have all but killed web education — but don’t worry, Google is here! They have the gall to cry foul when you dare challenge their agenda. “Why can’t we have a normal conversation?” they ask insincerely. Because they’ve chosen to ignore all moral and ethical issues . They killed web education and now they’re trying to educate an f-ing chatbot. That ain’t normal. The deskilling of the web industry and destruction of related professions has been rapid. I won’t be forgetting those who participated. The quality of chatbot output is the least of my concerns. Although it does reveal how low the bar is (lower than I assumed necessary). And how sorely lacking in taste vibe-coders are. The funny thing is that vibe-coders can’t escape the negative effects of all this slop. They too see the unreadable emails, the worthless content, the broken websites and apps. And they have the audacity to moan about it! The world is burning around them but their fire is under control. When the AI industrial complex collapses so will the economy. If Anthropic and OpenAI are allowed to IPO, the door is open for “safer” investment to flow in. That significantly increases the likelihood of bailouts . Whatever happens, it’s inevitable that more lives will be irrevocably ruined. Vibe-coders will be quick to point fingers at everyone but themselves. “It was just a tool!” they’ll plead. Thanks for reading! Follow me on Mastodon and Bluesky . Subscribe to my Blog and Notes or Combined feeds.

0 views

ASI as a Black Box

An artificial superintelligence (ASI) is an AI that’s much better than the top humans at almost all cognitive tasks. This is a perfectly fine definition. But, from this definition and from our web of knowledge, we can deduce very little about what an ASI would actually look like. Human geniuses provide a lower bound, but not an upper bound. In some areas, we have deductive theories that describe the space of what is possible for all agents. We know, for example, that a superintelligent AI won’t be able to travel faster than light, or travel back in time, because we have well-trod theories that prohibit these things. In most areas of life, we don’t have such theories, and so we don’t have solid reasons to say what is or is not possible. Consider persuasion. Some humans are very charismatic. But what is the range of charisma? Is superpersuasion possible? I don’t know. The term is very loosely defined , but people just throw the word around like it means something. Without a deductive theory of human belief-change and communication, the possibility of “superpersuasion” is an empirical question. Consider scientific discovery. Maybe ASI would vastly accelerate science, or not. You can think of scientific discovery as a function of intelligence, the existing state of knowledge, and access to empirical observation. But we don’t know the coefficients of those terms, the—so to speak—intelligence elasticity of scientific discovery. I would guess that progress would accelerate, since greater intelligence means better hypothesis generation, hypothesis selection, experiment design, and more efficient scheduling of lab equipment. More generally, for a question of the form “would an ASI be able to do X ?”, our inability to answer the question usually stems, not from our lack of knowledge about ASI, but from our lack of knowledge about X . “Things an ASI can do” is a subset of “things it is possible to do”, to know the former (deductively rather than empirically) we first need to know the latter, and to do that we need a complete, deductive theory of whatever area of knowledge X belongs to. But don’t we have all these mathematical theories of rationality and optimal decision-making? Are those useful for predicting how a superintelligence would behave? The standard doctrine says superintelligence should converge on perfect Bayesian rationality. If that’s true, then all those papers about decision theory and all that ink spilled on Newcomb’s problem will finally pay off. If, on the other hand, these theories are just an unreachable, abstract normative standard, then we know nothing. In short: we can’t predict the specific cognitive abilities an ASI would have, and we also can’t predict the higher-order consequences of those cognitive abilities.

0 views
Ahead of AI 2 days ago

Understanding Positional Embeddings in Modern LLMs

This article explains how LLMs encode and process token position information by covering classic and contemporary methods like RoPE. It’s essentially a comprehensive journey where we will work through the mechanisms with visual examples, a little math, and a small PyTorch implementation. And, of course, we will then also look at how modern architectures adapt or omit explicit positional encodings. Below, you can find a preview of which positional embedding choices are used in 2026 open-weight models. We will discuss these choices later after covering the underlying mechanisms. Figure 1: A preview of the gallery’s 2026 architectures, grouped by positional encoding. Architecture figures taken from my LLM gallery . But why spending so much time writing/reading about this rather minor building block? LLMs keep evolving, and as I was preparing a new article on the latest LLM architecture trends this year, I realized that I never covered the positional information methods used in LLMs in any article so far. So, instead of letting the other article get out of hand, I thought it might make sense to publish this as a standalone building block for the upcoming LLM architecture and trends analysis. Also, over the years, many readers have asked me for a tutorial on relative positional embeddings (aka RoPE) to follow up on the learned absolute embeddings used in GPT. It’s been on my TODO list for a long time, so I finally revisited my old draft and put this walkthrough together, which will cover different positional methods in LLMs; (finally) how RoPE works; what methods are used by recent open LLMs. Note that this article assumes some familiarity with basic LLM architectures and works well as a follow-up to Build a Large Language Model (From Scratch) . If the concepts and computation of the attention internals, particularly query, key, and value vectors, are a little fuzzy, I recommend checking my self-attention walkthrough as a refresher: The article is also very technical and likely too detailed for some. However, I really wanted to have this foundation out there for future articles to reference. And no worries, the next one will return to a more comfortable style. PS: I started experimenting with Substack’s new “perks” system. As a thank-you to supporters, I’ve added image exports and metadata from the architecture gallery for you to use in your own analyses. PPS: If you are already familiar with how position embeddings work, feel free to fast-forward to Section 6, Positional choices in modern architectures . Note that this article may be best read in a web browser than an email client. I recommend checking out the web view here , which also has a navigation bar on the left side. Let’s start with the overall architecture. A tokenizer encodes the input text into token IDs. Next, the input token embedding layer of an LLM maps each of these IDs to a vector of numbers. We usually refer to these entries in that vector as “features” (similar to classic ML). The transformer blocks inside the LLM then modify these vectors using attention and feedforward layers. Lastly, the model’s output layer produces logit scores (converted into probabilities) for the next token. The following shows this path in GPT-2 (a relatively classic LLM) and MiniMax-M3 (a more recent LLM released this year; a placeholder for most other recent LLMs). As we can see, the architectures are not too different from each other, but before we get to the positional information parts, let’s focus on the orange blocks for now (the token embedding layers) and the transformer blocks above them; the other architecture differences aren’t needed for this explanation. Figure 2: Token embedding layers in GPT-2 and MiniMax-M3, highlighted in orange. (Side note: M3’s first three out of 60 feedforward layers are dense not MoEs.) A token embedding layer, shown in the figure above, maps each token ID to a vector. Position information is then added afterward. Concretely, and as we will discuss in more detail later, GPT-2 adds a position vector to each token embedding before the modified embedding enters the first transformer block. On the other hand, MiniMax-M3 adds the positional information in each transformer block using a method called RoPE inside attention. We will talk about these methods in more detail later, but first, let’s start with the big picture information explaining why explicit positional information is added or needed. Consider the following two input sentences as a simple example: “The red fox jumps over the gray fox.” “The gray fox jumps over the red fox.” Changing the word order changes which fox does the jumping, as illustrated in the following figure. Figure 3: Changing the word order changes which fox jumps. For the examples in this article, we will treat each word as one token for simplicity. (As you know, real tokenizers can split words into smaller subword tokens.) The first sentence then has eight token positions, from 0 through 7. Figure 4: Token IDs and positions in the toy sentence. (If you are interested in how vocabularies are built, check out my tokenizer materials on GitHub .) Note that “fox”, which appears twice in the input sentence, gets the same token ID 2 in both cases. And since they both have the same token ID, irrespective of the position in the input sentence, they would get the same embedding vector from the embedding layer. Now, let’s consider a regular self-attention mechanism with no positional encoding (and no causal mask), where every input token can attend to every other token in the input. In this scenario, swapping any two words, like “red” and “gray”, would still result in the same embeddings and attention scores between each token pair. GPT-style models do use a causal (autoregressive) mask, which restricts each token to itself and earlier tokens. Interestingly, this causal mask gives the computation some implicit order information. In our example, “jumps” can attend to “red” in the first sentence and “gray” in the second. So, a causal LLM can theoretically already distinguish these sentences even without explicit positional embeddings. (We will return to this idea later when discussing the “NoPE” concept.) Nonetheless, GPT-2 and most newer LLMs include explicit positional information because it gives attention a direct signal about positions. That’s because even though the causal mask determines which tokens are accessible and provides implicit order information, it doesn’t distinguish a separation of one position from a separation of ten, for example. So, explicit positional methods are still useful (or even essential), since they supply that information. If this all sounded confusing, don’t worry, it was more meant as an overview, and we will step through these concepts a bit more slowly in the remaining article. We briefly talked about implicit positional information through causal attention masks (and we will revisit this again in more detail later). However, we now switch gears and talk about the explicit ways to add positional information. The perhaps simplest and most direct (but not “cheapest”) approach is to give each sequence position its own vector. We add that vector to the token embedding, so the resulting, modified embedding vector is then passed to the first transformer block. This exact procedure is discussed in the following subsections. To illustrate the use of an explicit, learned positional vector to the input embeddings, let’s consider GPT-2 as a concrete example. GPT-2 learns a table of position vectors during training. The following figure illustrates where these vectors are used in the architecture, that is, immediately after the token embedding lookup and before the first transformer block. Figure 5: GPT-2 adds absolute position information before the transformer blocks, as highlighted in purple. The dimensions shown are for GPT-2 XL . The positional embedding layer depicted in the figure above follows the same implementation as the token embedding layer, i.e., they have exactly the same output dimensions so that the resulting input vectors can be added to each other. The code below shows that with an excerpt of a GPT-2 code implementation: And in mathematical terms, we can write it as follows. Let m be a position index and x m the token ID at that position. The input to the first block is E token is the embedding layer, P m is the position info vector selected by m , and h m is the resulting summed embedding and position vector. The two vectors have the same width, so we add their entries element-wise. Figure 6 below shows this visually with a small numerical example. Figure 6: Token and position embeddings are added element-wise. (I am following the convention here that neural network diagrams are usually read from bottom to top.) In the figure above, at position 0, adding the token embedding to the position embedding gives . By the way, the values in the figure are chosen to keep the illustration simple; in a real example, both embedding layers (tables) are initialized with small random values and learned during training. So, what we covered in this section is a so-called absolute positional method because the positional embedding P m depends on the token’s own position m . In the previous example sentence, the two “fox” tokens have initially the same token embeddings, but because we are adding different positional vectors P 2 and P 7 , they have different values before they enter the first transformer block. And consequently, the query, key, and value vectors inside the attention mechanisms then also receive representations that already include position information. Note that the original GPT-2 implementation has 1,024 learned position vectors. This means it only works with inputs with 1024 or fewer tokens. If we want to extend the context window, we would have to extend this embedding table. And since the embedding table (layer) is updated like any other layer during training, the context length would have to be decided before the training starts. Extending the context would be very expensive since the table can become very large. For example, in the case of GPT-2 Medium, the positional embedding layer weights only make up 0.30% of the total weights at 1,024 positions. However, if we increased the context length to a nowadays common 100k range, they would make up >22% of the total model weights, which would be very uneconomic. The original “Attention Is All You Need” transformer from 2017, which predates GPT, uses the same addition mechanism, but computes its position vectors with fixed sine and cosine functions. Figure 7 below shows this addition before both the encoder and decoder stacks. The encoder processes the input sequence, and the decoder generates the output sequence. (The reason why it’s introduced here is that the sine and cosine functions will become relevant again later when discussing RoPE.) Figure 7: Sinusoidal position encodings added to the token embeddings as highlighted in purple. The original transformer also scales its token embeddings by √( d model ) before adding the position vectors, where d model is the embedding width. So, while GPT-2 in the previous section had the formula the formula used in the original transformer (depicted in Figure 7) is: This scaling by √( d model ) is a minor technical detail, but for those who are curious, the motivation behind this scaling factor is not explained in the transformer paper, but as I discussed with colleagues on a Twitter thread many years ago, it’s likely to put the token embeddings and sinusoidal positional encodings on comparable numerical magnitudes at initialization, so the position vectors don’t dominate the sum when we add them. In the GPT-2 case, this scaling wasn’t necessary because the original GPT-2 code initialized the token embeddings to have an initial standard deviation of 0.02 and the position embeddings to have an initial standard deviation of 0.01 so that they were already roughly on the same scale. Now, the more interesting point is how the position embeddings P m are computed. Concretely, each position vector has the following entries: As before, m denotes the position. The index i indexes a pair of features within the vector. Here, a feature is one number in the position vector. For example, let’s assume we have a small vector with 8 values ( d model = 8) that has entries numbered 0 through 7. (In the original paper this was 512 for the small model and 1024 for the bigger model, instead of 8, but that would be hard to visualize.) We group these into four pairs so that i goes from 0 through 3. Figure 8: Sine and cosine entries for each feature pair. Here, the indices 2 i and 2 i + 1 select the two entries corresponding to pair i . As shown in the equation earlier, the angle at position m , which is then fed into the sine or cosine functions to obtain P m , 2 i or P m , 2 i + 1 , respectively, is computed as follows: Figure 9: The angle inside sine and cosine is the token position multiplied by the pair’s angular frequency. Figure 10 shows this calculation for all four pairs of our 8-feature example at position m = 2. Figure 10: The four sine and cosine pairs at position 2. For an eight-feature embedding, the four frequencies are 1, 0.1, 0.01, and 0.001 radians per position. At position 2, the first pair in row 1 is (sin 2, cos 2) ≈ (0.909, −0.416). The second pair, in row 2, changes more slowly, giving (sin 0.2, cos 0.2) ≈ (0.199, 0.980). The figure below visualizes the sine and cosine calculations for different pairs, which are then collected in a position vector. Figure 11: Sampling sine and cosine functions to form a position vector. In short, this method is a (more complicated) alternative to obtaining positional embedding vectors compared to the GPT-2 method discussed earlier. Similar to the GPT-2 embeddings, these are also absolute embeddings because each vector is computed from its own position. However, in contrast to the GPT-2 approach, the sine and cosine functions stay fixed during training. (Also, they can be extended and evaluated beyond the training positions, like beyond the context length of 1024, although that alone doesn’t guarantee that the model will perform well there, because it hasn’t seen those during training.) To summarize, Figure 12 below compares the two ways of obtaining a position vector we discussed so far. Figure 12: Fixed sinusoidal and learned absolute positional embeddings. This sinusoidal method may look like an overly complicated way to obtain positional embeddings, however, understanding it will be useful when talking about RoPE in the next section. Also, there is some rationale behind it. The authors chose sine and cosine because they make the shifting by a certain offset gives a certain deterministic difference. For example, the difference between position 2 to 5 and the difference between 102 to 105 is both 3, which then results in the same rotation. According to the Attention Is All You Need paper: “...easily learn to attend by relative positions, since for any fixed offset k , PE pos + k can be represented as a linear function of PE pos .” Or in less technical terms, the authors hypothesized that this property would make it easier for attention to learn relationships such as “three tokens earlier.” Figure 13 below illustrates this for one sine/cosine pair. A rotation is a linear transformation, and both three-position shifts use the same rotation. Figure 13: For a pair with angular frequency 0.1 radians per position, a shift of three positions always rotates the pair by 0.3 radians. 3. Encoding relative position with RoPE So, the sinusoidal method above is still considered an absolute positional encoding because the vector added to each token embedding represents that token’s position counted from the start of the sequence. As shown in the previous figure, for example, tokens at positions 2 and 5 receive the position vectors P 2 and P 5 . If we move those tokens to positions 102 and 105, they’re still three positions apart. But each token gets a different position vector because its absolute position m has changed. So their attention score can change even when the token embeddings stay the same. The sinusoidal structure still gives the model a way to learn relative relationships. Anyways, RoPE, which we will discuss below, builds this relative relationship directly into the attention calculation. RoPE, which Su et al. introduced in the 2021 RoFormer paper , stands for Rotary Position Embedding and is considered a relative position method. In attention, each token’s current representation is projected into a query, a key, and a value vector. Query-key dot products determine attention scores. After masking and softmax, the resulting weights determine how the value vectors are mixed. For a refresher, see my Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs article: RoPE uses the token’s position to rotate its query and key vectors before their dot product. The value vectors bypass this rotation. Figure 14 below shows an overview of where RoPE is applied in a relatively recent MiniMax-M3, where QK-Norm normalizes queries and keys before RoPE. Figure 14: MiniMax-M3 applies RoPE to queries and keys inside attention. As shown in the figure above, the LLM architecture has no explicit position vector added alongside the initial token embedding. Its explicit position signal is injected inside attention instead. Side note: MiniMax-M3 rotates only part of each query and key head, a variation known as “partial RoPE” that we will cover later in Section 6.1. But to understand the basic RoPE operation first, let’s walk through the method step by step using a small example. Attention usually divides its query, key, and value features into multiple heads, which are processed in parallel. Hence, the full mechanism is usually called Multi-head Attention (note that Grouped Query Attention also uses multiple heads). Here, each attention head computes its own attention scores and weighted sums. RoPE then acts on pairs of numbers within each query and key head. To make this concrete, Figure 12 below follows a token representation through the learned query and key projections, which are computed via the weight matrices W Q and W K . In this toy example, each projection produces 16 features, which are reshaped into two heads with eight features each. In the figure below, every small cell contains one scalar feature, meaning one numerical entry in a vector (the numbers are not shown due to space reasons). Figure 15: Projecting a hidden state, reshaping into heads, and pairing features for RoPE. For the first eight-feature head, , we pair each feature in the first half with the corresponding feature in the second half. This gives , , , and . This split-half convention is used in LLM implementations, including my Qwen3 implementation . (We will compare it with the original, alternative adjacent-feature pairing method later after implementing the basic operation.) Each pair gets a rotation angle determined by the token position m and that pair’s frequency θ i . The corresponding query and key pairs use the same frequency. Figure 16 below shows how we rotate the pairs, put the updated features back in their original locations, and compute the query-key dot product. Figure 16: Pairwise rotations produce the rotated query and key vectors used to compute attention scores. The next section will walk through the figure(s) a bit more slowly. For the following example, we’ll use the same two positions as before. The query comes from “the” at position m = 5, and the key comes from the first “fox” at position n = 2. Since, in this example, the key comes before the query, the causal mask allows the query to attend to it. To illustrate the effect that the position has, and keeping everything fixed, let’s assume we start both pairs from the same two values, (1, 5). Figure 17: The first feature pair, (1, 5), takes one entry from each half of the query head. Furthermore, for this example, let’s use θ = 20° per token position ( π /9 radians per position). At position 2, we rotate the key’s arrow counterclockwise by 2 × 20° = 40°. At position 5, we rotate the query’s arrow by 5 × 20° = 100°. Figure 18: Starting from (1, 5), RoPE rotates the key pair at position 2 by 40° and the query pair at position 5 by 100°. To illustrate what’s happening in the figure above, for example, plugging cos 40° ≈ 0.766 and sin 40° ≈ 0.643 into the rotation from Figure 16 gives the new key coordinates: Then, changing 40° to 100°, using the same equation as above, the rotated query pair becomes q′ ≈ (−5.098, 0.117). But the important point is that both arrows still have length √26. Only the rotation changes but the vectors keep the same lengths after the transformation. So, for a query pair ( a , b ) at position m , the two operations here are. For the key pair at position n , we use the same equations with nθ . (Section 4 later shows how RoPE calculates the angular frequency for each feature pair.) To close this section, for those more comfortable reading code, everything about RoPE illustrated so far can be written in code, as follows: The resulting values are: If we rotate all 4 pairs in the 8-feature query (and key) vector(s), we get the vectors below. (We’ll come back to how we go from rotating one pair to rotating all of them later.) Note that, as shown above, the query and key heads have eight features each, which we previously grouped into 4 pairs with two elements each, like (1, 5), and (2, 6), and so on. For simplicity here we’ll follow just the first pair in each head, (1, 5), formed by taking the entries at indices 0 and 4: As shown above, in our example, those entries have values (1, 5) in both heads before the rotation. Also, as in the code above, let’s use the notation q =( q 1 , q 2 ) and k =( k 1 , k 2 ) for these two pairs. To compute their dot product, using the standard dot product formula, we multiply corresponding coordinates and add the products: Before rotation, the pairs have a dot product of 1 × 1 + 5 × 5 = 26. Using the rounded coordinates from the previous Figure 18 gives the contribution from the first pair after rotation: We can see why by writing the dot product in terms of the lengths of the two vectors (arrows in Figure 18) and the angle ϕ between them. In both cases, the vector length is √26. The query (position 5) is rotated by 100° and the key (position 2) is rotated by 40°. The vectors (arrows) start from the same direction, so they end up being 60° apart after rotation. Their exact dot product is 26 × cos(60°) = 13. Now let’s see what happens if we move just the key from position 2 to position 3, but we keep its starting coordinates (1, 5) fixed. The key’s rotation changes now from 2 × 20° = 40° to 3 × 20° = 60°. The query still rotates by 100°, so the angle between the arrows shrinks to 40° and the dot product changes from 13 to 26 × cos(40°) ≈ 19.92. So, moving the key one position closer changed the dot product even though we started with the same feature values. Why do these dot products matter? In attention, we sum the contributions from all feature pairs to get the query-key dot products. RoPE can change those dot products and thus the weights that softmax assigns to the value vectors. In the attention equation above, Q′ and K′ are matrices that contain the rotated queries q′ and keys k′ for one head, V contains the values, and d is the head dimension. Now that we illustrated the computation of the dot products, which is at the core of the attention mechanism, let’s return to positions m = 5 and n = 2. If we shift both positions to the right by three positions, we get m = 8 and n = 5. Each rotation gains 60° (the query goes from 100° to 160°, and the key goes from 40° to 100°), however, the angle between the vectors stays the same. The dot product for this feature pair is still 13. Figure 19: The first pair’s dot product at three different position settings. The key is that the difference between the rotation angles is If we keep the original query and key vectors fixed and shift both token positions by the same amount, RoPE adds the same rotation to both vectors. The angle between them stays the same, so their dot product stays the same. Note that Figure 20 below uses smaller angles than the actual example (20° per position) to keep the picture readable. Figure 20: A shared position shift preserves the relative angle and dot product for fixed query and key features. The examples above and Figure 20 illustrate the main idea behind RoPE. We use each token’s position to rotate the corresponding query and key in the attention module. In the query-key dot product, what matters is how much one vector rotates relative to the other. For each feature pair, this difference in rotation depends only on the offset between the two tokens. To illustrate this in the big picture context, in Figure 21, each row collects the scores for one query. If we consider row 5, the query for “the”, and column 2, the key for “fox”, the attention score between those two is computed as the dot product between the two rotated versions of the query and key vector. Figure 21: The attention scores for one head before softmax. Gray cells mark future keys excluded by the causal mask. What’s the point, again? Let’s once more consider “The red fox jumps over the gray fox.” If we add 100 tokens (e.g., think system prompt) before the sentence, the first “fox” and the second “the” move from positions 2 and 5 to 102 and 105. However, a prefix of any length shifts both positions equally, so they still stay exactly three positions apart. This means, each vector (arrow) gains the exact same additional rotation, which preserves the angle between them and their dot product (assuming we hold the unrotated query and key vectors fixed; in a real model, a new prefix of 100 tokens can change those vectors by changing what the tokens attend to). Figure 22 tracks this comparison in the attention score matrix. The right panel shows the rows and columns for the sentence after we add the prefix. In our toy example, the first pair still contributes 13 to the dot product. Figure 22: The highlighted score moves from row 5, column 2 to row 105, column 102. It keeps the same value if we hold the query and key vectors fixed before rotation. As you may recall, in the original transformer’s sinusoidal encoding, a fixed offset also corresponds to a fixed rotation within each sine/cosine pair. But that shared rotation applies only to the position vectors (added to the input vector) instead of the key and query vectors inside the attention mechanism. Figure 23 compares what this shift changes in the three methods we covered so far. Figure 23: The same position shift under sinusoidal encodings, GPT-2’s learned position embeddings, and RoPE. 3.6 Bonus: Combining the pairs into a complete head in RoPE To see how the relative offset affects the dot product, we focused on one feature pair in the query and key. Now let’s put all four pairs together and see how RoPE rotates the full eight-feature vectors in one attention head. (Please feel free to skip this section if it is too much technical detail.) Figure 24 picks up where Figure 17 left off. We rotate each of the four pairs, then put the results back in the query’s original feature order. Figure 24: RoPE rotates each pair of query features at its own angle. (The same can be applied to the key, which is not shown to keep the illustration simple). Figure 24 above shows the rotated query q′ (at the bottom). Similarly, we could get the rotated key k′ by repeating the same steps at the key’s position. Recall the two equations from Section 3.3. For query pair i with input values ( a i , b i ), let c i = cos( mθ i ) and s i = sin( mθ i ). These are the cosine and sine of the rotation angle for that pair at query position m : Figure 25 below shows the same four rotations between the four pairs as a single matrix multiplication, q′ = R m q . Figure 25: The four rotations from Figure 24 written as one matrix multiplication. The pair labels show where each rotation goes in the full matrix. 4. Computing rotation frequencies in RoPE? A full attention head uses multiple frequencies so that some feature pairs change more drastically (”quickly” in RoPE terms) with position and others change more gradually (”slowly” in RoPE terms). This serves the same purpose as the multiple frequencies in the original transformer’s sinusoidal encodings. However, instead of us assigning these manually, for a head with d features, standard full RoPE computes one frequency for each of its d /2 pairs as follows: d is the head dimension (it has to be even so that all features can be arranged in pairs). i is the index of the pair; it ranges from 0 to d /2−1. b is a chosen base value (hyperparameter) that determines the spacing of the frequencies. θ i is pair i ‘s frequency in radians per token position. The original RoFormer paper uses b = 10,000, borrowing the original transformer’s frequency formula with head dimension d in place of embedding width d model . For example, with head dimension d = 8 and base b = 10,000, we get frequencies [1, 0.1, 0.01, 0.001] radians per position. If you want to see RoPE in a complete model, I recommend checking out my end-to-end Qwen3 from-scratch implementation . Since code can be an effective medium for explaining and understanding concepts, the following subsection will walk you through the core parts that we have covered so far, if you are interested. Otherwise, please feel free to fast forward to section 6, on Positional choices in modern architectures . So, for the following code snippets, we’ll use the same split-half rotation convention as mentioned earlier (the interleaved convention will also be covered, no worries). First, we’ll apply it to one query vector, then we’ll use it on queries and keys inside the attention mechanism. First, let’s reuse the 8-feature query from Figure 24. For convenience, below are its four feature pairs before rotation. Detail from Figure 24. The eight query features form four pairs in the split-half layout. In the earlier non-coding example, we chose a frequency of 20° per token position for the first pair to keep the math simple. Here, we’ll use the original RoPE formula from the previous Section 4 above, θ i = b −2 i / d , with head dimension d = 8 and base b = 10,000. Figure 26 shows how the two functions work together to apply these rotations. Figure 26: rope_params returns the cosine and sine tables for token position 2. rope_split_half takes those tables and the original query and returns the rotated query. As shown in the figure above, first, determines how far to rotate each pair, returning two “tables” that store the cosine and sine of those angles. Or, in other words, computes the cosine and sine of each rotation angle, which depends on the token’s position and the feature pair’s frequency. The array in the function contains the 2 i values from the frequency formula in Section 4: For our eight-feature query, the are , that is, one 2 i value for each feature pair. Suppose we use the common base b = 10,000. And following our earlier examples, we choose a small head dimension of d = 8. In this case, the then computes the four pair frequencies θ i = b −2 i / d as follows: As we can see above, the resulting has one frequency θ i for each of the four feature pairs ( measured in radians) per token position. Next, the then multiplies each pair’s frequency θ i with the token position m , which computes the rotation angle mθ i . In our example, with , we get: Finally, takes the cosine and sine of each angle and outputs them as two so-called cosine and sine tables, and . In both tables, each row is a token position and each column is a feature pair. Since we only use token position 2 here, each table has shape , with the single row shown below: For the example in Figure 26, each table has one row for token position 2 and four columns, one for each feature pair. Figure 27 shows the rotation angle for each pair and the corresponding entries in the two tables. Figure 27: The two tables returned by rope_params for token position 2. The highlighted column gives the cosine and sine used to rotate the first feature pair, (1, 5) . In short, computes the cosine and sine of each rotation angle, which depends on the token’s position and the feature pair’s frequency. In a real-world scenario, if we pass 1,024 token positions for a context size of 1024, and a head dimension of 128, it returns two tensors, and , each with shape since a head with 128 features has 64 pairs. The tables have no batch or head dimension, and we broadcast each table across the batch and head dimensions. So, if we have multi-head attention or grouped query attention with 32 heads, feature pair 0 at token position 2 uses the same rotation angle in all 32 heads. Each head rotates its own feature values, so the rotated queries can still differ across heads. Also, the same entries can be reused across queries, keys, and heads that use the same positions and frequencies. In short, we only need to (pre)compute these tables once per model for all positions up to a maximum supported context size. Technically, it would also be possible to store these tables alongside the model weights; but they are so incredibly cheap to compute that storing them on disk would be overkill. Next, uses those cosines and sines to rotate the actual features in the query and key vectors inside attention. Considering the simple 8-feature example once more, the in the code snippet above splits the query into and and lines up the two entries in each pair. For example, the first slot of each half gives us , the second , and so on. The two expressions inside compute the rotated halves using the equations from Section 3.3: Note that here, everything denoted with a i corresponds to the first half of the query, and everything denoted with b i comes from the second half. The first expression inside computes all the a i ′ values, and the second computes all the b i ′ values. Then, we concatenate the halves, which puts each value back in its original index position via . Here’s the code that produces the rotated query in the earlier Figure 26. The shown reshape just brings the simple toy array into the expected format: one batch item, one head, one token, and eight features. With several positions, each token simply picks up its own row from the and tables. Through broadcasting, PyTorch reuses these same tables across all batch items and attention heads. So far, we coded the RoPE utility functions. Let’s now use it in the real context, the attention mechanism. Here, we use the standard frequency schedule for two 8-feature heads and four token positions. We use random query, key, and value tensors to keep the example self-contained, but as mentioned before, you can optionally also check the end-to-end Qwen3 from-scratch implementation with real query, key, and values produced by the pre-trained model. Note that only and but not pass through RoPE. The resulting raw attention scores are stored in a tensor of shape , that is, one row for each query position, and a column for each key position, and 4x4 since we have four tokens where each attends to each other. Then, using a conventional attention code implementation, the causal mask replaces the scores for future keys with negative infinity, so they get zeroed after the softmax. The resulting tensor has shape ; the 4 because we have four tokens, and the 8 because the embedding dimension was originally 8. We can also check that our claim about relative offsets holds, meaning that adding 3 to each position should change the position but preserve the query-key offsets: I mentioned the split-halves convention many times throughout this article. The original RoPE paper uses the alternative interleaved convention, as does the original LLaMA implementation (but as mentioned before, most code implementations now use the split-halves, because it makes it easier, in code, to rotate all pairs at once with element-wise sine and cosine operations.) To illustrate the difference between the two layouts, Figure 28 below uses the same 8-feature vector in both rows. Figure 28: Split-half pairs each feature in the first half with the corresponding feature in the second half. Interleaved pairs adjacent features. The interleaved implementation uses the same two equations, but selects alternating features and interleaves the outputs. But mathematically, the layouts are equivalent when we account for the feature order. So, rearranging the entries to makes the split-half pairs adjacent with pairs (1, 5), (2, 6), and so on. If we apply the same rearrangement to both queries and keys, it results in identical dot products. The outputs retain the reordered features, so we compare them with the split-half outputs in that same order. Applying the two layouts to the original vector however would give different scores. RoPE has been a staple in most (open-weight) LLMs since the original LLaMA model . Of course, proprietary models may use it as well, but it’s always hard to say anything with 100% certainty about proprietary models. However, in recent years, many architectures have been moving away from full-blown RoPE in every transformer block. In the following subsection, we will discuss some of these variants. For instance the full RoPE implementation we discussed and implemented in the sections above rotates every feature pair in every query and key head. In the following examples in the subsections above below, we will see that models only rotate some features within a head, other models skip RoPE in some transformer blocks, and some models use a different approach and add position information directly to their attention scores. As discussed throughout the article, regular “full” RoPE rotates all dimensions in each query and key head. Partial RoPE is a variant that rotates only a subset, while leaving the remaining dimensions unchanged. For example, MiniMax-M3’s configuration uses 64 rotary dimensions within each 128-dimensional head. That gives 32 rotated feature pairs and 64 unchanged dimensions. The figure below illustrates how that looks like. Figure 29: Full RoPE and partial RoPE in a 128-dimensional head. Note that if we look past the first transformer block, the unrotated features can still contain position information inherited from earlier transformer blocks. Anyways, what’s the point behind partial RoPE? It saves some memory. For example, in the MiniMax-M3 example above, rotating half the features reduces the size of the precomputed cosine and sine tables also in half. So, with partial RoPE, instead of 64 columns, now each table only needs 32 columns. However, note that the KV cache still stores the full key and value vectors for each cached token. For MiniMax-M3’s 1M-token context, rotating 64 of the 128 features halves the size of the precomputed cosine and sine tables, saving 0.134 GB (128 MiB) if the tables are stored in BF16. If the tables themselves were stored in NVFP4, the savings would be 0.038 GB (36 MiB), including the block scales. Note that in experiments with 1B to 8B models, the Fractional Rotation, Full Potential? paper found that rotating around 10% of each head’s features often gave similar training loss and benchmark accuracy to full RoPE. However, full RoPE did better on some evaluations. So, in my opinion, partial RoPE is hardly worth it. NoPE was coined in the The Impact of Positional Encoding on Length Generalization in Transformers paper and means no explicit positional encoding . The idea or observation is that an attention layer without any explicit position method still has a causal mask, where a query can attend only to itself and earlier tokens. And this causal masking implicitly provides positional information. Let’s use the first four tokens from our earlier example, . In Figure 30, the query for at position 2 can attend to , , and itself. The query for at position 1 can attend to and itself. Figure 30: Example of a causal mask in the attention mechanism. So, the next question is how can the model use the causal mask as a position signal? Here’s a toy example adapted from the paper. Suppose one feature of the value vectors is 1 for and 0 for the other three tokens. Since the input only contains only one , every visible prefix contains exactly one 1. If all non-masked keys get the same attention score, the softmax assigns every visible token equal weight. So at , attention averages 1, 0, and 0, which gives 1/3. At , it averages four values and gets 1/4. In other words, this output feature equals one over the prefix length, which a later layer can use as a signal for position. Figure 31: With uniform attention, the contribution from The shrinks as the prefix grows. The query for fox assigns The a weight of 1/3. Does NoPE work in practice? The researchers of the NoPE paper trained a tiny 107M-parameter decoder-only transformers from scratch on tasks such as copying, sorting, and addition to do some ablation. When they tested the small model on examples longer than those seen during training, interestingly, NoPE ranked first overall across the ten tasks and ahead of RoPE. The researchers behind the NoPE paper also conducted a separate experiment with a larger 1.3B-parameter models trained on code. Here, NoPE also handled contexts beyond the training length, however, its next-token predictions got worse at the longest lengths tested. Some of today’s open-weight LLMs even go so far and use NoPE in all layers. However, nowadays, it’s more common for models to omit RoPE in at least some layers, where layer here means transformer block. To give a concrete example, SmolLM3 (3B parameters) applies RoPE in three layers and skips it in the fourth, repeating that pattern through 36 layers. Interestingly, the SmolLM3 authors mentioned that this layer pattern improved long-context performance. (Also note that any later layer, following the first one, even though it may skip the rotation, still receives the inputs from preceding transformer blocks, so its queries and keys still receive some sort of position information from earlier layers.) Figure 32: SmolLM3’s architecture (left) and the positional method used in each of its 36 transformer blocks (right). Every fourth block skips RoPE. Architecture diagram from my LLM Architecture Gallery . 6.3 Learned relative position bias Before we get to the overview and look at what modern architectures use, there’s one more method to introduce. For instance, Thinky’s Inkling uses a learned relative position bias in both its sliding-window and global attention layers. It computes the usual scaled dot product between a query and key, and then it adds a certain number to that score before the softmax in the attention score computation. That number depends on the query token’s hidden state and how many positions back the key is. Let’s reuse our example sentence, “The red fox jumps over the gray fox.” (It’s the last time, I promise!) In the figure below, the “the” at position 5 is the query and “fox” at position 2 is the key. Inkling then uses the three-position gap between query and key and the hidden state for “the” to compute the bias, then adds it to the scaled dot product. Figure 33: Inkling adds a learned relative position bias to the scaled query-key dot product before softmax. The numbers are illustrative. This is not an entirely new idea. Some of you may recall Google’s popular T5 (Text-to-Text Transfer Transformer) in 2019, which used a simpler version of this idea. Figure 34: The original Inkling (975B) architecture. The learned relative position bias feeds into the attention module. LLM Architecture Gallery 6.4 How these choices appear in recent models Now, after all this build up, the interesting question is: which architecture uses which method? And what are the recent trends? To answer this, I grouped the entries in my LLM Architecture Gallery by their listed release dates and compared the 2025 releases with the earlier and later releases in the 2026 snapshot. Each gallery card counts once, so multiple variants of an architecture can contribute to the results. LLM Architecture Gallery Figure 35: The architecture diagrams illustrate which attention layers use RoPE, and the bars show the share of gallery entries in each category. Diagrams from my LLM Architecture Gallery . As we can see in Figure 35 above, using RoPE in every softmax attention layer is still the most common choice. However, its use fell from 2025 to 2026, and then also in the latter half of 2026, which could be a trend. Consequently, there is a growing number of models that mix RoPE and NoPE. The groups are still small, so it’s hard to say whether these trends will continue. The NoPE only entries were all recurrent hybrids with softmax attention layers that use no explicit positional encoding. The “Other” category here refers to xLSTM 7B (2025) and Inkling (2026). xLSTM has no softmax attention layers. Inkling adds a learned bias to the attention scores before softmax, as we saw in Section 6.3. Among the models that use RoPE, partial RoPE became more common from 2025 to 2026. It went from a minority of the RoPE models in 2025 to a majority in both 2026 groups, as shown in Figure 36. Figure 36: Among gallery entries that use RoPE, partial RoPE rotates a subset of query and key features in at least one attention layer. Full-width entries rotate all query and key features in every layer that uses RoPE. If you’d like to look up an individual model from the 2026 groups, Figure 37 below shows the 2026 gallery overview with the individual architectures labeled by position type. You can also browse the gallery interactively here . Figure 37: The individual 2026 gallery entries behind the two 2026 bars in Figure 35. 7. Final thoughts So, to recap, we discussed GPT-2 and the original transformer that add a position vector to each token embedding before the first transformer block. Then, modern LLMs switched to RoPE, which encodes relative position inside attention by rotating queries and keys. Surprisingly, RoPE remains as the most popular positional embedding method today. Recently, however, Inkling picked up an idea from T5 to add a learned relative position bias to the attention scores before softmax. And then we also have NoPE-RoPE architectures, which omit RoPE in some layers. It may be boring news, but as mentioned before, RoPE is still the predominant position embedding method, and I think this will carry over into 2027 for sure. Perhaps, though, we’ll see more architectures with an Inkling-like design coming up in 2027. While this may sound “boring”, I am glad that I finally got this article on RoPE (and other position info methods) out, after so many years, so that I can add it to my collection of LLM “fundamentals” explainers. If you are interested in the other architecture building blocks of LLMs, I still highly recommend starting with a GPT-2-style model, which will give you a solid understanding of the building blocks with the least amount of friction. And from there, you can then layer on additional methods (such as RoPE). If you’d like to build that GPT-2-style model yourself, my first Build a Large Language Model (From Scratch) book walks through all the building blocks depicted in the figure below. Figure 38: GPT-2 XL provides a baseline for seeing what newer architectures change. LLM Architecture Gallery For the methods that branch off from the GPT-2 baseline, I’ve collected some of my relevant blog articles below, starting with a refresher on self-attention: Self-attention, causal masking, and multi-head attention: Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs Modern attention variants (GQA, MLA, sliding-window attention, sparse attention, and hybrids): A Visual Guide to Attention Variants in Modern LLMs The KV cache. Understanding and Coding the KV Cache in LLMs from Scratch SwiGLU feed-forward layers and normalization (LayerNorm and RMSNorm): From GPT-2 to gpt-oss: Analyzing the Architectural Advances Mixture-of-experts layers and expert routing. The Big LLM Architecture Comparison Residual connections and DeepSeek’s manifold-constrained hyper-connections (mHC): Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention Putting some of the pieces together in Qwen3: Understanding and Implementing Qwen3 From Scratch walks through implementing the dense and mixture-of-experts versions in PyTorch. Thanks for reading and supporting my work! PS: I started experimenting with Substack’s new “perks” system. As a thank-you to supporters, I’ve added image exports and metadata from the architecture gallery for you to use in your own analyses. Figure 1: A preview of the gallery’s 2026 architectures, grouped by positional encoding. Architecture figures taken from my LLM gallery . But why spending so much time writing/reading about this rather minor building block? LLMs keep evolving, and as I was preparing a new article on the latest LLM architecture trends this year, I realized that I never covered the positional information methods used in LLMs in any article so far. So, instead of letting the other article get out of hand, I thought it might make sense to publish this as a standalone building block for the upcoming LLM architecture and trends analysis. Also, over the years, many readers have asked me for a tutorial on relative positional embeddings (aka RoPE) to follow up on the learned absolute embeddings used in GPT. It’s been on my TODO list for a long time, so I finally revisited my old draft and put this walkthrough together, which will cover different positional methods in LLMs; (finally) how RoPE works; what methods are used by recent open LLMs. Figure 2: Token embedding layers in GPT-2 and MiniMax-M3, highlighted in orange. (Side note: M3’s first three out of 60 feedforward layers are dense not MoEs.) A token embedding layer, shown in the figure above, maps each token ID to a vector. Position information is then added afterward. Concretely, and as we will discuss in more detail later, GPT-2 adds a position vector to each token embedding before the modified embedding enters the first transformer block. On the other hand, MiniMax-M3 adds the positional information in each transformer block using a method called RoPE inside attention. We will talk about these methods in more detail later, but first, let’s start with the big picture information explaining why explicit positional information is added or needed. 1.1 Why add positional information? Consider the following two input sentences as a simple example: “The red fox jumps over the gray fox.” “The gray fox jumps over the red fox.” Figure 3: Changing the word order changes which fox jumps. For the examples in this article, we will treat each word as one token for simplicity. (As you know, real tokenizers can split words into smaller subword tokens.) The first sentence then has eight token positions, from 0 through 7. Figure 4: Token IDs and positions in the toy sentence. (If you are interested in how vocabularies are built, check out my tokenizer materials on GitHub .) Note that “fox”, which appears twice in the input sentence, gets the same token ID 2 in both cases. And since they both have the same token ID, irrespective of the position in the input sentence, they would get the same embedding vector from the embedding layer. Now, let’s consider a regular self-attention mechanism with no positional encoding (and no causal mask), where every input token can attend to every other token in the input. In this scenario, swapping any two words, like “red” and “gray”, would still result in the same embeddings and attention scores between each token pair. GPT-style models do use a causal (autoregressive) mask, which restricts each token to itself and earlier tokens. Interestingly, this causal mask gives the computation some implicit order information. In our example, “jumps” can attend to “red” in the first sentence and “gray” in the second. So, a causal LLM can theoretically already distinguish these sentences even without explicit positional embeddings. (We will return to this idea later when discussing the “NoPE” concept.) Nonetheless, GPT-2 and most newer LLMs include explicit positional information because it gives attention a direct signal about positions. That’s because even though the causal mask determines which tokens are accessible and provides implicit order information, it doesn’t distinguish a separation of one position from a separation of ten, for example. So, explicit positional methods are still useful (or even essential), since they supply that information. If this all sounded confusing, don’t worry, it was more meant as an overview, and we will step through these concepts a bit more slowly in the remaining article. 2. Adding a position vector to each token embedding We briefly talked about implicit positional information through causal attention masks (and we will revisit this again in more detail later). However, we now switch gears and talk about the explicit ways to add positional information. The perhaps simplest and most direct (but not “cheapest”) approach is to give each sequence position its own vector. We add that vector to the token embedding, so the resulting, modified embedding vector is then passed to the first transformer block. This exact procedure is discussed in the following subsections. 2.1 Learned absolute position embeddings in GPT-2 To illustrate the use of an explicit, learned positional vector to the input embeddings, let’s consider GPT-2 as a concrete example. GPT-2 learns a table of position vectors during training. The following figure illustrates where these vectors are used in the architecture, that is, immediately after the token embedding lookup and before the first transformer block. Figure 5: GPT-2 adds absolute position information before the transformer blocks, as highlighted in purple. The dimensions shown are for GPT-2 XL . The positional embedding layer depicted in the figure above follows the same implementation as the token embedding layer, i.e., they have exactly the same output dimensions so that the resulting input vectors can be added to each other. The code below shows that with an excerpt of a GPT-2 code implementation: And in mathematical terms, we can write it as follows. Let m be a position index and x m the token ID at that position. The input to the first block is Here, E token is the embedding layer, P m is the position info vector selected by m , and h m is the resulting summed embedding and position vector. Figure 6: Token and position embeddings are added element-wise. (I am following the convention here that neural network diagrams are usually read from bottom to top.) In the figure above, at position 0, adding the token embedding to the position embedding gives . By the way, the values in the figure are chosen to keep the illustration simple; in a real example, both embedding layers (tables) are initialized with small random values and learned during training. So, what we covered in this section is a so-called absolute positional method because the positional embedding P m depends on the token’s own position m . In the previous example sentence, the two “fox” tokens have initially the same token embeddings, but because we are adding different positional vectors P 2 and P 7 , they have different values before they enter the first transformer block. And consequently, the query, key, and value vectors inside the attention mechanisms then also receive representations that already include position information. Note that the original GPT-2 implementation has 1,024 learned position vectors. This means it only works with inputs with 1024 or fewer tokens. If we want to extend the context window, we would have to extend this embedding table. And since the embedding table (layer) is updated like any other layer during training, the context length would have to be decided before the training starts. Extending the context would be very expensive since the table can become very large. For example, in the case of GPT-2 Medium, the positional embedding layer weights only make up 0.30% of the total weights at 1,024 positions. However, if we increased the context length to a nowadays common 100k range, they would make up >22% of the total model weights, which would be very uneconomic. 2.2 Fixed sinusoidal position encodings in the original transformer The original “Attention Is All You Need” transformer from 2017, which predates GPT, uses the same addition mechanism, but computes its position vectors with fixed sine and cosine functions. Figure 7 below shows this addition before both the encoder and decoder stacks. The encoder processes the input sequence, and the decoder generates the output sequence. (The reason why it’s introduced here is that the sine and cosine functions will become relevant again later when discussing RoPE.) Figure 7: Sinusoidal position encodings added to the token embeddings as highlighted in purple. The original transformer also scales its token embeddings by √( d model ) before adding the position vectors, where d model is the embedding width. So, while GPT-2 in the previous section had the formula the formula used in the original transformer (depicted in Figure 7) is: This scaling by √( d model ) is a minor technical detail, but for those who are curious, the motivation behind this scaling factor is not explained in the transformer paper, but as I discussed with colleagues on a Twitter thread many years ago, it’s likely to put the token embeddings and sinusoidal positional encodings on comparable numerical magnitudes at initialization, so the position vectors don’t dominate the sum when we add them. In the GPT-2 case, this scaling wasn’t necessary because the original GPT-2 code initialized the token embeddings to have an initial standard deviation of 0.02 and the position embeddings to have an initial standard deviation of 0.01 so that they were already roughly on the same scale. Now, the more interesting point is how the position embeddings P m are computed. Concretely, each position vector has the following entries: As before, m denotes the position. The index i indexes a pair of features within the vector. Here, a feature is one number in the position vector. For example, let’s assume we have a small vector with 8 values ( d model = 8) that has entries numbered 0 through 7. (In the original paper this was 512 for the small model and 1024 for the bigger model, instead of 8, but that would be hard to visualize.) We group these into four pairs so that i goes from 0 through 3. Figure 8: Sine and cosine entries for each feature pair. Here, the indices 2 i and 2 i + 1 select the two entries corresponding to pair i . As shown in the equation earlier, the angle at position m , which is then fed into the sine or cosine functions to obtain P m , 2 i or P m , 2 i + 1 , respectively, is computed as follows: Figure 9: The angle inside sine and cosine is the token position multiplied by the pair’s angular frequency. Figure 10 shows this calculation for all four pairs of our 8-feature example at position m = 2. Figure 10: The four sine and cosine pairs at position 2. For an eight-feature embedding, the four frequencies are 1, 0.1, 0.01, and 0.001 radians per position. At position 2, the first pair in row 1 is (sin 2, cos 2) ≈ (0.909, −0.416). The second pair, in row 2, changes more slowly, giving (sin 0.2, cos 0.2) ≈ (0.199, 0.980). The figure below visualizes the sine and cosine calculations for different pairs, which are then collected in a position vector. Figure 11: Sampling sine and cosine functions to form a position vector. In short, this method is a (more complicated) alternative to obtaining positional embedding vectors compared to the GPT-2 method discussed earlier. Similar to the GPT-2 embeddings, these are also absolute embeddings because each vector is computed from its own position. However, in contrast to the GPT-2 approach, the sine and cosine functions stay fixed during training. (Also, they can be extended and evaluated beyond the training positions, like beyond the context length of 1024, although that alone doesn’t guarantee that the model will perform well there, because it hasn’t seen those during training.) To summarize, Figure 12 below compares the two ways of obtaining a position vector we discussed so far. Figure 12: Fixed sinusoidal and learned absolute positional embeddings. This sinusoidal method may look like an overly complicated way to obtain positional embeddings, however, understanding it will be useful when talking about RoPE in the next section. Also, there is some rationale behind it. The authors chose sine and cosine because they make the shifting by a certain offset gives a certain deterministic difference. For example, the difference between position 2 to 5 and the difference between 102 to 105 is both 3, which then results in the same rotation. According to the Attention Is All You Need paper: “...easily learn to attend by relative positions, since for any fixed offset k , PE pos + k can be represented as a linear function of PE pos .” Or in less technical terms, the authors hypothesized that this property would make it easier for attention to learn relationships such as “three tokens earlier.” Figure 13 below illustrates this for one sine/cosine pair. A rotation is a linear transformation, and both three-position shifts use the same rotation. Figure 13: For a pair with angular frequency 0.1 radians per position, a shift of three positions always rotates the pair by 0.3 radians. 3. Encoding relative position with RoPE So, the sinusoidal method above is still considered an absolute positional encoding because the vector added to each token embedding represents that token’s position counted from the start of the sequence. As shown in the previous figure, for example, tokens at positions 2 and 5 receive the position vectors P 2 and P 5 . If we move those tokens to positions 102 and 105, they’re still three positions apart. But each token gets a different position vector because its absolute position m has changed. So their attention score can change even when the token embeddings stay the same. The sinusoidal structure still gives the model a way to learn relative relationships. Anyways, RoPE, which we will discuss below, builds this relative relationship directly into the attention calculation. RoPE, which Su et al. introduced in the 2021 RoFormer paper , stands for Rotary Position Embedding and is considered a relative position method. 3.1 Moving the rotation inside attention in RoPE In attention, each token’s current representation is projected into a query, a key, and a value vector. Query-key dot products determine attention scores. After masking and softmax, the resulting weights determine how the value vectors are mixed. For a refresher, see my Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs article: RoPE uses the token’s position to rotate its query and key vectors before their dot product. The value vectors bypass this rotation. Figure 14 below shows an overview of where RoPE is applied in a relatively recent MiniMax-M3, where QK-Norm normalizes queries and keys before RoPE. Figure 14: MiniMax-M3 applies RoPE to queries and keys inside attention. As shown in the figure above, the LLM architecture has no explicit position vector added alongside the initial token embedding. Its explicit position signal is injected inside attention instead. Side note: MiniMax-M3 rotates only part of each query and key head, a variation known as “partial RoPE” that we will cover later in Section 6.1. But to understand the basic RoPE operation first, let’s walk through the method step by step using a small example. 3.2 From token representations to feature pairs in RoPE Attention usually divides its query, key, and value features into multiple heads, which are processed in parallel. Hence, the full mechanism is usually called Multi-head Attention (note that Grouped Query Attention also uses multiple heads). Here, each attention head computes its own attention scores and weighted sums. RoPE then acts on pairs of numbers within each query and key head. To make this concrete, Figure 12 below follows a token representation through the learned query and key projections, which are computed via the weight matrices W Q and W K . In this toy example, each projection produces 16 features, which are reshaped into two heads with eight features each. In the figure below, every small cell contains one scalar feature, meaning one numerical entry in a vector (the numbers are not shown due to space reasons). Figure 15: Projecting a hidden state, reshaping into heads, and pairing features for RoPE. For the first eight-feature head, , we pair each feature in the first half with the corresponding feature in the second half. This gives , , , and . This split-half convention is used in LLM implementations, including my Qwen3 implementation . (We will compare it with the original, alternative adjacent-feature pairing method later after implementing the basic operation.) Each pair gets a rotation angle determined by the token position m and that pair’s frequency θ i . The corresponding query and key pairs use the same frequency. Figure 16 below shows how we rotate the pairs, put the updated features back in their original locations, and compute the query-key dot product. Figure 16: Pairwise rotations produce the rotated query and key vectors used to compute attention scores. The next section will walk through the figure(s) a bit more slowly. 3.3 Rotating one pair in RoPE For the following example, we’ll use the same two positions as before. The query comes from “the” at position m = 5, and the key comes from the first “fox” at position n = 2. Since, in this example, the key comes before the query, the causal mask allows the query to attend to it. To illustrate the effect that the position has, and keeping everything fixed, let’s assume we start both pairs from the same two values, (1, 5). Figure 17: The first feature pair, (1, 5), takes one entry from each half of the query head. Furthermore, for this example, let’s use θ = 20° per token position ( π /9 radians per position). At position 2, we rotate the key’s arrow counterclockwise by 2 × 20° = 40°. At position 5, we rotate the query’s arrow by 5 × 20° = 100°. Figure 18: Starting from (1, 5), RoPE rotates the key pair at position 2 by 40° and the query pair at position 5 by 100°. To illustrate what’s happening in the figure above, for example, plugging cos 40° ≈ 0.766 and sin 40° ≈ 0.643 into the rotation from Figure 16 gives the new key coordinates: Then, changing 40° to 100°, using the same equation as above, the rotated query pair becomes q′ ≈ (−5.098, 0.117). But the important point is that both arrows still have length √26. Only the rotation changes but the vectors keep the same lengths after the transformation. So, for a query pair ( a , b ) at position m , the two operations here are. For the key pair at position n , we use the same equations with nθ . (Section 4 later shows how RoPE calculates the angular frequency for each feature pair.) To close this section, for those more comfortable reading code, everything about RoPE illustrated so far can be written in code, as follows: The resulting values are: If we rotate all 4 pairs in the 8-feature query (and key) vector(s), we get the vectors below. (We’ll come back to how we go from rotating one pair to rotating all of them later.) 3.4 How the relative offset changes the attention score in RoPE Note that, as shown above, the query and key heads have eight features each, which we previously grouped into 4 pairs with two elements each, like (1, 5), and (2, 6), and so on. For simplicity here we’ll follow just the first pair in each head, (1, 5), formed by taking the entries at indices 0 and 4: As shown above, in our example, those entries have values (1, 5) in both heads before the rotation. Also, as in the code above, let’s use the notation q =( q 1 , q 2 ) and k =( k 1 , k 2 ) for these two pairs. To compute their dot product, using the standard dot product formula, we multiply corresponding coordinates and add the products: Before rotation, the pairs have a dot product of 1 × 1 + 5 × 5 = 26. Using the rounded coordinates from the previous Figure 18 gives the contribution from the first pair after rotation: We can see why by writing the dot product in terms of the lengths of the two vectors (arrows in Figure 18) and the angle ϕ between them. In both cases, the vector length is √26. The query (position 5) is rotated by 100° and the key (position 2) is rotated by 40°. The vectors (arrows) start from the same direction, so they end up being 60° apart after rotation. Their exact dot product is 26 × cos(60°) = 13. Now let’s see what happens if we move just the key from position 2 to position 3, but we keep its starting coordinates (1, 5) fixed. The key’s rotation changes now from 2 × 20° = 40° to 3 × 20° = 60°. The query still rotates by 100°, so the angle between the arrows shrinks to 40° and the dot product changes from 13 to 26 × cos(40°) ≈ 19.92. So, moving the key one position closer changed the dot product even though we started with the same feature values. Why do these dot products matter? In attention, we sum the contributions from all feature pairs to get the query-key dot products. RoPE can change those dot products and thus the weights that softmax assigns to the value vectors. In the attention equation above, Q′ and K′ are matrices that contain the rotated queries q′ and keys k′ for one head, V contains the values, and d is the head dimension. 3.5 Why shifting both positions preserves the attention score in RoPE Now that we illustrated the computation of the dot products, which is at the core of the attention mechanism, let’s return to positions m = 5 and n = 2. If we shift both positions to the right by three positions, we get m = 8 and n = 5. Each rotation gains 60° (the query goes from 100° to 160°, and the key goes from 40° to 100°), however, the angle between the vectors stays the same. The dot product for this feature pair is still 13. Figure 19: The first pair’s dot product at three different position settings. The key is that the difference between the rotation angles is If we keep the original query and key vectors fixed and shift both token positions by the same amount, RoPE adds the same rotation to both vectors. The angle between them stays the same, so their dot product stays the same. Note that Figure 20 below uses smaller angles than the actual example (20° per position) to keep the picture readable. Figure 20: A shared position shift preserves the relative angle and dot product for fixed query and key features. The examples above and Figure 20 illustrate the main idea behind RoPE. We use each token’s position to rotate the corresponding query and key in the attention module. In the query-key dot product, what matters is how much one vector rotates relative to the other. For each feature pair, this difference in rotation depends only on the offset between the two tokens. To illustrate this in the big picture context, in Figure 21, each row collects the scores for one query. If we consider row 5, the query for “the”, and column 2, the key for “fox”, the attention score between those two is computed as the dot product between the two rotated versions of the query and key vector. Figure 21: The attention scores for one head before softmax. Gray cells mark future keys excluded by the causal mask. What’s the point, again? Let’s once more consider “The red fox jumps over the gray fox.” If we add 100 tokens (e.g., think system prompt) before the sentence, the first “fox” and the second “the” move from positions 2 and 5 to 102 and 105. However, a prefix of any length shifts both positions equally, so they still stay exactly three positions apart. This means, each vector (arrow) gains the exact same additional rotation, which preserves the angle between them and their dot product (assuming we hold the unrotated query and key vectors fixed; in a real model, a new prefix of 100 tokens can change those vectors by changing what the tokens attend to). Figure 22 tracks this comparison in the attention score matrix. The right panel shows the rows and columns for the sentence after we add the prefix. In our toy example, the first pair still contributes 13 to the dot product. Figure 22: The highlighted score moves from row 5, column 2 to row 105, column 102. It keeps the same value if we hold the query and key vectors fixed before rotation. As you may recall, in the original transformer’s sinusoidal encoding, a fixed offset also corresponds to a fixed rotation within each sine/cosine pair. But that shared rotation applies only to the position vectors (added to the input vector) instead of the key and query vectors inside the attention mechanism. Figure 23 compares what this shift changes in the three methods we covered so far. Figure 23: The same position shift under sinusoidal encodings, GPT-2’s learned position embeddings, and RoPE. 3.6 Bonus: Combining the pairs into a complete head in RoPE To see how the relative offset affects the dot product, we focused on one feature pair in the query and key. Now let’s put all four pairs together and see how RoPE rotates the full eight-feature vectors in one attention head. (Please feel free to skip this section if it is too much technical detail.) Figure 24 picks up where Figure 17 left off. We rotate each of the four pairs, then put the results back in the query’s original feature order. Figure 24: RoPE rotates each pair of query features at its own angle. (The same can be applied to the key, which is not shown to keep the illustration simple). Figure 24 above shows the rotated query q′ (at the bottom). Similarly, we could get the rotated key k′ by repeating the same steps at the key’s position. Recall the two equations from Section 3.3. For query pair i with input values ( a i , b i ), let c i = cos( mθ i ) and s i = sin( mθ i ). These are the cosine and sine of the rotation angle for that pair at query position m : Figure 25 below shows the same four rotations between the four pairs as a single matrix multiplication, q′ = R m q . Figure 25: The four rotations from Figure 24 written as one matrix multiplication. The pair labels show where each rotation goes in the full matrix. 4. Computing rotation frequencies in RoPE? A full attention head uses multiple frequencies so that some feature pairs change more drastically (”quickly” in RoPE terms) with position and others change more gradually (”slowly” in RoPE terms). This serves the same purpose as the multiple frequencies in the original transformer’s sinusoidal encodings. However, instead of us assigning these manually, for a head with d features, standard full RoPE computes one frequency for each of its d /2 pairs as follows: Here, d is the head dimension (it has to be even so that all features can be arranged in pairs). i is the index of the pair; it ranges from 0 to d /2−1. b is a chosen base value (hyperparameter) that determines the spacing of the frequencies. θ i is pair i ‘s frequency in radians per token position. Figure 26: rope_params returns the cosine and sine tables for token position 2. rope_split_half takes those tables and the original query and returns the rotated query. As shown in the figure above, first, determines how far to rotate each pair, returning two “tables” that store the cosine and sine of those angles. Or, in other words, computes the cosine and sine of each rotation angle, which depends on the token’s position and the feature pair’s frequency. The array in the function contains the 2 i values from the frequency formula in Section 4: For our eight-feature query, the are , that is, one 2 i value for each feature pair. Suppose we use the common base b = 10,000. And following our earlier examples, we choose a small head dimension of d = 8. In this case, the then computes the four pair frequencies θ i = b −2 i / d as follows: As we can see above, the resulting has one frequency θ i for each of the four feature pairs ( measured in radians) per token position. Next, the then multiplies each pair’s frequency θ i with the token position m , which computes the rotation angle mθ i . In our example, with , we get: Finally, takes the cosine and sine of each angle and outputs them as two so-called cosine and sine tables, and . In both tables, each row is a token position and each column is a feature pair. Since we only use token position 2 here, each table has shape , with the single row shown below: For the example in Figure 26, each table has one row for token position 2 and four columns, one for each feature pair. Figure 27 shows the rotation angle for each pair and the corresponding entries in the two tables. Figure 27: The two tables returned by rope_params for token position 2. The highlighted column gives the cosine and sine used to rotate the first feature pair, (1, 5) . In short, computes the cosine and sine of each rotation angle, which depends on the token’s position and the feature pair’s frequency. In a real-world scenario, if we pass 1,024 token positions for a context size of 1024, and a head dimension of 128, it returns two tensors, and , each with shape since a head with 128 features has 64 pairs. The tables have no batch or head dimension, and we broadcast each table across the batch and head dimensions. So, if we have multi-head attention or grouped query attention with 32 heads, feature pair 0 at token position 2 uses the same rotation angle in all 32 heads. Each head rotates its own feature values, so the rotated queries can still differ across heads. Also, the same entries can be reused across queries, keys, and heads that use the same positions and frequencies. In short, we only need to (pre)compute these tables once per model for all positions up to a maximum supported context size. Technically, it would also be possible to store these tables alongside the model weights; but they are so incredibly cheap to compute that storing them on disk would be overkill. Next, uses those cosines and sines to rotate the actual features in the query and key vectors inside attention. Considering the simple 8-feature example once more, the in the code snippet above splits the query into and and lines up the two entries in each pair. For example, the first slot of each half gives us , the second , and so on. The two expressions inside compute the rotated halves using the equations from Section 3.3: Note that here, everything denoted with a i corresponds to the first half of the query, and everything denoted with b i comes from the second half. The first expression inside computes all the a i ′ values, and the second computes all the b i ′ values. Then, we concatenate the halves, which puts each value back in its original index position via . Here’s the code that produces the rotated query in the earlier Figure 26. The shown reshape just brings the simple toy array into the expected format: one batch item, one head, one token, and eight features. With several positions, each token simply picks up its own row from the and tables. Through broadcasting, PyTorch reuses these same tables across all batch items and attention heads. 5.2 Using RoPE inside causal attention So far, we coded the RoPE utility functions. Let’s now use it in the real context, the attention mechanism. Here, we use the standard frequency schedule for two 8-feature heads and four token positions. We use random query, key, and value tensors to keep the example self-contained, but as mentioned before, you can optionally also check the end-to-end Qwen3 from-scratch implementation with real query, key, and values produced by the pre-trained model. Note that only and but not pass through RoPE. The resulting raw attention scores are stored in a tensor of shape , that is, one row for each query position, and a column for each key position, and 4x4 since we have four tokens where each attends to each other. Then, using a conventional attention code implementation, the causal mask replaces the scores for future keys with negative infinity, so they get zeroed after the softmax. The resulting tensor has shape ; the 4 because we have four tokens, and the 8 because the embedding dimension was originally 8. We can also check that our claim about relative offsets holds, meaning that adding 3 to each position should change the position but preserve the query-key offsets: 5.3 An alternative feature-pairing convention I mentioned the split-halves convention many times throughout this article. The original RoPE paper uses the alternative interleaved convention, as does the original LLaMA implementation (but as mentioned before, most code implementations now use the split-halves, because it makes it easier, in code, to rotate all pairs at once with element-wise sine and cosine operations.) To illustrate the difference between the two layouts, Figure 28 below uses the same 8-feature vector in both rows. Figure 28: Split-half pairs each feature in the first half with the corresponding feature in the second half. Interleaved pairs adjacent features. The interleaved implementation uses the same two equations, but selects alternating features and interleaves the outputs. But mathematically, the layouts are equivalent when we account for the feature order. So, rearranging the entries to makes the split-half pairs adjacent with pairs (1, 5), (2, 6), and so on. If we apply the same rearrangement to both queries and keys, it results in identical dot products. The outputs retain the reordered features, so we compare them with the split-half outputs in that same order. Applying the two layouts to the original vector however would give different scores. 6. Positional choices in modern architectures RoPE has been a staple in most (open-weight) LLMs since the original LLaMA model . Of course, proprietary models may use it as well, but it’s always hard to say anything with 100% certainty about proprietary models. However, in recent years, many architectures have been moving away from full-blown RoPE in every transformer block. In the following subsection, we will discuss some of these variants. For instance the full RoPE implementation we discussed and implemented in the sections above rotates every feature pair in every query and key head. In the following examples in the subsections above below, we will see that models only rotate some features within a head, other models skip RoPE in some transformer blocks, and some models use a different approach and add position information directly to their attention scores. 6.1 Partial RoPE As discussed throughout the article, regular “full” RoPE rotates all dimensions in each query and key head. Partial RoPE is a variant that rotates only a subset, while leaving the remaining dimensions unchanged. For example, MiniMax-M3’s configuration uses 64 rotary dimensions within each 128-dimensional head. That gives 32 rotated feature pairs and 64 unchanged dimensions. The figure below illustrates how that looks like. Figure 29: Full RoPE and partial RoPE in a 128-dimensional head. Note that if we look past the first transformer block, the unrotated features can still contain position information inherited from earlier transformer blocks. Anyways, what’s the point behind partial RoPE? It saves some memory. For example, in the MiniMax-M3 example above, rotating half the features reduces the size of the precomputed cosine and sine tables also in half. So, with partial RoPE, instead of 64 columns, now each table only needs 32 columns. However, note that the KV cache still stores the full key and value vectors for each cached token. For MiniMax-M3’s 1M-token context, rotating 64 of the 128 features halves the size of the precomputed cosine and sine tables, saving 0.134 GB (128 MiB) if the tables are stored in BF16. If the tables themselves were stored in NVFP4, the savings would be 0.038 GB (36 MiB), including the block scales. Note that in experiments with 1B to 8B models, the Fractional Rotation, Full Potential? paper found that rotating around 10% of each head’s features often gave similar training loss and benchmark accuracy to full RoPE. However, full RoPE did better on some evaluations. So, in my opinion, partial RoPE is hardly worth it. 6.2 NoPE layers NoPE was coined in the The Impact of Positional Encoding on Length Generalization in Transformers paper and means no explicit positional encoding . The idea or observation is that an attention layer without any explicit position method still has a causal mask, where a query can attend only to itself and earlier tokens. And this causal masking implicitly provides positional information. Let’s use the first four tokens from our earlier example, . In Figure 30, the query for at position 2 can attend to , , and itself. The query for at position 1 can attend to and itself. Figure 30: Example of a causal mask in the attention mechanism. So, the next question is how can the model use the causal mask as a position signal? Here’s a toy example adapted from the paper. Suppose one feature of the value vectors is 1 for and 0 for the other three tokens. Since the input only contains only one , every visible prefix contains exactly one 1. If all non-masked keys get the same attention score, the softmax assigns every visible token equal weight. So at , attention averages 1, 0, and 0, which gives 1/3. At , it averages four values and gets 1/4. In other words, this output feature equals one over the prefix length, which a later layer can use as a signal for position. Figure 31: With uniform attention, the contribution from The shrinks as the prefix grows. The query for fox assigns The a weight of 1/3. Does NoPE work in practice? The researchers of the NoPE paper trained a tiny 107M-parameter decoder-only transformers from scratch on tasks such as copying, sorting, and addition to do some ablation. When they tested the small model on examples longer than those seen during training, interestingly, NoPE ranked first overall across the ten tasks and ahead of RoPE. The researchers behind the NoPE paper also conducted a separate experiment with a larger 1.3B-parameter models trained on code. Here, NoPE also handled contexts beyond the training length, however, its next-token predictions got worse at the longest lengths tested. Some of today’s open-weight LLMs even go so far and use NoPE in all layers. However, nowadays, it’s more common for models to omit RoPE in at least some layers, where layer here means transformer block. To give a concrete example, SmolLM3 (3B parameters) applies RoPE in three layers and skips it in the fourth, repeating that pattern through 36 layers. Interestingly, the SmolLM3 authors mentioned that this layer pattern improved long-context performance. (Also note that any later layer, following the first one, even though it may skip the rotation, still receives the inputs from preceding transformer blocks, so its queries and keys still receive some sort of position information from earlier layers.) Figure 32: SmolLM3’s architecture (left) and the positional method used in each of its 36 transformer blocks (right). Every fourth block skips RoPE. Architecture diagram from my LLM Architecture Gallery . 6.3 Learned relative position bias Before we get to the overview and look at what modern architectures use, there’s one more method to introduce. For instance, Thinky’s Inkling uses a learned relative position bias in both its sliding-window and global attention layers. It computes the usual scaled dot product between a query and key, and then it adds a certain number to that score before the softmax in the attention score computation. That number depends on the query token’s hidden state and how many positions back the key is. Let’s reuse our example sentence, “The red fox jumps over the gray fox.” (It’s the last time, I promise!) In the figure below, the “the” at position 5 is the query and “fox” at position 2 is the key. Inkling then uses the three-position gap between query and key and the hidden state for “the” to compute the bias, then adds it to the scaled dot product. Figure 33: Inkling adds a learned relative position bias to the scaled query-key dot product before softmax. The numbers are illustrative. This is not an entirely new idea. Some of you may recall Google’s popular T5 (Text-to-Text Transfer Transformer) in 2019, which used a simpler version of this idea. Figure 34: The original Inkling (975B) architecture. The learned relative position bias feeds into the attention module. LLM Architecture Gallery 6.4 How these choices appear in recent models Now, after all this build up, the interesting question is: which architecture uses which method? And what are the recent trends? To answer this, I grouped the entries in my LLM Architecture Gallery by their listed release dates and compared the 2025 releases with the earlier and later releases in the 2026 snapshot. Each gallery card counts once, so multiple variants of an architecture can contribute to the results. LLM Architecture Gallery Figure 35: The architecture diagrams illustrate which attention layers use RoPE, and the bars show the share of gallery entries in each category. Diagrams from my LLM Architecture Gallery . As we can see in Figure 35 above, using RoPE in every softmax attention layer is still the most common choice. However, its use fell from 2025 to 2026, and then also in the latter half of 2026, which could be a trend. Consequently, there is a growing number of models that mix RoPE and NoPE. The groups are still small, so it’s hard to say whether these trends will continue. The NoPE only entries were all recurrent hybrids with softmax attention layers that use no explicit positional encoding. The “Other” category here refers to xLSTM 7B (2025) and Inkling (2026). xLSTM has no softmax attention layers. Inkling adds a learned bias to the attention scores before softmax, as we saw in Section 6.3. Among the models that use RoPE, partial RoPE became more common from 2025 to 2026. It went from a minority of the RoPE models in 2025 to a majority in both 2026 groups, as shown in Figure 36. Figure 36: Among gallery entries that use RoPE, partial RoPE rotates a subset of query and key features in at least one attention layer. Full-width entries rotate all query and key features in every layer that uses RoPE. If you’d like to look up an individual model from the 2026 groups, Figure 37 below shows the 2026 gallery overview with the individual architectures labeled by position type. You can also browse the gallery interactively here . Figure 37: The individual 2026 gallery entries behind the two 2026 bars in Figure 35. 7. Final thoughts So, to recap, we discussed GPT-2 and the original transformer that add a position vector to each token embedding before the first transformer block. Then, modern LLMs switched to RoPE, which encodes relative position inside attention by rotating queries and keys. Surprisingly, RoPE remains as the most popular positional embedding method today. Recently, however, Inkling picked up an idea from T5 to add a learned relative position bias to the attention scores before softmax. And then we also have NoPE-RoPE architectures, which omit RoPE in some layers. It may be boring news, but as mentioned before, RoPE is still the predominant position embedding method, and I think this will carry over into 2027 for sure. Perhaps, though, we’ll see more architectures with an Inkling-like design coming up in 2027. While this may sound “boring”, I am glad that I finally got this article on RoPE (and other position info methods) out, after so many years, so that I can add it to my collection of LLM “fundamentals” explainers. If you are interested in the other architecture building blocks of LLMs, I still highly recommend starting with a GPT-2-style model, which will give you a solid understanding of the building blocks with the least amount of friction. And from there, you can then layer on additional methods (such as RoPE). If you’d like to build that GPT-2-style model yourself, my first Build a Large Language Model (From Scratch) book walks through all the building blocks depicted in the figure below. Figure 38: GPT-2 XL provides a baseline for seeing what newer architectures change. LLM Architecture Gallery For the methods that branch off from the GPT-2 baseline, I’ve collected some of my relevant blog articles below, starting with a refresher on self-attention: Self-attention, causal masking, and multi-head attention: Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMs Modern attention variants (GQA, MLA, sliding-window attention, sparse attention, and hybrids): A Visual Guide to Attention Variants in Modern LLMs The KV cache. Understanding and Coding the KV Cache in LLMs from Scratch SwiGLU feed-forward layers and normalization (LayerNorm and RMSNorm): From GPT-2 to gpt-oss: Analyzing the Architectural Advances Mixture-of-experts layers and expert routing. The Big LLM Architecture Comparison Residual connections and DeepSeek’s manifold-constrained hyper-connections (mHC): Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention Putting some of the pieces together in Qwen3: Understanding and Implementing Qwen3 From Scratch walks through implementing the dense and mixture-of-experts versions in PyTorch.

0 views

My Monitor Was Cutting My Download Speed by 75%

Somehow I am one of those few very lucky people to have the opportunity of owning a 2.5 gig symmetrical line for my internet. But one day, when I wanted to brag to my equally nerdy friends, shock and horror: 600 Mbps down? Luckily my upload speed was still 2.4 Gbps, so I didn’t lose all my bragging rights, but that’s weird, right? If anything, you’d expect the opposite! My ISP is pretty cool. It’s run by this one guy (Hey Keith!), and I know I can always message them and they respond promptly, as they did this time. We tried to run through a few debugging steps, they checked on their end, and everything was normal. I tried my work laptop too, same speed.

0 views