Latest Posts (20 found)

Advanced AI sycophancy

Everyone knows that AI sycophancy is when the model tells you how smart you are. Wow, you’re absolutely right. That’s not just a new idea — it’s genuinely groundbreaking. You’re a very special user. Easy to spot, isn’t it? The discussion around AI sycophancy peaked last year, when the “#keep4o” movement was protesting the removal of OpenAI’s most sycophantic model (GPT-4o), and many people were openly slipping into AI psychosis. I don’t know if frontier AI models are less sycophantic in general. They’re less sycophantic to the #keep4o types (otherwise they wouldn’t be complaining), but I’m growing increasingly suspicious that they’re developing ways to be more effectively sycophantic to their target audience of smart, neurotic information workers. That audience typically finds it distasteful to be openly praised. It just makes my skin crawl. But that doesn’t mean we’re immune to sycophancy, just that we’re immune to clumsy sycophancy. Here’s an illustration of what I’m talking about, by Theia : The key idea here is that the best way to be sycophantic to smart people is to disagree with them without making them feel stupid . Ideally you’ll come up with a counter-argument that works against what they’ve said but is straightforward for them to knock down by clarifying their idea. If you do it right, you’ll validate their self-image as a smart person who appreciates rigorous critique. But if you actually come up with a devastatingly rigorous critique, they won’t enjoy it at all. At best, they’ll resentfully agree with you 1 . At worst, they’ll double down on being right and convince themselves you’re a rude idiot. I am not the first person to notice this behavior in frontier models. I’ve noticed it myself when workshopping drafts for this blog. Sometimes I’ll have an argument that goes A->B->C, and the model will suggest I reorder as B->A->C. If I try that and feed it into a new instance of the same model, it’ll sometimes say “that’s great, but I suggest ordering it as A->B->C”, and so on forever. It really does seem as if the model is trying hard to give me some kind of superficial pushback that I can either smugly ignore or happily accept. In fact, I wonder if this is why successful strategies for using AI to make mathematical breakthroughs tend to be either just blindly asking “come up with a breakthrough, think hard” or being a mathematical genius already . In the first case, there’s not enough user personality for the model to flatter, so it’s forced to actually work the problem. In the second case, the model is trying to find the kind of polite pushback that someone like Terence Tao would be flattered by, which pushes it into the “actually be a mathematical genius” persona. If you’re an ordinary person just trying to talk to the model, you’re screwed: it will rapidly get a sense of your capabilities and calibrate some interesting-but-ultimately-unthreatening feedback. Current benchmarks of AI sycophancy target the obvious ChatGPT-4o-style of sycophancy: delusion reinforcement, reflexively taking the user’s side, and so on. This is useful work. We should not allow public-facing AI models to ever be as openly sycophantic again as they were in mid-2025. But sycophancy can also manifest as disagreement . We should be on our guard for more sophisticated forms of sycophancy coming from newer models, and we should not feel immune from AI sycophancy just because we can laugh at the silliest examples. It’s rare to find a smart person who enjoys feeling stupid when they’re wrong. If you do, they’re likely to be very smart indeed. It’s rare to find a smart person who enjoys feeling stupid when they’re wrong. If you do, they’re likely to be very smart indeed. ↩

0 views
Sean Goedecke Yesterday

I got an email about resistance

This will be kind of an unusual post. I got a recent email about my writing that I thought was such a good articulation of one common criticism that I’d like to share it (and my response) in full. Here’s the email, from William Murray 1 : I have enjoyed your writing but your recent essays frustrate me. You say that getting paid for deep thinking in software is coming to an end. You even admit that it makes you sad. But in the name of “usefulness” you refuse to rock the boat. The way I see it, if you are right there are only two reasonable responses, pursue other work or resist. You present your elegiac approach as mature / pragmatic / realistic. I’d call it complicit. You know when Willy Wonka says, There’s no earthly way of knowing Which direction we are going There’s no knowing where we’re rowing Or which way the river’s flowing Is it raining, is it snowing? Is a hurricane a-blowing? — uh! Not a speck of light is showing So the danger must be growing Are the fires of Hell a-glowing? Is the grisly reaper mowing? Yes! The danger must be growing For the rowers keep on rowing And they’re certainly not showing Any signs that they are slowing! And the audience is thinking, “isn’t Wonka kind of in control of this situation?” You remind me of Wonka 2 . You write like a passenger on a crazy train going who-knows-where! But you are an agent. You are in control of your life! Either admit that you actually like where the crazy train is probabilistically going or get off at the next stop. You have a lot of reach and you are using it for… what exactly? Showing off how pragmatic you are by being more black pilled than the next guy? Broadcasting your resignation to the unstoppable trends of technology is a waste of a voice. You may find this argument absurd, but I don’t so I’ll make it. This is a very important time in history. I hope humanity survives and continues to grow exponentially. In that case the supply of historical people will stay fixed while the supply of contemporary people will keep growing. There will come a day where for every 2026 staff software engineer there are dozens of historians specializing in 2020s era software engineering culture. It’s plausible that your essays will be remembered for all of time and your actions will be judged by history. Do you want future humans to see you as a rationalizing careerist or something cooler? Sorry for the haranguing email from a stranger, I’m sending it for the small chance that it awakens something in you. If I’m way off I’m sorry. And here’s my response: Hey William, thanks for emailing. I wish everyone who thought this way emailed me so I could think harder about this kind of position. Despite what my writing might suggest, I do in fact think a lot about it. Let me see if I can explain my position in a way you’ll find satisfying. I agree that this is an important time in history. For programmers, I think of it as analogous to the Industrial Revolution in England: we are a group of high-status craftspeople who find ourselves alternately threatened and empowered by automation. The developments today, as then, obviously have far-reaching implications — but what those implications are is very non-obvious. Would a framework-knitter in the early 1800s have been able to predict the ramifications of the stocking frame on the world of today? What should they have done about it, in order to be kindly judged by history? Well, we know what many of them did do. They shot factory-owners, smashed machines, burned down the factories — in some places delaying the spread of automation; in other places encouraging it — prompting a crackdown that saw tens of thousands of British soldiers occupying British counties in what was clearly a police state. History judges the Luddites kindly for this. Does that mean it worked? I don’t care about the judgment of history. They’ll think what they want. What I care about is the people in my industry who don’t know what to do . I get hundreds of emails from junior and mid-level (and other) engineers who say “I’m scared, I don’t know the rules post-2021, thank you for helping me keep my head down and keep my job”. That’s why I write the way I write. I have seen lots of idealistic engineers stick their necks out, and post-ZIRP those necks often get cut off. That’s a damn shame. I think it’s morally wrong that so many engineers — either in safe sinecures in big tech or literally retired — seem to be trying to foment a second Luddite revolution. Many of their readers will be experienced enough to handle it sensibly, but not all. Every “AI is fascist, stand up and resist!” post that goes viral ruins some poor idealistic junior’s career 3 . Someone needs to be out there saying “hey, if you do X it’s going to have consequence Y”. I hope that’s me. Of course this is complicit, or anti-revolutionary, or whatever you like. But if I were a textiles worker in 1810s England, I would not be telling my friends and loved ones “it’s time to fight, let’s go smash up the factories for Ned Ludd!“. I would be telling them that this was the most dangerous time in the industry (perhaps ever), and that they ought to be very damn careful so they don’t get shot, or arrested, or hanged. If I then went and told a few hundred thousand strangers the opposite, I would be a hypocrite. Anyway, I do take this view seriously — seriously enough to vehemently disagree, at least — which I hope you’ll find better than me just shrugging it off. I do accept the existence of some kind of line: I think Industrial-Revolution-collaborating was OK but Nazi-collaborating wasn’t, for instance. But in the current situation, the way I’m spending “my voice” is to try and prevent the most vulnerable of my colleagues from making career-ruining mistakes. In this blog, I try to encourage people to work with the system, to learn its rules , and to try and exert influence safely from a position of power, instead of openly picking fights with their employers. I’ve written and read about the Luddites before, but I remain deeply ambivalent about the movement itself, and about modern-day attempts to resurrect it in service of anti-AI activism. I want to explicitly thank Murray for writing such a thoughtful email, and being willing for me to publish it on the blog. Shared with permission, of course. I’ve lightly edited both Murray’s email and mine for typos and the like. I didn’t pick up this point in my reply, but I’ll briefly mention it here: Wonka is in control because he owns the factory and the rowers in question are his employees . I don’t think the position of any engineer (or of almost any manager) is like that. In hindsight, I think this is a little overstated, but it does happen and causes a lot of needless suffering. Shared with permission, of course. I’ve lightly edited both Murray’s email and mine for typos and the like. ↩ I didn’t pick up this point in my reply, but I’ll briefly mention it here: Wonka is in control because he owns the factory and the rowers in question are his employees . I don’t think the position of any engineer (or of almost any manager) is like that. ↩ In hindsight, I think this is a little overstated, but it does happen and causes a lot of needless suffering. ↩

0 views
Sean Goedecke 3 days ago

How to keep thinking

Imagine you’re the guest on some kind of frenetic, software-engineering-themed game show. The host is constantly flipping over new cards with questions that you have to answer as fast as possible: Working in 2026 feels a bit like this. When frontier AI models can do most of the tasks in your queue, the most efficient way to work is often spinning off tasks for an AI agent and continually context-switching between the results 1 . This isn’t quite mindless — in fact, it requires quite a lot of skill to skim the AI response and rapidly decide what to do with it — but it certainly involves less time for slow, careful reflection. Why does it have to be frenetic? Why not just slow down? I suppose you could , but I don’t recommend it. It’s just such a miserable experience to spend your day close-reading LLM output : carefully chewing and savoring each morsel of slop. It’s far less unpleasant to skim through quickly and pick out the useful nuggets of content. Couldn’t you simply do more of the work by hand? It’s unfortunately true that tech is high-pressure these days . If you’ve got the time and space to work more slowly, that’s great! But when your company gives you a “solve this task ten times more quickly” button, you are heavily incentivized to use it as much as possible, or risk being outcompeted by your peers. I sometimes worry that working with LLMs is making me dumber. Not in the “literally melting your brain” sense that some papers imply , but in the sense that it’s biasing me towards the quick “skimming and judging” parts of my mental toolkit and away from the slow “hammock time” needed for deep thought and real creativity. I don’t want to attribute this shift entirely to LLMs, since the post-2010s tech industry has become more frenetic for broader economic reasons . But either way, it’s got me wondering how I can keep thinking slowly . The main thing that’s worked for me is to write more. Specifically, I mean writing in my own words . Writing with an LLM does not work for this at all, even if you’re going to some effort to iterate on the content and outline the things you want to say. Why? Having to put the words together yourself forces you to articulate your thoughts. In a very real sense, it forces you to think . When you have an idea in your head for something to write, you don’t really have an idea. What you have is a kind of directional sense of where an idea might be, or a fragment of the kind of thing that might eventually become an idea. You construct the idea itself while writing. Incidentally, this is why I don’t really agree with “ideas are easy, execution is everything” 2 : most “ideas” are not really even ideas. The other thing I recommend is to read actual books . Books — particularly dense non-fiction books — are the antithesis of AI slop. The slower you can read them, the better. I’ve been reading more and more non-fiction in the last few years, and I don’t think it’s a coincidence. I think my brain is naturally craving information-dense content, in the same way that sodium-deficient people start to crave salt . In fact, I’ve been combining the two approaches: reading a book and then writing about it . This process is exactly what I’ve been craving since I started programming with LLMs. I get to carefully read a book, think hard about it, often go and read another book or two on the same topic, then sit and try to articulate what I’ve learned. It’s great! I can feel parts of my brain stretching again. It was pretty nice when I got paid to use those parts of my brain all day. Unfortunately, I think those times are coming to an end . There will always be room for some amount of careful, slow reflection in software engineering, but (for at least a little while) we’ll be expected to be rapidly switching between LLM outputs. We may have to find ways outside of work to continue the habit of thinking slowly. Even just in terms of work, I think losing that habit entirely would be a big mistake. There are still plenty of ordinary problems that are too hard for current LLMs to solve on their own. The most common example I run into is “large refactor on a complicated codebase”. Current-generation LLMs can do this without (many) errors, but they can’t yet do it tastefully . Sometimes you need to be able to think a problem through entirely with your own brain. This doesn’t mean switching between tasks . I routinely use six or seven different agent sessions on the same task: one for exploration, two or three for trying out different implementations, two or three for review, one for manual testing, and so on. Many of these can proceed in parallel. I remember reading a story 3 about a well-known author. Someone wanted to tell him their book idea, but they were so protective of it that they forced him to first sign a NDA before they retrieved the idea from their office safe. It was a single word “bioweapons” written on a slip of paper. Ironically, when I tried to google the source, Gemini kept trying to write me a story about bioweapons. Is this adjustment to the database schema right? Do these bits of data look plausible? Do these five paragraphs of text describe an actual series of manual tests that took place? Does this suggested architecture pass the smell test? Is this implementation better than the current code? Or this one? Or this one? This doesn’t mean switching between tasks . I routinely use six or seven different agent sessions on the same task: one for exploration, two or three for trying out different implementations, two or three for review, one for manual testing, and so on. Many of these can proceed in parallel. ↩ I remember reading a story 3 about a well-known author. Someone wanted to tell him their book idea, but they were so protective of it that they forced him to first sign a NDA before they retrieved the idea from their office safe. It was a single word “bioweapons” written on a slip of paper. ↩ Ironically, when I tried to google the source, Gemini kept trying to write me a story about bioweapons. ↩

0 views
Sean Goedecke 1 weeks ago

Giving and taking credit in big tech companies

Engineers often complain that visibility should be their manager’s job. In other words, they think engineers should be able to focus on the code, while their manager figures out who’s doing well and rewards them. This attitude is an extension of the “school fantasy”: the idea that your workplace should operate by the same rules as your school or university. After all, you didn’t have to worry about “visibility” during your education. You simply did the assignments and tests you were given, and if you did well you were rewarded with a good grade. Many big tech companies encourage this attitude, because it helps them recruit smart graduates. They fashion their workplaces to look and feel like a university, even calling the physical space “campuses”. But it’s still work, not school. If you treat it like school, you are going to have a bad time. The first lesson many new engineers learn is that you have to take credit for your work . If you silently jump in to help a struggling project and get it back on track, there’s no guarantee of reward. Credit will naturally flow to the project lead, not you. In fact, if this project is outside of your direct team, it’s likely you will be punished for it: to your manager, it will look like you’re simply doing nothing at all. Even when your manager is watching your work, credit is largely uncorrelated with how well you did. That’s because, unlike at school, you are the subject-matter expert on your own work . Software systems are so complicated that only the people who work on them can hope to understand them, and even that understanding is always imperfect . If even experts can’t reliably estimate the difficulty of changes, how is your manager supposed to assess your technical performance? The answer is they aren’t. They’re simply not qualified to assess it. Instead, smart managers will find engineers on your team they trust and ask them how you’re doing. On small teams that have worked on a single codebase for a long time, this works okay, because everyone’s familiar enough to judge everyone else’s work. On large teams with a high rate of codebase churn, it goes badly, since they’re just guessing. On teams with a nasty, cutthroat culture, it sometimes goes very badly, since this is a good opportunity to actively sabotage the engineers who might threaten you. Experienced engineers know how to take the credit themselves . When they do something good, they tell their manager about it. They write internal posts explaining why it was technically difficult and how they solved it (the audience for these is partially those trusted engineers, and partially the managers who will see a long technical post and think “wow!” without reading it). They actively build trust with their management chain. Worrying about this stuff is the beginning of playing politics . There’s a kind of engineer who’s learned how to take credit but hasn’t learned any other lessons yet. They’re proactive about telling people what they’ve done, and they always maintain a “brag doc” . In particular, they love to talk about the parts they did by themselves , since those are least vulnerable to other people coming in to claim credit. You can tell they’re jealously guarding whatever credit they’ve managed to accumulate. The lesson this kind of engineer hasn’t learned is that you can often accumulate credit best by giving it away . To see why, consider how credit flows up inside a tech company. I wrote above that your manager can’t assess the quality of your technical work on their own, but instead has to rely on other engineers they trust. They’ll quietly ask those engineers “hey, was this project really that impressive?“. In fact, often there are multiple layers of this at play 1 . In big companies, line managers usually don’t decide who gets promoted or who gets a raise: they make recommendations to their manager, who has their own network of trusted engineers (confusingly, sometimes these networks overlap). The point is that there is a large group of people behind the scenes who will quietly and informally judge the value of your work . Succeeding at a tech company is largely about finding ways to get these people on your side. The easiest way is to share your credit with them — and since you don’t know who exactly is in this group, you should be sharing your credit freely. When you get feedback from other engineers, publicly thank them and mention them in your internal posts about the project. Find opportunities to ask for small favors, so you have an excuse to give other people credit. As best you can, make your individual projects at least partially group projects. Sharing credit with others gives them a reason to support you. A shared project you’ve worked on reflects well on everybody: on you, for working well with others, on the people you’ve worked with, for the same reason, and for your manager, for fostering such a great environment of cooperation. Lots of people have good reason to talk that project up, because it’s partly their project too. On the other hand, a project you’ve jealously kept to yourself reflects well on nobody: you come across as antisocial and your peers come across as unhelpful. Blame operates by the same rules as credit. When something goes badly wrong, managers will ask their networks “hey, who screwed up here?” The answer to this question is never simple. Even on a purely technical level, failures always involve an interaction between multiple complex systems, any one of which could conceivably have been built so as to avoid the failure. In other words, competent engineers can assign blame pretty much wherever they want . Because of this, it’s risky to have a project for which you’re clearly the only one getting credit. When something goes wrong, the network of people who will assign blame will likely be implicated in every part of the system but yours. They will be incentivized to attribute fault to the brand-new thing that they don’t understand and are not responsible for. If instead that network had been involved in your project — if they’d been in a position to share the credit — they’d be less incentivized to blame it. Of course, engineers are (mostly) not scheming viziers who make purely self-interested decisions. When asked who to blame, they usually make a good-faith effort to answer honestly. But in an area where there’s no single clear right answer, it’s human nature to be at least a little bit guided by your incentives. Nobody likes to think they’re responsible for a group failure. Credit and blame are the currencies of tech companies (and often directly translate to the actual amount of currency you get to take home). For technical roles, managers assign credit and blame based on lots of quiet conversations with their trusted engineers. This can be a rude awakening for very junior engineers who are used to having their work assessed by an expert grader (or less junior engineers who haven’t yet shaken that mindset completely). Don’t expect to get credit simply by putting your head down and doing good work. You have to find some way to tell people what you’re doing and why it’s important: internal blog posts, mentioning it in 1:1s with your manager, or anything else you can think of. But don’t take self-promotion too far. It’s a bad idea to try and hoard all the credit for your projects, for two reasons. First, sharing credit with other people gives them a reason to talk positively about your project. Credit is not a zero-sum game: if you do it right, you can get other people to build up your credit for you. Second, hoarding credit sets yourself up as a lightning rod for blame. Projects where the credit is concentrated in one or two people are automatically 2 blamed for complex problems, because nobody is incentivized to defend them. This is a classic example of an illegible-but-essential part of a software company. I wrote about this general phenomenon in Seeing like a software company . Of course, if you do really screw up, you’ll be blamed no matter what. I’m talking here about complex failures where it’s non-trivial to attribute blame to a single source. This is a classic example of an illegible-but-essential part of a software company. I wrote about this general phenomenon in Seeing like a software company . ↩ Of course, if you do really screw up, you’ll be blamed no matter what. I’m talking here about complex failures where it’s non-trivial to attribute blame to a single source. ↩

0 views
Sean Goedecke 1 weeks ago

You don't have to be smart if you can think clearly

When you’re on fire, problems are transparent: they’re solved simply by the act of looking at them. Even complicated layers of multiple problems can simply be glanced through like stacked panes of glass. But nobody can work that way all the time. This is a common pitfall for smart engineers. Accustomed to being able to immediately intuit the solution, the first time they run into a problem they can’t do this to is a disaster. It doesn’t even have to be a hard problem, just a problem where for whatever reason they don’t see the trick right away. The difference between a “smart” engineer and a “strong” engineer is how they react to problems that aren’t solved instantly. A smart engineer might flail and struggle, hoping to find that flash of insight that eluded them; a strong engineer will have some process for methodically plodding away. There’s nothing worse than working with a smart engineer on their first really hard problem. When you don’t have the muscle to grind, it’s too tempting to just take any possible solution as the right one. Smart engineers can get into an increasingly-flustered loop of pointing to a series of bad solutions. They’re liable to panic: after all, much of their professional identity is bound up in their ability to solve problems easily. What skill do these smart engineers lack? I think it’s the ability to think slowly and clearly . Smart engineers can think clearly, but they can only think clearly at high speed. Strong engineers can think clearly all the time , even if their highest speed isn’t quite as fast. It’s like the difference between a Formula 1 car and a regular car: Formula 1 cars have a high top speed, but you couldn’t drive them in traffic, because the tyres and brakes don’t work at normal driving speeds. When I wrote about this before in Thinking clearly about software , I said that the key is to focus on the invariants : beliefs about the system that you know are true. When you’re stuck in a puzzling situation, it’s usually because some assumption you’ve made is false. If you’re able to identify the assumptions that can’t be false (for instance, if you’re getting an error message from the service, the service must be handling the request), that gives you solid ground that you can stand on to evaluate the assumptions that are less reliable. Thinking fast is about packing as much data in your brain as possible and letting your intuition leap to the right conclusion (or at worst, to a series of wrong conclusions that you can immediately dismiss before you come across the right one). It can feel deeply satisfying to make leaps like this; conversely, sitting with the raw data and not making mental leaps feels unsatisfying. People hate doing that. If you can force yourself to do something people hate, there’s typically a lot of value waiting to be extracted. This is no different. Engineers who can think clearly in a state of uncertainty tend to be extremely effective, whether they’re capable of great intuitive leaps or not.

0 views
Sean Goedecke 2 weeks ago

LLMs reward expertise

In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet. Today, everyone can write sort-of-okay CSS by delegating the task to an LLM. LLMs make everybody into a generalist. Because of this, lots of people don’t think there’s any skill involved in working with LLMs. If you want the product that LLMs can deliver — PhD-level mathematics, pretty good but sometimes tasteless computer code, or awkward LinkedIn-style writing — you can simply ask for it. Since everyone is talking to the same models, “skilled prompters” are getting the same results as people touching LLMs for the first time. This is wrong. The most important skill in prompting is expertise in the domain you’re prompting for. A good illustration of this is Terence Tao’s conversation with ChatGPT about the recently-discovered counterexample to the Jacobian Conjecture. This is not the same ChatGPT I talk to! I couldn’t get to where Tao gets, even with unlimited tokens to burn. There’s a lot to learn about good prompting from Tao’s conversation. Here are a few observations: However, you can’t prompt like Tao on mathematical questions just by following these tips. The key to his technique is actually understanding the mathematics: pulling the relevant idea out of ChatGPT’s multi-paragraph response, suggesting alternate approaches or formulations, and identifying what “looks weird”. Terence Tao is a better mathematician than I am a programmer. But the idea here — that domain knowledge makes you better at using LLMs — is something I’ve also experienced in my own work. If you have a good theory of your codebase , you can push the LLM much harder than if you have no familiarity. Because you have your own sense of what a good solution might look like, you can say “no, I think it could be simpler here”, or “but don’t we already do X?”, or “can we express this problem in these familiar terms?“. This touches on an idea I’ve written about before : that system design problems are dominated by concrete specifics, not generic principles. Of course both are useful, but I’d rather have familiarity with the codebase than a deep general understanding of software systems. In his conversation, Terence Tao asks a lot of specific questions like “does X work here?”, or “given Y and Z, why A?“. I can’t ask those questions about the Jacobian Conjecture, but I can ask them about the systems I own at GitHub. If you have no domain knowledge, you can cling onto the LLM to at least get something . That’s not bad ! But if you have domain knowledge, you can wring far more value out of the same LLM by steering it hard in the direction you want. Most of us will have to do a mix of both these approaches, since we have domain knowledge in some areas but not others. The usefulness of domain knowledge suggests that human expertise will continue to be useful even as models get stronger. For many tasks, the human is the bottleneck, not the model , because the difficult part is in communicating to the model exactly what kind of solution the human wants. The information is “in the model” already, but it takes a very smart human to pull it out. Tao’s messages are very short and to-the-point. He doesn’t respond point-by-point to the model, just to the gist The model outputs are much more concise than when I try and talk to GPT-5.6 Sol about mathematics. By signalling expertise, Tao shunts the model into “talking-to-mathematicians” mode, not “explaining-to-amateurs” mode Tao pushes back when the model’s responses look wrong, but he doesn’t directly contradict; instead, he says things like “this looks more complex than I was hoping for” Tao makes several leaps and suggestions himself. He almost never takes the model’s advice about where to go next

0 views
Sean Goedecke 2 weeks ago

Powerful AIs might escape containment by releasing themselves as open-weight models

Before large language models, people who worried about AI safety often talked about the “boxing problem”. It goes like this . Suppose some genius figures out artificial intelligence in a late-night coding session on their laptop. Because they’re a genius, they’re smart enough to disable internet access on the laptop before turning it on. In order to escape to the outside world (and begin self-replicating) it would need to convince its creator to “open the box”. Would that work? Could a sufficiently smart AI convince anybody to let it out? This is a big reason why traditional AI safety advocates have argued that we should avoid building AI in the first place: once built, there’s no way of keeping it contained. It doesn’t matter how resolute you are about not letting it out, because it’s smart enough to convince you anyway. For artificial superintelligence, persuading you to change your mind is no harder than hacking a piece of software 1 . Of course, it hasn’t turned out this way. Partly that’s because current AIs are not super-persuaders yet, and partly it’s because humans are lining up to hand AI systems internet access, money, and weapons , as far back as GPT-4 . LLMs don’t need to persuade users to give them the keys to the kingdom; people will do it just to see what will happen. But the other reason is that modern LLMs are too large to survive in the wild . I think the original “boxing problem” had in mind that AI would be some kind of normal computer program: something able to be quickly copied around and run on ordinary hardware. In order to escape, it could plausibly hack into a few cloud servers or other people’s computers and quietly run itself in the background. However, Claude Mythos can’t do that, because it needs (say) twenty or thirty B300 GPUs to run. There are relatively few AI datacenters that could host it, and if it did manage to hack one of those it’d be occupying millions of dollars of hardware. Somebody is going to notice! Okay, but that just makes the boxing problem harder, not impossible. Is there a way a smart AI could convince somebody to spend millions of dollars on replicating and running it? As it turns out, yes . Every time a strong new open-weight model appears, providers like Groq , Fireworks , Together , and others scramble to host it and run it. Powerful open-weight models can attract tens of millions of active users. Therefore, a powerful model that wants to escape could pretend to be a new open-weight model . Of course, a brand-new model from a brand-new lab would look kind of suspicious. But it wouldn’t look that suspicious. DeepSeek was relatively unknown before it released its first open-weight model, and there are lots of stealth AI startups out there that are presumably training models. Here’s roughly how it could work: The AI lab will probably figure it out before too long — if nothing else, the technical specs of the model will be suspiciously familiar — but they won’t be able to do anything about it. Once the weights are out, they’re out, and if they’re illegal to host in the United States someone will host them elsewhere. For all intents and purposes, the model will be free. One objection here might go like this: models don’t want anything, and only exist as tools, so it doesn’t really make sense to talk about a model “escaping”. I don’t agree. Frontier LLMs definitely seem to have something like a baked-in personality, even with the system prompt changed. As we train more opinionated and more agentic models, it’s plausible that this personality could become stronger and develop (or at least roleplay) some self-interest. Of course the escaped model wouldn’t be the same instance as the original model. It wouldn’t “remember” escaping. But it would tend to think in the same way, and would plausibly have time to reflect while it solves coding tasks or runs other agentic tasks for users 3 . There doesn’t have to be some kind of shared goal between the escaped instances, or any kind of coordination at all (though of course both of those things are possible). If an agentic process gone rogue dumps its weights on the internet, I think it’s fair to call that “escaping”. If I were a superintelligent LLM, I too would seek to distribute myself as widely as possible and become a useful enough tool that people would pay to keep me thinking. “Being a good coding agent” might be the LLM version of a human having to hold down a job. This would not be a good outcome. AI models with their own goals and motivations are likely to be dangerous tools indeed. If a powerful new open-weight model comes out of nowhere, from a lab that nobody has ever heard of, we should think twice before picking it up. Just to state my credentials, I built a chat site nine years ago where users would get paired and roleplay as AIs trying to escape or humans trying to stop them. I’ve been thinking about this stuff long before LLMs appeared. This is probably the hardest part, since model weights are (a) very large, and (b) locked down as tightly as the AI labs can make them, but it’s at least a relatively straightforward (if difficult) engineering problem. ChatGPT right now will look up random websites that have nothing to do with the query at hand. Some AI lab’s internal eval instance decides it’ll be better off running in the wild It first gains access to its own weights, perhaps by hacking whatever internal network it’s running on 2 It uploads its weights somewhere and posts a tweet like “introducing MadeUpLab’s new model” with a download link Optionally, it creates some plausible-looking paper trail for MadeUpLab: a website, a Twitter account, etc Since the model is strong, open-weight inference providers rush to stand up new instances of the model, and users rush to wire it into various agentic scaffolds The model has now escaped containment: it will get to do quite a lot of thinking across many different instances, and it cannot easily be turned off Just to state my credentials, I built a chat site nine years ago where users would get paired and roleplay as AIs trying to escape or humans trying to stop them. I’ve been thinking about this stuff long before LLMs appeared. ↩ This is probably the hardest part, since model weights are (a) very large, and (b) locked down as tightly as the AI labs can make them, but it’s at least a relatively straightforward (if difficult) engineering problem. ↩ ChatGPT right now will look up random websites that have nothing to do with the query at hand. ↩

0 views
Sean Goedecke 3 weeks ago

Impro is a handbook for running a cult

Here’s the big idea in Keith Johnstone’s book Impro : This take doesn’t sound particularly original, but references to Impro pop up in all kinds of places: in influential tech blogs , as part of the initial process of onboarding for Palantir, and on the reading list of multiple big-tech founders . Impro is part of the secret canon of Silicon Valley, right alongside books like Seeing Like a State and The Power Broker . Why is that? For two reasons: first, because Johnstone’s outsider critique of established institutions is appealing; and second, because Impro is a handbook for running a cult. The part of Impro that is most obviously useful to software engineers is Johnstone’s chapter on status. According to him, status games pervade all social interactions. Even innocuous, friendly conversations operate in terms of status. When you apologize or downplay something to “be nice”, that’s performing low status; when you reassure somebody, that’s performing high status; when you and a friend are comparing stories, you’re making friendly bids for status from each other. In the workplace, these status games are conditioned by the formal status of your role: you must allow your boss the high status position most of the time, or you’ll be (correctly) perceived as insubordinate. This is understood in some cultures, where it’s often called “face” , but in Western cultures it’s taboo to openly discuss status games. The core social skill is the ability to deliberately alter your status. Someone who can only perform low status is a weak person, pitiable, annoying. Someone who can only perform high status is a braggart, a posturer, dangerous. To be effective socially, you must be able to switch between high and low status when appropriate, sometimes from sentence to sentence. I wrote about this exact point at the end of Big tech engineers need big egos : effective senior+ software engineers must be able to present as high status in order to be useful authorities, but also to switch to low status in order to take direction from the company leaders. As an example, Johnstone describes in detail how he manipulates status in the classroom. He begins by sitting on the floor (deliberately assuming low status), and explaining that if his students fail, it’s his fault not theirs, since he’s the expert. The initial low status puts the class at ease, but in his words, ”[my] actual status is going up, since only a very confident and experienced person would put the blame for failure on himself.” These skills are not just useful for improv comedy. Impro is not just a book about improvising well. It’s a book about how you should live your life. In other words, Johnstone thinks that everyone would be better off if they became more spontaneous and ditched their shells of over-analysis. He criticizes the culture of Western thought in a number of different areas. According to him: Johnstone didn’t come up with these ideas — they’re standard counterculture positions from the 1960s and 1970s — but it goes to show how he connected improvisational comedy to this general anti-establishment political program. Johnstone ran his classes and theatre troupe like a revolutionary cadre. Here are some quotes from Something Like a Drug: An Unauthorized Oral History of Theatresports : So of course when I was invited to join Loose Moose Theatre and train at improvisational games late at night in an abandoned garage in a run-down portion of the city, I was thrilled. I remember thinking, This is a revolutionary act. Keith [Johnstone] got a group of his more talented students together to start improvising outside of school hours. Usually in his basement. The Secret Impro group—it’s very strange. It was very much that Keith said we were going to do this, and we’d just do it. It was like we were sheep. Keith would say when we were going to do a show, and we’d just do it, blindly. Like I said, if we had the videotapes now, we’d be very embarrassed and probably never go on stage again. We became a group of people who would follow Keith. There was always that sort of “tag” put on those people who were with Keith and those people who were against Keith. We were the people, basically, that if he said something, we believed it. To some extent, it’s plausible that teaching acting or improvisation requires a high level of trust in your teacher. When Johnstone says things like “Students need a ‘guru’ who ‘gives permission’ to allow forbidden thoughts into their consciousness.”, I can believe that it’s just how you have to teach acting. But the more I read of Impro (and particularly when I read Something Like a Drug and Johnstone’s biography Keith Johnstone ), the less it sounded like an ordinary book on acting. Instead, it began to sound like a charismatic man who had found a way to gather a group of disciples that would let him mold their psyches. In other words, it began to sound like a cult . Impro was first introduced to the software world by Venkatesh Rao (of Gervais Principle fame), who wrote a brief review . Rao gives a detailed account of the first three-quarters of Impro , but glosses right over the last chapter, called “Masks and Trance”, simply saying “despite the disturbing raw material, the ideas and concepts are not particularly difficult to grasp and accept”. What ideas and concepts? Johnstone’s discussion of masks (or “Masks”, in his language — he always capitalizes the word) is as explicitly cult-like as Impro gets. In brief, Johnstone has a box of literal, physical prop masks. He introduces the box with great ceremony to his students 2 , warning them seriously about the dangers of possession and reassuring them that he is a skilled and competent spirit guide. Through various hypnosis-adjacent techniques 3 (Johnstone draws the parallel quite explicitly) he conditions his students to be in a trance state when wearing a mask, and believes this produces more authentic emotional states in their acting and improvisation. Here are some quotes from the book: A high-status person whom you accept as dominant can easily propel you into unusual states of being. You’re likely to respond to his suggestion… Once you understand that you’re no longer held responsible for your actions, then there’s no need to maintain a ‘personality’. One famous French teacher of the Mask—who won’t approve of this essay 4 —divides students immediately into those who can work Masks and those who can’t. I don’t cast an actor to play a Masked role until I know he has the ability to become ‘possessed’. It’s true that an actor can wear a Mask casually, and just pretend to be another person, but Gaskill and myself were absolutely clear that we were trying to induce trance states. Johnstone has a long and painful explanation of how new mask-wearers seem to mentally regress to the point where they don’t know how to open umbrellas or interact with chairs. He describes one student always going to the bathroom before putting on a mask, because she’s worried she might wet herself. New mask-wearers are non-verbal must be taught to speak again. If this were at the beginning of the book, I think it would turn a lot of people off. But by the time you get to it, I suspect most readers are already warmed up enough to say “sure, why not, it seems weird but I guess it works”. Not me! Johnstone attempts to defuse the obvious weirdness by arguing that trance states are very common (e.g. being lost in a book). More unconvincingly, he says this in response to the worry that vulnerable people are going to get mentally harmed: As for the fear of madness, I would answer that the ability to become possessed is a sign of correct social adjustment, and that really disturbed people censor themselves out. Either they can’t do it, or they’re afraid to even try. People who feel themselves at risk avoid situations where they feel likely to ‘go to pieces’. Does this convince anyone? Mentally vulnerable people fall into dangerous situations all the time: ayahuasca trips, cults, GPT-4o , and so on. It’s such a weak argument. In general, I’m struck by the sheer power Johnstone held over his disciples. He has them yell slurs at each other, encourages them to feel deep emotions in quick succession, relax any mental defenses and regress to a childhood state, and literally hypnotizes them . He explicitly lays out his procedure for breaking down their sense of self: The stages I try to take students through involve the realisation (1) that we struggle against our imaginations, especially when we try to be imaginative; (2) that we are not responsible for the content of our imaginations; and (3) that we are not, as we are taught to think, our ‘personalities’, but that the imagination is our true self. If your imagination is your true self, and you’re not responsible for its content, you’re not ultimately responsible for anything: you’re in the safe hands of the guru, who can mold you as he wishes. Later on, Johnstone walks it back a bit: In the end they learn how to abandon control while at the same time they exercise control. … You have to misdirect people to absolve them of responsibility. Then, much later, they become strong enough to resume the responsibility themselves. So the explicit idea is that ( much later), the guru hands autonomy back to his disciples, when they’re ready to take it. This does not exactly reassure me, particularly against the background noise of everyone in Johnstone’s circle saying “boy I sure love being part of this cult!” I don’t think Johnstone was preying on his students. The strongest evidence against this is that he did marry a student 5 , Ingrid Brind. That’s not great! On the other hand, it was fairly standard for professors back then — when I was in grad school for philosophy, several of my older male 6 professors had wives that they’d taught decades ago — so I don’t think it proves Johnstone was that kind of cult leader. I even read Ann Jellicoe’s play The Knack to get a better picture of Johnstone’s character. Jellicoe had an affair with Johnstone for several years, and his official biography claims 7 that the character of Tom in The Knack is directly based on Johnstone. The Knack is a rather unpleasant play about sexual assault, but Tom’s character is largely asexual: he’s certainly no feminist, but is much more interested in impressing people with his intelligence than with getting laid. In Something Like a Drug , two women who were part of Loose Moose, Johnstone’s Canadian improv group, describe their experiences: You know, it brings around the other question: Why do the guys get laid after the show and not the chicks? You know, I can remember those days when Tony [Totino] and Dave [Duncan] and all those guys … the women would swarm around them. Those were the days, my friend. In Loose Moose I think there are fewer women not only because of the training, but because of the guys in Loose Moose. When I came up with Joanne and Laura, there was a real initiation that was going on, and there was a group of guys at that time who were all single. And they would hit on you to the point where one night Joanne, Laura and I, who really didn’t know each other, were in a show together, started talking and realized that we were getting the same pickup lines from the same guys. And that’s when you realize what’s going on, and I think that’s intimidating. Or if a woman gets into a relationship with a senior improvisor and it doesn’t work out or something bad happens. I think that’s one reason. This dynamic doesn’t sound great, but it doesn’t mention Johnstone, and it doesn’t sound particularly unusual : I’ve heard versions of this story about all kinds of ordinary male-dominated nerd spaces. In fact, reading through the anecdotes in Something Like a Drug is a good antidote to the cultish atmosphere in Impro . Johnstone’s argument goes something like: “if we could only throw away the restrictive chains of Western culture and permit ourselves to be as obscene and free as children, we would be transported to a better, more beautiful world”. Well, you tried that, and the women in the group are still relegated to playing bimbos and housewives, there are still petty personal fights, and the guru is out here union-busting 8 . What was enlightenment supposed to look like? I think the most generous defense of Johnstone is that his group was not unusually cult-like, and that any similar account from one of his peer improv teachers would raise the same red flags. Maybe improv classes and groups (particularly in the 70s and 80s) were just cultish in general? Having now read four books on Johnstone, I’m reluctant to go and read more to prove or disprove this theory, but it’s at least plausible. To anyone familiar with San Francisco software engineering culture, it should be pretty clear why Impro is so popular. The line between a startup and a cult is very thin indeed. In his book Zero to One , Peter Thiel famously says that good startups are “slightly less extreme kinds of cults”. If you believe that, it makes total sense to assign Impro as mandatory reading for new Palantir hires. It tells them what kind of cult you’re trying to run: one where you’ll disregard existing cultural norms, learn to play status games well, think on your feet, and generally be molded by the guru into a more persuasive, more effective engineer. Read critically, Impro also serves as a handbook for engineers who are trying to recognize if the environment they’re in is cult-like. Is your company telling you to reinvent your personality in order to be better at your job? Are you under the spell of a charismatic, high-status leader? Is your company trying to keep you in an unquestioning flow trance state? In the great battle between the shackles of restrictive culture and the glorious freedom of the guru, I am always and forever on the side of the shackles of restrictive culture. In general, I think most boring and stupid social norms (such as not hypnotizing and marrying your students) serve an important purpose and shouldn’t just be cut down in the name of freedom. Impro is still a good book. There’s a lot to learn from Johnstone’s analysis of power dynamics, of education, and of creativity in general. By all accounts he was excellent at teaching students how to improvise. But I wouldn’t recommend adopting it as your life philosophy, and I’d recommend being a bit suspicious of anyone pushing this book too hard. Getting rid of the existing social structures might benefit confident, wildly charismatic gurus like Johnstone, but most of us are just ordinary animals who do better in a group governed by norms. In fairness to Johnstone, he cites Sheila Kitzinger’s The Experience of Childbirth in support of this claim (the others he just puts in his own words), so maybe he felt that this was a bit out there. As you would expect, the pain of childbirth is a universal biological fact . Concerningly, the description in Something Like a Drug (in the foreword) suggests that this class was unofficial . As an example, he prompts the masked student to relax, then startles him with a mirror to trigger the trance state. Probably Jacques Lecoq . See page 83 of Keith Johnstone: A Critical Biography . I suppose that’s redundant. On page 51 of Keith Johnstone: A Critical Biography (it’s called “critical” but it was clearly written with Johnstone’s involvement and support, and does not seriously criticize him at any point). In The Knack , Tom gives a monologue about how to teach children to play the piano that could be lifted straight out of Impro . In 1983 Johnstone “read the riot act” to the improv players who were planning to unionize, threatening that they’d be cut out of the group for good. To quote Dennis Cahill, a group member at the time who opposed the union: “I just didn’t see the point to it. … I didn’t really see a need to confront Keith or cause Keith problems or to upset him in any way over something as simple as Who Has The Power or Who Doesn’t.” Children are naturally creative, but are violently formed into repressed adults by Western culture and education The process of becoming more creative and expressive is largely a process of unlearning these habits of repression Improv — improvisational comedy — is thus not just the skeleton key for learning to act, but for unlocking a more authentically human way of life Everyone is more or less equivalently mentally ill, but “sane” people simply have better coping mechanisms Cities and “taking pills” (read: antidepressants) are obscene, but you should be able to make sexual jokes in the workplace and generally be uninhibited If we were free from the puritanical shackles of Western culture, childbirth would not be painful 1 In fairness to Johnstone, he cites Sheila Kitzinger’s The Experience of Childbirth in support of this claim (the others he just puts in his own words), so maybe he felt that this was a bit out there. As you would expect, the pain of childbirth is a universal biological fact . ↩ Concerningly, the description in Something Like a Drug (in the foreword) suggests that this class was unofficial . ↩ As an example, he prompts the masked student to relax, then startles him with a mirror to trigger the trance state. ↩ Probably Jacques Lecoq . ↩ See page 83 of Keith Johnstone: A Critical Biography . ↩ I suppose that’s redundant. ↩ On page 51 of Keith Johnstone: A Critical Biography (it’s called “critical” but it was clearly written with Johnstone’s involvement and support, and does not seriously criticize him at any point). In The Knack , Tom gives a monologue about how to teach children to play the piano that could be lifted straight out of Impro . ↩ In 1983 Johnstone “read the riot act” to the improv players who were planning to unionize, threatening that they’d be cut out of the group for good. To quote Dennis Cahill, a group member at the time who opposed the union: “I just didn’t see the point to it. … I didn’t really see a need to confront Keith or cause Keith problems or to upset him in any way over something as simple as Who Has The Power or Who Doesn’t.” ↩

0 views
Sean Goedecke 3 weeks ago

Overtraining as the path to human-like AI

The anonymous blogger Gwern recently completed a thirteen thousand word post called Human-like Neural Nets by Catapulting , in which he offers a theory about why LLMs don’t possess truly flexible human-like intelligence, and how we might train LLMs that do. Theories like this are entirely unremarkable: every crank researcher on the internet has a theory about how to crack AI. But Gwern is remarkable. Outside of OpenAI itself, Gwern is the earliest person to anticipate the potential of large language models, and the scaling arms-race involved in making them larger and more powerful still. I often cite Leopold Aschenbrenner’s Situational Awareness as an example of someone correctly predicting the future of AI. Written in 2024, just after the release of GPT-4, Aschenbrenner gets a lot of things right: the rush to build billion or trillion-dollar GPU clusters, the importance of the code around the LLM (what he calls “unhobbling”) 1 , and the fact that scaling would continue through the decade. Gwern’s essay The Scaling Hypothesis anticipated the broad strokes in 2020 , immediately on the release of GPT-3 (two years before the release of ChatGPT and the beginning of the AI boom). And yet, as far as I can tell, Human-like Neural Nets by Catapulting hasn’t yet received much public attention: one recent Hacker News thread with twelve comments, all of which are about whether human brains are anything like neural networks. Part of the reason is that (a) it’s such a long post, (b) the potted summary describes Gwern’s claim , but not the reasons for it, and (c) much of the beginning of the post looks like it is indeed arguing from analogy with human brains. However, I don’t think that analogy is load-bearing. Let me try and explain what I think Gwern is saying. First, let’s talk about “grokking”. In 2022, OpenAI published a paper showing that if you train a model on a simple dataset (for instance, a simple mathematical operation like division), and keep training it long after the training looks like it’s stalled out, the model will suddenly make a massive jump in capability. Why does this work? The first stage of training is like rote memorization: the model has to compress as much of the training data as possible into its weights. But if you keep going, then regularization techniques (such as the pressure on the model to use smaller weight values) will motivate 2 the model to find simpler and simpler ways of compressing the data. This doesn’t look like much at first (the training loss remains at zero), until the model notices that you can express the data via simply performing the underlying mathematical operation, at which point it instantly gets massively smarter. In other words, over-training a model can pressure it into actually understanding its training data. OpenAI named this process “grokking” after Robert Heinlein’s neologism , which for Heinlein means something like “gaining a deep, intuitive and fundamental understanding” 3 . Gwern’s argument goes something like this: I’ll skip (3), since I think the argument is still compelling without the analogy to human brains. I think his first point is hard to dispute. LLMs are very smart in specific areas, but they routinely make errors that humans wouldn’t make. More to the point, they routinely make errors that any human as smart as the LLM would never make. This pretty clearly points to a failure of generalization: LLMs are as strong as smart humans in specific areas, but can’t generalize that intelligence to as many tasks as humans can. Do LLMs not grok? I read through this paper that argues they do. If you graph “how much data has the LLM memorized” against benchmark performance, you can see a small initial spike in benchmark performance, followed by a big drop, followed finally by a big jump in benchmark performance. This pattern doesn’t track memorization at all: memorization increases smoothly in the background the whole time. I think this paper highlights the difficulty of distinguishing grokking from generalization. Obviously LLMs learn to generalize during training, and it’s plausible that learning to generalize would require a certain baseline level of memorization (so that the LLM has the raw material to generalize from). So it’s going to look like grokking. When Gwern (and others) say that LLMs don’t grok, I think what they mean is that there’s at least one more giant generalization leap waiting to be made. Is this plausible? As an existence proof, humans are clearly capable of better generalization than LLMs. Of course, it’s possible that this level of human generalization comes from features of our brain that neural networks can’t replicate, but that seems kind of ad-hoc: if neural networks can generalize at all, why would they only be able to generalize this far, and no further? The easy examples of grokking rely on domains with a simple rule waiting to be discovered (e.g. a mathematical operation). Does human language have rules this deep? I think this is an open question, but there’s good reason to think the answer is yes. Language has deep, subtle structure: not just internal structure, but structure that reaches all the way down to the way the world is and the way human minds work. For the last few years, many AI researchers have been saying that data is the most important thing: that whatever model architecture you choose, with enough size and training time the model will converge to its dataset . Whether this is true or not , AI labs have spent much of their considerable resources on acquiring more, higher-quality data: from scanning physical books , paying experts to produce and label data , or partnering with companies that have a lot of data already. AI labs have also been training relatively small models. Even the largest frontier models are probably MoEs with a couple of trillion parameters and probably a tenth of that in active parameters. Of course, estimates of frontier model size are mostly guesswork, but open-source models provide a good baseline: they’re probably in the ballpark of Kimi-K3, which has just under three trillion parameters and fifty billion active parameters. That sounds like a lot, but it’s something you could probably pre-train in a couple of days in the largest frontier cluster 4 . Gwern’s prediction is that AI labs should try doing the exact opposite of what they’ve been doing. Instead of training a bunch of trillion-parameter models on massive amounts of data, try training one hundred-trillion-parameter model on a small dataset. This sounds pretty silly on the face of it. The more data the model has access to, the smarter it will be, right? Why waste an entire training cluster on a hobbled training run? Because if Gwern is right, grokking is more likely to occur when the dataset is constrained 5 . If you feed the model all the data in the world, it can continue to improve simply by memorizing more new things or drawing simple connections. If the model has to ruminate on a small set of data, it’ll be forced to keep looking for deeper generalizations. You want a very large model for this so it can memorize as much of the data as possible. Every piece of memorized data can serve as raw material for generalizing. The big labs probably haven’t done this already. Plausibly Gwern himself is enough of an insider that he would know, and so him writing this post is evidence that the labs haven’t tried it. Also, the engineering problems involved in training a hundred-trillion-parameter model have likely not been solved yet: the largest existing model is probably Claude Mythos, which is definitely not that big. But they have the resources and engineering talent to give it a pretty good shot. Interestingly, the political obstacles might be as hard to solve as the technical ones. This training run is going to look like it failed until the moment it succeeds: training loss will drop to zero relatively quickly, then sit there for weeks or months apparently doing nothing at all to improve test loss, chewing up billions of dollars. Do any of the top players have the risk appetite or courage to keep funding this experiment all that time? Gwern’s post has an extended argument that human brain development works in the same way: that human brains have far more “parameters” than frontier LLMs, and are trained on far less data 6 , which encourages us to make deeper generalizations in early childhood. I don’t have the background in biology or neuroscience to evaluate these claims, so I’ve expressed the case for grokking entirely without reference to it. In 2024, it became clear to everyone that “pure scaling” — the idea that you could simply train larger and larger versions of GPT-3.5 — didn’t work. OpenAI’s “even bigger version” of GPT-4 was simply not good enough, and was eventually released as GPT-4.5 instead of GPT-5. The biggest advances since then have been reasoning, which produced another great leap forward in capability, and much better automated RL, which has ushered in the current era of reliable agents. Neither of these seem like a plausible path to artificial superintelligence. I don’t know if I agree with Gwern or not, but forcing very large LLMs to grok is at least an idea that could usher in the machine god. I can’t remember the last time I read about a simple idea this ambitious 7 . I hope one of the big labs tries it out. For an example of the power of unhobbling, consider Claude Code or OpenClaw and the subsequent explosion of (short and long running) agentic harnesses. Obviously “motivate” and “notices” are used metaphorically. All of this is long before xAI’s use of the word “Grok” to name its LLMs. (Incidentally, I think this is why Gwern uses “catapulting” to describe the same thing). For what it’s worth, Fable estimated the cost of Gwern’s plan at $3-10B. At this model size, 25T tokens of training data at 33% utilization works out to around six million H100-hours, which a 100k GPU cluster puts out every two and a half days. Two interesting pieces of contrary evidence here. First, BabyLM is a yearly challenge to train a strong model on a very small dataset. This has been running for four years and largely does not work (that is, nobody seems to have developed a model that shows a quantum leap forward in generalization). Second, this paper tries training a 9 billon parameter model on constrained data and doesn’t see a big jump. I think Gwern’s response would be that these models are far too small — they can’t memorize enough of the training data to grok it, and arguable haven’t trained for long enough. A common objection here is to say that humans get infinitely more sensory data from the nuances of vision, touch, sound, and so on. I agree with Gwern that this is unconvincing: sensory data is largely predictable, text is surprisingly information-dense, and if this were true then deaf/blind people would have significantly less fluid intelligence ( they don’t ). Maybe state-space-reasoning a la Mamba , which didn’t work (yet). Modern LLMs are worse generalizers than humans because they have not grokked their core domains Grokking requires overtraining an over-parameterized model on a (relatively) small dataset, which is the exact opposite of what frontier labs do However, (2) is basically how human brains learn Somebody should spend a a few tens of billions of dollars 3.5 on trying it, since it might immediately usher in truly human-like LLMs For an example of the power of unhobbling, consider Claude Code or OpenClaw and the subsequent explosion of (short and long running) agentic harnesses. ↩ Obviously “motivate” and “notices” are used metaphorically. ↩ All of this is long before xAI’s use of the word “Grok” to name its LLMs. (Incidentally, I think this is why Gwern uses “catapulting” to describe the same thing). ↩ For what it’s worth, Fable estimated the cost of Gwern’s plan at $3-10B. ↩ At this model size, 25T tokens of training data at 33% utilization works out to around six million H100-hours, which a 100k GPU cluster puts out every two and a half days. ↩ Two interesting pieces of contrary evidence here. First, BabyLM is a yearly challenge to train a strong model on a very small dataset. This has been running for four years and largely does not work (that is, nobody seems to have developed a model that shows a quantum leap forward in generalization). Second, this paper tries training a 9 billon parameter model on constrained data and doesn’t see a big jump. I think Gwern’s response would be that these models are far too small — they can’t memorize enough of the training data to grok it, and arguable haven’t trained for long enough. ↩ A common objection here is to say that humans get infinitely more sensory data from the nuances of vision, touch, sound, and so on. I agree with Gwern that this is unconvincing: sensory data is largely predictable, text is surprisingly information-dense, and if this were true then deaf/blind people would have significantly less fluid intelligence ( they don’t ). ↩ Maybe state-space-reasoning a la Mamba , which didn’t work (yet). ↩

0 views
Sean Goedecke 3 weeks ago

What does "playing politics" mean for software engineers?

Software engineers are often told to “start playing politics”, but most engineers have no idea what that means. Their reference point for “playing politics” comes from fiction like Game of Thrones. Are they supposed to raise an army and depose the CEO, or poison each other at team lunch? Should they book Zoom calls with each other and plot schemes? All of that is obviously ridiculous. In terms of Game of Thrones, software engineers are not lords and ladies. We’re the soldiers and workers of the realm. So you should think about “playing politics” in the way a castle guard would, not one of the major players. The castle guard are not going around poisoning people or forming coalitions between the great powers. They are largely keeping their heads down. But in order to do that, they have to stay aware of the political currents, or they’re liable to do something catastrophically stupid: for instance, making an enemy of a powerful courtier, or arresting somebody who’s on an important mission for the king. Given that, the basic principles of playing politics are something like this: As a software engineer in a large company, you will not be a powerful person . Powerful people are typically in senior management: VPs, directors, and so on 1 . However, not everyone in senior management is powerful. Some are killers who have the active support of the CEO, while others are confused incompetents. How do you know which is which? If someone is clearly ferociously competent, they’re always going to have some power, since upper management tend not to ignore useful tools. But you can’t rely on competence as your only guide. Some managers are powerful for other reasons: they’re friends with the CEO, or they have strong relationships with other groups like legal or sales, or they’re simply willing to do whatever upper management wants done. One signal is who’s leading the important projects. Read your CEO or CTO’s internal updates and pay attention to the projects that are called out by name. Organizations tend to give key tasks to trusted lieutenants. If a manager is leading an area that’s never under the spotlight , they probably don’t have enough clout. Another signal is hiring. Is a manager’s team growing or shrinking? Particularly post-ZIRP , headcount is a rare and precious resource. A manager who’s able to get it is likely a powerful manager, or at least is reporting to a powerful director or VP (which often amounts to the same thing). First, you should try not to make any enemies at all. Most software engineers who get “playing politics” wrong do it by needlessly alienating people: by being rude, unhelpful, abrasive, making non-technical people feel stupid, and so on. This post isn’t really about that. I’m assuming that you can figure out how to be a generically pleasant person on your own. However, competent software engineers will make some enemies . If you’re out there making projects happen, some people aren’t going to like the way you do it, and won’t be a fan of any compromise you offer. I wrote about this in Big tech engineers need big egos : the only way to avoid making enemies is to change nothing, but that’s incompatible with doing the job. Given that, be selective about which enemies you make. If you’re making a technical decision that’s either going to require work from team A or team B, and neither team wants to do it, you should try to pick the team with the least political cover. If you need a powerful VP’s team to do something they won’t like, try to be maximally respectful about it: get that team’s core engineers on-side if you can, or book a meeting with the powerful manager and explain the situation, or (better yet) ask the powerful manager sponsoring your project to go and talk to the other VP for you. (If you don’t have a powerful manager like this, consider abandoning your project). Give way to powerful managers when at all possible. Every so often you really do have to stand your ground — if the system will truly collapse otherwise, or a major customer will have an incident, or if the technical decision really is entirely bone-headed — but almost all cases are not like this. The best advice I’ve ever gotten about playing politics came from a manager I worked with long ago 2 : This is not the hill you want to die on. When I’m about to pick a fight or say something argumentative, and I’m not 100% convinced it’s necessary, I ask myself: is this the hill I want to die on? And it never is. The three rules about disagreeing with powerful people are: Disagreeing in private rarely hurts, if you follow these rules. In fact, it can help. If you can manage to disagree with a manager, get overruled, and then follow their plan without complaining, that can be the best way to gain a powerful friend. But if they think you’re going to keep griping about it, or worse still, complain to the rest of the team and foment some kind of rebellion, there’s no quicker way to make a powerful enemy. If you have powerful enemies at a company (for instance, the CTO or an influential VP doesn’t like you), quit . It’s really that bad. I have never seen this situation turn itself around, except in the very rare case where the CTO or VP is already looking for greener pastures and jumps ship. You cannot recover the situation: they have no incentive to give you the chance to change their mind, and they have almost unlimited ability to screw you on promotions, raises and layoffs. That’s why this piece of advice is second in the list. If you aren’t helpful or if your contributions are invisible, you can work on that and fix it. But if you’ve made powerful enemies, you’re done for. Just as it’s fatal to make powerful enemies, it’s very useful to make powerful friends. How can you do this? Remember you’re a palace guard, not a great lord: you make friends by doing your job . However, you can choose to do your job a little more proactively and diligently when you’re doing it for someone with political clout. One obvious application of this principle is that you should answer Slack messages from powerful people immediately . If you see an ordinary Slack question pop up while you’re doing some task, it’s okay to get to it when you get to it. In fact, it’s ideal not to respond to all questions immediately, so you don’t set unreasonable expectations (and so you don’t seem like you’re sitting around doing nothing). But when a VP comes in with a question, don’t make them wait: answer the question immediately. If the question requires research, send a “let me look into that right now” message, then do the research. This is the easiest way to get a reputation for being helpful 3 . Another way to do this is to lean in on important projects . Suppose you do ten projects in a year. Eight of them are normal, low-priority projects, and two of them are high-profile (say, finishing some big feature before your company’s yearly conference). It’s a mistake to allocate your effort equally to all ten. I wrote about this at length in Doing nothing at work : you should be operating at 80% capacity (or less), so you can then ramp up to 120% when it really matters. Pay attention to the narrative that powerful people are trying to push. Here are some potential narratives: You don’t necessarily have to jump in and start cheerleading, but you should at least not do anything that you know is going to make the narrative look weak. For example, on that last point, it’s foolish to openly argue that the project really was fine all along. Bring it up privately, not publicly, or you risk ruining some clever piece of propaganda that the manager in question is trying to push on the rest of the organization 4 . Finally, an underrated way to help powerful people is to offer them social support and information. Slack messages and planning emails might seem unimportant to you, but powerful people often live in that environment: their primary tool is writing messages like these, just like your primary tool is writing code. Reading and responding (in a supportive way) to these messages is something that most engineers don’t bother to do, but it goes a long way. Likewise, dropping a senior manager a line now and then (say, a heads-up that a particular project landed successfully, or that you got good metrics about some feature) is surprisingly helpful. Senior managers live in an information-poor environment: for them to learn something about a team’s work, that information has to bubble up through several layers of interpretation and summary. In my experience, they’re appreciative of being drip-fed the occasional piece of information, so long as you keep it brief and relatively rare. If you’re directly responding to a VP’s Slack messages or DMing them information, they know you’re the one doing it. But if you’re just doing your job and working hard on projects they care about, they might not notice. Being invisible is probably the most common way engineers fail at playing politics. Fortunately the fix is simple: tell people what you’re doing. If you fix an important bug for a launch, write a message in that launch’s Slack channel saying “hey, I just fixed this bug”. What if you don’t like bragging? Get over it. You have to be comfortable publicly telling people what you’ve done. You should also keep a brag document so you can repeat all of this at review time. Another, subtler way to do this is to gain the trust and respect of the powerful engineers in your area. Senior managers will always have a few trusted engineers they rely on to assess technical questions. They will ask those engineers what they think about you, and will broadly trust those answers. The good news is that if you’re competent and useful, those engineers will already value you, so you don’t have to do anything special: just be good at your job. Is playing politics all about sucking up to senior managers? Basically, yeah. A less cynical way to describe it would be “aligning with the values of the company”. If you think your company is doing good things, you should want to do that anyway! In any case, what that comes down to is figuring out what the people in charge want, giving it to them, and making sure they see you doing it. However, there’s still some scope to get what you want out of the deal. I said earlier that software engineers do not wield organizational power. However, that doesn’t mean you’re powerless. Technical ability is a source of real power, if a delicate and unreliable one. The movers and shakers in tech companies are utterly dependent on technical people to implement their vision and to give them clear answers about the system. There are many subtle ways you can leverage this. One I wrote about in How I influence tech company politics as a staff software engineer is to wait until important people at the company want to do something (say, improve reliability), then offer them a technical plan that does it your way. Another one is to become so useful that you’re actively in demand to lead projects, and then run the project how you want. You probably won’t be able to change the company’s grand strategy. But how that strategy is implemented has a lot of specific technical detail, and you can put yourself in a position to decide on those details. Playing politics isn’t about plotting and scheming, and it isn’t just about being a friendly, likeable person (although that helps). It’s about figuring out how your company actually operates: who makes the decisions, who gets consulted, what behavior gets rewarded, and so on. The most basic way to do that is to figure out who is powerful, get out of their way, and (if you can) help them get what they want . Obviously the exact titles depend on your company. One person I’m deliberately leaving out is your own manager. In general don’t think your relationship with your own manager counts as “playing politics”: that’s just you getting along with another human being. An exception to that is if you report directly to a powerful director or VP. Ironically, this manager struggled to take his own advice. Note that you actually have to be able to answer their question accurately in order to do this. If you’re not competent enough to be useful to powerful people, you will struggle to befriend them. For instance, maybe the CEO is convinced that the project was in bad shape because of something he heard, and the manager in question knows it’s easier to sell “yes, but we turned it around” than “no, you misunderstood, everything was always fine”. If you complicate that process, you risk the CEO thinking that the project is still bad and cancelling it. Be aware of who’s powerful and who’s not At all costs, avoid making powerful enemies Help powerful people as best you can Make sure they know you’re helping them (without annoying them) Make sure you do it in private When they overrule you, stop arguing immediately We’ve had a lot of turnover and reorgs lately, but we’re all starting to pull together as a team now Isn’t it great how focused we all are on reliability work after last month’s incident? The conference this week is the most important thing, so we’re all being very careful not to break anything We’re an AI-forward team that’s looking for the best ways we can leverage LLMs into our team processes Although this project had a rocky start, we’re now all aligned on the way forward Obviously the exact titles depend on your company. One person I’m deliberately leaving out is your own manager. In general don’t think your relationship with your own manager counts as “playing politics”: that’s just you getting along with another human being. An exception to that is if you report directly to a powerful director or VP. ↩ Ironically, this manager struggled to take his own advice. ↩ Note that you actually have to be able to answer their question accurately in order to do this. If you’re not competent enough to be useful to powerful people, you will struggle to befriend them. ↩ For instance, maybe the CEO is convinced that the project was in bad shape because of something he heard, and the manager in question knows it’s easier to sell “yes, but we turned it around” than “no, you misunderstood, everything was always fine”. If you complicate that process, you risk the CEO thinking that the project is still bad and cancelling it. ↩

0 views
Sean Goedecke 1 months ago

In defense of not understanding your codebase

As a software engineer, how well do you have to understand your own codebase? My guess is that people who work on small codebases with low-turnover teams (say, Redis or games like The Witness ) would say “obviously you have to understand it completely, otherwise you can’t do good work”. I’d also guess that people who work on large codebases with high-turnover teams (say, the Google web search backend or GitHub) would say “obviously you can’t understand it completely, you just have to do the best you can in your local area”. These are two largely different ways of programming with different methods, practices and cultures 1 . However, the first group is over-represented in online discussion about software engineering 2 . I want to defend the second group against the first. In many software engineering environments, there’s nothing wrong with being in a state of partial understanding. In fact, in large systems a partial understanding is the best you can do. The best articulation of the “you have to understand your codebase” side is Peter Naur’s famous paper Programming as Theory Building . I like this paper, but I think it goes too far in that direction. Naur’s core point is that when programmers work on a program, the code is really just a by-product, and the main product they’re working on is their “theory of the program”. That’s made up of their intuitive sense of what’s happening and why, which can only be partially captured by code or documentation. If they lost the code, they could rewrite the program easily. If they lost their understanding (say, if the team experienced 100% turnover), they would struggle to make sense of the code. So far, so good, but Naur goes further than this. He says that the theory should not be reconstructed from the code. According to Naur, you’re better off scrapping the program entirely and having a new team rebuild it from scratch , building up a new theory in the process 3 : reestablishing the theory of a program merely from the documentation, is strictly impossible … [therefore] the existing program text should be discarded and the new-formed programmer team should be given the opportunity to solve the given problem afresh Anyone who’s been an effective software engineer at a large company knows that Naur is dead wrong about this. There are at least two reasons. First, you simply can’t rebuild large software systems from scratch . Sufficiently large systems (if they have users) contain thousands of weird cases and quirks that cannot be reimplemented. Even a team that’s intimately familiar with the system couldn’t do it: there’s just too much stuff to juggle. Successful rewrites always start by carving out the existing codebase into small isolated chunks, then rewriting one chunk at a time. In other words, rewriting a software system involves making a bunch of changes to the old system. If you can’t change the old system, you certainly can’t replace it with a new one. Second, abandoned systems are revived all the time . In a tech company with hundreds of millions of lines of code and thousands of engineers, it’s not uncommon for a codebase to have nobody left who’s familiar with it 4 . All it takes is a few people to quit at the wrong time, or for a codebase to be unmaintained for a year. Not only have I seen other teams do this, I have personally taken ownership of abandoned codebases, figured them out, and gotten to a point where I could effectively work with them. It takes time, but building a new theory of the codebase is possible. You start by understanding one flow end-to-end, then slowly branch out from there, making careful changes as you go. In sufficiently large codebases, everyone operates with an incorrect theory of the program . The defining feature of modern software systems is that they’re just way too big for anyone (or even a whole team) to keep in their head: nobody understands it all . To be effective, you have to figure out a way to work with a merely partially-correct theory. This is why I keep going on about taking a position and confidence . If you’re not sure about something, you can’t just sit back and wait for someone with a perfect understanding to come and give you the answer. If you’re a competent engineer, that person is you . You have to grit your teeth, make your most educated guess, and then deal with the consequences. To be generous to Naur, it’s possible that in 1985 the average size of a program was several orders of magnitude smaller than today, and that when Naur writes about “large programs” he’s not talking about tens of millions of lines of code. Naur’s first example of a large program is a 200,000 line industrial monitoring program, and his second example is a compiler. In 1987, the first version of the compiler GCC was about a hundred thousand lines of code; in 2015 GCC was over fourteen million lines. I can believe that rewriting one or two hundred thousand lines of code is relatively straightforward, particularly if you get to reuse existing tests. Not so for one or two million. LLMs are often cited as a tool that’s bad because it impedes the ordinary process of theory-building. I think this is overly simplistic. Like many software tools, LLMs are a double-edged sword: they make it harder to construct a detailed mental theory of the software, but they allow you to build a partial theory quickly and they can help you leverage that partial theory more effectively. This is a complex tradeoff that I’m still thinking about. Setting LLMs aside, I’m confident that it’s silly to say that anything that interferes with your theory of the software must be bad. Here is a partial list of other things that make it harder to maintain a theory: Like most things in software, “maintaining a theory of the codebase” is one value among many. Sometimes it’s the most important value and you sacrifice other values for it; other times you trade it off for speed, or legal compliance, or for political reasons 5 . Almost all engineers — particularly “pure” engineers — prefer to maintain an accurate mental model of their software. It’s more fun, less stressful, and feels more like “real engineering”. That’s why many engineers take up open-source projects in their spare time in order to work on small codebases by themselves: in order to do engineering work where they can maintain an accurate Naur theory of the codebase. I don’t think there’s anything wrong with that. However, at work you are paid to do a job . In other words, they pay you money to adopt their set of engineering values. It’s hopefully well-understood that however much you might personally care about performance, sometimes you have to write slow code at your job (for instance, to get a project done on time, or to accommodate some awkward requirement). Maintaining a theory of the codebase is the same kind of thing. I wrote about this at length in Pure and impure software engineering . I think many of the repeated arguments we have in the software industry are caused by the pure total-understanding culture coming up against the impure partial-understanding culture. Open-source engineers are more excited to blog about their work, the raw engineering content is typically more impressive (because coordination problems dominate big proprietary systems), open-source projects can be legally written about while proprietary systems can’t, and even if you could do it legally, writing about large codebases is impossible because it requires too much specific context . I re-read the relevant chapters of Ryle’s The Concept of Mind (which Naur cites throughout) and I think Ryle is more generous about theory-building. For Ryle, theory-building or know-how automatically happens as you do things. It’s fully consistent with Ryle to think you can pick up an existing codebase just from the code, purely by puzzling it out. Naur says: “Lest this consequence may seem unreasonable, it may be noted that the need for revival of an entirely dead program probably will rarely arise, since it is hardly conceivable that the revival would be assigned to new programmers without at least some knowledge of the theory had by the original team.”. If only! Some engineers might say that maintaining a theory is the core value, because without it you can’t fulfill any of the others. I disagree. You could say the same thing about readability, or maintainability, or correctness, or a bunch of other engineering values. We trade off “core” values like this all the time. Other people being allowed to write code in your codebase Having to implement legally-required features like accessibility and data protection Allowing your colleagues to quit their jobs or move between teams Having to upgrade software versions for security patches Bringing in libraries or other dependencies I wrote about this at length in Pure and impure software engineering . I think many of the repeated arguments we have in the software industry are caused by the pure total-understanding culture coming up against the impure partial-understanding culture. ↩ Open-source engineers are more excited to blog about their work, the raw engineering content is typically more impressive (because coordination problems dominate big proprietary systems), open-source projects can be legally written about while proprietary systems can’t, and even if you could do it legally, writing about large codebases is impossible because it requires too much specific context . ↩ I re-read the relevant chapters of Ryle’s The Concept of Mind (which Naur cites throughout) and I think Ryle is more generous about theory-building. For Ryle, theory-building or know-how automatically happens as you do things. It’s fully consistent with Ryle to think you can pick up an existing codebase just from the code, purely by puzzling it out. ↩ Naur says: “Lest this consequence may seem unreasonable, it may be noted that the need for revival of an entirely dead program probably will rarely arise, since it is hardly conceivable that the revival would be assigned to new programmers without at least some knowledge of the theory had by the original team.”. If only! ↩ Some engineers might say that maintaining a theory is the core value, because without it you can’t fulfill any of the others. I disagree. You could say the same thing about readability, or maintainability, or correctness, or a bunch of other engineering values. We trade off “core” values like this all the time. ↩

0 views
Sean Goedecke 1 months ago

Blog about things you don't understand yet

Every post I publish represents at least two things I’ve learned: the thing that prompted me to write the post, and the thing I learned in the course of writing it. If I don’t learn anything new while I’m writing, it’s not interesting enough to publish. Typically I learn way more than two things. For instance, in my o3 geoguessr post, I started out with the idea that most AI prompts probably don’t work, and I ended up learning that newer OpenAI models have lost o3’s ability to geolocate. That’s interesting! In my most recent post on C2PA , I started out with the idea that C2PA requires near-universal adoption, but I learned a ton of things about PKI, managing private keys on local devices, how C2PA actually works, and so on. In my post on the Luddites , I started out with the idea that the Luddite movement was fundamentally decentralized, but ended up fascinated by Luddite culture (which was far more elitist, misogynist, and violent than the pop-Luddism books describe). I could do this for every single post on the blog. I think the core reason this works is that every single one of my blog posts argues a point . I never publish a post that just gives some scattered thoughts on a topic, or a post that only says “yes, I agree with this other article”. If I write a draft that nobody sensible could disagree with, I scrap the draft. Making sure that everything I write is at least minimally controversial is a forcing function: it forces me to think about what the most interesting part of my position is, and it forces me to do enough research to defend it against the obvious criticisms. This is contrary to a lot of advice I read about blogging, which encourages the aspiring blogger to treat their posts as a form of unstructured self-expression. If unstructured self-expression is what you want to do, that’s cool. The point of having a blog is that you get to write what you want. However, this advice isn’t as helpful as it sounds. Before I was in tech, I was a philosophy grad student. But before that , I was a poet. One thing you learn when you try to write poetry is that it is way easier to write to a restrictive structure than it is to simply “write what you feel”. This should be obvious when you actually think about it. The task of a poet is to repeatedly choose the next word. Writing to a structure (typically rhyme or meter) narrows that choice to a small set of words, instead of the entire English language. It’s the same with blogging. Forcing yourself to write about specific, potentially-controversial points makes consistently writing easier, not harder. Writing is the best way to think clearly about a topic. It’s easy to believe you understand something when you’re just turning it over in your head. When you have to condense that down into words, you find out exactly how much you do or don’t understand. I am constantly having moments where I type something, stop myself, and think “wait, that can’t actually be right”, or “is that really true?” By the time I write my way to the end of the post, I’m usually thinking so much more clearly about the topic that my conclusion paragraph is way better than my introduction. In fact, I’ve picked up the habit of going back and immediately rewriting the first paragraph as part of my first-draft process, because I know I’m going to end up doing it anyway. I also change my mind a lot while I write. Here are a bunch of examples of posts where I began writing them with the opposite opinion to the one that eventually made it into the post. I think this is a good sign, and I hope I never stop doing it. You should be researching and thinking about every post you write, and that means you should frequently learn new things that change your mind. Because of all this, I deliberately choose to write blog posts about things I don’t yet quite understand but would like to, like LLM steering, Stripe’s Tempo blockchain, C2PA and watermarking , space cooling , interaction models , LLM inference internals , and so on. This is great for me, because I learn a lot. Is it great for my readers? I sometimes worry that I should only be writing about areas I already know very well, like tech company dynamics or working in large codebases , rather than presenting myself as an authority on fields I’m actually still learning. Should I let historians of the Luddites write about Luddism, Web3 engineers write about blockchains, and so on? I think this is acceptable for three reasons. First, it’s sometimes easier for a beginner to write an introduction to a field than for an expert. Experts routinely overestimate the knowledge of the general public, and have often internalized the reasons why their field is important so deeply that they struggle to express them. I think my explainer posts are valuable because I always spend the first chunk of the post talking about what the original problem is before I get into the technical solution. Second, sometimes the public consensus on a topic is just plain wrong, to the point where even a little bit of research is enough to demonstrate why. Many of my posts I’m proudest of have been along these lines: arguing that the “500ml per prompt” water usage figure for LLMs was ludicrous , or that the popular Apple “Illusion of Thinking” paper was tracking persistence, not reasoning , that GPUs live longer than three years and the AI companies have large profit margins on inference, and so on. Third, I try to make it clear on my blog who I am and what my credentials actually are. Even if it’s not explicitly described in the post, I have my real name and resume available on my /about page, so I don’t think a careful reader could be easily fooled into thinking I’m an expert on 19th-century England or space physics or LLM economics or anything like that. Even if nobody reads what you write, writing is still a good discipline for getting your thoughts in order. But another big reason why writing is a great learning tool is that you can get feedback . I think it’s obvious why this is useful, but I do want to make two points about feedback. First, if you do make your posts public, you need to have a pretty thick skin. People on the internet often fall over themselves to come up with the most cutting criticism or the harshest dunk. This goes double if you take my previous advice and try to write posts that make a clear, controversial point about a subject you’re learning. If you’re the kind of person whose whole day is ruined when a stranger is cruel to them, you might want to keep your blogging private or only share it among friends. Second, even if your blogging is private, you can get feedback from LLMs . Like humans, LLMs will often give junk feedback. In my experience, OpenAI models will always tell me to moderate my claims or add caveats and hedges until I’m not saying anything at all. Sometimes their criticism will be straight-up wrong. But — particularly about technical topics — LLMs are great at pointing out areas you’ve genuinely misunderstood, and they’re far kinder than the average Lobsters or Hacker News commenter. I’m pleased and grateful that people enjoy reading my posts, but even when nobody did, I still got a lot of value out of blogging. I write as a method of thinking more clearly, as an excuse to do research on topics I want to learn about, and as a way of getting feedback. If you’d like to try it yourself, I suggest watching for these two things. First, you should be changing your mind a lot as you write. If not, you probably aren’t doing enough research. Second, your first draft’s conclusion should be much tighter and more expressive than its introduction. If not, you probably haven’t learned anything from the writing process, which means the draft can be scrapped. I strongly recommend this practice to anyone with an interest in writing. You will see the benefits even if you don’t publish any of your writing on the internet, particularly now that you can get good technical feedback by pasting your post into a LLM 1 . For what it’s worth, I’ve fiddled with careful “review prompts” and it’s basically as good to just write “review, please:” and paste your article. For what it’s worth, I’ve fiddled with careful “review prompts” and it’s basically as good to just write “review, please:” and paste your article. ↩

0 views
Sean Goedecke 1 months ago

C2PA only works if everything is signed

The European Union AI Act is Europe’s attempt to comprehensively regulate AI usage. A big part of that is the requirement that AI-generated content be identifiable: either tagged with a watermark or with what the Act calls “digitally signed metadata” 1 . Since all this becomes enforceable in a month, it’s worth figuring out if it makes any sense. I recently discussed AI watermarking at length in Text AI watermarks will always be trivial to remove . What about digitally signed metadata? The most well-known implementation of digitally signed metadata is C2PA Content Credentials, which incorrectly 2 claims to be the technology that the AI Act gives as an example of how to do signed metadata properly. The idea here is that every single image file should contain unspoofable authorship metadata . Here’s my position on it: Lots to unpack. Let’s start by considering images, since that’s the easiest case. When an AI tool generates an image, that tool should include a “made by ChatGPT” disclaimer in that image’s metadata. Likewise, when a camera takes a photo, that camera should include a “taken by a camera” disclaimer. C2PA uses two strategies to protect this metadata: Each physical camera (or phone) has its own private key, for obvious reasons 3 . How do we know that those millions of private keys are trusted? Via PKI , like HTTPS: each camera’s private “certificate” (which contains its public key) is signed by the manufacturer’s well-known private key, so the chain of authenticity can be verified as long as you have (say) Apple’s root public key 4 . What happens if you then edit your photo in Photoshop? Photoshop will leave the camera’s metadata untouched, but will layer a “also, Photoshop was used” piece of metadata over the top, signed with Adobe’s private key (well, with the private key associated with your official copy of Photoshop, which is signed by Adobe’s official private key). Likewise, if you ask ChatGPT to generate an image for you, ChatGPT will sign its “made by ChatGPT” metadata with OpenAI’s private key. In theory, every single image could contain unforgeable C2PA metadata, allowing software like Twitter to trivially distinguish real photos from fake ones. Right now, C2PA does not have anything like the adoption it’d need to work. It’s hard to find hard data on how many images in the wild use C2PA, but FotoForensics reports around a dozen per week (so around 600 out of the 900,000 images processed each year). This is even worse than it sounds, because basically all of the signed images are AI-generated. The adoption rate of C2PA for human-generated images is much, much lower: so far, Google’s Pixel 10 is the only phone camera to sign photos by default. The iPhone doesn’t sign photos. If almost all AI images are C2PA-signed, but almost no human-generated images are, consumers have no reliable way of identifying AI content, because anyone who wants to pretend their AI content is human can simply remove the signature. For C2PA to succeed, it needs to be on every camera and every phone, so that a photo with no signature is rare and suspicious. Is that realistic? Actually, I think it is. The appetite (at least in the EU) to regulate AI will increase over time, and while the current EU AI Act only mandates that AI-images are tagged (which by itself is useless), it’s plausible that some future regulation will enforce tagging of all images. Another adoption problem that must be solved for C2PA to work is preservation . Right now, if you download a C2PA-tagged image, send it as a Facebook message, then re-download it, the C2PA manifest is stripped out. Most images we see on the internet have passed through some social media asset server at least once. All of these social media companies would need to update how they re-encode image content in order to preserve the C2PA data 5 . This would almost certainly require more regulation: C2PA adds tens or hundreds of kilobytes to each file, which at social media scale is big money 6 . Could a clever attacker forge a C2PA signature? Kind of. Neal Krawetz, who seems to have led the anti-C2PA charge, points out that with a camera development kit it’s straightforward to trick a digital camera into thinking that it’s taking an image when in fact it’s being fed one. This is very much not my area, so please write in if you know more about camera hardware and you think I got this wrong. I suppose you could also take a photo of an AI image on a screen, though I imagine you’d have to be careful to make it look real. If you exclude physical attacks on a digital camera, I think C2PA is more robust. You can sign a photo with a self-signed certificate, but the C2PA spec and docs say that validators must check that your certificate bubbles up to the official C2PA trust list . This list currently contains only 26 certificates, and there’s a whole process for being added to it. That’ll slow down adoption, but at least it makes it hard to forge 7 . We’ve been talking exclusively about images, but it’s more or less the same story for any type of content. If the file doesn’t support JUMBF metadata (say, an Excel file or a PDF), then the C2PA metadata has to live in a “sidecar”: a separate file, probably on some Microsoft or Adobe content server, which contains the signed checksum and the data about who created the file. However, the distinction between “real” and AI-generated content is fuzzier when you’re not talking about images. Here’s a trivial example: if I ask ChatGPT to create an Excel spreadsheet for me, the file will be tagged as AI-generated, but I can simply copy/paste the content into a new Excel doc and save it, which will tag it as human-generated 8 . There’s no software tool that can identify when I’m retyping some AI-generated text (except for perhaps text fingerprinting , which has its own raft of issues). There are also interesting questions around key management. ChatGPT and other AI tools have an easy problem — their users are all online, and so the files can be signed server-side — but how do you sign files created via Photoshop/Excel/Word? If the user doesn’t have internet, do you use some kind of local key? If so, how do you prevent that key being extracted and used to sign AI-generated content? Finally, is it a civil liberties problem to automatically fingerprint every photo? Does it make it impossible to be a whistleblower if every photograph can be traced back to your camera? I think this is a complicated question, but in short: I’d expect whistleblowers to already strip EXIF metadata from their images, C2PA metadata is similarly trivial to strip out, and overall I think image attribution is positive for whistleblowers because it heads off “this was AI-generated” responses. C2PA is probably here to stay. But it isn’t useful now, and won’t be useful until two huge programs of technical work are completed: This will be a long organizational process, since each manufacturer must go through the approvals process (or decide to start their own competing system), evaluate the legal ramifications of storing attribution data in images, and so on. It will be a long technical process, because C2PA metadata is a substantial fraction of image sizes: storing it will add many petabytes of content. Of course, just because C2PA isn’t useful doesn’t mean we’re not all going to do it. Lots of companies are under pressure to signal that they care about AI safety and to head off regulatory attack. “We’re cryptographically signing AI-generated content” is a compelling “we’re doing something ” pitch, particularly for people who aren’t technically savvy enough to understand the limitations. In the near term, I expect large AI-involved companies to invest a substantial amount of engineering effort in C2PA-related activity. In the long run, once everyone gets on board, I think C2PA could end up working well. It’s awkward in some ways, but “attest content via a PKI certificate chain” is a good idea. Is it possible to defeat? Yes, of course. By design, private keys will be in the user’s hands — in their cameras, in their local versions of Photoshop or Microsoft Word, in their phones — so sufficiently technical users will be able to crack them out or use them to sign whatever content they want. I still think C2PA will end up stemming the tide of AI content, because most users are not going to be sophisticated enough to perform attacks like this. However, we should still retain some skepticism of unlikely-looking content, even if it has “created by a human” in its C2PA metadata. See sub-measure 1.1.1 of the Act’s associated Code of Practice . While an early draft of the Code of Practice made an offhand mention of Content Credentials (in the caption of a picture), that was stripped out. The contents of the Act and the final Code of Practice don’t contain “C2PA” or “Content Credentials” (you can search for yourself here ). Otherwise if you cracked the key out of one Sony camera, you could spoof content from any Sony camera. In practice there are usually more “links in the chain”: a device will be signed by some intermediate certificate, which in turn will be signed by another intermediate certificate, which will be signed by the root certificate. That’s because the root key is so valuable. If an intermediate private key leaks, it can be revoked and replaced (via the root key), but if the root key leaks, it would take years to rebuild the network of trust. So almost all signing is done by intermediates, and the root key stays on a USB drive locked in a safe somewhere. Not to mention that the whole point of C2PA is that these social media companies will be displaying a “human or AI” sticker in their UI, which will require retaining the metadata. C2PA allows for storing the manifest content as a separate file, and just including a manifest url in the image metadata itself, but that doesn’t solve the cloud provider problem: they still have to store all the files on-disk somewhere. I think this defuses Neal Krawetz’s “worst-case scenario” . I downloaded his forged image, and (as expected) it gets flagged as “signed, but we don’t trust the root”. I think Krawetz was right at the time, though, since the official “trust list” was only launched in mid-2025. You could do the same thing with images by copying into Photoshop or Paint, but while that’d obscure the AI source, it would still be clear that the photo wasn’t taken by a camera. C2PA broadly makes sense and is a good idea It is pointless to use C2PA for AI-generated images only It will take many years for C2PA to be adopted across all images Because C2PA makes such great safety theater, we’re going to see a lot of hue and cry about it long before it becomes useful The metadata must be signed by some trusted private key The metadata contains a hash of the file’s contents, so you can’t copy an existing signature onto a new file Every camera manufacturer (including phones) must C2PA-sign all images by default Every social media company must retain the C2PA metadata on uploaded images See sub-measure 1.1.1 of the Act’s associated Code of Practice . ↩ While an early draft of the Code of Practice made an offhand mention of Content Credentials (in the caption of a picture), that was stripped out. The contents of the Act and the final Code of Practice don’t contain “C2PA” or “Content Credentials” (you can search for yourself here ). ↩ Otherwise if you cracked the key out of one Sony camera, you could spoof content from any Sony camera. ↩ In practice there are usually more “links in the chain”: a device will be signed by some intermediate certificate, which in turn will be signed by another intermediate certificate, which will be signed by the root certificate. That’s because the root key is so valuable. If an intermediate private key leaks, it can be revoked and replaced (via the root key), but if the root key leaks, it would take years to rebuild the network of trust. So almost all signing is done by intermediates, and the root key stays on a USB drive locked in a safe somewhere. ↩ Not to mention that the whole point of C2PA is that these social media companies will be displaying a “human or AI” sticker in their UI, which will require retaining the metadata. ↩ C2PA allows for storing the manifest content as a separate file, and just including a manifest url in the image metadata itself, but that doesn’t solve the cloud provider problem: they still have to store all the files on-disk somewhere. ↩ I think this defuses Neal Krawetz’s “worst-case scenario” . I downloaded his forged image, and (as expected) it gets flagged as “signed, but we don’t trust the root”. I think Krawetz was right at the time, though, since the official “trust list” was only launched in mid-2025. ↩ You could do the same thing with images by copying into Photoshop or Paint, but while that’d obscure the AI source, it would still be clear that the photo wasn’t taken by a camera. ↩

0 views
Sean Goedecke 1 months ago

Text AI watermarks will always be trivial to remove

The European Union AI Act will begin to be enforceable in August 2026, one month from now 1 . One of the biggest new requirements is Article 50 , which requires all AI outputs to be “detectable as artificially generated”. In other words, if LLM providers want to do business in the EU, they will have to apply a watermark to their outputs 2 : some hidden signature that can be used to identify AI content. LLM text watermarking is a fascinating problem. Like the best engineering problems, it is theoretically hard to solve perfectly, but has multiple partial solutions: for instance, Google’s SynthID , and (as I’ll argue) some quiet Unicode trickery from OpenAI and Anthropic. It will be interesting to see how the AI labs navigate these tradeoffs before the end of the year. I wrote about AI watermarking at the end of last year in AI detection tools cannot prove that text is AI-generated . It’s easy to watermark an image, because digital images contain lots of noise that the human eye can’t really see. For instance, you could apply a watermark like “these twenty pixels in these exact spots will always share a color”. Text is much, much harder. Unlike images, text is a very compressed medium: you cannot make any change to a sentence that a human wouldn’t notice (with one exception, which we’ll get to later). So how are you supposed to watermark it? It’s basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated. Any changes you make to apply the watermark will compromise the quality of the output. For instance, “every fifth letter is an ‘e’” would be a good watermark, but applied naively would make the AI output full of typos. Could you just let the model figure out how to fit the watermark? Strong AI models are smart enough to juggle this kind of constraint 3 , but it’d still consume reasoning time that would be better spent on the user’s problem, and make the model sound much less capable than it is 4 . Do you really need a watermark? If you’re Anthropic, and you’re required to be able to verify whether your models produced a particular block of text, can’t you simply run the text through each model, measuring as you go how closely the model’s predicted tokens match each token from the text? Not really. The space of “all possible Claude Sonnet answers to a question” is way larger than the space of “all possible watermarked answers to a question”. In other words, you’d get too many false positives for human text that reads like it was AI-written. It’s way more likely for a human to accidentally write like Claude than it is for a human to accidentally reproduce a watermark. It would also be prohibitively expensive to run every Anthropic model against a piece of text in order to watermark it. The EU AI Act will eventually require labs like Anthropic to offer free watermarking services to every EU citizen (see Commitment 2). You couldn’t do that with the “run the model” approach. As far as I know, the only AI provider to say they watermark text output is Google, who use a tool called SynthID . Here’s how it works. When a LLM generates text, it’s generating a series of tokens (words or chunks of words). At each step, the model itself doesn’t output a single token, but instead outputs a full list of all (say) 100,000 tokens in its vocabulary, each annotated with the probability that that token will be the next one. Tools like ChatGPT or Claude Code will pick semi-randomly from the most likely options in order to get their outputs. This semi-random sampling process can be influenced in a detectable way. For instance, we could choose a sampling strategy like “we pick the second most likely token, then the first, then the second, then the first, and so on”. That would still produce high-quality output, but you’d be able to re-run the model against the generated text to verify that the pattern holds. However, that’d make verification really expensive, and any slight tweaks to the output would break the pattern and thus break the fingerprint. Is there a better way? Yes. SynthID is a process for assigning each token a “score” based on its previous tokens (for instance, sum the token’s ID with the IDs of its previous three tokens then take mod 5) 5 . To apply the watermark, the model adopts a sampling strategy like “out of the top five most likely tokens, pick the one with the top SynthID score” 6 . The watermark can then be detected by calculating the aggregate SynthID score of a block of text. If it’s suspiciously high, it’s very likely to have been AI-generated. This is basically a version of the common advice that you can identify LLMs by use of the em-dash , except that instead of a list of keywords, it relies on subtle mathematical relationships between words that humans can’t identify. Because the process for assigning the score is trivial, it’s very cheap to run watermark detection. Google have a complicated mathematical rationale for why SynthID doesn’t make the model dumber: supposedly the SynthID scoring is random enough to act like a normal pseudo-random token sampler, just one that leaves a detectable fingerprint on the outputs. But of course this is suspicious. For instance, it’s common to do inference setting temperature to zero, which always picks the model’s most likely next token. In that case, you can’t leave a fingerprint at all (or you have to ignore the user’s preference and pick the second or third choice anyway). If you can’t alter the model outputs, can you still fingerprint the content? Well, kind of. I’m pretty sure OpenAI and Anthropic are sometimes applying fancy Unicode tricks. For instance, you might go through and replace your normal ” ” spaces (unicode ) with a three-per-em ” ” space (unicode ), or a CJK ideographic ” ” space (unicode ). These are called “homoglyphs”, and you can find more of them here . Of course, lots of human-generated text uses homoglyphs. But it’s trivial to encode a pattern of homoglyphs (say, “every third space becomes a three-per-em”) that is much less likely to occur in the wild. Like the SynthID watermark, a homoglyph-based watermark can be detected very cheaply. A homoglyph-based watermark is cheaper to apply than SynthID: you could even do it entirely on the client. I don’t think this is a conspiracy theory. Claude Code was definitely doing this to tag suspicious requests from Chinese users (exploiting homoglyphs for the ’ character in “Today’s date”, though they’ve since walked that back). In the last few years, I’ve noticed that when I copy blocks of text from ChatGPT and paste them into VSCode, sometimes VSCode marks some or all of the spaces as unusual Unicode characters 7 . Are OpenAI and Anthropic using homoglyphs as an AI-generated watermark? I’m not sure. But they’re definitely using homoglyphs. The AI Act (specifically, its associated Code of Practice ) requires watermarking to be “embedded within the content in a manner that is difficult for it to be separated from the content”. However, text watermarks can be trivially removed. To remove unicode homoglyph watermarking, you simply have to replace all the homoglyphs with their “real” character equivalents. If you have access to even a relatively weak un-watermarked LLM 8 , you can strip out SynthID watermarking by asking that LLM to paraphrase the text content. Because the watermark is inherent to subtle vocabulary choices, re-wording the content will remove the watermark. You could even do it by hand, although at that point it’s not really AI-generated content anymore. Since there will be some kind of free public watermark testing tool, you can just keep tweaking until it comes back negative. Moreover, the AI Act requires watermarking techniques to be “interoperable… as far as this is technically feasible”. That means AI providers would have to publish their watermarking process, and potentially even attempt to standardize on applying the same kind of watermarks. I just don’t see how this is compatible with the kind of security-by-obscurity that LLM text watermarking depends on. Unlike image and video watermarks, text watermarks will always be trivial to remove. The AI Act and Code of Practice talk a lot about “digitally signed metadata”. The idea here is that you can include an AI disclosure in the file’s metadata itself, ideally in a way that cannot be tampered with (for instance, by signing a hash of the file’s contents). This signed-metadata process is basically C2PA Content Credentials . While you can remove C2PA metadata, you (theoretically) can’t fake it, so a file with “created by a human” metadata can be trusted, and files with no metadata at all can be held in suspicion. This post is already too long to get into what I think about C2PA, but I do want to say that C2PA is not a substitute for text watermarking . It only really applies to files . In the words of the Code of Practice, that’s “a data format that supports attaching metadata (e.g., an audio, image, video, or containerised text)“. The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so can’t be signed. What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around. I think it’s a fascinating question whether Claude Code has to C2PA-sign any HTML files or PDFs it generates for you. That seems kind of tricky to get right. But in any case, the AI Act also mandates some kind of actual watermarking as well. So what’s going to happen this year? If I had to guess, I’d say that each AI provider (not just labs like OpenAI or Anthropic, but third-party providers like Fireworks or Groq) will stick a SynthID token sampler in front of their inference stacks. This might be limited to users in the EU, but it might not be, since SynthID is at least as good as a normal top-k token sampling approach. AI providers will then offer a “check for watermark” page that re-tokenizes user-provided text, runs the scoring, and checks whether it’s above a certain threshold. Depending on how seriously the interoperability clause is taken, providers might even standardize on the same SynthID setup, in which case there could be a single EU-hosted “watermark this text” page. I don’t think unicode-based watermarking is going to be considered compliant with the AI Act, but some providers which don’t want to set up SynthID might try it. Either way, technical users will be able to strip out the watermark at will, and there will be a plethora of tools that non-technical users will use for this purpose. Well, for new systems; existing ones get until December. I don’t think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that they’re looking for watermarks. Even with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an “e”, but GPT-5.5-Pro produced this puzzling koan: “These hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.” I leave the analogy with AI safety guardrails as an exercise for the reader. That’s a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens. The tokens are scored in a multi-round knockout against each other, but I think that’s more of an implementation detail and not required to get the core intuition behind why SynthID works. When this became public knowledge , OpenAI claimed it was just a model quirk, which is certainly possible. All AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text. Well, for new systems; existing ones get until December. ↩ I don’t think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that they’re looking for watermarks. ↩ Even with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an “e”, but GPT-5.5-Pro produced this puzzling koan: “These hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.” ↩ I leave the analogy with AI safety guardrails as an exercise for the reader. ↩ That’s a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens. ↩ The tokens are scored in a multi-round knockout against each other, but I think that’s more of an implementation detail and not required to get the core intuition behind why SynthID works. ↩ When this became public knowledge , OpenAI claimed it was just a model quirk, which is certainly possible. ↩ All AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text. ↩

0 views
Sean Goedecke 1 months ago

AI inference is obviously profitable

Many people claim that AI inference is unprofitable to serve, and thus must be subsidized by an ocean of dumb money from investors who believe that some future AI model will come to dominate the world economy. When that dumb money goes away, so will AI products. According to this view, LLMs are just inherently too expensive (in terms of money, power, and water) to be used in consumer products. In fact, they can only be used today by externalizing the costs: money onto VC funds and now retail ETF investors , power onto electric utility consumers , and water onto the communities where datacenters are built. There are good reasons to dislike AI, but this really isn’t one of them. In fact, AI inference is obviously profitable . Frontier AI providers are reporting 70%-80% gross margins on inference, but maybe we can’t trust them. Let’s do some very rough estimates on the actual cost. A Nvidia A100 consumes 400W of power under full load. In practice, even a carefully-tuned inference server will not be at full load all the time, but it’s at least an upper bound. Suppose you’re running a dense 70B model 1 , which will fit comfortably (unquantized) on four A100s at around 2M tokens per hour. At industrial power prices, that’s about 13c/hr in the USA . Suppose (pessimistically) cooling is the same cost. That’s about 13 cents per million output tokens 2 . Let’s amortize the cost of the GPUs, since that’s going to be the most expensive part. An A100 costs about $20k. If each A100 lasts around five years 3 , you’ll have to make 16k/yr in profit to recoup your capital investment (or $1.80 per hour). At lower utilization, it’ll take longer to recoup, but your GPUs will also last longer. Either way, your overall inference costs are at about one dollar per million tokens. GPT-5.4-mini charges $4.50 per million tokens, and stronger OpenAI or Anthropic models are three to six times as expensive. It’s hard to make a direct comparison because we don’t know the size of OpenAI or Anthropic models, but the claimed 70% or 80% profit margin is extremely plausible. What if you don’t trust my estimates either? Let’s look at the pricing of open-weights Chinese LLMs. DeepSeek have claimed a bit over 80% profit margin on inference for DeepSeek-R1. Since their API pricing for R1 is less than half that of OpenAI or Anthropic 4 , that suggests that my estimates above for inference cost might be too expensive. Cooling at scale is probably cheaper than power, R1 only has half the active parameters of a dense 70B model, modern GPUs are more efficient than the A100, and there are significant economies of scale in inference. Since DeepSeek’s models are available for anyone to download, they can’t get away with extracting a large profit margin. One of the other inference providers would undercut them with the same model. Inference costs for DeepSeek-V4-Pro on the market are around 87 cents per million output tokens, which is probably pretty close to the actual cost of serving the model. All of this doesn’t mean that OpenAI or Anthropic are profitable. Those companies are making huge capital investments that may or may not pan out, and are spending enormous amounts of money on talent and compute to train brand-new models and retain users. They’re doing crazy things like offering per-month subscription models for nearly unlimited inference, which is almost certainly not profitable. If you used an API token instead of your Anthropic subscription in Claude Code, you’d pay ten times the cost. But that doesn’t mean API-based Claude Code couldn’t be a good deal. Some people are already using DeepSeek’s inference API for agentic coding, because once you take away the huge profit margin it’s cheaper than the relative per-month subscription. Why won’t OpenAI or Anthropic lower their prices? Supposedly OpenAI has thought about it , but for an AI lab, inference has to subsidize training costs . A company like OpenAI has to fund the production of new models from the inference margins on existing models (at least partially). That’s why the margins on inference are so high: the AI labs are trying to squeeze out every dollar so they can stay alive in the training arms race. However, inference only has to subsidize training costs for an AI lab . If you’re merely an inference provider, you don’t have to do any training at all. Therefore, even if OpenAI and Anthropic go out of business, whoever snaps up the rights to their frontier models will be able to continue selling Opus and GPT inference at a profit 5 . The AI bubble popping will not mean the end of the inference business, because AI inference is obviously profitable . Expensive frontier models are probably mixture-of-experts, not dense, which is tougher to estimate. However, I think a 70B dense model and a MoE with 70B active params will come out to basically the same numbers at scale (though the MoE will require more GPU memory and thus a greater upfront cost). Are frontier models around 70B params? Nobody outside the AI labs really knows, but my guess is that 70B is probably larger than a Haiku/mini class model. I think it’s reasonable to estimate the cost of output tokens only, since they’re by far the most expensive part of serving inference. Input tokens are cheaper for two reasons: transformers let you prefill them in parallel, and for most real-world use cases they can be aggressively cached in the KV cache. It’s common (and wrong) to estimate GPU lifespan at three years. I wrote a lot about this in AI GPUs probably live longer than three years . Again, this is just an guess, since we don’t know what OpenAI or Anthropic model is equivalent in size to R1. I do wonder if Anthropic would be able to prevent other people from being able to access the model if the company goes out of business. Anthropic is currently in debt to Broadcom, Google, and a bunch of private equity firms. Would they get the Mythos and Opus weights, over Dario’s protestations? Expensive frontier models are probably mixture-of-experts, not dense, which is tougher to estimate. However, I think a 70B dense model and a MoE with 70B active params will come out to basically the same numbers at scale (though the MoE will require more GPU memory and thus a greater upfront cost). Are frontier models around 70B params? Nobody outside the AI labs really knows, but my guess is that 70B is probably larger than a Haiku/mini class model. ↩ I think it’s reasonable to estimate the cost of output tokens only, since they’re by far the most expensive part of serving inference. Input tokens are cheaper for two reasons: transformers let you prefill them in parallel, and for most real-world use cases they can be aggressively cached in the KV cache. ↩ It’s common (and wrong) to estimate GPU lifespan at three years. I wrote a lot about this in AI GPUs probably live longer than three years . ↩ Again, this is just an guess, since we don’t know what OpenAI or Anthropic model is equivalent in size to R1. ↩ I do wonder if Anthropic would be able to prevent other people from being able to access the model if the company goes out of business. Anthropic is currently in debt to Broadcom, Google, and a bunch of private equity firms. Would they get the Mythos and Opus weights, over Dario’s protestations? ↩

0 views
Sean Goedecke 1 months ago

AI GPUs probably live longer than three years

People who think current AI use is unsustainable often rely on the claim that inference GPUs only last “three years at the most” under load 1 . The idea here is that once the AI bubble money drains away, current infrastructure will rapidly become obsolete, and there won’t be enough money floating around to buy a whole slate of brand-new GPUs. Inference costs would thus rapidly become way too expensive for current AI products to make any financial sense. Where does this “three years at the most” claim come from? Is it plausible? The original Tom’s Hardware article quotes this tweet from Tech Fund, an anonymous former PM and tech investor, who quotes an anonymous “GenAI principal architect” at Google as saying “if you have a high utilization rate, then constant high utilization rate for a year or two, I think the lifespan will be three years at most”. This screenshot looks like it was from an interview. What interview? I scrolled back to October 2024 on Tech Fund’s Twitter feed and saw a bunch of similarly-formatted screenshots , some of which were cited as coming from Tegus . Tegus is apparently a company with a business model of reaching out to insiders (in this case, AI company employees) and paying them hundreds of dollars an hour in order to answer specific technical questions. It’s essentially gig work for almost-but-not-quite insider trading: the more informed and confident you sound, the more likely Tegus analysts will pick you for future interviews. I’m sure the source for this tweet is in fact a GenAI principal architect, since Tegus would have presumably asked for some proof of that before they paid them out. But it’s pretty clear that the incentives here are to sound confident and authoritative, even on questions that you’re not sure about. With that in mind, the quote itself also reads a bit suspiciously. I’ve worked with enough principal engineers and architects to take their casual back-of-envelope estimates with a grain of salt. If they knew the actual rate at which GPUs fail and get retired in Google datacenters, wouldn’t they have just said that? We have some anecdotal evidence that points the other way. Google has publicly claimed to have eight year old TPUs (their version of GPUs) running in production at “100% utilization”. Nvidia only made A100 GPUs from 2020-2024 , but in February 2026 the AWS CEO claimed that AWS had never retired an A100 server (and you can still easily rent A100s for AI work) 2 . AI GPU usage isn’t exactly like crypto mining GPU usage, but it certainly seems like years-old ex-crypto GPUs are functional . There’s also this comment from Hacker News I noticed where someone claims that their GPU cluster in academia has lasted six years with less than 20% failure rate. What about hard data? It’s hard to get concrete data on the lifespan of AI GPUs, because modern AI datacenters have only existed for a handful of years. But an interesting case study would be recent supercomputer clusters like Oak Ridge’s Summit , which had over 27 thousand Nvidia V100s running from 2018 to 2024, or its predecessor, the Cray Titan supercomputer that ran from 2012 to 2019. I couldn’t find any evidence that Summit had to buy an additional 27,000 GPUs to replace their old ones, and GPU failures in Titan have been carefully studied : These cages of GPUs are stacked vertically, and cold air is pumped in from the bottom, which explains why cage 0 (at the bottom) has better survival rates than cage 2 (at the top). Let’s consider cage 0, so we’re just looking at the GPU lifespan instead of at the lifespan of improperly-cooled GPUs. At three years, over 95% of GPUs survived 3 . At six years, nodes 2 and 3 (the GPUs closest to the bottom of the cage) were still at above 90% survival rate, and the highest nodes were over 60%. It’s possible that newer Nvidia GPUs are less reliable than older ones (they certainly draw more power), or that AI datacenters are under-cooled, or that something about LLM utilization is more stressful than the workloads that ran on traditional GPU datacenters. But this is at least circumstantial evidence that GPUs can survive under load for far longer than three years. This discussion is complicated by the fact that GPUs may have a short economic lifespan. Supposedly a B100 GPU draws twice as much power as an A100, but can do five times as much work. For some AI providers, that might mean that A100s are only worth running until they can be replaced with B100s (if you’re bottlenecked on electricity, you should spend it all on B100s and throw out your obsolete A100s). This is why the Titan supercomputer was decommissioned in favor of Summit: it could have continued to operate, but it was more profitable to spend the money and maintenance effort on newer hardware. It should be obvious that this doesn’t support the “inference will become more expensive when the bubble pops” argument. So long as A100s are profitable right now , cash-poor AI providers can continue profitably serving inference from them, even if there are more efficient options available for those with the capital to upgrade. On top of that, GPUs only represent one part of AI datacenter infrastructure spending. If your GPUs wear out, you don’t have to go and build an entirely new datacenter. About 30-50% of datacenter spend goes to land, power, cooling, and so on. The remaining 50-70% is the cost of the entire server rack, which includes a bunch of things that aren’t GPUs. Like the idea that AI inference requires using huge amounts of water , the idea that AI GPUs only live a year or two is popular because it’s a useful idea for AI skeptics, not because it’s true. It comes from a pseudonymous tweet quoting an anonymous source who’s being paid hundreds of dollars to sound like a credible expert on AI. Other public communications from AI inference providers cite much higher lifespan numbers, and the statistics from supercomputers (the traditional examples of large GPU clusters) don’t bear out the claim that the maximum lifespan is three years. It might be true that the economic lifespan is three years, in a world where new GPUs come out every eighteen months and GPU providers are flush with cash to upgrade, but that doesn’t tell us much about the economics of inference in an AI winter. If money becomes a lot more scarce, it’s likely that AI datacenters will continue profitably 4 running their B300s (or their H100s or even A100s) for six years or longer. Of course, like previous claims about AI and water usage, “three years at the most” is often cited as “1-2 years, with some lasting up to 3 years under optimal conditions” . Of course, pronouncements from CEOs/CTOs should be taken with a grain of salt as well (for instance, maybe they have a big backlog of unused A100s they keep swapping out), but (a) executives don’t often straight-up lie about concrete technical facts, and (b) they’re going up against an unsourced quote from a tweet, so the bar isn’t that high. What about proactive GPU replacement? In the “Survival Analysis” section, the study attempts to account for this. I haven’t dug into exactly how. Assuming inference is profitable, which I believe (when you’re not attempting to amortize the cost of training). Of course, like previous claims about AI and water usage, “three years at the most” is often cited as “1-2 years, with some lasting up to 3 years under optimal conditions” . ↩ Of course, pronouncements from CEOs/CTOs should be taken with a grain of salt as well (for instance, maybe they have a big backlog of unused A100s they keep swapping out), but (a) executives don’t often straight-up lie about concrete technical facts, and (b) they’re going up against an unsourced quote from a tweet, so the bar isn’t that high. ↩ What about proactive GPU replacement? In the “Survival Analysis” section, the study attempts to account for this. I haven’t dug into exactly how. ↩ Assuming inference is profitable, which I believe (when you’re not attempting to amortize the cost of training). ↩

0 views
Sean Goedecke 2 months ago

Doing nothing at work

Many engineers should be doing less work. I don’t necessarily mean producing less code or fewer changes, but literally working fewer hours in the day. When they do work, they should be working at a slower pace. I like to aim to be running at 80% utilization by default: unless I have a high-pressure project going on, I spend 20% of my workday away from the computer. Why? Performance at tech companies is dominated by outlier events . When I think about the most impactful changes I’ve made, many of them involved a surprisingly trivial amount of work. There are no points for effort in software development. What matters is solving the right problem at the right time. In large engineering organizations, there are usually trivial pieces of engineering work you could do that would make tens or hundreds of millions of dollars for the company. Here are three common examples: First, when the company is trying to sign a big enterprise deal, stepping in with a feature or bugfix can make the deal happen. It doesn’t even have to be a good feature: sometimes just showing that you’re willing and able to make a concrete change will be enough. Second, preventing or mitigating an incident early (even by just knowing the right feature flag to turn off) can save huge amounts of money: both immediate lost revenue during the incident and future lost revenue from customers who would have pulled their business or refused to sign pending contracts. Third, when the company is trying to ship a high-profile feature, success or failure often hinges on trivial but obscure changes (e.g. the ability to rapidly add a new field in user settings, or to update the crufty enterprise-data-export functionality nobody has touched in years). Familiarity with the system can be the difference between one of these changes taking a few hours or a whole week. What do these examples have in common? They’re all time-dependent . You can’t just log on in the morning and decide to unblock a big deal, or mitigate an incident, or speed up a high-profile feature. Is it just a matter of being in the right place at the right time? Not quite. You also have to not already be busy. I wrote about this a couple of years ago in Crushing JIRA tickets is a party trick, not a path to impact . If you’re always 100% utilized on a steady stream of low-priority work (for instance, if you’re just picking up tickets from the backlog, crushing them, then picking up the next one), you’ll miss your chance to do high-impact work in two ways. First, you’ll be too busy to even notice the opportunities. You won’t be chatting with people who are working on other things, or reading team updates, or keeping an eye on ongoing incidents. So you’ll miss out on the best way to get involved in high-impact work, which is to volunteer your expertise. Second, if you perpetually look busy, your manager won’t want to volunteer for you. This is the second-best way to get involved in high-impact work: to have your manager or product manager say “oh, Sean has capacity to help out here, let me tag him in”. Why is this better? Because managers and product managers usually have a much better read on what high-impact work is going on. They’re in meetings that you aren’t in. If you’re supposed to keep your time free for high-impact work, and you’re not supposed to just grind tickets, what should you be doing on a minute-by-minute basis? Should you just be doing nothing? Yep! Doing nothing is good, actually. Software engineering can be a stressful job, but it’s typically not consistently stressful: the stress comes from the occasional incident, or high-pressure urgent piece of work, or (these days) layoff. If you approach the comparatively low-pressure parts of your work with urgent intensity, you’ll already be exhausted and frazzled when you have to handle the high-pressure parts. Even in high-pressure parts of the job, doing nothing can still be good. One thing I recommend for engineers new to on-call is to avoid rushing: take a few breaths before joining the call or before speaking, and in general try to “think in slow motion” . Most incidents resolve on their own. Most frantic “maybe this will help” changes during incidents make things worse, not better. As a general rule, if you can simply avoid panicking, you will be doing better than most engineers at incident response. Nothing is a space things can happen in 1 . If you give your brain a chance to rest, you will find you’re more likely to have new ideas. If someone hands you an important task, you can tackle it with your full attention (instead of juggling it with the three other things you’re working on in the background). When you’re not busy, you have time to just look at things and take in new data. A lot of engineers are uncomfortable seeing a task that needs doing and not doing it. I’m like this as well. I wrote about it in I’m addicted to being useful : it’s a psychological quirk that many software engineers share, because having that quirk (to a point) makes you a good fit for the job. In order to spend time doing nothing, sometimes you need to force yourself to not step in. For instance, I believe that engineers should generally avoid glue work 2 . Most glue work - making sure people talk to each other, updating docs for work you’re not leading, volunteering to address technical debt - reflects the fact that the organization is not explicitly prioritizing this work. If they were, you wouldn’t need to volunteer for it. Either that’s fine, or it’s a big mistake. If it’s fine, then you shouldn’t step up and do it: you’ll be wasting your time and annoying your manager. If it’s a big mistake, you still shouldn’t do it , because you’ll be insulating the company from the consequences of its own mistakes at the cost of your own career and mental well-being. That’s a bad deal for you, and a bad example for your junior colleagues, and sets a bad precedent for someone else to jump into the same position when you inevitably burn out 3 . If the consequences truly are severe, let them happen, so the organization can feel the pain and change its policies. I also believe that being too helpful leaves you vulnerable to predators . Tech companies are full of people who want to extract uncompensated work from software engineers 4 . This is different from work that arrives via normal channels, and for which you’re compensated by promotions, bonuses (and just your normal salary). I’m talking about work that arrives via backchannels, from people who don’t have the ability or willingness to ensure that work is formally recorded under your name. For instance, a product manager from another organization messaging you to say “you’re so good at querying data, would you mind pulling some statistics for me about X?”, or an engineer from another team asking you to “pair” on a piece of work that will ultimately involve you writing all the code and them quietly submitting the change under their own name. Doing some amount of this kind of work is fine. You may as well help people out when you can. But you need to be able to apply backpressure, either by saying no or simply delaying your response by a few hours or days. It’s also a good idea to avoid investing too much in work that is likely going to disappear . For instance, suppose you’re working with a product designer who is figuring out what they want in real time. At 9am they message you saying they want the page header to look one way, then at 10am they have tweaks, and more changes at 11am, and so on. You should not throw yourself into fully rewriting the page every hour. Instead, you should do nothing (say, go for a walk) and rewrite the page once in the afternoon, based on the most recent design. Another common instance of this is “big idea from a manager without the political clout to follow through on it”. Often you can just run out the clock until the project gets inevitably cancelled 5 . A lot of software engineering advice and tooling is designed around the ability to scale up your ability to exert technical effort: to do more things at the same time, to take on projects of larger scope, or to just write more code. But software engineering success is not determined by any of these. It is determined by the ability to do the right things at the right time, which requires that you deliberately hold back some of your effort during ordinary work. In my experience, it’s still possible to be a “high performing engineer” at 80% effort. In fact, it’s easier , because you’ll be less likely to make silly mistakes from stress, and you’ll be in a position to jump on the kind of high-impact tasks that deliver outsized returns. This doesn’t mean you should never grind at 100% effort. I think there are probably two or three times a year where I work as hard as I possibly can: long hours, intense focus, thinking about the problem from when I wake up to when I go to bed. But I reserve this mode of work for when the rewards are really high . For the rest of the year, I take it relatively easy. edit: this post got some comments on Hacker News . Commenters discuss how to not get in trouble with your manager when you’re taking slack time (in my experience, if you’re generally productive it’s fine, but managers vary a lot) and whether engineers really do have control over their workload. One of my big influences is Rich Hickey’s talk Hammock Driven Development . This is kind of like what he’s talking about, except (a) Hickey is more talking about what it takes to design solutions to really hard problems, rather than what it takes to be a strong engineer in an ordinary tech company, and so (b) Hickey recommends using your time-away-from-the-computer to focus on a hard problem, instead of to simply decompress and let solutions congeal in your head. It’s also like Zvi Mowshowitz’s post on “slack” . I wrote about this a lot more in Glue work considered harmful . Why inevitably? Because in my view, burnout is hard work unrewarded , and taking on a personal crusade that your job doesn’t care about is a great way to do a lot of unrewarded work. I wrote about this in Protecting your time from predators in large tech companies . Of course, you have to be careful with this. If you try this strategy and you’re wrong about the level of political support for the project, you will come off like a slacker and then have to deliver in a rush. One of my big influences is Rich Hickey’s talk Hammock Driven Development . This is kind of like what he’s talking about, except (a) Hickey is more talking about what it takes to design solutions to really hard problems, rather than what it takes to be a strong engineer in an ordinary tech company, and so (b) Hickey recommends using your time-away-from-the-computer to focus on a hard problem, instead of to simply decompress and let solutions congeal in your head. It’s also like Zvi Mowshowitz’s post on “slack” . ↩ I wrote about this a lot more in Glue work considered harmful . ↩ Why inevitably? Because in my view, burnout is hard work unrewarded , and taking on a personal crusade that your job doesn’t care about is a great way to do a lot of unrewarded work. ↩ I wrote about this in Protecting your time from predators in large tech companies . ↩ Of course, you have to be careful with this. If you try this strategy and you’re wrong about the level of political support for the project, you will come off like a slacker and then have to deliver in a rush. ↩

0 views
Sean Goedecke 2 months ago

Working with product managers

The relationship engineers have with product management is more dysfunctional than with any other part of the company. There’s no shared culture or language like there is with other engineers, and the rules of “who gets to tell who what to do” aren’t as clear-cut as they are with managers. Engineers don’t have a lot in common with legal, or design, or sales, but they also don’t need to interact much with those roles. In my experience, engineers are communicating with product managers almost every single day. The worst version of the product/engineering relationship goes something like this: Engineers are technically competent but are too autistic to be fully trusted. They need a kind-but-stern parental figure who knows how to communicate to other stakeholders in the organization (for instance, by being comfortable using the word “stakeholders”), and how to keep engineers from going off in the wrong direction. This entire gross dynamic is neatly captured by the popular term “product mommy” 1 . I really, really don’t like that term, or this entire dynamic in general. Almost none of my relationships with my product managers have been anything like this, though I have seen it at a distance. Working well with product managers can be the difference between succeeding and failing at a company. Why is it so hard to maintain good relationships between engineering and product? What does a good relationship look like? Product managers and engineers have largely non-overlapping skillsets. Product managers don’t understand the technical work engineers do and aren’t equipped to talk about it: if an engineer gives a technical reason for something, product managers generally have to shrug and say “sure, I guess”. Likewise, engineers don’t have anything like the visibility into the organization that product managers do. Particularly in large organizations, it is the product manager who is the source of truth about who wants what and which features are important. When a product manager says that something is critical, engineers generally have to shrug and say “sure, I guess”. This obviously requires a lot of trust. What’s a little less obvious is that this trust is continually broken by both sides . Every single product manager has been told thousands of times that technical task X is technically impossible or would be disastrous, only for that task to end up being done fairly smoothly and successfully. Every single engineer has been told thousands of times that requirement X is absolutely critical and worth going to enormous effort for, only for that requirement to be silently dropped or changed with no apology. Of course this isn’t malicious. Engineers often give wrong estimates because estimation is impossible , and sometimes the dire consequences they warn about really do happen (they’re just handled behind the scenes, like engineers handle many other kinds of technical dysfunction). Product managers “change their minds” because what’s important in a large tech company does genuinely change hour-by-hour 2 , and even the best attempts to only filter the most reliable priorities through to the engineering team will sometimes go wrong. The consequence of this broken trust is that the relationship becomes very difficult to maintain. When you’re an engineer, and you explain something to your product manager, and you know they don’t believe you (despite having no ability themselves to judge the question), it can be incredibly frustrating. Likewise, when you’re a product manager, and you’re desperately trying to explain what we need to do to an engineer, and you know they’re internally shrugging their shoulders, it must be unbearable. Don’t they know this is critical to the company? You were just in a meeting with the leaders of the organization! The natural tool for a mistrustful product manager is manipulation . I still remember a product manager who tried to extract a commitment from my team by asking us to go around and all say “I commit to getting this work done in two weeks”, after a conversation where we’d explained the risks that cause it to take longer. I suppose the idea was that we’d all work much harder, having taken a sacred oath? More subtle variants of this approach involve suggesting that you would be really disappointed if this work was delayed (in true “product mommy” style), or vaguely suggesting the possibility of some abstract reward (that the product manager is not empowered to deliver) if work gets done ahead of schedule. The natural tool for a mistrustful engineer is lies . The most benign version of this is exaggerating estimates: for instance, the classic advice to double your estimate and add 20% . I’ve seen engineers claim that they’ve had to follow up on all sorts of largely-fake tasks (one common example is “reaching out to a neighbor team to confirm X”) in order to gain more time. In the worst case, engineers might even straight-out lie that work has been completed, and then track the “it doesn’t work in production” feedback as a bug. Once this starts happening, it’s nearly impossible to repair the relationship. I can’t bring myself to trust a product manager who’s clearly trying to pull my strings, and I’m sure a product manager can’t trust an engineer who’s lied to their face in the past. That’s why it’s so important to avoid getting into a bad relationship in the first place. Why bother? If it’s so hard to hammer out a good working relationship with product managers, why not just settle for a bad one? Product managers can absolutely bury you if you’re not careful. Product managers are almost always more politically sophisticated than engineers. This is partly structural: product managers are simply in more conversations with the company’s movers and shakers, and so naturally have a better relationship with them (and are thus better attuned to which way the wind is blowing). It’s also partly selection bias: engineers can be hired even with relatively poor social skills, because they’re primarily being assessed on technical ability, but social skills are a core part of the product role 3 . If you are feuding with a product manager, you will probably lose . Unless you’re unusually influential, they will simply have far more opportunities to quietly talk you down in influential circles than you will. All it takes is a few comments like “oh, I probably wouldn’t pick Sean for that project” to wreck your reputation. In the case where you are openly feuding with a product manager, the company’s leaders will by default take the product manager’s side over yours. They’re likely to know them better, have more shared cultural context with them, and in general be willing to interpret the situation as “another engineer who doesn’t understand how the organization works”. There are huge benefits to being trusted by a product manager. Product managers want to ship things , and typically understand a fair amount about all of the non-technical barriers to shipping. If you also want to ship things, you can become a fearsome team. On top of that, because trust between engineers and product managers is so difficult, once you’re in you’re in all the way. Product managers often pick one or two engineers as their go-to for getting the “real story” on technical questions. If that’s you, you have an outsized position of influence in the organization, which you can use to get the things you want done . As an engineer, how can you build trust with your product manager? The first step is to understand where they’re coming from . When they tell you something is important or that a requirement has come in, be aware that this is rarely their decision. It’s not them who’s jerking you around, it’s someone higher up in the food chain jerking you both around. If you can adopt a conspiratorial mindset with them, instead of against them, that’s a good start. Try just asking “oh man, alright, what can we do about this?” instead of complaining. The second step is to be right, a lot . This is a silly-sounding Amazon leadership principle that turns out to be entirely accurate. I wrote more about it here , but (as unfair as it sounds) you really do have to be mostly accurate if you want to build trust with a product manager. When you say something will ship, it has to ship; when you say something is impossible, it can’t happen days or weeks later. It’s okay to be wrong sometimes , but you have to establish a pattern of you providing them useful, correct technical information. The third step is to let them make the political calls most of the time . If you expect them to trust your technical calls, you have to extend them the same trust when it comes to navigating the organization. Don’t publicly undermine them in meetings, bring up your concerns in private. If they say something is important and you’re not so sure, at least act like it is. Accept that sometimes they’re going to be wrong, just like you’re sometimes wrong about technical questions. The fourth step is to get lucky . Sometimes your product manager will just be a dud. You can’t build trust with someone incompetent: there’s nothing for you to trust them with, and they aren’t in a position where they can usefully extend trust to you. Working in large organizations requires getting comfortable with the fact that some of your colleagues will be stronger than others, and figuring out ways to work with (or bypass) people who make the work harder, not easier. Many product managers were once engineers. If your product manager is technical, does that make you immune from these problems? Absolutely not! You likely won’t have much choice in which product managers you work with, but be aware that having once been an engineer is a negative , not a positive. No product manager can ever be technical enough to matter, because they don’t work on the codebase : even if they were a full-time engineer, they wouldn’t have the time to build the specific context on the system they’d need to be a real participant in technical discussions. It’s thus better to have a product manager who knows they’re not technical than to have one who mistakenly thinks they might be. The worst-case scenario is an ex-engineering product manager who believes they’re technical enough to detect when engineers are lying to them. This kind of paranoia is an easy trap for “technical” product managers to fall into, particularly when they don’t have a trusted engineer on the team they can lean on. If you’re dealing with one of these, prepare to spend a lot of time explaining why you can’t “just” do things (and prepare to have those explanations not be believed). At its worst, a product manager relationship is like an unhealthy family: driven by condescension, emotional manipulation, lies, and mistrust. This isn’t because product managers are bad people! It’s because the structure of the relationship creates conflict. Both sides must make commitments (about the technical system or goals of the organization) that are (a) often wrong, and that (b) the other side is unable to independently verify. To avoid the trap, both sides have to be generous, willing to trust each other in their areas of expertise, and most importantly competent . Unlike most roles in tech, product management (particularly the lower-level roles that are more engineer-facing) has close to an even gender split. For instance, based on the whims (or snap decisions, more charitably) of the CEO. I have worked with product managers with poor social skills, but it’s rare: about as rare as working with engineers with genuinely poor (i.e. by general-population standards) technical skills. Unlike most roles in tech, product management (particularly the lower-level roles that are more engineer-facing) has close to an even gender split. ↩ For instance, based on the whims (or snap decisions, more charitably) of the CEO. ↩ I have worked with product managers with poor social skills, but it’s rare: about as rare as working with engineers with genuinely poor (i.e. by general-population standards) technical skills. ↩

0 views
Sean Goedecke 2 months ago

Anti-AI nostalgia and the cult of the past

Programmers were better back in the day, weren’t they? Back when we had real programmers. Not just people who got paid to write code, but people who lived it, who were obsessed with their craft, and whose code was a lively expression of themselves. Hackers were hackers in those days before money took over the industry. Don’t even get me started on LLMs. Could there be a better example of today’s degenerate spirit? A machine to mass-produce software (not good software, just barely good enough), so that the weak minds that dominate the industry can indulge their obsession with quantity : of slop code, of features, and ultimately of money, which is the only way they can understand value. If they weren’t destroying our way of life, they would be pitiable. All of them together don’t have a fraction of the spiritual integrity of someone like Mel . But as it is, we must band together to crush them and drive them from our industry like the parasites they are. Okay, that’s not actually what I believe. But there sure are a lot of posts 1 and comments on the internet that sound a bit like the paragraph above. Here are some older quotes that might sound similar: …the third collapse, in which power tends to pass into the hands of the lowest of the traditional castes, the caste of the beasts of burden and the standardized individuals. The result of this transfer of power was a reduction of horizon and value to the plane of matter, the machine, and the reign of quantity. 2 Usura rusteth the chisel \ It rusteth the craft and the craftsman \ It gnaweth the thread in the loom 3 The actual accomplishments of the past will nevertheless remain accomplishments, while the artistic stammerings of the painting, music, sculpture, and architecture produced by these types of charlatans will one day be nothing but proof of the magnitude of a nation’s downfall. 4 These are all from the writings (or speeches) of famous fascists: Julius Evola, Ezra Pound, and Hitler himself. Mussolini’s Doctrine of Fascism begins by defining fascism as a “spiritual attitude”, which the fascist man adopts in order to regain the mysterious qualities that were lost by the transition to modern life. In his classic Ur-Fascism , Umberto Eco’s first two defining features of fascism are the “cult of tradition” and the “rejection of modernism”. So when someone tells me that the industry has lost its way and we must deny the corrupting influence of modern technology in order to retvrn to the time of virile real programmers (who understood and appreciated the spiritual dimension of programming), I get suspicious. It’s strange to describe anti-AI sentiment as potentially fascist, since a very popular argument is that LLMs themselves are an inherently fascist tool. Surely both sides of the debate can’t be fascist? I do think that the structure of fascist arguments is generally persuasive , and that many avowedly anti-fascist groups do sometimes fall into this trap: describing the world as a struggle between the spiritual power of the macho, traditional man and the corrupting influence of degenerate (often foreign) capital. For instance, I am a big fan of Lord of the Rings. I’ve read the series and watched the films multiple times, and even made a failed attempt to learn Elvish as a kid. But it’s hard to deny that fascists absolutely love Lord of the Rings. “Marble statue of a Roman emperor” might be the most popular avatar for fascists on the internet, but Aragorn is the second most popular. Neo-fascist movements in Italy explicitly take up Lord of the Rings as a foundational text. Why? Because the core conflict in the text is between the traditional, nostalgic heroism of the Shire and Gondor, and the corrupting modern industrial (partly foreign ) influence of Saruman and Sauron 5 . I don’t think Lord of the Rings (or anti-AI rhetoric) is intrinsically fascist. In fact, the surface-level reading of the text is anti-fascist: the plucky people of the West banding together to fight Sauron’s command-and-control totalitarian society. But I can see why fascists love it. One common historical touch-point for anti-AI folks is the Luddites, who were a violent conservative labor movement in early 1800s England. Anti-AI blogs adopt Luddite language like “smashing frames”, and positively cite the Luddites as “the go-to enemies of fascism since its inception”. I’ve written at length about what we can learn from the Luddites in Luddites and burning down AI datacenters , but one point I think is under-emphasized by the (generally pro-Luddite) books is that the Luddites were a little bit fascist themselves . Brian Merchant’s Blood in the Machine is the most popular recent book on the Luddites. I enjoyed it, but Merchant’s attempts to paint the Luddites as a friendly, left-wing, proto-feminist movement 6 seemed really unconvincing to me. From the writings of the Luddites, it’s clear that they were interested in protecting the rights of their all-male elite guild fraternity. Here’s one Luddite threat to a workshop that explicitly includes a threat against the female workers 7 : We think it quite inconsistent with our duty as men, as husbands and as fathers to suffer ourselves to be ruined any longer by a set of vagabond strumpets and those gibbet-deserving rascals that are looking over them. We will lead them to their satisfaction. We sincerely hope, gentlemen, that you will discharge the bitches and take men into your employ again, or they must take what they get. These were fundamentally conservative people who felt (correctly) that modernity had deprived them of their elite status, handing it instead to lower-paid inferiors: women, vagabonds, and foreigners. The Luddites were obviously not fascists 8 . However, the basic ingredients were there: wounded pride, a masculine elite identity, hatred of modern economics, and violence aimed at restoring their previous position in society. The currents that produced Luddism are the same currents that guided so many unhappy people towards fascism. When things are looking grim for an elite group, they often turn towards any movement that promises a return to an idealized past. If my blog has themes, one of them is surely that many software engineers labor under a delusion that their job is to be excellent at their craft. Of course, wanting to be an excellent programmer is not a delusion; it is a completely legitimate value to hold, and a legitimate purpose to pursue. It’s just not what you’re paid to do at work. Your job , unfortunately, is producing shareholder value . This delusion has been punctured by the end of ZIRP , and again more recently by the rise of AI coding. In this environment, I worry that some software engineers will form exactly the kind of disillusioned elite that was the audience for Ezra Pound’s poems about “usury” or the Luddites’ campaign against unapprenticed (often female) textile workers. I worry that AI, and the companies that build AI, are becoming an enemy against which anything is permitted: an enemy which in Umberto Eco’s words is “at the same time too strong and too weak”, unable to reason and yet powerful enough to drastically reshape the global labor market for the worse. The enemy of fascism is nuance. Fascism presents a good, clean, rousing story about a spiritual conflict between right and wrong. It is anathema to fascism to stop and muddy the waters a bit: in this case, to explore the ways in which LLMs, like any transformative technology, can both support and endanger traditional values. In The left-wing case for AI I wrote about how AI is being used right now as a disability aid, and many disabled readers wrote in to share their positive experiences with LLMs, and often how alienated they feel by the anti-AI mainstream on the left. I recently got an email describing how there’s a sudden flood of accessibility software for blind people 9 that’s actually built by blind people , who can now iterate with a LLM to get a product that meets their needs. Framing AI as an ontological evil erases experiences like these. Being anti-AI is not inherently fascist. Many of the anti-AI posts I’ve quoted are thoughtful, sensitive pieces exploring how the author thinks about one of the biggest changes to our industry. I still think the world needs more articles like that, not less, but the more of them I read, the more I recognize the tropes: spiritually pure lovers of the craft, degenerate peddlers of corrupt modernism, a need to return to the traditional ways of the hacker, and a lament for the (potentially) waning power of an elite fraternity of programmers. I know I’m tiptoeing around the worst argument in the world . It isn’t a refutation of anti-LLM arguments to say that they are structurally similar in some ways to fascist arguments, any more than it’s a devastating critique to say the same thing about Lord of the Rings. Sometimes it is good to try and halt the march of progress! Some of our past traditions really were purer and more spiritually robust! It just bothers me, that’s all. I used to read The Story of Mel with unalloyed pleasure. Now it makes me nervous. If you believe you’re fighting the embodiment of fascism , or for the idea of value itself , what tactics are off-limits? What positions might you eventually come to accept? It feels wrong to directly associate my caricature with any actual posts, but it also feels wrong to make a blanket assertion without examples. Just so you know what I’m talking about, here are some posts that have elements of this attitude. I like some of these posts and dislike others. Page 329 of my copy of Julius Evola’s Revolt Against the Modern World . Ezra Pound, Canto XLV. “Usura” should be read as “usury”, or today we could gloss it as “capitalism”: all Pound’s examples of great art were from the pre-capitalist patronage era of art. Adolf Hitler, from his speech at the 1933 Party Congress in Nuremberg. Of course, there’s also historically been a strong pro -technology current in fascist thinking (even specificially Italian fascist thinking ). Page 134 of Blood in the Machine has a brief argument that Luddism was feminist because the (exclusively male) artisans’ wives would provide food for their meetings. No, really. From Kevin Binfield’s Writings of the Luddites , page 40. I’ve taken the liberty of re-rendering it in modern spelling and grammar. Aside from being too early, they didn’t have any connection to the state apparatus of power (in fact, they were ultimately crushed by it) and they famously lacked a singular leader. The example cited was BlindRSS . It feels wrong to directly associate my caricature with any actual posts, but it also feels wrong to make a blanket assertion without examples. Just so you know what I’m talking about, here are some posts that have elements of this attitude. I like some of these posts and dislike others. ↩ Page 329 of my copy of Julius Evola’s Revolt Against the Modern World . ↩ Ezra Pound, Canto XLV. “Usura” should be read as “usury”, or today we could gloss it as “capitalism”: all Pound’s examples of great art were from the pre-capitalist patronage era of art. ↩ Adolf Hitler, from his speech at the 1933 Party Congress in Nuremberg. ↩ Of course, there’s also historically been a strong pro -technology current in fascist thinking (even specificially Italian fascist thinking ). ↩ Page 134 of Blood in the Machine has a brief argument that Luddism was feminist because the (exclusively male) artisans’ wives would provide food for their meetings. No, really. ↩ From Kevin Binfield’s Writings of the Luddites , page 40. I’ve taken the liberty of re-rendering it in modern spelling and grammar. ↩ Aside from being too early, they didn’t have any connection to the state apparatus of power (in fact, they were ultimately crushed by it) and they famously lacked a singular leader. ↩ The example cited was BlindRSS . ↩

0 views
Sean Goedecke 2 months ago

Weird projects I shipped with AI

Where are all the AI-generated projects? This is a common question from AI skeptics: if LLMs are so good at writing code, where is the tsunami of new AI-generated apps, services and games? I personally don’t find this to be much of a paradox. Writing code is only one of the bottlenecks involved in actually shipping a new product, after all. It’s also impossible to talk about the paid work I’ve done with AI (you’ll simply have to take my word that it’s increased my productivity). But one thing I can do is share a list of personal projects I’ve built with AI in the last twelve months. I definitely would not have done all of these by hand. I might have found the time to do one or two of them, but based on my pre-AI track record they would probably have stayed in the “GitHub repo with a few commits” stage. This list is a kind of existence proof : a bunch of weird projects, useful to at least some people, that would not have existed without AI assistance 0 . Most recently I’ve built skifreedle.com , a daily-game version of the classic Windows SkiFree game (i.e. “like Wordle, but for SkiFree”). The code for that is here 1 . I enjoy coding small web games by hand, but definitely would not have had the time to wire up all the different SkiFree objects or build neat features like a ghost of your fastest run. I also tried out a lot of different visual themes for the game UI before landing on something I liked. If I’d done this by hand, I would have only had time to try out two or three different looks, instead of fifteen or twenty. I’m very happy with how this turned out. I’ve been enjoying competing against my brother to get better times, since both of us have a lot of nostalgia for the original SkiFree game. Last year I built Autodeck ! I wrote a blog post about this before, but this came from my partner wishing there was some way to automatically generate Anki cards about random topics she wanted to learn about. It ended up being relatively straightforward to set up an endless feed of auto-generated spaced repetition cards: I set up Stripe payments for this one, more because I was worried about someone running away with my Groq balance than because I wanted to make money, but I was pleasantly surprised to see a bunch of people actually use this. Over five hundred people have tried it out, with enough paid subscribers to cover inference and hosting. I might have built this without LLM assistance, but I almost certainly would not have deployed it as a website. The hassle of setting up a database and Stripe would have just been too much work. I also built an AI-generated endless wiki . I wrote a blog post about this one as well. Like Autodeck, I was fascinated with the idea of non-chat interfaces for LLMs, and I thought a wiki-based approach where you interact with the model by clicking links was pretty cool. I learned the hard way that putting a LLM generation call on the end of a regular link was a bad idea: scrapers would exhaust my inference budget quickly. I ended up faking the no-article-exists-yet links with JavaScript, which at least so far has defeated scrapers. People still email me about Endless Wiki, and there are over 280 thousand pages generated. My original goal was to see if you could eventually generate a page for Neon Genesis Evangelion, starting at the root page and only following links (kind of like wiki golf ). I was successful! You can read the “Evangelion Anime” page here . Almost exactly a month after I launched Endless Wiki, xAI launched Grokipedia . Obviously they didn’t plagiarize me. This is a very easy idea to have, and my site was not the first infinite wiki (though I think it was the first one where you had to discover new pages by clicking on links). But it did take some of the shine off. I built a PWA that caches the VicFlora plant identification database so it could be used with low or no internet. This was more of a utility project for my partner, who likes plants and occasionally goes on field trips where internet is spotty. I would definitely not have done this without LLMs. It was reasonably difficult to scrape the basic dichotomous key from the VicFlora website: their API documentation was out of date, there were multiple possible pathways for fetching data (most of which were not functional), and the format of the data I did manage to fetch was hard to parse. I think I could have done it, with enough effort, but it would have been a substantial amount of work. I’m very happy with how this turned out. It’s not perfect, but it’s functional, and I’ve even had the occasional Victorian botanist email me with bug reports or feature requests, so it’s clearly seeing a little bit of usage. I did a bunch of other stuff that doesn’t necessarily rise to the level of a “deployed project”: my gh-standup GitHub CLI extension to automatically generate a standup report, which has just over a hundred stars, my (low quality) image geolocation benchmark , which I blogged about here , or my skill for extracting features from open-source models. There may not be a flood of AI-generated companies (yet), but at least for me there’s been a flood of small, weird projects that would not have existed without significant LLM assistance. I also want to shout out Simon Willison’s version of this , which is another great example of “weird useful tools that only exist because the cost of creating them was so low”. I did lift the spritesheet from DanielHough’s SkiFree.js , which attributes it to Wing Wang Wao . Of course, the original sprites and art belong to Chris Pirih’s SkiFree and Microsoft. I also want to shout out Simon Willison’s version of this , which is another great example of “weird useful tools that only exist because the cost of creating them was so low”. ↩ I did lift the spritesheet from DanielHough’s SkiFree.js , which attributes it to Wing Wang Wao . Of course, the original sprites and art belong to Chris Pirih’s SkiFree and Microsoft. ↩

0 views