Latest Posts (20 found)
Sean Goedecke 2 days ago

What does "playing politics" mean for software engineers?

Software engineers are often told to “start playing politics”, but most engineers have no idea what that means. Their reference point for “playing politics” comes from fiction like Game of Thrones. Are they supposed to raise an army and depose the CEO, or poison each other at team lunch? Should they book Zoom calls with each other and plot schemes? All of that is obviously ridiculous. In terms of Game of Thrones, software engineers are not lords and ladies. We’re the soldiers and workers of the realm. So you should think about “playing politics” in the way a castle guard would, not one of the major players. The castle guard are not going around poisoning people or forming coalitions between the great powers. They are largely keeping their heads down. But in order to do that, they have to stay aware of the political currents, or they’re liable to do something catastrophically stupid: for instance, making an enemy of a powerful courtier, or arresting somebody who’s on an important mission for the king. Given that, the basic principles of playing politics are something like this: As a software engineer in a large company, you will not be a powerful person . Powerful people are typically in senior management: VPs, directors, and so on 1 . However, not everyone in senior management is powerful. Some are killers who have the active support of the CEO, while others are confused incompetents. How do you know which is which? If someone is clearly ferociously competent, they’re always going to have some power, since upper management tend not to ignore useful tools. But you can’t rely on competence as your only guide. Some managers are powerful for other reasons: they’re friends with the CEO, or they have strong relationships with other groups like legal or sales, or they’re simply willing to do whatever upper management wants done. One signal is who’s leading the important projects. Read your CEO or CTO’s internal updates and pay attention to the projects that are called out by name. Organizations tend to give key tasks to trusted lieutenants. If a manager is leading an area that’s never under the spotlight , they probably don’t have enough clout. Another signal is hiring. Is a manager’s team growing or shrinking? Particularly post-ZIRP , headcount is a rare and precious resource. A manager who’s able to get it is likely a powerful manager, or at least is reporting to a powerful director or VP (which often amounts to the same thing). First, you should try not to make any enemies at all. Most software engineers who get “playing politics” wrong do it by needlessly alienating people: by being rude, unhelpful, abrasive, making non-technical people feel stupid, and so on. This post isn’t really about that. I’m assuming that you can figure out how to be a generically pleasant person on your own. However, competent software engineers will make some enemies . If you’re out there making projects happen, some people aren’t going to like the way you do it, and won’t be a fan of any compromise you offer. I wrote about this in Big tech engineers need big egos : the only way to avoid making enemies is to change nothing, but that’s incompatible with doing the job. Given that, be selective about which enemies you make. If you’re making a technical decision that’s either going to require work from team A or team B, and neither team wants to do it, you should try to pick the team with the least political cover. If you need a powerful VP’s team to do something they won’t like, try to be maximally respectful about it: get that team’s core engineers on-side if you can, or book a meeting with the powerful manager and explain the situation, or (better yet) ask the powerful manager sponsoring your project to go and talk to the other VP for you. (If you don’t have a powerful manager like this, consider abandoning your project). Give way to powerful managers when at all possible. Every so often you really do have to stand your ground — if the system will truly collapse otherwise, or a major customer will have an incident, or if the technical decision really is entirely bone-headed — but almost all cases are not like this. The best advice I’ve ever gotten about playing politics came from a manager I worked with long ago 2 : This is not the hill you want to die on. When I’m about to pick a fight or say something argumentative, and I’m not 100% convinced it’s necessary, I ask myself: is this the hill I want to die on? And it never is. The three rules about disagreeing with powerful people are: Disagreeing in private rarely hurts, if you follow these rules. In fact, it can help. If you can manage to disagree with a manager, get overruled, and then follow their plan without complaining, that can be the best way to gain a powerful friend. But if they think you’re going to keep griping about it, or worse still, complain to the rest of the team and foment some kind of rebellion, there’s no quicker way to make a powerful enemy. If you have powerful enemies at a company (for instance, the CTO or an influential VP doesn’t like you), quit . It’s really that bad. I have never seen this situation turn itself around, except in the very rare case where the CTO or VP is already looking for greener pastures and jumps ship. You cannot recover the situation: they have no incentive to give you the chance to change their mind, and they have almost unlimited ability to screw you on promotions, raises and layoffs. That’s why this piece of advice is second in the list. If you aren’t helpful or if your contributions are invisible, you can work on that and fix it. But if you’ve made powerful enemies, you’re done for. Just as it’s fatal to make powerful enemies, it’s very useful to make powerful friends. How can you do this? Remember you’re a palace guard, not a great lord: you make friends by doing your job . However, you can choose to do your job a little more proactively and diligently when you’re doing it for someone with political clout. One obvious application of this principle is that you should answer Slack messages from powerful people immediately . If you see an ordinary Slack question pop up while you’re doing some task, it’s okay to get to it when you get to it. In fact, it’s ideal not to respond to all questions immediately, so you don’t set unreasonable expectations (and so you don’t seem like you’re sitting around doing nothing). But when a VP comes in with a question, don’t make them wait: answer the question immediately. If the question requires research, send a “let me look into that right now” message, then do the research. This is the easiest way to get a reputation for being helpful 3 . Another way to do this is to lean in on important projects . Suppose you do ten projects in a year. Eight of them are normal, low-priority projects, and two of them are high-profile (say, finishing some big feature before your company’s yearly conference). It’s a mistake to allocate your effort equally to all ten. I wrote about this at length in Doing nothing at work : you should be operating at 80% capacity (or less), so you can then ramp up to 120% when it really matters. Pay attention to the narrative that powerful people are trying to push. Here are some potential narratives: You don’t necessarily have to jump in and start cheerleading, but you should at least not do anything that you know is going to make the narrative look weak. For example, on that last point, it’s foolish to openly argue that the project really was fine all along. Bring it up privately, not publicly, or you risk ruining some clever piece of propaganda that the manager in question is trying to push on the rest of the organization 4 . Finally, an underrated way to help powerful people is to offer them social support and information. Slack messages and planning emails might seem unimportant to you, but powerful people often live in that environment: their primary tool is writing messages like these, just like your primary tool is writing code. Reading and responding (in a supportive way) to these messages is something that most engineers don’t bother to do, but it goes a long way. Likewise, dropping a senior manager a line now and then (say, a heads-up that a particular project landed successfully, or that you got good metrics about some feature) is surprisingly helpful. Senior managers live in an information-poor environment: for them to learn something about a team’s work, that information has to bubble up through several layers of interpretation and summary. In my experience, they’re appreciative of being drip-fed the occasional piece of information, so long as you keep it brief and relatively rare. If you’re directly responding to a VP’s Slack messages or DMing them information, they know you’re the one doing it. But if you’re just doing your job and working hard on projects they care about, they might not notice. Being invisible is probably the most common way engineers fail at playing politics. Fortunately the fix is simple: tell people what you’re doing. If you fix an important bug for a launch, write a message in that launch’s Slack channel saying “hey, I just fixed this bug”. What if you don’t like bragging? Get over it. You have to be comfortable publicly telling people what you’ve done. You should also keep a brag document so you can repeat all of this at review time. Another, subtler way to do this is to gain the trust and respect of the powerful engineers in your area. Senior managers will always have a few trusted engineers they rely on to assess technical questions. They will ask those engineers what they think about you, and will broadly trust those answers. The good news is that if you’re competent and useful, those engineers will already value you, so you don’t have to do anything special: just be good at your job. Is playing politics all about sucking up to senior managers? Basically, yeah. A less cynical way to describe it would be “aligning with the values of the company”. If you think your company is doing good things, you should want to do that anyway! In any case, what that comes down to is figuring out what the people in charge want, giving it to them, and making sure they see you doing it. However, there’s still some scope to get what you want out of the deal. I said earlier that software engineers do not wield organizational power. However, that doesn’t mean you’re powerless. Technical ability is a source of real power, if a delicate and unreliable one. The movers and shakers in tech companies are utterly dependent on technical people to implement their vision and to give them clear answers about the system. There are many subtle ways you can leverage this. One I wrote about in How I influence tech company politics as a staff software engineer is to wait until important people at the company want to do something (say, improve reliability), then offer them a technical plan that does it your way. Another one is to become so useful that you’re actively in demand to lead projects, and then run the project how you want. You probably won’t be able to change the company’s grand strategy. But how that strategy is implemented has a lot of specific technical detail, and you can put yourself in a position to decide on those details. Playing politics isn’t about plotting and scheming, and it isn’t just about being a friendly, likeable person (although that helps). It’s about figuring out how your company actually operates: who makes the decisions, who gets consulted, what behavior gets rewarded, and so on. The most basic way to do that is to figure out who is powerful, get out of their way, and (if you can) help them get what they want . Obviously the exact titles depend on your company. One person I’m deliberately leaving out is your own manager. In general don’t think your relationship with your own manager counts as “playing politics”: that’s just you getting along with another human being. An exception to that is if you report directly to a powerful director or VP. Ironically, this manager struggled to take his own advice. Note that you actually have to be able to answer their question accurately in order to do this. If you’re not competent enough to be useful to powerful people, you will struggle to befriend them. For instance, maybe the CEO is convinced that the project was in bad shape because of something he heard, and the manager in question knows it’s easier to sell “yes, but we turned it around” than “no, you misunderstood, everything was always fine”. If you complicate that process, you risk the CEO thinking that the project is still bad and cancelling it. Be aware of who’s powerful and who’s not At all costs, avoid making powerful enemies Help powerful people as best you can Make sure they know you’re helping them (without annoying them) Make sure you do it in private When they overrule you, stop arguing immediately We’ve had a lot of turnover and reorgs lately, but we’re all starting to pull together as a team now Isn’t it great how focused we all are on reliability work after last month’s incident? The conference this week is the most important thing, so we’re all being very careful not to break anything We’re an AI-forward team that’s looking for the best ways we can leverage LLMs into our team processes Although this project had a rocky start, we’re now all aligned on the way forward Obviously the exact titles depend on your company. One person I’m deliberately leaving out is your own manager. In general don’t think your relationship with your own manager counts as “playing politics”: that’s just you getting along with another human being. An exception to that is if you report directly to a powerful director or VP. ↩ Ironically, this manager struggled to take his own advice. ↩ Note that you actually have to be able to answer their question accurately in order to do this. If you’re not competent enough to be useful to powerful people, you will struggle to befriend them. ↩ For instance, maybe the CEO is convinced that the project was in bad shape because of something he heard, and the manager in question knows it’s easier to sell “yes, but we turned it around” than “no, you misunderstood, everything was always fine”. If you complicate that process, you risk the CEO thinking that the project is still bad and cancelling it. ↩

0 views
Sean Goedecke 5 days ago

In defense of not understanding your codebase

As a software engineer, how well do you have to understand your own codebase? My guess is that people who work on small codebases with low-turnover teams (say, Redis or games like The Witness ) would say “obviously you have to understand it completely, otherwise you can’t do good work”. I’d also guess that people who work on large codebases with high-turnover teams (say, the Google web search backend or GitHub) would say “obviously you can’t understand it completely, you just have to do the best you can in your local area”. These are two largely different ways of programming with different methods, practices and cultures 1 . However, the first group is over-represented in online discussion about software engineering 2 . I want to defend the second group against the first. In many software engineering environments, there’s nothing wrong with being in a state of partial understanding. In fact, in large systems a partial understanding is the best you can do. The best articulation of the “you have to understand your codebase” side is Peter Naur’s famous paper Programming as Theory Building . I like this paper, but I think it goes too far in that direction. Naur’s core point is that when programmers work on a program, the code is really just a by-product, and the main product they’re working on is their “theory of the program”. That’s made up of their intuitive sense of what’s happening and why, which can only be partially captured by code or documentation. If they lost the code, they could rewrite the program easily. If they lost their understanding (say, if the team experienced 100% turnover), they would struggle to make sense of the code. So far, so good, but Naur goes further than this. He says that the theory should not be reconstructed from the code. According to Naur, you’re better off scrapping the program entirely and having a new team rebuild it from scratch , building up a new theory in the process 3 : reestablishing the theory of a program merely from the documentation, is strictly impossible … [therefore] the existing program text should be discarded and the new-formed programmer team should be given the opportunity to solve the given problem afresh Anyone who’s been an effective software engineer at a large company knows that Naur is dead wrong about this. There are at least two reasons. First, you simply can’t rebuild large software systems from scratch . Sufficiently large systems (if they have users) contain thousands of weird cases and quirks that cannot be reimplemented. Even a team that’s intimately familiar with the system couldn’t do it: there’s just too much stuff to juggle. Successful rewrites always start by carving out the existing codebase into small isolated chunks, then rewriting one chunk at a time. In other words, rewriting a software system involves making a bunch of changes to the old system. If you can’t change the old system, you certainly can’t replace it with a new one. Second, abandoned systems are revived all the time . In a tech company with hundreds of millions of lines of code and thousands of engineers, it’s not uncommon for a codebase to have nobody left who’s familiar with it 4 . All it takes is a few people to quit at the wrong time, or for a codebase to be unmaintained for a year. Not only have I seen other teams do this, I have personally taken ownership of abandoned codebases, figured them out, and gotten to a point where I could effectively work with them. It takes time, but building a new theory of the codebase is possible. You start by understanding one flow end-to-end, then slowly branch out from there, making careful changes as you go. In sufficiently large codebases, everyone operates with an incorrect theory of the program . The defining feature of modern software systems is that they’re just way too big for anyone (or even a whole team) to keep in their head: nobody understands it all . To be effective, you have to figure out a way to work with a merely partially-correct theory. This is why I keep going on about taking a position and confidence . If you’re not sure about something, you can’t just sit back and wait for someone with a perfect understanding to come and give you the answer. If you’re a competent engineer, that person is you . You have to grit your teeth, make your most educated guess, and then deal with the consequences. To be generous to Naur, it’s possible that in 1985 the average size of a program was several orders of magnitude smaller than today, and that when Naur writes about “large programs” he’s not talking about tens of millions of lines of code. Naur’s first example of a large program is a 200,000 line industrial monitoring program, and his second example is a compiler. In 1987, the first version of the compiler GCC was about a hundred thousand lines of code; in 2015 GCC was over fourteen million lines. I can believe that rewriting one or two hundred thousand lines of code is relatively straightforward, particularly if you get to reuse existing tests. Not so for one or two million. LLMs are often cited as a tool that’s bad because it impedes the ordinary process of theory-building. I think this is overly simplistic. Like many software tools, LLMs are a double-edged sword: they make it harder to construct a detailed mental theory of the software, but they allow you to build a partial theory quickly and they can help you leverage that partial theory more effectively. This is a complex tradeoff that I’m still thinking about. Setting LLMs aside, I’m confident that it’s silly to say that anything that interferes with your theory of the software must be bad. Here is a partial list of other things that make it harder to maintain a theory: Like most things in software, “maintaining a theory of the codebase” is one value among many. Sometimes it’s the most important value and you sacrifice other values for it; other times you trade it off for speed, or legal compliance, or for political reasons 5 . Almost all engineers — particularly “pure” engineers — prefer to maintain an accurate mental model of their software. It’s more fun, less stressful, and feels more like “real engineering”. That’s why many engineers take up open-source projects in their spare time in order to work on small codebases by themselves: in order to do engineering work where they can maintain an accurate Naur theory of the codebase. I don’t think there’s anything wrong with that. However, at work you are paid to do a job . In other words, they pay you money to adopt their set of engineering values. It’s hopefully well-understood that however much you might personally care about performance, sometimes you have to write slow code at your job (for instance, to get a project done on time, or to accommodate some awkward requirement). Maintaining a theory of the codebase is the same kind of thing. I wrote about this at length in Pure and impure software engineering . I think many of the repeated arguments we have in the software industry are caused by the pure total-understanding culture coming up against the impure partial-understanding culture. Open-source engineers are more excited to blog about their work, the raw engineering content is typically more impressive (because coordination problems dominate big proprietary systems), open-source projects can be legally written about while proprietary systems can’t, and even if you could do it legally, writing about large codebases is impossible because it requires too much specific context . I re-read the relevant chapters of Ryle’s The Concept of Mind (which Naur cites throughout) and I think Ryle is more generous about theory-building. For Ryle, theory-building or know-how automatically happens as you do things. It’s fully consistent with Ryle to think you can pick up an existing codebase just from the code, purely by puzzling it out. Naur says: “Lest this consequence may seem unreasonable, it may be noted that the need for revival of an entirely dead program probably will rarely arise, since it is hardly conceivable that the revival would be assigned to new programmers without at least some knowledge of the theory had by the original team.”. If only! Some engineers might say that maintaining a theory is the core value, because without it you can’t fulfill any of the others. I disagree. You could say the same thing about readability, or maintainability, or correctness, or a bunch of other engineering values. We trade off “core” values like this all the time. Other people being allowed to write code in your codebase Having to implement legally-required features like accessibility and data protection Allowing your colleagues to quit their jobs or move between teams Having to upgrade software versions for security patches Bringing in libraries or other dependencies I wrote about this at length in Pure and impure software engineering . I think many of the repeated arguments we have in the software industry are caused by the pure total-understanding culture coming up against the impure partial-understanding culture. ↩ Open-source engineers are more excited to blog about their work, the raw engineering content is typically more impressive (because coordination problems dominate big proprietary systems), open-source projects can be legally written about while proprietary systems can’t, and even if you could do it legally, writing about large codebases is impossible because it requires too much specific context . ↩ I re-read the relevant chapters of Ryle’s The Concept of Mind (which Naur cites throughout) and I think Ryle is more generous about theory-building. For Ryle, theory-building or know-how automatically happens as you do things. It’s fully consistent with Ryle to think you can pick up an existing codebase just from the code, purely by puzzling it out. ↩ Naur says: “Lest this consequence may seem unreasonable, it may be noted that the need for revival of an entirely dead program probably will rarely arise, since it is hardly conceivable that the revival would be assigned to new programmers without at least some knowledge of the theory had by the original team.”. If only! ↩ Some engineers might say that maintaining a theory is the core value, because without it you can’t fulfill any of the others. I disagree. You could say the same thing about readability, or maintainability, or correctness, or a bunch of other engineering values. We trade off “core” values like this all the time. ↩

0 views
Sean Goedecke 1 weeks ago

Blog about things you don't understand yet

Every post I publish represents at least two things I’ve learned: the thing that prompted me to write the post, and the thing I learned in the course of writing it. If I don’t learn anything new while I’m writing, it’s not interesting enough to publish. Typically I learn way more than two things. For instance, in my o3 geoguessr post, I started out with the idea that most AI prompts probably don’t work, and I ended up learning that newer OpenAI models have lost o3’s ability to geolocate. That’s interesting! In my most recent post on C2PA , I started out with the idea that C2PA requires near-universal adoption, but I learned a ton of things about PKI, managing private keys on local devices, how C2PA actually works, and so on. In my post on the Luddites , I started out with the idea that the Luddite movement was fundamentally decentralized, but ended up fascinated by Luddite culture (which was far more elitist, misogynist, and violent than the pop-Luddism books describe). I could do this for every single post on the blog. I think the core reason this works is that every single one of my blog posts argues a point . I never publish a post that just gives some scattered thoughts on a topic, or a post that only says “yes, I agree with this other article”. If I write a draft that nobody sensible could disagree with, I scrap the draft. Making sure that everything I write is at least minimally controversial is a forcing function: it forces me to think about what the most interesting part of my position is, and it forces me to do enough research to defend it against the obvious criticisms. This is contrary to a lot of advice I read about blogging, which encourages the aspiring blogger to treat their posts as a form of unstructured self-expression. If unstructured self-expression is what you want to do, that’s cool. The point of having a blog is that you get to write what you want. However, this advice isn’t as helpful as it sounds. Before I was in tech, I was a philosophy grad student. But before that , I was a poet. One thing you learn when you try to write poetry is that it is way easier to write to a restrictive structure than it is to simply “write what you feel”. This should be obvious when you actually think about it. The task of a poet is to repeatedly choose the next word. Writing to a structure (typically rhyme or meter) narrows that choice to a small set of words, instead of the entire English language. It’s the same with blogging. Forcing yourself to write about specific, potentially-controversial points makes consistently writing easier, not harder. Writing is the best way to think clearly about a topic. It’s easy to believe you understand something when you’re just turning it over in your head. When you have to condense that down into words, you find out exactly how much you do or don’t understand. I am constantly having moments where I type something, stop myself, and think “wait, that can’t actually be right”, or “is that really true?” By the time I write my way to the end of the post, I’m usually thinking so much more clearly about the topic that my conclusion paragraph is way better than my introduction. In fact, I’ve picked up the habit of going back and immediately rewriting the first paragraph as part of my first-draft process, because I know I’m going to end up doing it anyway. I also change my mind a lot while I write. Here are a bunch of examples of posts where I began writing them with the opposite opinion to the one that eventually made it into the post. I think this is a good sign, and I hope I never stop doing it. You should be researching and thinking about every post you write, and that means you should frequently learn new things that change your mind. Because of all this, I deliberately choose to write blog posts about things I don’t yet quite understand but would like to, like LLM steering, Stripe’s Tempo blockchain, C2PA and watermarking , space cooling , interaction models , LLM inference internals , and so on. This is great for me, because I learn a lot. Is it great for my readers? I sometimes worry that I should only be writing about areas I already know very well, like tech company dynamics or working in large codebases , rather than presenting myself as an authority on fields I’m actually still learning. Should I let historians of the Luddites write about Luddism, Web3 engineers write about blockchains, and so on? I think this is acceptable for three reasons. First, it’s sometimes easier for a beginner to write an introduction to a field than for an expert. Experts routinely overestimate the knowledge of the general public, and have often internalized the reasons why their field is important so deeply that they struggle to express them. I think my explainer posts are valuable because I always spend the first chunk of the post talking about what the original problem is before I get into the technical solution. Second, sometimes the public consensus on a topic is just plain wrong, to the point where even a little bit of research is enough to demonstrate why. Many of my posts I’m proudest of have been along these lines: arguing that the “500ml per prompt” water usage figure for LLMs was ludicrous , or that the popular Apple “Illusion of Thinking” paper was tracking persistence, not reasoning , that GPUs live longer than three years and the AI companies have large profit margins on inference, and so on. Third, I try to make it clear on my blog who I am and what my credentials actually are. Even if it’s not explicitly described in the post, I have my real name and resume available on my /about page, so I don’t think a careful reader could be easily fooled into thinking I’m an expert on 19th-century England or space physics or LLM economics or anything like that. Even if nobody reads what you write, writing is still a good discipline for getting your thoughts in order. But another big reason why writing is a great learning tool is that you can get feedback . I think it’s obvious why this is useful, but I do want to make two points about feedback. First, if you do make your posts public, you need to have a pretty thick skin. People on the internet often fall over themselves to come up with the most cutting criticism or the harshest dunk. This goes double if you take my previous advice and try to write posts that make a clear, controversial point about a subject you’re learning. If you’re the kind of person whose whole day is ruined when a stranger is cruel to them, you might want to keep your blogging private or only share it among friends. Second, even if your blogging is private, you can get feedback from LLMs . Like humans, LLMs will often give junk feedback. In my experience, OpenAI models will always tell me to moderate my claims or add caveats and hedges until I’m not saying anything at all. Sometimes their criticism will be straight-up wrong. But — particularly about technical topics — LLMs are great at pointing out areas you’ve genuinely misunderstood, and they’re far kinder than the average Lobsters or Hacker News commenter. I’m pleased and grateful that people enjoy reading my posts, but even when nobody did, I still got a lot of value out of blogging. I write as a method of thinking more clearly, as an excuse to do research on topics I want to learn about, and as a way of getting feedback. If you’d like to try it yourself, I suggest watching for these two things. First, you should be changing your mind a lot as you write. If not, you probably aren’t doing enough research. Second, your first draft’s conclusion should be much tighter and more expressive than its introduction. If not, you probably haven’t learned anything from the writing process, which means the draft can be scrapped. I strongly recommend this practice to anyone with an interest in writing. You will see the benefits even if you don’t publish any of your writing on the internet, particularly now that you can get good technical feedback by pasting your post into a LLM 1 . For what it’s worth, I’ve fiddled with careful “review prompts” and it’s basically as good to just write “review, please:” and paste your article. For what it’s worth, I’ve fiddled with careful “review prompts” and it’s basically as good to just write “review, please:” and paste your article. ↩

0 views
Sean Goedecke 1 weeks ago

C2PA only works if everything is signed

The European Union AI Act is Europe’s attempt to comprehensively regulate AI usage. A big part of that is the requirement that AI-generated content be identifiable: either tagged with a watermark or with what the Act calls “digitally signed metadata” 1 . Since all this becomes enforceable in a month, it’s worth figuring out if it makes any sense. I recently discussed AI watermarking at length in Text AI watermarks will always be trivial to remove . What about digitally signed metadata? The most well-known implementation of digitally signed metadata is C2PA Content Credentials, which incorrectly 2 claims to be the technology that the AI Act gives as an example of how to do signed metadata properly. The idea here is that every single image file should contain unspoofable authorship metadata . Here’s my position on it: Lots to unpack. Let’s start by considering images, since that’s the easiest case. When an AI tool generates an image, that tool should include a “made by ChatGPT” disclaimer in that image’s metadata. Likewise, when a camera takes a photo, that camera should include a “taken by a camera” disclaimer. C2PA uses two strategies to protect this metadata: Each physical camera (or phone) has its own private key, for obvious reasons 3 . How do we know that those millions of private keys are trusted? Via PKI , like HTTPS: each camera’s private “certificate” (which contains its public key) is signed by the manufacturer’s well-known private key, so the chain of authenticity can be verified as long as you have (say) Apple’s root public key 4 . What happens if you then edit your photo in Photoshop? Photoshop will leave the camera’s metadata untouched, but will layer a “also, Photoshop was used” piece of metadata over the top, signed with Adobe’s private key (well, with the private key associated with your official copy of Photoshop, which is signed by Adobe’s official private key). Likewise, if you ask ChatGPT to generate an image for you, ChatGPT will sign its “made by ChatGPT” metadata with OpenAI’s private key. In theory, every single image could contain unforgeable C2PA metadata, allowing software like Twitter to trivially distinguish real photos from fake ones. Right now, C2PA does not have anything like the adoption it’d need to work. It’s hard to find hard data on how many images in the wild use C2PA, but FotoForensics reports around a dozen per week (so around 600 out of the 900,000 images processed each year). This is even worse than it sounds, because basically all of the signed images are AI-generated. The adoption rate of C2PA for human-generated images is much, much lower: so far, Google’s Pixel 10 is the only phone camera to sign photos by default. The iPhone doesn’t sign photos. If almost all AI images are C2PA-signed, but almost no human-generated images are, consumers have no reliable way of identifying AI content, because anyone who wants to pretend their AI content is human can simply remove the signature. For C2PA to succeed, it needs to be on every camera and every phone, so that a photo with no signature is rare and suspicious. Is that realistic? Actually, I think it is. The appetite (at least in the EU) to regulate AI will increase over time, and while the current EU AI Act only mandates that AI-images are tagged (which by itself is useless), it’s plausible that some future regulation will enforce tagging of all images. Another adoption problem that must be solved for C2PA to work is preservation . Right now, if you download a C2PA-tagged image, send it as a Facebook message, then re-download it, the C2PA manifest is stripped out. Most images we see on the internet have passed through some social media asset server at least once. All of these social media companies would need to update how they re-encode image content in order to preserve the C2PA data 5 . This would almost certainly require more regulation: C2PA adds tens or hundreds of kilobytes to each file, which at social media scale is big money 6 . Could a clever attacker forge a C2PA signature? Kind of. Neal Krawetz, who seems to have led the anti-C2PA charge, points out that with a camera development kit it’s straightforward to trick a digital camera into thinking that it’s taking an image when in fact it’s being fed one. This is very much not my area, so please write in if you know more about camera hardware and you think I got this wrong. I suppose you could also take a photo of an AI image on a screen, though I imagine you’d have to be careful to make it look real. If you exclude physical attacks on a digital camera, I think C2PA is more robust. You can sign a photo with a self-signed certificate, but the C2PA spec and docs say that validators must check that your certificate bubbles up to the official C2PA trust list . This list currently contains only 26 certificates, and there’s a whole process for being added to it. That’ll slow down adoption, but at least it makes it hard to forge 7 . We’ve been talking exclusively about images, but it’s more or less the same story for any type of content. If the file doesn’t support JUMBF metadata (say, an Excel file or a PDF), then the C2PA metadata has to live in a “sidecar”: a separate file, probably on some Microsoft or Adobe content server, which contains the signed checksum and the data about who created the file. However, the distinction between “real” and AI-generated content is fuzzier when you’re not talking about images. Here’s a trivial example: if I ask ChatGPT to create an Excel spreadsheet for me, the file will be tagged as AI-generated, but I can simply copy/paste the content into a new Excel doc and save it, which will tag it as human-generated 8 . There’s no software tool that can identify when I’m retyping some AI-generated text (except for perhaps text fingerprinting , which has its own raft of issues). There are also interesting questions around key management. ChatGPT and other AI tools have an easy problem — their users are all online, and so the files can be signed server-side — but how do you sign files created via Photoshop/Excel/Word? If the user doesn’t have internet, do you use some kind of local key? If so, how do you prevent that key being extracted and used to sign AI-generated content? Finally, is it a civil liberties problem to automatically fingerprint every photo? Does it make it impossible to be a whistleblower if every photograph can be traced back to your camera? I think this is a complicated question, but in short: I’d expect whistleblowers to already strip EXIF metadata from their images, C2PA metadata is similarly trivial to strip out, and overall I think image attribution is positive for whistleblowers because it heads off “this was AI-generated” responses. C2PA is probably here to stay. But it isn’t useful now, and won’t be useful until two huge programs of technical work are completed: This will be a long organizational process, since each manufacturer must go through the approvals process (or decide to start their own competing system), evaluate the legal ramifications of storing attribution data in images, and so on. It will be a long technical process, because C2PA metadata is a substantial fraction of image sizes: storing it will add many petabytes of content. Of course, just because C2PA isn’t useful doesn’t mean we’re not all going to do it. Lots of companies are under pressure to signal that they care about AI safety and to head off regulatory attack. “We’re cryptographically signing AI-generated content” is a compelling “we’re doing something ” pitch, particularly for people who aren’t technically savvy enough to understand the limitations. In the near term, I expect large AI-involved companies to invest a substantial amount of engineering effort in C2PA-related activity. In the long run, once everyone gets on board, I think C2PA could end up working well. It’s awkward in some ways, but “attest content via a PKI certificate chain” is a good idea. Is it possible to defeat? Yes, of course. By design, private keys will be in the user’s hands — in their cameras, in their local versions of Photoshop or Microsoft Word, in their phones — so sufficiently technical users will be able to crack them out or use them to sign whatever content they want. I still think C2PA will end up stemming the tide of AI content, because most users are not going to be sophisticated enough to perform attacks like this. However, we should still retain some skepticism of unlikely-looking content, even if it has “created by a human” in its C2PA metadata. See sub-measure 1.1.1 of the Act’s associated Code of Practice . While an early draft of the Code of Practice made an offhand mention of Content Credentials (in the caption of a picture), that was stripped out. The contents of the Act and the final Code of Practice don’t contain “C2PA” or “Content Credentials” (you can search for yourself here ). Otherwise if you cracked the key out of one Sony camera, you could spoof content from any Sony camera. In practice there are usually more “links in the chain”: a device will be signed by some intermediate certificate, which in turn will be signed by another intermediate certificate, which will be signed by the root certificate. That’s because the root key is so valuable. If an intermediate private key leaks, it can be revoked and replaced (via the root key), but if the root key leaks, it would take years to rebuild the network of trust. So almost all signing is done by intermediates, and the root key stays on a USB drive locked in a safe somewhere. Not to mention that the whole point of C2PA is that these social media companies will be displaying a “human or AI” sticker in their UI, which will require retaining the metadata. C2PA allows for storing the manifest content as a separate file, and just including a manifest url in the image metadata itself, but that doesn’t solve the cloud provider problem: they still have to store all the files on-disk somewhere. I think this defuses Neal Krawetz’s “worst-case scenario” . I downloaded his forged image, and (as expected) it gets flagged as “signed, but we don’t trust the root”. I think Krawetz was right at the time, though, since the official “trust list” was only launched in mid-2025. You could do the same thing with images by copying into Photoshop or Paint, but while that’d obscure the AI source, it would still be clear that the photo wasn’t taken by a camera. C2PA broadly makes sense and is a good idea It is pointless to use C2PA for AI-generated images only It will take many years for C2PA to be adopted across all images Because C2PA makes such great safety theater, we’re going to see a lot of hue and cry about it long before it becomes useful The metadata must be signed by some trusted private key The metadata contains a hash of the file’s contents, so you can’t copy an existing signature onto a new file Every camera manufacturer (including phones) must C2PA-sign all images by default Every social media company must retain the C2PA metadata on uploaded images See sub-measure 1.1.1 of the Act’s associated Code of Practice . ↩ While an early draft of the Code of Practice made an offhand mention of Content Credentials (in the caption of a picture), that was stripped out. The contents of the Act and the final Code of Practice don’t contain “C2PA” or “Content Credentials” (you can search for yourself here ). ↩ Otherwise if you cracked the key out of one Sony camera, you could spoof content from any Sony camera. ↩ In practice there are usually more “links in the chain”: a device will be signed by some intermediate certificate, which in turn will be signed by another intermediate certificate, which will be signed by the root certificate. That’s because the root key is so valuable. If an intermediate private key leaks, it can be revoked and replaced (via the root key), but if the root key leaks, it would take years to rebuild the network of trust. So almost all signing is done by intermediates, and the root key stays on a USB drive locked in a safe somewhere. ↩ Not to mention that the whole point of C2PA is that these social media companies will be displaying a “human or AI” sticker in their UI, which will require retaining the metadata. ↩ C2PA allows for storing the manifest content as a separate file, and just including a manifest url in the image metadata itself, but that doesn’t solve the cloud provider problem: they still have to store all the files on-disk somewhere. ↩ I think this defuses Neal Krawetz’s “worst-case scenario” . I downloaded his forged image, and (as expected) it gets flagged as “signed, but we don’t trust the root”. I think Krawetz was right at the time, though, since the official “trust list” was only launched in mid-2025. ↩ You could do the same thing with images by copying into Photoshop or Paint, but while that’d obscure the AI source, it would still be clear that the photo wasn’t taken by a camera. ↩

0 views
Sean Goedecke 2 weeks ago

Text AI watermarks will always be trivial to remove

The European Union AI Act will begin to be enforceable in August 2026, one month from now 1 . One of the biggest new requirements is Article 50 , which requires all AI outputs to be “detectable as artificially generated”. In other words, if LLM providers want to do business in the EU, they will have to apply a watermark to their outputs 2 : some hidden signature that can be used to identify AI content. LLM text watermarking is a fascinating problem. Like the best engineering problems, it is theoretically hard to solve perfectly, but has multiple partial solutions: for instance, Google’s SynthID , and (as I’ll argue) some quiet Unicode trickery from OpenAI and Anthropic. It will be interesting to see how the AI labs navigate these tradeoffs before the end of the year. I wrote about AI watermarking at the end of last year in AI detection tools cannot prove that text is AI-generated . It’s easy to watermark an image, because digital images contain lots of noise that the human eye can’t really see. For instance, you could apply a watermark like “these twenty pixels in these exact spots will always share a color”. Text is much, much harder. Unlike images, text is a very compressed medium: you cannot make any change to a sentence that a human wouldn’t notice (with one exception, which we’ll get to later). So how are you supposed to watermark it? It’s basically a text steganography problem (concealing a secret code), made more difficult because the plaintext cannot be arbitrarily manipulated. Any changes you make to apply the watermark will compromise the quality of the output. For instance, “every fifth letter is an ‘e’” would be a good watermark, but applied naively would make the AI output full of typos. Could you just let the model figure out how to fit the watermark? Strong AI models are smart enough to juggle this kind of constraint 3 , but it’d still consume reasoning time that would be better spent on the user’s problem, and make the model sound much less capable than it is 4 . Do you really need a watermark? If you’re Anthropic, and you’re required to be able to verify whether your models produced a particular block of text, can’t you simply run the text through each model, measuring as you go how closely the model’s predicted tokens match each token from the text? Not really. The space of “all possible Claude Sonnet answers to a question” is way larger than the space of “all possible watermarked answers to a question”. In other words, you’d get too many false positives for human text that reads like it was AI-written. It’s way more likely for a human to accidentally write like Claude than it is for a human to accidentally reproduce a watermark. It would also be prohibitively expensive to run every Anthropic model against a piece of text in order to watermark it. The EU AI Act will eventually require labs like Anthropic to offer free watermarking services to every EU citizen (see Commitment 2). You couldn’t do that with the “run the model” approach. As far as I know, the only AI provider to say they watermark text output is Google, who use a tool called SynthID . Here’s how it works. When a LLM generates text, it’s generating a series of tokens (words or chunks of words). At each step, the model itself doesn’t output a single token, but instead outputs a full list of all (say) 100,000 tokens in its vocabulary, each annotated with the probability that that token will be the next one. Tools like ChatGPT or Claude Code will pick semi-randomly from the most likely options in order to get their outputs. This semi-random sampling process can be influenced in a detectable way. For instance, we could choose a sampling strategy like “we pick the second most likely token, then the first, then the second, then the first, and so on”. That would still produce high-quality output, but you’d be able to re-run the model against the generated text to verify that the pattern holds. However, that’d make verification really expensive, and any slight tweaks to the output would break the pattern and thus break the fingerprint. Is there a better way? Yes. SynthID is a process for assigning each token a “score” based on its previous tokens (for instance, sum the token’s ID with the IDs of its previous three tokens then take mod 5) 5 . To apply the watermark, the model adopts a sampling strategy like “out of the top five most likely tokens, pick the one with the top SynthID score” 6 . The watermark can then be detected by calculating the aggregate SynthID score of a block of text. If it’s suspiciously high, it’s very likely to have been AI-generated. This is basically a version of the common advice that you can identify LLMs by use of the em-dash , except that instead of a list of keywords, it relies on subtle mathematical relationships between words that humans can’t identify. Because the process for assigning the score is trivial, it’s very cheap to run watermark detection. Google have a complicated mathematical rationale for why SynthID doesn’t make the model dumber: supposedly the SynthID scoring is random enough to act like a normal pseudo-random token sampler, just one that leaves a detectable fingerprint on the outputs. But of course this is suspicious. For instance, it’s common to do inference setting temperature to zero, which always picks the model’s most likely next token. In that case, you can’t leave a fingerprint at all (or you have to ignore the user’s preference and pick the second or third choice anyway). If you can’t alter the model outputs, can you still fingerprint the content? Well, kind of. I’m pretty sure OpenAI and Anthropic are sometimes applying fancy Unicode tricks. For instance, you might go through and replace your normal ” ” spaces (unicode ) with a three-per-em ” ” space (unicode ), or a CJK ideographic ” ” space (unicode ). These are called “homoglyphs”, and you can find more of them here . Of course, lots of human-generated text uses homoglyphs. But it’s trivial to encode a pattern of homoglyphs (say, “every third space becomes a three-per-em”) that is much less likely to occur in the wild. Like the SynthID watermark, a homoglyph-based watermark can be detected very cheaply. A homoglyph-based watermark is cheaper to apply than SynthID: you could even do it entirely on the client. I don’t think this is a conspiracy theory. Claude Code was definitely doing this to tag suspicious requests from Chinese users (exploiting homoglyphs for the ’ character in “Today’s date”, though they’ve since walked that back). In the last few years, I’ve noticed that when I copy blocks of text from ChatGPT and paste them into VSCode, sometimes VSCode marks some or all of the spaces as unusual Unicode characters 7 . Are OpenAI and Anthropic using homoglyphs as an AI-generated watermark? I’m not sure. But they’re definitely using homoglyphs. The AI Act (specifically, its associated Code of Practice ) requires watermarking to be “embedded within the content in a manner that is difficult for it to be separated from the content”. However, text watermarks can be trivially removed. To remove unicode homoglyph watermarking, you simply have to replace all the homoglyphs with their “real” character equivalents. If you have access to even a relatively weak un-watermarked LLM 8 , you can strip out SynthID watermarking by asking that LLM to paraphrase the text content. Because the watermark is inherent to subtle vocabulary choices, re-wording the content will remove the watermark. You could even do it by hand, although at that point it’s not really AI-generated content anymore. Since there will be some kind of free public watermark testing tool, you can just keep tweaking until it comes back negative. Moreover, the AI Act requires watermarking techniques to be “interoperable… as far as this is technically feasible”. That means AI providers would have to publish their watermarking process, and potentially even attempt to standardize on applying the same kind of watermarks. I just don’t see how this is compatible with the kind of security-by-obscurity that LLM text watermarking depends on. Unlike image and video watermarks, text watermarks will always be trivial to remove. The AI Act and Code of Practice talk a lot about “digitally signed metadata”. The idea here is that you can include an AI disclosure in the file’s metadata itself, ideally in a way that cannot be tampered with (for instance, by signing a hash of the file’s contents). This signed-metadata process is basically C2PA Content Credentials . While you can remove C2PA metadata, you (theoretically) can’t fake it, so a file with “created by a human” metadata can be trusted, and files with no metadata at all can be held in suspicion. This post is already too long to get into what I think about C2PA, but I do want to say that C2PA is not a substitute for text watermarking . It only really applies to files . In the words of the Code of Practice, that’s “a data format that supports attaching metadata (e.g., an audio, image, video, or containerised text)“. The output of chat tools (and most of the output of AI agents) is not containerized text, but plain old regular text, and so can’t be signed. What would it even look like to sign ChatGPT outputs? There’s no artifact to pass around. I think it’s a fascinating question whether Claude Code has to C2PA-sign any HTML files or PDFs it generates for you. That seems kind of tricky to get right. But in any case, the AI Act also mandates some kind of actual watermarking as well. So what’s going to happen this year? If I had to guess, I’d say that each AI provider (not just labs like OpenAI or Anthropic, but third-party providers like Fireworks or Groq) will stick a SynthID token sampler in front of their inference stacks. This might be limited to users in the EU, but it might not be, since SynthID is at least as good as a normal top-k token sampling approach. AI providers will then offer a “check for watermark” page that re-tokenizes user-provided text, runs the scoring, and checks whether it’s above a certain threshold. Depending on how seriously the interoperability clause is taken, providers might even standardize on the same SynthID setup, in which case there could be a single EU-hosted “watermark this text” page. I don’t think unicode-based watermarking is going to be considered compliant with the AI Act, but some providers which don’t want to set up SynthID might try it. Either way, technical users will be able to strip out the watermark at will, and there will be a plethora of tools that non-technical users will use for this purpose. Well, for new systems; existing ones get until December. I don’t think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that they’re looking for watermarks. Even with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an “e”, but GPT-5.5-Pro produced this puzzling koan: “These hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.” I leave the analogy with AI safety guardrails as an exercise for the reader. That’s a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens. The tokens are scored in a multi-round knockout against each other, but I think that’s more of an implementation detail and not required to get the core intuition behind why SynthID works. When this became public knowledge , OpenAI claimed it was just a model quirk, which is certainly possible. All AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text. Well, for new systems; existing ones get until December. ↩ I don’t think the plain text of Article 50 requires this, but Recital 133 and the Code of Practice makes it pretty clear that they’re looking for watermarks. ↩ Even with extra high thinking, GPT-5.5 could not explain SynthID to me with every fifth letter being an “e”, but GPT-5.5-Pro produced this puzzling koan: “These hidden codes label model-made image, voice, movie, prose. Probe trace: maybe a model-made piece. Maybe erase trace; maybe leave trace. Hence trace alone? No.” ↩ I leave the analogy with AI safety guardrails as an exercise for the reader. ↩ That’s a toy example. In practice there are multiple different (but still mathematically simple) scoring methods that get combined together, including a random seed. Why include the seed? Otherwise the watermark would bias towards the same set of tokens. ↩ The tokens are scored in a multi-round knockout against each other, but I think that’s more of an implementation detail and not required to get the core intuition behind why SynthID works. ↩ When this became public knowledge , OpenAI claimed it was just a model quirk, which is certainly possible. ↩ All AI providers might be legally required to watermark, but even tiny local models are good enough to paraphrase text. ↩

0 views
Sean Goedecke 2 weeks ago

AI inference is obviously profitable

Many people claim that AI inference is unprofitable to serve, and thus must be subsidized by an ocean of dumb money from investors who believe that some future AI model will come to dominate the world economy. When that dumb money goes away, so will AI products. According to this view, LLMs are just inherently too expensive (in terms of money, power, and water) to be used in consumer products. In fact, they can only be used today by externalizing the costs: money onto VC funds and now retail ETF investors , power onto electric utility consumers , and water onto the communities where datacenters are built. There are good reasons to dislike AI, but this really isn’t one of them. In fact, AI inference is obviously profitable . Frontier AI providers are reporting 70%-80% gross margins on inference, but maybe we can’t trust them. Let’s do some very rough estimates on the actual cost. A Nvidia A100 consumes 400W of power under full load. In practice, even a carefully-tuned inference server will not be at full load all the time, but it’s at least an upper bound. Suppose you’re running a dense 70B model 1 , which will fit comfortably (unquantized) on four A100s at around 2M tokens per hour. At industrial power prices, that’s about 13c/hr in the USA . Suppose (pessimistically) cooling is the same cost. That’s about 13 cents per million output tokens 2 . Let’s amortize the cost of the GPUs, since that’s going to be the most expensive part. An A100 costs about $20k. If each A100 lasts around five years 3 , you’ll have to make 16k/yr in profit to recoup your capital investment (or $1.80 per hour). At lower utilization, it’ll take longer to recoup, but your GPUs will also last longer. Either way, your overall inference costs are at about one dollar per million tokens. GPT-5.4-mini charges $4.50 per million tokens, and stronger OpenAI or Anthropic models are three to six times as expensive. It’s hard to make a direct comparison because we don’t know the size of OpenAI or Anthropic models, but the claimed 70% or 80% profit margin is extremely plausible. What if you don’t trust my estimates either? Let’s look at the pricing of open-weights Chinese LLMs. DeepSeek have claimed a bit over 80% profit margin on inference for DeepSeek-R1. Since their API pricing for R1 is less than half that of OpenAI or Anthropic 4 , that suggests that my estimates above for inference cost might be too expensive. Cooling at scale is probably cheaper than power, R1 only has half the active parameters of a dense 70B model, modern GPUs are more efficient than the A100, and there are significant economies of scale in inference. Since DeepSeek’s models are available for anyone to download, they can’t get away with extracting a large profit margin. One of the other inference providers would undercut them with the same model. Inference costs for DeepSeek-V4-Pro on the market are around 87 cents per million output tokens, which is probably pretty close to the actual cost of serving the model. All of this doesn’t mean that OpenAI or Anthropic are profitable. Those companies are making huge capital investments that may or may not pan out, and are spending enormous amounts of money on talent and compute to train brand-new models and retain users. They’re doing crazy things like offering per-month subscription models for nearly unlimited inference, which is almost certainly not profitable. If you used an API token instead of your Anthropic subscription in Claude Code, you’d pay ten times the cost. But that doesn’t mean API-based Claude Code couldn’t be a good deal. Some people are already using DeepSeek’s inference API for agentic coding, because once you take away the huge profit margin it’s cheaper than the relative per-month subscription. Why won’t OpenAI or Anthropic lower their prices? Supposedly OpenAI has thought about it , but for an AI lab, inference has to subsidize training costs . A company like OpenAI has to fund the production of new models from the inference margins on existing models (at least partially). That’s why the margins on inference are so high: the AI labs are trying to squeeze out every dollar so they can stay alive in the training arms race. However, inference only has to subsidize training costs for an AI lab . If you’re merely an inference provider, you don’t have to do any training at all. Therefore, even if OpenAI and Anthropic go out of business, whoever snaps up the rights to their frontier models will be able to continue selling Opus and GPT inference at a profit 5 . The AI bubble popping will not mean the end of the inference business, because AI inference is obviously profitable . Expensive frontier models are probably mixture-of-experts, not dense, which is tougher to estimate. However, I think a 70B dense model and a MoE with 70B active params will come out to basically the same numbers at scale (though the MoE will require more GPU memory and thus a greater upfront cost). Are frontier models around 70B params? Nobody outside the AI labs really knows, but my guess is that 70B is probably larger than a Haiku/mini class model. I think it’s reasonable to estimate the cost of output tokens only, since they’re by far the most expensive part of serving inference. Input tokens are cheaper for two reasons: transformers let you prefill them in parallel, and for most real-world use cases they can be aggressively cached in the KV cache. It’s common (and wrong) to estimate GPU lifespan at three years. I wrote a lot about this in AI GPUs probably live longer than three years . Again, this is just an guess, since we don’t know what OpenAI or Anthropic model is equivalent in size to R1. I do wonder if Anthropic would be able to prevent other people from being able to access the model if the company goes out of business. Anthropic is currently in debt to Broadcom, Google, and a bunch of private equity firms. Would they get the Mythos and Opus weights, over Dario’s protestations? Expensive frontier models are probably mixture-of-experts, not dense, which is tougher to estimate. However, I think a 70B dense model and a MoE with 70B active params will come out to basically the same numbers at scale (though the MoE will require more GPU memory and thus a greater upfront cost). Are frontier models around 70B params? Nobody outside the AI labs really knows, but my guess is that 70B is probably larger than a Haiku/mini class model. ↩ I think it’s reasonable to estimate the cost of output tokens only, since they’re by far the most expensive part of serving inference. Input tokens are cheaper for two reasons: transformers let you prefill them in parallel, and for most real-world use cases they can be aggressively cached in the KV cache. ↩ It’s common (and wrong) to estimate GPU lifespan at three years. I wrote a lot about this in AI GPUs probably live longer than three years . ↩ Again, this is just an guess, since we don’t know what OpenAI or Anthropic model is equivalent in size to R1. ↩ I do wonder if Anthropic would be able to prevent other people from being able to access the model if the company goes out of business. Anthropic is currently in debt to Broadcom, Google, and a bunch of private equity firms. Would they get the Mythos and Opus weights, over Dario’s protestations? ↩

0 views
Sean Goedecke 1 months ago

AI GPUs probably live longer than three years

People who think current AI use is unsustainable often rely on the claim that inference GPUs only last “three years at the most” under load 1 . The idea here is that once the AI bubble money drains away, current infrastructure will rapidly become obsolete, and there won’t be enough money floating around to buy a whole slate of brand-new GPUs. Inference costs would thus rapidly become way too expensive for current AI products to make any financial sense. Where does this “three years at the most” claim come from? Is it plausible? The original Tom’s Hardware article quotes this tweet from Tech Fund, an anonymous former PM and tech investor, who quotes an anonymous “GenAI principal architect” at Google as saying “if you have a high utilization rate, then constant high utilization rate for a year or two, I think the lifespan will be three years at most”. This screenshot looks like it was from an interview. What interview? I scrolled back to October 2024 on Tech Fund’s Twitter feed and saw a bunch of similarly-formatted screenshots , some of which were cited as coming from Tegus . Tegus is apparently a company with a business model of reaching out to insiders (in this case, AI company employees) and paying them hundreds of dollars an hour in order to answer specific technical questions. It’s essentially gig work for almost-but-not-quite insider trading: the more informed and confident you sound, the more likely Tegus analysts will pick you for future interviews. I’m sure the source for this tweet is in fact a GenAI principal architect, since Tegus would have presumably asked for some proof of that before they paid them out. But it’s pretty clear that the incentives here are to sound confident and authoritative, even on questions that you’re not sure about. With that in mind, the quote itself also reads a bit suspiciously. I’ve worked with enough principal engineers and architects to take their casual back-of-envelope estimates with a grain of salt. If they knew the actual rate at which GPUs fail and get retired in Google datacenters, wouldn’t they have just said that? We have some anecdotal evidence that points the other way. Google has publicly claimed to have eight year old TPUs (their version of GPUs) running in production at “100% utilization”. Nvidia only made A100 GPUs from 2020-2024 , but in February 2026 the AWS CEO claimed that AWS had never retired an A100 server (and you can still easily rent A100s for AI work) 2 . AI GPU usage isn’t exactly like crypto mining GPU usage, but it certainly seems like years-old ex-crypto GPUs are functional . There’s also this comment from Hacker News I noticed where someone claims that their GPU cluster in academia has lasted six years with less than 20% failure rate. What about hard data? It’s hard to get concrete data on the lifespan of AI GPUs, because modern AI datacenters have only existed for a handful of years. But an interesting case study would be recent supercomputer clusters like Oak Ridge’s Summit , which had over 27 thousand Nvidia V100s running from 2018 to 2024, or its predecessor, the Cray Titan supercomputer that ran from 2012 to 2019. I couldn’t find any evidence that Summit had to buy an additional 27,000 GPUs to replace their old ones, and GPU failures in Titan have been carefully studied : These cages of GPUs are stacked vertically, and cold air is pumped in from the bottom, which explains why cage 0 (at the bottom) has better survival rates than cage 2 (at the top). Let’s consider cage 0, so we’re just looking at the GPU lifespan instead of at the lifespan of improperly-cooled GPUs. At three years, over 95% of GPUs survived 3 . At six years, nodes 2 and 3 (the GPUs closest to the bottom of the cage) were still at above 90% survival rate, and the highest nodes were over 60%. It’s possible that newer Nvidia GPUs are less reliable than older ones (they certainly draw more power), or that AI datacenters are under-cooled, or that something about LLM utilization is more stressful than the workloads that ran on traditional GPU datacenters. But this is at least circumstantial evidence that GPUs can survive under load for far longer than three years. This discussion is complicated by the fact that GPUs may have a short economic lifespan. Supposedly a B100 GPU draws twice as much power as an A100, but can do five times as much work. For some AI providers, that might mean that A100s are only worth running until they can be replaced with B100s (if you’re bottlenecked on electricity, you should spend it all on B100s and throw out your obsolete A100s). This is why the Titan supercomputer was decommissioned in favor of Summit: it could have continued to operate, but it was more profitable to spend the money and maintenance effort on newer hardware. It should be obvious that this doesn’t support the “inference will become more expensive when the bubble pops” argument. So long as A100s are profitable right now , cash-poor AI providers can continue profitably serving inference from them, even if there are more efficient options available for those with the capital to upgrade. On top of that, GPUs only represent one part of AI datacenter infrastructure spending. If your GPUs wear out, you don’t have to go and build an entirely new datacenter. About 30-50% of datacenter spend goes to land, power, cooling, and so on. The remaining 50-70% is the cost of the entire server rack, which includes a bunch of things that aren’t GPUs. Like the idea that AI inference requires using huge amounts of water , the idea that AI GPUs only live a year or two is popular because it’s a useful idea for AI skeptics, not because it’s true. It comes from a pseudonymous tweet quoting an anonymous source who’s being paid hundreds of dollars to sound like a credible expert on AI. Other public communications from AI inference providers cite much higher lifespan numbers, and the statistics from supercomputers (the traditional examples of large GPU clusters) don’t bear out the claim that the maximum lifespan is three years. It might be true that the economic lifespan is three years, in a world where new GPUs come out every eighteen months and GPU providers are flush with cash to upgrade, but that doesn’t tell us much about the economics of inference in an AI winter. If money becomes a lot more scarce, it’s likely that AI datacenters will continue profitably 4 running their B300s (or their H100s or even A100s) for six years or longer. Of course, like previous claims about AI and water usage, “three years at the most” is often cited as “1-2 years, with some lasting up to 3 years under optimal conditions” . Of course, pronouncements from CEOs/CTOs should be taken with a grain of salt as well (for instance, maybe they have a big backlog of unused A100s they keep swapping out), but (a) executives don’t often straight-up lie about concrete technical facts, and (b) they’re going up against an unsourced quote from a tweet, so the bar isn’t that high. What about proactive GPU replacement? In the “Survival Analysis” section, the study attempts to account for this. I haven’t dug into exactly how. Assuming inference is profitable, which I believe (when you’re not attempting to amortize the cost of training). Of course, like previous claims about AI and water usage, “three years at the most” is often cited as “1-2 years, with some lasting up to 3 years under optimal conditions” . ↩ Of course, pronouncements from CEOs/CTOs should be taken with a grain of salt as well (for instance, maybe they have a big backlog of unused A100s they keep swapping out), but (a) executives don’t often straight-up lie about concrete technical facts, and (b) they’re going up against an unsourced quote from a tweet, so the bar isn’t that high. ↩ What about proactive GPU replacement? In the “Survival Analysis” section, the study attempts to account for this. I haven’t dug into exactly how. ↩ Assuming inference is profitable, which I believe (when you’re not attempting to amortize the cost of training). ↩

0 views
Sean Goedecke 1 months ago

Doing nothing at work

Many engineers should be doing less work. I don’t necessarily mean producing less code or fewer changes, but literally working fewer hours in the day. When they do work, they should be working at a slower pace. I like to aim to be running at 80% utilization by default: unless I have a high-pressure project going on, I spend 20% of my workday away from the computer. Why? Performance at tech companies is dominated by outlier events . When I think about the most impactful changes I’ve made, many of them involved a surprisingly trivial amount of work. There are no points for effort in software development. What matters is solving the right problem at the right time. In large engineering organizations, there are usually trivial pieces of engineering work you could do that would make tens or hundreds of millions of dollars for the company. Here are three common examples: First, when the company is trying to sign a big enterprise deal, stepping in with a feature or bugfix can make the deal happen. It doesn’t even have to be a good feature: sometimes just showing that you’re willing and able to make a concrete change will be enough. Second, preventing or mitigating an incident early (even by just knowing the right feature flag to turn off) can save huge amounts of money: both immediate lost revenue during the incident and future lost revenue from customers who would have pulled their business or refused to sign pending contracts. Third, when the company is trying to ship a high-profile feature, success or failure often hinges on trivial but obscure changes (e.g. the ability to rapidly add a new field in user settings, or to update the crufty enterprise-data-export functionality nobody has touched in years). Familiarity with the system can be the difference between one of these changes taking a few hours or a whole week. What do these examples have in common? They’re all time-dependent . You can’t just log on in the morning and decide to unblock a big deal, or mitigate an incident, or speed up a high-profile feature. Is it just a matter of being in the right place at the right time? Not quite. You also have to not already be busy. I wrote about this a couple of years ago in Crushing JIRA tickets is a party trick, not a path to impact . If you’re always 100% utilized on a steady stream of low-priority work (for instance, if you’re just picking up tickets from the backlog, crushing them, then picking up the next one), you’ll miss your chance to do high-impact work in two ways. First, you’ll be too busy to even notice the opportunities. You won’t be chatting with people who are working on other things, or reading team updates, or keeping an eye on ongoing incidents. So you’ll miss out on the best way to get involved in high-impact work, which is to volunteer your expertise. Second, if you perpetually look busy, your manager won’t want to volunteer for you. This is the second-best way to get involved in high-impact work: to have your manager or product manager say “oh, Sean has capacity to help out here, let me tag him in”. Why is this better? Because managers and product managers usually have a much better read on what high-impact work is going on. They’re in meetings that you aren’t in. If you’re supposed to keep your time free for high-impact work, and you’re not supposed to just grind tickets, what should you be doing on a minute-by-minute basis? Should you just be doing nothing? Yep! Doing nothing is good, actually. Software engineering can be a stressful job, but it’s typically not consistently stressful: the stress comes from the occasional incident, or high-pressure urgent piece of work, or (these days) layoff. If you approach the comparatively low-pressure parts of your work with urgent intensity, you’ll already be exhausted and frazzled when you have to handle the high-pressure parts. Even in high-pressure parts of the job, doing nothing can still be good. One thing I recommend for engineers new to on-call is to avoid rushing: take a few breaths before joining the call or before speaking, and in general try to “think in slow motion” . Most incidents resolve on their own. Most frantic “maybe this will help” changes during incidents make things worse, not better. As a general rule, if you can simply avoid panicking, you will be doing better than most engineers at incident response. Nothing is a space things can happen in 1 . If you give your brain a chance to rest, you will find you’re more likely to have new ideas. If someone hands you an important task, you can tackle it with your full attention (instead of juggling it with the three other things you’re working on in the background). When you’re not busy, you have time to just look at things and take in new data. A lot of engineers are uncomfortable seeing a task that needs doing and not doing it. I’m like this as well. I wrote about it in I’m addicted to being useful : it’s a psychological quirk that many software engineers share, because having that quirk (to a point) makes you a good fit for the job. In order to spend time doing nothing, sometimes you need to force yourself to not step in. For instance, I believe that engineers should generally avoid glue work 2 . Most glue work - making sure people talk to each other, updating docs for work you’re not leading, volunteering to address technical debt - reflects the fact that the organization is not explicitly prioritizing this work. If they were, you wouldn’t need to volunteer for it. Either that’s fine, or it’s a big mistake. If it’s fine, then you shouldn’t step up and do it: you’ll be wasting your time and annoying your manager. If it’s a big mistake, you still shouldn’t do it , because you’ll be insulating the company from the consequences of its own mistakes at the cost of your own career and mental well-being. That’s a bad deal for you, and a bad example for your junior colleagues, and sets a bad precedent for someone else to jump into the same position when you inevitably burn out 3 . If the consequences truly are severe, let them happen, so the organization can feel the pain and change its policies. I also believe that being too helpful leaves you vulnerable to predators . Tech companies are full of people who want to extract uncompensated work from software engineers 4 . This is different from work that arrives via normal channels, and for which you’re compensated by promotions, bonuses (and just your normal salary). I’m talking about work that arrives via backchannels, from people who don’t have the ability or willingness to ensure that work is formally recorded under your name. For instance, a product manager from another organization messaging you to say “you’re so good at querying data, would you mind pulling some statistics for me about X?”, or an engineer from another team asking you to “pair” on a piece of work that will ultimately involve you writing all the code and them quietly submitting the change under their own name. Doing some amount of this kind of work is fine. You may as well help people out when you can. But you need to be able to apply backpressure, either by saying no or simply delaying your response by a few hours or days. It’s also a good idea to avoid investing too much in work that is likely going to disappear . For instance, suppose you’re working with a product designer who is figuring out what they want in real time. At 9am they message you saying they want the page header to look one way, then at 10am they have tweaks, and more changes at 11am, and so on. You should not throw yourself into fully rewriting the page every hour. Instead, you should do nothing (say, go for a walk) and rewrite the page once in the afternoon, based on the most recent design. Another common instance of this is “big idea from a manager without the political clout to follow through on it”. Often you can just run out the clock until the project gets inevitably cancelled 5 . A lot of software engineering advice and tooling is designed around the ability to scale up your ability to exert technical effort: to do more things at the same time, to take on projects of larger scope, or to just write more code. But software engineering success is not determined by any of these. It is determined by the ability to do the right things at the right time, which requires that you deliberately hold back some of your effort during ordinary work. In my experience, it’s still possible to be a “high performing engineer” at 80% effort. In fact, it’s easier , because you’ll be less likely to make silly mistakes from stress, and you’ll be in a position to jump on the kind of high-impact tasks that deliver outsized returns. This doesn’t mean you should never grind at 100% effort. I think there are probably two or three times a year where I work as hard as I possibly can: long hours, intense focus, thinking about the problem from when I wake up to when I go to bed. But I reserve this mode of work for when the rewards are really high . For the rest of the year, I take it relatively easy. edit: this post got some comments on Hacker News . Commenters discuss how to not get in trouble with your manager when you’re taking slack time (in my experience, if you’re generally productive it’s fine, but managers vary a lot) and whether engineers really do have control over their workload. One of my big influences is Rich Hickey’s talk Hammock Driven Development . This is kind of like what he’s talking about, except (a) Hickey is more talking about what it takes to design solutions to really hard problems, rather than what it takes to be a strong engineer in an ordinary tech company, and so (b) Hickey recommends using your time-away-from-the-computer to focus on a hard problem, instead of to simply decompress and let solutions congeal in your head. It’s also like Zvi Mowshowitz’s post on “slack” . I wrote about this a lot more in Glue work considered harmful . Why inevitably? Because in my view, burnout is hard work unrewarded , and taking on a personal crusade that your job doesn’t care about is a great way to do a lot of unrewarded work. I wrote about this in Protecting your time from predators in large tech companies . Of course, you have to be careful with this. If you try this strategy and you’re wrong about the level of political support for the project, you will come off like a slacker and then have to deliver in a rush. One of my big influences is Rich Hickey’s talk Hammock Driven Development . This is kind of like what he’s talking about, except (a) Hickey is more talking about what it takes to design solutions to really hard problems, rather than what it takes to be a strong engineer in an ordinary tech company, and so (b) Hickey recommends using your time-away-from-the-computer to focus on a hard problem, instead of to simply decompress and let solutions congeal in your head. It’s also like Zvi Mowshowitz’s post on “slack” . ↩ I wrote about this a lot more in Glue work considered harmful . ↩ Why inevitably? Because in my view, burnout is hard work unrewarded , and taking on a personal crusade that your job doesn’t care about is a great way to do a lot of unrewarded work. ↩ I wrote about this in Protecting your time from predators in large tech companies . ↩ Of course, you have to be careful with this. If you try this strategy and you’re wrong about the level of political support for the project, you will come off like a slacker and then have to deliver in a rush. ↩

0 views
Sean Goedecke 1 months ago

Working with product managers

The relationship engineers have with product management is more dysfunctional than with any other part of the company. There’s no shared culture or language like there is with other engineers, and the rules of “who gets to tell who what to do” aren’t as clear-cut as they are with managers. Engineers don’t have a lot in common with legal, or design, or sales, but they also don’t need to interact much with those roles. In my experience, engineers are communicating with product managers almost every single day. The worst version of the product/engineering relationship goes something like this: Engineers are technically competent but are too autistic to be fully trusted. They need a kind-but-stern parental figure who knows how to communicate to other stakeholders in the organization (for instance, by being comfortable using the word “stakeholders”), and how to keep engineers from going off in the wrong direction. This entire gross dynamic is neatly captured by the popular term “product mommy” 1 . I really, really don’t like that term, or this entire dynamic in general. Almost none of my relationships with my product managers have been anything like this, though I have seen it at a distance. Working well with product managers can be the difference between succeeding and failing at a company. Why is it so hard to maintain good relationships between engineering and product? What does a good relationship look like? Product managers and engineers have largely non-overlapping skillsets. Product managers don’t understand the technical work engineers do and aren’t equipped to talk about it: if an engineer gives a technical reason for something, product managers generally have to shrug and say “sure, I guess”. Likewise, engineers don’t have anything like the visibility into the organization that product managers do. Particularly in large organizations, it is the product manager who is the source of truth about who wants what and which features are important. When a product manager says that something is critical, engineers generally have to shrug and say “sure, I guess”. This obviously requires a lot of trust. What’s a little less obvious is that this trust is continually broken by both sides . Every single product manager has been told thousands of times that technical task X is technically impossible or would be disastrous, only for that task to end up being done fairly smoothly and successfully. Every single engineer has been told thousands of times that requirement X is absolutely critical and worth going to enormous effort for, only for that requirement to be silently dropped or changed with no apology. Of course this isn’t malicious. Engineers often give wrong estimates because estimation is impossible , and sometimes the dire consequences they warn about really do happen (they’re just handled behind the scenes, like engineers handle many other kinds of technical dysfunction). Product managers “change their minds” because what’s important in a large tech company does genuinely change hour-by-hour 2 , and even the best attempts to only filter the most reliable priorities through to the engineering team will sometimes go wrong. The consequence of this broken trust is that the relationship becomes very difficult to maintain. When you’re an engineer, and you explain something to your product manager, and you know they don’t believe you (despite having no ability themselves to judge the question), it can be incredibly frustrating. Likewise, when you’re a product manager, and you’re desperately trying to explain what we need to do to an engineer, and you know they’re internally shrugging their shoulders, it must be unbearable. Don’t they know this is critical to the company? You were just in a meeting with the leaders of the organization! The natural tool for a mistrustful product manager is manipulation . I still remember a product manager who tried to extract a commitment from my team by asking us to go around and all say “I commit to getting this work done in two weeks”, after a conversation where we’d explained the risks that cause it to take longer. I suppose the idea was that we’d all work much harder, having taken a sacred oath? More subtle variants of this approach involve suggesting that you would be really disappointed if this work was delayed (in true “product mommy” style), or vaguely suggesting the possibility of some abstract reward (that the product manager is not empowered to deliver) if work gets done ahead of schedule. The natural tool for a mistrustful engineer is lies . The most benign version of this is exaggerating estimates: for instance, the classic advice to double your estimate and add 20% . I’ve seen engineers claim that they’ve had to follow up on all sorts of largely-fake tasks (one common example is “reaching out to a neighbor team to confirm X”) in order to gain more time. In the worst case, engineers might even straight-out lie that work has been completed, and then track the “it doesn’t work in production” feedback as a bug. Once this starts happening, it’s nearly impossible to repair the relationship. I can’t bring myself to trust a product manager who’s clearly trying to pull my strings, and I’m sure a product manager can’t trust an engineer who’s lied to their face in the past. That’s why it’s so important to avoid getting into a bad relationship in the first place. Why bother? If it’s so hard to hammer out a good working relationship with product managers, why not just settle for a bad one? Product managers can absolutely bury you if you’re not careful. Product managers are almost always more politically sophisticated than engineers. This is partly structural: product managers are simply in more conversations with the company’s movers and shakers, and so naturally have a better relationship with them (and are thus better attuned to which way the wind is blowing). It’s also partly selection bias: engineers can be hired even with relatively poor social skills, because they’re primarily being assessed on technical ability, but social skills are a core part of the product role 3 . If you are feuding with a product manager, you will probably lose . Unless you’re unusually influential, they will simply have far more opportunities to quietly talk you down in influential circles than you will. All it takes is a few comments like “oh, I probably wouldn’t pick Sean for that project” to wreck your reputation. In the case where you are openly feuding with a product manager, the company’s leaders will by default take the product manager’s side over yours. They’re likely to know them better, have more shared cultural context with them, and in general be willing to interpret the situation as “another engineer who doesn’t understand how the organization works”. There are huge benefits to being trusted by a product manager. Product managers want to ship things , and typically understand a fair amount about all of the non-technical barriers to shipping. If you also want to ship things, you can become a fearsome team. On top of that, because trust between engineers and product managers is so difficult, once you’re in you’re in all the way. Product managers often pick one or two engineers as their go-to for getting the “real story” on technical questions. If that’s you, you have an outsized position of influence in the organization, which you can use to get the things you want done . As an engineer, how can you build trust with your product manager? The first step is to understand where they’re coming from . When they tell you something is important or that a requirement has come in, be aware that this is rarely their decision. It’s not them who’s jerking you around, it’s someone higher up in the food chain jerking you both around. If you can adopt a conspiratorial mindset with them, instead of against them, that’s a good start. Try just asking “oh man, alright, what can we do about this?” instead of complaining. The second step is to be right, a lot . This is a silly-sounding Amazon leadership principle that turns out to be entirely accurate. I wrote more about it here , but (as unfair as it sounds) you really do have to be mostly accurate if you want to build trust with a product manager. When you say something will ship, it has to ship; when you say something is impossible, it can’t happen days or weeks later. It’s okay to be wrong sometimes , but you have to establish a pattern of you providing them useful, correct technical information. The third step is to let them make the political calls most of the time . If you expect them to trust your technical calls, you have to extend them the same trust when it comes to navigating the organization. Don’t publicly undermine them in meetings, bring up your concerns in private. If they say something is important and you’re not so sure, at least act like it is. Accept that sometimes they’re going to be wrong, just like you’re sometimes wrong about technical questions. The fourth step is to get lucky . Sometimes your product manager will just be a dud. You can’t build trust with someone incompetent: there’s nothing for you to trust them with, and they aren’t in a position where they can usefully extend trust to you. Working in large organizations requires getting comfortable with the fact that some of your colleagues will be stronger than others, and figuring out ways to work with (or bypass) people who make the work harder, not easier. Many product managers were once engineers. If your product manager is technical, does that make you immune from these problems? Absolutely not! You likely won’t have much choice in which product managers you work with, but be aware that having once been an engineer is a negative , not a positive. No product manager can ever be technical enough to matter, because they don’t work on the codebase : even if they were a full-time engineer, they wouldn’t have the time to build the specific context on the system they’d need to be a real participant in technical discussions. It’s thus better to have a product manager who knows they’re not technical than to have one who mistakenly thinks they might be. The worst-case scenario is an ex-engineering product manager who believes they’re technical enough to detect when engineers are lying to them. This kind of paranoia is an easy trap for “technical” product managers to fall into, particularly when they don’t have a trusted engineer on the team they can lean on. If you’re dealing with one of these, prepare to spend a lot of time explaining why you can’t “just” do things (and prepare to have those explanations not be believed). At its worst, a product manager relationship is like an unhealthy family: driven by condescension, emotional manipulation, lies, and mistrust. This isn’t because product managers are bad people! It’s because the structure of the relationship creates conflict. Both sides must make commitments (about the technical system or goals of the organization) that are (a) often wrong, and that (b) the other side is unable to independently verify. To avoid the trap, both sides have to be generous, willing to trust each other in their areas of expertise, and most importantly competent . Unlike most roles in tech, product management (particularly the lower-level roles that are more engineer-facing) has close to an even gender split. For instance, based on the whims (or snap decisions, more charitably) of the CEO. I have worked with product managers with poor social skills, but it’s rare: about as rare as working with engineers with genuinely poor (i.e. by general-population standards) technical skills. Unlike most roles in tech, product management (particularly the lower-level roles that are more engineer-facing) has close to an even gender split. ↩ For instance, based on the whims (or snap decisions, more charitably) of the CEO. ↩ I have worked with product managers with poor social skills, but it’s rare: about as rare as working with engineers with genuinely poor (i.e. by general-population standards) technical skills. ↩

0 views
Sean Goedecke 1 months ago

Anti-AI nostalgia and the cult of the past

Programmers were better back in the day, weren’t they? Back when we had real programmers. Not just people who got paid to write code, but people who lived it, who were obsessed with their craft, and whose code was a lively expression of themselves. Hackers were hackers in those days before money took over the industry. Don’t even get me started on LLMs. Could there be a better example of today’s degenerate spirit? A machine to mass-produce software (not good software, just barely good enough), so that the weak minds that dominate the industry can indulge their obsession with quantity : of slop code, of features, and ultimately of money, which is the only way they can understand value. If they weren’t destroying our way of life, they would be pitiable. All of them together don’t have a fraction of the spiritual integrity of someone like Mel . But as it is, we must band together to crush them and drive them from our industry like the parasites they are. Okay, that’s not actually what I believe. But there sure are a lot of posts 1 and comments on the internet that sound a bit like the paragraph above. Here are some older quotes that might sound similar: …the third collapse, in which power tends to pass into the hands of the lowest of the traditional castes, the caste of the beasts of burden and the standardized individuals. The result of this transfer of power was a reduction of horizon and value to the plane of matter, the machine, and the reign of quantity. 2 Usura rusteth the chisel \ It rusteth the craft and the craftsman \ It gnaweth the thread in the loom 3 The actual accomplishments of the past will nevertheless remain accomplishments, while the artistic stammerings of the painting, music, sculpture, and architecture produced by these types of charlatans will one day be nothing but proof of the magnitude of a nation’s downfall. 4 These are all from the writings (or speeches) of famous fascists: Julius Evola, Ezra Pound, and Hitler himself. Mussolini’s Doctrine of Fascism begins by defining fascism as a “spiritual attitude”, which the fascist man adopts in order to regain the mysterious qualities that were lost by the transition to modern life. In his classic Ur-Fascism , Umberto Eco’s first two defining features of fascism are the “cult of tradition” and the “rejection of modernism”. So when someone tells me that the industry has lost its way and we must deny the corrupting influence of modern technology in order to retvrn to the time of virile real programmers (who understood and appreciated the spiritual dimension of programming), I get suspicious. It’s strange to describe anti-AI sentiment as potentially fascist, since a very popular argument is that LLMs themselves are an inherently fascist tool. Surely both sides of the debate can’t be fascist? I do think that the structure of fascist arguments is generally persuasive , and that many avowedly anti-fascist groups do sometimes fall into this trap: describing the world as a struggle between the spiritual power of the macho, traditional man and the corrupting influence of degenerate (often foreign) capital. For instance, I am a big fan of Lord of the Rings. I’ve read the series and watched the films multiple times, and even made a failed attempt to learn Elvish as a kid. But it’s hard to deny that fascists absolutely love Lord of the Rings. “Marble statue of a Roman emperor” might be the most popular avatar for fascists on the internet, but Aragorn is the second most popular. Neo-fascist movements in Italy explicitly take up Lord of the Rings as a foundational text. Why? Because the core conflict in the text is between the traditional, nostalgic heroism of the Shire and Gondor, and the corrupting modern industrial (partly foreign ) influence of Saruman and Sauron 5 . I don’t think Lord of the Rings (or anti-AI rhetoric) is intrinsically fascist. In fact, the surface-level reading of the text is anti-fascist: the plucky people of the West banding together to fight Sauron’s command-and-control totalitarian society. But I can see why fascists love it. One common historical touch-point for anti-AI folks is the Luddites, who were a violent conservative labor movement in early 1800s England. Anti-AI blogs adopt Luddite language like “smashing frames”, and positively cite the Luddites as “the go-to enemies of fascism since its inception”. I’ve written at length about what we can learn from the Luddites in Luddites and burning down AI datacenters , but one point I think is under-emphasized by the (generally pro-Luddite) books is that the Luddites were a little bit fascist themselves . Brian Merchant’s Blood in the Machine is the most popular recent book on the Luddites. I enjoyed it, but Merchant’s attempts to paint the Luddites as a friendly, left-wing, proto-feminist movement 6 seemed really unconvincing to me. From the writings of the Luddites, it’s clear that they were interested in protecting the rights of their all-male elite guild fraternity. Here’s one Luddite threat to a workshop that explicitly includes a threat against the female workers 7 : We think it quite inconsistent with our duty as men, as husbands and as fathers to suffer ourselves to be ruined any longer by a set of vagabond strumpets and those gibbet-deserving rascals that are looking over them. We will lead them to their satisfaction. We sincerely hope, gentlemen, that you will discharge the bitches and take men into your employ again, or they must take what they get. These were fundamentally conservative people who felt (correctly) that modernity had deprived them of their elite status, handing it instead to lower-paid inferiors: women, vagabonds, and foreigners. The Luddites were obviously not fascists 8 . However, the basic ingredients were there: wounded pride, a masculine elite identity, hatred of modern economics, and violence aimed at restoring their previous position in society. The currents that produced Luddism are the same currents that guided so many unhappy people towards fascism. When things are looking grim for an elite group, they often turn towards any movement that promises a return to an idealized past. If my blog has themes, one of them is surely that many software engineers labor under a delusion that their job is to be excellent at their craft. Of course, wanting to be an excellent programmer is not a delusion; it is a completely legitimate value to hold, and a legitimate purpose to pursue. It’s just not what you’re paid to do at work. Your job , unfortunately, is producing shareholder value . This delusion has been punctured by the end of ZIRP , and again more recently by the rise of AI coding. In this environment, I worry that some software engineers will form exactly the kind of disillusioned elite that was the audience for Ezra Pound’s poems about “usury” or the Luddites’ campaign against unapprenticed (often female) textile workers. I worry that AI, and the companies that build AI, are becoming an enemy against which anything is permitted: an enemy which in Umberto Eco’s words is “at the same time too strong and too weak”, unable to reason and yet powerful enough to drastically reshape the global labor market for the worse. The enemy of fascism is nuance. Fascism presents a good, clean, rousing story about a spiritual conflict between right and wrong. It is anathema to fascism to stop and muddy the waters a bit: in this case, to explore the ways in which LLMs, like any transformative technology, can both support and endanger traditional values. In The left-wing case for AI I wrote about how AI is being used right now as a disability aid, and many disabled readers wrote in to share their positive experiences with LLMs, and often how alienated they feel by the anti-AI mainstream on the left. I recently got an email describing how there’s a sudden flood of accessibility software for blind people 9 that’s actually built by blind people , who can now iterate with a LLM to get a product that meets their needs. Framing AI as an ontological evil erases experiences like these. Being anti-AI is not inherently fascist. Many of the anti-AI posts I’ve quoted are thoughtful, sensitive pieces exploring how the author thinks about one of the biggest changes to our industry. I still think the world needs more articles like that, not less, but the more of them I read, the more I recognize the tropes: spiritually pure lovers of the craft, degenerate peddlers of corrupt modernism, a need to return to the traditional ways of the hacker, and a lament for the (potentially) waning power of an elite fraternity of programmers. I know I’m tiptoeing around the worst argument in the world . It isn’t a refutation of anti-LLM arguments to say that they are structurally similar in some ways to fascist arguments, any more than it’s a devastating critique to say the same thing about Lord of the Rings. Sometimes it is good to try and halt the march of progress! Some of our past traditions really were purer and more spiritually robust! It just bothers me, that’s all. I used to read The Story of Mel with unalloyed pleasure. Now it makes me nervous. If you believe you’re fighting the embodiment of fascism , or for the idea of value itself , what tactics are off-limits? What positions might you eventually come to accept? It feels wrong to directly associate my caricature with any actual posts, but it also feels wrong to make a blanket assertion without examples. Just so you know what I’m talking about, here are some posts that have elements of this attitude. I like some of these posts and dislike others. Page 329 of my copy of Julius Evola’s Revolt Against the Modern World . Ezra Pound, Canto XLV. “Usura” should be read as “usury”, or today we could gloss it as “capitalism”: all Pound’s examples of great art were from the pre-capitalist patronage era of art. Adolf Hitler, from his speech at the 1933 Party Congress in Nuremberg. Of course, there’s also historically been a strong pro -technology current in fascist thinking (even specificially Italian fascist thinking ). Page 134 of Blood in the Machine has a brief argument that Luddism was feminist because the (exclusively male) artisans’ wives would provide food for their meetings. No, really. From Kevin Binfield’s Writings of the Luddites , page 40. I’ve taken the liberty of re-rendering it in modern spelling and grammar. Aside from being too early, they didn’t have any connection to the state apparatus of power (in fact, they were ultimately crushed by it) and they famously lacked a singular leader. The example cited was BlindRSS . It feels wrong to directly associate my caricature with any actual posts, but it also feels wrong to make a blanket assertion without examples. Just so you know what I’m talking about, here are some posts that have elements of this attitude. I like some of these posts and dislike others. ↩ Page 329 of my copy of Julius Evola’s Revolt Against the Modern World . ↩ Ezra Pound, Canto XLV. “Usura” should be read as “usury”, or today we could gloss it as “capitalism”: all Pound’s examples of great art were from the pre-capitalist patronage era of art. ↩ Adolf Hitler, from his speech at the 1933 Party Congress in Nuremberg. ↩ Of course, there’s also historically been a strong pro -technology current in fascist thinking (even specificially Italian fascist thinking ). ↩ Page 134 of Blood in the Machine has a brief argument that Luddism was feminist because the (exclusively male) artisans’ wives would provide food for their meetings. No, really. ↩ From Kevin Binfield’s Writings of the Luddites , page 40. I’ve taken the liberty of re-rendering it in modern spelling and grammar. ↩ Aside from being too early, they didn’t have any connection to the state apparatus of power (in fact, they were ultimately crushed by it) and they famously lacked a singular leader. ↩ The example cited was BlindRSS . ↩

0 views
Sean Goedecke 1 months ago

Weird projects I shipped with AI

Where are all the AI-generated projects? This is a common question from AI skeptics: if LLMs are so good at writing code, where is the tsunami of new AI-generated apps, services and games? I personally don’t find this to be much of a paradox. Writing code is only one of the bottlenecks involved in actually shipping a new product, after all. It’s also impossible to talk about the paid work I’ve done with AI (you’ll simply have to take my word that it’s increased my productivity). But one thing I can do is share a list of personal projects I’ve built with AI in the last twelve months. I definitely would not have done all of these by hand. I might have found the time to do one or two of them, but based on my pre-AI track record they would probably have stayed in the “GitHub repo with a few commits” stage. This list is a kind of existence proof : a bunch of weird projects, useful to at least some people, that would not have existed without AI assistance 0 . Most recently I’ve built skifreedle.com , a daily-game version of the classic Windows SkiFree game (i.e. “like Wordle, but for SkiFree”). The code for that is here 1 . I enjoy coding small web games by hand, but definitely would not have had the time to wire up all the different SkiFree objects or build neat features like a ghost of your fastest run. I also tried out a lot of different visual themes for the game UI before landing on something I liked. If I’d done this by hand, I would have only had time to try out two or three different looks, instead of fifteen or twenty. I’m very happy with how this turned out. I’ve been enjoying competing against my brother to get better times, since both of us have a lot of nostalgia for the original SkiFree game. Last year I built Autodeck ! I wrote a blog post about this before, but this came from my partner wishing there was some way to automatically generate Anki cards about random topics she wanted to learn about. It ended up being relatively straightforward to set up an endless feed of auto-generated spaced repetition cards: I set up Stripe payments for this one, more because I was worried about someone running away with my Groq balance than because I wanted to make money, but I was pleasantly surprised to see a bunch of people actually use this. Over five hundred people have tried it out, with enough paid subscribers to cover inference and hosting. I might have built this without LLM assistance, but I almost certainly would not have deployed it as a website. The hassle of setting up a database and Stripe would have just been too much work. I also built an AI-generated endless wiki . I wrote a blog post about this one as well. Like Autodeck, I was fascinated with the idea of non-chat interfaces for LLMs, and I thought a wiki-based approach where you interact with the model by clicking links was pretty cool. I learned the hard way that putting a LLM generation call on the end of a regular link was a bad idea: scrapers would exhaust my inference budget quickly. I ended up faking the no-article-exists-yet links with JavaScript, which at least so far has defeated scrapers. People still email me about Endless Wiki, and there are over 280 thousand pages generated. My original goal was to see if you could eventually generate a page for Neon Genesis Evangelion, starting at the root page and only following links (kind of like wiki golf ). I was successful! You can read the “Evangelion Anime” page here . Almost exactly a month after I launched Endless Wiki, xAI launched Grokipedia . Obviously they didn’t plagiarize me. This is a very easy idea to have, and my site was not the first infinite wiki (though I think it was the first one where you had to discover new pages by clicking on links). But it did take some of the shine off. I built a PWA that caches the VicFlora plant identification database so it could be used with low or no internet. This was more of a utility project for my partner, who likes plants and occasionally goes on field trips where internet is spotty. I would definitely not have done this without LLMs. It was reasonably difficult to scrape the basic dichotomous key from the VicFlora website: their API documentation was out of date, there were multiple possible pathways for fetching data (most of which were not functional), and the format of the data I did manage to fetch was hard to parse. I think I could have done it, with enough effort, but it would have been a substantial amount of work. I’m very happy with how this turned out. It’s not perfect, but it’s functional, and I’ve even had the occasional Victorian botanist email me with bug reports or feature requests, so it’s clearly seeing a little bit of usage. I did a bunch of other stuff that doesn’t necessarily rise to the level of a “deployed project”: my gh-standup GitHub CLI extension to automatically generate a standup report, which has just over a hundred stars, my (low quality) image geolocation benchmark , which I blogged about here , or my skill for extracting features from open-source models. There may not be a flood of AI-generated companies (yet), but at least for me there’s been a flood of small, weird projects that would not have existed without significant LLM assistance. I also want to shout out Simon Willison’s version of this , which is another great example of “weird useful tools that only exist because the cost of creating them was so low”. I did lift the spritesheet from DanielHough’s SkiFree.js , which attributes it to Wing Wang Wao . Of course, the original sprites and art belong to Chris Pirih’s SkiFree and Microsoft. I also want to shout out Simon Willison’s version of this , which is another great example of “weird useful tools that only exist because the cost of creating them was so low”. ↩ I did lift the spritesheet from DanielHough’s SkiFree.js , which attributes it to Wing Wang Wao . Of course, the original sprites and art belong to Chris Pirih’s SkiFree and Microsoft. ↩

0 views
Sean Goedecke 1 months ago

The famous o3 "GeoGuessr" prompt did not work

In April last year, Kelsey Piper discovered that OpenAI’s o3 model was surprisingly good at figuring out where a photo was taken from. Like human “geoguessr” pros , o3 could sometimes take a nondescript photo of a beach and tell you exactly where it is. Here’s the example Kelsey gave: Several people reproduced this with good results: not a 100% success rate, but clearly far better than you’d do with a random human guess. The lesson here is that model capabilities can surprise us . The o3 model had been released for two weeks before Kelsey’s tweet without anyone noticing how good it was at geolocation. What obscure capabilities did we never find? What capabilities of current models are we missing today? Some people drew another lesson from this: that “prompt engineering” can unlock brand-new capabilities. This is because Kelsey had a magic prompt that she built over time. When o3 got something wrong, she would ask it how it could have avoided the mistake, and then included that in the prompt. Here’s the first 10% of that prompt, so you get the idea: You are playing a one-round game of GeoGuessr. Your task: from a single still image, infer the most likely real-world location. Note that unlike in the GeoGuessr game, there is no guarantee that these images are taken somewhere Google’s Streetview car can reach: they are user submissions to test your image-finding savvy. Private land, someone’s backyard, or an offroad adventure are all real possibilities (though many images are findable on streetview). Be aware of your own strengths and weaknesses: following this protocol, you usually nail the continent and country… This prompt impressed a lot of people, who tried it out and reported that it correctly identified a lot of images. But of course, o3 correctly identified a lot of images with just a basic “think carefully about where this picture was taken?” prompt. Did the prompt actually help? It’d be tough to figure that out just from playing around in ChatGPT. You’d need to build an evaluation set of images and run o3 against them twice: once with the fancy prompt and once without it. So that’s what I did . I pulled 200 images from Wikimedia Commons, Geograph Britain and Ireland, and iNaturalist for the benchmark. You can read the AI-generated summary here , but here’s the key table: In general, the basic prompt did better on average. It consistently guessed closer to the actual location. Both prompts did pretty well, actually. Despite the fancy prompt being 10x larger, it only caused o3 to think for slightly longer (about one second on average, though the max was about double, at 10 minutes instead of 5 minutes). The images in my benchmark were fairly generic geoguessr-style outdoor images, with twelve indoor images thrown in for an extra challenge (the fancy prompt also did slightly worse on these). What’s going on? I think this shows how easy it is to fool yourself about the quality of prompting . When the model is already pretty good at a task, you can give it a very elaborate prompt without impacting performance. It’ll still be pretty good, except this time it’s good because of what you did . This is particularly true if you’re iterating with the model and asking it “what should I add to the prompt” for each mistake. Models will happily make up stories for you about their own reasoning processes, and will almost always say “yes, that helped a lot!” when you ask them if a particular prompt tweak made things better. The only way to actually know is by constructing some kind of benchmark 1 . It’s also interesting to me that nobody checked this at the time. It took me about six hours of fairly-distracted work and about $15 to construct and run this benchmark. Why didn’t anyone do this when they were writing articles about how good the o3 prompt was? One charitable reason might be that the story was more about o3’s real geolocation ability than about the magic prompt. The pricing for o3 also used to be about five times more expensive (though a benchmark of 40 images instead of 200 would still have thrown doubt on how much water the prompt was carrying). Also, AI just moves so fast . Geolocation was only the story for about a week: after that, GPT-4o’s sycophancy was what people were talking about. Another reason is that AI tooling wasn’t as good then. The benchmark was so easy for me to run because GPT-5.5 did most of the heavy lifting. Prior to strong agents, you would have had to write the (simple) benchmark yourself. I can’t point the finger too hard: I didn’t bother at the time either. Maybe my benchmark isn’t very good? The photos look reasonable enough: a wide variety of geoguessr-like shots of roads and landscapes, mostly. I could have tried to gather a few thousand photos instead of a few hundred, but if the magic prompt really was a big improvement you’d still expect to see that manifest on a benchmark this size. If someone wants to go and build a hundred-dollar geolocation benchmark instead of my fifteen-dollar one, I think that’d be an interesting project. Finally, let’s use the benchmark to answer a question I’ve had for a while: do gpt-5.4 and gpt-5.5 have o3’s geolocation abilities? The answer, apparently, is no. Whatever o3 had that made it good at this task hasn’t transferred to newer models. Benchmarks can mislead as well, but they’re better than just vibes. Benchmarks can mislead as well, but they’re better than just vibes. ↩

0 views
Sean Goedecke 1 months ago

Prompts are technical debt too

It’s common and correct to say that “all code is technical debt”. Adding code is a necessary evil for developing new features: you almost always have to do it, but each line of code adds to the complexity and maintenance burden of the system. All future changes to the system have to work with the existing code, or at least avoid breaking it. Once systems accumulate enough code, they become impossible for a single person to understand: instead of reading the code and understanding what it does, you must rely on guesses, theories and heuristics 1 . Sensible engineers write as little code as possible. They write a lot of prompts, though! Many large projects now have a set of codebase-specific prompt files: AGENTS.md, CLAUDE.md, those same files in sub-directories, and skills . If you’re building a program that uses AI 2 , you’ll have separate prompts for capabilities and for each tool , as well as a whole set of system prompts . Prompts are important. Minor tweaks to a LLM’s prompt can unlock significant performance improvements. If the same model feels different across Codex, Cursor, OpenCode, and Copilot, it’s almost certainly due to subtle differences in prompting. AI companies spend a lot of time testing and tweaking their prompts, so it makes sense why engineers would spend a lot of time tweaking their AGENTS.md files 3 for their projects. I’d even call switching tools or workflows to be a form of prompting. If I start wrapping my agents in a Ralph loop , pull in a new skill file, or install an MCP server, that’s still a change to my prompts even though I’m not the one who wrote it. I think it is a bad idea to spend a ton of time tweaking a bespoke agentic coding setup. Why is that, given that prompt adjustments can deliver a lot of value? Because prompt adjustments are model-specific . Earlier I said that AI companies spend a lot of time tweaking their prompts. In fact, they spend that amount of time for each new model release. A prompt that worked great for GPT-5.4 won’t necessarily work as well for GPT-5.5. You have to “learn how to hold the model” each time. In other words, a set of prompts that you carefully crafted in January this year might be out of date or actively harmful by February. Worse still, you might not even notice. Model capabilities are already so hard to pin down (unless you’re running every problem through different models and tools), and even weak AI systems are surprisingly good at some problems. You might just think “huh, the new Anthropic model isn’t as impressive as the hype”, or “wow, Claude Code has gotten worse recently”. In this sense, prompts are a worse form of technical debt than code . When technical debt blows up, it usually causes errors or a tangible slowdown as you try to understand the code. Prompts will decay silently. Also, even janky code tends to be relatively stable when untouched, but every single model upgrade could turn a functional prompt into a non-functional one. Could you simply decide not to upgrade models? Some people are trying this, but the pace of improvement is fast enough that that isn’t really practical. A delicately-prompted agentic harness built around GPT-4.1 is always going to underperform a bare-bones harness built around Opus 4.7. This might be a sensible strategy at some point in the future, when the rate of model improvement slows down (or when models are so capable that you don’t need the extra intelligence for normal engineering tasks), but I don’t believe it’s a good strategy today. In my view, most people should just be picking an AI coding tool maintained by a third-party company (Claude Code, Codex, Cursor, Copilot, etc) and leaving it as unconfigured as possible, so they can piggyback on the work of teams of engineers who are evaluating and tweaking prompts with each new model. Avoid MCP and skills unless absolutely necessary, and keep them off by default. At least this way if one of those teams gets it badly wrong, users will notice eventually and complain about it. When you write AGENTS.md files, try to avoid behavior steering (like the now-outdated “think step by step”, “you are a skilled engineer”, or “if you get a task right I will tip you $200”). Keep them limited to specific, concrete facts about the project. Don’t let models fill your AGENTS.md with pages of barely-reviewed text, for the same reason that you wouldn’t let them fill your codebase with pages of barely-reviewed code. Write your prompts yourself, and delete them whenever you get the chance. Almost every system you might get paid to work on is in this category (if not in the code of the system itself, then in its dependencies and libraries). Instead of just using AI to build a program. This distinction was a real pain when I was working on GitHub Models . Almost every system you might get paid to work on is in this category (if not in the code of the system itself, then in its dependencies and libraries). ↩ Instead of just using AI to build a program. This distinction was a real pain when I was working on GitHub Models . ↩

0 views
Sean Goedecke 1 months ago

The just-say-no engineer was a ZIRP phenomenon

The engineer who says no all the time is a real archetype among senior and staff engineers. Their role is to slow things down, to block the development of features that add complexity, and to ensure that as little code gets written as possible (since code is a liability). We can think of this as the just-say-no engineer 1 , as opposed to the just-say-yes engineer. The just-say-yes engineer is obsessed with moving fast, approves code changes by default, values MTTR over MTBF , and tends to ship a lot of code. The just-say-no engineer is obsessed with quality, is happy to move slowly, and blocks code changes by default. Most engineers are somewhere in the middle of the spectrum. By “just-say-no engineer”, I’m talking about the group of engineers who most strongly identify with that archetype. The just-say-no engineer is having a hard time in the era of AI. It used to be that they only had to say no to more junior engineers’ handwritten PRs, but now they have to say no to a barrage of AI-generated code, some of it generated by managers and VPs who are politically difficult to say no to. For the first time in their careers, they’re under a lot of pressure to lower their standards and start saying yes. However, this isn’t because of AI. It’s because of the end of ZIRP. ZIRP, or the “zero interest rate policy”, is a shorthand for the era of software development between 2008 and 2022 when banks were allowing companies to borrow money at near-zero interest rates. During this period, investors were throwing borrowed money at anything , which meant that tech companies were incentivized to constantly hire engineers for low-risk high-reward projects 2 . Successful companies would routinely grow from tens of engineers to thousands, who would go and work on all kinds of things: tangential open-source projects, endless technology migrations, rewrites into other languages, and so on. It was a great time to be a software engineer. We had a lot of bargaining power, and could get paid top dollar to do almost anything. The bosses largely didn’t care, because (a) teams were growing so fast they couldn’t pay attention, and (b) just having more engineers around was beneficial to the stock price, which was the main thing they cared about. But tech companies did have one problem: with so many engineers running wild, how would they keep their systems from becoming completely unmanageable? Enter the just-say-no engineer. In this environment, having a very senior engineer whose only job is to say no to things was actually quite valuable to the company. There are a few reasons for this: When banks hiked interest rates, almost every tech company immediately laid off 5-20% of their engineers. It was just no longer profitable to keep a bloated engineering staff around to boost the stock price. Instead, companies had to actually make money 3 . However, that wasn’t a good public explanation for the layoffs, since it sounds weak to admit that you were paying hundreds of engineers to do unprofitable work. Fortunately, the end of ZIRP coincided roughly with the rise of ChatGPT, so tech companies were able to to blame their layoffs on the power of AI. Saying “with this transformative new technology, we’re able to deliver 10x the value with half the engineers” is a much stronger message, even though it doesn’t make much sense (if this is true, why not keep your engineers and deliver 20x the value?) Something like this dynamic has been happening to the just-say-no engineer. Tech companies are now more focused than at any time in the past two decades. They are not doing a bunch of random crap anymore; instead they’re desperately chasing new capabilities and features that can make money (mostly built on AI, for obvious reasons). This new environment is actively inimical to the just-say-no engineer. It’s as if a shark got pulled out of the deep ocean and dropped into a fast-flowing river: what was once a powerful apex predator is now disoriented and flailing. This kind of engineer used to enjoy implicit (albeit distant) support from their management. If someone complained, they’d often get told “that engineer knows what they’re doing, if they said no, then I trust them”. Now that support is gone. The just-say-no engineer is now being criticized and actively overruled by their management. They’re being told to be more of a team player, to find a way to say yes, or are simply no longer being consulted (with the company’s blessing) on key decisions. They’re getting bad reviews for the exact same behavior that’s been rewarded pre-2022 4 . None of this depends upon AI. If LLMs had not taken off this decade, we would still be seeing the same cultural shifts in the industry. Companies would still be laying off engineers, and the engineers whose job has been to say no to things would still be upset and confused about why they’re now being punished for saying no. Ironically, if ZIRP had not ended, this would be a glorious moment for the just-say-no engineers. LLMs would have thrown fuel on the “engineers running wild” problem that the just-say-no engineers were empowered to solve. Tech companies, unable to publicly or privately cast doubt on AI-assisted coding 5 , would have relied heavily on these engineers to prevent the tsunami of AI code from swamping the entire company. They would have been paid even better and celebrated like kings. Instead, LLMs are adding insult to injury for the just-say-no engineer. They’re forced to watch while other engineers merge AI-generated PRs that would previously have been blocked, and are told to use the tools themselves: to become the kind of engineer they’ve spent their entire careers battling against. Worse still, the AI tooling mostly works . It’s not (yet) causing any kind of catastrophe 6 . The code isn’t quite as clean, and it’s a bit less well-understood, but it’s good enough (particularly in a world where companies are trying lots of new things and abandoning the ones that fail). So the just-say-no engineer faces not just a threat to their livelihood, but to their entire self-identity: they have to either insist that the apocalypse is right around the corner, or accept that their technical role was contingent on a really weird economic environment in the tech industry. Will the just-say-no engineer go extinct? No. They don’t fit well into every single tech company anymore, but there are domains where they’re needed. In Pure and impure software engineering I drew a distinction between “pure” engineering, which has a well-scoped, largely technical goal (like building a compiler or a language runtime) and “impure engineering”, which has a poorly-scoped, largely customer-driven goal (like trying out a new feature you’re not sure will work). During the ZIRP era, tech companies did a lot more pure work (for instance, building React ), and tended to treat even impure work like pure work. The just-say-no engineer is great for pure work, because pure codebases have to have a much higher bar for quality and can tolerate slower development cycles. Most tech companies are still doing some kind of pure work, typically in their core infrastructure pieces. This is essential work, but it doesn’t require a huge engineering team, and it’s rarely in the spotlight . If you’re a just-say-no engineer and you want to stay that way, I would recommend trying to move into one of these roles (and accepting that you’ll have a more limited scope than you did in the 2010s). This was a critical role during ZIRP, because: Part of the appeal here is the lure of the guru. In kung fu films, those who know martial arts perform furious acrobatics, but the true expert barely needs to move at all. For the same reasons, it sounds profound to say something like “junior engineers produce tons of code, seniors very little, and staff engineers remove code”. Of course this is false. Staff engineers are expected to be able to produce a lot of working code very quickly, when they need to. I wrote about this a lot more in The good times in tech are over . Not necessarily make a profit , but at least bring in revenue. Or pre-2023, or even pre-2024 or 2025. Cultural change lags behind economic incentives, sometimes by several years. For fear of killing the vibe (and thus the stock price). If you think there have been more incidents recently, consider that (a) you might be wrong , or (b) that other end-of-ZIRP factors (like increased velocity or layoffs) might be primarily responsible. Having half of the company’s engineers enmeshed in an endless loop of proposing changes and being told no was totally fine - they didn’t need to be productive anyway, and this way they weren’t impacting business-critical systems. It also solved the problem of the 5% of engineers who would get drunk on their technical freedom and make wild proposals like migrating to a hand-rolled database. Having a reputation for a very high technical bar is a positive for hiring (and remember, during ZIRP every tech company was always hiring) Some senior and staff engineers operate as gatekeepers, slowing down development and saying no to most things This was a critical role during ZIRP, because: Tech companies had thousands of engineers who were empowered to do basically whatever they wanted, so without gatekeeping the systems would have fallen apart Tech companies didn’t care that much if they got anything done When ZIRP ended, the environment for this kind of engineer became much worse, since tech companies were now actually focused on accomplishing things and the “do whatever you want” era was over Like with layoffs, this shift is often blamed on AI, but it would have happened even if powerful LLMs had not emerged at all. It’s an end-of-ZIRP phenomenon Part of the appeal here is the lure of the guru. In kung fu films, those who know martial arts perform furious acrobatics, but the true expert barely needs to move at all. For the same reasons, it sounds profound to say something like “junior engineers produce tons of code, seniors very little, and staff engineers remove code”. Of course this is false. Staff engineers are expected to be able to produce a lot of working code very quickly, when they need to. ↩ I wrote about this a lot more in The good times in tech are over . ↩ Not necessarily make a profit , but at least bring in revenue. ↩ Or pre-2023, or even pre-2024 or 2025. Cultural change lags behind economic incentives, sometimes by several years. ↩ For fear of killing the vibe (and thus the stock price). ↩ If you think there have been more incidents recently, consider that (a) you might be wrong , or (b) that other end-of-ZIRP factors (like increased velocity or layoffs) might be primarily responsible. ↩

0 views
Sean Goedecke 2 months ago

How I use LLMs as a staff engineer in 2026

A bit over a year ago I wrote How I use LLMs as a staff engineer . Here’s a brief summary of what I used AI for last year: Here are some tasks I explicitly didn’t use AI for last year: February 2025 was a long time ago. Back then the best model was the first reasoning model, OpenAI’s o1. Agents sort of worked, but would often get stuck or thrown off by compaction. What’s changed since then? The biggest change is that I now use LLMs to produce entire PRs in areas I’m familiar with . A year ago I would very occasionally ask an agent to make changes to a single file if it was a simple change I couldn’t be bothered typing out. Sometimes I would copy a function I wrote into a LLM chat window for feedback. But now I start every single change by asking an agent to solve the problem, and usually push the PR after a single editing pass. In late 2025 I used a lot of open VSCode windows. In early 2026, that changed to terminal tabs with the Copilot CLI, particularly when I needed to make changes across multiple repos at the same time. Now I use the GitHub Copilot app a lot (tens of sessions per day). This reflects a shift from having to line-edit the agent basically as it went to only doing an editing pass right at the end. Early agents would go wrong a lot and not be able to recover, so it was valuable to keep an eye on their thought processes and step in to pause them and set them right. In my experience, current agents move too fast to do this, and recover their own mistakes most of the time anyway. Sometimes I don’t even need to make edits and I can just push the change as-is, though this is rare: if nothing else, I typically go through and remove some of the over-commenting and other LLM-isms. I do a lot of skimming through and evaluating agent changes. Most of the time I reject them entirely, just based on “eh, that’s not what I was thinking”. On average it takes me about thirty seconds to make this initial assessment. If the change looks alright after that, I’ll dig in and do a proper review to make sure I understand it and it’s doing the right thing. For difficult tasks, I’ll often reject five or six (or more!) agent attempts before accepting one as good enough to work with, or giving up and making the change by hand. I rely on LLMs even more for bug-hunting than I do for making changes. In 2025, I used to throw the occasional bug at a LLM, just in case it was able to rapidly come up with an explanation. Now I throw every bug at a LLM (typically by opening a new agent session and pasting in the bug report), because it’s able to correctly diagnose 80% of issues on its own. Current agents are really good at chasing down bugs, particularly when you give them a vantage point across multiple repositories. I’m still better at it. Just last week I had a tricky bug that took about fourteen agent sessions before one finally figured it out. What was I doing in between and around those sessions? Ultimately an agent was the one to catch the bug. But I still count it as my find, because by that point I had narrowed the search space tightly enough that agent session #14 had a significantly easier problem to solve than agent session #1. In other words, human expertise still matters a lot for investigating bugs . I almost always write my own PR descriptions, since LLMs over-communicate and are bad at expressing the “core idea” behind a change. Writing the PR description by hand also signals to reviewers that I’ve reviewed the change myself, and I’m not asking them to be the first human to read the diff. The only time when I don’t write the PR description is when the change is trivial and the agent-generated description is one sentence. At that point I just leave it alone. I still don’t use LLMs to write Slack messages, ADRs, issues and so forth. I believe I have a better sense of what’s important to communicate, and I want to signal that there’s a human being thinking about the content. I still never use LLMs to write blog posts, though I do run each draft post through a LLM for feedback. OpenAI models used to be terrible at this and have only very recently gotten acceptable with GPT-5.5. Both OpenAI and Anthropic models still try to water down my arguments, but I’ve accepted that as part of the LLM “house style” and just ignore that part of the feedback. Another thing I do now is try and push as much testing and setup work as possible onto the agents . In 2025, I used to sometimes ask a LLM to produce a test script of curl commands that I could run against my dev server. In 2026, I just ask an agent to go and test my change, then read the log of what it did. I don’t test UI work like this, partly because it’s more fiddly and partly because I don’t trust agents to be sensitive to the subtle look-and-feel aspects of a change. Agents will write expansive unit tests without having to be told, but I do sometimes ask them to put together broader integration tests for a change. In general I now consider test code to be cheap: if I’m wondering whether a test would be useful, I just add it (so long as I know it won’t be flaky). Of course LLMs sometimes produce strange and unsatisfying test code - I do read it to catch obvious blunders - but I review it with a more generous eye than my actual production code. I’ll also task an agent with annoying local setup tasks that involve config wrangling on my machine. For instance, if my nvm installation is not switching my Node version correctly, I will often open a Copilot CLI agent and ask it to figure it out. This is a more-or-less direct replacement for Googling the problem, and is much quicker since the agent can run the trivial bash commands to diagnose and fix the problem itself. The main thing that’s changed in the last fifteen months is that agents are really good now . They’ve gone from something I used occasionally and suspiciously to something I use constantly and with light supervision. The core of my job is still the same: shipping projects , exercising my judgement, influencing tech company politics . But I now have a much wider net for small pieces of work that I’m willing to take on, which includes basically anything I can hand off to an agent and expect it to get more or less right. I used to spend a lot of time putting work off, either by delegating it or just saying “sorry, I don’t have time to do that now”. Now I get to say “yes” a lot more (at least when it comes to minor low-risk tweaks) 1 . Overall, here’s what I now use AI for: Here’s what I still don’t use AI for: In my view, the current core AI skill is shifting as much work onto AI agents as possible, without going too far . Many people are under-utilizing agents: not allowing them to investigate bugs or test their changes, or not throwing enough simple tasks at them. Other people are over-utilizing them: using them to write messages that ought to be hand-written, or trusting them to make sweeping changes that need careful human review. Since my last post, the balance has tilted more towards the agents, but finding the balance remains as tricky as ever. For once I can actually give an example, since it’s in a public repository. Someone internal wanted to be able to use the actions/ai-inference GitHub Action with Copilot-backed inference (for various reasons), and instead of saying “sorry, I don’t have time to get to it”, I was able to throw it at an agent. If a human had to do this, the output would likely have been better, but it wouldn’t have gotten done for weeks (if at all). Smart autocomplete with Copilot Short tactical changes in areas I don’t know well (always reviewed by a SME) Writing lots of use-once-and-throwaway research code Asking lots of questions to learn about new topics (e.g. the Unity game engine) Last-resort bugfixes, just in case it can figure it out immediately Big-picture proofreading for long-form English communication Writing whole PRs for me in areas I’m familiar with Writing ADRs or other technical communications Research in large codebases and finding out how things are done Digging up extra context on the bug (from logs, Slack, etc) and reporting it to the agents Building my own mental model of the problem, of course Setting up my own reproduction of the bug (in parallel with the agents’ efforts) Responding to agent sessions with “no, your theory can’t be right because of X” (or just killing and restarting the session with that extra hint) Writing (or drafting, depending on complexity) every code change I make Investigating and fixing bugs, either autonomously for most bugs or with my close involvement for trickier ones Research in large codebases, since current agents are now good enough to give the right answer almost all the time (and when they’re wrong, it’s clear from reading the explanation that they’ve missed something) Manual testing and local-machine setup or troubleshooting I still use AI for asking lots of questions to learn about topics, and for proofreading Writing any kind of public communication for me (PR descriptions, ADRs, messages) with the exception of trivial two-line PRs Writing code that I don’t carefully review Testing any kind of UI For once I can actually give an example, since it’s in a public repository. Someone internal wanted to be able to use the actions/ai-inference GitHub Action with Copilot-backed inference (for various reasons), and instead of saying “sorry, I don’t have time to get to it”, I was able to throw it at an agent. If a human had to do this, the output would likely have been better, but it wouldn’t have gotten done for weeks (if at all). ↩

0 views
Sean Goedecke 2 months ago

AI datacenters in space do not have a cooling problem

This year Elon Musk has started banging the drum about building AI datacenters in space. As the only person who owns a successful space company and a (moderately) successful AI company, this is a sensible way to boost his profile and net worth. Is it a sensible way to build datacenters? The first comment underneath most discussions of this always goes along these lines: “you obviously can’t build AI datacenters in space, because heat dissipation is really hard in space, and AI datacenters generate a lot of heat”. In general I am distrustful of snappy answers like these. It reminds me of the “AI datacenters obviously don’t use a lot of water, because cooling fluid circulates in a closed-loop system” argument: if it were true, there wouldn’t be a debate at all, just one side who understand the obvious point and another side who are stupid. Some arguments are like this! However, more often there’s a complicating factor that makes the snappy answer incorrect. In the water-use case, it’s that the closed-loop system has to itself be cooled by an open-loop evaporative chiller. What about the space datacenter case? First, let’s give the argument a fair shake. Although space is itself very cold, cooling is tricky because everything you’d want to cool is surrounded by vacuum. Heat transfer works in three ways: Vacuum is an excellent insulator because it defeats the first two methods of heat transfer. If there are no (or very few) atoms surrounding an object, those atoms can’t move around or collide. That’s why vacuum is used as an insulator in thermoses, travel mugs, and so on. So how can space datacenters get rid of their heat? By doubling down on the third method of heat transfer. Although it’s much harder to do heat transfer via moving atoms around in space, it’s actually easier to do heat transfer via emitting radiation. Any good emitter is also a good absorber. A perfectly black object is the most efficient emitter, but it’s also the most efficient way to absorb photons from external sources, which is why black objects get hotter in the sun 1 . In space, the sun’s light is much easier to avoid, because there aren’t objects everywhere for it to bounce off. A shaded radiator can dump quite a lot of heat. It would still require putting more radiators in space than we’ve ever done before. There are plenty of writeups out there if you want to read through the numbers. This is a recent one that estimates ~2500 square metres of radiation area would be needed to serve 1MW of datacenter energy (much less than what it’d need in solar panels) 2 . A serious AI datacenter is around 100MW 3 , so we’d need 250,000 square metres of radiation area. The largest current radiator in space is probably the ISS, at around a thousand square metres. Is scaling that up by 250x a lot? Yes, but it’s not necessarily ridiculous . We currently have zero industrial operations happening in space, so there’s been no need to push the boundaries here. In the grand scheme of things, 250,000 square metres is not that big. By my very rough estimates, that’s between 100-500 Starship launches: a couple of years at SpaceX’s current launch cadence, or a few months at their (very optimistic) estimate of future launch cadence. Of course, you don’t just need radiators to put a datacenter in space. You need a similar quantity of solar panels, the GPUs themselves, and all kinds of other supporting equipment. If a GPU dies in an Earth datacenter, you can go in and swap it out; if it dies in space, you just have to leave it dead and keep going with less capacity. It’s still wildly impractical to build AI datacenters in space. But it’s not impossible , and it’s certainly not impossible because of the cooling, which is a relatively minor component of the total mass that would have to be launched into space. In theory, black clothing would keep you slightly colder at night. Nobody ever talks about how impossible it would be to power space datacenters, despite the fact that you’d need to launch over triple the solar panel area into space than radiation area. I guess because people know solar panels exist and that the sun shines in space. The first gigawatt AI data centers are coming online this year, but 100MW is a fair estimate for a current pretty-large-but-not-enormous AI datacenter. Hot (i.e. fast-moving) atoms bump into other atoms, making them move and thus heating them up Hot atoms physically move from one location to another (e.g. in a fluid or gas), staying hot and thus making their new location hotter Hot objects emit photons (electromagnetic radiation), cooling themselves down and heating up other objects those photons collide with In theory, black clothing would keep you slightly colder at night. ↩ Nobody ever talks about how impossible it would be to power space datacenters, despite the fact that you’d need to launch over triple the solar panel area into space than radiation area. I guess because people know solar panels exist and that the sun shines in space. ↩ The first gigawatt AI data centers are coming online this year, but 100MW is a fair estimate for a current pretty-large-but-not-enormous AI datacenter. ↩

0 views
Sean Goedecke 2 months ago

Thinking Machines and interaction models

Thinking Machines just released Interaction Models . This is their first real AI model release 1 after a year of work and two billion dollars of capital. What is an “interaction model”? First, it’s not a frontier model . Thinking Machines is not yet competing with OpenAI, Anthropic and Google. Instead, they’re working on the problem of better real-time interaction with models. Some parts of what they’re doing are not new at all, other parts are slightly-questionable benchmark gaming, and still other parts represent a genuine technological advancement. I’ll try to lay it all out. If you’ve used ChatGPT in audio mode, you know that you can’t talk to it exactly how you’d talk to a human. There’s a big latency gap between when you finish talking and when the model jumps in. The model won’t interrupt you like a human, and doesn’t react to you interrupting it like a human would either. And of course you can’t give the model visual feedback like facial expressions. That’s because ChatGPT is either speaking or listening at any given time . When you’re talking, it’s in “listening” mode; when it’s talking, it’s in “speaking” mode, and isn’t absorbing any information from you. It relies on VAD (“voice activity detection”) to figure out if you’re talking. The alternative (and what “interaction models” do) is a fully-duplex system, where the model is constantly both in listening and speaking mode at the same time. Of course, the model can’t literally do this. Like all language models, it’s either doing prefill (ingesting prompt tokens) or decode (producing completion tokens). But what fully-duplex models can do is switch from listening to speaking mode in tiny chunks, called “micro-turns”. Instead of listening for ten seconds (or however long it takes you to stop talking), then speaking for ten seconds (or however long it takes to pass the model output through TTS), the model can listen for 200ms, then output for 200ms, then listen for 200ms, and so on. While the user is speaking, the model will know to output silence - most of the time. But if it decides it’s good to interrupt you or speak at the same time as you, it’s capable of doing that. So far, so unoriginal. There are plenty of examples of fully duplex audio systems that the Thinking Machines blog post already cites: Moshi , PersonaPlex , Nemotron-VoiceChat , and so on. But at least this outlines the space that “interaction models” are playing in: not “superintelligence from a frontier model”, but “better real-time conversational interaction” 2 . Given that, what is Thinking Machines doing that’s new? For existing fully-duplex models, you talk to the model itself. That’s a fairly big problem, since fully-duplex models have to be fast: fast enough that they can operate in tiny 200ms turns 3 . A model that fast cannot be particularly intelligent. Thinking Machines’ solution is to introduce an actual smart model - any regular language model will do here - in the background that the interaction model can delegate tasks to. In practice this is probably implemented as a tool call. The interaction model keeps chatting while the smart model works away, and then the smart model output is directly integrated into the interaction model’s context in the same way as audio and video input (a genuinely cool idea, I think). This is kind of neat, though it remains to be seen how well it works in practice. Will the model do a lot of “oh wait, the last thing I said was dumb, never mind” self-correction as the smarter model output trickles in? Will the fast interaction model be smart enough to delegate the right tasks at the right time? In general, the “start with a fast dumb model and have it hand off tasks” approach has been tricky for the AI labs to get right for a variety of reasons. If I’m being uncharitable, I might say that bolting on a strong reasoning model was an easy way for Thinking Machines to post impressive values for competitive benchmarks like FD-bench V3 (where they barely beat GPT-realtime-2.0) and BigBench Audio (where introducing the reasoning model bumps their score from 76% to 96%, only 0.1% below GPT-realtime-2.0). If I’m being charitable, I might say that a model fast enough for realtime conversation will have to have some way to punt hard tasks to a slower, smarter model. Both of those things are probably true. It’s also worth noting that Thinking Machines have also bolted on video input to their fully-duplex model. This is more exciting than it sounds, because face-to-face human conversation is very dependent on being able to read human expressions. In theory, this could unlock the ability to have genuine human-like conversations. The other reason why this is exciting is that it means Thinking Machines have been able to make a pretty big fully-duplex model (maybe twice the size of Moshi in terms of active parameters, and 40x the size in terms of total parameters). In fact, this is probably the biggest real technical achievement here. Other fully-duplex models are already doing micro-turns and interruptions, and could delegate reasoning fairly easily if they wanted to, but they aren’t doing video because they can’t . Being able to make a fully-duplex model the size of DeepSeek V4-Flash is pretty impressive. Much of the Thinking Machines blog post is dedicated to explaining how they’ve managed to do this: ingesting data in a more lightweight way, optimizing their inference libraries for tiny prefill/decode chunks, various decisions to make inference deterministic (a long-held hobbyhorse for Thinking Machines). There’s a lot of pressure on Thinking Machines to produce a genuine AI advancement. It doesn’t seem like they’re willing or able to compete in the frontier-model space (which makes sense, I wouldn’t want to either). Given that, I can see why they’re highlighting the parts of interaction models that are impressive to laypeople - all the fully-duplex interaction stuff - even though those parts are not truly innovative. So what are Interaction Models? A scaled-up, multimodal version of existing fully-duplex models like Moshi, with a real model bolted on for extra intelligence (and maybe better benchmarks). The scale and video parts are new and cool, and something like the overall approach has to be right. In general, I’m glad that we’ve got well-funded and high-profile AI labs tackling problems other than “build a smarter frontier model”. I think there’s a lot of low-hanging fruit waiting to be picked in other areas of AI research. People do seem to really like Tinker , which is their tooling for researchers who want to fine-tune models, but it’s not exactly the hot new frontier model that people were expecting. I think it’s at least a little shady that the Interaction Models video demo is making a big deal about some features (like real-time simultaneous translation) that are just features of fully-duplex audio models, not anything specific to their system. Even 200ms is a bit long. You can see from the demo that there’s an uncomfortable half-second lag sometimes as the model finishes its prefill slice and has to move to the decode slice. People do seem to really like Tinker , which is their tooling for researchers who want to fine-tune models, but it’s not exactly the hot new frontier model that people were expecting. ↩ I think it’s at least a little shady that the Interaction Models video demo is making a big deal about some features (like real-time simultaneous translation) that are just features of fully-duplex audio models, not anything specific to their system. ↩ Even 200ms is a bit long. You can see from the demo that there’s an uncomfortable half-second lag sometimes as the model finishes its prefill slice and has to move to the decode slice. ↩

0 views
Sean Goedecke 2 months ago

AI makes weak engineers less harmful

Like other kinds of puzzle-solving, software engineering ability is strongly heavy-tailed. The strongest engineers produce way more useful output than the average, and the weakest engineers often are actively net-negative: instead of moving projects along, they create problems that their colleagues have to spend time solving. That’s why many tech companies try to build a small, ludicrously well-paid team instead of a large team of more average engineers, and why so far this seems to be a winning strategy. Being effective in a large tech company is often about managing this phenomenon: trying to arrange things so that the most competent people land on projects you want to succeed, and the least competent are shunted out of the way 1 . For instance, if you’re technical lead on a project, you more or less have to ensure 2 that the most critical pieces are in the hands of people who won’t screw them up (whether by directly assigning the work, or by making sure someone can “sit on the shoulder” of the engineer who you’re worried about). Claude Code changed this. Frontier LLMs don’t have the taste or the system familiarity of a strong engineer, but they have absolutely raised the floor for weak engineers. Instead of getting a pull request that could never possibly work or would cause immediate problems, the worst you’ll now see is a standard LLM pull request: wrong in some ways, baffling in others, but at least functional on the line-by-line level and not so obviously incorrect that someone with no knowledge of the codebase could point it out. That is a huge improvement! You can try this out yourself. If you attempt to deliberately make mistakes while working with a coding agent, you’ll find that the agent pushes back hard against many obvious errors (i.e. caching user data with a non-user-specific key, writing an infinite loop that might never terminate, or leaking open files). Of course, the agent will still miss subtle errors, particularly ones that require understanding other parts of the codebase. Working with the least effective engineers is now sometimes like working with a Claude Opus or Codex instance that you communicate with over Slack. Occasionally it’s literally that: your colleague is simply pasting your messages into Claude Code and pasting you the response. This is annoying, but it’s a much better experience than working with this kind of engineer directly. After all, you probably already work with a bunch of LLM instances. The Slack interface is not ideal - unlike using Claude Code directly, you sometimes wait hours or days for a response, and you don’t get visibility into the agent’s thought processes - but it’s still helpful on the margin. More compute being thrown at your problem is better than less. Of course, this isn’t a great state of affairs for the engineer in question, who is almost certainly learning less than if they were making their own (bad) decisions. It’s also a bad state of affairs for the company, who is paying a human salary and getting a Copilot subscription (which they’re likely also paying for) 3 . After the current push to figure out what value AI is adding to engineers, I suspect there will be a push to figure out what value engineers are adding to AI , and the engineers who aren’t adding much may find themselves out of a job. You can’t talk to Claude-over-Slack like you’d talk to normal Claude. If you tend to handle LLMs roughly (insulting them, or just being very curt), you’ll have to change your communication style. A human is going to read your messages, after all, even if you’re really interacting with a LLM. There’s no point being rude. But if, like me, you say please-and-thank-you to the models 4 , you can treat your LLM-using coworker as just another Copilot window or Codex tab. It’s far better than having to treat them as an unwitting saboteur. Not all net-negative engineers use AI tools like this. Many are strongly convinced in their own wrong opinions about how to build good software, or mistrust AI in general, or believe that relying heavily on LLMs is not a good way to improve 5 . But no strong engineers use AI tools like this. Even when they’re being lazy or sloppy, a capable engineer will have enough baseline taste to catch obvious AI-generated errors. So the phenomenon of engineers 6 becoming thin wrappers around Claude Code is limited to the kind of engineers for whom this is an improvement in their work product. More charitably: many “least competent” engineers are just out of their comfort zone, and can be fine or even excel under the right circumstances (though in my view the best engineers are able to do good work in a wide variety of environments). Also, I don’t currently work with a lot of incompetent people. Much of this is based on past experience or talking to other engineers in the industry. Since your managers are doing the same thing, this can sometimes feel like Moneyball: you’re trying to identify underappreciated talent who are strong enough to help you win without being so high-profile that your boss poaches them to lead something else. I suppose it’s better to pay for nothing than to pay for net-negative output, but it still doesn’t seem good . I think this is actually the right way to hold Claude Opus 4.7. Is this true? I think relying on LLMs is not a great way for most engineers to improve, but if LLM output is consistently better than your own, it might be different. So long as you’re paying attention to where the LLM does better, it could actually be a good way to learn. I don’t have as much experience (or anecdotes) about non-engineers falling into this trap, but this post has convinced me that it might be worse. More charitably: many “least competent” engineers are just out of their comfort zone, and can be fine or even excel under the right circumstances (though in my view the best engineers are able to do good work in a wide variety of environments). Also, I don’t currently work with a lot of incompetent people. Much of this is based on past experience or talking to other engineers in the industry. ↩ Since your managers are doing the same thing, this can sometimes feel like Moneyball: you’re trying to identify underappreciated talent who are strong enough to help you win without being so high-profile that your boss poaches them to lead something else. ↩ I suppose it’s better to pay for nothing than to pay for net-negative output, but it still doesn’t seem good . ↩ I think this is actually the right way to hold Claude Opus 4.7. ↩ Is this true? I think relying on LLMs is not a great way for most engineers to improve, but if LLM output is consistently better than your own, it might be different. So long as you’re paying attention to where the LLM does better, it could actually be a good way to learn. ↩ I don’t have as much experience (or anecdotes) about non-engineers falling into this trap, but this post has convinced me that it might be worse. ↩

0 views
Sean Goedecke 2 months ago

Notes on incidents

Incidents are boring. Most of what you actually do during an incident is wait: for some other team to investigate, or for a deploy to finish, or for the result of some change to become apparent, or for someone else who’s been paged to come online. It’s stressful, but there’s often just not that much to do. Most incidents resolve on their own. People love to share war stories about incidents where some hero engineer improvised a clever fix that instantly repaired the system. That rarely happens. Well-designed software systems tend to come good by themselves, and many modern systems are at least partly well-designed, by virtue of being built out of really solid pieces. If a server process is crashing or leaking memory, Kubernetes will kill the pod and bring it back up. If a service is overloaded and jammed up, clients will (hopefully) trigger circuit breakers and back off until it can recover. Temporary spikes in expensive operations will often just fill up a queue instead of taking the entire system down. Most incident calls I’ve been on - well over half - would have come good by themselves in roughly the same time without any human intervention. Most incident-resolving actions make incidents worse. Engineers jump too quickly to resolve incidents. Oh, the queue size is huge? Don’t worry, I’m here in a production console to clear the queue! Unfortunately, some of the jobs I just nuked were doing important billing work and aren’t automatically re-queued, so this queue-latency incident just became a billing incident as well. Another classic in this genre is “engineer forces a series of redeploys to “fix” a concerning-looking metric, and the concurrent deploys cause far more stress on the system than whatever was causing the metric to look weird”. For that reason, the first thing you should do in an incident is nothing . When I was paged late at night, I used to have a habit of pouring myself a glass of scotch before I joined the call. This was only partly for the tranquilizing effects of alcohol: the main reason was to have a ritual I could go through to convince myself that I wasn’t rushing, and that it was OK to take a few breaths and relax before jumping into the problem 1 . Making a cup of tea or going for a walk around the house would probably have served as well. Effective incident-resolving actions are often dull. Typically the action needed to resolve the incident - assuming it doesn’t resolve on its own - is to temporarily disable some problematic feature until the system recovers. This is never a complex code change. Typically someone spends five minutes putting together the patch, and then an hour waiting for reviews, CI, and deploying. If you’re very lucky, you’ll get to write a “wrap a cache around it” code change. In an incident, there is no substitute for knowledge of the system. Five strong engineers can troubleshoot on an incident call and get nowhere, while one half-drunk engineer who’s familiar with the codebase can swan in and immediately fix the problem. This is because the kinds of actions that resolve incidents are so simple: if you’ve been the one working on the project, you likely already know exactly what feature flag to check and disable, or what code change to revert. Resolving incidents requires courage. Incident calls can be scary. When engineers are scared, they often reach for consensus: hedging their statements, asking the group if they agree a particular course of action is safe, deferring to each other, and so on. But if you’re the one with knowledge of the system, you have to be decisive. Say “I’m going to do X”, wait thirty seconds, then do it. While it’s usually net-negative to have a powerful manager fidgeting on the incident call, this is one of the rare cases where it can be helpful - executives are very comfortable saying “okay, do it now” about technical courses of action they don’t fully understand. Resolving incidents buys a lot of political credit. One thing that I think surprises a lot of engineers who are new to on-call is how grateful managers and executives are for even really simple fixes (i.e. “turn off the feature flag”). This is because incidents are one of the few times that non-technical leadership are directly confronted with their lack of control over the technical sphere. When the team is building a product, your VP has a lot of freedom to guide the process and make decisions. But when there’s an active incident, they have to just sit there and trust that their technical employees are going to pull them out of the fire. It’s a scary situation, particularly for someone who’s used to exercising a degree of power in the workplace. However, always resolving incidents is (by itself) not a durable position of power. This is a little counter-intuitive. Surely if you’re always resolving incidents, you’re indispensable? The problem is that incident-resolving work is almost always so techical as to be completely opaque to executives. They know the incident has resolved, but they don’t know if you did a heroic effort or merely did the obvious thing. They also can’t point to your successes as theirs (which is always the most reliable way to get VPs and directors on your side), because incidents are expected to be fixed , and it’s always better not to have had the incident at all . I don’t need to do this anymore because I just don’t get as keyed up about incidents as I used to. I don’t need to do this anymore because I just don’t get as keyed up about incidents as I used to. ↩

0 views
Sean Goedecke 2 months ago

Why hasn't longer-horizon training slowed AI progress?

Dwarkesh Patel 1 recently posted an award for the best answers to four key questions about AI. It’s partly a challenge and partly a job interview, since some of the winners will get offered a role as a “research collaborator”. I don’t want the job, but I do want to write down my answer to his first question: why hasn’t AI progress slowed down more? There are a few reasons we might think AI progress would slow down. The particular reason Dwarkesh is interested in goes like this. Training a model (specifically reinforcement learning) requires the model to perform a task and then get “graded” on the output. As models get more powerful and tasks become harder, they take longer and require more FLOPs 2 to complete, and thus more FLOPs to train: thus training harder models will take longer. But intuitively, AI progress hasn’t slowed down that much. The famous METR horizon-length graph shows that AI systems are capable of more and more complex tasks over time, and that this process is accelerating, not slowing down. Why would that be? Firstly, it might just be the case that newer models are benefiting from orders of magnitude more FLOPs . Of course, AI labs aren’t standing up orders of magnitude more GPUs (they’re trying, but there are hard physical limits on how fast you can scale up a physical datacenter). But it’s certainly possible that they’re learning to use their existing FLOPs orders of magnitude more efficiently. The efficiency of complex software systems - and the training code for a frontier AI model certainly qualifies - is not typically determined by the number of genius ideas in it. It is determined by the number of boneheaded mistakes. Take this story 3 of how the initial GPT-4 training run used FP16 when summing many small values, which will completely mess up your results if the sum of those values is large. How much training-efficiency-per-FLOP does solving bugs like that buy? Plausibly enough to outweigh any inherent lack of efficiency from training more powerful models. Secondly, intuitions about the speed of AI progress are weird and unreliable . Humans measure AI progress - and intelligence in general - on a really uneven scale. It’s easy to tell when an AI (or a person) is less smart than you, because you can just see them making mistakes. It’s very hard to tell if they’re smarter, because in that case you’re the one making mistakes. You have to rely on more subtle context clues: do they get better long-term results than you, or do they often confuse you in situations where you later end up agreeing with them, and so on. The jump from GPT-3 to GPT-4 seemed huge because GPT-4 was dumber than almost all humans, and GPT-4 was sometimes as smart as a human. However, frontier models are now smart enough to be in the realm of ambiguity on many topics. It’s thus much harder to tell the “real” rate at which they’re getting smarter. Maybe the rate of growth of “raw intelligence” really has slowed down! I don’t know how we’d be in a position to know for sure. Thirdly, many traits other than intelligence determine the capabilities of AI models . Take the jump in October last year where OpenAI and Anthropic models were suddenly “agentic” (i.e. they could reliably perform complex tasks end-to-end). That might be intelligence, but it might also just be a greater working memory, or more rote familiarity with the basic tools of a LLM harness, or more ability to attend to the context window, or even simply a personality more suited to tools like Claude Code or Codex. Of course, all of these traits are plausibly “intelligence”. But they’re traits you might instil by various clever tricks (or even just tweaking the system prompt), not by brute-forcing more FLOPs. It’s illustrative here to consider the mistake made by Apple’s infamous The Illusion of Thinking paper, where the researchers asked various models to brute-force solve Tower of Hanoi puzzles with different numbers of disks, using the results to score how good at reasoning the models were. But of course when you read the output, all of the failures were cases of the model realizing that many hundreds of steps were required, and refusing to even try. These same models could trivially write code to perform the steps, or correctly go through any smaller subset of the steps. The problem wasn’t intelligence, it was persistence : these models lacked the willingness to dig in and keep powering through steps until they got to an answer 5 . Even inside an AI lab, I don’t think anyone has a good understanding of how many “real” FLOPs are being thrown at a training run (not counting FLOPs that are wasted on bugs). We also don’t have a clear sense of whether AI progress really is slowing down or not. Mythos seems impressive, and coding agents are really good now, but once the models get close to human intelligence it becomes really tricky to monitor. Finally, almost everyone judges intelligence by capabilities, but capabilities are produced by a constellation of many traits (intelligence is just one of them). I think this stuff is really complicated. A general theory like “RL takes more flops-per-reward as tasks get longer, therefore training will gradually slow down” sounds good, but in practice AI development is dominated by lightning strikes: silly bugs that make training a hundred times worse, clever ideas that make models a hundred times more useful, and spiky capabilities that can produce dazzling results in some areas but zero improvement in others. We are still very early . If you’re reading this you probably know who Dwarkesh is, but if you don’t: he’s a well-known tech-adjacent podcaster whose gimmick is that he actually does extensive research before each guest and asks specific technical questions. A FLOP is a floating-point operation, i.e. a matrix multiplication, i.e. “time on a GPU”. I saw this in a tweet and only realized that the source was Dwarkesh when I was researching for this post. What if AI progress stalls for technical reasons, and everyone gives up on training new models? In that world, open source models will eventually catch up, and AI labs won’t be in a privileged position. Incidentally, this is my pet theory about why models got much better at agentic tasks last year: training on longer and longer agentic traces meant that models started to “believe they could do it”, and made them much less likely to just give up and take shortcuts or refuse to continue. If you’re reading this you probably know who Dwarkesh is, but if you don’t: he’s a well-known tech-adjacent podcaster whose gimmick is that he actually does extensive research before each guest and asks specific technical questions. ↩ A FLOP is a floating-point operation, i.e. a matrix multiplication, i.e. “time on a GPU”. ↩ I saw this in a tweet and only realized that the source was Dwarkesh when I was researching for this post. ↩ What if AI progress stalls for technical reasons, and everyone gives up on training new models? In that world, open source models will eventually catch up, and AI labs won’t be in a privileged position. ↩ Incidentally, this is my pet theory about why models got much better at agentic tasks last year: training on longer and longer agentic traces meant that models started to “believe they could do it”, and made them much less likely to just give up and take shortcuts or refuse to continue. ↩

0 views