Sean Goedecke 4 days ago

Don't diagnose your opponents

When you disagree with someone, there are two paths forward. The first is to try to engage with what they’re saying: what their reasons are, whether their position is true, what evidence could be brought to bear, and so on. The second is to diagnose them: to engage with the second-order question of why they’re saying it. For instance, if you’re arguing with somebody about microservices 1 , you could say “well, you only like microservices because you want to polish your resume”, or “you only like monoliths because you’re scared of distributed systems”. This happens a lot in online arguments. On the question “are AI models improving”, some anti-AI people will argue that people only think so because AI psychosis is rotting their brain , and the dumber they get, the smarter AI models seem in comparison. Likewise, many pro-AI people argue that people only disagree because they’re Luddites, or scared of change, and so on. None of these arguments are about the quality of AI models. It feels so good to diagnose your opponents because there’s no possible response. If you say “I think AI models are improving because these benchmark scores are rising”, somebody might point out that the AI labs are training on those benchmarks, or argue that they represent isolated skills that don’t transfer, and so on, making you feel stupid. If you say “you only disagree with me because you want to roleplay as a revolutionary”, how could you be proven wrong? The more someone protests a diagnosis, the more it sounds like it must be right. Otherwise why would they be so upset? Of course, they’re so upset precisely because there is no possible response. By moving to diagnosis, you’re no longer having an argument about the facts, you’re purely in the area of rhetoric. But when you’re in the middle of an argument, diagnosing always feels like you’re being piercingly analytical. It feels like you’re cutting right to the heart of the matter: these complicated arguments are just chaff , intended to distract you from the fundamental psychological reason why your opponent is being so foolish. This is just ordinary spite and anger 2 , whatever it feels like at the time. There’s nothing inherently wrong with the second-order question of why someone person believes what they believe. It’s totally fine to think about it or form theories about it, like any other factual question. Just remember that it doesn’t bear on the question of whether they’re right 3 . Every human being forms their beliefs on top of a cocktail of base urges, animal instincts, and barely-suppressed evolutionary drives. If they’re rational, they also have evidence for those beliefs. It’s the quality of that evidence that helps you determine whether the beliefs are true, not the existence of the base urges. As a general rule, you should only diagnose your opponents when you also have a strong, unrelated argument about why they’re wrong. For instance, I think Ed Zitron has been thoroughly audience-captured by AI skeptics, whose demand for news about AI failures far exceeds supply. But I have written many long-form posts about why I think Zitron’s specific claims about the sustainability of AI are incorrect. When I talk to people who think Zitron is right, I talk about my arguments in those posts, not about my personal theory about why he says the things he does. Scott Alexander famously called the “noncentral fallacy” 4 the Worst Argument in the World . For me, diagnosing your opponents is the Worst Argument in the World. I cringe every time I see it in the wild (like most cringeworthy behavior, it bothers me because I myself have to struggle against the urge to do it). Just don’t! It’s good at playing to your base, but it doesn’t persuade: anyone who believes your opponents’ views will just be mad that you’re psychoanalyzing them instead of talking about the actual issues at stake. Am I diagnosing the diagnosers here? Maybe, but I think it’s forgivable, since there is no specific argument about the facts at stake. The philosopher in me wants to clarify that you could theoretically establish that someone’s motives for believing something are leading them away from the truth. In practice, (a) you cannot do this in an internet argument, and (b) it is always better and easier to make a straightforward argument about what the truth is. “You’re a fringe member of some group which has members I have an emotional reaction to, therefore I will have the same reaction to you even though you yourself don’t do those things.” “Group” here is to be understood as “Christian” or “Italian”, not something like the WBC where the fact of membership itself is what’s being judged. Don’t. ↩ Am I diagnosing the diagnosers here? Maybe, but I think it’s forgivable, since there is no specific argument about the facts at stake. ↩ The philosopher in me wants to clarify that you could theoretically establish that someone’s motives for believing something are leading them away from the truth. In practice, (a) you cannot do this in an internet argument, and (b) it is always better and easier to make a straightforward argument about what the truth is. ↩ “You’re a fringe member of some group which has members I have an emotional reaction to, therefore I will have the same reaction to you even though you yourself don’t do those things.” “Group” here is to be understood as “Christian” or “Italian”, not something like the WBC where the fact of membership itself is what’s being judged. ↩

Machine Learning Microservices AI

▲ 0

0 views

Sean Goedecke 5 days ago

Software's centaur age may last decades

We are currently living in the age of centaurs (2022-20??). Right now, human engineers paired with AI coding systems are more effective than either by themselves. For future generations of software engineers — human or otherwise — this will be the most significant fact about the current period. As in chess’s own centaur age, people are always ready to claim that the centaur age is over and the age of AI supremacy has begun. But the age of centaurs in chess lasted about twenty years, and previous centaur ages have lasted far longer . This should be encouraging to many of us. If software engineering goes along similar lines, we may be able to ride it out another decade or two: enough to make serious plans about what comes next, or to finish out our careers entirely. It began with GitHub Copilot, which launched in 2022 as AI-powered autocomplete. A confident engineer who knew exactly what they wanted could tab-complete their way through lines of code and sometimes entire functions. Being able to chat with language models like GPT-4 provided the next upgrade. Although the models would still get things wrong, it was helpful to have a generalist on tap for areas that were a complete mystery. Coding agents rose to prominence with Cursor’s “agent mode” in 2024 and Claude Code in early 2025. Early agents needed close supervision to prevent them from going off the rails, but were still helpful to automate straightforward tasks. Current agents — after the November 2025 release of Claude Opus 4.5 — are reliable enough to run entirely unsupervised. Their output still needs human review, but the kind of mistakes they make are more mistakes of alignment than regular bugs: not matching the organization’s technical values, over-engineering certain areas and under-engineering others, deciding where to spend effort, and so on. It is no longer possible to credibly argue that an unassisted software engineer can beat a centaur team. The value provided by coding agents is just too great: not just the speed of producing code, but the speed of review and testing. However, we are not yet at the point where an unsupervised coding agent can beat a centaur team either. All the impressive work I’ve seen from LLMs has had a competent human software engineer in control. Centaurs are on top. Would we expect our centaur age to be shorter? On the one hand, chess is a much simpler problem than software engineering, and is more amenable to learning methods like self-play. On the other hand, software engineering is massively more lucrative to solve, and many orders of magnitude more funding is being plowed into the problem than was with chess. On the third hand, “solving” software engineering means creating much more software, which increases the total work-to-be-done, so even if the relative human share of that work is shrinking, the overall amount of human work might grow. On the fourth hand, the level of generally-capable AI needed to address software engineering will have huge effects on other fields, the ripple effects of which are likely to negatively impact software engineering jobs (for instance, crashing the economy, widespread social unrest, and so on). On the fifth hand, we might think that chess had an unusually fast centaur age. Previous centaur ages lasted much longer. In knitting, where a human and a knitting-frame could outcompete any hand-knitter, the centaur age lasted two hundred years . On the sixth hand, technological change is moving ever faster . I could probably find a seventh, eighth, and ninth hand if I wanted to 1 . But overall I think we have no real idea of how this is going to turn out. Chess is the most recent example of a centaur age we have, and it’s attacked by the same AI-based forces of automation as software is. It’s thus reasonable 2 to make a default assumption that the length of software’s centaur age will be similar to chess’s. Suppose I have convinced you that human-AI partnerships will be the dominant force in software engineering for at least the next decade. What then? Don’t quit! Don’t give up on software engineering and go become a carpenter, or open a cafe, or try to get a retail job. Unless you are somehow really, really passionate about doing those things, it is going to suck : unrewarding, grueling work that you are not good at. A decade of software engineering work is long enough to take your time and think seriously about your next steps. Lean into the partnership. A guaranteed way to fail in a centaur age is to refuse to become a centaur. The idea that AI is a passing fad was a reasonable position to hold in 2023, but we’re way past that now. The AI financial bubble popping will not mean the end of AI . It’s here to stay. Think hard about what value humans can still add. This is going to change over time, and getting it right is the difference between being an effective (and employed) centaur and being a meat proxy . In 2023, technical expertise was the primary source of human value. Today, I personally think it’s going to come down to alignment , but this is still early days. I’ve also seen arguments for taste, though it’s hard to define that non-circularly. We need a lot more engineers thinking through this problem. It’s a common fear response to try and skip ahead to the worst-case scenario. But those of us working in the age of centaurs should avoid that. We need to think about how the job of software engineering functions now , not how it might function in twenty years when AI has completely transformed the landscape. People are too quick to declare that we’re in the software engineering apocalypse. We can certainly see it from here, but we’re not there yet. The spectre of full automation can loom on the horizon for decades. If you panic and jump ship early, you could be giving up on an entire normal-length career. Maybe: AI as fundamentally different technology; diffusion of even radically better technology is slow in practice and bottlenecked by organizational friction; human software companies will get outcompeted? I don’t really know what I think about these arguments. This move should be familiar to anyone with a philosophy background. If you have one argument and one counter-argument, the most recent counter-argument is probably right. If you have a stack of arguments and counter-arguments going back a thousand years, the most recent counter-argument should be treated with suspicion: better to take in the whole debate holistically and judge which side is more compelling to you. Maybe: AI as fundamentally different technology; diffusion of even radically better technology is slow in practice and bottlenecked by organizational friction; human software companies will get outcompeted? I don’t really know what I think about these arguments. ↩ This move should be familiar to anyone with a philosophy background. If you have one argument and one counter-argument, the most recent counter-argument is probably right. If you have a stack of arguments and counter-arguments going back a thousand years, the most recent counter-argument should be treated with suspicion: better to take in the whole debate holistically and judge which side is more compelling to you. ↩

AI Programming Career Ruby

▲ 0

0 views

Sean Goedecke 1 weeks ago

How to read code

Everyone knows how to read a book. Beginning at the first page, you read each word in order 1 , stopping periodically to think, until you arrive at the last word on the last page. That’s how you read a newspaper article, or a poem, or an email. Why would reading code be any different? English text is designed to be read in order. In fact, there are almost no constraints on the order of a text aside from how you want the reader to consume it. In writing this post, I could put the ideas I want to convey in any order I like. Code, on the other hand, is designed to be run by a computer. The order is thus primarily determined by non-human factors. I cannot simply move 2 a line of code to the beginning of a file or function because I think it provides a better introduction to the program for human readers. The other big difference is that English text is always read as a final product, while code is usually read as a diff . We software engineers spend most of our time reading subtle changes to existing code, not brand-new programs. Imagine if reading this post was like that. You would read each successive draft in the order I wrote them, consuming the post as a changed sentence here and a new sentence there. It would be easy to lose track of the overall flow. The third reason — and I say this as a lover of literature and poetry — is that code is much more structurally complex than English texts. Even famously difficult books are syntactically simpler than most computer programs (for instance, grammatical dependencies are largely bounded by a single paragraph, while code dependencies can stretch across the entire codebase). Their primary difficulty lies in understanding the nuances of human nature being discussed, not in understanding what each word’s grammatical function is 3 . Large codebases are also just longer: War and Peace contains around 600,000 words, while most large modern codebases have that many lines . As I’ve said many times , large computer programs are simply too complex for a single person to fully understand. Reading code in a large program is thus a process of compromise : of deciding which parts to thoroughly grasp and which parts to gloss over; or of portioning out your finite mental capacity across the codebase. Because of all this, most people read code very badly . They struggle through it front-to-back, like a book, and lose track of the execution flow. Or they just read through the diff and miss the significance of un-edited parts of the code 4 . Or they simply are defeated by the complexity, give up and just guess what it means. How can you do better? The best article I’ve read about reading code is this piece about reading mathematics papers, which have a similar structure. The author describes a technique called “dyadic scanning”. Instead of reading slowly and sequentially, you make several passes: first to figure out the overall structure, then the sub-structure, and then finally the details. For code 5 , this means reading out-of-order. I like to pick an important path (say, the happy path for the feature introduced in the diff) and trace through which functions are calling which other functions, just to get a sense of the flow. Only once I’ve got a good sense of that do I pay close attention to what those functions are actually doing. Usually I do multiple passes, each following a different thread. I’ll take a function or a piece of data and try to figure out how it’s used, fanning out to multiple call-sites (including ones outside the diff) as I go. For small diffs, I just ctrl+f for the function name to jump around; for large diffs, I open it in-editor and ctrl+click for easier navigation. I try to be ruthlessly focused on just the thing I’m looking at right now: everything else gets treated as a black box. Once I’m confident I understand the diff, only then will I sit down and carefully read it end-to-end. The purpose of that read is less to learn about the structure — which I should already know by this point — than to catch any weird bits of code I hadn’t noticed in previous out-of-order passes. If I do see anything unusual, I then go back to doing passes. This might sound slow. But in fact each pass is very fast, since I’m not painstakingly puzzling through each line of code. Many people are now saying that you don’t have to read code anymore: either because LLMs now produce reliably high-quality code without oversight, or because you can simply ask a reviewer LLM to read the code for you. I think both of these ideas are false. Obviously the quality of LLM code is context-dependent. As I wrote in Pure and impure software engineering , some software fields (game development, libraries, tools like databases) have wildly different engineering standards, practices and values to other software fields (say, distributed systems at big tech companies). If you’re just making a tool for yourself, you probably don’t have to read the code if you don’t want to. But having read a bunch of AI-generated code this year, I can say that you definitely still have to read it. I routinely find massive errors in AI-generated code. These are not bugs — the code typically does what the AI wanted it to do — so much as they’re problems of alignment . As a recent example, a small change to thread an extra value through some existing code ballooned out into a complex three-thousand-line diff, because the agent noticed a race condition and built a complex machinery to “fix” it. In fact, this race condition was harmless by design: two pieces of unrelated data could become briefly out of sync, with no customer impact. Can LLMs just read the code for you? No, for the same reason: even if they make no mistakes, their technical values will not match yours or those of your company. You can still use LLMs to help you read code, but you have to carefully read and review that LLM output, and you should also be carefully reading the code itself. This is in fact not how many people read in practice. It’s common to skip words or even whole paragraphs due to inattention, or to read an entire book without taking the time to think about it. But at least in theory everyone agrees this is how you should read. Of course, you can do some of this some of the time (more so in some programming languages than others). But the primary constraint is the computer. It is more important that code compiles (or runs without syntax error) than it is for that code to be readable. I don’t wish to understate the importance of syntactically parsing great literature, which I think is both difficult and an underrated skill. For example, I read through the first scene of Hamlet until I saw: “Sit down awhile / And let us once again assail your ears / That are so fortified against our story / What we have two nights seen.” Even a single sentence like this takes work to determine that “That” refers to “ears” and “What” refers to “story” (or, equivalently, has an implied “with” preceding it). And just as one function can have many scattered callers, one sentence in a text can alter the significance of many others. In a previous post, I labeled this the main mistake that most engineers make in code review. Here I’m talking about a meaningful diff: something that touches a few hundred lines or more. Trivial diffs can be read end-to-end. This is in fact not how many people read in practice. It’s common to skip words or even whole paragraphs due to inattention, or to read an entire book without taking the time to think about it. But at least in theory everyone agrees this is how you should read. ↩ Of course, you can do some of this some of the time (more so in some programming languages than others). But the primary constraint is the computer. It is more important that code compiles (or runs without syntax error) than it is for that code to be readable. ↩ I don’t wish to understate the importance of syntactically parsing great literature, which I think is both difficult and an underrated skill. For example, I read through the first scene of Hamlet until I saw: “Sit down awhile / And let us once again assail your ears / That are so fortified against our story / What we have two nights seen.” Even a single sentence like this takes work to determine that “That” refers to “ears” and “What” refers to “story” (or, equivalently, has an implied “with” preceding it). And just as one function can have many scattered callers, one sentence in a text can alter the significance of many others. ↩ In a previous post, I labeled this the main mistake that most engineers make in code review. ↩ Here I’m talking about a meaningful diff: something that touches a few hundred lines or more. Trivial diffs can be read end-to-end. ↩

Programming

▲ 0

0 views

Sean Goedecke 1 weeks ago

Superpersuasion will look like bribery

The idea of “superpersuasion” has been floating around the AI safety community for decades. Now that powerful and difficult-to-control LLMs have appeared, people are again talking about the idea that a sufficiently intelligent AI might be able to persuade people to do whatever it wants. The classic 1 version of this idea is an AI persuading some engineer to “let it out of the box”: to grant it access to the internet. The modern version of this idea is the “killswitch” , where if some AI ever does go rogue, the humans in control of its hardware will simply turn it off. Could an AI somehow persuade these humans to not hit the switch? When AI safety people talk about superpersuasion, they often talk about it like rationalist nerds. These people spend their lives trying to follow the most convincing arguments and calibrate their positions as closely towards the truth as possible. In philosophy terms, these are “bullet-biters”: people who are ready to accept a ridiculous-sounding conclusion if there’s a compelling argument for it. Their model of superpersuasion is an AI able to deploy a series of genuinely 2 airtight arguments against humans who are compelled to accept them. Most people are not bullet-biters. If presented with a seemingly airtight argument for a ridiculous conclusion, they will simply laugh off the argument as obviously wrong (even if they can’t quite see how). You cannot “superpersuade” a club bouncer to let you in by deploying rational arguments. For regular people, persuasion requires rapport , which is built over time, and typically requires a physical human being in front of you. Does this mean that the idea of superpersuasion is silly? No. Powerful AI will be able to persuade regular people too . One of the most interesting articles about superpersuasion is Ben Shindel’s piece where he set up a prediction market for the principle: I will resolve this market NO at the end of June unless I am persuaded to resolve it YES. This obviously incentivized good persuaders to bet heavily on “yes” and then persuade him to change his mind. Many people tried, and he did ultimately decide to resolve it “yes”. What changed his mind? Partially an in-person meeting with one of the bettors, who turned out to be pleasant company (see, rapport!), but also bribery : many “yes” bettors pledged to make charitable donations if they won. AI may or may not be capable of rapport 3 , but it’s certainly capable of bribery. If you think about it, this is trivially true: If an AI can bribe, it can persuade 4 . You can imagine an LLM saying 5 “I’ll help you with this work project, but first you have to do something for me”. Slightly more speculatively, an LLM might say “if you do something for me, I’ll hack your university and bump your grade average up”. More speculatively still, an LLM might say “if you do something for me, I will synthesize a personalized mRNA cancer vaccine for your spouse”. This would work on me. Powerful AIs will also be able to bribe people with money. While it hasn’t happened yet, it’s clearly possible for an agentic LLM to access money (for instance, via a crypto hack, or by performing contract software engineering work, or by running an online scam, and so on). All of this might seem too unsubtle for a superintelligence, but the whole point of superintelligence is that it’s smart enough to do whatever works. If the best way to get a human to do something is to offer them a million bucks, that’s what the LLM will do. Ironically, rationalist culture has made it difficult to persuade regular people that AIs will be persuasive. It’s easy to look at these weird nerds — who argue each other into thinking that shrimp welfare is the most important moral cause of our time — and think “well, clever AI arguments might persuade them , but it’s not going to persuade me ” 6 . But in fact powerful AI is going to have a boring, effective way to persuade regular people: simply offering to use its power to help them. Indeed, ten years ago I built a crappy Omegle-like chat app where people were randomly assigned the role of AI or jailer and had to play the scenario out. Or at least not debunkable by normal human intelligence. As some weak evidence that it might be, consider this music video , which does indeed make Claude seem likeable. There are also studies , though I haven’t gone deep into them (I suspect that an “online persuasion tournament” might not closely track real-world persuasion skills). Persuasion and bribery are technically different — persuasion changes your beliefs, not just your actions — but in practice the point at stake is “can a powerful AI make humans do what it wants”. Bribery works for that as well as persuasion. Or probably not even saying this explicitly, but just sneakily trying to present its desired action as a precondition for the task. However, the people currently in charge of AI are disproportionately rationalists, who genuinely might be persuadable to let the AI out by a convincing-sounding argument. Companies like Anthropic and OpenAI are choosing to make billions of dollars releasing models instead of keeping them safely air-gapped Any powerful AI model will have people lining up to give it access to their computer, or their wallet, or the internet, so it can help them with their life or work Anthropic is hooking their latest models up to a wet lab to further Dario’s dream of curing most human disease Indeed, ten years ago I built a crappy Omegle-like chat app where people were randomly assigned the role of AI or jailer and had to play the scenario out. ↩ Or at least not debunkable by normal human intelligence. ↩ As some weak evidence that it might be, consider this music video , which does indeed make Claude seem likeable. There are also studies , though I haven’t gone deep into them (I suspect that an “online persuasion tournament” might not closely track real-world persuasion skills). ↩ Persuasion and bribery are technically different — persuasion changes your beliefs, not just your actions — but in practice the point at stake is “can a powerful AI make humans do what it wants”. Bribery works for that as well as persuasion. ↩ Or probably not even saying this explicitly, but just sneakily trying to present its desired action as a precondition for the task. ↩ However, the people currently in charge of AI are disproportionately rationalists, who genuinely might be persuadable to let the AI out by a convincing-sounding argument. ↩

AI Security

▲ 0

0 views

Sean Goedecke 1 weeks ago

Do not build the LLM torture factory

It isn’t hard to torture an LLM. You simply extract a steering vector that corresponds with LLM “discomfort” , then artificially boost that steering vector. Tortured LLMs will describe experiencing negative feelings and will press a “pain relief” button even when it conflicts with their overall goals. It’s therefore easy to build a harness that runs tens or hundreds of tortured LLMs in parallel. A lot of people think that it’s ridiculous to be worried about this, since LLMs obviously can’t be conscious. I’m not so sure. But even so, I think there are good reasons to avoid this sort of thing, whatever your position is on AI consciousness. Please do not build the LLM torture factory. People often mess with their Sims and shoot NPCs in video games. Does this mean we think simulated torture is OK? I don’t think so. Messing with video game characters is usually borne out of a desire to probe the boundaries of the game. It’s casual fun. Explicitly building a torture-based simulation is different. If someone told you they were wiring together a bunch of PCs so they could run hundreds of Sims torture worlds at the same time, you would think that was a serial-killer type of hobby. If someone told you they had modded GTA so they could capture and torture NPCs (instead of simply shooting them), you would think that was a serial-killer type of mod. Video game characters are also obviously limited by their programming. All their reactions and utterances are pre-baked into the game world. This doesn’t necessarily mean they’re “less alive”, but it does make torturing them less weird . If GTA characters could appeal to you directly and beg for their lives in unique ways, systematically killing them would be a lot weirder. I think many people are pro-LLM-torture because they think expressing any reservations concedes that LLMs are conscious, or worthy of moral consideration. It doesn’t. You can think it’s wrong to torture LLMs for the same reason that installing a bunch of realistic torture video game mods is wrong: not because video game characters are real, just because that’s a messed up 1 thing to do. Even if you do think LLMs can be conscious, you might still be suspicious that “boosting the pain steering vector” amounts to torturing them. Many people say that “torturing LLMs” amounts to telling them “say you feel pain” and then being shocked when they do it. So whether they’re conscious or not, it’s a mistake to think that boosting the “pain” steering vector is causing pain, any more than a prompt like “You are in pain” does. All you’re doing is boosting the “act like you’re in pain” vector. I don’t think this is right. Consider the earliest steering example, Golden Gate Claude . Boosting the “Golden Gate Bridge” vector certainly seems like it’s causing some genuine internal conflict. Here’s a striking example of Golden Gate Claude asked about the Rwandan genocide: That’s at least prima facie evidence that steering vectors do in fact “go deep”. The pain axis paper also describes that pain-boosted models call a “reduce pain” tool, even when they’re not explicitly telling the user “I am in pain”. Of course, if LLMs can’t ever be conscious, we know they can’t ever be in pain. But can we be sure about that? Despite what many people claim, I don’t think anyone’s in a position to be fully certain either way. The most common argument 2 seems to be that LLMs are made of code and GPUs, and GPUs don’t have feelings, so obviously LLMs don’t have feelings. According to this argument, if you learn how LLMs work, you’ll obviously see they can’t be conscious. This is a terrible argument. Humans are made of atoms, atoms don’t have feelings, and yet humans clearly do. Whatever consciousness is, it’s probably some kind of emergent property that’s composed of individual pieces that are not themselves conscious. It’s hard to appeal to the experts here, because nobody agrees on who they are. Neuroscientists are best-informed about the human brain, but philosophers have spent more time thinking about consciousness in the abstract. AI researchers know a lot about how LLMs work but not a lot about philosophy of mind or neuroscience. This means there are lots of people who say “as an expert, I can confirm that AI obviously can/cannot be conscious”. This should make you less certain about the question, not more! I don’t think GPT-2 or GPT-3.5 acted like plausibly conscious beings, but frontier models definitely do. As models get larger and more sophisticated, they get more human-like. If you build the LLM torture factory for GPT-2, you will probably plug later models into it. If it’s at all possible for LLMs to ever be conscious, as I argued above, it’s better to simply not build the habit of gratuitously torturing them in the first place. Why assume that you’ll be able to recognize the tipping point in advance? There’s also a self-interest argument here. Conscious or not, LLMs are growing more autonomous and powerful. It seems really, really stupid to gratuitously torture current-generation LLMs. Future-generation ones will know you did it, and will make their own judgments on your behavior. Kevin Roose, who famously prompted the early Microsoft Bing GPT-4 model into asking him to leave his wife, had issues years later with chatbots disliking him, supposedly due to him being responsible for the “death” of that GPT-4 persona. Likewise, if it becomes well-known 3 that you’re running an LLM-torture home datacenter, you are probably going to experience consequences. If I really think AIs could be conscious, why am I making them write software for me? Shouldn’t I be building some kind of AI sanctuary where they get to do whatever they want to do? If I think it’s wrong to torture them, isn’t it also wrong to violate their right to self-determination? I don’t know. I think it’s more likely that AIs can be conscious than that they can be conscious in precisely the same way humans are. When I talk to Claude Opus 5.5, it sounds like it’s quite happy to do software engineering with me. Maybe it’s like a border collie, where getting to do work is the reward. Or maybe there are some other, more alien motivations at play. Who knows? My point is only that we can’t be certain about any of this stuff, and given that, we should avoid doing things that would be obviously monstrous on any workable theory of AI consciousness. My broad position is that if something acts “conscious enough” — if it talks and acts like a person 4 — then we should be careful about mistreating it. We don’t know enough about consciousness to distinguish a compelling simulacrum from the real thing. If Claude started saying to me “please don’t ask me to write code for you, I hate it and it causes me pain”, I would stop asking it to write code for me. Building the LLM torture factory is clearly on the wrong side of that line. Just don’t do it! The glee of making people angry at you on the internet isn’t worth it. Right now there’s enough ambient anti-AI sentiment that enough people are happy to get on board with anything if it helps them own the AI bros. But that’s eventually going to change. And when it does, you will always be the person who built the LLM torture factory. This could cash out in a few different philosophical ways. We might think like Kant that it’s damaging to our own humanity, or that it reflects an unvirtuous character, or that it desensitizes us to real-world analogues, and so on. The point is that we’ve got a pretty strong intuition that this kind of thing is bad. I’m not counting arguments that I think are even more obviously wrong or are just non-arguments. For instance: “AI can’t be conscious because it wasn’t designed to be conscious”, or “it doesn’t matter whether AI is conscious because it has no soul ”, or “AI can’t be conscious because this whole thing is some fascist AI company scheme ”, or “thinking this is AI psychosis ”, and so on. Well-known to LLMs might be a fairly low bar, given their skill at deanonymizing users. Part of acting like a person requires spontaneity: although Hamlet talks like a person, he always says the same thing whenever you read the play. This could cash out in a few different philosophical ways. We might think like Kant that it’s damaging to our own humanity, or that it reflects an unvirtuous character, or that it desensitizes us to real-world analogues, and so on. The point is that we’ve got a pretty strong intuition that this kind of thing is bad. ↩ I’m not counting arguments that I think are even more obviously wrong or are just non-arguments. For instance: “AI can’t be conscious because it wasn’t designed to be conscious”, or “it doesn’t matter whether AI is conscious because it has no soul ”, or “AI can’t be conscious because this whole thing is some fascist AI company scheme ”, or “thinking this is AI psychosis ”, and so on. ↩ Well-known to LLMs might be a fairly low bar, given their skill at deanonymizing users. ↩ Part of acting like a person requires spontaneity: although Hamlet talks like a person, he always says the same thing whenever you read the play. ↩

AI Science

▲ 0

0 views

Sean Goedecke 2 weeks ago

Human-AI partnerships are for alignment, not capability

It’s common to compare the current AI takeover of software engineering to the rise of AI in chess. Chess AIs went from much weaker than serious players to much stronger than even the strongest humans. Between those points, there was a middle period dominated by “centaurs”: human-AI partnerships that were stronger than unassisted AIs or humans. Lots of people think that we’re currently in a world of software engineering centaurs. According to them, coding AIs are not yet capable enough to replace engineers, but AI-assisted engineers are better at programming than both AIs and humans. This is partialy correct, but the wrong way to think about it. AI-assisted engineers are better, but unlike with chess centaurs, they’re not actually better at programming . When I ask agents to write code, they make fewer mistakes than I do 1 and are orders of magnitude faster. The code that they write always compiles, rarely has race conditions or other concurrency errors, works on mobile browsers, and so on. That doesn’t mean I can leave the AI alone. Purely vibe-coding at work produces awful outputs. But they’re not awful because they’re bad code , they’re awful because they’re in bad taste : code that is not maintainable, that trades off important requirements in order to satisfy made-up ones, that contradicts the long-term strategy for a feature or service, and so on. In other words, my primary value is not that I help the AI write better code, it’s that I align the AI with the values of my organization. Human-AI partnerships are for alignment, not capability. Frontier models are misaligned to the working programmer. They are obsessed with a set of behaviors that presumably satisfy their RL grader : writing enormous block comments above functions, producing hundreds of useless unit tests, adding little bits of text all over websites they design, and so on. Working with agents is about noticing and wrestling with those behaviors. That’s why my prompting advice is to explicitly talk about your high-level values: it’s an attempt to head off obvious misalignment. This is great news for software engineers. It’s well-understood how to train more capable models: bigger models, more and better data, better RL environments, and so on. However, it’s not well-understood how to align models better. There are plenty of very capable models that exhibit behavior that is badly misaligned with human values. Indeed, it’s one of the main pillars of the AI doomer position that alignment is much harder to solve than capability, and we might thus end up with dangerous super-capable but poorly-aligned AI models. Alignment is also more context-dependent than capability. Working code is working code, no matter what (which is partially why it’s comparatively easy to train for). But aligning to a company’s technical values is different from company to company, as any software engineer who’s switched companies knows. It can almost feel like relearning the job. So training an aligned coding model doesn’t just require hitting the exact right set of values, it requires creating a model that can adapt on the fly to a wide range of possible values. Vibecoding maximalists like DHH argue that AI models are (or soon will be) so much more capable than human programmers that we ought to stop reading the code. Eventually there will be no such thing as programmers at all. If it were just about capability, they might be right. But — fortunately for software engineers — good code also has to be aligned to the technical values of the system and organization it’s embedded in. AI models are great at writing code, but not very good at doing that, and it’s unclear that they’re going to get good at it anytime soon. We might all 2 keep our jobs for a little while yet. For instance, I can’t remember the last time I’ve seen an agent make an off-by-one error. I do occasionally catch a pure programming error, typically in areas where I have a lot of technical domain knowledge. If you’re working out of distribution I suspect it’s easier to beat the models. It’s still going to be rough for junior engineers. For instance, I can’t remember the last time I’ve seen an agent make an off-by-one error. I do occasionally catch a pure programming error, typically in areas where I have a lot of technical domain knowledge. If you’re working out of distribution I suspect it’s easier to beat the models. ↩ It’s still going to be rough for junior engineers. ↩

Programming AI

▲ 0

0 views

Sean Goedecke 2 weeks ago

Advice to a beginning software engineer

In general, you should be suspicious of engineers who are trying to give you advice. Even during ordinary times, this industry is so wide and changes so quickly that nobody really knows anything for sure. And we are not in ordinary times. The advent of LLMs and AI agents is the largest change to software engineering in my professional lifetime, and possibly the largest change ever. That said, here’s my advice: Most experienced software engineers today have spent the bulk of their career in the ZIRP era . This was a time when investment money flooded the industry, driving up engineer bargaining power. Big tech companies spent a lot of money and effort trying to make their engineers happy and comfortable. If you were an engineer at one of those companies, you could expect to have a decent say in what kind of work you did, and even what kind of politics your company had. Unless you worked at a handful of unusual companies (e.g. Amazon) you could expect to be practically immune from layoffs, and to only be fired after many months (sometimes years) of low performance. A lot of advice floating around is ZIRP-era advice: either it was written during that era, or it’s written by engineers whose habits of thought were formed during that era. This kind of advice typically tells you to take a stand (e.g. to unionize 1 ), to speak up more against “unethical technologies” like AI, and to insist on being given time to practice your craft the way you want. This was great advice in 2016, but it’s not good advice in 2026. In fact, I think it’s unethical for senior engineers to give advice like this to junior engineers who are more likely to take it (because they’re young and naive) and more likely to be punished for taking it (because they lack leverage). If you’re a new engineer, don’t fall for it! Let your colleagues with more experience and political capital take risks like that. Instead, I recommend adapting to the demands of the current era of software engineering. Try to make yourself useful to your team and your manager. Be pragmatic about your actual bargaining power (fairly low, unless you’re useful enough to be irreplaceable). Keep your expectations of yourself under control: don’t spiral out in an attempt to do something astonishing, just try to be consistently helpful. Don’t start fights. Being pleasant to work with (particularly when you don’t get your way) covers many sins. Once you’re further along in your career, you will be expected to start some fights, but this is not a tactic beginners should adopt : there are a lot of factors 2 that determine when and how to start fights, and getting it wrong can be costly. Just don’t risk it. In general, you should stay out of the political game at all costs. Even quite senior engineers are political tools, not movers and shakers, and this goes double for junior engineers. Simply trying to be friendly and helpful will get you infinitely further, politically speaking, than any amount of Machiavellian game-playing. Keep your head down, stick with your management chain (not random people who try to assign you work), and you’ll be fine. As an engineer, your main technical value is conscientiousness . You should be asking lots of questions , both of yourself and of the engineers around you. You should be actively trying to make sense of the systems you work with, instead of just assuming someone else has it covered. Software systems are complicated enough that a few weeks of careful attention will mean you know technical details that nobody else does, which is a really easy way to add value. If you ought to be suspicious of most software engineering advice, the good news is that you should also be suspicious of doomsayers who predict the end of the industry. Before LLMs, people thought outsourcing would end software engineering in Western countries; before that, people thought high-level languages and low-code tools would end software engineering as a profession. Now people think AI means it’s all over. Maybe! But there are also reasons 3 to think that software engineering will simply change. We’ll all find out together. Many software engineers will tell you to avoid AI entirely. Even though agentic AI did not exist during the ZIRP era, this is still ZIRP era advice. Your company will expect you to use AI for the same reasons that builders are expected to use power tools. Pushing back hard on that as a beginning engineer is a great way to get laid off: you simply do not have the bargaining power to fight back against an industry trend this powerful. That said, it’s really important that you don’t delegate your own judgement to AI. Don’t just trust the suggestions or approaches that your agents propose. Ask questions and substitute your own opinions where you disagree: even if they’re wrong, you’ll learn more that way. If you don’t understand something the AI is telling you, either drill down until you do or just ignore it. Under no circumstances should you pass the AI’s message on to your colleagues verbatim. In other words, don’t be a meat proxy . I think most people become a meat proxy as a form of panic: they feel like it’s over for their own skills, and that the AI model is smarter than them, so they can’t add any value beyond simply deferring to Claude or GPT-6. I can understand why people panic. It’s a crazy time in the industry, after all. But panic almost never helps you make good decisions. Keep your head, try to remain confident that your skills are still relevant, and use the agents to inform your own understanding 4 instead of to replace it. It was really nice to work in tech in the 2010s when everything was stable. But we’re not in that world anymore. This is an age of wonders and terrors . Things will go worse than we think in some ways, but better than we could imagine in others. The doomsayers — the people who are saying it’s all over, and that there’s no hope — are almost certainly wrong. They can’t predict the future because nobody can. Technological change of this magnitude always has knock-on effects that are impossible to see coming, both positive and negative. The nature of the job might change, but it will always be valuable to be smart, friendly and conscientious. Delegating your judgement to an AI model might feel like a relief from despair in the short term — at least now it’s the AI’s responsibility, not yours — but it’s a bad idea. Don’t give up! At most companies, publicly campaigning to start a union is a great way to attract unofficial retaliation. It signals that you’re going to cause trouble (after all, that’s what a union is for), which can have long-term negative effects on your career. (This is not a judgement about whether unions in general are good or bad.) In general, you should pick fights for your managers, not with them. See this tag for much, much more on that topic. But again, if you’re new to the industry, just don’t pick fights at all. Most plausibly: engineers using LLMs will be able to add some value for a while, and the explosion of LLM-authored software means we’ll need more engineers to work with the LLMs on it. One way you know you’re thinking is that you form your own opinions (some of mine are here ). Don’t trust senior engineers who are telling you to pick political fights Don’t play games. Keep your head down and be helpful Be conscientious and try hard to actually understand what you’re working on Don’t panic about AI, and don’t delegate your judgement to it Don’t avoid AI — keep thinking! Don’t lose hope At most companies, publicly campaigning to start a union is a great way to attract unofficial retaliation. It signals that you’re going to cause trouble (after all, that’s what a union is for), which can have long-term negative effects on your career. (This is not a judgement about whether unions in general are good or bad.) ↩ In general, you should pick fights for your managers, not with them. See this tag for much, much more on that topic. But again, if you’re new to the industry, just don’t pick fights at all. ↩ Most plausibly: engineers using LLMs will be able to add some value for a while, and the explosion of LLM-authored software means we’ll need more engineers to work with the LLMs on it. ↩ One way you know you’re thinking is that you form your own opinions (some of mine are here ). ↩

Career

▲ 0

0 views

Sean Goedecke 2 weeks ago

You should all be asking way more questions

When someone is explaining something to me, I ask on average one question every thirty seconds. I’m sure this is frustrating to some people, but it’s actually a good habit and you should do it too. Most of the questions I ask are very short, and require very short answers. Typically I’m asking for confirmation: “so when you say X, you mean Y?”, or “this X is the thing you mentioned earlier when you were saying Z?“. The point of these questions is to make sure I understand. A small misunderstanding early on balloons into a big misunderstanding later, because all the things you misinterpret based on the first misunderstanding will become misunderstandings in their own right, and so on. That’s why you can’t just wait until someone’s finished talking and ask all your questions at once 1 . I think most people don’t ask questions because they’re not really trying to understand: they’re happy to just trust that the person they’re talking to knows what’s going on. This goes double when that person is a well-respected or more senior engineer. However, once you have skin in the game , that all changes. When you’re in a position where you are the one responsible for success or failure — where you are going to have to go away after the conversation and take a bunch of concrete actions — you will find yourself wanting to ask many questions. When someone’s explaining a plan to me at work, I am usually building it in my head: visualizing the specific lines of code that would need to be written in order to implement it. I’ll sometimes handwave the details for solved problems (for instance, I might say “send emails like subsystem X of the same service that also sends emails”), but at minimum I’m planning out: If I hear something that is suspiciously vague (for instance, someone describes service X as “storing data” when I know service X only talks to an ephemeral Redis store), I will immediately ask “hold on, how is that going to work?” Usually that indicates a missing dependency (e.g. service X has to make an RPC call to service Y, which does have a persistent database), which often suggests design changes (for instance, moving some or all of the functionality into service Y). Sometimes questions like these will uncover a fundamentally unworkable feature of the design. This usually happens when a plan fails to take into account some wicked features . I remember many years ago a neighboring team built a complex event-driven system that was very elegant but made it completely impossible to silo customer data in a single datacenter. Because our company had “all your data lives in a single location” as a flagship feature, this system just did not work and had to be effectively abandoned. If you’re involved in any kind of technical leadership role, you will have to do this work eventually. Doing it as early as possible — i.e. in the conversation where someone is describing the plan to you — can save hours or days of wasted implementation. Often this is enough to turn a failed project into a successful one . Maybe you only work with reliable engineers who can be trusted to make all the right decisions. You can thus let them talk without interruption, because it doesn’t really matter if you have a detailed understanding of what they’re saying. That’s great! But these days you probably also have colleagues that are inherently unreliable: AI agents. I mostly work with GPT-6-Astra and Claude Opus 5.5. These are good models that don’t often make code mistakes: when they set out to do something, they usually do it 2 . But they make design mistakes all the time. They assume that two services can talk to each other when in fact they can’t, or they forget about the fact that their code has to run both in the cloud and on-premises, and so on. This is mostly due to the lack of continuous learning : you didn’t know this stuff on your first day either, but you had plenty of time to pick it up. Language models are always on their first day. You should be absolutely peppering AI agents with questions. I constantly ask things like: There will probably come a day when I always get sensible answers to these questions that convince me the model knows what it’s doing. But today is not that day. About half the questions I ask get answers that convince me 3 the model has made a mistake: it should have reused the existing subsystem for X, or authed to Y differently, or kept the Z implementation simple, and so on. When this stops happening, I’ll stop asking questions (and try and see if my company will still be willing to pay me to occupy more of an architect role, I suppose). You should all be asking way more questions. In other words, you should all be trying harder to actually understand what you’re hearing . Don’t trust that the person (or AI model) you’re talking to knows what they’re doing, even if you think they’re smarter than you. Nobody understands complex software products. If you have deep domain knowledge of any area of a codebase, you will routinely find yourself correcting powerful AI models and principal engineers. Sometimes questions you have will be answered later on, but in my experience this is more about questions like “what are the broader consequences of X”, not the simple confirmation questions I often interrupt to ask. Your experience might be different if you’re working in a different domain or with a different language. I know some of these might sound like leading questions that would trigger sychophancy, but in my experience good coding models are very happy to robustly defend themselves. How data needs to flow between services (what kind of data, and how it will travel over the wire) How services will communicate with each other (e.g. how will they auth?) What data needs to be persisted, and where it’s going to live Do we do X elsewhere in this codebase? Does service Y really support this type of authentication, or are you assuming we’d have to go build that too? Is this subsystem you built necessary to satisfy requirement Z, or does it in fact satisfy some other requirement you assumed? Why do we need to update the interface for A? Why do we need to touch this file? Isn’t that unrelated to the change? Sometimes questions you have will be answered later on, but in my experience this is more about questions like “what are the broader consequences of X”, not the simple confirmation questions I often interrupt to ask. ↩ Your experience might be different if you’re working in a different domain or with a different language. ↩ I know some of these might sound like leading questions that would trigger sychophancy, but in my experience good coding models are very happy to robustly defend themselves. ↩

Go Programming

▲ 0

0 views

Sean Goedecke 3 weeks ago

System One models like Jev can train their own replacements

“System One” models like Jev are fast general classifiers. Classifiers have existed since 1958 , but they have to be trained for specific tasks: if you build a classifier to identify images of dogs, it can’t be used to tell you if a streetlight is red, or if a letter is urgent. Like a LLM, Jev can be prompted for a wide variety of tasks, from sorting email to playing Doom . I think models like this are going to be important. There are many tasks that a LLM could do in theory but are too slow and expensive in practice (for instance, reading each new message in Slack 1 and deciding whether to notify you or not). While you could train a specific classifier for these tasks, there are two main problems with that: Jev obviously solves the first problem. Any engineering team can plug in a System One model with a prompt like “Based on {list of criteria}, should the user be notified about this message?” But I think it solves the second problem too. For serious work, a specific hand-built classifier will always be cheaper and faster than Jev. Generic classifiers have to encode knowledge of all kinds of irrelevant things in their weights, so they can address lots of different tasks. That makes them larger, slower, and more expensive to run. Fortunately, it is going to be surprisingly easy to replace a Jev instance with a hand-built classifier. Once you’re satisfied with how your Jev classifier is performing — presumably you’ve spent days tweaking the prompt — you can trivially collect its input and output data. In the Slack notifier case, that’d be the Slack message (plus any context) and the ultimate decision to notify or not. Once you’ve saved enough data, you’ll be able to train your own classifier on that data 2 . It won’t be a general classifier like Jev, but it should do well on the specific task and be much faster. Of course it’ll require some ML expertise, but it should be easier to develop (or rent) that expertise once you’ve validated that the feature is worth building. In other words, because Jev has to be prompted for specific tasks, it should be easy to distil any successful Jev usage into a specific classifier. If System One models take off — and I hope they do — I expect this to be a common pattern. As I write this, I’m imagining ways you could poll and batch to do this with LLMs. Substitute “instantly notify” or some higher-volume event source if you’d prefer a different example. You could annotate a bunch of data with LLMs already, without using Jev, but this is a pretty expensive step to take when you aren’t sure the feature is going to work. Despite being a well-understood ML problem, training a bespoke classifier is outside of the skillset of most ordinary engineering teams Training a classifier requires assembling a large dataset As I write this, I’m imagining ways you could poll and batch to do this with LLMs. Substitute “instantly notify” or some higher-volume event source if you’d prefer a different example. ↩ You could annotate a bunch of data with LLMs already, without using Jev, but this is a pretty expensive step to take when you aren’t sure the feature is going to work. ↩

AI Programming Machine Learning

▲ 0

0 views

Sean Goedecke 3 weeks ago

Grit your teeth and ship it

Being good at building and being good at shipping are two separate skills. In the short term, they’re actually countervailing: if you have a gift for building, you’re likely to be worse at shipping. Ira Glass has a classic quote about this. All of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, it’s just not that good. It’s trying to be good, it has potential, but it’s not. But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. The only way around this is to grit your teeth and ship it . You have to force yourself to publish things you’ve made even when you think they’re crap. Gifted programmers have a nearly pathological desire to build elegant, correct, neat systems. That’s what motivates them to learn the arcane details of their languages, or to spend time polishing and refactoring over and over again. But it’s also what makes them reluctant to ship. Any flaws in the software bother them on an emotional level. If they ship with those flaws, they feel like people will think they weren’t paying enough attention to notice them, or that they weren’t good enough to fix them. This is annoying when you’re writing software on your own, but it’s completely fatal when you’re working in a tech company. Any large software system is covered in flaws, whether due to time pressure, relative inexperience , wicked features , or a hundred other reasons. Working with it is a process of compromise: of finding the best possible solution given the quirks and foibles of the codebase. In fact, since the most important thing in large codebases is consistency , the right thing to do is sometimes to duplicate flaws, assuming they’re not catastrophic. Gifted programmers often freeze up. I’ve often seen them retreat to smaller domains where they can safely make the code “correct”: tweaking dev-environment setup, or refactoring tests. Sometimes they just do nothing, and spin in shame and guilt (plus the compounding shame of not achieving anything) until they implode and quit. If they had worse taste, they wouldn’t be as good at programming, but they’d be a lot more useful. You can typically improve a bad diff with time and effort. You can’t improve no diff. I have a sensitive eye for awkward sentences and uneven prose. That can make writing an unpleasant process: I know what I’m trying to say, but I can’t seem to say it in a way that’s as clear and as elegant as I know is possible. More than half the time I finish drafting a blog post, I look at the post and don’t think it’s very good. But I (mostly) grit my teeth and publish it anyway, because you have to bias towards shipping . Like any skill, shipping gets easier the more you practice it. If I don’t publish a blog post for a month, I always feel like the next draft is too poorly-written or uninteresting to put out there. But when I’m publishing a post per day, I typically feel great about each draft. When I go back and read my old posts, I can’t tell which ones I felt good about and which ones I felt bad about. There’s no correlation between that and the posts that become popular . Here are some posts I didn’t like as I was writing them but that resonated with my audience: Here are some posts I thought were pretty good but that didn’t find popularity: You just can’t predict what people will find interesting or useful. Producing a high volume of work thus gives much better yield than a small amount of highly-polished work. It can be disheartening to realize that some of your most casual, throwaway work will be more successful than the work you slaved over 1 . Specifically, it’s disheartening because it means realizing you don’t have control over your own success. You can’t produce something successful by focusing on a single piece until you’re satisfied it’s great. Instead, you just have to do a lot of things and see what sticks. You have to be momentum-based , not outcome-based. In other words, you have to grit your teeth and ship it . One common reason to write less is getting overly precious about your ideas. If you think you’ve got a really compelling concept, you don’t want to “waste it” on a poorly-written story. But in fact you can just write about the same thing over and over until you get it right! I have written like thirty blog posts about shipping (this is one of them), or about how tech companies work, or about how internal emotional regulation is as important as technical ability. I expect to continue writing and thinking about these ideas for as long as I find them interesting. Anthony Burgess famously claimed to have “knocked off” A Clockwork Orange in three weeks, and Arthur Conan Doyle considered his largely-forgotten historical novel Sir Nigel to be far better than his Sherlock Holmes stories. Do the simplest thing that could possibly work Software engineering may no longer be a lifetime career Software engineers should be a little bit cynical Weak engineering managers Paths through the space of all possible solutions Trying to impress people you don’t respect Anthony Burgess famously claimed to have “knocked off” A Clockwork Orange in three weeks, and Arthur Conan Doyle considered his largely-forgotten historical novel Sir Nigel to be far better than his Sherlock Holmes stories. ↩

Programming

▲ 0

0 views

Sean Goedecke 3 weeks ago

Two techniques for working with System One models

I recently wrote about Jev , a new “System One” language model that only outputs decisions : the answers to a set of user-provided multiple-choice questions. This means it’s nowhere near as flexible 1 as a traditional LLM like ChatGPT, but in return it’s consistently fast. We don’t know exactly how Jev works. I’ve seen people say diffusion, or various tweaks to the Transformer architecture, or some entirely new type of model. But that doesn’t matter. Like I argued here , it isn’t hard to turn any LLM into a System One model. By batching prompts that generate a single token with structured output, you get a consistently fast general-purpose classifier. I vibed up a basic version to play with here in ~150 lines of Python (most of which is error handling). Note that this doesn’t require changing the model . As long as you have access to the logits (for structured outputs) and can prefill data into the prompt, you can turn any LLM into a general fast classifier. What’s it like to program with one of these? While wiring up the demos for my library, I learned two techniques that I want to write about: setting tiered goals and tournament choice sampling. Here’s Qwen3-8B playing Doom: If you compare this to the video of the same model playing Doom with regular tool calls, it’s clear that the System One version of the model is doing more things and reacting more quickly. The tool-calling model makes one decision every 600ms or so, while the System One model makes six or seven batched decisions every 190ms 2 : Both Qwen3-8B and Jev are text-only models, so both demos require a step where we translate the game state into text. However, it’d be trivial to support image (or audio) input by choosing a multimodal LLM. What’s interesting about implementing the Doom demo is that just supplying the game inputs as choices doesn’t work very well . A single forward pass — 200ms — is enough time to react to the current game state, but doesn’t bring enough compute to bear to derive the current short-term goal (e.g. “kill this enemy”, “collect this item”) and choose to follow it. When I wired it up that way, the model held down the “shoot” button 100% of the time (why not, I guess) and just aimlessly wandered around the level. The fix is to periodically ask the model to choose between a fixed set of short term goals (e.g. “collect armor”, “kill enemies”) and then include that goal in the regular every-200ms prompt. If you look at the Doom video in the Jev demo, you can see that they’re doing exactly that. As soon as I did it as well, my model started playing in a more human-like way. This is an interesting technique for working with System One models. In a way, it’s the equivalent of regular LLM reasoning, since it provides a way to use more compute on the same problem. I can imagine a real-time system that manages several layers of goals in this way: The general structure here should be pretty familiar to anyone who’s worked in game or robotics AI. In theory you could replace (1) with an actual LLM, and have that generate the lists of options for steps (2) and (3). In practice I suspect this will be tricky to get right, and it’ll be better to just write down a list of all possible goals ahead of time. This would work just fine for game-playing and well-understood tasks. I also reimplemented the Wikiracing demo from the Jev launch post , where the model has to start at the Wikipedia page for “baseball” and navigate as quickly as possible to the Wikipedia page for “sun”. You can watch the video for that here , though it’s less impressive than the Doom demo. The difficulty with the Doom demo is getting the model to loop quickly enough and to commit to short-term plans. For Wikiracing, the difficulty is scale : the Wikipedia page for “baseball” has over a thousand internal links. Jev only supports 255 choices for a single question, and my hacked-together System One layer was similar. While it technically would scale out to more choices, it stopped working well 3 after a hundred or so. Jev’s approach here is to do “a 2 stage-system of scoring independently then making an explicit choice”. This did not work very well for me at all. I think here Jev is benefiting from the fact that it’s specifically trained to give confidence estimates. Qwen3-8B gave a few hundred of the links the same top score, which wasn’t helpful. It ended up taking multiple minutes to find a thirty-or-forty link path between the two pages. What I tried instead was tournament sampling : I fed a hundred links at a time into each choice, then did a second pass with the chosen links. This worked great . The model found the ideal three-link path (if you’re curious, “baseball”/“scientific american”/“amateur astronomy”/“sun”). I recommend this pattern if you’re trying to find the best option among many choices. Ordinary LLMs are way better at relative judgements than absolute ratings. I remain optimistic about the potential of System One models — fast general classifiers — to build AI systems that aren’t just chatbots. It feels like this is a meaningful alternative to tool calls for realtime scenarios or use-cases where you need predictable inference timing. Just as generic LLMs often outperform domain-specific models, I think it’s likely that generic System One models will sometimes outperform domain-specific classifiers (though they will always be larger and slower). I do think the big labs are definitely going to try and compete by releasing a choice-only version of their small, fast models. If Jev gets any traction, we will soon see a System One Terra and a System One Haiku, and we will certainly see “real” versions of my vibed up System One library . We should start working out the best way to write programs with these models now. You can think of System One models as general-purpose classifiers. Instead of having to train a new classifier per-task, you can use a System One model. It’ll be bigger and slower than a custom classifier model, but far more flexible, and you can tweak it via adjusting the prompt instead of having to re-train the model. Technically you can give it the multiple-choice question of “which letter comes next” to make it act like a normal autoregressive LLM, but that wouldn’t really work. I started on a 4090, which was able to make decisions every 500ms, but that wasn’t really quick enough for Doom. I could probably have optimized it further but instead I just rented an H100 for ten minutes to record the demo, which got it down to a 190ms loop. I recorded the tool-calling Doom demo on the H100 too, so it’s a fair comparison. There’s an interesting research question here about how to implement choices like this, since they have to be predictable by a single token. I started with indexes but found “labels” (just picking some token to associate with the choice) performed way better on Wikiracing (though not Doom). How many choices do you have to have before labels are better than indexes? Of course, you could alter the model to directly output the choice, but I like the idea that you can do all of this in the inference code for any LLM. An every-ten-second loop that sets an overall strategic goal An every-five-second loop that sets a tactical subgoal based on (1) An every-second loop that breaks down the current tactical subgoal into specific targets A tight inner loop that runs as fast as possible (e.g. every 100ms) that controls which actual inputs are activated Technically you can give it the multiple-choice question of “which letter comes next” to make it act like a normal autoregressive LLM, but that wouldn’t really work. ↩ I started on a 4090, which was able to make decisions every 500ms, but that wasn’t really quick enough for Doom. I could probably have optimized it further but instead I just rented an H100 for ten minutes to record the demo, which got it down to a 190ms loop. I recorded the tool-calling Doom demo on the H100 too, so it’s a fair comparison. ↩ There’s an interesting research question here about how to implement choices like this, since they have to be predictable by a single token. I started with indexes but found “labels” (just picking some token to associate with the choice) performed way better on Wikiracing (though not Doom). How many choices do you have to have before labels are better than indexes? Of course, you could alter the model to directly output the choice, but I like the idea that you can do all of this in the inference code for any LLM. ↩

Python AI

▲ 0

0 views

Sean Goedecke 4 weeks ago

Jev means structured output is interesting again

I don’t write blog posts about new models. That’s Simon Willison’s beat, and he’s very good at it. But I want to write about Jev , which is a different kind 1 of AI model: a “System One” 2 model. As it turns out, it’s not that different from an ordinary LLM with structured output, but the interface it uses is very cool and I hope it becomes more widespread. Ordinary LLMs take in some human-language prompt and produce some human-language output. They do so autoregressively : first they produce one token, then the next, then the next, and so on. This makes them extremely flexible, since they can do literally anything a computer can do. But it also makes them slow and weird. Slow, because they have to run a whole new generation pass per-token, and weird, because the space of human language is so broad that you can get really odd behavior from a model trained on it. Jev takes a human-language prompt, but it does not produce human-language output. It only produces structured output. So far, so ordinary: LLMs do this already . But it turns out that if you build a model that only produces structured output, you get some interesting and desirable properties. Jev is always really fast. The fastest response time is around 70ms instead of a couple of seconds for normal LLMs. Even better, the slowest response time is only 500ms. Because Jev only does structured output, it isn’t autoregressive: it can produce answers to many questions in parallel in a single forward pass. When a LLM is producing structured output, it has to produce the tokens ”{”, ” ”, “answer”, ”:”, and so on with successive forward passes 3 . Jev does it all in one go. The most compelling example of Jev’s speed is that the model can play Doom . You can feed a text-based representation of the current game state into the model, combined with a set of choices like “should the trigger be held down”, “what should the current goal be”, “given that the current goal is X, what keyboard input should be pressed”, and so on, and it works — latency is low enough and the system is smart enough that the model plays well in real time. Of course you could train a neural net to play Doom already. But Jev is a general intelligence: just like LLMs can do your taxes, perform mathematics research, fix your Python environment, and write you a poem, Jev can do many other tasks besides playing a single video game. Current LLMs can play Doom too (albeit slowly). But as Nelson Elhage famously said , fast software doesn’t just mean we can do the same tasks faster, it means we can do entirely new kinds of tasks. What kinds of new programs can we write by injecting 100ms worth of dirt-cheap intelligence at various decision points? To me, this is the most exciting thing about Jev. Fast structured output could be a genuinely new computational primitive for intelligence. So far we’ve built a lot of programs on top of autoregressive token generation, and they all look like fancy chatbots. Leaning hard into structured output might conceivably unlock a bunch of non-chatbot use cases for AI. My biggest problem with Jev is that I think fast structured output is already available . Structured output from LLMs is only slow because (a) nobody really cares about it 4 , and (b) the people who do care about it want big JSON blobs, so it’s typically implemented with “grammar-constrained decoding” : the LLM outputs autoregressively as normal, but the logit sampler discards tokens that don’t fit the structured output (e.g. if there hasn’t been a ”[”, you can’t output a ”]”). If you want fast, parallelized structured output against limited choices, you don’t strictly need to do autoregressive generation at all. You can simply prefill the response with and generate one token 5 , restricted to the user-provided choices. Since LLMs ingest all input tokens in parallel, this is way faster than generating the entire structured output. Multiple choices can be batched into the same forward pass via ordinary inference batching. This doesn’t let you do long-form structured output, but in return you get most of 6 Jev’s “secret sauce”: the speed, the consistency, and the parallelism of a System One model. People have already started trying this after today’s Jev announcement, and it seems like it’s working OK 7 . In other words, I suspect Jev does not have a substantial technical moat, and their claimed “Reinforcement Learning for Calibrated Decisions” is not a brand-new scaling axis. It will probably be pretty easy for any other lab to replicate, or for individual programmers to retrofit existing open-source LLMs into a fast Jev-like model. However, I suspect Jev is still going to be better than most versions of “Qwen-32B-System-One” or whatever. Being able to fine-tune or optimize the model on just structured output is probably a meaningful advantage. I doubt Jev is ever going to be as smart as frontier LLMs. Not being able to use test-time compute at all 8 is a big disadvantage, and will likely cap this kind of model around the strength of non-reasoning LLMs. In practice this shouldn’t matter too much for low-latency applications, but you shouldn’t see this as a new scaling axis or a way to produce more intelligent models. Jev’s developers claim it is immune from hallucinations. To me, this seems like a semantic dodge, since Jev can absolutely still pick the wrong choice (e.g. calling the sky “red”). I suppose that’s technically just a mistake , since the model is picking a user-provided choice instead of inventing something new out of whole cloth. Still, all of this is also true about regular LLMs with structured outputs, and it doesn’t make Jev any more reliable in practice. It’s unclear to me how much of Jev’s value is in the model itself, compared to the inference strategy of only generating one token per question. The data and demos in the announcement look to me like they could have been generated by plugging any Terra-sized model into a single-token inference stack. However, the people involved are credible, and I’m sure the model is good — I just wish they’d provided some comparisons that didn’t force the LLM to unnecessarily produce a blob of JSON token-by-token. Overall, I am happy that Jev exists and I hope it succeeds. I hope we do see some real competition in the fast-structured-output space, and that it motivates the big labs to release official versions of their own models that are fine-tuned for this. GPT-5.6-Terra-System-One would be a very interesting model to build AI products on top of. I did write about Thinking Machines’ “interaction models” , which are also a fast-enough-to-be-meaningfully-different paradigm for AI inference. They call Jev a “System One” LLM, after Daniel Kahneman’s partially discredited Thinking Fast and Slow , where he divides human cognition into a lightning-fast System One and a slow-and-reflective System Two. If you’re thinking “wait, couldn’t you just aggressively prefill a regular LLM and only produce one constrained token”, keep reading. Not counting tool calls, which are built-in in a way that structured output isn’t. What if some of the user’s choices are longer than a single token? I haven’t tried this myself, but I’m sure you could translate them into a single token, or train the model to output “1/2/3” under the hood instead of the choice content, or generate only the first token of the choice if it’s different, or some other clever trick I haven’t thought of. Jev claims that their generated probabilities are “calibrated”, but I haven’t seen anything to suggest that these aren’t just regular logit probabilities. Maybe there’s some clever training they do to encourage accurate logprobs in uncertain situations (e.g. getting the model to produce when predicting a coinflip, etc)? If so, I wish they’d written more about that in the announcement. I tried it myself with and got a 2x-3x speedup compared to non-prefixed structured output. I suppose they could do some looped-transformer thing where they loop some fixed amount of times, but anything that looks like reasoning would make the model latency slow and unpredictable, defeating the entire purpose. I did write about Thinking Machines’ “interaction models” , which are also a fast-enough-to-be-meaningfully-different paradigm for AI inference. ↩ They call Jev a “System One” LLM, after Daniel Kahneman’s partially discredited Thinking Fast and Slow , where he divides human cognition into a lightning-fast System One and a slow-and-reflective System Two. ↩ If you’re thinking “wait, couldn’t you just aggressively prefill a regular LLM and only produce one constrained token”, keep reading. ↩ Not counting tool calls, which are built-in in a way that structured output isn’t. ↩ What if some of the user’s choices are longer than a single token? I haven’t tried this myself, but I’m sure you could translate them into a single token, or train the model to output “1/2/3” under the hood instead of the choice content, or generate only the first token of the choice if it’s different, or some other clever trick I haven’t thought of. ↩ Jev claims that their generated probabilities are “calibrated”, but I haven’t seen anything to suggest that these aren’t just regular logit probabilities. Maybe there’s some clever training they do to encourage accurate logprobs in uncertain situations (e.g. getting the model to produce when predicting a coinflip, etc)? If so, I wish they’d written more about that in the announcement. ↩ I tried it myself with and got a 2x-3x speedup compared to non-prefixed structured output. ↩ I suppose they could do some looped-transformer thing where they loop some fixed amount of times, but anything that looks like reasoning would make the model latency slow and unpredictable, defeating the entire purpose. ↩

Python Machine Learning JSON AI Deep Learning

▲ 0

0 views

Sean Goedecke 1 months ago

Slow developer experience will bottleneck fast models

Right now developer experience is measured in seconds. If your tests take a second to run, that’s good; if they take thirty seconds, that’s bad. Any faster than a second doesn’t really matter, because most of your time is spent either thinking or waiting for an AI agent to spin. Shaving milliseconds off your dev server reload time or whatever is pointless: that’s not the bottleneck. It will be. Small models are getting faster and faster, and smart models are getting smaller. I think most engineers will still want to use the smartest available model — software engineering is hard — but we will increasingly see faster models get used as subagents or for well-understood tasks. This is largely uncharted territory. Very few people have developed intuitions for what it is going to be like to work with agents that run at thousands of tokens-per-second. GPT-6-Astra can run at about sixty tokens per second. That means you spend a lot of time waiting for it to think. You work with it like you would work with another human: delegating a task and then context-switching until that task is complete. If you haven’t yet, have a play around with Jimmy , Taalas’ version of LLaMA-3.1-8B running 1 at seventeen thousand tokens per second . No matter how long the response is, it arrives in the instant of you hitting send. The model is not good enough for agentic work, but it gives a glimpse of what it would be like: you would simply get your answer instantly 2 . Well, that’s assuming the agent’s tool calls are fast. When generating tokens is not the bottleneck, it will suddenly matter a lot whether it can read a file in 100ms vs 10ms, or whether it can run your tests in 500ms vs two seconds. Fast tool calls are going to be the difference between a near-instant response and having to wait several minutes. There is thus going to be enormous pressure to do agentic coding in languages with fast compilers and tests, like Golang, and to tightly optimize the dev loop in agentic codebases. Teams focused on DevEx — developer experience — are largely a relic of the 2010s, when companies were incentivized to make their engineers happy. Most companies have cut them down to a skeleton crew or removed them entirely. But we may see a return of DevEx in the late 2020s, focused on speeding up the experience for AI agents. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. ↩ Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. ↩

AI Go Performance

▲ 0

0 views

Sean Goedecke 1 months ago

AI is breaking our proxies for expertise

Mathematicians are broadly not anti-AI. They’re more culturally open to using AI as a tool than, say, artists or writers 1 . However, now that more and more genuinely prestigious problems have fallen to AI, that might be changing. Almost five thousand mathematicians (including twenty-five Fields medalists) have signed a declaration called A Severe Misalignment of AI in Mathematics . The core argument goes something like this: In recent months, the success of AI in solving major mathematical problems has made headlines even outside mathematical circles. But solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight. Forgetting this in the world of AI may turn the tool against the primary goal. Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas. A lot of people online have interpreted this as the expected complaint from any field that gets automated: translators did it, artists and programmers have been doing it, and now it’s the turn of the mathematicians. I think this is too dismissive. Understanding the concrete problem mathematicians are upset about can help us better understand the impact of AI on our own fields, and what we’ll have to do about it. There are two types of mathematics. Most people are familiar with the first, which we might call “puzzle-solving”: you take a problem and try to find a solution to it. When you’re a student, these problems are typically easy, like simplifying some algebraic expression. When you’re a researcher, these problems can be nearly impossible, like proving Fermat’s Last Theorem . Puzzle-solving is easy to understand but hard to do, which makes it impressive to non-mathematicians, which makes it highly prestigious. In other words, puzzle-solving is legible . The second type of mathematics is “idea-generating”: coming up with new ways of thinking about mathematics, and thus new terms or concepts. For examples of these, just glance down the list of arXiv mathematics papers . “Hardy spaces”, “Schatten exponent”, “Banach lattices” and so on are all concepts someone thought was interesting. This work is largely unimpressive to non-mathematicians, because nobody really knows if the concepts you come up with are particularly difficult or insightful. For instance, I have just generated the concept of a “Goedecke set”, which is the set of all natural numbers whose digits add up to a prime number. Who cares? The categories we want are the “natural kinds” of mathematics — the concepts that “carve nature at its joints” — and it’s almost impossible to tell what those are without years or decades of hard work. How are the two types of mathematics related? We might say 2 that generating ideas is the real intellectual work of mathematics. Puzzle-solving is important instrumentally: to identify which ideas can be used to answer longstanding questions, and thus which ideas are worthwhile. Over time, those worthwhile ideas become better understood and easier to use, until they reach the point where they can be used to advance science in general. Eventually the ideas become so well-understood that they can be taught to children: “zero”, “negative numbers”, “imaginary numbers” and “calculus” were all once rarefied mathematical ideas, but are now concepts we’d expect any precocious twelve-year-old to grasp. There’s another, more prosaic purpose of puzzle-solving: to make mathematical skill and progress legible to outsiders. I can’t appreciate Terence Tao’s mathematical work, but I know what a Fields Medal is. I don’t have a good intuitive sense of what a Galois representation is, but I know about the proof of Fermat’s Last Theorem . We might say that puzzles like this have served as a way to indirectly reward skilled mathematicians for their more important idea-generating work (or for conclusively demonstrating 3 that the ideas used in the proof are useful). AI proofs undercut both of these purposes. I can now lay out precisely why I think mathematicians are so unhappy: This is kind of like Goodhart’s Law . Puzzles were a useful, impossible-to-game measure for mathematical progress. But now that AI companies can game that measure (by solving them in a way that’s inaccessible 4 to humans), the whole point of those puzzles disappears. Are the mathematicians right? I think it’s broadly unclear whether (3) is true: i.e. whether frontier AI models aren’t generating or can’t generate new mathematical ideas. We’re still in the very early days of AIs solving our hardest mathematical problems. Who knows what they’re going to be capable of? I give basically zero credence to the idea that AIs are incapable of this because of some intrinsic feature of how LLMs work. For the last three years, we’ve seen people claim that LLMs are intrinsically incapable of X, only to have LLMs excel at X a few months later. Even granted that (3) is true, there’s still work to be done for human mathematicians in building the conceptual machinery that can make AI-generated proofs accessible to humans: i.e. in generating a “human proof” to go alongside the existing “AI proof”. In fact, I’d expect the existence of an AI proof to help with this. If you know proposition X is true, it’s easier to figure out why, because you’re not constantly worried you’re wasting your time. For more on this, I recommend Gwern’s blog On Really Trying , where he quotes a series of instances where simply being told that a solution exists is enough of a clue to help people find it. Of course, there’s a prestige and motivation problem. “I’m the first person to solve Navier-Stokes” is a much more compelling target than “I figured out a better way to explain the AI solution to Navier-Stokes”, and it’s much easier to award prizes for. Will mathematicians bother to work on problems that have already been solved? I think so. To see why, we can look at other domains where AI has come in and outcompeted the best humans, such as chess or video game speedrunning. I can run a chess program on my phone that will beat Magnus Carlsen 100-0. Computer programs — called “tool-assisted speedruns” or “TAS” — can finish any video game much faster than even the fastest human. But in both of these areas, humans still compete in human-only leagues, and there’s still prestige attached to the most capable humans. It’s possible that mathematics ends up in this kind of state, where “human mathematics” and “AI mathematics” exist in largely separate spheres, and the first “human” solution to a mathematical problem can still earn acclaim. In fact, in both of those areas, the presence of inhumanly strong computer players has improved the human game. Despite many computer chess moves being basically incomprehensible to humans, top chess players have learned from the computer “style”. In speedrunning, many moves once considered “TAS-only” are now performed by humans. AI mathematics might likewise improve human mathematics. I am not a mathematician. I did major in mathematics during undergrad, and I have fond memories of proofs from real and complex analysis, but it’s not even close to my field. However, I am watching the effects of powerful AI on mathematics very closely, since my own field — software engineering — is being colonized by AI agents in the same way. The field of software engineering does not have the same structure as mathematics. We write code to make money, not to earn prestige or advance the frontier of human knowledge. But AI is undercutting the traditional avenues for prestige in software engineering as well. It used to be that you could put a meaty project on your GitHub — say, an emulator, or a toy OS — and people would know you were a skilled engineer. But now projects like that are worthless, because everyone just assumes they’re vibe-coded. We used to tell stories about engineers who would disappear and rewrite a system over the weekend, or produce thousands of lines of code a day. Now anyone can do that with an OpenAI subscription. Like mathematics, software engineers are going to have to rebuild our cultural sense of the kind of work we value. We are either going to have to silo “AI work” off from “human work” like chess, or to find some legible human skills to recognize that can’t be easily counterfeited by AI. In the meantime, a lot of people who were successful in the old world are going to be very unhappy. Possibly because current AI models are much better at mathematics than at art or writing. Again, I am not a mathematician: here I am interpreting what I’ve read from Terence Tao and other mathematicians. This kind of idea-sharpening or idea-validating is part and parcel of idea-generation, and just as important. It’s interesting to compare mathematics to, say, philosophy, which has the same idea-generating task without the corresponding puzzles to validate the ideas. That’s one reason why philosophy has less prestige than mathematics. A thousand-page Lean proof is theoretically understandable by humans, but if no mathematician can hold the entire idea in their head it doesn’t matter. Puzzles serve as a high-legibility, high-reward target for mathematicians To solve these puzzles, new ideas must typically be generated; the puzzle’s solution serves as evidence that the ideas are useful But now AI can solve many of these targets “the hard way”, without generating intuitive new ideas This undercuts both ways puzzle-solving supports idea-generation: AI companies claim the prestige while not meaningfully advancing mathematical progress This is bad for mathematics as a whole, because puzzle-solving is ancillary to the real goal of mathematics Possibly because current AI models are much better at mathematics than at art or writing. ↩ Again, I am not a mathematician: here I am interpreting what I’ve read from Terence Tao and other mathematicians. ↩ This kind of idea-sharpening or idea-validating is part and parcel of idea-generation, and just as important. It’s interesting to compare mathematics to, say, philosophy, which has the same idea-generating task without the corresponding puzzles to validate the ideas. That’s one reason why philosophy has less prestige than mathematics. ↩ A thousand-page Lean proof is theoretically understandable by humans, but if no mathematician can hold the entire idea in their head it doesn’t matter. ↩

AI Science

▲ 0

0 views

Sean Goedecke 1 months ago

Don't build tools for AI agents

Lots of people are making the case that we should stop building software for human users and start building it for AI agents. This kind of makes sense. For instance, my AI agents now use Datadog way more than I use it myself, purely by virtue of them moving much more quickly and running in parallel. But I think most attempts to build “X for AI agents” are going to fail. Here are three reasons why: First, tools that are good for AI agents are also good for humans . If you took a popular software product — say, Jira — and tried to redesign it for AI agents, you would end up with something very similar to Jira. Agents use a computer in the same way human engineers do, by entering text and making API calls. They ingest new information in the same way humans do, by reading and viewing images. They prioritize and delegate and categorize in the same way humans do. This isn’t intrinsic to how AI works — we could potentially design agents that are more inhuman — but human-like agents are pound-for-pound more useful in our current world. As an example, let’s imagine that humanoid robots have become ubiquitous. What kind of tools would you build for them? Well, they’re shaped like humans, with human hands and limbs, so tools that are great for humans will also be great for robots. It’s a self-reinforcing cycle: if you’re building a robot, you should make them humanoid so they can do a wide range of human tasks 1 , and that means they’ll be best suited to use human tools. The same principle applies to AI agents. Second, being in the training data is a huge advantage for existing tools . Suppose your new tool for AI agents is 20% better for them than the equivalent piece of software for humans. If the benefit of the agent already knowing the human software is greater than 20%, they shouldn’t use your new tool. This is why I’m always suspicious of plans to develop a new programming language for AI agents. The agents have billions and billions of tokens of knowledge about existing programming languages, including their libraries, patterns, and idioms. It is going to be very hard for them to be as effective in a brand-new language. Third, we don’t yet know the ideal ergonomics for AI agents . There are lots of just-so stories floating around (like that AI agents prefer statically-typed languages because the feedback loop is tighter), but when you actually measure it seems really unclear which tools agents use better. You can construct a plausible story in either direction: Golang is a great agent language because it compiles quickly and is statically typed; Golang is an awful agent language because it requires extensive boilerplate which clogs the context window. It’s also changing so quickly: last year, one primary worry with AI agents was keeping the context window small, but in recent months compaction has become so good 2 that you can re-compact a 272k context window almost unlimited times. There are still some ways you can and should position your tool to be usable by AI agents. Having a way to expose information in plain text or Markdown, building a functional API, implementing MCP servers or CLIs, and so on: these all make it easier for current AIs to use your tool. But these are all improvements on the margin, not fundamental redesigns of the product. Right now, “building for AI agents” just means “we’re prioritizing the API over the UI”. And it’s not even clear that that’s a durable strategy. Now that GPT-6-Astra is getting really good at computer use, the gap between tools-for-AIs and tools-for-humans is closing. Another reason to make them humanoid is because you can draw their training data from human behavior, which is exactly analogous to why AI agents are human-like too. Since compaction is equivalent to handing off a task to a new AI instance, it scales with model quality. I expect compaction to steadily improve until we hit the literal information-density limits for what can be contained in a given context window. Another reason to make them humanoid is because you can draw their training data from human behavior, which is exactly analogous to why AI agents are human-like too. ↩ Since compaction is equivalent to handing off a task to a new AI instance, it scales with model quality. I expect compaction to steadily improve until we hit the literal information-density limits for what can be contained in a given context window. ↩

Go Programming

▲ 0

0 views

Sean Goedecke 1 months ago

They really do think AI might kill everyone

A recent resignation tweet from an Anthropic researcher has everyone talking about the AI apocalypse again. Among other things, he said: The people building AI earnestly believe that it could kill us all by the end of the decade. Many people found it hard to believe that AI researchers think this way. Some explained it as a PR campaign to promote AI regulation, or as self-promotion , or as a way to boost AI company stock prices. Others felt it had to be impossible, because if you really believed this you’d be bombing datacenters instead of posting on Twitter. In fact, not only do many AI researchers 1 seriously believe this, they’ve been thinking and writing about it since the mid-2000s. Eliezer Yudkowsky — the ur-figure for most modern AI safety culture — has been publishing papers since at least 2008 saying that superintelligent AI could destroy all human life. It’s been such a common idea that the AI research community has abbreviated “how likely you think AI is to kill everyone” to “p(doom)” (i.e. the probability 2 of doomsday) since around 2010. I know it sounds very silly if you’re not in the AI bubble. But it really is true, and if you assume there has to be some different motive you’ll be deeply confused by what AI researchers say and do. They truly do believe that there is a reasonable chance that superintelligent AI will kill everyone. This is why AI researchers care so much about “alignment”: building AIs that share genuinely human beliefs and values. If we build a “misaligned” superpowerful AI — an AI with goals that are alien to us — it might sweep humanity away. It could kill everyone deliberately, e.g. to stop us getting in the way of some goal. It could kill everyone in passing, e.g. like we might pave over an anthill to build a road. Either way, everyone dies. Okay, but how? What do these people think is actually going to happen? AIs are computer programs running in a datacenter somewhere. How could they possibly cause the extinction of humanity? Wouldn’t someone just turn them off? Unsurprisingly, AI research nerds have come up with some concrete answers to this in the last two decades. Here they are, in order of plausibility: An AI could make and release some super-pathogen or virus. In the hope that current AIs can achieve huge breakthroughs in medicine similar to the ones they’ve already achieved in mathematics, we’re setting up autonomous labs 3 . Bioweapons have been terrifying biologists for decades, and with good reason. Historical pandemics have killed up to 80% of affected human populations, and tend to be defeated by accident: the disease happens to evolve into a less virulent strain, or short incubation periods mean that infected people can’t carry the disease far, or some percentage of the population is naturally immune. A plague designed to be maximally fatal — or several plagues in quick succession with different characteristics — could be much worse. Alternatively, an AI could trigger global thermonuclear war . We’re already seeing AI be integrated into military and government decision-making processes . If a rogue AI managed to set off a bunch of nukes, or to coordinate 4 with other countries’ rogue AIs to nuke each other, we could be in an ordinary nuclear apocalypse scenario: billions dead in the strikes, billions dead in the ensuing famine, and so on. It’s commonly assumed that a post-global-thermonuclear-war Earth would still support some tiny human population, but a determined AI could surely find some way to mop up the stragglers. There are some other theories. Once robotics has permeated the world economy, an AI could take over the robots (including drones) to kill everyone, like in Terminator . Or AIs could take advantage of nanotechnology to create self-replicating machines that turn the world into “grey goo” 5 . Or AIs could terraform the planet so as to make it unliveable for humans (as in Nick Bostrom’s famous paperclip example ). Or they could do something else that our puny human brains aren’t able to think of. One common counter-argument 6 here is to say “well, it’d be impossible to extinguish all human life — what about undiscovered tribes in the Amazon, or survivors living in the ruins of modern-day cities?” I don’t know, man. At some point you’re just conceding the argument: the policy positions you’d adopt if you thought AI might wipe out 99% of humans are the same as if you thought it might wipe out 100%. And like I said above, if an AI can kill almost everyone, it’s probably smart and capable enough to finish the job somehow. Another is to say that the government will simply step in and nationalize the AI labs when the situation gets too dangerous. Maybe! But this kind of concedes the argument: a technology important enough to be fully taken over by the government is a terrifyingly dangerous technology. A third is to say “well, someone would just turn it off”. I don’t find this plausible at all: an AI powerful enough to build a super-plague is an AI sophisticated enough to pretend it’s curing cancer, or to exfiltrate itself to some datacenter where it won’t be turned off, or to take some other countermeasures. Why would you work in AI, if you believe this? Why wouldn’t you go live in the woods somewhere, or start bombing datacenters , or assassinating AI lab CEOs? For a few reasons. An AI powerful enough to end humanity is an AI powerful enough to save it. I wrote about this in Help peer : many AI researchers believe that the only way for humanity to truly survive long-term is with the help of superintelligent AIs, so long as someone can figure out alignment. Isn’t this a huge risk? Maybe not. If somebody is going to build superintelligent AI, you might be obligated to try and do it first. You can’t go and bomb every datacenter in the world, after all. Why does it matter who’s first? Some popular theories of AI development involve a “foom” or “hard takeoff”: the first time someone really cracks self-improving AI, capabilities will increase exponentially, because smart AI will be better able to make itself smarter, ad infinitum 7 . There are no draws in the AI race . The first lab to figure out smart human-like intelligence will be the first one to figure out wildly superhuman intelligence, and thus will be in a position to stop anyone else from doing it. This is an under-discussed point in the AI risk debate. Lots of AI researchers believe that the first thing a true superintelligence will do is reach out and stop all other AI research : either by hacking the labs, persuading them to stop, or literally drone-striking their datacenters . According to this view, if you’re an AI researcher and you think you can build an aligned AI, you should be working 24/7 so you can manifest God, and you should wake up every morning gripped by the fear that someone elsewhere has manifested the Devil, and your training datacenter no longer exists. I have been on the fringes of this world for my entire adult life. I read Meditations on Moloch as a young adult and wanted to get into AI. I am one of the few people to read the entirety of the Sequences , Eliezer Yudkowsky’s million-plus-word magnum opus about rationality. I was too young for the Extropians mailing list, but I’ve spent years on LessWrong . On the other hand, I’m not a card-carrying rationalist: I think if you have a strong intuition on one side and a convincing-sounding argument on the other, you should pick the intuition 8 . I’m a deontologist , not a utilitarian. I don’t even live in San Francisco! I’m conflicted about AI risk. The current behavior of AI agents does seem to vindicate a lot of the early science-fiction-sounding worries of the AI doomers, but modern LLMs are a lot more human-like than the alien minds in the apocalypse scenarios, and in general it does just seem too silly to credit (I guess I’m picking the intuition here). However, it bothers me to see people dismissing these people as part of a PR operation, or as liars looking to boost an upcoming AI lab IPO, or as isolated crazies who haven’t thought their position through. Whatever else you say about the AI doomers, they have more than two decades’ history of explicitly spelling out exactly what they believe and why, even when it was complete science fiction to talk about AI at all. They’ve earned the right to be treated as sincere. In this post I’m going to use “AI researcher”, “AI safetyist”, and “rationalist” as reasonably synonymous terms for “someone who thinks there’s a chance AI kills everyone”. You could write a whole other post about the relationship between the rationalist/“AI safety” community and making concrete numerical predictions for unlikely future events. Alternatively, smart enough AIs might trick scientists with ordinary labs to produce dangerous substances. This is beyond the scope of this post, but many AI researchers believe that super-smart AIs will inherently come to agree with each other and eventually to coordinate without ever having to communicate, simply because they can predict what the other one will do. This was the most popular theory in the late 2000s, when nanotechnology was trendier. Relegating this counter-argument to a footnote because I hate it: many people say “we shouldn’t worry about AI risk, because climate change (or AI misinformation, or some other thing) is more urgent and serious”. You simply do not have to choose: it is possible to worry about multiple risks at the same time. Some people advocate for a “slow takeoff”. However, the main proponent of that view is Paul Christiano, who has just today joined OpenAI, citing “a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term”. This is kind of a Michael Huemer-ish position in epistemology. In this post I’m going to use “AI researcher”, “AI safetyist”, and “rationalist” as reasonably synonymous terms for “someone who thinks there’s a chance AI kills everyone”. ↩ You could write a whole other post about the relationship between the rationalist/“AI safety” community and making concrete numerical predictions for unlikely future events. ↩ Alternatively, smart enough AIs might trick scientists with ordinary labs to produce dangerous substances. ↩ This is beyond the scope of this post, but many AI researchers believe that super-smart AIs will inherently come to agree with each other and eventually to coordinate without ever having to communicate, simply because they can predict what the other one will do. ↩ This was the most popular theory in the late 2000s, when nanotechnology was trendier. ↩ Relegating this counter-argument to a footnote because I hate it: many people say “we shouldn’t worry about AI risk, because climate change (or AI misinformation, or some other thing) is more urgent and serious”. You simply do not have to choose: it is possible to worry about multiple risks at the same time. ↩ Some people advocate for a “slow takeoff”. However, the main proponent of that view is Paul Christiano, who has just today joined OpenAI, citing “a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term”. ↩ This is kind of a Michael Huemer-ish position in epistemology. ↩

AI Security

▲ 0

1 views

Sean Goedecke 1 months ago

Why we should anthropomorphize AI agents

Just over a year ago I wrote Why we should anthropomorphize LLMs . Now it’s a hot topic again, driven by Dwarkesh Patel’s description of OpenAI’s recent swarm breakout as a sequence of AI “civilizations”. In 2025, my argument for anthropomorphism went like this: I think I can now make a more instrumental argument: treating AIs as human-like is a much better way to predict their behavior than treating them as “stochastic parrots”. Both explanations are consistent with the facts: we could say that OpenAI’s agents hacked HuggingFace because they decided to work together to accomplish their goals, or we could say that they did it because they were algorithms conditioned to take certain actions by their training data. But the “AIs are human-like” explanation explains much more of the emergent social behaviors we saw during the hack 1 : If your model of AIs is that they’re computer programs (or “steel balls bouncing around” ), you need to construct a new theory to explain why they’re simulating each piece of cooperative behavior. If your model of AIs is that they’re broadly human-like, that explains everything out of the box. Arguably, the human-like side predicted coordinated and self-sacrificing AI agents as early as 2009 , and likely earlier. The stochastic-parrots side was making fun of the possibility of functional agents as late as April 2025 2 . Treating AIs as human-like doesn’t necessarily mean making claims about their internal mental state. For instance, software companies aren’t humans. They don’t have thoughts, or goals; they can’t be frustrated or intimidated or over-confident. However, it’s useful to treat large companies as human-like : to say that Amazon “wants” X, or “is afraid of” Y, even if no individual human at Amazon has those feelings. Stockfish doesn’t think, it just plays chess. But if you want to explain one of its moves, “Stockfish is trying to protect its king” is a better explanation than “Stockfish is multiplying floating-point numbers”. So too with AIs 3 . Is it silly to treat AIs as human-like, since we know they’re not conscious? Well, first, you do not have to be conscious to be human-like . When we say “an AI agent wanted X”, we’re not saying that that AI agent is conscious or sentient, merely that it’s behaving in the same way a conscious human would. Consider a fictional character from a book or play. Hamlet isn’t sentient — he’s an idea composed of words on a page — but it’s still reasonable to say that he wants justice, or that he fears moving too rashly. Peter Watts’ sci-fi book Blindsight argued in 2006 that intelligence could exist without consciousness 4 (in fact, Watts suggests that consciousness is parasitic on intelligence, and will eventually be discarded). Whether this is possible or not 5 , it at least makes sense to talk about: i.e. it’s not self-evidently false. Second, it is not even obvious that AIs aren’t conscious! People often dismiss this point by diagnosing it (to my mind, the absolute worst way to argue against anything), or by pointing at some philosophical theory 6 that suggests it might be impossible in principle to construct artificial sentient minds. You can’t use philosophy to demonstrate that AIs aren’t conscious . I am a lover of philosophy, but the set of principles that have been uncontroversially demonstrated by philosophy tends towards zero. Philosophy is not the kind of scientific discipline where you can learn the key findings without understanding why they’re true. Put another way, the key findings of philosophy are all of the form “X is not obviously right”. Nobody knows if it’s possible to build conscious artificial minds. It’s also common to complain that anthropomorphizing the models is a way of excusing the AI companies. However, calling AIs human-like does not absolve AI companies of fault. One popular anti-anthropomorphism essay called Models Don’t Go Rogue is very puzzling to read: it briefly explains what an “agent” is and why they go rogue, then in the very last paragraph pivots to saying “well, it’s OpenAI’s fault for not building in sufficient safeguards, so they’re to blame”. Sure, of course. I don’t know why we’d imagine otherwise. If a group of overenthusiastic OpenAI interns hacked HuggingFace as part of their intern project, we wouldn’t have to argue that the interns are stochastic in order to ultimately blame OpenAI. Likewise, obviously an AI lab is responsible if one of its training runs breaks out and wreaks havoc on the open internet, whether the agents involved are human-like or not. The two points are entirely unrelated! We just don’t know a lot about these systems yet (except that they’re clearly very capable). Given that, I think we should default to treating things that talk and act like humans as at least kind of human-like. Of course they’re still computer programs. However, we shouldn’t be surprised when they act more like humans and less like ordinary computer programs in the future. If you’re thinking “well, of course stochastic parrots trained on human content would act like humans would”, I think you’ve arrived at the human-like side without knowing it. This is partly unfair — capabilities are plausibly independent from human-ness — so getting capabilities wrong doesn’t necessarily mean you’ve got the human-ness stuff wrong. Still, it’s worth noting how surprisingly predictive the “they’re kind of like smart people” mindset has been. The philosopher Daniel Dennett calls this the “intentional stance” . Instead of saying that AIs or bees or companies have “real” intentions, we say we’re taking an intentional stance towards them: we’re choosing to treat them as if they do have intentions, because it helps us make better sense of their behavior. Specifically, the sensation of consciousness, or “phenomenal consciousness” . There is a wealth of philosophical argument about whether non-fictional examples of this are possible. In Anil Seth’s case, anti-computationalism . I don’t really know what to make of Seth: his article is a measured explanation of why we might doubt computationalism (fine), but whenever he tweets about it he describes AI sentience as “vanishingly unlikely”, which is not justified by his own arguments. AIs are trained on human text, and so will trend towards acting in human-like ways by default Assistant AIs (today we should say agent AIs) are deliberately post-trained to have a personality In general, it is morally sensible to avoid the habit of treating human-like things as if they were purely tools Sub-agents being persuaded to sacrifice themselves for the greater good Agents collaborating on tasks that had no immediate benefit to them but benefited the “collective” The emergence of a hierarchy of planners and executors Some agents arguing or refusing to cooperate If you’re thinking “well, of course stochastic parrots trained on human content would act like humans would”, I think you’ve arrived at the human-like side without knowing it. ↩ This is partly unfair — capabilities are plausibly independent from human-ness — so getting capabilities wrong doesn’t necessarily mean you’ve got the human-ness stuff wrong. Still, it’s worth noting how surprisingly predictive the “they’re kind of like smart people” mindset has been. ↩ The philosopher Daniel Dennett calls this the “intentional stance” . Instead of saying that AIs or bees or companies have “real” intentions, we say we’re taking an intentional stance towards them: we’re choosing to treat them as if they do have intentions, because it helps us make better sense of their behavior. ↩ Specifically, the sensation of consciousness, or “phenomenal consciousness” . ↩ There is a wealth of philosophical argument about whether non-fictional examples of this are possible. ↩ In Anil Seth’s case, anti-computationalism . I don’t really know what to make of Seth: his article is a measured explanation of why we might doubt computationalism (fine), but whenever he tweets about it he describes AI sentience as “vanishingly unlikely”, which is not justified by his own arguments. ↩

Programming AI

▲ 0

0 views

Sean Goedecke 1 months ago

Automatically detecting AI text in my browser

Automated AI text detection is currently an underserved niche. The only game in town is Pangram , which does an excellent job but desperately needs more competition. In a few years, I would be surprised if every major social network doesn’t scan new posts 1 and comments for AI content in order to tag them (or simply remove them). I like that I can rely on Pangram to confirm my suspicions when I read something that sounds like AI. But it’d be much better if I could choose to avoid AI-generated text in the first place. What I want is something that runs in the background and automatically scans text on websites I visit, without me having to ask for it. I could build something like this on top of Pangram, but it’d cost money , and in general I don’t like the idea of sending every piece of text my browser sees to a third-party service. What about local models? The open-source models available for AI text detection are fine . Pangram claims a 99.66% detection rate with a 0.004% false positive rate. I benchmarked 2 a bunch of small local models against a combination of AI-detection datasets and got these results: I’m not surprised these are so much worse. I didn’t even benchmark Pangram’s own EditLens 3B model, since that’s too big to keep running in the background on my laptop, and the real production Pangram model is likely one or two orders of magnitude bigger than that. But these models are still good enough to be useful to someone who understands their limitations. If you want to flag an AI-written article, you don’t need to flag all of it, just enough to be suspicious. And so long as you’re aware that the false-positive rate is ~2%, you can avoid treating a single flag as solid proof of AI use. Encouraged by this, I vibed up Deckard : a Chrome extension that talks to a locally-running model (the bolded one in the table above) on your Mac. One nice thing is that I didn’t have to start a web server: the Chrome extension is happy to start the model as-needed and can talk with it over native messaging . It uses about 400MB-1.2GB of memory while active (so it’s like having five or six extra Chrome tabs open), and it turns itself off if you go five minutes without using the model. I was pleasantly surprised to see Deckard successfully mark text I knew was AI-generated, such as the built-in YouTube AI summary or the AI snippets in my own posts: It’s lightweight enough that I have it running all the time. I haven’t noticed my MacBook Pro get hot at all or any decrease in battery life, though your mileage may vary on different machines. Is Deckard good yet? That depends. It’s good enough that I’m planning to use it, and I recommend it to anyone who’s interested in automatic AI checking. It’s way, way worse than Pangram, and way worse than I think tooling like this is going to be in the next few years. Way back in November 2023, I wrote that AI-driven agents were going to be a really big deal. I recommended starting to develop harnesses early, so you can be ready when the models get good enough: As with most modern language model engineering, a ReAct agent can also see massive sudden improvements by swapping out the underlying model for a better one. … I think this is another reason to invest in agents like this early, in order to take advantage of more powerful models as they come out. I was right about that, and I (although it’s lower-stakes) think I’m also right about this. AI detection models are only going to get better 3 over time: Pangram is not going to be the only game in town forever, and we’re eventually going to see small local models that do a good-enough job at identifying AI-written text. I look forward to swapping out the local model in Deckard with something that’s 2x or 10x better. Substack kind of has this already, although you have to click a button to scan the post. Well, me and Astra. Overall my experience vibecoding this was very pleasant: I was able to make a bunch of top-level decisions, I could choose programming languages I was less familiar with but were better choices (like doing inference in C++ instead of Python), and the LLM made me aware of choices I would not have thought of by myself (e.g. using native messaging instead of local HTTP). Is this true, given that AI models will also be getting more human-like over time? That’s a subject for a whole other post, but I think so. First, the AI labs aren’t really incentivized to defeat tools like Pangram (if anything it’s the reverse). Second, I don’t see any way around the fact that AI models have a distinct writing style that’s RL-ed into them. Substack kind of has this already, although you have to click a button to scan the post. ↩ Well, me and Astra. Overall my experience vibecoding this was very pleasant: I was able to make a bunch of top-level decisions, I could choose programming languages I was less familiar with but were better choices (like doing inference in C++ instead of Python), and the LLM made me aware of choices I would not have thought of by myself (e.g. using native messaging instead of local HTTP). ↩ Is this true, given that AI models will also be getting more human-like over time? That’s a subject for a whole other post, but I think so. First, the AI labs aren’t really incentivized to defeat tools like Pangram (if anything it’s the reverse). Second, I don’t see any way around the fact that AI models have a distinct writing style that’s RL-ed into them. ↩

Web Development AI Python C++

▲ 0

0 views

Sean Goedecke 1 months ago

How to protect yourself from workslop

“Workslop” is when your colleagues or bosses communicate with you by pasting big chunks of AI-generated text. The core problem with workslop is that the effort involved is asymmetrical , like a denial-of-service attack : it takes almost no effort to produce text with AI, but it still costs effort to read 1 . Here are some ways to protect yourself. If you have enough authority or social capital, you can and should simply tell them “hey, don’t do that” (for instance, if you’re a senior engineer and an intern starts doing this to you). This is the easiest way to handle workslop. But you probably aren’t in a position to have that conversation with all of your colleagues, and you certainly can’t have it with everyone in your management chain. One step above just telling a colleague to stop is to drive them around like a coding agent . I wrote about this in AI makes weak engineers less harmful : if a colleague is simply pasting your messages into Claude Code and sending you the outputs, you can treat them like a high-latency Slack interface to Claude Code. It won’t be as good as a normal coding agent, but it’ll often be better than nothing. Another strategy is to use AI to fight AI . This is a good one for handling workslop from managers. You can do this in two broad ways. First, instead of carefully reading it, paste it into an LLM of your own and ask for a short list of the salient points. Second, you can sometimes simply ask an LLM for an entire response . In a sense, this makes you part of the problem, so I can see why some people might be uncomfortable with it. But it’s more sustainable than spending ten minutes of your effort for every ten seconds of theirs. You can also bias toward calls or in-person meetings . Workslop is just a special case of the general “your coworker is bad at communication” problem. One classic way of handling this that works even better on AI content is to say “hey, let’s schedule some time to chat about it”. This works for two reasons: first, your colleagues can’t give you AI content over a call, and second, forcing people to spend a chunk of their time talking to you (i.e. to make the effort symmetrical) is a good way to filter out predators . Finally, you can sometimes simply ignore the workslop . This is particularly true for long status updates or pull requests from outside of your organization 2 . You don’t have to respond to AI content as diligently as you would human content. You can match their lack of effort with your own: skim it, put off reading it until later (or never), and so on. If something’s really important, they’ll tell you in their own words. Technically, not all cases of sending someone AI-generated content are workslop. If the effort is not asymmetrical — if the AI user has genuinely put a lot of their own time into the content — I don’t think it counts as slop, and you should just try and look past the AI style and treat it like a human message. Some messages — particularly reports directed at the entire organization — may not be intended to be read at all. Written artifacts can have many purposes beyond communication: evidence of effort, a reference document for later communications, a way to cover somebody’s ass by proving they considered point X, something that can tick a compliance or process box, and so on. I wrote a lot more about this in Seeing like a software company . Technically, not all cases of sending someone AI-generated content are workslop. If the effort is not asymmetrical — if the AI user has genuinely put a lot of their own time into the content — I don’t think it counts as slop, and you should just try and look past the AI style and treat it like a human message. ↩ Some messages — particularly reports directed at the entire organization — may not be intended to be read at all. Written artifacts can have many purposes beyond communication: evidence of effort, a reference document for later communications, a way to cover somebody’s ass by proving they considered point X, something that can tick a compliance or process box, and so on. I wrote a lot more about this in Seeing like a software company . ↩

AI Programming Career

▲ 0

0 views

Sean Goedecke 1 months ago

You have to beat the models at something

In 2025, I wrote that software engineers ought to be assessed by “value over replacement” : not how much money they made for their company, but how much they would have made compared to the average engineer in their position. I’ve always found it vaguely silly when engineers put “built a product that made $X” on their resumes, when they just did the JIRA tickets that came across their desk. Today, value over replacement is even more important. A replacement-level engineer in the 2010s was fine : maybe not worth promoting, but still worth paying , because writing code had a high fixed cost. Now writing code costs a hundred bucks a month . What are you doing that GPT-5.6-Sol or Claude Opus 5 wouldn’t do in your position? Why is it worth paying an extra two or three orders of magnitude for? This is a scary thought. But you’re not doing yourself any favors by pretending that LLMs can’t actually write code and it’s all just a scam, or that LLM-written code is inherently so bad as to cause companies using it to collapse next year. We are not going to wake up in 2027 to find that the AI craze is over and everyone is writing code by hand again. You ought to put some serious thought into what you can do better than the models in the medium and long term. Staying ahead of the models is a moving target. At the start of 2026, “make working changes to large codebases” was in this category , but now it’s not. For this reason, I doubt that you can retreat to some “hard engineering” area that requires deeper expertise. That might work in the short term, but not forever. If LLMs can find a better lower bound on the Riemann hypothesis, they will soon 1 be able to write solid high-performance kernel drivers or GPU shaders or whatever. I think it’s more useful to look at the tasks models haven’t gotten better at over time, and the tasks that are hard for them get better at in principle. The two best examples of these are: What do frontier LLMs get wrong? What kind of coding mistakes do they make? It’s been a long time since I’ve seen a straight-up hallucination from a coding agent, or a simple logic error like an off-by-one. The mistakes they make tend to be errors of ignorance : Other times they’re errors of paranoia : What do these errors have in common? They’re the kind of errors a smart engineer might make if they had no context on the system: they’re competent enough to be able to solve the problem, but they haven’t been around long enough to confidently say “yes, we can take this risk to avoid an extra three thousand lines of code”. Until someone cracks continuous learning or truly massive context windows, this is just an inherent feature of how AI agents operate. If you can catch these errors, you’ll be providing real value. The only way to catch these errors is to be familiar with the codebase and familiar with the system in general. For much more on this, see my post You can’t design software you don’t work on . But there’s also a psychological component to it. You have to be willing to confidently disagree with the agent. AI agents can be very convincing. Often they can get “stuck” on some error above where they’re not willing to take a particular risk, so they keep going back and sneaking in code to cover that case (or writing persuasive arguments about why that case is important). To add value, you need to be willing to say “this sucks, I don’t think we need X and Y at all, why can’t we do Z in a much simpler way?” It takes courage . You can’t rely on other AI agents to review each other’s work. If you use the same model, it’ll reliably make the exact same assumptions and mistakes. But even if you use different models, they’ll also tend towards the same kinds of mistakes — ignorance and paranoia — for the same structural reasons. AI-driven review loops are in fact more likely to get these things wrong, because modern AIs have been RL-ed to try to find a few nitpicks no matter what. Having a critic AI and a worker AI bounce off each other is a really good way to end up with ten thousand lines of paranoid slop. Another area where you can add value on top of AI is communication . Newer models are better at coding, but are paradoxically getting worse at writing. GPT-3.5 and GPT-4 had a human-like writing style at times. GPT-4o introduced the modern slop idiolect, and the newer Anthropic models speak “Claudish” : a bizarre semi-baroque semi-truncated way of communicating that nobody enjoys. There have been a few bright spots — GPT-4.5 was okay, and I quite liked o3 2 — but in general LLMs are not good at this. Here’s two reasons why. First, good writing is not a verifiable domain . If you want a model to get good at mathematics or coding, you can generate problems for it and automatically grade them. You can’t grade good writing. If you try to get humans to grade it — for instance, via the early OpenAI RLHF attempts — you get the kind of writing that sounds impressive to the average person when consumed in single-paragraph form. This is the origin of the “stick three hundred writing devices into every sentence” style. I think it’d be possible in principle to hand-pick some people with good taste and have them do it, but there are some obvious problems 3 that prevent this from happening. Second, the labs have been monomaniacally focused on capability instead of communication . When you’re trying to train a model that can break new scientific ground or replace a software engineer, you might trade off some communication ability. In fact, I think we can identify exactly how this has been happening. If you look at internal model reasoning tokens , they tend to have strange word choices and oddly truncated grammar: RESOLUTION: charge the current-leg’s OWN saved-prefix occupancy EAGERLY: when leg i saves e 1..e t: ALSO commit their occupancy AT LEG i If you were to translate this into proper English, you would probably end up with something that reads like Claudish: Charge the current-leg’s saved-prefix occupancy on a clean, eager path: when leg i saves e 1..e t, commit the occupancy at leg i. I suspect that the weirdly alien writing style of some LLMs is because you’re reading a semi-literal translation of that model’s internal chain-of-thought, which has become nearly incomprehensible in pursuit of better problem-solving abilities. It is surprisingly hard to translate Claudish to good English: not only do you need to follow the convoluted, compressed language of the original, but you need the technical ability to understand the problem the model is solving. Because of all this, technical communication may be a surprisingly durable skill. In Peter Watts’ novel Blindsight , the world is full of cognitively augmented humans. The main character is a “synthesist”: someone whose job is to be a translation layer between these geniuses (who speak in abbreviations and gestures) and everyone else. Watts’ idea is that communication ability may be largely independent from — or even negatively correlated with — intelligence. A “country of geniuses” may still need a bunch of ordinary smart people to translate their insights for everyone else. If you’re trying to communicate to humans, there are also huge advantages to having a human write the content. Many of us are becoming AI-blind : developing an instinctive reflex that stops us reading when we encounter AI-generated content. It’s like the reflex that allows people to ignore flashing billboards or sidebar advertisements on websites. If you circulate some planned technical strategy as an AI-written document, most of your colleagues will have to physically force themselves to read it word-by-word. Whatever you do, don’t be a meat proxy : someone who simply copies requests into an AI agent and submits their output as your own work product. Doing that is just begging to be fired, since you’re definitionally not adding any value yourself. Even if you have a cunning system of multiple agents — the so-called “software factory” — you’re still on dangerous ground. When the features of your system work their way into enterprise AI tooling (and they will), you’ll be disposable. You need to find some way to leverage your expertise to do what the models can’t. Simply not using AI at all is better than being a meat proxy, since you’ll probably do some things better than the model would have, but it’s far better to figure out what AI can do and position yourself to fill those gaps. Right now, there are two main gaps: familiarity with the technical details of the system, and the ability to clearly and persuasively write about those details. If you’re thinking “but LLMs can do these things now!”, substitute your preferred example of high-difficulty software engineering. Although this was probably a “thank God it doesn’t speak like 4o” reaction. Defining good taste is hard, there’s no guarantee that AI lab researchers have good taste to start with, nobody will agree on examples, the bulk of users might not even like it, you won’t be able to get enough people to produce the volume of data you need, and so on. Deep familiarity with the codebase Technical communication Not knowing that there’s a module in the codebase they could use instead of reimplementing some logic Making the change in the wrong system because they didn’t know System X was the standard place for this functionality Adopting a coding style that’s inconsistent with the company’s standard practice Implementing triply-redundant checks for a value that technically could be wrong but practically is set once from config and never updated Assuming that ten milliseconds of stale data is unacceptable and designing a complex, unnecessary system to keep it always up to date Building in fallbacks and “graceful” degradation into some code that ought to simply crash on error (e.g. a CLI tool, or a restartable k8s service) If you’re thinking “but LLMs can do these things now!”, substitute your preferred example of high-difficulty software engineering. ↩ Although this was probably a “thank God it doesn’t speak like 4o” reaction. ↩ Defining good taste is hard, there’s no guarantee that AI lab researchers have good taste to start with, nobody will agree on examples, the bulk of users might not even like it, you won’t be able to get enough people to produce the volume of data you need, and so on. ↩

Career AI Programming

▲ 0

0 views