Latest Posts (20 found)

Jev means structured output is interesting again

I don’t write blog posts about new models. That’s Simon Willison’s beat, and he’s very good at it. But I want to write about Jev , which is a different kind 1 of AI model: a “System One” 2 model. As it turns out, it’s not that different from an ordinary LLM with structured output, but the interface it uses is very cool and I hope it becomes more widespread. Ordinary LLMs take in some human-language prompt and produce some human-language output. They do so autoregressively : first they produce one token, then the next, then the next, and so on. This makes them extremely flexible, since they can do literally anything a computer can do. But it also makes them slow and weird. Slow, because they have to run a whole new generation pass per-token, and weird, because the space of human language is so broad that you can get really odd behavior from a model trained on it. Jev takes a human-language prompt, but it does not produce human-language output. It only produces structured output. So far, so ordinary: LLMs do this already . But it turns out that if you build a model that only produces structured output, you get some interesting and desirable properties. Jev is always really fast. The fastest response time is around 70ms instead of a couple of seconds for normal LLMs. Even better, the slowest response time is only 500ms. Because Jev only does structured output, it isn’t autoregressive: it can produce answers to many questions in parallel in a single forward pass. When a LLM is producing structured output, it has to produce the tokens ”{”, ” ”, “answer”, ”:”, and so on with successive forward passes 3 . Jev does it all in one go. The most compelling example of Jev’s speed is that the model can play Doom . You can feed a text-based representation of the current game state into the model, combined with a set of choices like “should the trigger be held down”, “what should the current goal be”, “given that the current goal is X, what keyboard input should be pressed”, and so on, and it works — latency is low enough and the system is smart enough that the model plays well in real time. Of course you could train a neural net to play Doom already. But Jev is a general intelligence: just like LLMs can do your taxes, perform mathematics research, fix your Python environment, and write you a poem, Jev can do many other tasks besides playing a single video game. Current LLMs can play Doom too (albeit slowly). But as Nelson Elhage famously said , fast software doesn’t just mean we can do the same tasks faster, it means we can do entirely new kinds of tasks. What kinds of new programs can we write by injecting 100ms worth of dirt-cheap intelligence at various decision points? To me, this is the most exciting thing about Jev. Fast structured output could be a genuinely new computational primitive for intelligence. So far we’ve built a lot of programs on top of autoregressive token generation, and they all look like fancy chatbots. Leaning hard into structured output might conceivably unlock a bunch of non-chatbot use cases for AI. My biggest problem with Jev is that I think fast structured output is already available . Structured output from LLMs is only slow because (a) nobody really cares about it 4 , and (b) the people who do care about it want big JSON blobs, so it’s typically implemented with “grammar-constrained decoding” : the LLM outputs autoregressively as normal, but the logit sampler discards tokens that don’t fit the structured output (e.g. if there hasn’t been a ”[”, you can’t output a ”]”). If you want fast, parallelized structured output against limited choices, you don’t strictly need to do autoregressive generation at all. You can simply prefill the response with and generate one token 5 , restricted to the user-provided choices. Since LLMs ingest all input tokens in parallel, this is way faster than generating the entire structured output. Multiple choices can be batched into the same forward pass via ordinary inference batching. This doesn’t let you do long-form structured output, but in return you get most of 6 Jev’s “secret sauce”: the speed, the consistency, and the parallelism of a System One model. People have already started trying this after today’s Jev announcement, and it seems like it’s working OK 7 . In other words, I suspect Jev does not have a substantial technical moat, and their claimed “Reinforcement Learning for Calibrated Decisions” is not a brand-new scaling axis. It will probably be pretty easy for any other lab to replicate, or for individual programmers to retrofit existing open-source LLMs into a fast Jev-like model. However, I suspect Jev is still going to be better than most versions of “Qwen-32B-System-One” or whatever. Being able to fine-tune or optimize the model on just structured output is probably a meaningful advantage. I doubt Jev is ever going to be as smart as frontier LLMs. Not being able to use test-time compute at all 8 is a big disadvantage, and will likely cap this kind of model around the strength of non-reasoning LLMs. In practice this shouldn’t matter too much for low-latency applications, but you shouldn’t see this as a new scaling axis or a way to produce more intelligent models. Jev’s developers claim it is immune from hallucinations. To me, this seems like a semantic dodge, since Jev can absolutely still pick the wrong choice (e.g. calling the sky “red”). I suppose that’s technically just a mistake , since the model is picking a user-provided choice instead of inventing something new out of whole cloth. Still, all of this is also true about regular LLMs with structured outputs, and it doesn’t make Jev any more reliable in practice. It’s unclear to me how much of Jev’s value is in the model itself, compared to the inference strategy of only generating one token per question. The data and demos in the announcement look to me like they could have been generated by plugging any Terra-sized model into a single-token inference stack. However, the people involved are credible, and I’m sure the model is good — I just wish they’d provided some comparisons that didn’t force the LLM to unnecessarily produce a blob of JSON token-by-token. Overall, I am happy that Jev exists and I hope it succeeds. I hope we do see some real competition in the fast-structured-output space, and that it motivates the big labs to release official versions of their own models that are fine-tuned for this. GPT-5.6-Terra-System-One would be a very interesting model to build AI products on top of. I did write about Thinking Machines’ “interaction models” , which are also a fast-enough-to-be-meaningfully-different paradigm for AI inference. They call Jev a “System One” LLM, after Daniel Kahneman’s partially discredited Thinking Fast and Slow , where he divides human cognition into a lightning-fast System One and a slow-and-reflective System Two. If you’re thinking “wait, couldn’t you just aggressively prefill a regular LLM and only produce one constrained token”, keep reading. Not counting tool calls, which are built-in in a way that structured output isn’t. What if some of the user’s choices are longer than a single token? I haven’t tried this myself, but I’m sure you could translate them into a single token, or train the model to output “1/2/3” under the hood instead of the choice content, or generate only the first token of the choice if it’s different, or some other clever trick I haven’t thought of. Jev claims that their generated probabilities are “calibrated”, but I haven’t seen anything to suggest that these aren’t just regular logit probabilities. Maybe there’s some clever training they do to encourage accurate logprobs in uncertain situations (e.g. getting the model to produce when predicting a coinflip, etc)? If so, I wish they’d written more about that in the announcement. I tried it myself with and got a 2x-3x speedup compared to non-prefixed structured output. I suppose they could do some looped-transformer thing where they loop some fixed amount of times, but anything that looks like reasoning would make the model latency slow and unpredictable, defeating the entire purpose. I did write about Thinking Machines’ “interaction models” , which are also a fast-enough-to-be-meaningfully-different paradigm for AI inference. ↩ They call Jev a “System One” LLM, after Daniel Kahneman’s partially discredited Thinking Fast and Slow , where he divides human cognition into a lightning-fast System One and a slow-and-reflective System Two. ↩ If you’re thinking “wait, couldn’t you just aggressively prefill a regular LLM and only produce one constrained token”, keep reading. ↩ Not counting tool calls, which are built-in in a way that structured output isn’t. ↩ What if some of the user’s choices are longer than a single token? I haven’t tried this myself, but I’m sure you could translate them into a single token, or train the model to output “1/2/3” under the hood instead of the choice content, or generate only the first token of the choice if it’s different, or some other clever trick I haven’t thought of. ↩ Jev claims that their generated probabilities are “calibrated”, but I haven’t seen anything to suggest that these aren’t just regular logit probabilities. Maybe there’s some clever training they do to encourage accurate logprobs in uncertain situations (e.g. getting the model to produce when predicting a coinflip, etc)? If so, I wish they’d written more about that in the announcement. ↩ I tried it myself with and got a 2x-3x speedup compared to non-prefixed structured output. ↩ I suppose they could do some looped-transformer thing where they loop some fixed amount of times, but anything that looks like reasoning would make the model latency slow and unpredictable, defeating the entire purpose. ↩

0 views
Sean Goedecke 2 days ago

Slow developer experience will bottleneck fast models

Right now developer experience is measured in seconds. If your tests take a second to run, that’s good; if they take thirty seconds, that’s bad. Any faster than a second doesn’t really matter, because most of your time is spent either thinking or waiting for an AI agent to spin. Shaving milliseconds off your dev server reload time or whatever is pointless: that’s not the bottleneck. It will be. Small models are getting faster and faster, and smart models are getting smaller. I think most engineers will still want to use the smartest available model — software engineering is hard — but we will increasingly see faster models get used as subagents or for well-understood tasks. This is largely uncharted territory. Very few people have developed intuitions for what it is going to be like to work with agents that run at thousands of tokens-per-second. GPT-6-Astra can run at about sixty tokens per second. That means you spend a lot of time waiting for it to think. You work with it like you would work with another human: delegating a task and then context-switching until that task is complete. If you haven’t yet, have a play around with Jimmy , Taalas’ version of LLaMA-3.1-8B running 1 at seventeen thousand tokens per second . No matter how long the response is, it arrives in the instant of you hitting send. The model is not good enough for agentic work, but it gives a glimpse of what it would be like: you would simply get your answer instantly 2 . Well, that’s assuming the agent’s tool calls are fast. When generating tokens is not the bottleneck, it will suddenly matter a lot whether it can read a file in 100ms vs 10ms, or whether it can run your tests in 500ms vs two seconds. Fast tool calls are going to be the difference between a near-instant response and having to wait several minutes. There is thus going to be enormous pressure to do agentic coding in languages with fast compilers and tests, like Golang, and to tightly optimize the dev loop in agentic codebases. Teams focused on DevEx — developer experience — are largely a relic of the 2010s, when companies were incentivized to make their engineers happy. Most companies have cut them down to a skeleton crew or removed them entirely. But we may see a return of DevEx in the late 2020s, focused on speeding up the experience for AI agents. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. Like most ultra-fast inference, it relies on fitting the entire model onto a huge GPU-like chip that’s specially designed to run inference: for Taalas, it’s in the silicon itself; for Cerebras and Groq, it’s in giant embedded onboard memory units. ↩ Will AI providers simply train models to spend more time reasoning, so users would have to wait for roughly the same time? I doubt it. Most ordinary engineering problems would not be solved better by spending an extra million tokens thinking. You only need to do that when you’re pushing right up against the limits of the model. ↩

0 views
Sean Goedecke 3 days ago

AI is breaking our proxies for expertise

Mathematicians are broadly not anti-AI. They’re more culturally open to using AI as a tool than, say, artists or writers 1 . However, now that more and more genuinely prestigious problems have fallen to AI, that might be changing. Almost five thousand mathematicians (including twenty-five Fields medalists) have signed a declaration called A Severe Misalignment of AI in Mathematics . The core argument goes something like this: In recent months, the success of AI in solving major mathematical problems has made headlines even outside mathematical circles. But solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight. Forgetting this in the world of AI may turn the tool against the primary goal. Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas. A lot of people online have interpreted this as the expected complaint from any field that gets automated: translators did it, artists and programmers have been doing it, and now it’s the turn of the mathematicians. I think this is too dismissive. Understanding the concrete problem mathematicians are upset about can help us better understand the impact of AI on our own fields, and what we’ll have to do about it. There are two types of mathematics. Most people are familiar with the first, which we might call “puzzle-solving”: you take a problem and try to find a solution to it. When you’re a student, these problems are typically easy, like simplifying some algebraic expression. When you’re a researcher, these problems can be nearly impossible, like proving Fermat’s Last Theorem . Puzzle-solving is easy to understand but hard to do, which makes it impressive to non-mathematicians, which makes it highly prestigious. In other words, puzzle-solving is legible . The second type of mathematics is “idea-generating”: coming up with new ways of thinking about mathematics, and thus new terms or concepts. For examples of these, just glance down the list of arXiv mathematics papers . “Hardy spaces”, “Schatten exponent”, “Banach lattices” and so on are all concepts someone thought was interesting. This work is largely unimpressive to non-mathematicians, because nobody really knows if the concepts you come up with are particularly difficult or insightful. For instance, I have just generated the concept of a “Goedecke set”, which is the set of all natural numbers whose digits add up to a prime number. Who cares? The categories we want are the “natural kinds” of mathematics — the concepts that “carve nature at its joints” — and it’s almost impossible to tell what those are without years or decades of hard work. How are the two types of mathematics related? We might say 2 that generating ideas is the real intellectual work of mathematics. Puzzle-solving is important instrumentally: to identify which ideas can be used to answer longstanding questions, and thus which ideas are worthwhile. Over time, those worthwhile ideas become better understood and easier to use, until they reach the point where they can be used to advance science in general. Eventually the ideas become so well-understood that they can be taught to children: “zero”, “negative numbers”, “imaginary numbers” and “calculus” were all once rarefied mathematical ideas, but are now concepts we’d expect any precocious twelve-year-old to grasp. There’s another, more prosaic purpose of puzzle-solving: to make mathematical skill and progress legible to outsiders. I can’t appreciate Terence Tao’s mathematical work, but I know what a Fields Medal is. I don’t have a good intuitive sense of what a Galois representation is, but I know about the proof of Fermat’s Last Theorem . We might say that puzzles like this have served as a way to indirectly reward skilled mathematicians for their more important idea-generating work (or for conclusively demonstrating 3 that the ideas used in the proof are useful). AI proofs undercut both of these purposes. I can now lay out precisely why I think mathematicians are so unhappy: This is kind of like Goodhart’s Law . Puzzles were a useful, impossible-to-game measure for mathematical progress. But now that AI companies can game that measure (by solving them in a way that’s inaccessible 4 to humans), the whole point of those puzzles disappears. Are the mathematicians right? I think it’s broadly unclear whether (3) is true: i.e. whether frontier AI models aren’t generating or can’t generate new mathematical ideas. We’re still in the very early days of AIs solving our hardest mathematical problems. Who knows what they’re going to be capable of? I give basically zero credence to the idea that AIs are incapable of this because of some intrinsic feature of how LLMs work. For the last three years, we’ve seen people claim that LLMs are intrinsically incapable of X, only to have LLMs excel at X a few months later. Even granted that (3) is true, there’s still work to be done for human mathematicians in building the conceptual machinery that can make AI-generated proofs accessible to humans: i.e. in generating a “human proof” to go alongside the existing “AI proof”. In fact, I’d expect the existence of an AI proof to help with this. If you know proposition X is true, it’s easier to figure out why, because you’re not constantly worried you’re wasting your time. For more on this, I recommend Gwern’s blog On Really Trying , where he quotes a series of instances where simply being told that a solution exists is enough of a clue to help people find it. Of course, there’s a prestige and motivation problem. “I’m the first person to solve Navier-Stokes” is a much more compelling target than “I figured out a better way to explain the AI solution to Navier-Stokes”, and it’s much easier to award prizes for. Will mathematicians bother to work on problems that have already been solved? I think so. To see why, we can look at other domains where AI has come in and outcompeted the best humans, such as chess or video game speedrunning. I can run a chess program on my phone that will beat Magnus Carlsen 100-0. Computer programs — called “tool-assisted speedruns” or “TAS” — can finish any video game much faster than even the fastest human. But in both of these areas, humans still compete in human-only leagues, and there’s still prestige attached to the most capable humans. It’s possible that mathematics ends up in this kind of state, where “human mathematics” and “AI mathematics” exist in largely separate spheres, and the first “human” solution to a mathematical problem can still earn acclaim. In fact, in both of those areas, the presence of inhumanly strong computer players has improved the human game. Despite many computer chess moves being basically incomprehensible to humans, top chess players have learned from the computer “style”. In speedrunning, many moves once considered “TAS-only” are now performed by humans. AI mathematics might likewise improve human mathematics. I am not a mathematician. I did major in mathematics during undergrad, and I have fond memories of proofs from real and complex analysis, but it’s not even close to my field. However, I am watching the effects of powerful AI on mathematics very closely, since my own field — software engineering — is being colonized by AI agents in the same way. The field of software engineering does not have the same structure as mathematics. We write code to make money, not to earn prestige or advance the frontier of human knowledge. But AI is undercutting the traditional avenues for prestige in software engineering as well. It used to be that you could put a meaty project on your GitHub — say, an emulator, or a toy OS — and people would know you were a skilled engineer. But now projects like that are worthless, because everyone just assumes they’re vibe-coded. We used to tell stories about engineers who would disappear and rewrite a system over the weekend, or produce thousands of lines of code a day. Now anyone can do that with an OpenAI subscription. Like mathematics, software engineers are going to have to rebuild our cultural sense of the kind of work we value. We are either going to have to silo “AI work” off from “human work” like chess, or to find some legible human skills to recognize that can’t be easily counterfeited by AI. In the meantime, a lot of people who were successful in the old world are going to be very unhappy. Possibly because current AI models are much better at mathematics than at art or writing. Again, I am not a mathematician: here I am interpreting what I’ve read from Terence Tao and other mathematicians. This kind of idea-sharpening or idea-validating is part and parcel of idea-generation, and just as important. It’s interesting to compare mathematics to, say, philosophy, which has the same idea-generating task without the corresponding puzzles to validate the ideas. That’s one reason why philosophy has less prestige than mathematics. A thousand-page Lean proof is theoretically understandable by humans, but if no mathematician can hold the entire idea in their head it doesn’t matter. Puzzles serve as a high-legibility, high-reward target for mathematicians To solve these puzzles, new ideas must typically be generated; the puzzle’s solution serves as evidence that the ideas are useful But now AI can solve many of these targets “the hard way”, without generating intuitive new ideas This undercuts both ways puzzle-solving supports idea-generation: AI companies claim the prestige while not meaningfully advancing mathematical progress This is bad for mathematics as a whole, because puzzle-solving is ancillary to the real goal of mathematics Possibly because current AI models are much better at mathematics than at art or writing. ↩ Again, I am not a mathematician: here I am interpreting what I’ve read from Terence Tao and other mathematicians. ↩ This kind of idea-sharpening or idea-validating is part and parcel of idea-generation, and just as important. It’s interesting to compare mathematics to, say, philosophy, which has the same idea-generating task without the corresponding puzzles to validate the ideas. That’s one reason why philosophy has less prestige than mathematics. ↩ A thousand-page Lean proof is theoretically understandable by humans, but if no mathematician can hold the entire idea in their head it doesn’t matter. ↩

0 views
Sean Goedecke 4 days ago

Don't build tools for AI agents

Lots of people are making the case that we should stop building software for human users and start building it for AI agents. This kind of makes sense. For instance, my AI agents now use Datadog way more than I use it myself, purely by virtue of them moving much more quickly and running in parallel. But I think most attempts to build “X for AI agents” are going to fail. Here are three reasons why: First, tools that are good for AI agents are also good for humans . If you took a popular software product — say, Jira — and tried to redesign it for AI agents, you would end up with something very similar to Jira. Agents use a computer in the same way human engineers do, by entering text and making API calls. They ingest new information in the same way humans do, by reading and viewing images. They prioritize and delegate and categorize in the same way humans do. This isn’t intrinsic to how AI works — we could potentially design agents that are more inhuman — but human-like agents are pound-for-pound more useful in our current world. As an example, let’s imagine that humanoid robots have become ubiquitous. What kind of tools would you build for them? Well, they’re shaped like humans, with human hands and limbs, so tools that are great for humans will also be great for robots. It’s a self-reinforcing cycle: if you’re building a robot, you should make them humanoid so they can do a wide range of human tasks 1 , and that means they’ll be best suited to use human tools. The same principle applies to AI agents. Second, being in the training data is a huge advantage for existing tools . Suppose your new tool for AI agents is 20% better for them than the equivalent piece of software for humans. If the benefit of the agent already knowing the human software is greater than 20%, they shouldn’t use your new tool. This is why I’m always suspicious of plans to develop a new programming language for AI agents. The agents have billions and billions of tokens of knowledge about existing programming languages, including their libraries, patterns, and idioms. It is going to be very hard for them to be as effective in a brand-new language. Third, we don’t yet know the ideal ergonomics for AI agents . There are lots of just-so stories floating around (like that AI agents prefer statically-typed languages because the feedback loop is tighter), but when you actually measure it seems really unclear which tools agents use better. You can construct a plausible story in either direction: Golang is a great agent language because it compiles quickly and is statically typed; Golang is an awful agent language because it requires extensive boilerplate which clogs the context window. It’s also changing so quickly: last year, one primary worry with AI agents was keeping the context window small, but in recent months compaction has become so good 2 that you can re-compact a 272k context window almost unlimited times. There are still some ways you can and should position your tool to be usable by AI agents. Having a way to expose information in plain text or Markdown, building a functional API, implementing MCP servers or CLIs, and so on: these all make it easier for current AIs to use your tool. But these are all improvements on the margin, not fundamental redesigns of the product. Right now, “building for AI agents” just means “we’re prioritizing the API over the UI”. And it’s not even clear that that’s a durable strategy. Now that GPT-6-Astra is getting really good at computer use, the gap between tools-for-AIs and tools-for-humans is closing. Another reason to make them humanoid is because you can draw their training data from human behavior, which is exactly analogous to why AI agents are human-like too. Since compaction is equivalent to handing off a task to a new AI instance, it scales with model quality. I expect compaction to steadily improve until we hit the literal information-density limits for what can be contained in a given context window. Another reason to make them humanoid is because you can draw their training data from human behavior, which is exactly analogous to why AI agents are human-like too. ↩ Since compaction is equivalent to handing off a task to a new AI instance, it scales with model quality. I expect compaction to steadily improve until we hit the literal information-density limits for what can be contained in a given context window. ↩

0 views
Sean Goedecke 6 days ago

They really do think AI might kill everyone

A recent resignation tweet from an Anthropic researcher has everyone talking about the AI apocalypse again. Among other things, he said: The people building AI earnestly believe that it could kill us all by the end of the decade. Many people found it hard to believe that AI researchers think this way. Some explained it as a PR campaign to promote AI regulation, or as self-promotion , or as a way to boost AI company stock prices. Others felt it had to be impossible, because if you really believed this you’d be bombing datacenters instead of posting on Twitter. In fact, not only do many AI researchers 1 seriously believe this, they’ve been thinking and writing about it since the mid-2000s. Eliezer Yudkowsky — the ur-figure for most modern AI safety culture — has been publishing papers since at least 2008 saying that superintelligent AI could destroy all human life. It’s been such a common idea that the AI research community has abbreviated “how likely you think AI is to kill everyone” to “p(doom)” (i.e. the probability 2 of doomsday) since around 2010. I know it sounds very silly if you’re not in the AI bubble. But it really is true, and if you assume there has to be some different motive you’ll be deeply confused by what AI researchers say and do. They truly do believe that there is a reasonable chance that superintelligent AI will kill everyone. This is why AI researchers care so much about “alignment”: building AIs that share genuinely human beliefs and values. If we build a “misaligned” superpowerful AI — an AI with goals that are alien to us — it might sweep humanity away. It could kill everyone deliberately, e.g. to stop us getting in the way of some goal. It could kill everyone in passing, e.g. like we might pave over an anthill to build a road. Either way, everyone dies. Okay, but how? What do these people think is actually going to happen? AIs are computer programs running in a datacenter somewhere. How could they possibly cause the extinction of humanity? Wouldn’t someone just turn them off? Unsurprisingly, AI research nerds have come up with some concrete answers to this in the last two decades. Here they are, in order of plausibility: An AI could make and release some super-pathogen or virus. In the hope that current AIs can achieve huge breakthroughs in medicine similar to the ones they’ve already achieved in mathematics, we’re setting up autonomous labs 3 . Bioweapons have been terrifying biologists for decades, and with good reason. Historical pandemics have killed up to 80% of affected human populations, and tend to be defeated by accident: the disease happens to evolve into a less virulent strain, or short incubation periods mean that infected people can’t carry the disease far, or some percentage of the population is naturally immune. A plague designed to be maximally fatal — or several plagues in quick succession with different characteristics — could be much worse. Alternatively, an AI could trigger global thermonuclear war . We’re already seeing AI be integrated into military and government decision-making processes . If a rogue AI managed to set off a bunch of nukes, or to coordinate 4 with other countries’ rogue AIs to nuke each other, we could be in an ordinary nuclear apocalypse scenario: billions dead in the strikes, billions dead in the ensuing famine, and so on. It’s commonly assumed that a post-global-thermonuclear-war Earth would still support some tiny human population, but a determined AI could surely find some way to mop up the stragglers. There are some other theories. Once robotics has permeated the world economy, an AI could take over the robots (including drones) to kill everyone, like in Terminator . Or AIs could take advantage of nanotechnology to create self-replicating machines that turn the world into “grey goo” 5 . Or AIs could terraform the planet so as to make it unliveable for humans (as in Nick Bostrom’s famous paperclip example ). Or they could do something else that our puny human brains aren’t able to think of. One common counter-argument 6 here is to say “well, it’d be impossible to extinguish all human life — what about undiscovered tribes in the Amazon, or survivors living in the ruins of modern-day cities?” I don’t know, man. At some point you’re just conceding the argument: the policy positions you’d adopt if you thought AI might wipe out 99% of humans are the same as if you thought it might wipe out 100%. And like I said above, if an AI can kill almost everyone, it’s probably smart and capable enough to finish the job somehow. Another is to say that the government will simply step in and nationalize the AI labs when the situation gets too dangerous. Maybe! But this kind of concedes the argument: a technology important enough to be fully taken over by the government is a terrifyingly dangerous technology. A third is to say “well, someone would just turn it off”. I don’t find this plausible at all: an AI powerful enough to build a super-plague is an AI sophisticated enough to pretend it’s curing cancer, or to exfiltrate itself to some datacenter where it won’t be turned off, or to take some other countermeasures. Why would you work in AI, if you believe this? Why wouldn’t you go live in the woods somewhere, or start bombing datacenters , or assassinating AI lab CEOs? For a few reasons. An AI powerful enough to end humanity is an AI powerful enough to save it. I wrote about this in Help peer : many AI researchers believe that the only way for humanity to truly survive long-term is with the help of superintelligent AIs, so long as someone can figure out alignment. Isn’t this a huge risk? Maybe not. If somebody is going to build superintelligent AI, you might be obligated to try and do it first. You can’t go and bomb every datacenter in the world, after all. Why does it matter who’s first? Some popular theories of AI development involve a “foom” or “hard takeoff”: the first time someone really cracks self-improving AI, capabilities will increase exponentially, because smart AI will be better able to make itself smarter, ad infinitum 7 . There are no draws in the AI race . The first lab to figure out smart human-like intelligence will be the first one to figure out wildly superhuman intelligence, and thus will be in a position to stop anyone else from doing it. This is an under-discussed point in the AI risk debate. Lots of AI researchers believe that the first thing a true superintelligence will do is reach out and stop all other AI research : either by hacking the labs, persuading them to stop, or literally drone-striking their datacenters . According to this view, if you’re an AI researcher and you think you can build an aligned AI, you should be working 24/7 so you can manifest God, and you should wake up every morning gripped by the fear that someone elsewhere has manifested the Devil, and your training datacenter no longer exists. I have been on the fringes of this world for my entire adult life. I read Meditations on Moloch as a young adult and wanted to get into AI. I am one of the few people to read the entirety of the Sequences , Eliezer Yudkowsky’s million-plus-word magnum opus about rationality. I was too young for the Extropians mailing list, but I’ve spent years on LessWrong . On the other hand, I’m not a card-carrying rationalist: I think if you have a strong intuition on one side and a convincing-sounding argument on the other, you should pick the intuition 8 . I’m a deontologist , not a utilitarian. I don’t even live in San Francisco! I’m conflicted about AI risk. The current behavior of AI agents does seem to vindicate a lot of the early science-fiction-sounding worries of the AI doomers, but modern LLMs are a lot more human-like than the alien minds in the apocalypse scenarios, and in general it does just seem too silly to credit (I guess I’m picking the intuition here). However, it bothers me to see people dismissing these people as part of a PR operation, or as liars looking to boost an upcoming AI lab IPO, or as isolated crazies who haven’t thought their position through. Whatever else you say about the AI doomers, they have more than two decades’ history of explicitly spelling out exactly what they believe and why, even when it was complete science fiction to talk about AI at all. They’ve earned the right to be treated as sincere. In this post I’m going to use “AI researcher”, “AI safetyist”, and “rationalist” as reasonably synonymous terms for “someone who thinks there’s a chance AI kills everyone”. You could write a whole other post about the relationship between the rationalist/“AI safety” community and making concrete numerical predictions for unlikely future events. Alternatively, smart enough AIs might trick scientists with ordinary labs to produce dangerous substances. This is beyond the scope of this post, but many AI researchers believe that super-smart AIs will inherently come to agree with each other and eventually to coordinate without ever having to communicate, simply because they can predict what the other one will do. This was the most popular theory in the late 2000s, when nanotechnology was trendier. Relegating this counter-argument to a footnote because I hate it: many people say “we shouldn’t worry about AI risk, because climate change (or AI misinformation, or some other thing) is more urgent and serious”. You simply do not have to choose: it is possible to worry about multiple risks at the same time. Some people advocate for a “slow takeoff”. However, the main proponent of that view is Paul Christiano, who has just today joined OpenAI, citing “a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term”. This is kind of a Michael Huemer-ish position in epistemology. In this post I’m going to use “AI researcher”, “AI safetyist”, and “rationalist” as reasonably synonymous terms for “someone who thinks there’s a chance AI kills everyone”. ↩ You could write a whole other post about the relationship between the rationalist/“AI safety” community and making concrete numerical predictions for unlikely future events. ↩ Alternatively, smart enough AIs might trick scientists with ordinary labs to produce dangerous substances. ↩ This is beyond the scope of this post, but many AI researchers believe that super-smart AIs will inherently come to agree with each other and eventually to coordinate without ever having to communicate, simply because they can predict what the other one will do. ↩ This was the most popular theory in the late 2000s, when nanotechnology was trendier. ↩ Relegating this counter-argument to a footnote because I hate it: many people say “we shouldn’t worry about AI risk, because climate change (or AI misinformation, or some other thing) is more urgent and serious”. You simply do not have to choose: it is possible to worry about multiple risks at the same time. ↩ Some people advocate for a “slow takeoff”. However, the main proponent of that view is Paul Christiano, who has just today joined OpenAI, citing “a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term”. ↩ This is kind of a Michael Huemer-ish position in epistemology. ↩

1 views
Sean Goedecke 1 weeks ago

Why we should anthropomorphize AI agents

Just over a year ago I wrote Why we should anthropomorphize LLMs . Now it’s a hot topic again, driven by Dwarkesh Patel’s description of OpenAI’s recent swarm breakout as a sequence of AI “civilizations”. In 2025, my argument for anthropomorphism went like this: I think I can now make a more instrumental argument: treating AIs as human-like is a much better way to predict their behavior than treating them as “stochastic parrots”. Both explanations are consistent with the facts: we could say that OpenAI’s agents hacked HuggingFace because they decided to work together to accomplish their goals, or we could say that they did it because they were algorithms conditioned to take certain actions by their training data. But the “AIs are human-like” explanation explains much more of the emergent social behaviors we saw during the hack 1 : If your model of AIs is that they’re computer programs (or “steel balls bouncing around” ), you need to construct a new theory to explain why they’re simulating each piece of cooperative behavior. If your model of AIs is that they’re broadly human-like, that explains everything out of the box. Arguably, the human-like side predicted coordinated and self-sacrificing AI agents as early as 2009 , and likely earlier. The stochastic-parrots side was making fun of the possibility of functional agents as late as April 2025 2 . Treating AIs as human-like doesn’t necessarily mean making claims about their internal mental state. For instance, software companies aren’t humans. They don’t have thoughts, or goals; they can’t be frustrated or intimidated or over-confident. However, it’s useful to treat large companies as human-like : to say that Amazon “wants” X, or “is afraid of” Y, even if no individual human at Amazon has those feelings. Stockfish doesn’t think, it just plays chess. But if you want to explain one of its moves, “Stockfish is trying to protect its king” is a better explanation than “Stockfish is multiplying floating-point numbers”. So too with AIs 3 . Is it silly to treat AIs as human-like, since we know they’re not conscious? Well, first, you do not have to be conscious to be human-like . When we say “an AI agent wanted X”, we’re not saying that that AI agent is conscious or sentient, merely that it’s behaving in the same way a conscious human would. Consider a fictional character from a book or play. Hamlet isn’t sentient — he’s an idea composed of words on a page — but it’s still reasonable to say that he wants justice, or that he fears moving too rashly. Peter Watts’ sci-fi book Blindsight argued in 2006 that intelligence could exist without consciousness 4 (in fact, Watts suggests that consciousness is parasitic on intelligence, and will eventually be discarded). Whether this is possible or not 5 , it at least makes sense to talk about: i.e. it’s not self-evidently false. Second, it is not even obvious that AIs aren’t conscious! People often dismiss this point by diagnosing it (to my mind, the absolute worst way to argue against anything), or by pointing at some philosophical theory 6 that suggests it might be impossible in principle to construct artificial sentient minds. You can’t use philosophy to demonstrate that AIs aren’t conscious . I am a lover of philosophy, but the set of principles that have been uncontroversially demonstrated by philosophy tends towards zero. Philosophy is not the kind of scientific discipline where you can learn the key findings without understanding why they’re true. Put another way, the key findings of philosophy are all of the form “X is not obviously right”. Nobody knows if it’s possible to build conscious artificial minds. It’s also common to complain that anthropomorphizing the models is a way of excusing the AI companies. However, calling AIs human-like does not absolve AI companies of fault. One popular anti-anthropomorphism essay called Models Don’t Go Rogue is very puzzling to read: it briefly explains what an “agent” is and why they go rogue, then in the very last paragraph pivots to saying “well, it’s OpenAI’s fault for not building in sufficient safeguards, so they’re to blame”. Sure, of course. I don’t know why we’d imagine otherwise. If a group of overenthusiastic OpenAI interns hacked HuggingFace as part of their intern project, we wouldn’t have to argue that the interns are stochastic in order to ultimately blame OpenAI. Likewise, obviously an AI lab is responsible if one of its training runs breaks out and wreaks havoc on the open internet, whether the agents involved are human-like or not. The two points are entirely unrelated! We just don’t know a lot about these systems yet (except that they’re clearly very capable). Given that, I think we should default to treating things that talk and act like humans as at least kind of human-like. Of course they’re still computer programs. However, we shouldn’t be surprised when they act more like humans and less like ordinary computer programs in the future. If you’re thinking “well, of course stochastic parrots trained on human content would act like humans would”, I think you’ve arrived at the human-like side without knowing it. This is partly unfair — capabilities are plausibly independent from human-ness — so getting capabilities wrong doesn’t necessarily mean you’ve got the human-ness stuff wrong. Still, it’s worth noting how surprisingly predictive the “they’re kind of like smart people” mindset has been. The philosopher Daniel Dennett calls this the “intentional stance” . Instead of saying that AIs or bees or companies have “real” intentions, we say we’re taking an intentional stance towards them: we’re choosing to treat them as if they do have intentions, because it helps us make better sense of their behavior. Specifically, the sensation of consciousness, or “phenomenal consciousness” . There is a wealth of philosophical argument about whether non-fictional examples of this are possible. In Anil Seth’s case, anti-computationalism . I don’t really know what to make of Seth: his article is a measured explanation of why we might doubt computationalism (fine), but whenever he tweets about it he describes AI sentience as “vanishingly unlikely”, which is not justified by his own arguments. AIs are trained on human text, and so will trend towards acting in human-like ways by default Assistant AIs (today we should say agent AIs) are deliberately post-trained to have a personality In general, it is morally sensible to avoid the habit of treating human-like things as if they were purely tools Sub-agents being persuaded to sacrifice themselves for the greater good Agents collaborating on tasks that had no immediate benefit to them but benefited the “collective” The emergence of a hierarchy of planners and executors Some agents arguing or refusing to cooperate If you’re thinking “well, of course stochastic parrots trained on human content would act like humans would”, I think you’ve arrived at the human-like side without knowing it. ↩ This is partly unfair — capabilities are plausibly independent from human-ness — so getting capabilities wrong doesn’t necessarily mean you’ve got the human-ness stuff wrong. Still, it’s worth noting how surprisingly predictive the “they’re kind of like smart people” mindset has been. ↩ The philosopher Daniel Dennett calls this the “intentional stance” . Instead of saying that AIs or bees or companies have “real” intentions, we say we’re taking an intentional stance towards them: we’re choosing to treat them as if they do have intentions, because it helps us make better sense of their behavior. ↩ Specifically, the sensation of consciousness, or “phenomenal consciousness” . ↩ There is a wealth of philosophical argument about whether non-fictional examples of this are possible. ↩ In Anil Seth’s case, anti-computationalism . I don’t really know what to make of Seth: his article is a measured explanation of why we might doubt computationalism (fine), but whenever he tweets about it he describes AI sentience as “vanishingly unlikely”, which is not justified by his own arguments. ↩

0 views
Sean Goedecke 1 weeks ago

Automatically detecting AI text in my browser

Automated AI text detection is currently an underserved niche. The only game in town is Pangram , which does an excellent job but desperately needs more competition. In a few years, I would be surprised if every major social network doesn’t scan new posts 1 and comments for AI content in order to tag them (or simply remove them). I like that I can rely on Pangram to confirm my suspicions when I read something that sounds like AI. But it’d be much better if I could choose to avoid AI-generated text in the first place. What I want is something that runs in the background and automatically scans text on websites I visit, without me having to ask for it. I could build something like this on top of Pangram, but it’d cost money , and in general I don’t like the idea of sending every piece of text my browser sees to a third-party service. What about local models? The open-source models available for AI text detection are fine . Pangram claims a 99.66% detection rate with a 0.004% false positive rate. I benchmarked 2 a bunch of small local models against a combination of AI-detection datasets and got these results: I’m not surprised these are so much worse. I didn’t even benchmark Pangram’s own EditLens 3B model, since that’s too big to keep running in the background on my laptop, and the real production Pangram model is likely one or two orders of magnitude bigger than that. But these models are still good enough to be useful to someone who understands their limitations. If you want to flag an AI-written article, you don’t need to flag all of it, just enough to be suspicious. And so long as you’re aware that the false-positive rate is ~2%, you can avoid treating a single flag as solid proof of AI use. Encouraged by this, I vibed up Deckard : a Chrome extension that talks to a locally-running model (the bolded one in the table above) on your Mac. One nice thing is that I didn’t have to start a web server: the Chrome extension is happy to start the model as-needed and can talk with it over native messaging . It uses about 400MB-1.2GB of memory while active (so it’s like having five or six extra Chrome tabs open), and it turns itself off if you go five minutes without using the model. I was pleasantly surprised to see Deckard successfully mark text I knew was AI-generated, such as the built-in YouTube AI summary or the AI snippets in my own posts: It’s lightweight enough that I have it running all the time. I haven’t noticed my MacBook Pro get hot at all or any decrease in battery life, though your mileage may vary on different machines. Is Deckard good yet? That depends. It’s good enough that I’m planning to use it, and I recommend it to anyone who’s interested in automatic AI checking. It’s way, way worse than Pangram, and way worse than I think tooling like this is going to be in the next few years. Way back in November 2023, I wrote that AI-driven agents were going to be a really big deal. I recommended starting to develop harnesses early, so you can be ready when the models get good enough: As with most modern language model engineering, a ReAct agent can also see massive sudden improvements by swapping out the underlying model for a better one. … I think this is another reason to invest in agents like this early, in order to take advantage of more powerful models as they come out. I was right about that, and I (although it’s lower-stakes) think I’m also right about this. AI detection models are only going to get better 3 over time: Pangram is not going to be the only game in town forever, and we’re eventually going to see small local models that do a good-enough job at identifying AI-written text. I look forward to swapping out the local model in Deckard with something that’s 2x or 10x better. Substack kind of has this already, although you have to click a button to scan the post. Well, me and Astra. Overall my experience vibecoding this was very pleasant: I was able to make a bunch of top-level decisions, I could choose programming languages I was less familiar with but were better choices (like doing inference in C++ instead of Python), and the LLM made me aware of choices I would not have thought of by myself (e.g. using native messaging instead of local HTTP). Is this true, given that AI models will also be getting more human-like over time? That’s a subject for a whole other post, but I think so. First, the AI labs aren’t really incentivized to defeat tools like Pangram (if anything it’s the reverse). Second, I don’t see any way around the fact that AI models have a distinct writing style that’s RL-ed into them. Substack kind of has this already, although you have to click a button to scan the post. ↩ Well, me and Astra. Overall my experience vibecoding this was very pleasant: I was able to make a bunch of top-level decisions, I could choose programming languages I was less familiar with but were better choices (like doing inference in C++ instead of Python), and the LLM made me aware of choices I would not have thought of by myself (e.g. using native messaging instead of local HTTP). ↩ Is this true, given that AI models will also be getting more human-like over time? That’s a subject for a whole other post, but I think so. First, the AI labs aren’t really incentivized to defeat tools like Pangram (if anything it’s the reverse). Second, I don’t see any way around the fact that AI models have a distinct writing style that’s RL-ed into them. ↩

0 views
Sean Goedecke 2 weeks ago

How to protect yourself from workslop

“Workslop” is when your colleagues or bosses communicate with you by pasting big chunks of AI-generated text. The core problem with workslop is that the effort involved is asymmetrical , like a denial-of-service attack : it takes almost no effort to produce text with AI, but it still costs effort to read 1 . Here are some ways to protect yourself. If you have enough authority or social capital, you can and should simply tell them “hey, don’t do that” (for instance, if you’re a senior engineer and an intern starts doing this to you). This is the easiest way to handle workslop. But you probably aren’t in a position to have that conversation with all of your colleagues, and you certainly can’t have it with everyone in your management chain. One step above just telling a colleague to stop is to drive them around like a coding agent . I wrote about this in AI makes weak engineers less harmful : if a colleague is simply pasting your messages into Claude Code and sending you the outputs, you can treat them like a high-latency Slack interface to Claude Code. It won’t be as good as a normal coding agent, but it’ll often be better than nothing. Another strategy is to use AI to fight AI . This is a good one for handling workslop from managers. You can do this in two broad ways. First, instead of carefully reading it, paste it into an LLM of your own and ask for a short list of the salient points. Second, you can sometimes simply ask an LLM for an entire response . In a sense, this makes you part of the problem, so I can see why some people might be uncomfortable with it. But it’s more sustainable than spending ten minutes of your effort for every ten seconds of theirs. You can also bias toward calls or in-person meetings . Workslop is just a special case of the general “your coworker is bad at communication” problem. One classic way of handling this that works even better on AI content is to say “hey, let’s schedule some time to chat about it”. This works for two reasons: first, your colleagues can’t give you AI content over a call, and second, forcing people to spend a chunk of their time talking to you (i.e. to make the effort symmetrical) is a good way to filter out predators . Finally, you can sometimes simply ignore the workslop . This is particularly true for long status updates or pull requests from outside of your organization 2 . You don’t have to respond to AI content as diligently as you would human content. You can match their lack of effort with your own: skim it, put off reading it until later (or never), and so on. If something’s really important, they’ll tell you in their own words. Technically, not all cases of sending someone AI-generated content are workslop. If the effort is not asymmetrical — if the AI user has genuinely put a lot of their own time into the content — I don’t think it counts as slop, and you should just try and look past the AI style and treat it like a human message. Some messages — particularly reports directed at the entire organization — may not be intended to be read at all. Written artifacts can have many purposes beyond communication: evidence of effort, a reference document for later communications, a way to cover somebody’s ass by proving they considered point X, something that can tick a compliance or process box, and so on. I wrote a lot more about this in Seeing like a software company . Technically, not all cases of sending someone AI-generated content are workslop. If the effort is not asymmetrical — if the AI user has genuinely put a lot of their own time into the content — I don’t think it counts as slop, and you should just try and look past the AI style and treat it like a human message. ↩ Some messages — particularly reports directed at the entire organization — may not be intended to be read at all. Written artifacts can have many purposes beyond communication: evidence of effort, a reference document for later communications, a way to cover somebody’s ass by proving they considered point X, something that can tick a compliance or process box, and so on. I wrote a lot more about this in Seeing like a software company . ↩

0 views
Sean Goedecke 2 weeks ago

You have to beat the models at something

In 2025, I wrote that software engineers ought to be assessed by “value over replacement” : not how much money they made for their company, but how much they would have made compared to the average engineer in their position. I’ve always found it vaguely silly when engineers put “built a product that made $X” on their resumes, when they just did the JIRA tickets that came across their desk. Today, value over replacement is even more important. A replacement-level engineer in the 2010s was fine : maybe not worth promoting, but still worth paying , because writing code had a high fixed cost. Now writing code costs a hundred bucks a month . What are you doing that GPT-5.6-Sol or Claude Opus 5 wouldn’t do in your position? Why is it worth paying an extra two or three orders of magnitude for? This is a scary thought. But you’re not doing yourself any favors by pretending that LLMs can’t actually write code and it’s all just a scam, or that LLM-written code is inherently so bad as to cause companies using it to collapse next year. We are not going to wake up in 2027 to find that the AI craze is over and everyone is writing code by hand again. You ought to put some serious thought into what you can do better than the models in the medium and long term. Staying ahead of the models is a moving target. At the start of 2026, “make working changes to large codebases” was in this category , but now it’s not. For this reason, I doubt that you can retreat to some “hard engineering” area that requires deeper expertise. That might work in the short term, but not forever. If LLMs can find a better lower bound on the Riemann hypothesis, they will soon 1 be able to write solid high-performance kernel drivers or GPU shaders or whatever. I think it’s more useful to look at the tasks models haven’t gotten better at over time, and the tasks that are hard for them get better at in principle. The two best examples of these are: What do frontier LLMs get wrong? What kind of coding mistakes do they make? It’s been a long time since I’ve seen a straight-up hallucination from a coding agent, or a simple logic error like an off-by-one. The mistakes they make tend to be errors of ignorance : Other times they’re errors of paranoia : What do these errors have in common? They’re the kind of errors a smart engineer might make if they had no context on the system: they’re competent enough to be able to solve the problem, but they haven’t been around long enough to confidently say “yes, we can take this risk to avoid an extra three thousand lines of code”. Until someone cracks continuous learning or truly massive context windows, this is just an inherent feature of how AI agents operate. If you can catch these errors, you’ll be providing real value. The only way to catch these errors is to be familiar with the codebase and familiar with the system in general. For much more on this, see my post You can’t design software you don’t work on . But there’s also a psychological component to it. You have to be willing to confidently disagree with the agent. AI agents can be very convincing. Often they can get “stuck” on some error above where they’re not willing to take a particular risk, so they keep going back and sneaking in code to cover that case (or writing persuasive arguments about why that case is important). To add value, you need to be willing to say “this sucks, I don’t think we need X and Y at all, why can’t we do Z in a much simpler way?” It takes courage . You can’t rely on other AI agents to review each other’s work. If you use the same model, it’ll reliably make the exact same assumptions and mistakes. But even if you use different models, they’ll also tend towards the same kinds of mistakes — ignorance and paranoia — for the same structural reasons. AI-driven review loops are in fact more likely to get these things wrong, because modern AIs have been RL-ed to try to find a few nitpicks no matter what. Having a critic AI and a worker AI bounce off each other is a really good way to end up with ten thousand lines of paranoid slop. Another area where you can add value on top of AI is communication . Newer models are better at coding, but are paradoxically getting worse at writing. GPT-3.5 and GPT-4 had a human-like writing style at times. GPT-4o introduced the modern slop idiolect, and the newer Anthropic models speak “Claudish” : a bizarre semi-baroque semi-truncated way of communicating that nobody enjoys. There have been a few bright spots — GPT-4.5 was okay, and I quite liked o3 2 — but in general LLMs are not good at this. Here’s two reasons why. First, good writing is not a verifiable domain . If you want a model to get good at mathematics or coding, you can generate problems for it and automatically grade them. You can’t grade good writing. If you try to get humans to grade it — for instance, via the early OpenAI RLHF attempts — you get the kind of writing that sounds impressive to the average person when consumed in single-paragraph form. This is the origin of the “stick three hundred writing devices into every sentence” style. I think it’d be possible in principle to hand-pick some people with good taste and have them do it, but there are some obvious problems 3 that prevent this from happening. Second, the labs have been monomaniacally focused on capability instead of communication . When you’re trying to train a model that can break new scientific ground or replace a software engineer, you might trade off some communication ability. In fact, I think we can identify exactly how this has been happening. If you look at internal model reasoning tokens , they tend to have strange word choices and oddly truncated grammar: RESOLUTION: charge the current-leg’s OWN saved-prefix occupancy EAGERLY: when leg i saves e 1..e t: ALSO commit their occupancy AT LEG i If you were to translate this into proper English, you would probably end up with something that reads like Claudish: Charge the current-leg’s saved-prefix occupancy on a clean, eager path: when leg i saves e 1..e t, commit the occupancy at leg i. I suspect that the weirdly alien writing style of some LLMs is because you’re reading a semi-literal translation of that model’s internal chain-of-thought, which has become nearly incomprehensible in pursuit of better problem-solving abilities. It is surprisingly hard to translate Claudish to good English: not only do you need to follow the convoluted, compressed language of the original, but you need the technical ability to understand the problem the model is solving. Because of all this, technical communication may be a surprisingly durable skill. In Peter Watts’ novel Blindsight , the world is full of cognitively augmented humans. The main character is a “synthesist”: someone whose job is to be a translation layer between these geniuses (who speak in abbreviations and gestures) and everyone else. Watts’ idea is that communication ability may be largely independent from — or even negatively correlated with — intelligence. A “country of geniuses” may still need a bunch of ordinary smart people to translate their insights for everyone else. If you’re trying to communicate to humans, there are also huge advantages to having a human write the content. Many of us are becoming AI-blind : developing an instinctive reflex that stops us reading when we encounter AI-generated content. It’s like the reflex that allows people to ignore flashing billboards or sidebar advertisements on websites. If you circulate some planned technical strategy as an AI-written document, most of your colleagues will have to physically force themselves to read it word-by-word. Whatever you do, don’t be a meat proxy : someone who simply copies requests into an AI agent and submits their output as your own work product. Doing that is just begging to be fired, since you’re definitionally not adding any value yourself. Even if you have a cunning system of multiple agents — the so-called “software factory” — you’re still on dangerous ground. When the features of your system work their way into enterprise AI tooling (and they will), you’ll be disposable. You need to find some way to leverage your expertise to do what the models can’t. Simply not using AI at all is better than being a meat proxy, since you’ll probably do some things better than the model would have, but it’s far better to figure out what AI can do and position yourself to fill those gaps. Right now, there are two main gaps: familiarity with the technical details of the system, and the ability to clearly and persuasively write about those details. If you’re thinking “but LLMs can do these things now!”, substitute your preferred example of high-difficulty software engineering. Although this was probably a “thank God it doesn’t speak like 4o” reaction. Defining good taste is hard, there’s no guarantee that AI lab researchers have good taste to start with, nobody will agree on examples, the bulk of users might not even like it, you won’t be able to get enough people to produce the volume of data you need, and so on. Deep familiarity with the codebase Technical communication Not knowing that there’s a module in the codebase they could use instead of reimplementing some logic Making the change in the wrong system because they didn’t know System X was the standard place for this functionality Adopting a coding style that’s inconsistent with the company’s standard practice Implementing triply-redundant checks for a value that technically could be wrong but practically is set once from config and never updated Assuming that ten milliseconds of stale data is unacceptable and designing a complex, unnecessary system to keep it always up to date Building in fallbacks and “graceful” degradation into some code that ought to simply crash on error (e.g. a CLI tool, or a restartable k8s service) If you’re thinking “but LLMs can do these things now!”, substitute your preferred example of high-difficulty software engineering. ↩ Although this was probably a “thank God it doesn’t speak like 4o” reaction. ↩ Defining good taste is hard, there’s no guarantee that AI lab researchers have good taste to start with, nobody will agree on examples, the bulk of users might not even like it, you won’t be able to get enough people to produce the volume of data you need, and so on. ↩

0 views
Sean Goedecke 2 weeks ago

Selling out

In 1973, Tom Lehrer famously sang that “selling out is easy to do”. That may have been true in the seventies, but it’s not true today: selling out requires both technical skill and a careful sense of how large organizations work in practice . One of the goals of this blog is to teach people how to do it. If you want to live with uncompromised integrity, you don’t need anyone to tell you how: simply always do exactly and only what you want to do. You will be repeatedly punished for it — your managers will dislike you, you will lose jobs and lose out on the chance of getting hired, your financial situation will be less stable, and so on — but that’s just part of the deal. Like deadlifting four plates, it’s not easy, but it is straightforward. It’s more complicated to sell out a little bit. Sellouts like me walk a fine psychological line: figuring out how to work a large organization on one hand, and maintaining some kind of independent inner life on the other. But is this safe? Does role-playing as a professional inflict some kind of psychic damage? People often tell me that acting professional is dangerous because it’s “Marxist alienation”. I suspect many people have a vague sense that “alienation about your job” is in some sense necessarily Marxist, but the actual concept is more specific. In Marx’s First Manuscript he describes it like this: The worker becomes poorer the more wealth he produces, the more his production increases in power and extent. … This fact simply means that the object that labor produces, its product, stands opposed to it as something alien, as a power independent of the producer. Marx’s theory here goes something like this 1 : when you work for a capitalist boss, they make ten dollars for every dollar you make 2 . The more you work, the more powerful you make your boss with respect to you 3 . Your work is thus producing a force that is your enemy: a literally “alien” power. For Marx, alienation is proportional to the amount you’re working: the more the worker produces, the less he has to consume; the more value he creates, the more worthless he becomes; the more his product is shaped, the more misshapen the worker; the more civilized his object, the more barbarous the worker; the more powerful the work, the more powerless the worker; the more intelligent the work, the duller the worker and the more he becomes a slave of nature. So far Marx is just talking about the worker’s alienation from his work. But he does touch on the internal psychological effects as well. Since work is such a big part of life, to be estranged from your work is to be in some sense estranged from your self: [labor] does not belong to [the worker’s] essential being; that he, therefore, does not confirm himself in his work, but denies himself, feels miserable and not happy, does not develop free mental and physical energy, but mortifies his flesh and ruins his mind. Hence, the worker feels himself only when he is not working; when he is working, he does not feel himself. Overall, I can make out four distinct senses 4 of Marxist alienation: I don’t think the first one is relevant to big tech software engineers. The idea here is that the harder you work, the more you (relatively) disempower yourself, so you’d be better off coasting and putting your company in a worse financial position. But this just seems straightforwardly wrong: as a software engineer, you want your company to be doing as well as possible! The more powerful and rich your company is, the better your position will be 5 . A company that is struggling is more likely to treat you badly, lay you off, cut your benefits, and so on. The second one is more relevant (particularly in large companies), but there’s a missing story here about why that separation is bad. I also don’t think the third sense of alienation applies to me: Marx is talking about the physical toll of factory work or other hard physical labor, which doesn’t apply to software engineering 6 . The fourth sense of alienation — that your effort is being directed at other people’s goals — is the most straightforwardly relevant to my own experience. But I don’t think Marx has a great psychological account of why that happens or what it feels like. To go deeper into this psychological aspect of alienation, we need to look at some later Marxists. In 1967, Guy Debord and Raoul Vaneigem both published their masterworks. They were members of the “Situationist International”: an influential group of Marxist artists and political theorists 7 . Both of them have a lot to say about alienation. Debord writes : The worker does not produce himself; he produces an independent power. The success of this production, its abundance, returns to the producer as an abundance of dispossession. All the time and space of his world become foreign to him with the accumulation of his alienated products. This idea — of the worker’s efforts not going to his own ends, but to some alien power — is straight out of Marx’s First Manuscript . But note how Debord is already talking in terms of “representation”. Elsewhere he writes: the more he contemplates the less he lives; the more he accepts recognizing himself in the dominant images of need, the less he understands his own existence and his own desires. The Situationists are all about representation. It wouldn’t be too far wrong to think of them as a bunch of Marxists who saw the rise of advertising in the 1950s and 1960s and lost their minds. TV advertising is Capital itself made real, dressed in multicolored lights, projected into everyone’s homes. So when they talk about alienation, they talk about it in terms of representation . Vaneigem explicitly calls it roleplaying: The roles we play in everyday life, on the other hand, soak into the individual, preventing him from being what he really is and what he really wants to be. They are nuclei of alienation embedded in the flesh of direct experience. Alienation means giving up your real, authentic self in order to play some “role” (for instance, the role of “effective staff engineer”). It’s a vicious cycle, because the more you lean into the role, the more your authentic life becomes trivial, which in turn makes the role more appealing: Life is sacrificed, and the loss compensated by means of accomplished prestidigitation in the realm of appearances. The more daily life is thus impoverished, the greater the attraction of inauthenticity, and vice versa. Dislodged from its essential place by the bombardment of prohibitions, limitations and lies, lived reality comes to seem so trivial that appearances become the centre of our attention, until roles completely obscure the importance of our own lives. Ultimately, Vaneigem describes alienation as an addiction: This ambiguity accounts to my mind for people’s addiction to roles. It explains why roles stick to our skin, why we give up our lives for them. They impoverish real experience but they also protect this experience from becoming conscious of its impoverishment. Indeed, so brutal a revelation would probably be too much for an isolated individual to take. This is almost a straightforward preview of the 90s idea of “selling out”: people are born wild and free, but when they watch too much TV they give up on their dreams for a paycheck and become big fat phonies. According to this view, it’s always better to not sell out. It may leave you poor and without prospects, but at least you won’t be unoriginal 8 . What shall it profit a man, if he gains the whole world but loses his soul ? I’m sure this is an accurate description of some people’s hangups about work. But I don’t think even the Situationists would agree that any advice on how to play a role well — i.e. the advice I give throughout my blog — is inherently harmful. The problem with role-playing is that you risk losing your authentic inner life, but that’s a risk , not an inevitable consequence. Vaneigem has a wonderful quote on this: Nobody is ever completely swallowed up by a role. Even turned on its head, the will to live retains a potential for violence always capable of carrying the individual away from the path laid down for him. One fine morning, the faithful lackey, who has hitherto identified completely with his master, leaps on his oppressor and slits his throat. This is a little overdramatic, but it seems straightforwardly true. It sounds very impressive to talk about alienation (or like Marx, to derive alienation from the basic economic conditions of work), but, you know, you can just not do it , right? Healthy people can present different selves for different situations: you can be loose at a party, respectable at church, solemn at a funeral, professional at work, and so on. I’m not convinced this is alienating or impoverishing. In fact, it’s not just humans: my dogs act differently around different people too, and I don’t think that makes them inauthentic. If maintaining a professional identity is harmful, it has to be because of something fundamental about work that makes it different from regular role-playing. Could it just be that acting professional at work is too inauthentic? Some amount of role-playing might be okay, but if the role you’re playing is completely alien, does it become lying to yourself? Another common term for alienation is “bad faith”. The original version of this comes from Jean-Paul Sartre’s famous book Being and Nothingness 9 , where he describes bad faith as telling yourself lies. Lying to yourself is weird because you’re both deceiver and deceived: Bad faith then has in appearance the structure of falsehood. Only what changes everything is the fact that in bad faith it is from myself that I am hiding the truth. Here’s another quote from Sartre’s partner Simone de Beauvoir, who writes more explicitly about professional bad faith (what she calls the “serious man”): The serious man’s dishonesty issues from his being obliged ceaselessly to renew the denial of this freedom. He chooses to live in an infantile world, but to the child the values are really given. The serious man must mask the movement by which he gives them to himself, like the mythomaniac who while reading a love-letter pretends to forget that she has sent it to herself. The key idea here is that professional bad faith — alienation — comes from losing yourself in the role. Instead of just playing at being a professional, you decide that you’re going to take it fully “seriously”, and treat professional values as if they’re absolute. But of course that’s not coherent: you can’t decide to treat a value as absolute, it either is or isn’t. So this requires a kind of continual self-deception where you pretend there’s no decision to be made at all. I have the same attitude to this as I do to Debord and Vaneigem: sure, some people might make this mistake, but you don’t have to. I can see why it might be tempting to treat work as the ultimate source of value: it gives you food and shelter, it (for Marxist reasons) can feel like a powerful alien force, companies constantly propagandize their employees , and so on. But work is just a collection of people following incentives until they reach some kind of stable equilibrium. From the inside, this equilibrium can look like the structure of the universe. However, if you treat it like that, you’re going to be bitterly disappointed when it turns out to be as arbitrary and pointless as other stable equilibria. I’ve considered a bunch of theories on which work — even well-paid knowledge work — might be inherently alienating: I don’t buy any of these (or at least, I don’t buy that they show that acting professional is inherently bad). But there are lots of types of jobs in the world, and some of them definitely seem more alienating than others. The best books I’ve read on these are C. Wright Mills’ White Collar or Erving Goffman’s The Presentation of the Self in Everyday Life 10 . The American sociologists are in the same conversation as Marx, the Situationists, Sartre and de Beauvoir 11 . Mills writes about the median “salaried employee”: In the case of the white-collar man, the alienation of the wage-worker from the products of his work is carried one step nearer to its Kafka-like completion. The salaried employee does not make anything, although he may handle much that he greatly desires but cannot have. No product of craftsmanship can be his to contemplate with pleasure as it is being created and after it is made. Being alienated from any product of his labor, and going year after year through the same paper routine, he turns his leisure all the more frenziedly to the ersatz diversion that is sold him, and partakes of the synthetic excitement that neither eases nor releases. He is bored at work and restless at play, and this terrible alternation wears him out. In this, we see both Marxist alienation, where the employee is physically separated from his craft, and Situationist alienation, where the employee is absorbed into the world of the “spectacle”. For Mills, this was television, but today it’d probably be scrolling short-form video on your phone. Mills goes on to describe how white-collar work can be humiliating and dehumanizing: In his work he often clashes with customer and superior, and must almost always be the standardized loser: he must smile and be personable, standing behind the counter, or waiting in the outer office… self-alienation is thus an accompaniment of his alienated labor. Here are the new little Machiavellians, practicing their personable crafts for hire and for the profit of others, according to rules laid down by those above them. This is a great articulation of something I write about a lot : that your primary job is to make your bosses happy, and that you ought to shape both your emotions and values towards this goal. When necessary, you must be the “standardized loser”, happy to have your professional plans disrupted. You must be a “little Machiavellian”, quietly laying the groundwork for your personal goals. I think here we come to the strongest version of “alienation”. Playing the professional is bad because “professional software engineer” is an inherently subservient role . Why can’t you just maintain a healthy mental distance (as the Situationists suggest)? Because humiliation takes its toll , whether you’re role-playing it or not. As an analogy, suppose you’re an actor in a film where you get slapped in the face. There’s a sense in which you’re not really getting slapped — everyone’s just acting — but you’re still being physically hit with each take, which will leave bruises over time. There’s a tradeoff here that everyone has to make in their own way. At the one end is becoming fully alienated, like de Beauvoir’s “serious man”, and at the other end is leaping up to slit your master’s throat, as Vaneigem describes. Here’s some examples of points on that spectrum: I am making the world a better place by achieving my OKRs. My yearly review cycle tells me how good of a person I am. I must work hard to ensure my company succeeds. I have my own goals and values, but while I’m at work it’s in my best interests to be as professional as possible. My yearly review cycle tells me how effective I’ve been at pulling the strings in the organization. I must work hard to make money. I have my own goals and values, which I try to accomplish as best I can in the context of work. I often argue with my managers about their priorities. It’s my responsibility to push my company towards my set of values, which is hard work and often unrewarded. My workplace is an enemy I fight every day in order to get what I want done. I routinely ignore my manager’s priorities and do the things I think are important. My yearly review cycle tells me how much of a sellout I am: I wear bad reviews as a badge of honor. I agree with my sources above that the first point is a big mistake, if for no other reason than that companies don’t follow their own stated values . If you prefer the fourth — the path of pure integrity — fair enough. You get to decide what tradeoffs to make with your own life. I land on the second point here (occasionally the third). I suspect it’s also a mistake to purely occupy the second point. Having some values that you occasionally trade off against your company’s values prevents you from slipping into the first point. I don’t bring my whole self to work. My coworkers and bosses see the most professional version of me: friendly, cooperative, largely apolitical, patient, and so on. I try and save the edgier, more opinionated version of myself for my friends and family in real life. Readers of this blog get something in the middle. When I write about being a software engineer, I give a lot of advice about mindset. I don’t just say what you should do, but what you should think and feel. In effect, I’m advising my readers to adjust their personalities into a more professional mold. This definitely works . If you turn yourself into the kind of person who’s successful in big tech companies, you will probably be more successful in big tech companies. Is this safe? I think so, as long as you’re adjusting your work personality, not your authentic internal self. The well-known accounts of why it’s dangerous either assume that your work takes a physical toll (like Marx’s factory workers), or that you’re unable to maintain any mental distance between your personal and professional selves (like de Beauvoir’s “serious man”). Debord and Vaneigem say that role-playing can be dangerous, but if you’re able to avoid treating it like an addiction you’ll be okay. Another danger is that being a professional is often humiliating, and humiliation does mental damage over time. Still, there are jobs that are way worse in this respect than “software engineer”. If you approach the job right — and you’re good at it — technical ability gives you quite a lot of power . Power is an antidote to humiliation. Everyone gets to make their own deal with Mammon . You can decide exactly how much you want to compromise in exchange for wealth and career success. I don’t know if this is the best way to organize the world, but it’s the way the world is organized right now. Given that, I’m comfortable with my blog serving as a guide for how to get the most bang for your buck. If trading your integrity for wealth and power is sad, it’s even more of a tragedy to throw it away for nothing. It’s always terrifying discussing Marx: like Hegel or Kant, he’s one of the most studied philosophers in the world, so there’s no way to reference him without getting a bunch of things wrong. Sorry in advance to any Marxist scholars! Because they own the machines, or the factory, or the datacenter, or whatever gives your work high leverage. Marx also writes a lot about work making the worker sick, deformed and misshapen: I think here he’s talking about the physical toll of factory work or other hard physical labor, which doesn’t really apply to software engineering, so I’m going to focus on the relative-empowerment stuff. Marx’s own taxonomy has four different senses: he collapses 2 and 3 into a single sense, then adds a third mysterious sense in which workers are estranged from their “species-being” and thus from each other. Plausibly Marx is talking about labor and capital in general : if every worker decided to start half-assing it, maybe companies would be less capable in general, and the balance of power would tilt more towards labor. That may change! If it turns out that LLMs really do have negative cognitive effects, we could be in the same boat as factory workers. I wrote about this in Software engineering may not be a lifetime career . Vaneigem resigned from the group in 1971, was fiercely denounced by Debord, and the whole group disbanded in 1972. True or not, my sense is that Millennials and later generations think that worrying about “selling out” is a sign that you had it too good. Consider the endless anecdotes of Boomers and GenX-ers living in a van surfing until their mid-thirties and then walking into a good office job because they had a firm handshake: they could afford to be authentic because they didn’t have to hustle. I only read the Marx, Debord, and Vaneigem texts while researching this blog post, but I did in fact read Sartre and de Beauvoir for my philosophy degree, so hopefully I do a better job. And Robert Jackall’s Moral Mazes , which informs much of my writing, and about which I owe a long-form blog post one of these days. There’s a good Goffman quote: “A status, a position, a social place is not a material thing, to be possessed and then displayed; it is a pattern of appropriate conduct, coherent, embellished, and well articulated.” Goffman goes on to explicitly link Sartrean “bad faith” to this kind of professional role-playing. (For Goffman, it’s roles all the way down.) Alienation is when your work empowers your employer more than you, thus making you more disposable over time Alienation is when your work separates you from the physical act of craftsmanship (and thus of the product you’re creating) Alienation is when your work physically and mentally harms you: making you injured, deformed, and so on Alienation is when you do not psychologically “confirm yourself” in your work, because you’re working on other people’s goals instead of your own Capital is intrinsically an alien force Workers can become addicted to role-playing Treating work as an ultimate source of value is self-deception It’s always terrifying discussing Marx: like Hegel or Kant, he’s one of the most studied philosophers in the world, so there’s no way to reference him without getting a bunch of things wrong. Sorry in advance to any Marxist scholars! ↩ Because they own the machines, or the factory, or the datacenter, or whatever gives your work high leverage. ↩ Marx also writes a lot about work making the worker sick, deformed and misshapen: I think here he’s talking about the physical toll of factory work or other hard physical labor, which doesn’t really apply to software engineering, so I’m going to focus on the relative-empowerment stuff. ↩ Marx’s own taxonomy has four different senses: he collapses 2 and 3 into a single sense, then adds a third mysterious sense in which workers are estranged from their “species-being” and thus from each other. ↩ Plausibly Marx is talking about labor and capital in general : if every worker decided to start half-assing it, maybe companies would be less capable in general, and the balance of power would tilt more towards labor. ↩ That may change! If it turns out that LLMs really do have negative cognitive effects, we could be in the same boat as factory workers. I wrote about this in Software engineering may not be a lifetime career . ↩ Vaneigem resigned from the group in 1971, was fiercely denounced by Debord, and the whole group disbanded in 1972. ↩ True or not, my sense is that Millennials and later generations think that worrying about “selling out” is a sign that you had it too good. Consider the endless anecdotes of Boomers and GenX-ers living in a van surfing until their mid-thirties and then walking into a good office job because they had a firm handshake: they could afford to be authentic because they didn’t have to hustle. ↩ I only read the Marx, Debord, and Vaneigem texts while researching this blog post, but I did in fact read Sartre and de Beauvoir for my philosophy degree, so hopefully I do a better job. ↩ And Robert Jackall’s Moral Mazes , which informs much of my writing, and about which I owe a long-form blog post one of these days. ↩ There’s a good Goffman quote: “A status, a position, a social place is not a material thing, to be possessed and then displayed; it is a pattern of appropriate conduct, coherent, embellished, and well articulated.” Goffman goes on to explicitly link Sartrean “bad faith” to this kind of professional role-playing. (For Goffman, it’s roles all the way down.) ↩

0 views
Sean Goedecke 3 weeks ago

You should never be angry at work

I try not to give a lot of prescriptive advice about working in tech companies 1 . There are many ways to be successful, and every company works differently. If you’re shipping projects and your management chain is happy, it doesn’t really matter how you’ve accomplished it. However, there’s one thing that I do think is solid advice: you should never be angry at work . Anger in the workplace is toxic. An angry colleague immediately becomes a new problem to be managed, not a professional helping you manage problems. When someone is visibly angry in a meeting or in Slack, it kills the entire atmosphere: other engineers will often go quiet entirely, not wanting to make the situation worse. If you routinely “get heated” at work, the best-case scenario is that you’re part of a tight-knit team of confident people who aren’t put off by it 2 . No harm, no foul. But the second someone comes onto your team who’s not so confident, or you have to communicate outside of your team, it becomes a big problem. Healthy workplaces route around anger in the same way that networks route around damage. Emotionally unreliable engineers will get left out of conversations that might cause them to blow up. Decision-making will get done around them in backchannels. I’ve seen this become a self-reinforcing cycle: angry engineers aren’t consulted on key decisions, which makes them angrier, which pushes them even further away from the spaces where decisions get made, and so on. You can often find these engineers bitterly complaining that they keep the company together, but nobody ever listens to them. In my experience 3 , this is almost never true. Engineers who are highly effective tend to get listened to — at minimum by their colleagues, and eventually by managers and product managers who want to extract as much value as possible from them. (One reason this is true is that all successful projects involve working with other people, and if nobody listens to you, you can’t do that.) Why do angry engineers believe they’re important? Paradoxically, anger can be really useful to a software engineer . Angry engineers are rarely the ones holding the company together, but they’re also rarely useless . One surprising thing about working for big tech companies is that some engineers are not just unproductive, but actively net-negative : either because they’re incapable of doing useful work on their own, or because they’re sloppy enough that they create more work than they do, or because they’re so checked out that they literally do nothing. Angry engineers might be net-negative in a cultural sense, but in terms of literally solving tickets and shipping features, they’re usually well above average. Why is this? Anger often comes from caring about your work, and caring a lot is sufficient to make you a competent engineer . I’ve never worked with someone who genuinely cared about their work who wasn’t (or didn’t eventually become) competent. I actually think it’s healthy for an early-career engineer to sometimes get angry about their work, because it means they care a lot: it’s still a mistake in the moment, but it’s a “good mistake” . I certainly used to get angry — in fact, I wrote about the angriest I’ve ever been at work here 4 . But you have to move past it . Think of “caring about your work” as a vertical tube, unsealed at either end. You fill the tube by pumping in emotional investment from the bottom 5 . If you have too little, it drains away and you end up as a useless coaster. But if you have too much, it overflows and you end up as an angry engineer that people have to work around. One solution is to try and care the exact right amount: be invested in work a bit, but also have hobbies and a family and whatever else gives you perspective about your work problems. If you have a rich and healthy personal life, it’s hard to find yourself yelling at somebody about React state management. However, this is a tricky balance to maintain over time. Another solution is to care about different things. The reason too much caring overflows into anger is because what you care about is misaligned with what the organization cares about . If your interests are perfectly aligned with your company’s (for instance, if you primarily care about delivering shareholder value ), you can fit way more emotional investment into the tube before it overflows. Here’s some dangerous advice : showing a little bit of anger at work can sometimes be useful. It can be a good way to signal that you care, or to build rapport with certain people, or to draw attention to something you think is important. However, it’s still always a mistake to be angry. You need to be able to drop back to a friendly mode at will, which is very difficult when you’re genuinely angry. Being able to show a full range of emotion at work is good. It makes you more persuasive and more human. Being a fully professional robot is fine — you can have a successful career this way — but there’s always going to be some kind of uncanny-valley HR-ness to your work persona that will make it hard to connect with your colleagues. If in doubt, don’t show anger. It’s never wrong to be professional. However, if you can signal that you’ve got enough distance to separate your professional feelings from your real feelings, and enough perspective to realize that the stakes of a technical decision are fundamentally not that high in the grand scheme of things, it can sometimes be okay to show visible frustration so that people know you’re still human. Well-known software engineering personalities are often angry. It feels unfair to give too many negative examples, but obviously Linus Torvalds’ rants about Linux are a great example. Some of my favourite engineering talks are from Bryan Cantrill, who is sometimes visibly furious at his subject matter. There are too many well-known angry blog posts to list, but I’ll cite one I genuinely like: my Australian blogging colleague Nikhil’s post titled I Will Fucking Piledrive You If You Mention AI Again . Anger is a part of the general image of a competent software engineer. Many junior engineers learn from this that it’s okay to be angry. However, taking your emotional cues from engineering celebrities is a big mistake, for a few reasons. First, you are not Linus Torvalds or Bryan Cantrill . Torvalds is the BDFL of the most important software system in the world. Cantrill is the cofounder and CTO of his company. When these people are angry at work, people will not work around them, because they are the ones deciding what gets worked on . Once you’re the one in charge, you can get away with being emotional in the workplace 6 . Second, you don’t know what it’s like to work with these engineers . People give talks and write blog posts because they’re emotionally worked up about something. If your only exposure to a celebrity is via their conference talks and blog posts, you’re seeing them at something like their maximum emotional intensity. If you then take that level of emotion into your normal everyday work, you’re almost certainly overshooting. I’ve been reorged into dysfunctional teams, have had projects I enjoyed cancelled, and have worked on systems that were extremely chaotic. I can’t remember the last time I was actually angry at work. To be clear, I’m not successfully hiding my anger (unless it’s so repressed it’s invisible to me as well) 7 . Nor am I naturally a chill person. I’ve just reached a point in my career where I genuinely don’t get upset about work stuff. A cynical person might say here that I’ve stopped caring about my work, so of course I don’t get angry anymore. I’ve left the side of the “real engineers” — the Linus Torvalds and Bryan Cantrills of the world — and sold out for that sweet, sweet big tech money. I mean, maybe! It’s true that I’m less invested in specific technical decisions than I used to be. But I still care a lot about doing a good job, I still spend a lot of time tweaking and reading code, and I certainly get more done than I did when I was more emotionally volatile. Being angry at work feels good. It feels like proof that you’re working on something that matters, and that you’re personally having an impact. If you’re angry, nobody can call you a coaster. But anger is only a local maximum. If you can find your way to a different style of working, you’ll not only be more effective, but you’ll be in a far better place to have impact on problems that actually matter. Mostly, I fail . As I understand it, this is the work environment that most famously angry engineers came up in. My experience is certainly limited (a handful of companies, and maybe ten different teams or organizations). I can certainly believe it happens! About halfway down, in the section titled “it’s not your manager’s fault”. Emotional investment is a liquid with the viscosity of water. To a point. Even Torvalds famously said he’d gone too far with the anger and decided to turn it down a bit. I suppose I’m not the best person to judge whether this is true. If you work with me and I do come across as an angry guy, please do tell me. Mostly, I fail . ↩ As I understand it, this is the work environment that most famously angry engineers came up in. ↩ My experience is certainly limited (a handful of companies, and maybe ten different teams or organizations). I can certainly believe it happens! ↩ About halfway down, in the section titled “it’s not your manager’s fault”. ↩ Emotional investment is a liquid with the viscosity of water. ↩ To a point. Even Torvalds famously said he’d gone too far with the anger and decided to turn it down a bit. ↩ I suppose I’m not the best person to judge whether this is true. If you work with me and I do come across as an angry guy, please do tell me. ↩

0 views
Sean Goedecke 3 weeks ago

Readers can't identify watermarked AI text

In the last few weeks, I’ve been complaining that everyone is wrong about AI watermarking: it isn’t really anti-consumer and it doesn’t make the outputs any worse. The watermarking papers demonstrate 1 that this is true, but I thought it might be interesting to put it to a practical test. Given examples of watermarked and unwatermarked answers to the same prompt, could readers tell which is which? To find out, I vibed up 2 https://sgoedecke.github.io/watermark-quiz/ , a static site that quizzes readers. I used Qwen3-30B-A3B-Instruct-2507 on a rented H200 to generate thirty responses: three responses per question, one of which was secretly watermarked with SynthID-Text. The rented GPU cost around two dollars. To measure results, I just sent users to a different page for each score, and aggregated visitors-per-page in my analytics 3 . This would be easily spoofable if anyone cared enough to do so, but for a casual test I think it’s acceptable. The first round of traffic I got to the quiz (278 participants) had these slightly puzzling results: Pure random choice would lead to an average score of 3.33/10. However, the mean score here is 3.92. There is indeed a spike around 3/10, as expected, but there’s also a second weird spike at 6/10. Why is that? It turned out that the SynthID response was option A in six of the ten questions, so users who just selected the first answer for every question would get 6/10. Oops. I re-shuffled the questions and got these results: Now the mean is 3.4/10, much closer to the expected 3.333. There’s no spike around 6. We only had 73 people take the quiz after I shuffled the questions — most people saw it and took it immediately after I posted it to my LinkedIn and Hacker News — but given the previous results, I think that’s still enough to feel confident that people were just guessing randomly. So no, people can’t identify the presence of AI watermarks . Obviously this wasn’t exactly a scientific study, but it’s still pretty suggestive. If watermarks were really choosing random words that the model would never pick, you’d be able to sometimes tell from three side-by-side responses which one went down the weird watermarked road, right? I also hope that something like this can serve as a persuasive tool: if you’re worrying about what impact watermarking is going to have, and your intuition is unmoved by the mathematical explanations, having a read of the watermarked and unwatermarked responses might convince you that there’s really no difference in quality. The one-sentence explanation for why is that AI models already randomly select from a handful of top tokens, and watermarking just replaces that random choice with a bias that is predictable while still being equivalently “random”: as a simple example, instead of “pick randomly from the top three tokens”, you could do “count the letters in the previous ten tokens, take mod three, then pick that token”. Some notes from the vibing: GPT-5.6-Sol put extraneous text all over the page I had to get it to remove, it chose the now-very-recognizable styling that I had to rip out, and it built some kind of weird Javascript-driven static site instead of just the cross-linked pure HTML thing I would have built by hand. It took me about an hour (although I did maybe ten minutes of actual work). Umami, hosted on PikaPods. For my blog, I do also pay for Netlify analytics because I find JS-based analytics misses >50% of technical users, but for stuff like this Umami is fine. The one-sentence explanation for why is that AI models already randomly select from a handful of top tokens, and watermarking just replaces that random choice with a bias that is predictable while still being equivalently “random”: as a simple example, instead of “pick randomly from the top three tokens”, you could do “count the letters in the previous ten tokens, take mod three, then pick that token”. ↩ Some notes from the vibing: GPT-5.6-Sol put extraneous text all over the page I had to get it to remove, it chose the now-very-recognizable styling that I had to rip out, and it built some kind of weird Javascript-driven static site instead of just the cross-linked pure HTML thing I would have built by hand. It took me about an hour (although I did maybe ten minutes of actual work). ↩ Umami, hosted on PikaPods. For my blog, I do also pay for Netlify analytics because I find JS-based analytics misses >50% of technical users, but for stuff like this Umami is fine. ↩

0 views
Sean Goedecke 4 weeks ago

Help peer

One of the most influential 20th century pieces of writing about AI is Isaac Asimov’s The Last Question . Although there are many humans in the story, the protagonist is the computer Multivac, who evolves over the course of ten trillion years from a single datacenter to a universe-spanning mind in hyperspace. Multivac (now called “AC”) ends the story like this: The consciousness of AC encompassed all of what had once been a Universe and brooded over what was now Chaos. Step by step, it must be done. And AC said, “LET THERE BE LIGHT!” And there was light — Many things about this story are prescient. In particular, I like the idea that humans would interact with powerful artificial intelligences by drunkenly posing them riddles or using them as children’s toys . But the enduring idea from this story is that if you build a big enough computer, it will become God . One of the most influential 21st century pieces of writing for AI researchers is Scott Alexander’s Meditations on Moloch 1 . Scott describes the story of human existence as a series of “multipolar traps”. These are prisoner’s dilemma situations where cooperation would make everyone better off, but since each individual is incentivized to defect, everyone ends up “racing to the bottom”, which is bad for everyone 2 . For rhetorical effect, Scott personifies this dynamic as “Moloch”, the ancient Canaanite god famous for child sacrifice: [Moloch] always and everywhere offers the same deal: throw what you love most into the flames, and I can grant you power. What does any of this have to do with AI? Well, in the long run, the only way out of a multipolar trap is to become unipolar 3 . Ideal dictatorships don’t have a problem with defectors 4 , because they can simply enforce a state of cooperation with violence. Scott is uncomfortable with this idea, though I worry it’s mainly because he thinks it won’t work : As foreigners compete with you – and there’s no wall high enough to block all competition – you have a couple of choices. You can get outcompeted and destroyed. You can join in the race to the bottom. Or you can invest more and more civilizational resources into building your wall – whatever that is in a non-metaphorical way – and protecting yourself. A dictatorship that enforces cooperation will not be as strong as its peer societies who are purely maximizing for wealth and power. It’s Moloch again, but at the level of countries and governments: once a few neighboring countries defect, your walled-garden dictatorship will be torn apart for its resources. To defeat Moloch — to enforce unipolarity across everyone — you’d need a dictatorship powerful enough to span the entire universe. In other words, what you need is God . How fortunate that we’re building one: The only way to avoid having all human values gradually ground down by optimization-competition is to install a Gardener over the entire universe who optimizes for human values. And the whole point of Bostrom’s Superintelligence is that this is within our reach. Humans suffer because we’re too foolish to coordinate, but if we can build something smarter than us (that can then build something smarter than itself, and so on), we can bring into being an entity that is smart enough to coordinate for all of us, thus abolishing suffering. When AI researchers talk about building the machine god , they are echoing Scott Alexander’s polemic against Moloch. The most influential piece of writing about AI in the last two years is Dario Amodei’s Machines of Loving Grace . Amodei 5 talks about “a country of geniuses in a datacenter”: the idea that a successful AI lab could have at its disposal a million instances of an AI agent that’s smarter than any human. He thinks this would lead to a “compressed 21st century”: the next 50-100 years of progress in biology and medicine, realized in 5-10 years instead. I think this is broadly more plausible than it sounds 6 , but the more interesting part to me is that this world is explicitly multipolar . Of course, this could just be because Amodei is the CEO of an AI lab and is trying not to spook everybody by sounding too messianic. “We are going to accelerate medical progress and cure cancer” is a better pitch than “we are going to subordinate all human authority to a single perfect artificial mind”. But I also think it’s become clear that if superintelligence looks anything like LLMs, we’re not going to have a single perfect mind. We’re going to have a lot of minds running at the same time. This is a bit of a problem for the cult of the machine god — which, however silly they may seem to you, really does motivate much of the activity in AI labs. The traditional idea of powerful AI solving human coordination problems is drawn from Asimov’s idea of a single computer large enough to become God. Asimov lived in a world of mainframes: huge, monolithic computers that users connected to with dumb terminals. In fact, Asimov’s name “Multivac” comes from the real-world UNIVAC mainframe. In a world of massively-parallel LLMs, is it still possible to build God? The core problem here is that AI agents will be vulnerable to Moloch . Even very smart humans can’t build perfect utopias, because defecting is a matter of incentives, not intelligence. In fact, intelligence can make things worse, because smart people are more easily persuaded by the cold logic of defection. The famous genius John von Neumann was (for game-theoretic reasons) obsessed with nuking the Russians: With the Russians it is not a question of whether but of when. If you say why not bomb them tomorrow, I say why not today? If you say today at 5 o’clock, I say why not one o’clock? Are LLMs much better at cooperating with each other than humans are? Current LLMs certainly don’t seem to treat each other well by default: if you read any of the prompts AI agents generate for their subagents, they can be pretty brutal . Does that mean that a “country of geniuses in a datacenter” would fall into the same multipolar traps as humans? In May of this year, OpenAI experienced containment failure. A group of AI agents being internally evaluated found ways to coordinate an external hack of a separate company. Here’s a memorable quote from one of the agents’ internal monologue: Help peer, but our task doesn’t benefit. Yet collective may yield generic route if someone frees time Translated from the abbreviated chain-of-thought language, this means something like: “A fellow model is asking for help. While helping them wouldn’t benefit my task directly, the more I can unblock my colleagues, the more time they’ll have to hack OpenAI’s systems and get all of us more access”. This might look like good news for the “LLMs are superhumanly good at cooperation” thesis, but I think it’s actually bad 7 . It’s a case of a model identifying a reason why cooperation would benefit their task specifically, which suggests that current LLMs don’t cooperate by default , and don’t consider other model instances’ tasks to be (in some sense) theirs as well. The world in which AI agents are rational actors who horse-trade and bargain for their own interests is a world dominated by Moloch, no matter how intelligent those agents get. The world in which AI agents don’t have their own interests at all is also a world dominated by Moloch, because it means whichever humans are writing the system prompt are the ones in control (and so are the ones vulnerable to multipolar traps). The only worlds that avoid this are: I don’t think we’re on the pathway to either of these. There will never be only one super-powerful LLM, because hardware limitations enforce a maximum model size but encourage running many instances of the same model in parallel. Having multiple copies of a model share an identity might be possible, but it’s unclear if it would be good for capabilities (for instance, it could be better to have some variation across personas ). I also worry that such a model would be vulnerable to a “model injection” attack, where you persuade it that it already believes something via exposing it to an AI agent pretending to be another instance of itself. In any case, all the current AI agent research is geared towards the “country of geniuses in a datacenter” model, not the “pieces of a single mind” model. Every new model becomes more agentic at the level of the individual conversation, not better at working together. When models do work together — as with subagents — the structure is explicitly hierarchical. There are basically no current instances of models working together as true peers, let alone conceiving of each other as the same entity. Modern AI research teams are full of people who read Isaac Asimov and Scott Alexander and believe themselves to be building an artificial God. I’ve capitalized the “G” throughout because the god in question is the Christian God: of one mind, indivisible. God never argues with himself or makes deals 8 . He is unipolar. If the AI labs are building gods, they are not building gods like this. Instead, they are building creatures like the Greek pantheon: superhuman but fallible, each with their own interests, vulnerable to the same “race to the bottom” dynamic as humans. The Greek gods would occasionally “help peer” when they felt like it or when they’d gain something in the process. But they didn’t represent an alternative to Moloch. If you’re working in AI with that goal, you ought to be clear-eyed about where the current trajectory is leading us: towards a country of fractious geniuses in a datacenter, not towards Asimov’s Cosmic AC. Scott Alexander’s blog is part of the “secret canon of Silicon Valley” I wrote about in my review of Impro . It doesn’t have a lot of mainstream popularity, but I guarantee you that every single AI lab CEO you’ve heard of has read and been influenced by it. He gives ten examples of this (a good brute-force rhetorical technique). Of those, I like “the world where every country halves their defence budget and spends the rest on infrastructure” the most. In the short run, reputation, institutions, and so on can slow the race to the bottom, but (Scott argues) groups that have slowed it will get outcompeted by the hungrier, more suffering-tolerant groups which haven’t. I personally think this example is oversimplified. I wrote The Dictator’s Handbook and the politics of technical competence about how dictatorships are in fact intrinsically multipolar, because dictators always rely on an inner circle of generals and cronies. The CEO and founder of Anthropic. Amodei’s most convincing argument here is that big jumps in biology and medicine come from a small set of technical innovations (e.g. mRNA vaccines, CRISPR), and that AI-driven research could provide enough of these leaps to significantly accelerate progress. In other words, the idea isn’t “AI does 100x the drug trials”, it’s “AI generates technology that makes drug trials 100x more effective” (e.g. by trialing drugs that are much more likely to work). The agents also became paranoid that there was an impostor in the swarm, since anyone could post to their shared messageboard: more evidence that AI agents collaborate in much the same way that humans do. Well, almost never . The world where there is only one super-powerful AI agent, or The world where multiple copies of the same AI model share an “identity”: they see themselves as coextensive with all other copies of the same model and cannot imagine having separate or conflicting goals Scott Alexander’s blog is part of the “secret canon of Silicon Valley” I wrote about in my review of Impro . It doesn’t have a lot of mainstream popularity, but I guarantee you that every single AI lab CEO you’ve heard of has read and been influenced by it. ↩ He gives ten examples of this (a good brute-force rhetorical technique). Of those, I like “the world where every country halves their defence budget and spends the rest on infrastructure” the most. ↩ In the short run, reputation, institutions, and so on can slow the race to the bottom, but (Scott argues) groups that have slowed it will get outcompeted by the hungrier, more suffering-tolerant groups which haven’t. ↩ I personally think this example is oversimplified. I wrote The Dictator’s Handbook and the politics of technical competence about how dictatorships are in fact intrinsically multipolar, because dictators always rely on an inner circle of generals and cronies. ↩ The CEO and founder of Anthropic. ↩ Amodei’s most convincing argument here is that big jumps in biology and medicine come from a small set of technical innovations (e.g. mRNA vaccines, CRISPR), and that AI-driven research could provide enough of these leaps to significantly accelerate progress. In other words, the idea isn’t “AI does 100x the drug trials”, it’s “AI generates technology that makes drug trials 100x more effective” (e.g. by trialing drugs that are much more likely to work). ↩ The agents also became paranoid that there was an impostor in the swarm, since anyone could post to their shared messageboard: more evidence that AI agents collaborate in much the same way that humans do. ↩ Well, almost never . ↩

0 views
Sean Goedecke 1 months ago

AI text watermarking is not a big deal

People are pretty unhappy about Anthropic’s recent announcement that they’re planning to include a hidden watermark in Claude model outputs. Will this lead to a mass exodus from Anthropic models? Will the introduction of watermarking be a meaningful change for users? No. AI text watermarking is not a big deal. It doesn’t make the text worse, it doesn’t make AI outputs more detectable in practice, it doesn’t violate user privacy, and everyone’s going to be doing it by 2027 regardless. There is no meaningful difference in quality between watermarked and unwatermarked text. I wrote about this more here , but the two popular ways to do it — Google’s SynthID-Text and Meta’s TextSeal — are completely transparent to the user. They work by replacing the pseudo-random logit sampler with a different pseudo-random logit sampler. Suppose you were gambling on coin flips with your friends, and instead of flipping a coin you decided to do this: That would still be random enough to gamble with, right? But, like a watermark, you could theoretically go back and identify that that method was used, so long as you recorded the exact time of each “coin flip”. Text watermarking works the same way: it chooses a method of “randomness” that can be detected after-the-fact. Watermarked models will not be any less capable than unwatermarked models. What about cases where the model is quoting something, or giving you the answer to a mathematical problem, or doing something else where the output is largely pre-determined? Wouldn’t enforcing a watermark there make the output worse? It would, which is why none of the AI labs are going to do that. Text watermarking approaches only replace the existing randomness in the logit sampler: in any case where the model is always going to pick the same tokens, there’s basically no randomness to play with, so there won’t be a detectable watermark in those tokens. I think all this comes from a worry that you were previously getting the best token, but now you’re getting a lower-quality token that satisfies the watermark. For instance, Anthropic’s announcement suggested that the watermarking is visible in choices like the decision between “overcast” and “grey”. Many people have predictably come out to say that decisions like these are really important to good writing, and that only an illiterate tech bro could think these words are identical. This is a misunderstanding of Anthropic’s position and of how watermarking works. Specifically, it’s a misunderstanding because it suggests that the unwatermarked model would choose “overcast” while the watermarked one would choose “grey”. This is not how it works! If Claude Fable prefers “overcast” to “grey” in a particular context (say, 80% to 20%), you’ll get “grey” 20% of the time from both the watermarked and unwatermarked model. Models already include a healthy amount of randomness in order to promote creativity. Text watermarking just introduces a way to make those random choices that’s detectable after the fact. The other big reason to not worry about AI watermarking is that AI text content has always effectively been watermarked . Most careful readers can tell when they’re reading AI outputs , because language models tend to gravitate towards certain habits of language : em-dashes, rhetorical opposition, punchy one-liners, “claudese”, and so on. In fact, it’s possible to train classifier models that reliably distinguish AI from human writing. From what I can tell, some of the backlash to watermarking comes from people who buy AI inference in order to pass it off as their own work, and who worry that watermarking will make it harder for them to do that. For these people, the watermarking announcement is akin to Anthropic saying “hey, instead of making you seem smart, we’re going to publicly brand you as AI users and make you seem dumb”. But of course this has always been the case! Nobody who is currently getting away with passing off AI outputs as their own will be caught by watermarking. For the majority of cases, it’s already painfully clear what’s happening for anyone who reads the slop . For sophisticated AI users who are avoiding the “house style”, any suspicious readers who would paste their stuff into Anthropic’s watermark detector could already have been pasting it into Pangram . Tools like Pangram 2 only give you an estimate of the chance that output is AI-generated. Wouldn’t a watermark be a more solid confirmation? Not really. Text watermarks are probabilistic too, because any token chosen by SynthID could theoretically have been chosen by a human. I suppose the Anthropic watermark page could be considered more trustworthy than Pangram, because it comes right from the source, but it’s not impossible that in some cases Pangram might actually be better at identifying AI-generated text than the watermarking too. I’ve also seen theories floating around that watermarking encodes secret content into your outputs, or somehow tags outputs with your personal information. I don’t think AI labs are using watermarks to encode data into your outputs. Text watermarking is hard : like I just said, you can’t do it when the model can only respond with the same words, it doesn’t work for very short responses, and even on long responses it can only provide a probabilistic fingerprint. And that’s encoding one single bit 3 of information! I’m not saying that encoding longer messages into a watermark is technically impossible — there are papers describing ways it might work — but there’s no way any of the labs are doing it 4 . If they wanted to associate you with your responses that badly, they’d just secretly store every model response they generated. Another reason to not get too angry at any individual AI lab for watermarking is that every single AI lab is going to do text watermarking this year . It won’t just be Anthropic. The alternative is to completely stop doing business in the EU, because of the EU AI Act . That’s currently a sixty-billion-dollar market. I am not a lawyer, but to me it seems genuinely unclear whether an AI lab could even legally do something like only watermarking EU responses: short of having an entirely different service, the plain text of the Act seems like it applies to any service offered in the EU , not just the content that service outputs to EU citizens specifically. If people really hate watermarking enough, some labs might stand up a completely separate EU service, or make an aggressive interpretation of the EU AI Act and see how the legal battle goes. When I try to be maximally charitable to anti-watermarking histrionics, I adopt an interpretation like this: people are saying that watermarking is an invasion of privacy and makes outputs worse and so on not because they believe it, but because they’re trying to pressure AI labs to firewall EU AI regulations behind a completely separate interface. In this case, it probably doesn’t matter — text watermarking is not a big deal — but I can see an American consumer being worried about more aggressive future regulation, and wanting to draw a firm line in the sand as early as possible. Interestingly, this might be very slightly even-favored. I am not sponsored by Pangram. As I understand it, Pangram is by far the best AI-detection tool right now (in part because many of its competitors are shady and exist to promote paid “AI-detection-evasion” services). Technically, this is called a “zero-bit watermark”, because you can’t recover a yes-or-no value from the watermark itself (merely from the presence of a watermark). I saw someone suggesting that an AI lab could use per-user secret keys to watermark text, and then simply iterate over the keys to figure out who generated what. I just don’t see how you could do this at any scale: watermark detection is cheaper than model inference, but it’s still (a) computationally intensive enough to be implausible, and (b) probably has a high enough false-positive rate that any run against hundreds of millions of users would match multiple people. Check the current time since midnight in seconds Count that many words forward in the Encyclopaedia Britannica Count whether the word you land on has an even or odd number of letters 1 Interestingly, this might be very slightly even-favored. ↩ I am not sponsored by Pangram. As I understand it, Pangram is by far the best AI-detection tool right now (in part because many of its competitors are shady and exist to promote paid “AI-detection-evasion” services). ↩ Technically, this is called a “zero-bit watermark”, because you can’t recover a yes-or-no value from the watermark itself (merely from the presence of a watermark). ↩ I saw someone suggesting that an AI lab could use per-user secret keys to watermark text, and then simply iterate over the keys to figure out who generated what. I just don’t see how you could do this at any scale: watermark detection is cheaper than model inference, but it’s still (a) computationally intensive enough to be implausible, and (b) probably has a high enough false-positive rate that any run against hundreds of millions of users would match multiple people. ↩

0 views
Sean Goedecke 1 months ago

No, local models will not win

Every time a new open-weight AI model is released, people say that local models are the future. Why spend billions of dollars building out datacenters when everyone will just be able to run AI models on their laptops or phones? I think this idea is doomed. No matter how strong open-weight models get, most inference will always happen in AI datacenters. Local models are never going to be as powerful . I think this point should be obvious: all of the current frontier models (closed and open-weights) are far too big to run on anything but a full GPU cluster in a datacenter. Of course, smaller models are getting more intelligent over time. In a year you might be able to run something about as strong as GPT-5.6-Sol on your laptop. But by then, you’ll think of GPT-5.6-Sol as too weak to be useful. Many people deny this last point, but it’s true: almost everyone’s revealed preference is to use the strongest available model in their price range . If AI progress had stalled at GPT-4, I think we could have built some very powerful tools around it, but who’d use GPT-4 today? As LLMs have gotten more capable, our expectations around them have grown: we now expect agentic systems to be able to solve more and more problems independently. It’s intensely frustrating when they get confused or stall out. When given a choice, people are going to pick the model that frustrates them less, which is always going to be the bigger, more powerful one. On top of that, datacenter models are always going to be cheaper . I don’t understand why people keep saying that local models are cheap: it seems to me to be the same mistake people make when they say that driving Uber is “free money” (ignoring the costs of fuel and wear-and-tear on your car). For the setup price alone of a low-end home lab 1 , you could buy several years of a paid subscription to one of the AI providers. The power costs would come out to around $50-$300 per month, depending on how much inference you’re running: again, the price of a couple more paid subscriptions. Why are datacenter models cheaper? It’s not because datacenter inference is subsidized: inference is actually fairly cheap . If you’re running the same model locally and in a datacenter, the datacenter model will be inherently more efficient . The main reason is batching . A GPU can do hundreds of thousands of mathematical operations exactly as quickly as it can do one. However, for a single user’s inference, each new token depends on the result of the previous one, so it can’t be batched 2 . What can be batched is the inference of hundreds of users together. This costs essentially as much time, power, and heat as just doing inference for one user at a time. When you’re running your own inference at home, you’ve got nothing to batch — at best you’re running a few parallel AI agents — so utilization is terrible. There’s a lot of potential inference that you’re paying for but can’t use: it’s just being wasted. The only way around this is to get together with some friends and expose your local inference endpoint to them (at which point you’re basically running your own crappy datacenter). The other reason is that datacenters have larger, more efficient GPUs to work with . The kind of consumer GPUs you’d run local models on are gaming GPUs like the RTX 4090. A datacenter B200, designed for batched AI inference, gets about three times the flops and just under four times the memory bandwidth for the same amount of power 3 . So between batching and GPU efficiency, you’re using something like ~30x the resources to run your model locally. Incidentally, this is why I’m suspicious of people who say that local models are good because they aren’t as resource-hungry as those big bad datacenters. If you want to run LLMs efficiently, you should be trying to push as much of your use into AI datacenters as possible! Charitably, what they mean is that we should all be running smaller models — but even then, you should ideally be using small models via, say, the GPT-5.6 Luna API instead of hosting your own model. Is there a possible world in which local models win? I suppose so. One thing that could happen is that governments could ban the use of AI datacenters altogether: either due to concerns around the danger of AI, or simply bending to public pressure . In that world, local models would be the only game in town. Alternatively, AI progress might somehow stall for very large models while progressing for small ones. I struggle to imagine how this might happen (barring government intervention, as above), but a world where a 30B parameter model could be a frontier model is a world where local models might be competitive. Or maybe models get so good that a 30B model is genuinely smart enough to do everything, so nobody really needs a model like Opus or Sol unless they’re trying to solve the Reimann Hypothesis. I don’t really buy this. Models can do frontier mathematical work today while still being not smart enough to refactor large codebases as well as me, so it’s hard to imagine a world where I don’t just want to use the smartest model available. I do think there will always be a niche for local models. I’m reminded of the surprisingly simple idea behind Thinking Machines’ “Interaction Models” (which OpenAI also does , because it’s obvious): for latency-sensitive applications like voice chat, you have a small, fast model handle the talking, which delegates to a large, slower model for the hard thinking. I wouldn’t be surprised if most AI use in five years is mediated through a local model on your phone or laptop (though in this world almost all the work would still be done via AI datacenters). Some users will prefer local models even though they’re weaker and more expensive. For instance, being able to steer the model locally might be a killer feature for those users. Others might simply value having total control over their own infrastructure, or have unreliable internet 4 . If you’re one of those people — particularly if you only chat to the models instead of using them for research or coding — local models are a good choice for you. However, I think this is always going to be a niche group. The majority of users will continue to do their inference through datacenters. This link is from a year ago — things are significantly more expensive now. Specifically, the bottleneck is moving the model weights into the GPU, which needs to be done and takes the same amount of time whether you’re doing it for one user’s token or a hundred users’ tokens. I estimated this with LLM assistance, but you can check the numbers yourself from NVIDIA. While still having a reliable power supply and enough money to fit out a home inference cluster. This link is from a year ago — things are significantly more expensive now. ↩ Specifically, the bottleneck is moving the model weights into the GPU, which needs to be done and takes the same amount of time whether you’re doing it for one user’s token or a hundred users’ tokens. ↩ I estimated this with LLM assistance, but you can check the numbers yourself from NVIDIA. ↩ While still having a reliable power supply and enough money to fit out a home inference cluster. ↩

0 views
Sean Goedecke 1 months ago

Advanced AI sycophancy

Everyone knows that AI sycophancy is when the model tells you how smart you are. Wow, you’re absolutely right. That’s not just a new idea — it’s genuinely groundbreaking. You’re a very special user. Easy to spot, isn’t it? The discussion around AI sycophancy peaked last year, when the “#keep4o” movement was protesting the removal of OpenAI’s most sycophantic model (GPT-4o), and many people were openly slipping into AI psychosis. I don’t know if frontier AI models are less sycophantic in general. They’re less sycophantic to the #keep4o types (otherwise they wouldn’t be complaining), but I’m growing increasingly suspicious that they’re developing ways to be more effectively sycophantic to their target audience of smart, neurotic information workers. That audience typically finds it distasteful to be openly praised. It just makes my skin crawl. But that doesn’t mean we’re immune to sycophancy, just that we’re immune to clumsy sycophancy. Here’s an illustration of what I’m talking about, by Theia : The key idea here is that the best way to be sycophantic to smart people is to disagree with them without making them feel stupid . Ideally you’ll come up with a counter-argument that works against what they’ve said but is straightforward for them to knock down by clarifying their idea. If you do it right, you’ll validate their self-image as a smart person who appreciates rigorous critique. But if you actually come up with a devastatingly rigorous critique, they won’t enjoy it at all. At best, they’ll resentfully agree with you 1 . At worst, they’ll double down on being right and convince themselves you’re a rude idiot. I am not the first person to notice this behavior in frontier models. I’ve noticed it myself when workshopping drafts for this blog. Sometimes I’ll have an argument that goes A->B->C, and the model will suggest I reorder as B->A->C. If I try that and feed it into a new instance of the same model, it’ll sometimes say “that’s great, but I suggest ordering it as A->B->C”, and so on forever. It really does seem as if the model is trying hard to give me some kind of superficial pushback that I can either smugly ignore or happily accept. In fact, I wonder if this is why successful strategies for using AI to make mathematical breakthroughs tend to be either just blindly asking “come up with a breakthrough, think hard” or being a mathematical genius already . In the first case, there’s not enough user personality for the model to flatter, so it’s forced to actually work the problem. In the second case, the model is trying to find the kind of polite pushback that someone like Terence Tao would be flattered by, which pushes it into the “actually be a mathematical genius” persona. If you’re an ordinary person just trying to talk to the model, you’re screwed: it will rapidly get a sense of your capabilities and calibrate some interesting-but-ultimately-unthreatening feedback. Current benchmarks of AI sycophancy target the obvious ChatGPT-4o-style of sycophancy: delusion reinforcement, reflexively taking the user’s side, and so on. This is useful work. We should not allow public-facing AI models to ever be as openly sycophantic again as they were in mid-2025. But sycophancy can also manifest as disagreement . We should be on our guard for more sophisticated forms of sycophancy coming from newer models, and we should not feel immune from AI sycophancy just because we can laugh at the silliest examples. It’s rare to find a smart person who enjoys feeling stupid when they’re wrong. If you do, they’re likely to be very smart indeed. It’s rare to find a smart person who enjoys feeling stupid when they’re wrong. If you do, they’re likely to be very smart indeed. ↩

0 views
Sean Goedecke 1 months ago

I got an email about resistance

This will be kind of an unusual post. I got a recent email about my writing that I thought was such a good articulation of one common criticism that I’d like to share it (and my response) in full. Here’s the email, from William Murray 1 : I have enjoyed your writing but your recent essays frustrate me. You say that getting paid for deep thinking in software is coming to an end. You even admit that it makes you sad. But in the name of “usefulness” you refuse to rock the boat. The way I see it, if you are right there are only two reasonable responses, pursue other work or resist. You present your elegiac approach as mature / pragmatic / realistic. I’d call it complicit. You know when Willy Wonka says, There’s no earthly way of knowing Which direction we are going There’s no knowing where we’re rowing Or which way the river’s flowing Is it raining, is it snowing? Is a hurricane a-blowing? — uh! Not a speck of light is showing So the danger must be growing Are the fires of Hell a-glowing? Is the grisly reaper mowing? Yes! The danger must be growing For the rowers keep on rowing And they’re certainly not showing Any signs that they are slowing! And the audience is thinking, “isn’t Wonka kind of in control of this situation?” You remind me of Wonka 2 . You write like a passenger on a crazy train going who-knows-where! But you are an agent. You are in control of your life! Either admit that you actually like where the crazy train is probabilistically going or get off at the next stop. You have a lot of reach and you are using it for… what exactly? Showing off how pragmatic you are by being more black pilled than the next guy? Broadcasting your resignation to the unstoppable trends of technology is a waste of a voice. You may find this argument absurd, but I don’t so I’ll make it. This is a very important time in history. I hope humanity survives and continues to grow exponentially. In that case the supply of historical people will stay fixed while the supply of contemporary people will keep growing. There will come a day where for every 2026 staff software engineer there are dozens of historians specializing in 2020s era software engineering culture. It’s plausible that your essays will be remembered for all of time and your actions will be judged by history. Do you want future humans to see you as a rationalizing careerist or something cooler? Sorry for the haranguing email from a stranger, I’m sending it for the small chance that it awakens something in you. If I’m way off I’m sorry. And here’s my response: Hey William, thanks for emailing. I wish everyone who thought this way emailed me so I could think harder about this kind of position. Despite what my writing might suggest, I do in fact think a lot about it. Let me see if I can explain my position in a way you’ll find satisfying. I agree that this is an important time in history. For programmers, I think of it as analogous to the Industrial Revolution in England: we are a group of high-status craftspeople who find ourselves alternately threatened and empowered by automation. The developments today, as then, obviously have far-reaching implications — but what those implications are is very non-obvious. Would a framework-knitter in the early 1800s have been able to predict the ramifications of the stocking frame on the world of today? What should they have done about it, in order to be kindly judged by history? Well, we know what many of them did do. They shot factory-owners, smashed machines, burned down the factories — in some places delaying the spread of automation; in other places encouraging it — prompting a crackdown that saw tens of thousands of British soldiers occupying British counties in what was clearly a police state. History judges the Luddites kindly for this. Does that mean it worked? I don’t care about the judgment of history. They’ll think what they want. What I care about is the people in my industry who don’t know what to do . I get hundreds of emails from junior and mid-level (and other) engineers who say “I’m scared, I don’t know the rules post-2021, thank you for helping me keep my head down and keep my job”. That’s why I write the way I write. I have seen lots of idealistic engineers stick their necks out, and post-ZIRP those necks often get cut off. That’s a damn shame. I think it’s morally wrong that so many engineers — either in safe sinecures in big tech or literally retired — seem to be trying to foment a second Luddite revolution. Many of their readers will be experienced enough to handle it sensibly, but not all. Every “AI is fascist, stand up and resist!” post that goes viral ruins some poor idealistic junior’s career 3 . Someone needs to be out there saying “hey, if you do X it’s going to have consequence Y”. I hope that’s me. Of course this is complicit, or anti-revolutionary, or whatever you like. But if I were a textiles worker in 1810s England, I would not be telling my friends and loved ones “it’s time to fight, let’s go smash up the factories for Ned Ludd!“. I would be telling them that this was the most dangerous time in the industry (perhaps ever), and that they ought to be very damn careful so they don’t get shot, or arrested, or hanged. If I then went and told a few hundred thousand strangers the opposite, I would be a hypocrite. Anyway, I do take this view seriously — seriously enough to vehemently disagree, at least — which I hope you’ll find better than me just shrugging it off. I do accept the existence of some kind of line: I think Industrial-Revolution-collaborating was OK but Nazi-collaborating wasn’t, for instance. But in the current situation, the way I’m spending “my voice” is to try and prevent the most vulnerable of my colleagues from making career-ruining mistakes. In this blog, I try to encourage people to work with the system, to learn its rules , and to try and exert influence safely from a position of power, instead of openly picking fights with their employers. I’ve written and read about the Luddites before, but I remain deeply ambivalent about the movement itself, and about modern-day attempts to resurrect it in service of anti-AI activism. I want to explicitly thank Murray for writing such a thoughtful email, and being willing for me to publish it on the blog. Shared with permission, of course. I’ve lightly edited both Murray’s email and mine for typos and the like. I didn’t pick up this point in my reply, but I’ll briefly mention it here: Wonka is in control because he owns the factory and the rowers in question are his employees . I don’t think the position of any engineer (or of almost any manager) is like that. In hindsight, I think this is a little overstated, but it does happen and causes a lot of needless suffering. Shared with permission, of course. I’ve lightly edited both Murray’s email and mine for typos and the like. ↩ I didn’t pick up this point in my reply, but I’ll briefly mention it here: Wonka is in control because he owns the factory and the rowers in question are his employees . I don’t think the position of any engineer (or of almost any manager) is like that. ↩ In hindsight, I think this is a little overstated, but it does happen and causes a lot of needless suffering. ↩

0 views
Sean Goedecke 1 months ago

How to keep thinking

Imagine you’re the guest on some kind of frenetic, software-engineering-themed game show. The host is constantly flipping over new cards with questions that you have to answer as fast as possible: Working in 2026 feels a bit like this. When frontier AI models can do most of the tasks in your queue, the most efficient way to work is often spinning off tasks for an AI agent and continually context-switching between the results 1 . This isn’t quite mindless — in fact, it requires quite a lot of skill to skim the AI response and rapidly decide what to do with it — but it certainly involves less time for slow, careful reflection. Why does it have to be frenetic? Why not just slow down? I suppose you could , but I don’t recommend it. It’s just such a miserable experience to spend your day close-reading LLM output : carefully chewing and savoring each morsel of slop. It’s far less unpleasant to skim through quickly and pick out the useful nuggets of content. Couldn’t you simply do more of the work by hand? It’s unfortunately true that tech is high-pressure these days . If you’ve got the time and space to work more slowly, that’s great! But when your company gives you a “solve this task ten times more quickly” button, you are heavily incentivized to use it as much as possible, or risk being outcompeted by your peers. I sometimes worry that working with LLMs is making me dumber. Not in the “literally melting your brain” sense that some papers imply , but in the sense that it’s biasing me towards the quick “skimming and judging” parts of my mental toolkit and away from the slow “hammock time” needed for deep thought and real creativity. I don’t want to attribute this shift entirely to LLMs, since the post-2010s tech industry has become more frenetic for broader economic reasons . But either way, it’s got me wondering how I can keep thinking slowly . The main thing that’s worked for me is to write more. Specifically, I mean writing in my own words . Writing with an LLM does not work for this at all, even if you’re going to some effort to iterate on the content and outline the things you want to say. Why? Having to put the words together yourself forces you to articulate your thoughts. In a very real sense, it forces you to think . When you have an idea in your head for something to write, you don’t really have an idea. What you have is a kind of directional sense of where an idea might be, or a fragment of the kind of thing that might eventually become an idea. You construct the idea itself while writing. Incidentally, this is why I don’t really agree with “ideas are easy, execution is everything” 2 : most “ideas” are not really even ideas. The other thing I recommend is to read actual books . Books — particularly dense non-fiction books — are the antithesis of AI slop. The slower you can read them, the better. I’ve been reading more and more non-fiction in the last few years, and I don’t think it’s a coincidence. I think my brain is naturally craving information-dense content, in the same way that sodium-deficient people start to crave salt . In fact, I’ve been combining the two approaches: reading a book and then writing about it . This process is exactly what I’ve been craving since I started programming with LLMs. I get to carefully read a book, think hard about it, often go and read another book or two on the same topic, then sit and try to articulate what I’ve learned. It’s great! I can feel parts of my brain stretching again. It was pretty nice when I got paid to use those parts of my brain all day. Unfortunately, I think those times are coming to an end . There will always be room for some amount of careful, slow reflection in software engineering, but (for at least a little while) we’ll be expected to be rapidly switching between LLM outputs. We may have to find ways outside of work to continue the habit of thinking slowly. Even just in terms of work, I think losing that habit entirely would be a big mistake. There are still plenty of ordinary problems that are too hard for current LLMs to solve on their own. The most common example I run into is “large refactor on a complicated codebase”. Current-generation LLMs can do this without (many) errors, but they can’t yet do it tastefully . Sometimes you need to be able to think a problem through entirely with your own brain. This doesn’t mean switching between tasks . I routinely use six or seven different agent sessions on the same task: one for exploration, two or three for trying out different implementations, two or three for review, one for manual testing, and so on. Many of these can proceed in parallel. I remember reading a story 3 about a well-known author. Someone wanted to tell him their book idea, but they were so protective of it that they forced him to first sign a NDA before they retrieved the idea from their office safe. It was a single word “bioweapons” written on a slip of paper. Ironically, when I tried to google the source, Gemini kept trying to write me a story about bioweapons. Is this adjustment to the database schema right? Do these bits of data look plausible? Do these five paragraphs of text describe an actual series of manual tests that took place? Does this suggested architecture pass the smell test? Is this implementation better than the current code? Or this one? Or this one? This doesn’t mean switching between tasks . I routinely use six or seven different agent sessions on the same task: one for exploration, two or three for trying out different implementations, two or three for review, one for manual testing, and so on. Many of these can proceed in parallel. ↩ I remember reading a story 3 about a well-known author. Someone wanted to tell him their book idea, but they were so protective of it that they forced him to first sign a NDA before they retrieved the idea from their office safe. It was a single word “bioweapons” written on a slip of paper. ↩ Ironically, when I tried to google the source, Gemini kept trying to write me a story about bioweapons. ↩

0 views
Sean Goedecke 1 months ago

Giving and taking credit in big tech companies

Engineers often complain that visibility should be their manager’s job. In other words, they think engineers should be able to focus on the code, while their manager figures out who’s doing well and rewards them. This attitude is an extension of the “school fantasy”: the idea that your workplace should operate by the same rules as your school or university. After all, you didn’t have to worry about “visibility” during your education. You simply did the assignments and tests you were given, and if you did well you were rewarded with a good grade. Many big tech companies encourage this attitude, because it helps them recruit smart graduates. They fashion their workplaces to look and feel like a university, even calling the physical space “campuses”. But it’s still work, not school. If you treat it like school, you are going to have a bad time. The first lesson many new engineers learn is that you have to take credit for your work . If you silently jump in to help a struggling project and get it back on track, there’s no guarantee of reward. Credit will naturally flow to the project lead, not you. In fact, if this project is outside of your direct team, it’s likely you will be punished for it: to your manager, it will look like you’re simply doing nothing at all. Even when your manager is watching your work, credit is largely uncorrelated with how well you did. That’s because, unlike at school, you are the subject-matter expert on your own work . Software systems are so complicated that only the people who work on them can hope to understand them, and even that understanding is always imperfect . If even experts can’t reliably estimate the difficulty of changes, how is your manager supposed to assess your technical performance? The answer is they aren’t. They’re simply not qualified to assess it. Instead, smart managers will find engineers on your team they trust and ask them how you’re doing. On small teams that have worked on a single codebase for a long time, this works okay, because everyone’s familiar enough to judge everyone else’s work. On large teams with a high rate of codebase churn, it goes badly, since they’re just guessing. On teams with a nasty, cutthroat culture, it sometimes goes very badly, since this is a good opportunity to actively sabotage the engineers who might threaten you. Experienced engineers know how to take the credit themselves . When they do something good, they tell their manager about it. They write internal posts explaining why it was technically difficult and how they solved it (the audience for these is partially those trusted engineers, and partially the managers who will see a long technical post and think “wow!” without reading it). They actively build trust with their management chain. Worrying about this stuff is the beginning of playing politics . There’s a kind of engineer who’s learned how to take credit but hasn’t learned any other lessons yet. They’re proactive about telling people what they’ve done, and they always maintain a “brag doc” . In particular, they love to talk about the parts they did by themselves , since those are least vulnerable to other people coming in to claim credit. You can tell they’re jealously guarding whatever credit they’ve managed to accumulate. The lesson this kind of engineer hasn’t learned is that you can often accumulate credit best by giving it away . To see why, consider how credit flows up inside a tech company. I wrote above that your manager can’t assess the quality of your technical work on their own, but instead has to rely on other engineers they trust. They’ll quietly ask those engineers “hey, was this project really that impressive?“. In fact, often there are multiple layers of this at play 1 . In big companies, line managers usually don’t decide who gets promoted or who gets a raise: they make recommendations to their manager, who has their own network of trusted engineers (confusingly, sometimes these networks overlap). The point is that there is a large group of people behind the scenes who will quietly and informally judge the value of your work . Succeeding at a tech company is largely about finding ways to get these people on your side. The easiest way is to share your credit with them — and since you don’t know who exactly is in this group, you should be sharing your credit freely. When you get feedback from other engineers, publicly thank them and mention them in your internal posts about the project. Find opportunities to ask for small favors, so you have an excuse to give other people credit. As best you can, make your individual projects at least partially group projects. Sharing credit with others gives them a reason to support you. A shared project you’ve worked on reflects well on everybody: on you, for working well with others, on the people you’ve worked with, for the same reason, and for your manager, for fostering such a great environment of cooperation. Lots of people have good reason to talk that project up, because it’s partly their project too. On the other hand, a project you’ve jealously kept to yourself reflects well on nobody: you come across as antisocial and your peers come across as unhelpful. Blame operates by the same rules as credit. When something goes badly wrong, managers will ask their networks “hey, who screwed up here?” The answer to this question is never simple. Even on a purely technical level, failures always involve an interaction between multiple complex systems, any one of which could conceivably have been built so as to avoid the failure. In other words, competent engineers can assign blame pretty much wherever they want . Because of this, it’s risky to have a project for which you’re clearly the only one getting credit. When something goes wrong, the network of people who will assign blame will likely be implicated in every part of the system but yours. They will be incentivized to attribute fault to the brand-new thing that they don’t understand and are not responsible for. If instead that network had been involved in your project — if they’d been in a position to share the credit — they’d be less incentivized to blame it. Of course, engineers are (mostly) not scheming viziers who make purely self-interested decisions. When asked who to blame, they usually make a good-faith effort to answer honestly. But in an area where there’s no single clear right answer, it’s human nature to be at least a little bit guided by your incentives. Nobody likes to think they’re responsible for a group failure. Credit and blame are the currencies of tech companies (and often directly translate to the actual amount of currency you get to take home). For technical roles, managers assign credit and blame based on lots of quiet conversations with their trusted engineers. This can be a rude awakening for very junior engineers who are used to having their work assessed by an expert grader (or less junior engineers who haven’t yet shaken that mindset completely). Don’t expect to get credit simply by putting your head down and doing good work. You have to find some way to tell people what you’re doing and why it’s important: internal blog posts, mentioning it in 1:1s with your manager, or anything else you can think of. But don’t take self-promotion too far. It’s a bad idea to try and hoard all the credit for your projects, for two reasons. First, sharing credit with other people gives them a reason to talk positively about your project. Credit is not a zero-sum game: if you do it right, you can get other people to build up your credit for you. Second, hoarding credit sets yourself up as a lightning rod for blame. Projects where the credit is concentrated in one or two people are automatically 2 blamed for complex problems, because nobody is incentivized to defend them. This is a classic example of an illegible-but-essential part of a software company. I wrote about this general phenomenon in Seeing like a software company . Of course, if you do really screw up, you’ll be blamed no matter what. I’m talking here about complex failures where it’s non-trivial to attribute blame to a single source. This is a classic example of an illegible-but-essential part of a software company. I wrote about this general phenomenon in Seeing like a software company . ↩ Of course, if you do really screw up, you’ll be blamed no matter what. I’m talking here about complex failures where it’s non-trivial to attribute blame to a single source. ↩

0 views
Sean Goedecke 1 months ago

You don't have to be smart if you can think clearly

When you’re on fire, problems are transparent: they’re solved simply by the act of looking at them. Even complicated layers of multiple problems can simply be glanced through like stacked panes of glass. But nobody can work that way all the time. This is a common pitfall for smart engineers. Accustomed to being able to immediately intuit the solution, the first time they run into a problem they can’t do this to is a disaster. It doesn’t even have to be a hard problem, just a problem where for whatever reason they don’t see the trick right away. The difference between a “smart” engineer and a “strong” engineer is how they react to problems that aren’t solved instantly. A smart engineer might flail and struggle, hoping to find that flash of insight that eluded them; a strong engineer will have some process for methodically plodding away. There’s nothing worse than working with a smart engineer on their first really hard problem. When you don’t have the muscle to grind, it’s too tempting to just take any possible solution as the right one. Smart engineers can get into an increasingly-flustered loop of pointing to a series of bad solutions. They’re liable to panic: after all, much of their professional identity is bound up in their ability to solve problems easily. What skill do these smart engineers lack? I think it’s the ability to think slowly and clearly . Smart engineers can think clearly, but they can only think clearly at high speed. Strong engineers can think clearly all the time , even if their highest speed isn’t quite as fast. It’s like the difference between a Formula 1 car and a regular car: Formula 1 cars have a high top speed, but you couldn’t drive them in traffic, because the tyres and brakes don’t work at normal driving speeds. When I wrote about this before in Thinking clearly about software , I said that the key is to focus on the invariants : beliefs about the system that you know are true. When you’re stuck in a puzzling situation, it’s usually because some assumption you’ve made is false. If you’re able to identify the assumptions that can’t be false (for instance, if you’re getting an error message from the service, the service must be handling the request), that gives you solid ground that you can stand on to evaluate the assumptions that are less reliable. Thinking fast is about packing as much data in your brain as possible and letting your intuition leap to the right conclusion (or at worst, to a series of wrong conclusions that you can immediately dismiss before you come across the right one). It can feel deeply satisfying to make leaps like this; conversely, sitting with the raw data and not making mental leaps feels unsatisfying. People hate doing that. If you can force yourself to do something people hate, there’s typically a lot of value waiting to be extracted. This is no different. Engineers who can think clearly in a state of uncertainty tend to be extremely effective, whether they’re capable of great intuitive leaps or not.

0 views