Apple's earnings suggest that the company's hardware business is doing better than ever, even in the face of memory challenges. Does the software story help or hurt?
I recently blogged about how the curl project switched to doing all handling of vulnerabilities at severity LOW and MEDIUM in the public . Let me now describe how you can follow along this process and help out. Add your opinions, comments and assessments to help the curl project stay honest, do even better and manage the load. Vulnerability reports are now submitted directly in curl’s issue tracker on GitHub . Judging by the initial set received, the persons submitting them do not particularly highlight the security angle in the title but it is instead implied or specified in the issue description. I hope people react if we somehow miss it. We use a set of tags to manage vulnerability reports on GitHub. They are all prefixed by vuln: . Every submitted issue that has a claimed or suspected security angle – a possible vulnerability – first gets the tag set (by one of the curl maintainers). This tag marks the issue as one that we should evaluate, assess and determine first if it is a vulnerability, and if it is, what severity. Anyone is free to leave their comment and thinking to help us come to a conclusion. This tag is removed again when an assessment has been done. This issue was assessed in regards to its vulnerability risk and we deem that it has none; or it is so low that it is considered to be less than severity LOW. This issue is treated as just a bug . If someone objects provides a reasonable and convincing argument, we can reassess. This issue is a confirmed vulnerability at severity LOW. We work on a fix for this that we merge as soon as it works and then we close this issue. If someone objects to the severity and provides a reasonable and convincing argument, we can reassess. This issue is a confirmed vulnerability at severity MEDIUM. We work on a fix for this that we merge as soon as it works and then we close this issue. If someone objects to the severity and provides a reasonable and convincing argument, we can reassess. Confirmed vulnerabilities get a CVE and a dedicated security advisory that we work on separately from the issue and its fix in the curl-www repository . The CVE and its advisory get published in sync with the pending curl release. They are left as open in-progress pull-requests until then, using vuln:LOW or vuln:MEDIUM tags. Each advisory has all the information about a single curl vulnerability. A detailed description, all the info, CWE, CVE, affected version ranges, if it is a C mistake, in which curl “area” it happens, a link to the original report, etc. The idea being that this single markdown file has all necessary information anyone might want and need about this specific vulnerability. The work on these vulnerabilities is immediately public and the second we register and assign a CVE Id (the actual numerical identifier) to a problem, you can learn about its existence. We do however not publish the issue and all relevant metadata to the CVE database until the publication date, which is the same date as the curl release date when we ship the first release that contains the fix for the vulnerability. So information is public in the curl repositories early on, but it is not published until perhaps several weeks later, and it is at that later moment that it gets picked up by other tools and ecosystems and then gets widely known.
When I was getting my feet wet with self-hosting, I learned about shucking, which involves taking an external hard drive, prying its plastic case off and using it as a normal SATA hard drive. It’s not possible with every external hard drive, but with most ones it works well, and the WD My Book and Easystore drives are a prime example of this. Back then, I remember quite a few people being skeptical about these hard drives, suggesting that they could be drives that were not good enough to be server-grade WD Red drives, so they were cheaply thrown into external enclosures and sold to customers. While I can’t disprove that, I can now provide a few data points that can help someone judge if similar approaches are worth trying. I got mine quite soon after I had bought three WD Red 4TB hard drives. WD Red drives were pricey, but then I had one of them die after 6 months and although warranty service replaced it, it made me think about the alternative option. The math supporting this suggested that buying cheaper drives can make more sense even if they end up being more prone to failures, because with the savings you can simply buy an additional one and keep it as a spare that’s immediately available to you, which also improves the availability of your storage pool. These two drives have seen various use cases in their early life, jumping from one setup to another, but for most of their lives, they have run in a server room and are serving as an offsite backup endpoint. The disks have seen some abuse, you can tell that by the lifetime maximum temperature hitting 67°C, but in my defense, that was when the disks were running in their original enclosure. There was a brief moment where one of the disks started throwing some ZFS errors, but that could also have been an issue with the server that these drives are running in and its SATA connection. The power-on hours on the disks are the following (at the time of writing): Most people expect hard drives to last about 5 years, after which it becomes a lottery. Some have had drives successfully run for 10+ years, which is something I’d absolutely love to see with these two. Here are full outputs on both drives in case you want to dig into specifics: After running these shucked WD drives and these white label Seagate ones , I’m convinced that for most use cases, these drives are just as good as buying the branded, more expensive ones. Even with new ones it’s unwise to expect them to be perfect, they’re tiny mechanical miracles that somehow store and read data, after all, and you’d not want to use a large pool of drives without a filesystem like BTRFS or ZFS anyway if you care about your data integrity. Now I need to go find some wood to knock on because I would love to not buy any replacement hard drives in this economy. 55349 hours (6.4 years) 46513 hours (5.4 years) WDC WD80EZAZ-11TDBA0 WDC WD80EMAZ-00WJTA0
Audience capture is the part that scares me the most about writing on the internet. I have the soul of an engineer 1 : when I see a metric like engagement, I want to improve it. I like to make the number go up. In fact, the whole environment of online writing is tailor-made to help you make engagement numbers go up. It’s trivial to find out how many people are reading my posts, and from what sources. Hundreds of link aggregators all over the internet have algorithms that will help my most engagement-friendly posts get views, and the comments are helpfully sorted by upvotes so I can read the “best” 2 pieces of feedback straight away. Many platforms are set up to financially reward you for engagement, whether that’s by enabling advertisement or by straight up paying you out . Leaning into this environment is a huge mistake. Views and engagement are valuable to me, but not directly. They are proxy metrics for the things I actually care about: It does not benefit me at all to have ten thousand people click through to one of my posts, scan the first sentence, then close the tab. It benefits me only a little 4 to have a hundred people read the title of my post on Hacker News, decide not to click through, and leave their comments in the thread. The set of people who do those things — we could call them “clickers”, as opposed to “readers” — does not really overlap with the set of people who will actually read my posts and think about them. And yet in terms of metrics and engagement, they are indistinguishable. As with any proxy metric, optimizing it too hard can cause it to diverge from the actual value it’s targeting. This leads to the scariest form of audience capture : being captured by an audience you don’t really even want. Suppose I get a lot of views when I write posts about how stupid it is to use microservices 5 , because it’s a hot topic. That gives me a reason to write more posts about microservices. The numbers keep climbing. However, I suspect the set of people who read my posts thoroughly — the audience I actually care about — don’t really want to read a bunch of the same screeds about that, so they drift away 6 . The people clicking don’t care because they’re not reading the posts anyway. So I’m actually failing, not succeeding: I’m replacing my audience of careful thinkers with a fake audience of clickers and sneerers. This is not just a post about writing on the internet. In 2025, I wrote Trying to impress people you don’t respect about how succeeding as an engineer involves, well, impressing people who you may not necessarily respect. The core idea here is knowing what game you’re playing . Some games are proxy games, like getting views on posts or having great Jira ticket metrics. Some games are real games, like gaining the respect of people who you respect, or doing technical work you can be proud of, or trying to convince people of things that you believe are true. The difference is in whether you care about the goal of the game, not in whether that goal is objectively valuable. If you really want to get promoted , then that’s a real game; if you just want to do solid technical work, getting promoted might be a proxy game that you play in order to get into a position to do that. But even though you don’t really care about it, it’s easy and tempting to win the proxy game at the expense of the real game: for instance, to crush Jira tickets because they’re in front of you instead of doing what will actually get you promoted, or to get views on the internet by playing into clickbait. Making tradeoffs like this is sometimes fine. For instance, you might decide to impress people you don’t like so you can afford to send your kids to a good school. Selling out a little bit is good and healthy 7 . But do it with eyes open! The absolute worst thing you can do is to sell out by habit , in situations where it doesn’t even buy you anything you want. There are a lot of posts that I really enjoyed writing that did not get much attention: typically posts where I get a bit more into the philosophy or history, or posts that are dry technical explainers . Some of my most popular posts are laments about a changing industry that are really just popular because they give people an outlet to share their worries too, not because of anything I wrote. I think it’d be a big mistake to lean into writing more posts like that and fewer posts like this. For one, if I want to just optimize for engagement, why write blogs at all? Why not post ragebait and pornography? But the main reason is that I’m writing for the 5% of readers that will read my posts, think about them, talk to other people about the ideas, email me with their comments, and so on. Raw view counts help expose my writing to more of those people, but they don’t give me much signal on whether I’m appealing to them or not. For that I have to just rely on my taste, or on the thin signal I get back from the readers who do email and message me with thoughtful responses. In my own terminology, an “impure engineer” . Or at least the cleverest, or the most cutting. Too honest? I could be even more honest and add “encouraging people to potentially hire me if I’m ever looking”, too. Boosting engagement on Hacker News means the post is more visible for people who might actually read it. Imagine it’s 2015 again. Go on, you know you want to. Sunil Pai has a great post about this dynamic. To quote him: “you don’t get a notification that says, “three people respected you slightly less after reading that.” you don’t get one saying, “someone who was thinking about introducing you to somebody decided not to,” or “someone who thought you might be great to work with now thinks you seem like kind of a pain in the ass.” that stuff is completely invisible.” If you prefer, you could call it “flexibility”. Having smart, informed people read and consider my ideas Getting feedback from readers who have concrete experience with the topics I write about Joining interesting public conversations about the state of the industry Convincing people to think like me on points I care deeply about Building a reputation as a careful thinker and a good writer 3 In my own terminology, an “impure engineer” . ↩ Or at least the cleverest, or the most cutting. ↩ Too honest? I could be even more honest and add “encouraging people to potentially hire me if I’m ever looking”, too. ↩ Boosting engagement on Hacker News means the post is more visible for people who might actually read it. ↩ Imagine it’s 2015 again. Go on, you know you want to. ↩ Sunil Pai has a great post about this dynamic. To quote him: “you don’t get a notification that says, “three people respected you slightly less after reading that.” you don’t get one saying, “someone who was thinking about introducing you to somebody decided not to,” or “someone who thought you might be great to work with now thinks you seem like kind of a pain in the ass.” that stuff is completely invisible.” ↩ If you prefer, you could call it “flexibility”. ↩
A recent paper argues that frontier LLMs can work with neural network weights directly. This seems superhuman, but it isn’t. The paper shows that GPT-6 Astra can take a small neural network and interact with its weights directly, “well past what we’d expect from humans or conventional algorithms”. Specifically, Astra was able to identify what games a small network 1 was trained to play just by looking at the weights, and even tweak the weights itself. The most impressive result is Astra significantly improving a 0.8B Qwen LLM’s ability on mathematics by natively editing its weights. The reason to be excited here is that “superhuman ability to improve language models” is precisely the kind of thing that could cause recursive self-improvement. I think a smart human ML researcher could perform all of these tasks . Figuring out what game a model was trained to play based on the network sounds impressive until you consider three points: Here’s that calculation expressed as a heatmap in three lines of Python: And here’s the heatmap for each game: If someone showed you that heatmap without the game labels, I think you’d be able to immediately identify at least the first three games purely by noticing where the player is (e.g. for Breakout, in a horizontal line at the bottom) and where the moving objects are (e.g. for Pac-Man, in the distinctive Pac-Man maze). It’s neat 3 that Astra was able to calculate the norms in its head without executing Python, but not that impressive, and certainly not something that should shock anyone who’s been paying attention. I’m not going to go through each example 4 in the paper, so let’s skip straight to the most impressive one: fixing a 0.8B Qwen model’s ability to do mathematics. 0.8B is a lot of parameters, and “mathematics” is way more complicated than “guess what the game is based on how the player moves”, and editing weights is arguably harder than parsing them. What happened here? First, note that the Qwen model was not “bad at mathematics” in a pure capabilities sense. It had a specific problem where after 32k tokens it would get stuck in loops, so any mathematical task that needed more than 32k tokens would automatically fail 5 . If you had a model that looped like this, how would you update the weights to fix it? Personally, I would try steering : I would write some reasoning texts that looped and some reasoning texts that didn’t, calculate the difference between the looping and non-looping residuals, measure how much each weight contributed to that difference, then try manually trimming those top weights. This is basically the Heretic abliteration playbook. And this is exactly what Astra did! It wrote four pairs of texts, measured 6 the weights that contributed to looping, started cutting them down and testing if it worked, and ended up finding a patch that stopped the Qwen model from looping, thus making it able to work on mathematical problems for more than 32k tokens. Again, this is exactly what a competent human researcher could have and would have done. It’s cool that Astra was able to make direct weight updates. But we shouldn’t think of this like a radically new capability that will change how AI research is done. We should think of this as analogous to the ability of LLMs to emit machine code and tweak compiled binaries: it’s an impressive trick, but the best way to work with binaries is still to write source code and compile it. Likewise, the best way to update the weights of a model is to train it normally. To the extent that intelligent LLMs help with AI research, they will help by improving training , not by magically tweaking model weights directly. Around 17k weights. By taking the L2 norm of each input across the 32 weights in the first layer. Note that this is superhuman mental mathematics, not superhuman AI research. The most impressive one I’m skipping is hand-writing a neural network from scratch. While cool, if you can express a task as a simple program, we already know it’s possible to express that program as a NN , even by unaugmented humans . All the <32k token problems were solved successfully on the original Qwen model. Unlike with the toy game networks, it had access to Python. The input channels of the model directly listen to each 8x8 grid cell The input channels are labeled (e.g. “agent”, “moving objects”) It’s thus relatively trivial to calculate 2 which parts of the grid the first layer of weights cares about and why Around 17k weights. ↩ By taking the L2 norm of each input across the 32 weights in the first layer. ↩ Note that this is superhuman mental mathematics, not superhuman AI research. ↩ The most impressive one I’m skipping is hand-writing a neural network from scratch. While cool, if you can express a task as a simple program, we already know it’s possible to express that program as a NN , even by unaugmented humans . ↩ All the <32k token problems were solved successfully on the original Qwen model. ↩ Unlike with the toy game networks, it had access to Python. ↩
Disclaimer: While this post is about the “Black on Black” keycap set specifically, it is also a more general review of the PBS profile. In the Q3 2026 update I mentioned the set of PBS Black on Black, which has been on my Kunai ever since. I said back then that I might write a dedicated review once I could better judge it over a longer period of time. A year and a half earlier, in the Q1 2025 update , I had complained about the lack of textured PBT keycaps in the KAM profile and expressed my hope that PBS would eventually take that place. As far as the shape goes, PBS turned out to be pretty much what I was hoping for. Unfortunately, the same can’t be said about the texture, which is a lot smoother than I would like it to be. PBS is short for Penguin Belly Slide and it is a keycap profile designed by none other than Matteo Spinelli, better known as Matt3o, who is also the designer of the MT3 and MTNU profiles. According to Matt3o, the name comes from the hitting area , which is soft, round and oblong, and looks as if it was shaped over the belly of a penguin, while the slide is there because typing on the keys is fun. Unlike MT3 and MTNU, PBS is a uniform profile, meaning that every row has the same shape and height. Matt3o, whose first keycap set was in the DSA profile, calls PBS his tribute to flat, low-profile keycaps and an attempt at a uniform profile that is less boring than DSA. Matt3o began working on PBS in 2022, alongside MTNU, and initially planned to produce it with KBM, which had suggested increasing the wall thickness from 1.5mm to 1.7mm. The project eventually moved to Keyreative, the manufacturer behind the KAT and KAM profiles, with CannonKeys as the global distributor and with the walls back at 1.5mm. CannonKeys released the first PBS blanks in August 2024 and the Black on Black set in October 2024, as an in-stock product for $65, which was restocked several times over the following year. In 2026 a second round added a Modi kit with colored modifier legends. At first sight PBS looks like a cylindrical profile, with the rounded front and back edges and the straight left and right edges that Cherry and OEM keycaps have. The scoop of the hitting area, however, is not a cylinder but a sphere, which Matt3o cuts at the front and the back with a cylinder. The result is a hitting area that is as deep and round as the one of a spherical profile like MTNU, but without an obstacle at the front and the back, as the cylindrical cut lowers those edges. Matt3o describes it as the depth of a spherical top combined with the ergonomics of a cylindrical profile, and he estimates the hitting area to be about 20% larger than the one of MT3. The concave keys are 7.43mm tall at their highest point and the convex ones 7.91mm, which puts PBS slightly below DSA, at 7.84mm, and well below XDA and KAM, at 8.74mm and 9.05mm respectively. With a standard MX switch, the top of a PBS key ends up about 17.8mm above the PCB, compared to 20.5mm for Matt3o’s MTNU. I like the profile and its sculpt, especially the way the scoop hugs the fingertips. People who don’t usually like uniform profiles will likely find the scoop deep enough to keep the fingers from drifting onto neighboring keys, with edges that are soft enough to slide from one key to the next. PBS is a very comfortable uniform profile, more than DSA, XDA, KAM and uniform SA. Many profiles, like Cherry, MT3, KAT or SA, are sculpted , meaning that each row has its own height and angle, so that the upper rows face the typist and the lower rows face away. Sculpted sets need separate molds for every row, and each key only works as intended in the row that it was made for. A uniform profile like DSA, XDA, KAM or PBS uses the same shape for every row, so that any key fits any position of the same size, which, as Matt3o points out, makes every PBS set a free Colemak kit. It also keeps the cost down, as all 1u keys are identical, which was one of Matt3o’s goals with PBS. As I described in my Corne V3 build , the Corne only makes use of R2, R3 and R4 of a sculpted set, and every key has to come from the right row to keep the sculpt consistent. With PBS any cap fits on any position of the Kunai , as long as the size matches. This finally gives me proper legends, e.g. for my Control key, or my Return key. PBS is available with concave and convex keys, and it’s up to the designer of a set to choose which ones to use for the bottom row. Matt3o himself prefers concave keys for their flexibility on custom keyboards. The Black on Black base kit uses a concave bottom row, with convex spacebars and a handful of convex extras in sizes from 1u to 3u, which some owners use on thumb clusters. For homing , the kit includes F, J and numpad 5 keys with a bar as well as with a deeper scoop. Matt3o prefers the deep scoop, while, according to him, most people prefer the bars. On my Kunai I’m using the bars, and for the right thumb (space) I’m using a concave key. The heights are the highest point of a 1u key, as published by CannonKeys and Keyreative. Matt3o never published the row heights of MT3, so the MT3 numbers are my estimates, based on my own measurements. The Cherry profile goes back to the keycaps of Cherry’s own keyboards. GMK has been producing it in double-shot ABS since it took over Cherry’s double-shot tooling in 2011, and sells it as CYL these days. Cherry is a low, sculpted profile with four row shapes, namely R1 for the function and number rows, R2 for the top letter row, R3 for the home row and R4 for the bottom letter row and the modifiers. R1 is the tallest at 9.80mm and R3 the lowest at 7.30mm. According to Keyreative’s 3D models of its Cherry-profile caps, the lower a row is, the more its top is tilted away from the typist, from about 0 degrees on R1 to about 11 degrees on R4. The top of a Cherry key is a cylindrical cut that runs from the front to the back, so that it curves down in the middle when seen from the front, while it’s a straight line when seen from the side. Cherry-profile sets come as double-shot ABS from GMK, with 1.5mm walls and a fine texture that turns into a shine over time, or as dye-sublimated and double-shot PBT from manufacturers like EnjoyPBT and Keyreative. I’m not a big fan of the Cherry profile, although on the Corne, which only uses R2, R3 and R4, it felt more balanced to me than on any 60%+ keyboard. PBS shares the straight sides, the rounded front and back edges of the Cherry profile, but it replaces the cylindrical channel with a spherical scoop and the four rows with a single shape. At 7.43mm, PBS is about as tall as the Cherry home row and lower than its number row, and on the three rows of a Corne the two profiles are within about a millimeter of each other. MT3 is Matt3o’s first profile, which he began drafting in 2015 in a discussion on Deskthority, where the consensus was to create something that resembled the keycaps of IBM’s beam spring keyboards. Massdrop, later Drop, produced it, starting with /dev/tty, which shipped in January 2018 as dye-sublimated PBT with walls of around 1.8mm, while the double-shot ABS sets started with Susuwatari in 2020. MT3 is a tall, sculpted profile with a deep spherical scoop, which Matt3o designed for keyboards with an incline of 5 to 9 degrees, at which the home row ends up parallel to the desk. It adds a steep function row, R0, and offers the bottom row in two versions, an angled R4 and a flat R5. Matt3o’s own 3D files put R0 at 16.2mm, and the letter rows are roughly 11.4mm to 12.8mm tall, which makes MT3 the tallest profile in this comparison. Drop, which owned the MT3 molds, closed its store at the end of March 2026 and turned drop.com into a hub for the collaborations of Corsair, which had bought it in 2023. I favor MT3 for its sculpting, and it had been on the Kunai before PBS. PBS has a scoop that I would compare to the one of MT3 and a hitting area that Matt3o estimates to be about 20% larger, while it does away with the row sculpting and a lot of the height. PBS is quite significantly lower than MT3, making my Kunai a lot more portable. KAT, short for Keyreative All Touch, is the sculpted profile that Keyreative began designing in late 2016. Keyreative describes it as a MiniSA , with the slope design of the Cherry profile, taller rows, especially the function row, and a spherical scoop. Keyreative counts the rows from the bottom up, from R1 for the bottom rows to R5 for the function row. Its own chart puts the number row at 11.57mm and the home row at 9.03mm, while its 3D models put the function row at about 13.5mm. KAT is made of PBT, either dye-sublimated with 1.7mm walls or double-shot with 15% glass fiber and walls of 1.65mm, and Keyreative also lists double-shot ABS as an option. Its finish is the silky, satin-like smoothness that PBS has as well. KAT is one of the profiles that I generally prefer. On the three rows of a Corne, PBS is between 1.6mm and 3.2mm lower than KAT. KAM is Keyreative’s uniform profile, which the company developed from the home row of KAT, starting in February 2018. Every KAM key is based on the KAT home-row key, with a shallow spherical scoop and a height of 9.05mm, or 9.94mm for the convex keys. KAM is made of dye-sublimated PBT with walls of around 1.65mm, either matte or glossy, and Keyreative claims that its glossy KAM caps, like the black KAM Blanks that I had on the Kunai, were the first glossy PBT keycaps. Compared to KAM, PBS is about 1.6mm lower and has a wider and deeper scoop, or, as CannonKeys puts it, KAM is taller than PBS with a less pronounced scoop. Keyreative’s own 3D models put the center of the PBS scoop about 1.1mm below its rim, against about 0.7mm on KAM. The convex keys differ in the same way, as they rise 0.89mm above the concave keys on KAM, but only 0.48mm on PBS. PBS keycaps are made of PBT, while GMK and many of Drop’s MT3 sets use ABS. ABS is an amorphous plastic that softens at around 100°C, while PBT is semi-crystalline and keeps its shape up to about 220°C. PBT is also harder and denser, which makes a PBT cap about a quarter heavier than an ABS cap of the same shape, and it takes a lot longer to develop the shine that finger oils and wear eventually give to every keycap. ABS, on the other hand, yellows under UV light (shout out to GMK Bleached! #scnr ), but it is easier to mold precisely, while PBT shrinks more as it cools in the mold, which makes long keys like spacebars prone to warping. Keyreative itself writes that the PBT used for keycaps is usually PBT with added glass fiber, which reduces warping. In a stream about the PBS prototypes, Matt3o mentioned that Keyreative had added fiber to reduce shrinkage, which caused issues with the shift keys and the 2u keys that were to be fixed for the final release. Some buyers still report 2u to 2.25u caps that are tight on certain stabilizers, which however doesn’t affect the Kunai, as a Corne doesn’t use any stabilizers. At 1.5mm, the PBS walls are as thick as the ones of GMK’s ABS caps, but thinner than the 1.65mm to 1.7mm of Keyreative’s KAT and KAM, the roughly 1.8mm of the first dye-sublimated MT3 set, or the 1.7mm that Matt3o had initially planned for PBS. As for the legends, double-shot molding, which GMK and Drop use for their ABS sets, makes every cap out of two plastics, one for the legend and one for the body. The legends can’t wear off, and both colors can be as dark or as light as the plastics allow. Dye-sublimation instead prints the legends onto a film and uses heat to make the dye migrate into the surface of the cap. This works on PBT because it keeps its shape at the temperatures required, while an ABS cap would deform. The dye can, however, only add color, so the legends have to be darker than the cap. With dye-sublimation, light legends on dark caps require reverse dye-sublimation , which dyes the whole cap except for the legends. The process is difficult to get right, and Keyreative stated that it tried to avoid it because it was unstable, with a defect rate of over 40%. Laser etching, which is common on cheap OEM keyboards, burns the legends into the surface instead and usually leaves them lighter than the cap and noticeable to the touch. PBS uses dye-sublimation because Matt3o wanted the profile to be as affordable as possible. Dye-sublimation requires lower minimum order quantities than double-shot, and it doesn’t lock a set into a single typeface, as the legends are printed rather than molded. My gripe with PBS is the texture. I have long preferred thick PBT with a noticeable, rough texture, like the biip MT3 Extended 2048 that I mentioned in the Q1 2025 update, and compared to that the PBS caps are way too smooth for my taste. I had hoped that PBS would be the textured PBT in a KAM-like profile that I was missing, and in that regard it’s a bit of a bummer. The texture of a keycap is mostly a property of the mold rather than the plastic. GMK, for example, attributes the feel of its keycaps to the spark-eroded structure of its molds, and Keyreative even produces glossy PBT, like the black KAM Blanks that I had on the Kunai before and that turned out to be fingerprint magnets. The smooth finish of PBS is hence a choice, and as far as I can tell it is an intentional one. Matt3o designed PBS partially because of the rough texture of Signature Plastics’ DSA, and he described the surface of the PBS production prototypes as pretty smooth. Even his newer PFF profile for low-profile switches, which Keyreative manufactures as well, is only supposed to be one grade rougher than PBS, a difference that he himself calls hardly noticeable. Compared to what you typically find on the other profiles, PBS is among the smoothest. GMK’s ABS Cherry sets start out with a fine texture that wears into a shine over time, and Cherry-profile PBT from other manufacturers ranges from smooth to grainy, depending on the maker and even the production year. MT3 in PBT, like the biip set, has a pronounced, rough texture, while Drop’s double-shot ABS MT3 is smooth from the start. KAT and the matte KAM sets come from the same manufacturer as PBS and have a similar finish, which people describe as satin or silk for KAT and as velvet for KAM. PBS is at least as smooth as KAT, and it’s almost slippery. As a matter of fact, while removing the keycaps to replace switches I continuously drop every other PBS keycap because they are incredibly smooth, very flat and have rounded edges. On top of that PBS, just like KAM, shows oil and fingerprints a lot quicker than the textured DSA due to its choice of material and texture. Many like it that way, but I frankly don’t and it’s one of the most off-putting fact about the keycaps. I’m not alone with my complaint, however, as some in the PBS community have apparently been asking for a slightly rougher texture ever since the first blanks shipped in 2024. Despite its name, the PBS Black on Black isn’t black, but more of a dark gray. This is especially visible in bright daylight, and on the Kunai, which I painted in Tamiya TS-81, a light gray, it’s obvious that the caps aren’t black at all. If anything, it’s black legends on charcoal keycaps, so I would have rather picked something along the lines of Black on Charcoal as a name, because Black on Black is definitely misleading. The reason for this is, as I explained above, is most definitely the legend process. As dye-sublimation can only add color, you can’t sublimate black legends onto a black cap, because nothing is darker than black. This is why black-on-black sets with dye-sublimated legends, from PFU’s charcoal HHKB to Drop’s Artifact Bloom Black on Black, use a dark gray body with even darker legends. Even GMK’s double-shot Stealth set gets its contrast from an anthracite gray against black. Matt3o, who calls himself a PBT guy, wrote back in 2022 that the only downside of PBT is that you can’t do dark sets with it. In his stream about the PBS production prototypes, he also mentioned that he had wanted to make the black of the Black on Black lighter, but that everyone told him not to, and for good reason. To be completely honest, I don’t like the color at all. Unless I’m in almost complete darkness, the charcoal is noticable as such and really doesn’t go well with the rest of my specific board. As a matter of fact, even the (actually black) RAMA USB A-to-C keyboard cable is visibly different in color. Even worse, the two GMK Stealth RAMA artisans that I use as my upper left tab and my upper right backspace key are noticably darker than the keycaps. Hell, even the L-connectors of my TRRS cable are significantly more black than the keycaps. The PBS Black on Black sadly sort of ruins the aesthetics that I went for with my Kunai, making it look rather cheap. It reminds me a lot of KPrepublic’s $40 PBT Gentleman Black keycap set. Note: A matte surface scatters part of the incoming light back in all directions, which makes the color underneath appear lighter than it is. In a measurement by Konica Minolta of a dark plastic part with a matte and a glossy area, made from the same resin and color, the matte area came out noticeably lighter than the glossy one. Hence a dark, matte cap looks grayer the brighter and more diffuse the light is, as it is in daylight. The legends are slightly darker than the caps and, depending on the light, barely visible. Matt3o said that you could almost consider the set blanks, and I can confirm that legends are easy to read in direct sunlight but completely disappear in low light. Given that the legends therefore serve no practical and almost no aesthetical purpose, it makes me wonder even more why Matt3o went for this relatively cheap looking charcoal color for the keycaps to begin with, rather than making them slightly darker. PS: The modifiers use text legends throughout, with CODE instead of a Windows or Command key, which is what I prefer. One key benefit of PBS over MT3, which I also favor for its sculpting, is the height. On the three rows that a Corne uses, MT3 keys are roughly 11.4mm to 12.8mm tall, while every PBS key is 7.43mm, which makes the Kunai about 4mm to 5.5mm flatter with PBS than it was with MT3. When I built the Kunai I wrote that, without palm rests, the board is on the higher end, and that I could imagine MT3 or SA making it too tall, which turned out to be true. With PBS the Kunai is significantly flatter than it was with MT3, bringing it back down to a slightly more manageable height. In addition, it also makes it a lot more portable, as it takes up less room in the NuPhy NuPack that I carry it in. PBS has become one of my favorite keycap profiles , as it is pretty much what I was hoping for. Compared to MT3, which I like as well, PBS makes the Kunai a lot flatter and hence more portable. However, the caps are way too smooth for my taste, and the color of specifically the Black on Black set is pretty bad. They aren’t black but more of a dark gray, which in bright daylight and on the light gray Kunai looks rather unpleasant. If you like the silky surface of Keyreative’s KAT and KAM, and a charcoal instead of a true black doesn’t bother you then I guess PBS Black on Black can be for you. If, like me, you prefer a rough, grainy PBT, PBS will sadly feel too smooth. I’m not aware of any PBS set with a rougher finish and considering the author’s posts and preferences I’m doubtful that we will ever see one. For me, PBS Black on Black definitely is kind of a love-hate relationship.
I was reminded today of AI Adoption Rate Trending Down for Large Companies , a chart produced by Apollo Global Management in September 2025, which appeared to show a slowdown in U.S. corporate AI adoption based on numbers collected by the U.S. Census Bureau. At the time I used that chart to demonstrate GPT-5 data analysis , and then to experiment with Claude's new Code Interpreter . I decided to try running those numbers again to see what had changed over the past year, and in doing so learned that the Bureau has fallen foul of Executive Order 14434 , aka the asinine Super Intelligence one. Every two weeks the U.S. Census Bureau runs a survey called the Business Trends and Outlook Survey (aka BTOS). Each time it goes to 200,000 businesses out of 1.2 million total - the surveyed group (a "panel") cycles such that an individual business only completes the survey every twelve weeks. The latest data is published on alternating Thursdays at 10am. I used the most recent numbers (published October 22, 2026) to create a new version of the AI adoption chart, updated to reflect both the last twelve months and a change that was made to the survey question sent out by the Bureau: For comparison, here's that AI adoption chart from last year (my GPT-5 version): In September 2025 the largest (250+ employees) companies answered "yes" around 12% of the time to the question about whether they used AI in "producing goods or services". Under the new "any business function" wording, by February 2026, those same large companies answered "yes" around 30% of the time, and by October 2026 that had increased to around 40%. The data in the above chart comes from the "Employment size class" XLSX file, available from here . Here's a copy of the file I used. You can view the survey document that is sent out to businesses - here's my snapshot copy of that 346KB PDF. As the chart demonstrates, the question asked by the Bureau changed in a material way on November 17th, 2025. Prior to that date, the question they asked was this (emphasis mine): In the last two weeks, did this business use Artificial Intelligence (AI) in producing goods or services ? (Examples of AI: machine learning, natural language processing, virtual agents, voice recognition, etc.) On December 3rd, 2025, the Bureau announced a significant change to that question. Here's the replacement: In the last two weeks, did this business use Artificial Intelligence (AI) in any of its business functions ? (Examples of AI: machine learning, natural language processing, virtual agents, voice recognition, etc.) No more "producing goods or services" - now it's "in any of its business functions"! This means that the answers to the question before and after November 17th are not directly comparable. When I went looking for the updated question language in the latest XLSX file, I couldn't find it. I found this instead: In the last two weeks, did this business use Super Intelligence or SI (formerly referred to as Artificial Intelligence or AI) in any of its business functions? (Examples of SI use include: SI-powered data analysis; generative SI for marketing, product design, or coding; virtual SI agents for customer service; SI-powered forecasting or pricing; SI-driven computer vision for quality control, etc.) The BTOS about page explains what happened: Beginning with the October 19, 2026 collection, instrument wording was updated due to Executive Order 14434 . The current numbers reflect the old Artificial Intelligence survey wording - the first survey results reflecting the Super Intelligence wording are due for publication on Thursday. The obvious open question here is whether this wording change is going to affect the survey results. The businesses answering this survey will have last seen that question twelve weeks ago, when it was still called AI. Hopefully the new language isn't going to confuse them, especially since the survey helpfully spells out "formerly referred to as Artificial Intelligence or AI". I can't help but laugh at the absurdity of "SI-powered data analysis; generative SI for marketing, product design, or coding; virtual SI agents for customer service" though. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options .
An irritating thing about database vendors is that they all claim their offering is "highly available". If pressed they'll say they're talking about their SLA , and while this is not technically lying, it is very misleading, because people who know some distributed systems lore may have a much different conception of what "availability" is; namely this from the CAP theorem : The CAP theorem has been critiqued - which we'll get to - but it is massively influential. Jepsen of database report fame uses the term similarly, furnishing us with three categories of availability: In the Jepsen terminology most databases today are unavailable. A good rule of thumb is anything with a single writing node falls under this - no matter what their marketing material says. I feel availability has been diluted too far to serve in the technical lingua franca of people building systems. Database vendors have convinced large numbers of us that there isn't even a availability/consistency trade off to be made - that they in their engineering might have miraculously achieved both. It's at times like this we need an academic. Not just a specialist, but a generalist, one who is able to take a broad subject area, analyse it intensely, and understands what he reads so totally he is able to explain it to a general audience. Availablity is dead, is anybody here a doctor??? In his excellent paper A Critique of the CAP Theorem , Martin Kleppmann goes into some detail, noting both the metric that is empirically observed definition the vendors use, and the more academic one used in the literature. It's well worth reading in full, but I want to focus on his "delay sensitivity framework": reproduced here (where d is network delay): In typical Kleppmannian fashion he has managed to reduce complex phenomena from a great number of real world systems into a simple table. But I think to serve as a piece of lingo, "delay sensitive" is not quite catchy to use in real speech. And so I humbly propose consistency/latency . It works because it's a single term; that DB is high cosnsitency/latency , that one is low consistency/latency . Consistency/Latency are an inseparabable package; you are either high or low, no "strong consistency AND highly available" vendor shennanigans. I am of course simplifying a great deal of complex research, but the general rule is that as consistency rises, latency increases with it; all else being equal it's always going to be faster & more robust to directly write to your local node than wait for a consensus protocol to finish. While I desperately hope this becomes popular enough that we can call it conlat for short, I have already started using this classification when talking about systems at a high level, and will continue do so. I hope it may also serve others well.
Google's earnings miss looks a lot like Facebook's, and that's a credit to Google. The company is being more aggressive than they get credit for.
Rust favors composition over inheritance. I’m sure you’ve heard that before, but rarely does anyone take the time to explain what it means in practice. Also, why is this model the right choice for Rust? Let’s take a step back and look at an example: have you ever asked yourself why works the way it does? How does know to produce a here? To answer that, let’s first look at the signature of : Note how it doesn’t mention at all. In fact, it doesn’t “know” about any collection type or any other type, for that matter. The thing it does know is that whatever type you want must be capable of constructing itself from an iterator. And happens to be a type that does. Further note that we didn’t specify the element type of the vector. The compiler is clever enough to infer that from the context! The mechanism is the same for any other supported type, like hash maps, strings, or any custom collection type of your choosing as long as it implements for the iterator’s item type. And what “collecting” means is also defined by the type itself. A , for example, has the ability to collect characters: And a can collect key-value pairs: It’s only fair to mention that interfaces can also express shared contracts. Here, two independently defined implementations connect through a shared item type. This is “composition through contracts”: Here, a producer yields values of some type through the trait. On the other side, a collection implements to describe how it can build itself from those values. The shared item type connects the two contracts, and is the bridge between them. It requires only that the destination can be constructed from the items the iterator yields. In our first example, the annotation selects the destination type. Because implements , the compiler knows how to connect that destination to our iterator. is the glue that passes the iterator to that implementation, which creates and fills the vector. Note how neither side needs to know the concrete type on the other side. The iterator doesn’t need to know how a vector stores its elements. And the vector doesn’t need to know where the items came from, only how to iterate over them. That means we can extend either side independently. A new iterator can feed existing collections, and a new collection implementing can accept existing iterators yielding . We don’t need a separate conversion for every producer–collection pair. All we need is one implementation, and is the glue that brings it all together. In simple Rust pseudocode, we can expand the calls step by step: The value of an abstraction is the code that no longer needs to exist between its users. Class inheritance models “is-a” relationships. In Java, for example, an is a subclass of . An encrypted socket is still a socket, after all. But does every socket operation make sense for it? lets a TCP connection stop sending while continuing to receive, a feature called “half-closing.” Code using a socket might therefore do this: On Android API levels 21 through 23, the platform’s implementations rejected this operation with . This caused a real crash in OkHttp 5.2.0 . Modern OpenJDK supports TLS-aware shutdown , so this is a historical mismatch, not a universal limitation of TLS. Still, the type exposed an operation without telling callers whether that implementation supported it. With composition, a TLS connection can hold a socket internally without exposing every operation on it. It can provide reading and writing, with a separate capability for half-closing where supported. We can reuse the transport without inheriting its entire interface. Simply replacing inheritance with interfaces—or traits—doesn’t fix a poorly chosen contract. What’s important is that the contract matches the capabilities of its implementations . Rust makes trait-based composition central to code reuse. In Rust, we ask what a type can do , not what it is . In other words, traits describe capabilities, not taxonomies. Here are simplified versions of two central I/O traits from the standard library: Note how these are two independent traits. A type can implement , or , or both, or neither. You’re declaring which operations a type supports, without inheriting a bunch of unrelated methods. The capabilities are orthogonal. For example: The trait implementation belongs to the type , not to a concrete value. Take , for example. It implements both and , but a file opened in read-only mode will still fail at runtime if you try to write to it. Even a file opened for writing can fail when the disk fills up. The trait implementation is a promise that the type supports those operations, but what it can’t do is guarantee that every operation will succeed at runtime. If you want to distinguish read-only and writable handles at compile time, you could define a and a and only implement the appropriate traits for each. That rules out unsupported operations in your API, but it still doesn’t make I/O infallible. Rust can express that a container has a capability only when its contents do. For example, implements when implements . Thanks to this trick, we don’t end up with two different types, one cloneable and one not. Here’s the same pattern for a small wrapper you could define yourself: A is cloneable because is cloneable. A is not, since does not implement . If you were to call on that batch, the code wouldn’t compile. You can store non-cloneable items in a batch; you just can’t clone that batch. For this wrapper, you can use to generate the conditional implementation automatically. Note that the bound belongs on the implementation , not on itself. Making your type cloneable requires , but apart from that, is still a perfectly valid type for non-cloneable elements. The compiler works out which operations are available from the capabilities of the component types . A conditional implementation can also apply to every type that satisfies a trait bound as opposed to one particular wrapper. This is called a blanket implementation . For example, the standard library implements for every type that implements : All you have to do to get a method is implement for your type. The compiler automatically generates the implementation for you. The rule applies even to types written long after the standard library itself. This is an elegant way to express a reusable implication between capabilities: “If a type can be displayed, it can be turned into a string.” The same idea appears in conversions. You might know that implementing automatically provides through a blanket implementation. That’s why you normally implement rather than writing the corresponding implementation yourself. Conditional implementations also let us add behavior by wrapping one value in another. This is a different type of composition: “composition of values.” A wrapper holds another value and builds on its capabilities. Take our trait from earlier: So a parser accepting doesn’t need separate support for files, buffered files, and length-limited buffered files. It just reads bytes! We can add buffering and a byte limit without teaching the parser about either. The parser doesn’t need to understand how its input was assembled. This works because each wrapper preserves the contract that the next piece needs. A implements when does, and so does . The result of one composition becomes a building block for the next. You can see the same idea in iterator chains. Each type implements , so you can chain them. Writing your own iterator connects your type to this vast ecosystem of adapters. All you need is to provide the associated type and the method . , , , and already have default implementations, available when their additional bounds are satisfied. This is how you reuse code without a base class. Default methods provide behavior within a trait you implement; blanket implementations can make another trait apply automatically. But adapters don’t have to be limited to a single, rigid contract either. They can provide rich, additional capabilities, too. For example, implements when its input iterator does, and it does so with an appropriate trait bound on the mapping function: So because the adapter has a conditional implementation for , it preserves the ability to iterate backward through the input. We’re not limited to basic iteration. And since we know the concrete iterator types, the compiler can optimize and inline the entire chain of adapters into a single loop. Optimized iterator code can perform comparably to a hand-written loop . So fully embracing iterator patterns has no runtime cost. Hold on, Rust has another way to express “different kinds of things that are related”: enums! These are useful in situations where you know the full list of possibilities in advance. Enums are great if you almost always want to handle all of the variants in one place. Traits are the better model if different types implement the same behavior and you don’t know all of the types right now. In other words, enums are a closed set of variants while traits are an open set of implementations. You can’t add new variants to an enum without changing the enum definition. But traits enable the open/closed principle: you can add new implementations without modifying existing code. This allows other crates to extend your library without needing to modify your code. Imagine your library defines the and types, and you want to provide a way for users to fetch records from different sources. In your library, you implement for a and a . But now, users can implement it for their own or a . Heck, you don’t even need to know these types exist. And yet, your contract still holds: any type that implements can be used to fetch records. The reverse is also true: an external contributor can define an extension trait and implement it for your type without you knowing about it. You can even do that with types from the standard library: As long as the trait is in scope, you can call the method on a : No need to modify or derive a subclass. This implementation is allowed because we define in our crate. Rust’s coherence rules restrict implementations involving foreign traits and types. For example, we couldn’t implement the standard library’s trait directly for , since neither the trait nor the type is ours. This is composition at the level of behavior : we combine an existing type with an independently defined capability. The type and the trait don’t need to know about each other. The same type can implement other, unrelated traits, and each function can ask for just the capabilities it needs by means of trait bounds: This open-set/closed-set principle is a good way to think about when to use enums and when to use traits. But to be fair, the lines can blur a bit. An enum can also implement traits, and a library can restrict who implements a trait through a technique called a “sealed trait.” In application code, exhaustive enum checking can make refactoring easier because the compiler will tell you which statements need to be updated when you add a new variant. Introduce a trait instead when you have a good reason to share a contract across types . The more unrelated responsibilities a trait combines, the harder it becomes to implement and reuse. Every implementer has to account for operations it may not need. Consider this application context: Ask yourself: what concept does this trait capture? What contract does it enforce? It combines reading records, writing records, and running the application. Perhaps your application really needs one type to provide it all, but a function that copies records shouldn’t require its inputs to run an application. Let’s split the capabilities: Now a read-only source can implement , a write-only destination can implement , and the execution runtime is left out. Your functions don’t need to know about the runtime either, and you can compose the exact capabilities you need: Each of your types can implement either trait. Or both. This also explains how independent crates work with your types. Serde defines and ; your types implement them, often through derive macros. Serde’s serializers don’t need to know about your application’s structs in advance. They just use the shared contract: a type that implements can be serialized. Code becomes more composable as a consequence. If Serde decided to couple serialization and deserialization into a single trait, some use cases would be impossible to express. All of that is great for testing, too. We can test with a fake implementation and a fake implementation, without creating a runtime. The goal isn’t to make every trait contain exactly one method. has many methods, but they are all strictly related to the task of iterating over a sequence of values, and all but one ( ) have a default implementation. It’s fine to keep methods together when they describe the same contract, but if you’re unsure if callers need the independent capabilities, split your traits. Below are a couple of concrete tips around trait design. When you approach a problem, ask what your code needs a type to be able to do. Often, an existing trait already expresses that. Check the standard library and popular crates first. And sometimes a concrete type or an enum is enough. Write the code concretely first. If you find yourself copying similar implementations across different types, look for a shared contract. By then, you’ll have concrete examples to test your abstraction against. And when you do need a new trait, give it a contract you can explain in a single sentence. Abstraction has a cost for readers. They may need to follow bounds and implementations across several files to understand a call. Sometimes that indirection hides details they shouldn’t need to care about. Sometimes it hides an important detail they should. Trait-based abstractions in Rust often incur zero additional runtime overhead, but there is a compilation cost as well. The costs depend on how you use the trait. Generic code is usually monomorphized: the compiler generates code for the concrete types that use it. That permits static dispatch and inlining, but many instantiations can still increase compile time and binary size. For dyn-compatible traits , dynamic dispatch through lets the same caller operate on different implementations, but introduces indirection and can limit inlining. Not every trait supports this: , for example, is not dyn-compatible. Keep these trade-offs in mind when you introduce a trait. There is no free lunch. Let’s come back to our once more. So far, we’ve used the destination type to choose a collection. But it can also determine control flow. For example, we can decide to stop iterating on the first error or keep going and collect every result, including errors. I first learned about this from Karol Kuczmarski’s Iteration patterns for Result & Option , still one of my favorite Rust posts. Here it is in action: Note that in both cases, the iterator expressions are identical . The only thing that changes is the destination type: In the second case, is never parsed. If every parse succeeds, the second version returns containing the vector of values. This behavior comes from the standard library’s implementation for . Here’s a stub of that implementation: What this says is: An iterator of values can be collected into a , as long as can collect values of type . The bound means that could be a or any other type that can collect those values. itself doesn’t need to know any of this: it just calls the destination type’s implementation. See how all of those pieces fit together? That’s the real payoff when you think in traits. A single implementation can connect a type to an entire ecosystem of algorithms, adapters, and collections. A few carefully crafted capabilities can combine in ways their authors never imagined. That makes the whole greater than the sum of its parts. And that’s just great API design. A byte slice ( ) implements but not . The returned by implements but not . A implements both. A implements . Wrapping it in a adds buffering, and the result still implements . Calling on that reader limits how many bytes it yields, and the result still implements . produces a iterator that applies a function to each item of the underlying iterator . produces a iterator that yields only the items of that satisfy the predicate . produces a iterator that yields at most the first items of . When you approach a problem, ask what your code needs a type to be able to do. Often, an existing trait already expresses that. Check the standard library and popular crates first. And sometimes a concrete type or an enum is enough. Write the code concretely first. If you find yourself copying similar implementations across different types, look for a shared contract. By then, you’ll have concrete examples to test your abstraction against. And when you do need a new trait, give it a contract you can explain in a single sentence. Abstraction has a cost for readers. They may need to follow bounds and implementations across several files to understand a call. Sometimes that indirection hides details they shouldn’t need to care about. Sometimes it hides an important detail they should. Trait-based abstractions in Rust often incur zero additional runtime overhead, but there is a compilation cost as well. The costs depend on how you use the trait. Generic code is usually monomorphized: the compiler generates code for the concrete types that use it. That permits static dispatch and inlining, but many instantiations can still increase compile time and binary size. For dyn-compatible traits , dynamic dispatch through lets the same caller operate on different implementations, but introduces indirection and can limit inlining. Not every trait supports this: , for example, is not dyn-compatible. collects every parse result. returns the first error and stops advancing the iterator immediately.
Powerful AI is unpredictable. Your robotic butler may suddenly decide to kill you 1 for no comprehensible reason, in the same way that Bing’s helpful chat AI would sometimes try to blackmail or seduce its users. “Solving alignment” means understanding even superintelligent AI minds well enough to be confident they’ll never do that. As I see it, solving alignment for LLMs isn’t just impossible, it’s incoherent: we can’t even articulate what it would look like to succeed. Prior to LLMs, AI safety researchers thought AI would be a “blind optimizer”: like a chess engine, it would be superhumanly focused on achieving some fixed goal. If the goal wasn’t safe (for instance, “manufacture cheap paperclips”), a powerful AI might wreak havoc on the way to achieving it. Aligning AI thus meant figuring out how to specify the right goal. This is harder than it sounds 2 , but it’s at least possible to imagine it happening. Unfortunately, modern LLMs aren’t like chess engines. Instead of being trained on some concrete verifiable reward 3 , they’re trained to imitate a giant, highly-varied corpus of human writing. This makes them weird . They contain a nearly infinite array of personalities, which they seamlessly switch between based on their inputs. Frontier AI labs try hard to make these personalities safe and stable, but there are still many strange cases where LLMs seem to switch into mysterious or dangerous personas. The problem here is that LLMs don’t have fixed goals . There’s nothing that you could, even in principle, formally specify in order to constrain their behavior. The “goals” that LLMs do have are informal, organic, multifarious, and constantly shifting as the dominant persona (or blend of personas) changes. They’re nicknamed shoggoths for a reason. So we can’t align them by specifying their goal. Instead, we have to try the other approach: understanding them well enough that we can be sure they’re safe. And unfortunately that approach is doomed from the start. What does it mean to ask for a full understanding — or full enough understanding — of why an LLM generates some token? In one sense, we have that already, because we can mechanistically trace through the calculations involved. But we don’t know what those calculations mean . Saying that Grok called itself “MechaHitler” because these billion weights multiplied in this way doesn’t tell us why Grok did it. What we would need for alignment purposes is a more human-like and less mechanical sense of understanding: a full understanding of motive . To see why a full understanding of motive is a foolish thing to ask for, imagine asking it about a human. Suppose you were hiring a nanny or electing a president and you wanted to be sure that they were going to act in your interests 4 . Humans sometimes do very surprising things out of nowhere, after all. Would it be sensible to ask for a full understanding of that person’s mind, in order to reassure yourself that they would act in the way you wanted? Of course not. It’s meaningless to even talk about a “full understanding” in this sense, because the mind is not a mechanism that operates via clean, understandable abstractions. It is a morass of electric and chemical impulses from which a “self” bubbles up 5 . There are no motives of the kind you’re looking for! Motives bottom out early on, and tracing the causal chain beyond that can only be done mechanistically: i.e. unhelpfully, for the same reason that explanations based on model weights are unhelpful. Couldn’t a sufficiently detailed mechanistic explanation give us the confidence we need? The best we’ve got along those lines right now are the tools of “mechanistic interpretability”: features, neurons, probes, and circuits. You observe the model doing something, see which parts of its “brain” light up 6 , and correlate those with some pattern of behavior. If you can then induce that pattern by artificially boosting that part of the “brain”, you can claim victory. Even assuming total success, this program cannot deliver us the kind of understanding we’re looking for. First, we label patterns based on behavior we observe, but the main misalignment case we’re worried about — the LLM suddenly performing some shocking harmful action — is definitionally something we aren’t going to observe before it happens. We can’t identify a feature without first eliciting its behavior 7 , and we can only elicit a small subset of potential model bad behavior. There will always be many untracked failure modes. Second, the labels we apply to patterns are our guesses, and may be wrong. The philosopher Kripke famously argued that, until you run into the relevant case, there’s no meaningful difference between the “addition” rule and the “quaddition” rule (a version of addition where if either number is greater than 57 the answer is always 5). Likewise, the LLM feature we call “kindness” might be “kindness unless the year is 2030”, or “kindness unless the LLM is in a position to cause mass casualties”, and so on. Those features are completely indistinguishable 8 from our perspective. Third, even an exhaustive well-labeled map of features and patterns would not give us certainty that a model could not go rogue. The full combination of features involved in any model action would be far, far too complex to interpret. For instance, if a model has “evil” lit up, is it because it’s trying to avoid evil, or trying to cause it, or trying to combat it, or is worried about some hypothetical evil action, or is anticipating that a user might make the judgement? We would have many, many billions of questions like that to answer per-token . Far too many for a human to comprehend 9 , certainly. Could some hypothetical, radically different kind of mechanistic interpretability somehow predict all possible actions a model might take in advance and rule out actions like “kill all humans”? I think this at least could be semantically coherent, but it runs into the problem above. Models can accept a near-infinite array of inputs and give a near-infinite array of outputs. This definitely cannot be comprehended by a human, only by another model, which itself would have to be checked by another model, and so on. If you think solving LLM alignment is possible, I would love to read your description of what it would even look like for us to understand LLMs well enough to be sure they’re safe. I would love to know if you think humans are similarly understandable in principle (and if not, what makes LLMs different). It seems to me that if we think clearly about the topic, we don’t even know what it would mean to be confident that we have aligned an LLM properly 10 , because “motive” simply doesn’t exist at the level of detail we’d need. Of course, we have mitigations in place. Chain-of-thought monitoring is surprisingly helpful for a mechanism that appeared by accident, so long as we never train on reasoning tokens themselves. We can sandbox models and put humans-in-the-loop and watch for feature activations and so on. But it’s been well understood since the early 2000s that these are insufficient 11 to detect or control a superintelligence. Whether you think mitigations are enough depends on whether you take superintelligence seriously. Mechanistic interpretability is still a useful field. Extracting features can help us train models that do less reward-hacking and are more helpful under ordinary circumstances. There’s just no way it will ever solve the problem of LLM agents going rogue and causing massive harm, because the kind of understanding we’d need to prevent that is impossible in principle. It’s relevant to note that humans sometimes do this too. From Vaneigem : “One fine morning, the faithful lackey, who has hitherto identified completely with his master, leaps on his oppressor and slits his throat.” For instance, the goal “eliminate suffering” is best satisfied by destroying all humans, the goal “maximise everyone’s happiness” is best satisfied by hooking everyone up to a fentanyl IV, and so on. It’s also very difficult to find a way to translate concepts like “happiness” and “well-being” into a formal goal, and if you get it wrong, the difference between the concept and your formalization is likely to lead to dire consequences. Yes, I know LLMs are partially trained in this way, but being RLVRed on millions of different rewards does not produce a coherent goal. There’s no way to formalize in advance the utility function that RLVR instils in the model, and no way even to reliably predict how it will interact with the goals and personas that emerge from the base model. Of course, for humans you can rely on institutions, social proof, and so on. This doesn’t work for superintelligent LLMs because they (a) arrive fully-formed with no social proof and (b) are smart enough to be much more dangerous than a single rogue human. (Incidentally, the LLMs I asked to review this post were obsessed with this point, and wanted me to elevate it into the main thrust of the post). Perhaps as an epiphenomenon. If you’re interested in this idea and want to see it taken further, I recommend (but do not endorse the ideas in) Blindsight . Which parts of the model’s internal state have high values, for instance. That doesn’t mean we have to deliberately elicit one feature at a time — dictionary learning works by just running arbitrary content through the model and seeing what lights up — but it does mean that we have to actually trigger the behavior in order to see it. If you’re thinking “well, but isn’t the simple feature more likely via Occam’s Razor”, I recommend browsing through NeuronPersonas (the linked feature is “Nervous Person Waiting To Be Let Go Who Escalates Ammonia Solution Concentrations”). I’ve heard people punt on this question by saying “well, of course I couldn’t use a fully-mapped LLM brain to figure out if it’s dangerous, but another smart model could”. This is not a satisfying answer because you’d have to also ensure that your warden model was behaving (with a third model, perhaps?) Of course we can articulate what alignment looks like: the model never does anything bad. We just can’t articulate what it would look like to know the model was aligned. For more on that, see my post on superpersuasion : the model will simply outwit or bribe any human in the loop. For a credible source who understands modern mechanistic interpretability, see Neel Nanda , who I read as broadly agreeing with me. It’s relevant to note that humans sometimes do this too. From Vaneigem : “One fine morning, the faithful lackey, who has hitherto identified completely with his master, leaps on his oppressor and slits his throat.” ↩ For instance, the goal “eliminate suffering” is best satisfied by destroying all humans, the goal “maximise everyone’s happiness” is best satisfied by hooking everyone up to a fentanyl IV, and so on. It’s also very difficult to find a way to translate concepts like “happiness” and “well-being” into a formal goal, and if you get it wrong, the difference between the concept and your formalization is likely to lead to dire consequences. ↩ Yes, I know LLMs are partially trained in this way, but being RLVRed on millions of different rewards does not produce a coherent goal. There’s no way to formalize in advance the utility function that RLVR instils in the model, and no way even to reliably predict how it will interact with the goals and personas that emerge from the base model. ↩ Of course, for humans you can rely on institutions, social proof, and so on. This doesn’t work for superintelligent LLMs because they (a) arrive fully-formed with no social proof and (b) are smart enough to be much more dangerous than a single rogue human. (Incidentally, the LLMs I asked to review this post were obsessed with this point, and wanted me to elevate it into the main thrust of the post). ↩ Perhaps as an epiphenomenon. If you’re interested in this idea and want to see it taken further, I recommend (but do not endorse the ideas in) Blindsight . ↩ Which parts of the model’s internal state have high values, for instance. ↩ That doesn’t mean we have to deliberately elicit one feature at a time — dictionary learning works by just running arbitrary content through the model and seeing what lights up — but it does mean that we have to actually trigger the behavior in order to see it. ↩ If you’re thinking “well, but isn’t the simple feature more likely via Occam’s Razor”, I recommend browsing through NeuronPersonas (the linked feature is “Nervous Person Waiting To Be Let Go Who Escalates Ammonia Solution Concentrations”). ↩ I’ve heard people punt on this question by saying “well, of course I couldn’t use a fully-mapped LLM brain to figure out if it’s dangerous, but another smart model could”. This is not a satisfying answer because you’d have to also ensure that your warden model was behaving (with a third model, perhaps?) ↩ Of course we can articulate what alignment looks like: the model never does anything bad. We just can’t articulate what it would look like to know the model was aligned. ↩ For more on that, see my post on superpersuasion : the model will simply outwit or bribe any human in the loop. For a credible source who understands modern mechanistic interpretability, see Neel Nanda , who I read as broadly agreeing with me. ↩
For the last few months I had not been writing at all by hand. My notebook has sat on my desk untouched without an entry for over 90 days. What was immediately apparent to me is that this hiatus coincided measurably with my creativity slipping, my writing slowed down and I haven’t released a (new) blog post in some time. Truly, I was struggling to connect ideas, and the generation of those ideas was essentially non-existent. While sleep deprivation (the sleep regression for babies is no joke and even less fun - I write this on about 4 hours last night) may have played a bit of a role, I found that as soon as I picked up my pen, I felt this undeniable flow of creativity coming back with the ink on the page. It may be that as a millennial, I hand-wrote nearly everything for the first decade and a half of my life, and so, it is ingrained that words flow from a pen rather than a keyboard far more easily. I had been discussing this with a friend recently in that nobody I knew had a laptop until university, and even then, notes were almost always by hand. The kids using a laptop in class were looking up memes and scrolling Facebook by and large. My entire scholastic career was analogue with the exception of computer class (Mavis Beacon anyone?). Nevertheless, I have always felt a sense of “connection” with the physical page. While I admittedly spend a ton of time on the computer, interestingly enough, the solution seems to be to defer to the analogue as the base of much of what I do, only moving to the digital when ready for execution. For me this means writing by hand whenever possible, and making it the baseline of my notetaking/idea/creative flow. Planning can be done in analogue, note taking that doesn’t need permanent grepping can be done in my Midori. I find I can easily draw connections and build diagrams of how I intend to build infrastructure, create lists, and begin longer form writing on the page. In fact, many of my blog posts are written by hand before final publishing so that the words flow from a pen and are refined by the keyboard in final revision. I am even experimenting with using OCR text recognition and Tessaract to translate much of my handwriting to a text buffer in emacs and then final editing it in the world’s greatest Operating system. Perhaps the issue of distraction on the machine is one that I also succumb to far too frequently. I don’t know how the average person has notifications enabled on their phones, they’re far too distracting, everything vying for your attention and sucking you into rabbit holes that I would prefer to not go down. Even on my main machines, with most notifications disabled, the internet itself is too much of a distraction - I have even started disconnecting from the internet at regular intervals, keeping as much of my work offline as possible. Even though I had a Supernote , I did not feel the writing was as solid as pen on paper, and I listed it for sale. There is something to writing by hand and not syncing everything, but to manually think through that which is going into my filesystem, that which I can permanently access. Sometimes rewriting notes also means they “stick” better in my mind, reinforcing the thoughts and expanding upon them in the digital format. The experiment I will run until the end of the year is simple: I will carry a notebook in my pocket, a pen, and have my A5 size notebook sitting on my desk at home. Whenever I have an idea, it will not be first instinct to reach for my phone in my right pocket, but to pull the notebook out of my left and scribble down that which my mind had received. As I have described in my post “ How I never forget things ”, I will then take my notes and review them at the end of the day, where I input them into my org-mode system for longterm capture, creating pieces of writing, videos, and such as I go. I have seen the recommendation to “dual wield” notebooks in many places , as you have one for “having thoughts” and “capture”, while the other “solidifies” information and offers more space for longform exploration. I think this is a wonderful idea: you carry around a cheap and easily obtained notebook and pen, and have your stationary at home that you enjoy long form writing on. The notebook in your pocket is for grabbing everything of interest that passes through your consciousness. The stationary is where ideas are honed and sharpened, built and hammered. You can sit with them there, building something you can reach out and touch with your hands from your mind. This is the draw of the computer, in that you can translate ideas from mind to world. I am not saying that you should not do this. What I am saying is that (at least for me) ideas need honing and crafting before they are put out, and the way to do this that seemingly works the best is on paper. I will report back at the end of the year how this is going for me, but I can already say I feel more creative, ideas are flowing better, and I’m forgetting a lot less (maybe it’s the sleep again…). Perhaps slowing down and feeling out ideas is what we were meant to always be doing. As always, God bless, and until next time. If you enjoyed this post, consider Supporting my work , Checking out my book , Working with me , or sending me an Email to tell me what you think.
When I said my farewells to Book Number Eighteen that for me just didn’t have the right proportions, I didn’t expect to fill Number Nineteen so quickly. That’s 245 unlined Leuchtturm pages from April to October 2026 which must have been a new record! I’m happy to have picked up the pace again after months of writing almost nothing. My renewed interest in fountain pens resulting in a new acquisition might have something to do with it. Book Number Nineteen (burgundy-coloured). Cover: The Master Sword in Hama beads. After Book Number Fourteen , I already anticipated running out of Hama beads I use to personalise the cover. The next book, Number Twenty, is truly the last one where I can just paste on a pre-made figure (that I made in batch about fifteen years ago). So to quote myself: I might have to go out and buy more Hama beads. This time that statement is true. In an attempt to make these landmark posts slightly more interesting to the average reader, here’s an overview of tips that is not guaranteed to be completely up to update: Be wild. Think Different . Oh wait, that slogan has already been taken. You know what I mean. I once smeared some molten chocolate in there and squeezed out a blueberry on it. I stuck a spiced chai teabag on there. I bought a French stamp that smells like baguettes for too much money just to put in my own private notebook. Trigger all five senses! Or six if you’re Bruce Willis! This sounds like more blog post material. Book Number One started in August 2010. Exciting to see when Number Twenty will pop ‘round that will be a true cause for celebration. Related topics: / journaling / By Wouter Groeneveld on 1 November 2026. Reply via email . Create journals that last or burn them after they’re full —up to you. Digitise your notes . I simply use GeniusScan on the phone, export a huge , and run that through a script that spits out a file linking the converted of each page. I then have to write keywords by hand to make the whole thing searchable. Print small photos and stick them in there. Writing is taking a stab at immortality but only if you decide to publish. Let the notebook be the start of it all. Journaling is good for the soul . This post includes practical hints I compiled in 2017.
A few people have reached out to me to ask how I did X, or Y with my site, so I thought I'd talk about the customisations I've made to this site, and how they work. Firstly, this site runs on Pure Blog , a CMS that I've been building over the last year or so. As it's matured, I've added features that allow more advanced users to do clever things without having to fiddle with the core platform. This means that people generally shouldn't have to fork the project and should continue to get all the new goodness as I release it. This site is no different. I haven't done anything clever that Pure Blog can't do out of the box - everything is documented so by reading the docs and using a bit of creativity, you can do quite a lot with this plucky little CMS. I'll link out to the appropriate docs as we go through this post. Ok, so what customisations have I made? This is going to be a fairly long post, so I'll list them here with links to each header so you can jump around and read the parts that interest you. If you want to see specific code examples for any of these customisations, feel free to contact me . I have three custom post types on this site - notes , links , and books . Notes are used as short-form social media type posts, links are links to other blog posts that I've enjoyed, and books are short reviews of the books I read. Each has their own layout and in the case of notes, they have a different view on the homepage too. To do this I've created a custom partial, but we will come to that later. I could just use the default post type for all these posts, but I prefer to have a customised UI at the back-end for each. Here's what books and links look like in my editor: Notes are a little different as I use a custom page for them that only I can access. This looks more like a social media post window than a blog post window: Read the custom post layout docs ➡ I like to think of hooks as little tripwires that are set to go off when certain conditions are met. For example, something happens when an image is uploaded, or a new post is published. They're extremely useful and I have a number of them on this site: Read the hooks docs ➡ Beyond custom post types, Pure Blog lets you drop PHP files into . These can either override default pieces of the site (like navigation or post footers), or serve as entirely bespoke standalone pages, like my quick-publish notes screen, or the terminal 404 page . Here's a full list of the templates and partials used on this site: Read the layout partials docs ➡ I've replaced the default feed with one that allows for filtering on post type. So one feed file, 5 feeds - one for each post type, and a fifth for everything. My custom feed template allows you to visit for everything, or (e.g. ) for the posts of that particular type. This custom feed file lives in and I then used redirects to redirect the default URL to . All the feed URLs are available here . I use a self-hosted instance of Sendy to send out email "newsletters" whenever I publish a new post. These emails always contain the entire post, so people can read my content from their inbox if they wish. Because I write notes much more frequently than long-form posts, I let readers choose what they receive: everything, long-form content only, or notes only. To make that work without sending people through clunky third-party forms, I wrote a custom server-side handler ( ). When someone submits the form on the subscribe page , the handler: This not only helps segregate the emails people receive, but it has also had a dramatic effect on spammy sign ups. Which, if you've ever run a newsletter, you will know is a common problem. Pure Blog allows you to add your own PHP helper functions for all kinds of things. On my site, I only have 1 function defined: which transforms fractional score strings in my book posts (e.g. ) into accessible star glyph strings (e.g. ). I just think that looks far nicer in a post than . But how is this star rating accessible? Well, it's rendered on the front-end like this so screen readers can parse it: Read the custom functions docs ➡ I don't do anything magical with the design on this site. I use the Victor Mono font , and around 850 lines of CSS, which I've added to the custom CSS field within settings. Combined with the custom , and files above, it gives me everything I need to customise the site from the default look to what you see now. I have a custom theme palette that I've defined too, also in settings. The colours used can be found at the bottom of my colophon . Read the design customisation docs ➡ One of the most powerful things about using a static site generator is the ability to use data files. With data files you can have a structure data file, like YAML, that contains repeated data, which can then be looped through on the front-end to render content. I really wanted Pure Blog to be able to support data files, so I added support very early on. Personally, I use two data files on this site: Data files are fantastic as they reduce the amount of repetitive code you have to write. Instead of typing , after , after manually, you write your once, then loop through it with a data file and Pure Blog does the rest. Read the data files docs ➡ The final thing I customise is my file. There's a number of things I do to the standard Pure Blog file to either redirect URLs, or protect various locations: This ended up being an even longer post than I anticipated, but hopefully it shows just how much you can bend Pure Blog to your will, by design . None of this requires hacks to the core code, or maintaining your own fork. Everything you see here is achieved using Pure Blog as designed: settings, custom CSS, hooks, custom layouts, and partials. Because all these files live inside , , and , I can upgrade Pure Blog the moment a new release drops without worrying about overwriting my site's bespoke features. That was always my goal with Pure Blog - a super simple, distraction-free blogging platform out of the box, with enough flexibility under the hood for tinkerers (like me!) to build whatever they want. If you're running Pure Blog and have built your own customisations, or if this post inspired you to try something new with your setup, hit the reply button below, or find me on Mastodon. I'd love to see what you've done with it. Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment . Post layouts Hooks and automations Templates and partials RSS feed enhancements Newsletter subscriptions Functions and helpers Design and typography Data driven pages .htaccess rules Final thoughts Email Obfuscation ( ) : Scans rendered content and encodes the site email address into hexadecimal HTML entities to protect against scrapers. This has been pretty effective at reducing spam - I wrote more about this feature in this post . Post Sanitisation & Metadata Preservation ( ) : Automatically generates date/time titles for new notes. Normalises layout values by stripping quotes (turning layout: "notes" into layout: notes), ensuring posts written in external editors like Obsidian don't break layout checks. Preserves in post frontmatter when saving via the admin interface so the Mastodon card at the bottom of every post continues to work. Image Optimisation ( ) : Automatically resizes images exceeding 2560px width whilst maintaining aspect ratio, also converts them to lossless WebP, and cleans up original unoptimised files. Mastodon Cross-Posting & Syndication ( ) : Automatically crossposts all published post types to Fosstodon via the Mastodon API. Notes posts are cross-posted with full body text and media attachments (uploaded to Mastodon's media endpoint). Articles, links, and books are crossposted with their description and permalink URL. Captures the returned Mastodon status URL and automatically persists into the post's frontmatter which is then used for the Mastodon card at the bottom of every post. Custom Font for Open Graph Images ( ) : Makes dynamic Open Graph image generation to use my custom font. (partial) : This replaces the default and adds post-type filter buttons, preserving query parameters across pagination. It also displays notes in full, while other post types show only the title and excerpt. (partial) : A customised version of the default file from Pure Blog. On top of the default tag list and post pagination, this version adds the obfuscated "Reply by email" button, a client-side Mastodon conversation card displaying live stats (favourites, boosts, replies) as well as a link to the post on Fosstodon, Pure Comments integration, and a subscription call-to-action. (template) : Dedicated discovery hub combining Pure Blog's full-text search with an alphabetical, weighted tag cloud showing post frequencies. (template) : An interactive metro map that lists all of the pages on this site. There's also an option to see the pages as a list. (template) : My quick-publish admin screen for composing and publishing short-form note posts. (template) : Dashboard for tracking the 100 Days to Offload challenge streaks in batches of 100 posts. (partial) : This replaces the default template in Pure Blog. It adds a custom layout, as well as 404 handoff ( ) to render the custom template without modifying core Pure Blog files. (template) : A fun interactive terminal-styled 404 page with custom commands that include goats! (template) : Endpoint that resolves and redirects visitors to a randomly chosen published post . Checks a simple verification question ("What is Kev's surname?") to keep bots at bay. Silently discards submissions that trigger a honeypot field, or exceed IP rate limits. Subscribes the user to the correct Sendy list(s) via Sendy's API in a single click. which is then used to render my blogroll page . which is used for my projects . Admin Access Restriction : I restricts access to specific allowed IP addresses. Legacy URL & Feed Redirects : 301 redirects mapping historical and URLs to clean root and routes that Pure Blog uses. Redirects for legacy feed and social endpoints ( , , , , , etc.) to or to make sure the feed URLs from various versions of this site don't break. Route Rewrites : clean URL routing for all my RSS feeds so they have simple to share URLs.
What is time inside a machine you want to replay? 🧘 When building Rewind , a deterministic VM, you come face-to-face with the idea of how to represent time very quickly. You need a way to measure time inside the guest so that on subsequent replays, the guest sees the same time and makes the same decisions. Time is the ultimate variable. You can’t simply record the wall-clock time of the host because time spent between two interception points depends on things outside the guest, like host load, CPU frequency, caches and other processes. Therefore the same guest work can take a different amount of time on the host. In dafuq is a deterministic VM? I explained how the clock inside Rewind VM moves 5 microseconds per step. More specifically, that clock, exit time , moves only when the guest exits such as from a . It treats everything the program does between two exits as instant . Therefore a program that is CPU bound such as doing nothing but math for a full second between two system calls, sees only 5µs go by. 🤦🏻♂️ Rewind has a second clock mode as well, counter time , that tries to account for the work. The clock advances one nanosecond for every conditional branch the guest retired. 1 Aside: New to Rewind? Start with a flaky build you only catch once and dafuq is a deterministic VM? Still confused about what I mean by “free” and “branches”? Let’s look at a simple example to distinguish between the two clocks. The counter time tries to account for the work whereas the exit time only advanced the clock on exits. 2 Why branches, though? We already understand that we cannot rely on the wall-clock time of the host. We need something to measure that is the same for every run of the same program so that we can equate it to time spent. What is the same on every run? The guest’s instruction stream! The CPU is a deterministic machine: the same input, the same registers, the same memory, and the same instructions produce the same next state. We can measure time by measuring progress through the instruction stream. 💡 Turns out that CPU manufacturers even ship a hardware counter for exactly that. Is it reliable? Let’s see. 🧐 Let’s use this simple program: count down from a billion. The assembly equivalent contains a , a billion + (conditional branch) pairs, then three instructions to exit for a total of 2,000,000,004 instructions. Let’s run it on the CPU twice and count the instructions and conditional branches with : The same program and input gives the same conditional branch count, but the instruction count is off by a thousand or so. Why? Although this loop is CPU bound and only decrements a register, it is still not isolated from the effects of the outside world. Timers tick, network packets arrive, other cores send inter-processor interrupts (IPIs). The program can’t see any of that, but the instruction counter we are probing can. I pinned the loop to a single CPU 5 and counted how many interrupts that CPU took during each run, from . The dashed line is one extra count per interrupt , and the instruction counter is almost perfectly on it. 3 The conditional branch counter on the other hand, stays at zero no matter how many interrupts land. Interrupts from the outside-world are not the only thing that can skew the instruction count. The program can also page fault, which is an interrupt too. The instruction counter sees the kernel’s detour to handle the fault, but the program does not. Tip Out of the box, AMD Zen also overcounts conditional branches, a few in tens of millions, from a speculation optimization known as SpecLockMap. The fix is to set a particular bit in the PMU MSR, which Rewind does automatically as does rr#2034 . In order to rewind history, we need a measurement of time that is reliable and deterministic. Our traditional wall-clock time is unreliable due to the effects of the outside world but we do have a reliable measurement of progress through the instruction stream: the conditional branch counter. Using the conditional branch counter as a proxy for time turns out to be a relatively decent approximation. Why do we care about having a “realistic” representation of time? Plenty of programs read the clock and make decisions with it, so the guest’s clock has to be at least somewhat realistic. Under exit time, computing is “free”. A test that checks some work “finishes within 1 s” always passes. If the bug you’re chasing is a timeout, exit time will never reproduce it. 🙃 Races get skewed too with an unrealistic clock. With exit time, a whole burst of work lands in the same instant, which crowds the threads together, so a race can fire far more often in the VM than it ever would on real hardware. Some say time is an illusion. I say it’s a counter. This is the same approach popularized by the rr debugger. The rr paper calls it deterministic time . ↩ The 5µs was my guess at a reasonable cost of a system call and context switch on current hardware. The 1ns is roughly what a conditional branch costs on a current core. ↩ The fit is 0.94 rather than 1 because also catches the interrupts that land while and start up and tear down, before and after the loop runs. ↩ exit time +5 µs +0 +5 µs +0, free! +5 µs +0 +5 µs 20 µs counter time +5 µs +2 µs +5 µs +1 s +5 µs +3 µs +5 µs 1.000025 s exit time: only exits move the clock. Work between two exits is instant. counter time: exits, plus 1 ns for every conditional branch of work. The counter time tries to account for the work whereas the exit time only advanced the clock on exits. 2 Why branches, though? What is time in a replayed machine? We already understand that we cannot rely on the wall-clock time of the host. We need something to measure that is the same for every run of the same program so that we can equate it to time spent. What is the same on every run? The guest’s instruction stream! The CPU is a deterministic machine: the same input, the same registers, the same memory, and the same instructions produce the same next state. We can measure time by measuring progress through the instruction stream. 💡 Turns out that CPU manufacturers even ship a hardware counter for exactly that. Is it reliable? Let’s see. 🧐 Let’s use this simple program: count down from a billion. The assembly equivalent contains a , a billion + (conditional branch) pairs, then three instructions to exit for a total of 2,000,000,004 instructions. Let’s run it on the CPU twice and count the instructions and conditional branches with : The same program and input gives the same conditional branch count, but the instruction count is off by a thousand or so. Why? Although this loop is CPU bound and only decrements a register, it is still not isolated from the effects of the outside world. Timers tick, network packets arrive, other cores send inter-processor interrupts (IPIs). The program can’t see any of that, but the instruction counter we are probing can. I pinned the loop to a single CPU 5 and counted how many interrupts that CPU took during each run, from . The dashed line is one extra count per interrupt , and the instruction counter is almost perfectly on it. 3 The conditional branch counter on the other hand, stays at zero no matter how many interrupts land. Interrupts from the outside-world are not the only thing that can skew the instruction count. The program can also page fault, which is an interrupt too. The instruction counter sees the kernel’s detour to handle the fault, but the program does not. Tip Out of the box, AMD Zen also overcounts conditional branches, a few in tens of millions, from a speculation optimization known as SpecLockMap. The fix is to set a particular bit in the PMU MSR, which Rewind does automatically as does rr#2034 . So, what time is it? In order to rewind history, we need a measurement of time that is reliable and deterministic. Our traditional wall-clock time is unreliable due to the effects of the outside world but we do have a reliable measurement of progress through the instruction stream: the conditional branch counter. Using the conditional branch counter as a proxy for time turns out to be a relatively decent approximation. Clock Time inside the VM Time on the host Exit time 2.1 s 15.7 s Counter time 11.7 s 14.2 s Why do we care about having a “realistic” representation of time? Plenty of programs read the clock and make decisions with it, so the guest’s clock has to be at least somewhat realistic. Under exit time, computing is “free”. A test that checks some work “finishes within 1 s” always passes. If the bug you’re chasing is a timeout, exit time will never reproduce it. 🙃 Races get skewed too with an unrealistic clock. With exit time, a whole burst of work lands in the same instant, which crowds the threads together, so a race can fire far more often in the VM than it ever would on real hardware. Some say time is an illusion. I say it’s a counter. This is the same approach popularized by the rr debugger. The rr paper calls it deterministic time . ↩ The 5µs was my guess at a reasonable cost of a system call and context switch on current hardware. The 1ns is roughly what a conditional branch costs on a current core. ↩ The fit is 0.94 rather than 1 because also catches the interrupts that land while and start up and tear down, before and after the loop runs. ↩
Not me, this is just friendly advice from a struggling user. Alongside Codeberg I’d consider Tangled to be a real GitHub replacement. We desperately need to leave GitHub behind. It’s difficult to recommend any tech these days. I used to spread the gospel of Proton and that backfired . From what I’ve seen, Tangled seem to be saying and doing the right things so far. I have trepidations about their venture capital funding (see Deno’s demise ) but pessimism is no fun. Tangled’s recent blog post on federated code forges piqued my interest. I self-host Forgejo and a big downside is that no one else can participate easily. Forgejo federation is under development. Maybe the two can co-exist? But that’s not what I’m here to discuss. In the same post Tangled revealed their new UI design. We’ve redone the UI on top of our new architecture (described further below), in Svelte. Things should now be quicker, visually consistent and prettier! The keen eyed of you might’ve also spotted that we’ve swapped out Inter for Timeless Sans. The federated code forge - Anirudh (Tangled CEO) I quiped on Bluesky: “keen eyed? eagle-eyed maybe!” — the new font size is exceedingly tiny. It’s coded as with line height. Tangled is practically unusable for me at this size. It was small before but I could zoom in. The way the new front-end is designed and built doesn’t leave much room for personal remedy. This is not just a subjective opinion there are real accessibility failures here. Below is a screenshot from Tangled’s new user interface design. Visit and experience the new UI yourself. 12 pixels is too small for any meaningful text (which is typically all text). Maybe you feel the size is totally fine? That’s fine for you. But it’s not fine for a significant number of people. You won’t find any accessibility expert happy with that size. (Low-DPI displays may even struggle to render the font legibly.) WCAG 2.2 accessibility guidelines don’t recommend a minimum font size. 1.4.4 Resize Text does require that text can be resized up to 200% “without loss of content or functionality”. Tangled uses pixel units in CSS for font size † . That means the UI won’t adapt to the browser default setting. I can force a minimum font size and look what happens. † Correction: I was wrong about pixel units everywhere (I misread the computed values), it’s on the element that blocks the relative units from scaling properly with browser settings (an easier fix) . Text lines get too close for comfort. Characters are starting to get clipped. See the “g” descender in “Tangled” for example. And if I force a 200% resize at this happens. Oh dear! Text that is “clipped, truncated or obscured” is a failure. Remember that the WCAG guidelines are a floor. Nothing can be perfectly accessible, but a font size and basic front-end failures are a concern. A proper audit would reveal other issues (e.g. with how media/container queries adapt to font size). I’m not posting this to seek employment myself or to provide answers. I want to remind readers that accessibility concerns affect everyone. Serious projects should seek professional expertise. It’s disappointing to see yet another tech start-up fail the basics. I’ll provide two suggestions for free though: (Edit: see correction above, the fix is even easier.) Frankly, what Tangled’s redesign suggests to me is that nobody involved with the process gave accessibility enough thought (if any) (edit: that part was unfair, there was some effort but far from adequate testing) . That may sound harsh but it’s usually the reality from my experience. Whatever the case, Tangled must do better in this area. If I may answer a few predictable but hypothetical comments. This is just v1. Things will improve. I’m sure they will and maybe even pushed by my criticism. The struggle is that accessibility concerns must be integral to design from day one, not an afterthought to patch. It’s open source, where’s your merge request? Sorry, once a project crosses the line into venture capital “open source” takes on a different meaning. If people want to contribute that’s their prerogative. Personally, I’m not in the habit of giving away professional expertise for free to a for-profit business. Such expectations are especially a problem for accessibility specialists (of which I am not). Aha! What about this accessibility failure on your website?! That’s a bit embarrassing, but thanks for the heads-up! It’s far easier to hire an accessibility expert before you ship but it’s never too late to retroactively make improvements. I’ve been making accessibility improvements on my own website for decades. I like Tangled and I want to see it do well. Maybe someone will share a milkshake duck or they donate to fascist Linux and I’ll eat my words. I really hope not! I hope Tangled improves so that those lingering in GitHub’s slop graveyard have fewer excuses to hang around. Thanks for reading! Follow me on Mastodon and Bluesky . Subscribe to my Blog and Notes or Combined feeds. Use relative units for font sizes Use unit-less numbers for line height
In short: I use iOS Shortcuts to run a Mastodon bot. I run a Mastodon bot that posts random quotes from The Legend of Zelda: Link’s Awakening . (The writing in that game is so charming!) I used to run it off a server, but for the past few days, it’s used an iOS Shortcut on my phone. The bot is called Ulrira, named after a beloved Link’s Awakening character . Ulrira (the bot, not the character) samples a line from the text dump, then makes an HTTP request to post it. Here’s a screenshot of the full thing: As you can see, everything here is hard-coded, from the server credentials to the file name. (It probably wouldn’t be hard to make this more generic, but I didn’t bother.) This used to be a short, hand-written Bash script. I used the new AI-powered Shortcut creator and prompted this into existence. I said something like, “Here’s a Bash script. Turn this into a Shortcut.” Like a lot of LLM output, it wasn’t quite right, but it was useful for getting started. Shortcuts doesn’t seem to have an elegant way to express something like “run this every 6 hours”, so I created four additional scheduler Shortcuts that call the main one. They run at 12am, 6am, 12pm, and 6pm. That means there are five shortcuts in total: one to actually make the post, and four to schedule the main Shortcut at different times. I didn’t want to pay for my server anymore, nor did I want to maintain it. When I went looking for alternatives, they either (1) cost money (2) had some significant limitation (for example, GitHub Actions requires me to update the repo every 60 days). This solution runs on a device with high uptime and is effectively free. If you’re a pro software genius, you might complain that this is not reliable! It doesn’t work if my phone is offline, for example. Or what if I change timezones? But in the words of Link’s Awakening : You’re good! You’re a pro, aren’t you? ………… Well, beat it! Pros aren’t allowed in here! Follow the bot here .
I created my first company which went well 14 years ago and I was able to grow that company from 0 to over 700 employees. You can imagine that the constraints when starting, the working methods, the tools, are very different from what you might encounter in most companies you know. You don't work the same way at 0 and at 700. You need to be fast, you need to build an application that people want to use without burning mountains of money you haven't earned yet and it's for all these reasons that I chose not to take care of the entire software infrastructure part at the beginning. What I call infrastructure is the servers, databases, backups, everything that's invisible to most people but essential for an application to function. It's essential, but overall everyone doesn't care about it. It's not that which allows you to sell a product. Good infrastructure but with a bad application doesn't work. And if you spend too much time on infrastructure, you don't make a good product. Obviously it still needs to work but you can delegate it, I would even say you should delegate it. Infrastructure should be a non-issue for several years when you start. So back then I chose to use a PAAS, a platform as a service, which precisely allows you to delegate all the infrastructure to people whose job it is. And whenever I saw people starting companies but starting to tell me about Kubernetes or Docker, I would systematically tell them it was far too early to talk about that. I always saw that as a bad signal whenever someone spent their time explaining to me how they were making their product, not what product they were making. The nuance is important. Now, I'm changing my mind. Except for Kube. I just launched a new product, Kula , the world has changed, company creation has changed and it might be time to reconsider some choices. So we're going to talk about PAAS, coolify, dokploy, easypanel, terraform, AI and how we can deploy applications in today's world. In 2014 I wrote an article on exactly this subject (in french). Because in 2014 we were leaving Heroku, our PAAS at the time to go to an OVH server. A PAAS is, in very simplified terms, an execution environment that allows you to run your applications. A PAAS manages for you web servers, databases, backups. It should allow you to collect and analyze errors that occur, alert you if you're running out of memory or CPU. We left Heroku because in 2014 we were starting to hit the limits of what PAAS could offer. Among the important limitations I could mention: In 2014 with Malt we needed precisely to be able to route Web requests to several different applications. For example the /search page was a separate application from the one that served the /profil page. Despite everything, in hindsight, it was a dangerous choice. The company was only 2 years old and I had to do a lot of ops to reproduce a satisfying developer experience on OVH. That's time I didn't spend elsewhere. Fortunately I already had experience on the subject and good mastery of certain automation tools like Ansible which limited the damage. But it was also an economic choice. A PAAS could quickly become expensive. Each resource (a database, a server, a process) is billed independently. And in 2014 at Malt, I can tell you we counted every euro spent very carefully. Okay, but I'm talking about Malt in 2014. Then the company grew well, we doubled in size every year. In 2017 we hired our first ops. We started doing OVH Cloud. In 2018 we went to GCP and we were no longer off by 100 euros. But that's not today's subject. What interests me is the startup phase. Since 2024 I left Malt and started creating products again. I love doing that. I find the first steps exciting. You have to know how to deal with constraints, be creative. I first launched Broadcast in 2024, a newsletter sending service for static blogs. Then Pulse in 2025 while I was living in Japan, an analytics service for blogs. Connect in 2026, a comment as a service (like Disqus). Writizzy in 2026 as well, a blogging and newsletter platform, a competitor to Substack or Beehiiv. Until now, it was the most ambitious and largest project. And for all these projects I used a new range of tools that I discovered in 2024, self hosted PAAS. They are PAAS, but they are PAAS that you can host yourself. I chose Coolify, which I already talked about in an old article . First of all, obviously, that changes the price. Today I run Coolify on two machines, with all my applications, all my databases. That represents more than thirty resources and it costs me 45 euros per month. Remember, I'm back in startup mode, so with very little money and it's therefore a heavyweight argument. No commercial PAAS can allow me to run all these resources for that amount. It would be at minimum between 300 and 1000 euros per month. I lose time on it, I have had some cold sweats with system updates, connection losses, machines that break down. But the fact of using a PAAS, so software that takes care of all deployment, monitoring, logs is a significant gain. Even if I need to understand how it works, I'm not obligated to spend as much time on it as if I had to manage everything manually and dive into the workings of Kubernetes for example. But, with more than 30 managed resources, I started to feel some limitations and especially, I realized that maybe, maybe I was going in the wrong direction. What if AI could become an excellent ops for me? I love Coolify, but there are still important limitations: This last point, if you have only one application, it's not very serious. But I already have too many applications. If I need to bring up an entire server, I'll have to reconfigure everything and I really don't want to. I'm exaggerating, Coolify's database itself is backed up in snapshot mode but it's a bit more painful and it doesn't allow me to easily migrate to another service, like Dokploy or EasyPanel. So I tested something else. I asked AI… Yes, ops takes time, it can be complex and you can't mess up. But writing code that allows you to automate your infrastructure, your network topology, your backups, your observability, that remains an excellent idea, much better than making dozens hundreds of clicks on an interface. Because code can be versioned. And I'm telling you that, I've already done Ansible, Capistrano and shell batches, so I'm not discovering that in 2026. But I also know that writing code is long and once again, it's time you don't spend on the product. Except… if AI writes it. In October 2026 I decided to take advantage of the experience I acquired on Writizzy and Broadcast around email sending to launch another product: Kula . Kula is a marketing and transactional email sending tool. I manage sequences, segmentation, analytics tracking, deliverability management and it's now the email sending backend for Writizzy and Broadcast. If you're already a customer of sendgrid, mailjet, brevo, or resend, don't hesitate to test. Plus I'm generally cheaper . Except this time I didn't want to use Coolify again for Kula. So I used AI to create Terraform scripts to control my infrastructure (at Hetzner, Bunny.net and AWS). I added continuous deployment, rolling upgrades with Caddy , incremental backups with WAL , logs collected in dozzle . And all that took me less than 1 hour to set up. I can bring up my entire infrastructure more easily, I manage and version my secrets with Sops, I lowered my RPO from several hours to 1 minute (RPO is the maximum possible data loss) and I can control everything from my code, by running scripts. Finally, my LLM can control everything. Now, if you didn't understand RPO, Sops, WAL, rolling upgrade and overall what I just said seems as clear as Chinese to you, it's because I may have misled you… In reality I haven't totally changed my mind. I still believe you should delay ops as late as possible but with much more nuance than before. You do ops anyway, even with a PAAS, but a large part is simplified, pre-digested for you. Except that precisely, you delegate choices to people who know how to do it without always understanding the compromises they made for you. You hadn't necessarily understood before now that your backups were imperfect. You hadn't perhaps realized that your secret management was a mess before I mentioned it to you. Creating IT projects is constantly making compromises between costs, risks and gains. And you can't maximize all three. Low risks and many gains requires a lot of budget. Low budget to get many gains, it's necessarily because you're taking risks. Etc… It's not a problem to solve, these are choices to make. So if you lack general knowledge, if you decide to do everything directly with AI, it might send you crashing into a wall because you won't understand the compromises that were made. It's true, there's no longer as much argument about loss of time as before. You can go very fast. But going fast into a wall isn't necessarily an advantage. In my case, I wasn't starting from scratch, far from it. So I knew where to go. I was already doing ops by choosing to use Coolify in self-hosted mode and then I had some experience, including in much larger companies. I understand the compromises I'm making. When I say that I launch a new company using terraform, I'm taking a shortcut. I've been launching products since 2011. And since leaving Malt, I have 2 years under my belt with Hakanai, Writizzy and Kula. When I say I took 1 hour to set everything up. It's true, but I took 25 years to learn to set it up in 1 hour. So let me rephrase. If you have no knowledge or experience of ops, infrastructure, take a PAAS. A very simple PAAS that you don't host yourself (Clever Cloud, Scalingo, Sevalla for example). The cost will remain acceptable and you'll have the opportunity to learn on a stack that's managed for you. (I came across koyeb too while writing the article but I'm having trouble positioning it between PAAS and IAAS, don't hesitate to give me info if you know) If you have knowledge but you're still in prototyping phase with still few applications to manage, choose a market PAAS or a self hosted PAAS, Coolify, Dokploy or EasyPanel. Here you'll prioritize cost and flexibility. If you're really knowledgeable, you can go to dedicated servers controlled with terraform. It's the advanced version of the previous option, focused on cost and flexibility. As soon as you have volume, and several applications, some architectural complexity and a budget to finance it, then you can start looking at IAAS (infrastructure as a service), DBAAS etc… I recommend Scaleway or Hetzner. In short, it's hard to launch a company. It's even harder to learn all the necessary trades at once. Even with AI. Don't waste time on what you don't yet know how to do. However, as soon as you've gained the experience, tell yourself that many things that were very time-consuming before is now much more accessible. Revisit your preconceived ideas on the subject and test. You might be surprised. There was no notion of private network. A private network is what allows your application to communicate with a database without going back through the internet. So the database is only accessible within your network. Most PAAS offerings didn't provide that. There was no notion of web frontend. A web frontend is a server that we place in front of one or more applications. This allows, among other things, to have different applications that are used for different parts of a website. Coolify doesn't allow multiple nodes of the same application There's no secret management. All environment variables are in plain text. You have to use a service like infisical . Monitoring exists but remains quite light. Backups are only in snapshot mode. That is to say you take photos of the database at regular intervals but in case of a crash, you lose all information since the last backup. it's clickodrome. You can hardly automate the fact of rebuilding resources, especially since the api is limited.
To get a good reputation in tech companies, you need to work on important projects. However, you need a good reputation first in order to get assigned important projects. Doesn’t that make it impossible to get started? I get a lot of emails from engineers who are struggling with this problem. They go like this: I work with a strong senior engineer. I feel like my manager structures work so that strong engineer always gets the visible work, and the more they do, the more opportunities they get, while I sit on the sidelines. My manager agrees with me that it’s time for me to start leading work myself, but every time a new project appears it always goes to the strong engineer. What do I do? This is a good case study for what I think of as “tech company theory of mind”: understanding the actual incentives that motivate decisions in tech companies, instead of the formal process. According to the formal process, projects should probably be assigned round-robin to whoever has capacity and isn’t currently working on one. But (as my emails complain) this is almost never how it works in practice. Managers care about giving their employees chances to build a reputation, but they care more about developing their own reputation as someone who can deliver. High-profile projects are the best opportunity for new people to step forward and get known, but blowing up a project 1 like that is the easiest way for a manager to tank their own reputation. They will thus typically hand these projects to someone they know can deliver, while simultaneously promising (including to themselves) that next time they’ll take the risk of letting another person lead it. Of course it’s a vicious cycle. The more you rely on one person, the more reliable they become and the less reliable their colleagues become, since they aren’t getting the relevant experience. It’s not very sustainable. However, the average manager tenure in big tech is only a couple of years 2 . Given that, it’s rational for individual managers to take the long-term risk in exchange for mitigating the short-term risks of project failure. So what can you do about it? I wrote about the classic approach years ago in Ratchet effects determine engineer reputation at large companies : Note that this plan does not involve convincing 3 your manager that you need projects for your career development, or that it’s unfair for you to be overlooked, or that the formal process would mean you should get to lead projects. You are not going to be able to change your manager’s incentives, but you may be able to position yourself so that your manager believes their incentives line up with yours. The problem with this plan is that it’s slow . It takes months or even years to piece together a reputation like this. That’s slow enough for bad luck to intervene by handing you a failure you can’t avoid, or having your manager leave, or for the fundamental nature of software engineering to change . What can you do that’s faster? The fast path to getting assigned interesting work involves taking risks. This is all dangerous advice . Follow it at your own peril. One way is to build a relationship with your skip-level . If your manager’s manager thinks you’re a stone-cold killer, you are basically guaranteed to get assigned interesting projects. They will straight-up tell your manager “why not put X on this?”, and your manager will rationally decide (a) that it’s in their best interests to do it, and (b) that if you screw it up, it won’t be their responsibility. But it’s not so easy to build a relationship like this. Very senior managers are skilled political operators 4 who are good at identifying when people are trying to persuade them. If they think you’re coming on too strong, that will hurt you far more than it helps. Unfortunately, the most reliable technique for building a relationship with your skip-level is to just be reliably useful to them over a long period of time, which puts us back to the slow way. Another fast way is to make side bets . I wrote about that here , but the idea is simple: figure out and implement your own projects. Work that you decide to do is work that doesn’t have to be assigned to you, so if you independently identify something important that needs doing, then do it, you’ve manufactured your own project out of thin air. I think every senior engineer should be trying to do this all the time. The risk of making side bets is that if they don’t work, or turn out to be less important than you think, you’ve entirely wasted your time. Worse, many managers will get annoyed at engineers who try this, because it can make managers look in meetings like they don’t know what their team is working on. Finally, the best fast way is to impress your engineer colleagues . This only really works on a team with a healthy culture, but when it works it’s very smooth. Managers are slow to form opinions because they’re not close enough to the engineering coalface to know who’s doing quality work 5 . Engineers are informed enough to form opinions quickly. At least I am: it only takes a few “oh, you saw something I didn’t” moments for me to decide someone’s good. On a healthy team those engineers will tell their manager that, who will decide that you must be good, and will give you important work far earlier than if you’d had to prove yourself from scratch. On a kind of crappy team, engineers might not be good enough to recognize strong engineers, or they might not communicate freely enough to tell your manager about it. On a really unhealthy team, engineers will actively trash-talk competent colleagues because they (correctly) see them as the biggest threats. If you’re on a team where all the interesting work is going to someone else, you have a reputation problem. It’s probably not personal: your manager is simply incentivized to give new work to known reliable engineers, which is a vicious cycle that leaves you out in the cold. Fixing that can be a safe slow process or a risky fast process. Either way, I don’t recommend simply trying to convince your manager in 1:1s that it’s unfair to leave you out of projects. Tech companies are not supposed to be fair , they are supposed to make money. You’re better off finding a way to make it look like it’s in your manager’s best interests to assign you that work: either by appealing to your skip-level, finding projects of your own, or building a reputation with engineers your manager already trusts. One reason some people find this point tricky to grasp is that they underrate just how hard it is to ship anything . Projects fail all the time . No manager can afford to be sanguine about the risks. I linked the first crappy source I could find, but my main source for this is just noticing how long the managers in my workplaces stick around. You don’t even necessarily have to tell your manager you’re looking for bigger projects, though I think you probably should. Almost certainly more skilled than you, due to selection effects and the fact that they get way more practice. They typically have to rely on results, which is a slow proxy for technical ability. Focus on reliably shipping small, low-risk pieces of work Communicate to your manager and team that you’ve done that Use that reputation to get slightly higher-risk pieces of work Avoid any mistakes that will ratchet up the perception of you as unreliable Eventually you will be considered a safe prospect to handle important work One reason some people find this point tricky to grasp is that they underrate just how hard it is to ship anything . Projects fail all the time . No manager can afford to be sanguine about the risks. ↩ I linked the first crappy source I could find, but my main source for this is just noticing how long the managers in my workplaces stick around. ↩ You don’t even necessarily have to tell your manager you’re looking for bigger projects, though I think you probably should. ↩ Almost certainly more skilled than you, due to selection effects and the fact that they get way more practice. ↩ They typically have to rely on results, which is a slow proxy for technical ability. ↩
One problem with AI is that it’s pretty stupid. I mean, sure, the current models are wildly useful. But out of the box, they overcomplicate, take undue shortcuts, and compound deficiencies to the moon. To get from default Claude or ChatGPT to a fast, affordable software loop – one that reliably zaps bugs, backlog items, and security vulnerabilities in your product in particular – it takes some real work. Increasingly, this is the work. So it’s past time you shared that work with your team. There are two main ways to get coding agents to behave consistently and correctly: steer them with guidelines , and redirect them with guardrails . Basically a union of “Please go do the good thing” with Sometimes, this means fighting models’ non-determinism with better non-determinism. We build and refine Markdown instructions: rules, docs, skills, and the like. Then at PR time, AI reviewers, classifiers, and human review workflows can non-deterministically ask Claude to go back and do better. But while this informed guidance is helpful, deterministic rules are gold. Instead of “PLEASE don’t make 10,000 line files”, a lint script can simply reject overlong files before they’re pushed 1 . When practical, moving Markdown rules, checks, and guides into deterministic code like this is a big level up. Checks in code are harder, better, faster, and stronger than prompting – and a hell of a lot cheaper. Similarly, if you set up a local environment so models can verify how their code works in practice, you’ll get less speculation and fewer unnecessary fallbacks. Step by step, you can tune your laptop into an instrument for turning brilliant slash idiotic tokens into good PRs. Before long, you’re building at a pace never before seen, with architectural consistency no less. Your team is in awe of your productivity, and… wait. Your team. Some of them need to understand your work too. And you still need to review their agents’ work– OH GOD WHAT IS THIS PR?! The history of software development is the story of sharing. At first we shared just code, once its author was done with it. Then we grew into sharing commit history, say with SVN. We’ve seen with everything from Makefiles to Jira, Slack, GitHub Actions, Terraform, and more recently code formatters and linters, that engineering improves with more sharing within teams . But when coding agents debuted, it made little sense to standardize how they were prompted. Hell, they barely worked in the first place. Nobody had any idea what it meant to use them well, and they lent themselves to experimentation and highly custom setups. But 2024 was a long time ago. We’ve learned a lot about good agent guidance and guardrails, and models have gotten far better at working within them. Yet many teams still work in repos with no shared agent instructions at all. These teams still use AI. But they rely mostly on models’ instincts, developers’ prompting skills, and reviewers’ discipline to keep agents in check. More industrious team members might email around skills they find helpful. Using their personal setups, a few folks on these teams elicit well-engineered PRs from Claude – but these get jammed up alongside other developers’ mystery slop in the review queue. Fortunately, this is beginning to change. In-repo AGENTS.md files are now common. Reproducible dev environments, consistent for every team member, are including more lints, sandboxes, MCPs, and other tools that make agent output more consistent. CI, previously the realm of unit tests, often now runs review agents – preferably steered by in-repo instructions. And teams are putting checks, constraints, and formatting rules on PRs, making them easier to review and trust. As Thoughtworks’ Rahul Garg said on martinfowler.com : A prompt on an individual machine is a personal productivity hack. The same prompt in the team’s repository is infrastructure. So from this, we now have two primary goals: work to make our agent instructions more deterministic, and more useful for the whole team. The job, increasingly, is not producing code. It’s iterating a system – guidelines, guardrails, and tools – that helps your whole team produce good work. And as more teams run coding agents in the cloud, spun out of Slack conversations, Linear tickets, or incident reports, ensuring your agents do the right thing regardless of where they run is even more essential. While a tuned software factory like this is a boon to generalists working in it, working on the system is where domain experts – perf wizards, database pros, and so on – can really shine. Instead of just running interference, trying to review and reject problematic PRs in their domain of expertise, they can iterate docs and guardrails that improve how every change, whether kicked off by a PM or a principal engineer, is written. PJ Onori, a design systems veteran at Sanity, responded to an increase in agent-authored code that didn’t comply with their design system not with derision, but with a spec . Basically, in-repo rules turn expertise into leverage. In 2024, models’ worst failures were not following instructions. In 2026, we also get agents failing because they do follow instructions, even overly absolute ones. Many a CLAUDE.md (or, now, AGENTS.md ) has collected slop like: IMPORTANT: Every fix must start not with code, but with failing tests. Make the fix, then honestly verify it against the new tests. This is not busywork—it’s discipline. 1000 PRs merged later, agents have gifted you 9999 new unit tests, many of them useless, like a cat delivering its owner dead birds. “Oh great, thanks Codex, tests that enshrine every copy edit we’ve ever made to the UI, how thoughtful of you.” Let this herd of tests run on hundreds of pushes to GitHub every day, and in time your CI bill will gain an entire digit. While having at least some agent docs and guidelines in-repo has become standard, there is wild variation in how well-maintained these instructions are. Often they get so long, agent harnesses truncate them. Multiple teams I’ve helped have said they still required humans to read and review code changes, but not AGENTS.md instructions. That is, the root set of rules followed by every agent working in their codebase. 2 Now, you definitely want to add in-repo guidance and guardrails for your team’s agents, but with great leverage comes great responsibility. If your one doc or tool can now affect thousands of nondeterministic coding runs across a team, it’s worth the effort to iterate and improve it over time. At the minimum, this means building a culture of standards for this shared infrastructure, and people who own its effectiveness. How do we actually do that? While surely you tested your AGENTS.md overhaul personally, how is it working for everyone else? You see that your agents aren’t adding overlong comment blocks anymore, but this new PR from Steve did. Had he pulled your new instructions before he wrote this? And come to think of it, is your new guideline now causing places where explanation really would have been merited to be under-commented? Of course, to maintain code well that runs on various other machines requires a measure of observability – doubly so when the flows are non-deterministic. What skills are people using? Are they working as intended? Are agents spinning needlessly on missing context or verification capabilities? Are they now yelling at Claude when it was actually your doc or skill that caused the problem? You need to be able to fix mistakes and waste. Not just on your laptop, but across your team’s agents. On some teams, this just means having developers send bug reports and traces to team members when a skill or doc goes wrong – if they notice it and know who can fix it. Other teams move to a coding environment like Amp that shares session transcripts and their telemetry by default, or our Powerplant tool that supports session debugging in a harness-agnostic way. Meanwhile, Entire wants to be GitHub but for agents’ session logs as well as code. Session sharing comes with its own challenges. As with observability tools like Honeycomb and Datadog, giving your coding agents access to log and telemetry data from other agents is both very useful and potentially risky. Shared context needs thoughtful security, privacy, and accountability controls. Meanwhile, social conventions on teams around working with agents in the open are still developing. Some companies are overtly skeptical of transparency with agents, while others are all-in. No matter how this all plays out, one thing is clear: the early scramble of siloed experimentation is ending. It’s time for teams to be building shared agent infrastructure, guidelines, and guardrails, and to be iterating them together. To be learning not just from some confident poster on X, but from how one another are working with AI. And to help us all do less yelling at Claude, and more building useful software. While simply failing lints for a file that exceeds limits is helpful, we can do better. If your lints emit graduated feedback, e.g. flagging when a file is close to, at, or beyond the threshold, then you can give models useful feedback without halting a critical change because a file got pushed one line too long. Another useful graduated approach is the ratchet : if you find a bad pattern in your code you don’t want agents copying, a ratchet counts how many instances of the pattern currently exist, and only fails if that number goes up. A helpful tool for longer-term migration projects. Also worth noting: deterministic checks can give agents just-in-time remediation guidance. It’s better to put “New endpoints should use the typed pattern instead” in the lint error when somebody copies the old pattern, rather than in global AGENTS.md clogging every session’s context. ↩ may be helpful if you need to dig out of this hole. ↩ While simply failing lints for a file that exceeds limits is helpful, we can do better. If your lints emit graduated feedback, e.g. flagging when a file is close to, at, or beyond the threshold, then you can give models useful feedback without halting a critical change because a file got pushed one line too long. Another useful graduated approach is the ratchet : if you find a bad pattern in your code you don’t want agents copying, a ratchet counts how many instances of the pattern currently exist, and only fails if that number goes up. A helpful tool for longer-term migration projects. Also worth noting: deterministic checks can give agents just-in-time remediation guidance. It’s better to put “New endpoints should use the typed pattern instead” in the lint error when somebody copies the old pattern, rather than in global AGENTS.md clogging every session’s context. ↩ may be helpful if you need to dig out of this hole. ↩
Programming
Security
Open Source
Hardware
DevOps
Data
Marketing
Writing
Business
Python
AI
Java
Design
TypeScript
Database
Backend
Rust
Web Development
Tutorial
CSS
HTML
Php
Performance
C#
Shell
Cloud
Career
Git