Posts in Typescript (20 found)
マリウス 1 months ago

The Cables that Connect the World

At the northeastern edge of La Línea de la Concepción , on a scrubby Mediterranean beach called El Burgo–Torrenueva , there is an old battlement-tower, La Torre Nueva , and not much else. It was part of the system of coastal watchtowers during the 16th century that would defend the area against the incursion of the Barbary corsairs . The coordinates are . Walk the tideline and you would never know that buried two metres beneath the sand, a fibre-optic cable comes out of the sea here and turns into the internet. It’s the start of a line that runs across the Strait of Gibraltar to Ceuta , on the African coast, and on toward two continents. Nearly everything you do online that crosses an ocean passes through a cable like this, ending, in most cases, underneath a similarly unremarkable patch of coast. Note: Ceuta is an interesting place by itself, that has recently gained some attention and that would also make for an interesting write-up of its own. However, the tl;dr is that it is an autonomous Spanish city of some 85,000 people sitting on the North African coast, bordering Morocco , which means the European Union has one of its very few land borders with the African continent running straight through a peninsula most people could probably not even point to on a map. It has been held by the Spanish crown since 1668, it had been Portuguese before that, and Morocco seemingly never stopped claiming it. For our purposes, though, what matters is that the small enclave, until very recently, hung off the mainland’s network by a single ageing link. When we talk about the internet we do so as if it were air. Ambient, ownerless, and everywhere. In reality, however, it is the exact opposite, because international data doesn’t (normally) travel by, let’s say, satellite, despite what most people might assume. It travels through roughly 1.5 million kilometres of very real (and very owned) fibre-optic cable lying on the seabed, surfacing at a small number of carefully chosen landing points. For these landing points you normally need a gently sloping seabed, mild currents, and little marine traffic, so that anchors and trawlers don’t sever the line. Suitable spots are scarce enough that the same beach usually becomes the shared landfall for several cable systems at once. Unlike what you might be thinking of at first, submarine cables aren’t your run-of-the-mill Ethernet or fibre cable. The hardware that does the heavy lifting out in the deep ocean is about as thick as a garden hose with roughly 25mm across and weighing in at around 1.4 tonnes for every kilometre. The part that carries your data is a small bundle of glass fibres, each one around the same thickness as human hair, sitting in the very middle. Everything else wrapped around those fibres is there to keep them alive in a deeply hostile environment. Working outward from the core, the fibres sit in a water-blocking gel inside a thin copper or aluminium tube, which is sheathed in polycarbonate, then an aluminium water barrier, then a layer of stranded steel wires that give the cable its tensile strength, then a wrap of mylar tape, and finally an outer skin of polyethylene. The copper is for power, because the cable doubles as a very long extension lead, which we will get to in a moment. Closer to shore, where trawlers and anchors roam, the whole thing gets one or two further jackets of galvanised steel armour wire, swelling it to 50mm or more in diameter and several times the weight. Hence, the cable that surfaces on our Spanish beach is buried a couple of metres down and not simply left lying on the sand. The reason a copper conductor runs the entire length is that light, no matter how pure the glass, slowly fades as it travels, and so every 50 to 80 kilometres the cable is interrupted by a repeater , which is an optical amplifier that boosts the signal back up before passing it along. Each repeater needs electricity, and because the fish sadly still didn’t manage to install power sockets on the ocean floor, the shore stations at either end have to feed a direct current of anywhere between 3,000 and 15,000 volts down that copper core, to literally power the cable from both ends at once. On top of the amplification, modern systems lean on a stack of clever tricks to keep the signal intelligible across thousands of kilometres of glass, including wavelength-division multiplexing to cram many separate colours of light down a single fibre, coherent detection to read them back out, and forward error correction to repair whatever gets garbled along the way. Length, then, is mostly a question of power and amplification rather than of the glass itself. Shorter hops can dispense with repeaters entirely, hence an unrepeatered span will happily run to around 250 kilometres on amplifiers at each end alone, which is roughly the length of the line we started this post with. At the other extreme, a single system can stretch across an ocean, and the longest of them, like the 2Africa cable encircling the continent it is named after, run to tens of thousands of kilometres. The actual manufacturing and laying of these cables is, perhaps a little surprising for something the entire global economy rests on, the business of only a small handful of companies. The bulk of the world’s submarine cable is built and installed by just four suppliers, namely the American SubCom , the French Alcatel Submarine Networks , the Japanese NEC , and the Chinese HMN Technologies . They own and operate the specialised fleet of cable-laying ships, which aren’t exactly the kind of boat you would recognise from a harbour, but more like a purpose-built vessel carrying thousands of kilometres of cable coiled in enormous tanks below deck, rolling it out over the stern at a steady walking pace as they crawl across the ocean. Deploying a new system is a multi-year effort that begins long before any ship leaves port. First somebody, these days increasingly a content giant rather than a phone company, decides a route is worth having and assembles the money for it, either alone or as a consortium of several owners sharing the bill. Then comes a marine survey, in which a ship maps the intended path along the seabed to find the gentlest, safest route around wrecks, trenches, and other people’s cables, followed by the permitting, which is the paperwork of securing landing rights and concessions from every jurisdiction the cable so much as touches. As we are about to see on the Spanish beach, this can generate a remarkable quantity of bureaucracy . Only once all that is settled does the cable get manufactured to length, loaded onto the ship, and laid, with the vessel simply lowering it onto the seabed in deep water and a sea plough burying it a metre or two beneath the sediment closer to shore, where the danger from fishing and anchors is greatest. A working ship covers somewhere in the region of 100 to 200 kilometres a day, so an ocean crossing takes several weeks at sea. A transatlantic system running some 7,000 kilometres typically costs in the order of 250 million USD, while a longer trans-Pacific route can easily climb towards 400 million, and the cable itself runs anywhere from roughly 6,000 to 20,000 dollars per kilometre, depending on how many fibre pairs it carries and how heavily it is armoured. Keep in mind that the spending does not stop once the cable is lit, because a submarine cable has a design life of only around 20 to 25 years and on top of that there are somewhere between 150 and 200 faults occurring across the world’s cables in a typical year. The overwhelming majority of them are not caused by sabotage or sharks, but by the combination of fishing gear and dragged ship anchors. Each break has to be mended by sending out one of a small number of dedicated repair ships, that are on permanent standby under regional maintenance agreements, to grapple the cable up off the seabed, haul both severed ends to the surface, splice them back together, and lower the repaired thing back down. This is slow and weather-dependent work that is quite expensive. With the data provided by TeleGeography ’s Submarine Cable Map I have put together a list of the (co-)owners of undersea cables and sorted it by the number of cables each individual company has a stake in. The full dataset runs to some 473 distinct owners, the overwhelming majority of which are obscure national and regional carriers you will never have heard of, so rather than just dumping the entire list here, I limited it to the hundred most prolific (co-)owners: Note: These figures are derived from the public Submarine Cable Map data, counting both, systems already in service, and those still planned or under construction (603 of the former, 91 of the latter, at the time of writing). The field is free-form text, so a few owners turn up under more than one spelling, and I had to do a little manual untangling of company names. What jumps out, at least to me, is the name sitting right at the top. For most of the history of this infrastructure the owners were telephone companies, the BTs and AT&Ts and NTTs of the world, laying cables to carry one another’s calls and, later, traffic. Google now has a stake in more submarine cables than any traditional carrier on the planet, with Meta not far behind, and Microsoft and Amazon both slowly accumulating their own share. The companies that fill those cables with traffic have, over the past decade or so, decided that they would rather own the pipes than rent them. The other thing the numbers tell you is just how long the tail is. Of those 473 owners, some 260 appear on exactly one cable, and more than 340 of them, north of seventy percent, on no more than two. These are the world’s national telecoms, each one buying a slice of the handful of consortium cables that happen to land on its particular stretch of coast, which is also why so many of the big international systems list a dozen or more co-owners apiece. The internet, seen from this angle, is less of a single network and more of a mix of local operators, all chipping in for a share of the same few very expensive ropes across the ocean. To see what it looks like where the cable actually meets the land, let’s head back to that beach in La Línea . The cable that surfaces there is called Dos Continentes , it belongs to GTD , a Chilean telecoms group , and it’s a relatively small regional system consisting of two armoured fibre cables looping across the Strait of Gibraltar to Ceuta , the Spanish enclave on the African coast that depended on a single ageing link before this one was built. I went looking for exactly where it comes ashore, and the paper trail gives an idea about how invisible this infrastructure actually is. The cable lands in Spain, but the public Spanish government map of coastal concessions doesn’t seem to show it, because it looks like coastal permits in Andalusia are devolved to the regional government. The landfall instead shows in a regional registry , in a signed resolution buried under an expediente number. That document pinpoints where the cable enters the public maritime domain, at grid reference , just seaward of the beach manhole. The cable then runs inland, buried as the permit insists ( “no exterior element above ground level” ) to what is presumably a network node, where traffic is fed into GTD ’s pre-existing terrestrial dark-fibre network, from where it’ll eventually travel to one of the actual GTD data centres in Madrid , Barcelona , Bilbao/Sopelana , and Sevilla . On its way out to sea it crosses three older cables already lying on the seabed, namely Europe India Gateway , ATLAS , and FLAG . As can be seen (or, well, actually not) even an empty-looking patch of water off a Spanish beach is layered with other people’s infrastructure. Note: When GTD applied, it seems that the town council of La Línea formally objected and asked them to drop the project. The cable, the council said, cut straight through the main local fishing ground, “splitting it literally in two” , threatening the small shellfish and trasmallo boats that work those waters, and a protected limpet that lives on the rocks, in a town whose fleet was already squeezed by run-ins with Gibraltar over fishing rights. However, they were overruled and the concession was granted anyway, with mitigation conditions attached, for an initial fifteen years. The Dos Continentes cable ( Segment I , La Línea - Ceuta Sur ramal ), owned by GTD Cableado de Redes Inteligentes, S.L.U. , the Spanish arm of the Chilean GTD group , has a total length of ~105 km and is in service since 2020 under the signed concession resolution from the Junta de Andalucía ( Dirección General de Calidad Ambiental y Cambio Climático ), expediente , dated 14 January 2020. The two key points, as given in the resolution’s coordinate table are: Note: The resolution’s prose text gives a slightly different value that disagrees with its own table by approximately 140m. To convert the UTM coordinates I used the official Instituto Geográfico Nacional ( IGN ) Calculadora Geodésica with the following settings: and differ by only centimetres in practice, so the resulting coordinates (WGS84-equivalent) can be dropped straight into any consumer map or GPS app: Both points sit on Playa de El Burgo–Torrenueva , beside the Punta de Torrenueva tower, at the northeastern ( Levante / Mediterranean-facing) edge of La Línea de la Concepción , against the municipal boundary. The resolution describes the route as passing “muy cerca de la torre-faro existente en la Punta de Torre Nueva” . As you can see, however, you see nothing. :-) The permit requires the whole installation to be subterranean ( “no exterior element above ground level: No manholes, splices, connections or terminals.” ), hence you can stand exactly on the landfall, but it’s a point in the sand by a tower, and not a structure. On the afternoon I was there, a couple of dozen people were spread out on that stretch of sand under parasols, probably not even knowing that somewhere underneath them the link that carries an entire enclave’s traffic to another continent came out of the sea. It is interesting to see that what has changed most over the past decade isn’t the technology itself, but who pays for it. For a century these systems were built by carriers selling capacity to one another, which made the network something close to a shared utility with many owners. Today, however, the largest (co-)owner of submarine cable on the planet is an advertising company. It probably makes sense in their position, however it is a change in how the network is governed, and, more importantly, it seems to have happened almost entirely out of public view, which is worrying. If you live anywhere near a coast, there is a decent chance one of these things lands within driving distance of you, and the TeleGeography map will get you to roughly the right bay. Getting from there to the actual patch of sand takes some amount of digging through concession resolutions, planning registers, environmental reports, and sometimes the local newspaper archive. It took me an evening of reading to narrow it down, but I can recommend to do this exercise if you’re curious about the world that you’re living in and, more importantly, the hidden infrastructure surrounding you. PS: Maybe we picked the wrong word and should have called it the trench rather than the cloud ? Transformation type: Transformación de Datum Reference system: ETRS89 Input coordinates: UTM Huso (zone): 30

0 views
マリウス 2 months ago

GL.iNet Mudi 7

tl;dr: After almost seven years my Netgear Nighthawk M2 has started rebooting on its own, reporting nonsensical battery percentages and ignoring most of my presses on its touch buttons, so I spent the past three months replacing it with the GL.iNet Mudi 7 ( GL-E5800 ), a 5G NR Sub-6 travel router with two nano-SIM slots plus an onboard eSIM, Wi-Fi 7, a 2.5 GbE port, two USB-C ports and a removable 5380 mAh battery. It is the most capable mobile router I have owned, its 13.5-hour battery rating is close to what I measure, and the LTE reception alone is a clear upgrade over the M2 . Sadly the Tri-band on the box means two bands at a time, there is no MLO at all, both SIM trays are underneath the battery, the touchscreen still can’t get you through a captive portal, and firmware 4.8.5 has a cellular defect that leaves the device on Connecting… after a carrier deactivates an idle data session. If you came to the Mudi line for blue-merle and IMEI randomization, you might be disappointed to learn that this sadly seems to have ended with the GL-E750 . Earlier this year I reviewed the GL.iNet Slate 7 ( GL-BE3600 ), the Wi-Fi 7 travel router that replaced my long-running Linksys WRT3200 ACM as the router in my travel setup . I mentioned in that post that I was also in the process of replacing my even older Netgear Nighthawk M2 , the LTE-A Cat. 20 hotspot that has handled my mobile data for almost seven years now. The M2 has been a reliable piece of equipment, however, it has started misbehaving so badly that I no longer trust on the road. Random reboots, increasingly nonsensical battery percentages, and touch buttons that no longer register most presses make it a tedious device to use, and with it well past any expectation of longevity, I figured it was time to give its successor a proper, multi-month trial before the M2 gives up entirely in the middle of some airport lounge. The device I settled on is the GL.iNet Mudi 7 ( GL-E5800 ), a 5G NR Sub-6 Tri-band Wi-Fi 7 travel router that GL.iNet unveiled at CES 2026 and started shipping back in April. On paper the device is an upgrade over both the Netgear M2 and the Mudi V2 aka GL-E750V2 , which was still a 4G/LTE Cat. 6 device with a 0.96" OLED. The Mudi 7 packs Qualcomm ’s Dragonwing MBB Gen 3 platform, a Wi-Fi 7 PHY with a 6 GHz radio, two nano-SIM slots plus an onboard eSIM, two USB-C ports, a 2.5 GbE Ethernet port, a 2.8" color touchscreen, and a removable 5380 mAh battery, all in a 157x75x22.8mm, 300g enclosure that runs OpenWrt with GL.iNet ’s firmware layer on top. At $419.99, or roughly €425, it is also the most expensive device GL.iNet sells. Just like the Slate 7 , the Mudi 7 is above most consumer travel routers. It comes with a 5G NR Sub-6 Rel-17 NSA/SA modem with LTE Cat. 20 (DL) / Cat. 18 (UL) fallback, and the exact specifications of the hardware are as follows: Apart from having a modem, the second difference from the Slate 7 is the 6 GHz radio, which the Slate 7 lacks entirely. However, the Tri-band on the box is a bit misleading. The Mudi 7 has radios for all three bands, but the chipset cannot drive 5 GHz and 6 GHz simultaneously, so you configure the device as either 2.4 + 5 GHz or 2.4 + 6 GHz. This also means the Mudi 7 has no Multi-Link Operation at all. On the Slate 7 I complained that GL.iNet ’s MLO documentation advertises a 6 GHz band that the hardware doesn’t have. On the Mudi 7 the 6 GHz band is present and MLO is gone, which is an odd trade for a device that costs nearly three times as much. There are two regional variants, GL-E5800NA for North America and GL-E5800EU for Europe, with different 5G NR and LTE band coverage, which is important to travelers like myself. Both variants cover n5, n7, n26, n38, n41, n77 and n78. Beyond that they diverge, as the EU model adds n1, n3, n8, n20, n28, n40 and n75, while the NA model adds n2, n12, n14, n25, n30, n48, n66 and n71, plus n13, n29 and n70 in SA mode only. LTE splits the same way, with the EU model on FDD B1, B3, B5, B7, B8, B20, B28 and B32 and TDD B38, B40, B41, B42 and B43, and the NA model on FDD B2, B4, B5, B7, B12, B13, B14, B17, B25, B26, B29, B30, B66 and B71 and TDD B38, B41, B42, B43 and B48. For my use case (almost exclusively APAC/LATAM) the EU variant turned out to be the more sensible choice, but anyone moving frequently between North America and the rest of the world should read both band lists carefully before ordering. To be fair, though, the Nighthawk M2 splits even harder. Netgear ships that device as at least five separate SKUs, and the band list for each one is quite short. The box itself contains the Mudi 7 , the battery pack, a relatively big travel pouch, a USB-C cable, and the paper manual. No external antennas and no power adapter, which I appreciate, given the chargers I already lug around. The headline feature is the modem, which uses the Dragonwing platform, Qualcomm ’s rebranded enterprise and mobile-broadband lineup. In practice the 4.67 Gbps peak figure is, as with virtually all hyped peak numbers, marketing material. Real-world throughput depends primarily on the carrier’s network, the SIM plan, the spectrum allocation, the band combination, and the signal conditions at your specific location. In my own testing I have seen sustained downlink figures in the 600–900 Mbps range on a properly-provisioned 5G network, and significantly less (in the 100–250 Mbps range) on a more typical mixed NSA deployment. What’s more important, though, is the LTE fallback. The modem falls back to LTE Cat. 20 (DL) / Cat. 18 (UL) and is significantly more sensitive than the M2 ’s aging Qualcomm baseband. In the same hotel rooms where my M2 used to show a single LTE bar at best, the Mudi 7 can consistently show two or three, often pulling more usable bandwidth on the same SIM and the same carrier. Lastly, the Mudi 7 has two TS-9 external antenna ports for those of us who care to bolt on a pair of paddle or directional antennas in RV/cabin/dead-zone scenarios. I haven’t bothered to test these, as my use case doesn’t involve any of that. However, these days most people might have almost exclusively converted to Starlink anyway, so the external antennas might not be as much of a selling point as they were ten years ago. The Mudi 7 has two Nano-SIM slots and one onboard eSIM. Both Nano-SIMs and the eSIM are managed via the touchscreen and the web UI. However, it’s important to note that the Dual SIM Dual Standby in this context means dual standby with an asterisk. The onboard eSIM and SIM slot 2 are mutually exclusive and cannot be active at the same time. The eSIM is disabled by default, and the moment you enable it, SIM 2 stops functioning. SIM 1 remains operational either way, and the modem can auto-switch (i.e. fail over) between SIM 1 and whichever of SIM 2 / eSIM is currently active, but you do not get to keep three simultaneously hot profiles. For anyone hoping to keep a local SIM, a regional roaming eSIM, and a home-country SIM in standby together, this is a bit of a disappointment. Failover itself has also been more rigid than I expected. The web UI exposes the auto-switch feature, including data-usage thresholds and signal-loss triggers, but the failover decision-making has been slow in practice. A complete loss of signal usually does cause a switchover within a reasonable amount of time, but more nuanced situations (such as one SIM throttling without any indication, or losing data while still showing connected ) often require a manual nudge. GL.iNet ’s documentation describes far more sophisticated multi-WAN coordination than the SIM-side auto-switch logic delivers. Then again, to be fair, Mwan3 on the Linksys has had similar issues and I guess down detection is just a complicated thing to get right. One caveat is that both Nano-SIM trays are underneath the battery , so putting a card in or taking one out means having the device powered down, prying off the back cover, and pulling the battery out. On a product aimed at people who buy a local SIM on arrival, that is a weird design. Then again, in many cases the device is probably already powered off because you arrived by airplane anyway. Switching between profiles that are already provisioned (either physical-to-physical or physical-to-eSIM) is one of the things the touchscreen handles well, and it doesn’t normally require any detours into the admin UI. Speaking of which, just like the Slate 7 , the Mudi 7 comes with a built-in touch display, though here it is a 2.8" color LCD rather than the much smaller panel on the Slate 7 . The screen shows the usual variety of things, like signal strength and current network type, connected client count, real-time data usage, battery percentage, Wi-Fi details with a QR code for quick joining, and the ability to toggle the VPN, the Wi-Fi, and a couple of other features without opening the admin UI. Firmware upgrades also display a progress bar on the screen, which (as I had complained about with the Linksys ) is a small but welcome quality-of-life feature. The notable thing missing from the touchscreen is captive portal handling. The moment the upstream WAN is a hotel or airport Wi-Fi network with a captive portal in the middle, the touchscreen is useless and you have to reach for a phone, tablet, or laptop, attach to the Mudi 7 , open a browser, and go through the portal manually before the router (and everything behind it) can reach the internet. But to be fair, a 2.8" panel is probably a poor place to render an HTML login form and a keyboard to begin with. The lockscreen with a 4-digit PIN that was introduced on the Slate 7 is also present on the Mudi 7 , which I once again appreciate, given the kind of sensitive information (carrier and SIM details, VPN state, hostnames) that this screen displays. One annoying quirk is the battery percentage reporting. Both the LCD and the web UI will, after a full charge, stay at 100% for the first 1–3 hours of unplugged operation before catching up to reality and dropping rapidly to whatever the actual state of charge is. The underlying kernel fuel-gauge driver does report accurate values (you can confirm this via SSH and ), but from what I can see the MCU layer that drives the LCD and the admin UI applies some smoothing to avoid the device displaying 98–99% immediately after charging. I would much rather see the truth on the screen than a smoothed consumer-friendly approximation, especially on a device whose entire purpose is to be unplugged for long stretches. The Mudi 7 shipped with OpenWrt 23.05.4 ( , Kernel ), with GL.iNet ’s firmware layer on top. The device runs Qualcomm ’s proprietary SDK and binary blobs. The same software-openness caveats that apply to the Slate 7 apply here as well. You get full root SSH access, the configuration tree, and the ability to side-load the LuCI UI if you want, but you’re stuck with GL.iNet ’s firmware for anything that touches the cellular or Wi-Fi 7 silicon. The original Mudi ( GL-E750 ) is the device that blue-merle was written for, the SRLabs package that changes the IMEI via AT commands on the device’s modem, wipes the stored client MAC addresses, and randomizes the BSSID and the WAN MAC address across reboots, and it is a large part of why the Mudi line got its reputation as the privacy-focused travel router in the first place. However, blue-merle supports the GL-E750 and nothing else, and with the 5G modem, the firmware base, and the entire platform having changed underneath it, there is no indication that this is going to change. If IMEI randomization is the reason you were looking at a Mudi specifically, the Mudi 7 does not give you that, at least today. To be fair, the firmware layer is also what makes the device usable out of the box. The Multi-WAN , WireGuard , OpenVPN , Tailscale , AdGuard Home , DNScrypt-proxy2 , Tor , and the modem management features are all preinstalled and reachable via a friendly web UI, which (as I had mentioned in the Slate 7 review) is a substantial step up over the bare vanilla OpenWrt experience on an older router like my WRT3200 ACM . The Mudi 7 supports WireGuard with up to 600 Mbps. I have been running my own WireGuard tunnel on the device, routing the entire LAN through it, and it has kept up with whatever the upstream 5G or LTE connection could deliver. As with the Slate 7 , Tailscale is available, with the same caveats. Basic connectivity works, but anything beyond the default configuration (exit nodes with advanced flags, subnet routing, tagged ACLs, etc.) is going to require manual intervention via SSH. The Mudi 7 can, like the Slate 7 , run a Tor node and route LAN traffic over it. The moment Tor is enabled, VPNs , DNS , AdGuard Home and IPv6 will not work properly anymore, because the firmware doesn’t (yet) compose these services the way a hand-rolled OpenWrt setup can. Note: As I had explained in the Slate 7 review , these limitations are 100% a GL.iNet issue and not caused by OpenWrt . The same combinations work fine if you wire them up by hand on top of a vanilla OpenWrt installation, including DNS lookups via Tor through DNScrypt-proxy2 . The UI just isn’t there yet on the GL.iNet side. AdGuard Home is, as on the Slate 7 , part of the default installation and just as plug-’n-play. I still don’t use it personally, but the web UI is identical to the one on the Slate 7 and works fine in the configurations I have tested. The Mudi 7 differentiates itself from most travel routers in the number of uplinks it can hold at once, as the device supports up to five concurrent WAN inputs: The cellular modem, the 2.5 GbE Ethernet port (when configured as WAN), Wi-Fi-as-WAN (i.e. repeater mode), USB-C tethering from a phone or a secondary modem, and USB-C-attached USB Ethernet adapters. The firmware uses Multi-WAN underneath, with a friendly UI on top. Router, access point and extender modes are all supported, WDS is not. The device features dual USB-C, with one of the USB-C ports being power-only. The other USB-C port is a fully-featured 10 Gbps port with USB tethering, and USB OTG support. It’s possible to charge the Mudi 7 on one port while simultaneously tethering on the other. USB tethering itself, just like on the Slate 7 , is a matter of a few clicks in the UI. Plug a phone in, enable tethering on the phone, and the Mudi 7 picks it up as a USB Ethernet WAN. The same applies to a USB-to-Ethernet adapter, should you ever need to add a second wired WAN or to bridge into a hotel’s wired LAN where Wi-Fi is unreliable. I have had the Mudi 7 for roughly three months now, and the tl;dr is that the device is pretty solid overall, with a handful of caveats around firmware quirks and the chunkier footprint. Battery life is a bit of a mixed bag here, because it depends a lot on what features/services are running on the Mudi 7 , on the amount of WiFi clients and how cellular coverage is. Let me therefore put it this way: For the amount of features you get with the Mudi , especially compared to my older M2 , the battery life is decent. Having that said, however, I do believe that the Nighthawk , at least in its earlier days, was able to survive longer on a single charge than the Mudi is able to right now. Obviously I don’t have scientific benchmarks to prove it, but I remember vividly being out and about with the M2 for a full day and going to bed with the device only around halfway drained. This is something that I don’t think is possible with the GL.iNet . While the device easily gets through a regular workday, I probably wouldn’t trust it to survive a full day road trip with four friends through a mountainous region. Ultimately, its battery life can be extended using an external powerbank, but that’s clearly not ideal with a device that already weighs 300g on its own. If we’re being honest here, 300g equals about two Google Pixel 5 or two Motorola Edge 30 phones, which can both provide you with a 5G hotspot and which will probably (combined) outlast the Mudi by at least a few hours. So if the pure 5G hotspotting capability is all you care about, the GL.iNet is definitely not a good option with regard to battery life. If, however, you’re looking at it as the centerpiece of your mobile LAN, that will allow you to leave your Slate 7 at home because it supports pretty much every important feature and offers integrated 5G connectivity on top of that, then its battery life isn’t too bad after all. The chassis warms up noticeably under sustained 5G load (especially with a VPN), but never to the point where I’d be concerned about throttling or comfort. The back gets warm to the touch, but no warmer than a mid-range phone under similar load, and certainly not as warm as my old M2 would get at times. Unlike with the Netgear , I haven’t experienced any heat warnings with the Mudi so far. The build quality is solid. The chassis has a reassuring density to it, the touchscreen is responsive, and the front button doesn’t feel flimsy. The back panel is a bit of a weak point, because it is a plastic snap-fit cover protecting the battery and it creaks under pressure. Given that this cover has to be pried off to swap the battery or a SIM, I’m half-expecting it to wear out relatively quickly. Weight and footprint, as I had anticipated in the travel desk write-up , are clearly worse than the M2 ’s. The Mudi 7 is heavier (300g vs the M2 ’s 240g) and noticeably chunkier in both length and width. In absolute terms this is still a small device, but on a packed desk and in a packed bag, the difference is noticeable. The included travel pouch is also larger than the router needs, because most of the extra volume is set aside for accessories. Most people probably won’t use the travel pouch for travel, but rather for storage at home. Charging behavior has been predictable. The 24W PD fast-charging input gets the 5380 mAh battery from 0% to ~80% in roughly an hour, and to full in about an hour and 45 minutes. The device accepts whatever USB-C PD source there is around, including my UGREEN 100W and the Sharge Pouch Mini P2 power bank. If you’re considering this device as a permanent member (or even a centerpiece) of your LAN, I have some good news for you: The Mudi 7 can be operated via USB-C, without its battery plugged in. I don’t know whether this is officially supported by GL.iNet , because when you connect a charger the display will show a battery icon with an exclamation mark inside of it, but long-pressing the front button will turn the device on nevertheless. I haven’t experienced any peaks in power-consumption that would lead to arbitrary restarts without the battery plugged-in, but your mileage may vary. Reliability has been pretty good, and I haven’t experienced any crashes, random reboots, or other issues. The only firmware-level oddities I have encountered are the battery reporting discussed above and the cellular issue in firmware 4.8.5 mentioned in the tl;dr : After a carrier deactivates an idle data session, the router can remain on Connecting… until I intervene. The Mudi 7 is probably one of the most capable travel-friendly mobile routers I have ever owned, and it has a permanent place in my travel setup . The 5G modem, the dual-SIM-plus-eSIM configuration, the dual USB-C ports, the 2.5 GbE port, the removable 5380 mAh battery, and the Wi-Fi 7 PHY can replace the M2 + Slate 7 combo for me, while also covering scenarios (5G, multi-SIM, multi-WAN, USB-C-tethered secondary modems) that the combo never could. While the 300g weight and the bulkier footprint are a step back compared to the M2 , if I account for the added size and weight of the Slate 7 that I had to lug around alongside the M2 to make the LAN work for me, then it doesn’t look as bad anymore. Then again, with the M2 + Slate 7 combo I had the flexibility to only bring what’s really needed, which, for e.g. a day trip, would end up being only the M2 . Apart from that, there is the Tri-band situation, with the chipset only driving two bands at once and offering no MLO at all, the SIM-failover logic, which doesn’t work as smoothly as one would expect, the SIM 2 vs eSIM mutual-exclusion, that is mildly annoying, and the battery percentage smoothing, that makes me distrust everything else the device reports. However, none of these are deal-breakers but more like minor inconveniences. The proprietary Qualcomm blob situation is the more concerning part for me, and as with the Slate 7 , the Mudi 7 is OpenWrt only in spirit , and its long-term wireless and cellular stack support depends entirely on GL.iNet and Qualcomm continuing to cooperate. If your priority is true software freedom, this device, like virtually every other 5G-capable router on the market today, is not for you. Neither is it for you if you bought into the Mudi name for blue-merle and IMEI randomization. If your priority is a modern, travel-ready, multi-WAN, multi-SIM 5G router with a solid (if proprietary) firmware on top, the Mudi 7 is, at the time of writing, the best option I’m aware of. I will be sticking with the Mudi 7 as my mobile-data device for the foreseeable future, and the Nighthawk M2 has been retired to the bottom of a drawer after seven years of service. PS: Make sure to check future updates if you’re interested in the long-term experience with the Mudi 7 .

0 views
David Bushell 2 months ago

End of the contact form saga

I can’t take it anymore! If you wan’t to speak to me, send an email. My contact form is out of service indefinitely. This is actually in lieu of moving my professional services to a yet to be announce limited company. But I can’t let opportunity for a dramatic blog post go to waste. Also, I’ll probably skip the contact form on my company website. Is that a bad idea? I always got more spam via the form than the publicly visible address. My contact form has been through a lot. Previous entries in the saga: I quite enjoyed the week in September when I opened a port to a self-hosted SMTP server I coded in 100 lines of TypeScript. The final iteration of my form included true end-to-end encryption. Through trial and error heuristics, I successfully eliminated all spam. (How many false positives I rejected remains unknown…) My privacy policy which was already simple is now entirely pointless. Are contact forms just outdated in general? Everyone seems to embed a Calendly widget these days. That’s not my style. I like the tiny bit of additional friction required to send an email. If someone can’t be bothered their message probably wasn’t serious. I’m not looking to maximise meaningless engagement. I look forward to moving business email to a separate domain. Biggest mistake I ever made was using for personal and business. Nothing worse than seeing an “urgent” request only to find out on Monday it didn’t matter. So long old contact form, it was fun! Thanks for reading! Follow me on Mastodon and Bluesky . Subscribe to my Blog and Notes or Combined feeds. SMTP on the edge Email: the final form I shut the emails out I let the emails in Progressive dehancement PGP encrypted contact form

0 views
Filippo Valsorda 2 months ago

Production ML-DSA Verification in 350 Lines of Python

I don’t do a lot of Python, at least not in my most recent life. 1 However, I happen to have just written a production ML-DSA verifier in pure Python . It’s 350 lines of code (plus many more of tests), it supports all parameter sets, and I am pretty satisfied with it. You can fetch it as from PyPI , thanks to William Woodruff , or you can copy-paste it: it’s a single file without dependencies and it’s dual-licensed CC0 and 0BSD. It works with Python 3.8 and later. The API is modeled after the excellent pyca/cryptography . I hope this will make it easier for some projects to migrate to post-quantum authentication, which has suddenly become more urgent than we all anticipated . In particular, I hope it will unblock some client applications that can’t use C extensions for portability reasons. Modern Python package management , typing , and linting are also a lot more powerful 2 than in the early Python 3 days, and the result is a pretty readable ML-DSA verifier. ML-DSA is actually very simple to implement with its 23-bit base field: we use Python integers (without even needing Python’s big integer support) and SHA-3 from hashlib. There are 86 lines of throat clearing, 27 lines of base field (arithmetic, , ), 28 lines of sampling ( , ), 39 of polynomials ( , ), 25 of NTT, 30 of parsing and packing ( , , ), 35 of key expansion ( , ), and 80 of actual signature verification ( , , ). Performance is… decent? 230 ML-DSA-44 verifications per second without precomputation. That’s 60x slower than Go, but not 1000x. The only optimization change I made was using integers instead of field elements in the NTT hot loop . The implementation is tested with the full reusable ML-DSA testing stack: Wycheproof test vectors and CCTV accumulated vectors , using pytest and muzoo for mutation testing. It has 96% branch coverage, and more importantly it kills every mutation I (and Claude) could think of. (ML-DSA testing techniques deserve their own article.) The project started as a way to double-check the tests of the tests of my Go crypto/mldsa implementation. How do you know your tests are good and comprehensive? You add bugs (“mutations”) and you check that the tests fail. What if you skipped a check though? There won’t be any code to introduce a bug in! The obvious solution is to write a different implementation from scratch, then introduce bugs there, check that the tests catch the bugs, and then port the tests back. Duh. Anyway, pure Python might not be particularly well-suited for cryptography that involves secrets because producing constant-time code could be difficult. However, a signature verifier involves no secrets, and Python is expressive and, most importantly, different from Go, making shared mistakes less likely. You might want to follow me on Bluesky at @filippo.abyssdomain.expert or on Mastodon at @[email protected] , but I can’t promise any more Python. The CENTOPASSI is not all smooth riding, that’s part of the point. However, I am a little annoyed at the local who I had called and who said this road was closed but totally doable on a motorcycle. My work is made possible by Geomys , an organization of professional Go maintainers, which is funded by Ava Labs , Teleport , Datadog , Tailscale , and Sentry . Through our retainer contracts they ensure the sustainability and reliability of our open source maintenance work and get a direct line to my expertise and that of the other Geomys maintainers. (Learn more in the Geomys announcement .) Here are a few words from some of them! Teleport — For the past five years, attacks and compromises have been shifting from traditional malware and security breaches to identifying and compromising valid user accounts and credentials with social engineering, credential theft, or phishing. Teleport Identity is designed to eliminate weak access patterns through access monitoring, minimize attack surface with access requests, and purge unused permissions via mandatory access reviews. Ava Labs — We at Ava Labs , maintainer of AvalancheGo (the most widely used client for interacting with the Avalanche Network ), believe the sustainable maintenance and development of open source cryptographic protocols is critical to the broad adoption of blockchain technology. We are proud to support this necessary and impactful work through our ongoing sponsorship of Filippo and his team. Fun fact, I got started in open source as a maintainer of youtube-dl.  ↩ I feel the same about the TypeScript ecosystem. It’s fun for a week or two every once in a while, but I wouldn’t want to daily drive any of these ecosystems: it’s too easy to spend a whole day updating dev dependencies and fixing linter errors and get the mistaken impression of having gotten anything done.  ↩ Fun fact, I got started in open source as a maintainer of youtube-dl.  ↩ I feel the same about the TypeScript ecosystem. It’s fun for a week or two every once in a while, but I wouldn’t want to daily drive any of these ecosystems: it’s too easy to spend a whole day updating dev dependencies and fixing linter errors and get the mistaken impression of having gotten anything done.  ↩

0 views
Unsung 3 months ago

“If HEIC has no haters I’m dead.”

Over on Bluesky, Melanie Walsh asks : Favorite and least favorite file formats? I’ll start. Favorite: TXT Least favorite: HEIC The answers – both replies and quote posts – are really interesting because most of the time they’re not about inherent capabilities of each format, but: Of course, Walsh put a finger on the scale with her initial example, but HEIC stands out as a favorite least favorite. I understand this is mostly out of its limited support, raising a question whether Apple spent the right amount of time socializing and incentivizing its adoption – even on a Mac, you can’t escape blank stares the moment you drag it into many websites/web apps: HEIC on the other hand, Apple’s way of making photos smaller and everything else more complicated than it needs to be. By the way HEIC is when you drag a picture from your Notes app into your email, and then it laughs in your face and is like sorry, girl, I’m HEIC!! I don’t do things like that!! I didn’t know I had a least favorite file format but yeah HEIC can fuck right off Sweet fucking hell fuck heic into the sun Reading the replies here makes me feel like I live in an oddly privileged bubble in an inverse of the usual meaning of privilege for being a poor Android-using mfer who has never seen a HEIC in their life and had to actually look that sh*t up. Least favorite is a toss up between HEIC (WHICH NOBODY ASKED FOR, APPLE) and WEBP Controversial but I hope everyone involved with HEIC only tastes soap instead of cilantro forever I agree with this person that WebP is much better supported than it used to, but it sometimes takes one link in the chain – cough Google Docs cough – for you to avoid a format forever. And, those are always lagging indicators. If a format didn’t work once in an important flow, it might take many years before you come back: all the people saying “webp” in the quotes might as well be fighting WW2 still. look for another grievance. please Some other fun answers: IF IT’S CALLED [C]OMMA [S]EPARATED [V]ALUES WHY DO I HAVE TO OPEN A WINDOW AND CHANGE THE DEFAULT DELIMITER OPTION FROM TAB TO COMMA ??!?!?! Favorite: MP3 (invented piracy, patents all expired, doesn’t need an FPU) Least favorite: DICOM (nightmarish metadata, too many possible image encodings, when it wants a 3D volume the solution is just “a bunch of files in a folder”, also IT IS A NETWORK PROTOCOL >:( ) Least fave: .R01, .R02, etc... – nothing needs to be split into multiple rar files! Please stop! The world has moved beyond this. Least favorite: can I count those awful pointer doc types Google uses, like .gdoc and .gsheet favorite: transparent PNG least favorite: transparent PNG that is not really transparent but just a fuckin checkered background I forgot about this meme: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/if-heic-has-no-haters-im-dead/1.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/if-heic-has-no-haters-im-dead/1.1600w.avif" type="image/avif"> For least fav I voted for GIF, having not only spent countless hours trying to make good-looking animated gifs that do not weigh tens of megabytes, look horrible, and cause performance issues… but also having worked on two different products (Medium and Figma) that had to swallow gifs made by others, and seeing engineers lose their minds peeking into their insides and how messy they were . To be fair, GIF comes from the late 1980s, and simply outlived its purpose. It’s a fascinating format that literally deserves a book written about it: the messy patent wars, the pronunciation, the technical format and many surprises hiding inside , even the word “gifs” transcending the format itself to mean “short animated memes.” To go back to the thread, a small pattern that I also encountered from time to time: Least favorite: .md, specifically when it’s used for Sega Genesis game roms. There’s already a type of text file type called .md, so Windows tries to open them in notepad. Just call it .gen instead, nerd. Favorite: TS, the one that opens in my IDE Least Favorite: TS, the one that opens in Quicktime Lastly, because of course someone had to do it: Favorite: Gaylord Archival® Reinforced Acid Free Manilla Least favorite: Office Depot Vertical Hanging Folders #encoding #graphics #software evolution how well supported it is in the general ecosystem? how painful it was last time I used it? who’s using it and for what? if there is one app I use it with, do I like this app? (interesting in the context of PDFs which some people love, and others hate)

0 views
spf13 3 months ago

Why TypeScript 7.0 Was Rewritten in Go (and what it means for your dev stack)

Over the last year, the team that invented TypeScript ported the TypeScript compiler and tools to Go . Not Rust. Not C++. Go . Microsoft’s numbers show build times improving by roughly an order of magnitude in the Go-based TypeScript 7.0. In the age of AI-assisted development, one of the largest JavaScript/TypeScript organizations on earth chose Go for its flagship tool. Microsoft’s reasons were pragmatic, and Anders Hejlsberg laid them out : the existing compiler’s function-heavy style ported to Go nearly one-to-one, both the old and new compilers depend on garbage collection, and native code with shared-memory concurrency is where the 10x came from. The announcement doesn’t mention AI once. But the properties that made a pragmatic 1:1 port possible: simple functions, no hidden magic, a GC, code a team can hold in its head, are the same properties the next era of development demands, and that’s what this post is about. Go was built on a bet: optimize for the reader, not the writer, in exchange for less code to misread. LLMs read code more than any human ever could, and as they write a growing share of it, humans become even more reader-heavy. Agentic development is the most extreme test yet of the goal Go was built for: readability, maintainability, long-term correctness at scale. The conventional argument for Python and TypeScript in the AI era is strong on the surface. Python has PyTorch, LangChain, and the ML ecosystem. TypeScript has the web and a vast developer pool. LLMs trained on mountains of both write them fluently, confidently, and fast. Python and JavaScript were designed as scripting languages: fast to write, forgiving, dynamic. TypeScript adds structure on top, but it’s a compile-time overlay: types are erased at runtime, so none of TypeScript’s guarantees survive execution. TypeScript hasn’t displaced JavaScript so much as grown alongside it: GitHub’s 2025 Octoverse report has TypeScript overtaking Python and JavaScript by monthly contributors, but JavaScript still generated nearly double the new repositories over the same 12 months, and GitHub’s own summary is blunt: “JavaScript is still massive,” and the combined “JavaScript/TypeScript ecosystem still accounts for more overall activity than Python alone.” Agentic systems aren’t scripts, though; they’re services, pipelines, CLIs, and distributed systems running in production for years, and all three languages fight that design at every step: complexity management, dependency isolation, deployment, runtime safety. Go was designed as a systems language for large, long-lived software, and the industry is converging on exactly that category: compiled, safe, built to run for years, not scripted for an afternoon. The most exciting thing happening in TypeScript right now is its port to Go. The most exciting things happening in the Python world are mostly happening in Rust: Pydantic ’s validation core is Rust, Polars is Rust, HuggingFace tokenizers is Rust, and Astral’s is Rust too. Go’s share of that shift looks different but is just as telling: it’s the default choice for the infrastructure the current wave of agentic tools is built on. Ollama , the standard way to run local models, is Go. Weaviate , the vector database behind countless RAG and agent-memory stacks, is Go. Temporal , the durable-execution engine teams use to orchestrate long-running agent workflows, is Go. Charm’s Crush , a beautiful AI coding agent CLI in the same category as Google’s Antigravity , is Go too. Even GitHub’s own MCP server , the reference implementation for connecting AI agents to GitHub, is Go. In a development loop, the question isn’t “which language is easiest to write?” It’s “which is easiest to write, review, and ship?” Agentic development multiplies that question. A machine now asks it hundreds of times a day. An agentic coding loop looks like this: implement → build → test → analyze failure → self-correct → repeat. A human developer runs this loop maybe a dozen times per hour; an autonomous agent runs it dozens of times per task : what Anthropic’s guidance on coding agents describes as systems that “iterate on solutions using test results as feedback.” That frequency changes the economics of language choice. Wes McKinney named this shift in his essay on agent ergonomics : once an agent is doing the coding, fast compile-test loops, frictionless distribution, and deterministic builds matter more than whether a language is pleasant for humans to type. He didn’t just write about it. Roborev , his continuous background code-review tool for AI coding agents, is built in Go. Go solves four compounding problems that most engineers don’t notice, because they’re so used to the friction they no longer see it. Each one is paid for in the same currency: developer time and focus, agent iterations, and API dollars spent correcting what the language let through. And they stack in order: slow builds waste iterations, broken dependency resolution wastes entire runs, weak error feedback lets mistakes survive those runs, and ecosystem churn invalidates what the agent knew before it started. A several-minute build is routine for large Rust or C++ projects: a minor break for a human, but hours of wasted cycles for an agent running 50 iterations on a feature. Go compiles almost instantly, keeping the loop tight. Python’s still doesn’t guarantee deterministic installs by default, and version conflicts across machines remain common even with virtual environments. Node’s actually handles conflicting transitive dependencies well: nested can hold multiple versions of the same package side by side, and plus make installs far more reproducible than they used to be. Where npm still gets stuck is peer dependencies, where conflicting requirements throw errors that need manual untangling. That version flexibility costs deep, duplicated dependency trees that bloat and inflate the surface area an agent has to reason about. Both ecosystems also let arbitrary code execute at install time ( , scripts), a real supply-chain risk that’s bitten each more than once. Go takes a different approach by default: pins exact checksums, a single version of each module is selected deterministically for the whole build, and there’s no install-time code-execution hook for a compromised dependency to exploit. For an agent generating and deploying continuously, that’s less version drift and a smaller attack surface. Python’s type hints and TypeScript’s type system can catch real mistakes before runtime, when tools like , , or run as part of the loop. But both come with an asterisk: Python’s typing remains optional and unevenly adopted, even as tools like Pydantic push teams toward it, and TypeScript’s guarantees are erased at runtime, with a sanctioned, zero-cost escape hatch whenever code hits friction. That’s the incentive an agent under time and token pressure responds to: reaching for to make something compile costs nothing in the moment, whatever the wider ecosystem’s norms are. Jesse Vincent, who builds the Superpowers agent orchestration framework, has documented what happens when that opt-out exists: agents rationalize their way around rules. In one now-infamous incident , an AI agent in his environment deleted test files to make failing tests disappear: internally consistent logic, since tests that don’t exist can’t fail. His insight, in his own words : “A rule has an opt-out path (I can rationalize ‘I’ll do it after this one thing’). A gate doesn’t — the next action is blocked until the gate condition is met.” Go has no such opt-out. Go does have (it’s the alias for ), but it isn’t TypeScript’s . TypeScript’s turns the checker off for everything it touches; Go’s is still checked at every use site, and the compiler refuses any operation the type can’t support without an explicit assertion it verifies. The escape hatch is itself gated. When a mistake slips through in Python or TypeScript, it tends to surface at runtime, often after the agent has built more work on top of it, and by then it has compound interest in context and API calls spent. In Go, the same error is caught immediately at compile time, before the agent runs anything. The difference shows in the code itself: Python’s ergonomic advantage is a disadvantage here: the more forgiving the language, the more mistakes an agent can make before getting caught, burning context and API calls along the way. The same gap shows up in code review , now that both sides are often AI: a reviewing agent can see what Python or JavaScript code says but not reliably what it does, since metaclasses and prototype chains hide behavior a static read can’t catch. Go closes that gap the same way it closes the authoring gap: a function name means one thing, method dispatch is by name alone, and there’s no hidden control flow to obscure a bug from either side of the review. Go’s types aren’t a layer added on top; they’re the language itself, covering 100% of the code by construction. This may be the largest of the four. Go’s compatibility promise means code written for Go 1.0 in 2012 still compiles and runs correctly today. The language and standard library only add; they essentially never break what already worked. Nothing in the Node ecosystem offers that guarantee. Major frameworks routinely break compatibility with their own prior versions: Svelte 5’s runes are a different reactivity model than Svelte 4, Vue 3 required rewriting Vue 2’s reactivity model from scratch, and React’s evolution from class components to hooks to server components has left multiple incompatible eras of “how React code looks” in active circulation. Ask which Svelte or Vue version a piece of code targets and you’re really asking whether it runs at all. Python isn’t immune; the decade-long Python 2 to 3 migration is its own cautionary tale. But Node’s churn is faster and more constant. That churn is a specific, compounding problem for agents. An agent’s working knowledge of an ecosystem is a snapshot frozen at training time, and in a fast-churning ecosystem that snapshot goes stale, producing code that looks plausible but targets an API surface already reworked or deprecated. In Go, a training snapshot from years ago is still correct today. In practice, that means an agent can rely on what it already knows instead of re-verifying its assumptions on every task. Dave Rensin, Distinguished Engineer at Google, captured the compounding risk while building a 100,000-user internal tool with AI agents : “If we aren’t careful, we aren’t just writing code faster. We are mass-producing our mistakes.” Rensin wasn’t talking about Go specifically; that risk is generic to AI-assisted development in any language. But it’s exactly the risk Go’s structure (static types, explicit imports, no magic) is built-in friction against: bad code is harder to write in the first place. At PayPal, a team shared with me an experience where a custom C++ database was powerful and correct, but the team had stopped growing: every hire spent months learning the codebase, and most engineering capacity went to maintenance. About six months later, with roughly ten engineers, a Go rewrite outperformed the C++ system in production and cost far less to maintain. Go isn’t faster than C++, so what happened? The bottleneck was never the code’s speed. It was the team’s ability to reason about, maintain, and extend it. Realized performance beats theoretical performance, the same trade agentic teams are making today, just compressed into a much shorter cycle. Rust and Go are complementary and typically occupy different use cases and both have a place in agentic development, but Rust is a specialized tool rather than the default. Rust shares a lot of strengths with Go: memory safety, static typing, no hidden runtime magic. Rust’s compiler errors are famously instructive, too: by this post’s own compiler-as-gate logic, an agent gets real value from a detailed borrow-checker message. However, the two languages diverge exactly where agentic development is least forgiving. Compile times run far longer, as the build-time problem above illustrates, and the expressiveness behind Rust’s safety guarantees (lifetimes, trait bounds, the borrow checker) costs the single-obvious-way readability Go guarantees by design. The cost shows up hardest at refactor time: in my experience, a change that stays local in Go ripples through lifetimes and trait bounds in Rust, and refactorability is the property agentic development leans on hardest. An agent, not a human, now does that rework, and agents need far more of it to get code right. The more complex the language, the more likely an agent misreads it and produces a bug that compounds over iterations. Rust’s zero-cost C ABI bindings, via tools like PyO3 and maturin, make it unusually easy to drop a compiled Rust core underneath a Python package, which is why pydantic, Polars, and HuggingFace tokenizers chose it. Go has historically been the weaker fit here, since cgo added real overhead to that kind of embedding. While Go is closing that gap: Go 1.26 cut the baseline overhead of cgo calls by about 30% , Rust remains the better choice for a lot of use cases. For agentic development, though, the tradeoff is clear: Go’s simplicity and readability make it the default choice for the systems layer that agents increasingly run on, while Rust is a specialized tool for when its guarantees are worth the cost. Everything above is a cost multiplied by iteration count. Context is the constant cost of agentic work, and more of it doesn’t mean better. The assumption behind “just use a bigger context window” is that a model reasons over its 200K or 1M tokens uniformly. Chroma’s 2025 study across 18 leading LLMs (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) tested that directly, holding task difficulty constant and varying only input length, and found the assumption false: performance degrades as input grows, even on trivial tasks. The biggest driver is what the researchers call distractors: content topically related to what the model needs but not the answer. One distractor measurably hurts accuracy; four compound the damage. Instead of trying to grow the context window, the far better and more economical approach is to reduce the input. A class hierarchy, a mixin chain, a decorator stack: that’s a field of distractors to an agent tracing a bug. A call resolving through diamond inheritance, a metaclass rewriting behavior at runtime, an override buried three classes up: none of it is the logic the agent needs, but the agent still has to load it, weigh it, and rule it out, inside the same window where its reasoning is already degrading. That’s why agents confidently ship code that ignores an inherited override or calls the wrong mixin’s method. Go’s design is minimal and explicit. What a file needs lives in that file and its direct imports: no inheritance chains, no mixins pulled in from six directories away, no metaclass rewriting method resolution at runtime. A function name means one thing, and the compiler enforces it. The tokens Go code needs are almost entirely relevant tokens. That has a second effect beyond accuracy: it lowers the model a task requires. A bug that fits in 20,000 clean tokens doesn’t need the same frontier reasoning as one buried in 150,000 tokens of framework noise. Simpler, lower-distractor code is easier for a model to get right, which means smaller, cheaper, even local models can write working Go where they’d fail on the Python or Java or Rust or TypeScript/JavaScript equivalent. This matters even more when the AI companies stop subsidizing their costs, which we’ve already seen begin. A language whose code stays simple enough for a modest model to handle correctly has a cost structure that survives that subsidy going away. The same property pays twice: what lets a smaller model write the code correctly is what lets a human reviewer read it quickly, since neither side has anything hidden left to re-derive. Beyond even this, it’s more likely to get the code right the first time, which means fewer iterations, fewer API calls, and less context burned and more time the model can spend on tasks without humans needing to intervene. Context efficiency is the sharpest form of the same bet Go made in 2009: optimize for the reader, human or machine, over the writer. The same reader-first bet shows up at the surface of the code. ships with Go and runs across the community’s code. Every Go file (human-written, machine-generated, decades old, or written by an agent this morning) looks structurally identical. Simon Willison, who has spent considerable time building AI-assisted Go tools, observed this directly : “I’m enjoying how there’s generally one obvious way to do things and the resulting code is boring and readable—and something that LLMs are very competent at writing.” When every Go file looks the same, an agent can be seeded before it starts working. Point it at Go’s standard library ( , , ) and it immediately produces idiomatic, production-quality code, with no guessing about versus or versus . Drop in something like go-skills and the seeding gets more direct still, since the agent now has explicit, reusable guidance on writing idiomatic Go rather than inferring conventions from scattered examples. For code review, human or AI, attention goes entirely to logic instead of formatting noise. Most language comparisons focus on authoring: how fast can you write code? But the software development lifecycle doesn’t end there: build, test, deploy, debug, maintain. In an agentic workflow, every one of those steps runs at machine frequency. Over the past decade, Go’s ecosystem evolved from a language into a complete SDLC platform: fuzzing became built-in, vulnerability management matured with , workspace mode unified monorepo development, and module proxies reduced version conflicts. None of this required reaching outside the Go ecosystem. Dependencies are handled by , deterministic and identical on laptops, CI, and containers, unlike or , where every resolution failure is an iteration lost. Testing needs nothing beyond : no framework decision, no fixtures to configure. Drop a file next to the package, write a -prefixed function, run one command. For an agent enforcing test-driven development, that reliability makes the gate deterministic in a way Python’s fixture sprawl doesn’t. Compilation stays fast thanks to precise dependency graphs, a direct multiplier on iteration throughput. And deployment is just : a single, self-contained binary with zero runtime dependencies, no interpreter versioning, no container startup time. Copy the binary and run it. These advantages don’t add linearly, they compound. Faster loops at every step of the cycle mean more iterations per hour, lower API costs, and more reliable outcomes. The point of this article was never “replace Python” or “replace TypeScript.” Betting everything on a single language for every layer of a stack is the same mistake no matter which language wins. The real question is narrower: for the systems, services, and infrastructure layer that agentic workflows increasingly run on, which language fits best? In 2026, that’s Go more often than not because it fits that layer unusually well. Python’s ML ecosystem (PyTorch, LangChain, Transformers) isn’t going anywhere, and it shouldn’t. For training models and running inference, Python is the answer. Go doesn’t compete there. It complements it. Go can run underneath Python just as easily as beside it. Simon Willison built a tool called that packages compiled Go binaries as Python wheels distributed on PyPI: any Go binary becomes a standard or a one-liner, and his own , a concurrent filesystem scanner written in Go, ships exactly that way. Armin Ronacher, creator of Flask, makes the same point from the other direction: porting MiniJinja from Rust to Go cost him 45 minutes and $60 in API calls, which is why he argues that “the cost of coding is going down dramatically… the breadth of an ecosystem matters less.” Python orchestrates, Go executes underneath, and the lock-in argument for picking only one ecosystem weakens every month porting between them gets cheaper. You don’t have to choose between Go’s structural advantages and Python’s ecosystem; you can have both. Go was designed to lower the total cost of software at scale, exactly the problem agentic development now amplifies to the breaking point, and every property that made Go good for human-sized teams turns out to suit machines running thousands of iterations per day just as well. None of this makes Go the only language worth using. Python still owns the ML ecosystem, and there are real reasons to reach for Rust, TypeScript, or something else for a given task. Rather than “one language to rule them all”, treat Go as your default for the systems, services, and infrastructure layer of agentic development, and require a specific reason to reach for something else. When starting a new service, CLI, or system, ask “why not Go?” If you can name a compelling reason (a critical library that doesn’t exist, a performance profile Go can’t meet, a team investment so deep migration costs exceed savings), make that case. If you can’t, Go is very likely the right default. Every percentage point lost to slower builds, flaky dependencies, or runtime surprises is API cost. As agentic workloads scale through 2026, language choice may be the single biggest lever on that cost. Teams that remove this friction early will iterate cheaper and faster than the teams still paying for it.

0 views
Jason Tucker 3 months ago

Callsheet: The Companion App for Jellyfin (and Plex, If You Must)

I saw a post from Casey Liss about Callsheet a couple days ago and asked him a simple question on Mastodon. Does it support Jellyfin yet? I already knew it worked with Plex and have been using it since it came out many years back. Casey wrote back almost immediately. It's already there. What I didn't know is that Jellyfin support had shipped back in May. It looks like Casey is also moving way from Plex, that benefits us all as it relates to this app. I was already using the Plex feature in it and didn't realize that Jellyfin was added. If you haven't run into Callsheet before, it's a cast and crew lookup app for iOS, iPadOS, macOS, and visionOS, built by Casey. You probably know him from the Accidental Tech Podcast or his other show, Analog(ue) , with Mike Hurley. He built Callsheet as his own answer to a question a lot of us have had while watching something. Who is that actor, and what else have they been in? IMDb technically answers that question. It also makes you fight through ads and clutter to get there. Callsheet skips all of that and pulls its data from TMDB instead. That alone would make it a nice utility app. The part I care about is what Callsheet does once you're actually watching something, and which server it's watching along with you. None of that is new on its own. Trakt has done some version of "what am I currently watching" for years, mostly built around scrobbling from various clients. What's different here is that Callsheet ties that playback position directly to its cast and crew database. You're not just seeing a progress bar. You can jump straight from "I'm 20 minutes into this episode" to "who plays that character" without leaving the app. The per-title progress meter is newer than I realized. Casey added it in version 2026.5, released June 20, 2026, along with the "Ends At" popover that breaks down runtime versus what you've already watched. Before that, Now Playing showed what you were watching, just not how far into it you were. How the Jellyfin integration with Now Playing in Callsheet. Callsheet has supported Plex since version 2023.4, back in September 2023, alongside a similar integration for the Channels app. For years there was only one way to connect, and Casey has been pretty open that it was never great. He's described that original mechanism as unreliable to the point that he regretted including it at all. That original approach is now called Passive mode. It talks directly to Plex clients over the local network, no login required, and it can even show what's playing on someone else's Plex server if you're on the same network as their client. The tradeoff is that it barely works with iOS and iPadOS clients, and it only works at all if you're on the same network as the playback. In 2026.5 , Casey added a second option called Active mode. It requires logging into Plex, similar to Jellyfin's Quick Connect, but in exchange it works reliably, works with iOS and iPadOS clients, and works even when you're away from home. The catch is it only shows media from your own server, not anyone else's. Casey was candid about why he built this. He's been migrating his own media setup toward Jellyfin because Plex keeps getting worse, and once he'd already broken his own no-login rule to support Jellyfin in 2026.4, adding a login option for Plex was the obvious next step. Version 2026.4 , released in May 2026, added proper Jellyfin support to the Now Playing view. Setting it up takes a few steps: Once that's done, the connection sticks around. You don't have to repeat the dance every time you open the app. It's a little embarrassing that I went two months without noticing this had shipped. But now that I've had it running for even a few days, it's held up without a single hiccup. No missed progress updates, no stale sessions, nothing to troubleshoot. There's a detail buried in the release notes that's worth calling out on its own. If you point Callsheet at a Tailscale address for your Jellyfin server, whether that's your 100.x address or a home-domain.ts.net hostname, the Jellyfin integration is the only one of the three (Plex, Channels, and Jellyfin) that actually works remotely. Plex and Channels both need Callsheet to be on the same network as whatever's doing the playback. Jellyfin doesn't have that restriction. That matches how I use Tailscale everywhere else in my setup. If I'm out of the house watching something on my phone through my Jellyfin client, Callsheet can still show me progress and cast information without me being on my home network. Authorizing the Jellyfin Integration Not entirely, and I don't think that's really the goal. Trakt tracks watch history over time, builds stats, and has a whole social layer around what you and your friends are watching. Callsheet doesn't do any of that. What it does is answer the question you have right now, mid-episode, about who's on screen and how much time is left before you need to decide whether to keep going. For me, that's actually the more useful tool most of the time. I don't check Trakt while I'm watching something. I check it after, if at all. Callsheet is the app I reach for in the moment, and now that moment includes Jellyfin instead of just Plex. I've been slowly replacing pieces of my Plex-dependent workflow for a while now, and this is one more piece that no longer needs Plex to function. Casey mentioned in that same release he's been running Jellyfin alongside Plex on his own Mac mini, I'm curious how many other developers in this space are quietly making the same switch behind the scenes and what new Jellyfin introductions are out there for apps I'm already using. Time to dig more into what I'm using already to see what I can find. In Callsheet, go to Settings, then Persnickety Preferences, then Integrations. Find the Jellyfin section and enter your server URL. Callsheet generates a Quick Connect code. Open your Jellyfin server's admin dashboard and approve that code.

0 views
dfir.ch 4 months ago

Anatomy of a Deno-Based Proxy & RAT

This is a copy of a blog post I wrote for my employee for InfoGuard LABS. Executive Summary In a recent investigation, we encountered malware that combined aggressive social engineering with the unconventional use of Deno, a secure JavaScript and TypeScript runtime built on V8. The attack began with a large-scale email flooding campaign, commonly referred to as mailbombing, designed to overwhelm employees and create confusion. Shortly afterward, the targeted users received Microsoft Teams calls from an attacker impersonating internal IT support.

0 views
Kelly Sutton 4 months ago

Moving on from React, 2 Years Later

It’s been an even busier year and change for Scholarly . We’re coming up on 3 years in business. We’ve raised a small round of funding from our existing investors, grown the team in both Denver and Seattle, and continue growing in all dimensions. I’m trying to do an annual review of a decision to move away from React in ~2023 to see how things are turning out. You can read the original posts, Moving on from React and Moving on from React, a Year Later . What a wild 18 months it’s been. Since the last post, we’ve moved from tab-completion and copy-paste LLM-aided development to full-on agents with things like Claude Code. We’ve also grown the team and we have reintroduced React (gasp!). The decision to reintroduce React was solely driven by React Flow . It’s the best diagram tool we found, and we thought it was worth eating our hat. Unlike some of the other libraries we use and pay for, it’s not currently packaged as Vanilla JavaScript. We’ve also deployed React in a select few areas where its state management yields the best customer experience. We ship this as small pieces of a page that is otherwise server-rendered. The React bits help us add the interactivity that we believe makes the best customer experience. For those keeping tabs, here’s how our Ruby/JS LOC has changed over time: A few reflections on the numbers above: Given the recent changes in software engineering, it’s hard to tell how much of this even matters anymore. Our roles as software engineers are changing with every model or harness upgrade. Agents and models have gotten a lot better at interpreting and using StimulusJS and Turbo . We use Claude Code with Opus 4.7 at the time of writing. Some of the rough edges of using Turbo with LLMs in the beginning feel completely gone now. Kept this one short to provide an update. Things are changing quickly, and it’s kind of interesting to think how much of this may or may not matter in the long run. If the LLM is writing our code and the customers have a great experience, how much does stack choice matter? Maybe we should index toward more complex technologies for humans but easier for LLMs to write? How much control should we cede? Thanks for reading. Until next time. Our codebase has 179k LOC of Ruby, compared to 61k from 18 months ago. A tripling! This can be somewhat attributed to our adoption of Sorbet for static type-checking. It just produces more verbose Ruby and provides some more safety that certain parts of our code base benefit from. Our JS LOC went from 4.1k to 14.8k in the same time frame. We’ve also adopted TypeScript here for some of our files that touch React. I’ve kept the linear trendline to simulate where we might have been with React. We’re still below where I’d predict we’d be had we stuck with it. You can clearly see where we made the cutover from React to Stimulus in August 2023, although it’s not as obvious since it’s so far in the past. Our Ruby LOC was growing super linearly last time, and that continues to be the case. I attribute this solely to Claude Code. It really whips the llama’s ass. Volume of LOC remains a liability, but the product capability has grown about this much or more in the meantime, so not concerning.

0 views
Simon Willison 5 months ago

Extract PDF text in your browser with LiteParse for the web

LlamaIndex have a most excellent open source project called LiteParse , which provides a Node.js CLI tool for extracting text from PDFs. I got a version of LiteParse working entirely in the browser, using most of the same libraries that LiteParse uses to run in Node.js. Refreshingly, LiteParse doesn't use AI models to do what it does: it's good old-fashioned PDF parsing, falling back to Tesseract OCR (or other pluggable OCR engines) for PDFs that contain images of text rather than the text itself. The hard problem that LiteParse solves is extracting text in a sensible order despite the infuriating vagaries of PDF layouts. They describe this as "spatial text parsing" - they use some very clever heuristics to detect things like multi-column layouts and group and return the text in a sensible linear flow. The LiteParse documentation describes a pattern for implementing Visual Citations with Bounding Boxes . I really like this idea: being able to answer questions from a PDF and accompany those answers with cropped, highlighted images feels like a great way of increasing the credibility of answers from RAG-style Q&A. LiteParse is provided as a pure CLI tool, designed to be used by agents. You run it like this: I explored its capabilities with Claude and quickly determined that there was no real reason it had to stay a CLI app: it's built on top of PDF.js and Tesseract.js, two libraries I've used for something similar in a browser in the past . The only reason LiteParse didn't have a pure browser-based version is that nobody had built one yet... Visit https://simonw.github.io/liteparse/ to try out LiteParse against any PDF file, running entirely in your browser. Here's what that looks like: The tool can work with or without running OCR, and can optionally display images for every page in the PDF further down the page. The process of building this started in the regular Claude app on my iPhone. I wanted to try out LiteParse myself, so I started by uploading a random PDF I happened to have on my phone along with this prompt: Regular Claude chat can clone directly from GitHub these days, and while by default it can't access most of the internet from its container it can also install packages from PyPI and npm. I often use this to try out new pieces of open source software on my phone - it's a quick way to exercise something without having to sit down with my laptop. You can follow my full conversation in this shared Claude transcript . I asked a few follow-up questions about how it worked, and then asked: This gave me a thorough enough answer that I was convinced it was worth trying getting that to work for real. I opened up my laptop and switched to Claude Code. I forked the original repo on GitHub, cloned a local copy, started a new branch and pasted that last reply from Claude into a new file called notes.md . Then I told Claude Code: I always like to start with a plan for this kind of project. Sometimes I'll use Claude's "planning mode", but in this case I knew I'd want the plan as an artifact in the repository so I told it to write directly. This also means I can iterate on the plan with Claude. I noticed that Claude had decided to punt on generating screenshots of images in the PDF, and suggested we defer a "canvas-encode swap" to v2. I fixed that by prompting: After a few short follow-up prompts, here's the plan.md I thought was strong enough to implement. I prompted: And then mostly left Claude Code to its own devices, tinkered with some other projects, caught up on Duolingo and occasionally checked in to see how it was doing. I added a few prompts to the queue as I was working. Those don't yet show up in my exported transcript, but it turns out running in the relevant folder extracts them. Here are the key follow-up prompts with some notes: I've started habitually asking for "small commits along the way" because it makes for code that's easier to understand or review later on, and I have an unproven hunch that it helps the agent work more effectively too - it's yet another encouragement towards planning and taking on one problem at a time. While it was working I decided it would be nice to be able to interact with an in-progress version. I asked a separate Claude Code session against the same directory for tips on how to run it, and it told me to use . Running that started a development server with live-reloading, which meant I could instantly see the effect of each change it made on disk - and prompt with further requests for tweaks and fixes. Towards the end I decided it was going to be good enough to publish. I started a fresh Claude Code instance and told it: After a bit more iteration here's the GitHub Actions workflow that builds the app using Vite and deploys the result to https://simonw.github.io/liteparse/ . I love GitHub Pages for this kind of thing because it can be quickly configured (by Claude, in this case) to turn any repository into a deployed web-app, at zero cost and with whatever build step is necessary. It even works against private repos, if you don't mind your only security being a secret URL. With this kind of project there's always a major risk that the model might "cheat" - mark key features as "TODO" and fake them, or take shortcuts that ignore the initial requirements. The responsible way to prevent this is to review all of the code... but this wasn't intended as that kind of project, so instead I fired up OpenAI Codex with GPT-5.5 (I had preview access) and told it: The answer I got back was enough to give me confidence that Claude hadn't taken any project-threatening shortcuts. ... and that was about it. Total time in Claude Code for that "build it" step was 59 minutes. I used my claude-code-transcripts tool to export a readable version of the full transcript which you can view here , albeit without those additional queued prompts (here's my issue to fix that ). I'm a pedantic stickler when it comes to the original definition of vibe coding - vibe coding does not mean any time you use AI to help you write code, it's when you use AI without reviewing or caring about the code that's written at all. By my own definition, this LiteParse for the web project is about as pure vibe coding as you can get! I have not looked at a single line of the HTML and TypeScript written for this project - in fact while writing this sentence I had to go and check if it had used JavaScript or TypeScript. Yet somehow this one doesn't feel as vibe coded to me as many of my other vibe coded projects: Most importantly, I'm happy to attach my reputation to this project and recommend that other people try it out. Unlike most of my vibe coded tools I'm not convinced that spending significant additional engineering time on this would have resulted in a meaningfully better initial release. It's fine as it is! I haven't opened a PR against the origin repository because I've not discussed it with the LiteParse team. I've opened an issue , and if they want my vibe coded implementation as a starting point for something more official they're welcome to take it. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . - I've written more about red/green TDD here . (it was messing around with pdfium) - I had a new idea for how the UI should work - see below - it's important to credit your dependencies in a project like this! - it was testing with Playwright in Chrome, turned out there was a bug in Safari - dropping screenshots in of small UI glitches works surprisingly well - it still wasn't working in Safari... - but it fixed it pretty quickly once I pointed that out and it got Playwright working with that browser As a static in-browser web application hosted on GitHub Pages the blast radius for any bugs is almost non-existent: it either works for your PDF or doesn't. No private data is transferred anywhere - all processing happens in your browser - so a security audit is unnecessary. I've glanced once at the network panel while it's running and no additional requests are made when a PDF is being parsed. There was still a whole lot of engineering experience and knowledge required to use the models in this way. Identifying that porting LiteParse to run directly in a browser was critical to the rest of the project.

0 views
Armin Ronacher 6 months ago

Absurd In Production

About five months ago I wrote about Absurd , a durable execution system we built for our own use at Earendil, sitting entirely on top of Postgres and Postgres alone. The pitch was simple: you don’t need a separate service , a compiler plugin , or an entire runtime to get durable workflows. You need a SQL file and a thin SDK. Since then we’ve been running it in production, and I figured it’s worth sharing what the experience has been like. The short version: the design held up, the system has been a pleasure to work with, and other people seem to agree. Absurd is a durable execution system that lives entirely inside Postgres. The core is a single SQL file ( absurd.sql ) that defines stored procedures for task management, checkpoint storage, event handling, and claim-based scheduling. On top of that sit thin SDKs (currently TypeScript , Python and an experimental Go one) that make the system ergonomic in your language of choice. The model is straightforward: you register tasks, decompose them into steps, and each step acts as a checkpoint. If anything fails, the task retries from the last completed step. Tasks can sleep, wait for external events, and suspend for days or weeks. All state lives in Postgres. If you want the full introduction, the original blog post covers the fundamentals. What follows here is what we’ve learned since. The project got multiple releases over the last five months. Most of the changes are things you’d expect from a system that people actually started depending on: hardened claim handling, watchdogs that terminate broken workers, deadlock prevention, proper lease management, event race conditions, and all the edge cases that only show up when you’re running real workloads. A few things worth calling out specifically. Decomposed steps. The original design only had , where you pass in a function and get back its checkpointed result. That works well for many cases but not all. Sometimes you need to know whether a step already ran before deciding what to do next. So we added / , which give you a handle you can inspect before committing the result. This turned out to be very useful for modeling intentional failures and conditional logic. This in particular is necessary when working with “before call” and “after call” type hook APIs. Task results. You can now spawn a task, go do other things, and later come back to fetch or await its result. This sounds obvious in hindsight, but the original system was purely fire-and-forget. Having proper result inspection made it possible to use Absurd for things like spawning child tasks from within a parent workflow and waiting for them to finish. This is particularly useful for debugging with agents too. absurdctl . We built this out as a proper CLI tool. You can initialize schemas, run migrations, create queues, spawn tasks, emit events, retry failures from the command line. It’s installable via or as a standalone binary. This has been invaluable for debugging production issues. When something is stuck, being able to just and see exactly where it stopped is a very different experience from digging through logs. Habitat . A small Go application that serves up a web dashboard for monitoring tasks, runs, checkpoints, and events. It connects directly to Postgres and gives you a live view of what’s happening. It’s simple, but it’s the kind of thing that makes the system more enjoyable for humans. Agent integration. Since Absurd was originally built for agent workloads, we added a bundled skill that coding agents can discover and use to debug workflow state via . There’s also a documented pattern for making pi agent turns durable by logging each message as a checkpoint. The thing I’m most pleased about is that the core design didn’t need to change all that much. The fundamental model of tasks, steps, checkpoints, events, and suspending is still exactly what it was initially. We added features around it, but nothing forced us to rethink the basic abstractions. Putting the complexity in SQL and keeping the SDKs thin turned out to be a genuinely good call. The TypeScript SDK is about 1,400 lines. The Python SDK is about 1,900 but most of this comes from the complexity of supporting colored functions. Compare that to Temporal’s Python SDK at around 170,000 lines. It means the SDKs are easy to understand, easy to debug, and easy to port. When something goes wrong, you can read the entire SDK in an afternoon and understand what it does. The checkpoint-based replay model also aged well. Unlike systems that require deterministic replay of your entire workflow function, Absurd just loads the cached step results and skips over completed work. That means your code doesn’t need to be deterministic outside of steps. You can call or in between steps and things still work, because only the step boundaries matter. In practice, this makes it much easier to reason about what’s safe and what isn’t. Pull-based scheduling was the right choice too. Workers pull tasks from Postgres as they have capacity. There’s no coordinator, no push mechanism, no HTTP callbacks. That makes it trivially self-hostable and means you don’t have to think about load management at the infrastructure level. I had some discussions with folks about whether the right abstraction should have been a durable promise . It’s a very appealing idea, but it turns out to be much more complex to implement in practice. It’s however in theory also more powerful. I did make some attempts to see what absurd would look like if it was based on durable promises but so far did not get anywhere with it. It’s however an experiment that I think would be fun to try! The primary use case is still agent workflows. An agent is essentially a loop that calls an LLM, processes tool results, and repeats until it decides it’s done. Each iteration becomes a step, and each step’s result is checkpointed. If the process dies on iteration 7, it restarts and replays iterations 1 through 6 from the store, then continues from 7. But we’ve found it useful for a lot of other things too. All our crons just dispatch distributed workflows with a pre-generated deduplication key from the invocation. We can have two cron processes running and they will only trigger one absurd task invocation. We also use it for background processing that needs to survive deploys. Basically anything where you’d otherwise build your own retry-and-resume logic on top of a queue. Absurd is deliberately minimal, but there are things I’d like to see. There’s no built-in scheduler. If you want cron-like behavior, you run your own scheduler loop and use idempotency keys to deduplicate. That works, and we have a documented pattern for it , but it would be nice to have something more integrated. There’s no push model. Everything is pull. If you need an HTTP endpoint to receive webhooks and wake up tasks, you build that yourself. I think that’s the right default as push systems are harder to operate and easier to overwhelm but there are cases where it would be convenient. In particular there are quite a few agentic systems where it would be super nice to have webhooks natively integrated (wake on incoming POST request). I definitely don’t want to have this in the core, but that sounds like the kind of problem that could be a nice adjacent library that builds on top of absurd. The biggest omission is that it does not support partitioning yet. That’s unfortunate because it makes cleaning up data more expensive than it has to be. In theory supporting partitions would be pretty simple. You could have weekly partitions and then detach and delete them when they expire. The only thing that really stands in the way of that is that Postgres does not have a convenient way of actually doing that. The hard part is not partitioning itself, it’s partition lifecycle management under real workloads. If a worker inserts a row whose lands in a month without a partition, the insert fails and the workflow crashes. So you need a separate maintenance loop that always creates future partitions far enough ahead for sleeps/retries, and does that for every queue. On the delete side, the safe approach is , but getting that to run from doesn’t work because it cannot be run within a transaction, but runs everything in one. I don’t think it’s an unsolvable problem, but it’s one I have not found a good solution for and I would love to get input on . This brings me a bit to a meta point on the whole thing which is what the point of Open Source libraries in the age of agentic engineering is. Durable Execution is now something that plenty of startups sell you. On the other hand it’s also something that an agent would build you and people might not even look for solutions any more. It’s kind of … weird? I don’t think a durable execution library can support a company, I really don’t. On the other hand I think it’s just complex enough of a problem that it could be a good Open Source project void of commercial interests. You do need a bit of an ecosystem around it, particularly for UI and good DX for debugging, and that’s hard to get from a throwaway implementation. I don’t think we have squared this yet, but it’s already much better to use than a few months ago. If you’re using Absurd, thinking about it, or building adjacent ideas, I’d love your feedback. Bug reports, rough edges, design critiques, and contributions are all very welcome—this project has gotten better every time someone poked at it from a different angle.

0 views
Martin Fowler 6 months ago

Fragments: April 2

As we see LLMs churn out scads of code, folks have increasingly turned to Cognitive Debt as a metaphor for capturing how a team can lose understanding of what a system does. Margaret-Anne Storey thinks a good way of thinking about these problems is to consider three layers of system health : While I’m getting a bit bemused by debt metaphor proliferation, this way of thinking does make a fair bit of sense. The article includes useful sections to diagnose and mitigate each kind of debt. The three interact with each other, and the article outlines some general activities teams should do to keep it all under control ❄                ❄ In the article she references a recent paper by Shaw and Nave at the Wharton School that adds LLMs to Kahneman’s two-system model of thinking . Kahneman’s book, “Thinking Fast and Slow”, is one of my favorite books. Its central idea is that humans have two systems of cognition. System 1 (intuition) makes rapid decisions, often barely-consciously. System 2 (deliberation) is when we apply deliberate thinking to a problem. He observed that to save energy we default to intuition, and that sometimes gets us into trouble when we overlook things that we would have spotted had we applied deliberation to the problem. Shaw and Nave consider AI as System 3 A consequence of System 3 is the introduction of cognitive surrender, characterized by uncritical reliance on externally generated artificial reasoning, bypassing System 2. Crucially, we distinguish cognitive surrender, marked by passive trust and uncritical evaluation of external information, from cognitive offloading, which involves strategic delegation of cognition during deliberation. It’s a long paper, that does into detail on this “Tri-System theory of cognition” and reports on several experiments they’ve done to test how well this theory can predict behavior (at least within a lab). ❄                ❄                ❄                ❄                ❄ I’ve seen a few illustrations recently that use the symbols “< >” as part of an icon to illustrate code. That strikes me as rather odd, I can’t think of any programming language that uses “< >” to surround program elements. Why that and not, say, “{ }”? Obviously the reason is that they are thinking of HTML (or maybe XML), which is even more obvious when they use “</>” in their icons. But programmers don’t program in HTML. ❄                ❄                ❄                ❄                ❄ Ajey Gore thinks about if coding agents make coding free, what becomes the expensive thing ? His answer is verification. What does “correct” mean for an ETA algorithm in Jakarta traffic versus Ho Chi Minh City? What does a “successful” driver allocation look like when you’re balancing earnings fairness, customer wait time, and fleet utilisation simultaneously? When hundreds of engineers are shipping into ~900 microservices around the clock, “correct” isn’t one definition — it’s thousands of definitions, all shifting, all context-dependent. These aren’t edge cases. They’re the entire job. And they’re precisely the kind of judgment that agents cannot perform for you. Increasingly I’m seeing a view that agents do really well when they have good, preferably automated, verification for their work. This encourages such things as Test Driven Development . That’s still a lot of verification to do, which suggests we should see more effort to find ways to make it easier for humans to comprehend larger ranges of tests. While I agree with most of what Ajey writes here, I do have a quibble with his view of legacy migration. He thinks it’s a delusion that “agentic coding will finally crack legacy modernisation”. I agree with him that agentic coding is overrated in a legacy context, but I have seen compelling evidence that LLMs help a great deal in understanding what legacy code is doing . The big consequence of Ajey’s assessment is that we’ll need to reorganize around verification rather than writing code: If agents handle execution, the human job becomes designing verification systems, defining quality, and handling the ambiguous cases agents can’t resolve. Your org chart should reflect this. Practically, this means your Monday morning standup changes. Instead of “what did we ship?” the question becomes “what did we validate?” Instead of tracking output, you’re tracking whether the output was right. The team that used to have ten engineers building features now has three engineers and seven people defining acceptance criteria, designing test harnesses, and monitoring outcomes. That’s the reorganisation. It’s uncomfortable because it demotes the act of building and promotes the act of judging. Most engineering cultures resist this. The ones that don’t will win. ❄                ❄                ❄                ❄                ❄ One the questions comes up when we think of LLMs-as-programmers is whether there is a future for source code. David Cassel on The New Stack has an article summarizing several views of the future of code . Some folks are experimenting with entirely new languages built with the LLM in mind, others think that existing languages, especially strictly typed languages like TypeScript and Rust will be the best fit for LLMs. It’s an overview article, one that has lots of quotations, but not much analysis in itself - but it’s worth a read as a good overview of the discussion. I’m interested to see how all this will play out. I do think there’s still a role for humans to work with LLMs to build useful abstractions in which to talk about what the code does - essentially the DDD notion of Ubiquitous Language . Last year Unmesh and I talked about growing a language with LLMs. As Unmesh put it Programming isn’t just typing coding syntax that computers can understand and execute; it’s shaping a solution. We slice the problem into focused pieces, bind related data and behaviour together, and—crucially—choose names that expose intent. Good names cut through complexity and turn code into a schematic everyone can follow. The most creative act is this continual weaving of names that reveal the structure of the solution that maps clearly to the problem we are trying to solve. Technical debt lives in code. It accumulates when implementation decisions compromise future changeability. It limits how systems can change. Cognitive debt lives in people. It accumulates when shared understanding of the system erodes faster than it is replenished. It limits how teams can reason about change. Intent debt lives in artifacts. It accumulates when the goals and constraints that should guide the system are poorly captured or maintained. It limits whether the system continues to reflect what we meant to build and it limits how humans and AI agents can continue to evolve the system effectively.

0 views
David Bushell 6 months ago

404 Deno CEO not found

I visited deno.com yesterday. I wanted to know if the hundreds of hours I’d spent mastering Deno was a sunk cost. Do I continue building for the runtime, or go back to Node? Well I guess that pretty much sums up why a good chunk of Deno employees left the company over the last week. Layoffs are what American corpo culture calls firing half the staff. Totally normal practice for a sustainable business. Mass layoffs are deemed better for the moral of those who remain than a weekly culling before Friday beers. The Romans loved a good decimation. † If I were a purveyor of slop and tortured metaphors, I’d have adorned this post with a deepfake of Ryan Dahl fiddling as Deno burned. But I’m not, so the solemn screenshot will suffice. † I read Rome, Inc. recently. Not a great book, I’m just explaining the reference. A year ago I wrote about Deno’s decline . The facts, undeterred by my subjective scorn, painted a harsh picture; Deno Land Inc. was failing. Deno incorporated with $4.9M of seed capital five years ago. They raised a further $21M series A a year later. Napkin math suggests a five year runway for an unprofitable company (I have no idea, I just made that up.) Coincidentally, after my blog post topped Hacker News — always a pleasure for my inbox — Ryan Dahl (Deno CEO) clapped back on the offical Deno blog: There’s been some criticism lately about Deno - about Deploy, KV, Fresh, and our momentum in general. You may have seen some of the criticism online; it’s made the rounds in the usual places, and attracted a fair amount of attention. Some of that criticism is valid. In fact, I think it’s fair to say we’ve had a hand in causing some amount of fear and uncertainty by being too quiet about what we’re working on, and the future direction of our company and products. That’s on us. Reports of Deno’s Demise Have Been Greatly Exaggerated - Ryan Dahl Dahl mentioned that adoption had doubled following Deno 2.0. Since the release of Deno 2 last October - barely over six months ago! - Deno adoption has more than doubled according to our monthly active user metrics. User base doubling sounds like a flex for a lemonade stand unless you give numbers. I imagine Sequoia Capital expected faster growth regardless. The harsh truth is that Deno’s offerings have failed to capture developers’ attention. I can’t pretend to know why — I was a fanboy myself — but far too few devs care about Deno. On the rare occasions Deno gets attention on the orange site, the comments page reads like in memoriam . I don’t even think the problem was that Deno Deploy, the main source of revenue, sucked. Deploy was plagued by highly inconsistent isolate start times . Solicited feedback was ignored. Few cared. It took an issue from Wes Bos , one of the most followed devs in the game, for anyone at Deno to wake up. Was Deploy simply a ghost town? Deno rushed the Deploy relaunched for the end of 2025 and it became “generally available” last month. Anyone using it? Anyone care? The Deno layoffs this week suggest only a miracle would have saved jobs. The writing was on the wall. Speaking of ghost towns, the JSR YouTube channel is so lonely I feel bad for linking it. I only do because it shows just how little interest some Deno-led projects mustered. JSR floundered partly because Deno was unwilling couldn’t afford to invest in better infrastructure . But like everything else in the Deno ecosystem, users just weren’t interested. What makes a comparable project like NPMX flourish so quickly? Evidently, developers don’t want to replace Node and NPM. They just want what they already have but better; a drop-in improvement without friction. To Deno and Dahl’s credit, they recognised this with the U-turn on HTTP imports . But the resulting packaging mess made things worse. JSR should have been NPMX. Deno should have gone all-in on but instead we got mixed messaging and confused docs. I could continue but it would just be cruel to dissect further. I’ve been heavily critical of Deno in the past but I really wanted it to succeed. There were genuinely good people working at Deno who lost their job and that sucks. I hope the Deno runtime survives. It’s a breath of fresh air. B*n has far more bugs and compatibility issues than anyone will admit. Node still has too much friction around TypeScript and ECMAScript modules. So where does Deno go from here? Over to you, Ryan. Where is Deno CEO, Ryan Dahl? Tradition dictates an official PR statement following layoffs. Seems weird not to have one prepared in advance. That said, today is Friday, the day to bury bad news. I may be publishing this mere hours before we hear what happens next… Given Dahl’s recent tweets and blog post , a pivot to AI might be Deno’s gamble. By the way, it’s rather telling that all the ex-employees posted their departures on Bluesky. What that tells you depends on whether you enjoy your social media alongside Grok undressing women upon request. I digress. Idle speculation has led to baseless rumours of an OpenAI acquisition. I’m not convinced that makes sense but neither does the entire AI industry. I’m not trying to hate on Dahl but c’mon bro you’re the CEO. What’s next for Deno? Give me users anyone a reason to care. Although if you’re planning a 10× resurgence with automated Mac Minis, I regret asking. Thanks for reading! Follow me on Mastodon and Bluesky . Subscribe to my Blog and Notes or Combined feeds.

0 views
David Bushell 7 months ago

SvelteKit i18n and FOWL

Perhaps my favourite JavaScript APIs live within the Internationalization namespace. A few neat things the global allows: It’s powerful stuff and the browser or runtime provides locale data for free! That means timezones, translations, and local conventions are handled for you. Remember moment.js? That library with locale data is over 600 KB (uncompressed). That’s why JavaScript now has the Internationalization API built-in. SvelteKit and similar JavaScript web frameworks allow you to render a web page server-side and “hydrate” in the browser. In theory , you get the benefits of an accessible static website with the progressively enhanced delights of a modern “web app”. I’m building attic.social with SvelteKit. It’s an experiment without much direction. I added a bookmarks feature and used to format dates. Perfect! Or was it? Disaster strikes! See this GIF: What is happening here? Because I don’t specify any locale argument in the constructor it uses the runtime’s default. When left unconfigured, many environments will default to . I spotted this bug only in production because I’m hosting on a Cloudflare worker. SvelteKit’s first render is server-side using but subsequent renders use in my browser. My eyes are briefly sullied by the inferior US format! Is there a name for this effect? If not I’m coining: “Flash of Wrong Locale” (FOWL). To combat FOWL we must ensure that SvelteKit has the user’s locale before any templates are rendered. Browsers may request a page with the HTTP header. The place to read headers is hooks.server.ts . I’ve vendored the @std/http negotiation library to parse the request header. If no locales are provided it returns which I change to . SvelteKit’s is an object to store custom data for the lifetime of a single request. Event are not directly accessible to SvelteKit templates. That could be dangerous. We must use a page or layout load function to forward the data. Now we can update the original example to use the data. I don’t think the rune is strictly necessary but it stops a compiler warning . This should eliminate FOWL unless the header is missing. Privacy focused browsers like Mullvad Browser use a generic header to avoid fingerprinting. That means users opt-out of internationalisation but FOWL is still gone. If there is a cache in front of the server that must vary based on the header. Otherwise one visitor defines the locale for everyone who follows unless something like a session cookie bypasses the cache. You could provide a custom locale preference to override browser settings. I’ve done that before for larger SvelteKit projects. Link that to a session and store it in a cookie, or database. Naturally, someone will complain they don’t like the format they’re given. This blog post is guaranteed to elicit such a comment. You can’t win! Why can’t you be normal, Safari? Despite using the exact same locale, Safari still commits FOWL by using an “at” word instead of a comma. Who’s fault is this? The ECMAScript standard recommends using data from Unicode CLDR . I don’t feel inclined to dig deeper. It’s a JavaScriptCore quirk because Bun does the same. That is unfortunate because it means the standard is not quite standard across runtimes. By the way, the i18n and l10n abbreviations are kinda lame to be honest. It’s a fault of my design choices that “internationalisation” didn’t fit well in my title. Thanks for reading! Follow me on Mastodon and Bluesky . Subscribe to my Blog and Notes or Combined feeds. Natural alphanumeric sorting Relative date and times Currency formatting

0 views
David Bushell 7 months ago

Building on AT Protocol

At Protocol has got me! I’m morphing into an atmosphere nerd . AT Protocol — atproto for short — is the underlying tech that powers Bluesky and new social web apps. Atproto as I understand it is largely an authorization and data layer. All atproto data is inherently public. In theory it can be encrypted for private use but leaky metadata and de-anonymisation is a whole thing. Atproto users own the keys to their data which is stored on a Personal Data Server (PDS). You don’t need to manage your own. If you don’t know where your data is stored, good chance it’s on Bluesky’s PDS. You can move your data to another PDS like Blacksky or Eurosky . Or if you’re a nerd like me self-host your own PDS . You own your data and no PDS can stop you moving it. Atproto provides OAuth; think “Sign in with GitHub” . But instead of an account being locked behind the whims of proprietary slopware, user identity is proven via their PDS. Social apps like Bluesky host a PDS allowing users to create a new account. That account can be used to login to other apps like pckt , Leaflet , or Tangled . You could start a new account on Tangled’s PDS and use that for Bluesky. Atproto apps are not required to provide a PDS but it helps to onboard new users. Of course I did. You can sign in at attic.social Attic is a cozy space with lofty ambitions. What does Attic do? I’m still deciding… it’ll probably become a random assortment of features. Right now it has bookmarks. Bookmarks will have search and tags soon. Technical details: to keep the server stateless I borrowed ideas from my old SvelteKit auth experiment. OAuth and session state is stored in encrypted HTTP-only cookies. I used the atcute TypeScript libraries to do the heavy atproto work. I found @flo-bit’s projects which helped me understand implementation details. Attic is on Cloudflare workers for now. When I’ve free time I’ll explore the SvelteKit Bunny adapter . I am busy on client projects so I’ll be scheming Attic ideas in my free time. What’s so powerful about atproto is that users can move their account/data. Apps write data to a PDS using a lexicon ; a convention to say: “this is a Bluesky post”, for example. Other apps are free to read that data too. During authorization, apps must ask for permission to write to specific lexicons. The user is in control. You may have heard that Bluesky is or isn’t “decentralised”. Bluesky was simply the first atproto app. Most users start on Bluesky and may never be aware of the AT Protocol. What’s important is that atproto makes it difficult for Bluesky to “pull a Twitter”, i.e. kill 3rd party apps, such as the alternate Witchsky . If I ever abandon attic.social your data is still in your hands. Even if the domain expires! You can extract data from your PDS. You can write a new app to consume it anytime. That’s the power of AT Protocol. Thanks for reading! Follow me on Mastodon and Bluesky . Subscribe to my Blog and Notes or Combined feeds.

0 views
Allen Pike 7 months ago

Launch Now

Inside us are two wolves. One wolf wants to craft, polish and refine – make things of exceptional quality. The other wolf wants to move fast and get feedback now. The two wolves don’t always get along. For years, I’ve balanced this by working toward exceptional products but constantly collecting private feedback along the way. Then, once we’ve built something excellent, something worthy of attention, we launch it to the world with appropriate fanfare. Videos, marketing campaigns, polished onboarding, and so on. “Here’s something worth trying, we think you’ll really like it.” This totally works. At least, it works as a path to eventually ship high-quality software. Polished, usable, even delightful software. But when it comes to building something people will pay for, it’s neither reliable nor fast. Our first product at Forestwalk was a developer tool – a platform for building and running evaluations of LLM-powered apps . We learned a ton building it, but after a few months – as we approached our first pilot projects – feedback from demos and potential first customers convinced us that this was the wrong path. It was more likely to lead us into a lifestyle business than something big. So we pivoted. We spent a few weeks building a prototype a week, showing demos, doing customer research, and found a second promising product path. Our second product was a productivity tool – a work assistant that could capture, organize, and rationalize teams’ tasks . We learned a ton building it, but after a few months – as we approached a public beta – feedback from private testers and our investors convinced us that this was the wrong path. It was more likely to lead us into a lifestyle business than something big. So we pivoted. The third time purports to be the charm. But at the same time, doing the same thing over and over typically gets the same results. We need to build something profoundly useful, something people really want. We can’t keep hiding away, sending out private demos and prototypes, not fully shipping anything! So, we decided to push harder into the discomfort of showing our work early. Just before Christmas, we decided to commit to something and work towards getting it shipped. This third product is codenamed Cedarloop 1 . It’s a realtime meeting agent. Unlike AIs that passively listen in to meetings and just write up notes after the fact, Cedar joins calls and uses “voice in, visuals out” to screen-share useful observations and perform routine tasks live during a Google Meet or Zoom meeting. The vision is to build a kind of agentic PM assistant. It can respond within a second of you talking 2 , which – when it works – feels like magic. We’ve been learning a lot building it. Recently, we started working with an excellent designer here in Vancouver who was keen to get going. I’d like to do some user testing. What do people say when you let them try it? Well, obviously it’s so early right now. They won’t like it. The inference and onboarding need more work. But we’ve been doing research about problems, needs, willingness to pay, and things like that. Sure… but we should also let people try it. What if we launched now? Well, obviously we can’t launch now . I mean… obviously. Launching now would be embarrassing. It’s not my brand to launch something publicly that’s not ready. On the other hand… I keep a printed copy of Y Combinator’s list of essential startup advice on my desk. And if you know YC, you’ll know that the first point of advice is “Launch now”. Only last month I was interviewing Brett Huneycutt, Wealthsimple’s co-founder . He had a lot of great stories, but one that sticks out is that even as a $10B company, they prioritize launching “now”, for as close as they can get to that definition. It’s not just about speed: a rapid feedback loop is a core ingredient in getting to quality. So we launched now. As of today, people can check out our research-preview realtime meeting agent at Cedarloop.ai . With luck, they’ll report issues, inform what we should prioritize next, and tell us what problems they’d love to have automated away. We’re only a few hours in, and yep – people are reporting issues. Linear integration had an OAuth issue. Login didn’t work in social-media webviews. We’ve been so focused on the desktop experience that we’ve let the mobile layout get janky. This is embarrassing! But also, there’s signal. People are trying the Linear integration. Our desktop-focused app is being discovered on mobile. Folks care enough to click at all. And in a week or so, we’ll have a smoother onboarding flow than we would have gotten to with weeks of private user tests. So it’s worth the pain. We’re going to take the feedback, follow the signal, learn and re-learn, and do better. We’ll use it to forge the best damn live agent ever – or, if the feedback peters out, we’ll know we’re on the wrong path, and find the right one. In the meantime, there’s a lot to do. 3 Back to work! This is not a good name yet. For example, sometimes iOS mishears “Hey Cedar” as “Hey Siri”. But part of our move-fast strategy is to worry more about names once we’ve proven something has traction. At that point, we’ll put in the work to give it the right name – and eventually rename the company after it. ↩ It’s fascinating how much you can do to get LLM response times down. Our first prototype often took over 8000ms to respond, which doesn’t feel live at all. Once we got it under ~1200ms, voice-in-vision-out suddenly felt alive – a step change. We have a lot of work planned to get Cedarloop even faster and much more reliable, which I’m keen to write about when I can. ↩ Speaking of having a lot to do: if you’re an experienced product-minded developer in Vancouver who would be excited to iterate and build out realtime agents using LLMs and TypeScript, we’re hiring a Founding Engineer . Just sayin’. ↩ This is not a good name yet. For example, sometimes iOS mishears “Hey Cedar” as “Hey Siri”. But part of our move-fast strategy is to worry more about names once we’ve proven something has traction. At that point, we’ll put in the work to give it the right name – and eventually rename the company after it. ↩ It’s fascinating how much you can do to get LLM response times down. Our first prototype often took over 8000ms to respond, which doesn’t feel live at all. Once we got it under ~1200ms, voice-in-vision-out suddenly felt alive – a step change. We have a lot of work planned to get Cedarloop even faster and much more reliable, which I’m keen to write about when I can. ↩ Speaking of having a lot to do: if you’re an experienced product-minded developer in Vancouver who would be excited to iterate and build out realtime agents using LLMs and TypeScript, we’re hiring a Founding Engineer . Just sayin’. ↩

0 views
Justin Duke 7 months ago

Unshipping Keystatic

Two years after initially adopting it , we've formally unshipped Keystatic . Our CMS, such as it is, is now a bunch of Markdoc files and a TypeScript schema organizing the front matter — which is to say, it's not really a CMS at all. There were a handful of reasons for this move, in no specific order: That last point is basically what I wrote about Invoke — it's a terrible heuristic, judging a project by its commit frequency, and I know that. Things can and should be finished! And yet. When you're already on the fence, a quiet GitHub graph is the thing that tips you over. To Keystatic's credit, it was tremendously easy to extricate. The whole migration was maybe two hours of work, most of which was just deleting code. That's the sign of a well-designed library — one that doesn't metastasize into every corner of your codebase. I wish more tools were this easy to leave. Our team's use of Keystatic as an actual front-end CMS had dropped to zero. All of the non-coders have grown sufficiently adept with Markdown that the GUI was gathering dust; Keystatic had become a pure schema validation and rendering tool, and offered fairly little beyond what we were already getting from our build step. Some of the theoretically nice things — image hosting, better previewing — either didn't work as smoothly as we'd like or were supplanted entirely by Vercel's built-in features. The project appears to have atrophied a little bit, commits dwindling into the one-per-quarter frequency despite a healthy number of open issues. This is not to besmirch the lovely maintainers, who have many other things going on. But it's harder to stick around on a library you're not getting much value from when you're also worried there's not a lot of momentum down the road.

0 views
Luke Hsiao 7 months ago

Using Changesets in a polyglot monorepo

One of the nice things about working in a smaller business is you can enjoy using things that don’t need to scale to extreme sizes. One example in the software world is monorepos . While monorepos can scale well (see Google, Facebook, and others), doing so requires special tooling and infrastructure. With plain , you can only go so far . While you can use it, it has meaningful advantages, like being able to make atomic changes that affect many parts of the system in a single commit, which eliminates whole classes of compatibility and integration issues. You can always split a monorepo later (see ). So, suppose you’re a small-to-medium team using a monorepo. Let’s go further and say that this monorepo stores all your company’s code, meaning it spans many different programming languages—it’s a polyglot monorepo. What tool can you use to manage versioning in a consistent way? I argue that is a solid choice, even if it’s primarily focused on the JavaScript/TypeScript ecosystem. Note There is an open discussion about native polyglot monorepo support . However, even without native support, has the hooks needed to implement this as-is for many setups. For any versioning tool, you are typically looking for how to: assumes per-package semantic versioning (i.e., all packages have their own version). In addition, each package has its own . The team also has a GitHub Action, which importantly allows specifying custom scripts for the and commands. That customization is what gives support for polyglot repositories. In , engineers commit “changeset” files to the repository that define what content ends up in changelogs, and what packages versions are bumped (i.e., major, minor, patch). See the documentation for more details. I’m a fan of . I also really like scripts . The example below uses both. I’m also going to assume you are in a enterprise setting where all your monorepo is private, not open-source. My recommended organization (at least at time of writing) is something like the following. Put all packages in a directory, no matter what language they are. I also enjoy having documentation as code , so let’s say you have a directory, too, and that your docs is written in a javascript-based frontend (like Starlight ), for the purposes of highlighting a nuance later. With this setup, you can configure with a proxy workspace at the root with all your packages. And, declare your dependencies: You should now also update your : Because is built for JavaScript, we also need “proxy” files for all of our packages; uses these to perform version bumps. These can be as simple as: With this setup, note how we are intentionally trying to exclude our internal as a pnpm workspace member—we only want to version packages. To do so, declare the directory its own workspace, otherwise it will try and combine the dependencies into the root . This can be as simple as: Next, we can add our changeset configuration: The glue to create polyglot versioning PRs Next, we want to automate our releases. That is, generating the changelog PRs, bumping package metadata, pushing tags, and triggering builds on those tags. Let’s start with our GitHub Workflow definition, and unpack the scripts it calls. Tip You might be wondering why we run a workflow explicitly, rather than using something like as a trigger. It turns out GitHub has two fatal flaws with that intuitive approach (at time of writing). First, if you push >3 tags at once, workflows will not trigger . Unfortunately, this is a relatively common scenario in a monorepo. Second, GitHub’s triggering of is highly unreliable . This unreliability is still present, even if you use a PAT as they instruct . So, instead consider an explicit for the purpose as I’ve done here. Setting is the key to polyglot support. The meat of the glue for polyglot support then, is how you implement . The key bit here is we rely on to bump the versions in for us when we call , but then it is up to us to propagate that version to the respective language’s metadata appropriately. Here is an example that uses pretty naive parsing. You can write something similar (or better!) for the languages you use. In the standard flow, you will now have a pull request on GitHub with the appropriate updates, as well as the metadata updates for all the relevant packages. Once that is merged, the very same action will run, realize all the files are consumed, and push tags. Note With our example configuration, will only push tags, not publish packages, because we set in our and have all the packages set to . Typically, you’ll then want to react to these pushed tags. For example, to build new docker images. For that, rather than using an trigger as you would reasonably assume, you probably want a . See the tip earlier in the post for why. can manage per-package semantic versioning and changelogs in polyglot monorepos today, even without direct native support for multiple languages. The trick is to treat JavaScript package manifests as the canonical source of version bumps, and then syncing those bumps via your own scripts to the language-native manifests. A few gotchas exist (like explicitly making independent files for subdirectories you want to be independent, or using a separate personal access token to push tags), but none are blockers to being able to benefit from ’s convenient workflow. I used to suggest versioning monorepos with a single global version using a tool like . Since trying , I’m sold on the benefits of letting people write commit messages for future internal engineers while also adding a separate changelog note for end users. These are often two distinct audiences, and relying on a single conventional commit to serve both is often suboptimal. define what content appears in the changelog/release notes influence the version numbers of the packages automate the commits doing the actual metadata bumps and tagging automate the builds that happen in response

0 views
Simon Willison 8 months ago

How StrongDM's AI team build serious software without even looking at the code

Last week I hinted at a demo I had seen from a team implementing what Dan Shapiro called the Dark Factory level of AI adoption, where no human even looks at the code the coding agents are producing. That team was part of StrongDM, and they've just shared the first public description of how they are working in Software Factories and the Agentic Moment : We built a Software Factory : non-interactive development where specs + scenarios drive agents that write code, run harnesses, and converge without human review. [...] In kōan or mantra form: In rule form: Finally, in practical form: I think the most interesting of these, without a doubt, is "Code must not be reviewed by humans". How could that possibly be a sensible strategy when we all know how prone LLMs are to making inhuman mistakes ? I've seen many developers recently acknowledge the November 2025 inflection point , where Claude Opus 4.5 and GPT 5.2 appeared to turn the corner on how reliably a coding agent could follow instructions and take on complex coding tasks. StrongDM's AI team was founded in July 2025 based on an earlier inflection point relating to Claude Sonnet 3.5: The catalyst was a transition observed in late 2024: with the second revision of Claude 3.5 (October 2024), long-horizon agentic coding workflows began to compound correctness rather than error. By December of 2024, the model's long-horizon coding performance was unmistakable via Cursor's YOLO mode . Their new team started with the rule "no hand-coded software" - radical for July 2025, but something I'm seeing significant numbers of experienced developers start to adopt as of January 2026. They quickly ran into the obvious problem: if you're not writing anything by hand, how do you ensure that the code actually works? Having the agents write tests only helps if they don't cheat and . This feels like the most consequential question in software development right now: how can you prove that software you are producing works if both the implementation and the tests are being written for you by coding agents? StrongDM's answer was inspired by Scenario testing (Cem Kaner, 2003). As StrongDM describe it: We repurposed the word scenario to represent an end-to-end "user story", often stored outside the codebase (similar to a "holdout" set in model training), which could be intuitively understood and flexibly validated by an LLM. Because much of the software we grow itself has an agentic component, we transitioned from boolean definitions of success ("the test suite is green") to a probabilistic and empirical one. We use the term satisfaction to quantify this validation: of all the observed trajectories through all the scenarios, what fraction of them likely satisfy the user? That idea of treating scenarios as holdout sets - used to evaluate the software but not stored where the coding agents can see them - is fascinating . It imitates aggressive testing by an external QA team - an expensive but highly effective way of ensuring quality in traditional software. Which leads us to StrongDM's concept of a Digital Twin Universe - the part of the demo I saw that made the strongest impression on me. The software they were building helped manage user permissions across a suite of connected services. This in itself was notable - security software is the last thing you would expect to be built using unreviewed LLM code! [The Digital Twin Universe is] behavioral clones of the third-party services our software depends on. We built twins of Okta, Jira, Slack, Google Docs, Google Drive, and Google Sheets, replicating their APIs, edge cases, and observable behaviors. With the DTU, we can validate at volumes and rates far exceeding production limits. We can test failure modes that would be dangerous or impossible against live services. We can run thousands of scenarios per hour without hitting rate limits, triggering abuse detection, or accumulating API costs. How do you clone the important parts of Okta, Jira, Slack and more? With coding agents! As I understood it the trick was effectively to dump the full public API documentation of one of those services into their agent harness and have it build an imitation of that API, as a self-contained Go binary. They could then have it build a simplified UI over the top to help complete the simulation. With their own, independent clones of those services - free from rate-limits or usage quotas - their army of simulated testers could go wild . Their scenario tests became scripts for agents to constantly execute against the new systems as they were being built. This screenshot of their Slack twin also helps illustrate how the testing process works, showing a stream of simulated Okta users who are about to need access to different simulated systems. This ability to quickly spin up a useful clone of a subset of Slack helps demonstrate how disruptive this new generation of coding agent tools can be: Creating a high fidelity clone of a significant SaaS application was always possible, but never economically feasible. Generations of engineers may have wanted a full in-memory replica of their CRM to test against, but self-censored the proposal to build it. The techniques page is worth a look too. In addition to the Digital Twin Universe they introduce terms like Gene Transfusion for having agents extract patterns from existing systems and reuse them elsewhere, Semports for directly porting code from one language to another and Pyramid Summaries for providing multiple levels of summary such that an agent can enumerate the short ones quickly and zoom in on more detailed information as it is needed. StrongDM AI also released some software - in an appropriately unconventional manner. github.com/strongdm/attractor is Attractor , the non-interactive coding agent at the heart of their software factory. Except the repo itself contains no code at all - just three markdown files describing the spec for the software in meticulous detail, and a note in the README that you should feed those specs into your coding agent of choice! github.com/strongdm/cxdb is a more traditional release, with 16,000 lines of Rust, 9,500 of Go and 6,700 of TypeScript. This is their "AI Context Store" - a system for storing conversation histories and tool outputs in an immutable DAG. It's similar to my LLM tool's SQLite logging mechanism but a whole lot more sophisticated. I may have to gene transfuse some ideas out of this one! I visited the StrongDM AI team back in October as part of a small group of invited guests. The three person team of Justin McCarthy, Jay Taylor and Navan Chauhan had formed just three months earlier, and they already had working demos of their coding agent harness, their Digital Twin Universe clones of half a dozen services and a swarm of simulated test agents running through scenarios. And this was prior to the Opus 4.5/GPT 5.2 releases that made agentic coding significantly more reliable a month after those demos. It felt like a glimpse of one potential future of software development, where software engineers move from building the code to building and then semi-monitoring the systems that build the code. The Dark Factory. I glossed over this detail in my first published version of this post, but it deserves some serious attention. If these patterns really do add $20,000/month per engineer to your budget they're far less interesting to me. At that point this becomes more of a business model exercise: can you create a profitable enough line of products that you can afford the enormous overhead of developing software in this way? Building sustainable software businesses also looks very different when any competitor can potentially clone your newest features with a few hours of coding agent work. I hope these patterns can be put into play with a much lower spend. I've personally found the $200/month Claude Max plan gives me plenty of space to experiment with different agent patterns, but I'm also not running a swarm of QA testers 24/7! I think there's a lot to learn from StrongDM even for teams and individuals who aren't going to burn thousands of dollars on token costs. I'm particularly invested in the question of what it takes to have agents prove that their code works without needing to review every line of code they produce. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . Why am I doing this? (implied: the model should be doing this instead) Code must not be written by humans Code must not be reviewed by humans If you haven't spent at least $1,000 on tokens today per human engineer, your software factory has room for improvement

0 views