Posts in Open-source (20 found)
Unsung 2 days ago

“They had no concept of a duty of care to their users.”

A Mastodon post by computer scientist David Chisnall has a very Unsung opener: I have used vim since around 2000. I have written five books, a PhD thesis, a few dozen papers and over 150 articles with it. At this point, my higher brain functions are not engaged at all when I use a bunch of common vim commands, they just happen. Documents I wrote with anything else have random :w in the middle. Chisnall goes on to talk about one specific vim feature: Persistent undo is one of my favourite features of vim. […] I don’t often need the persistent undo. But on the few occasions when I have needed it, it’s been invaluable: ooops, I deleted something from this file, maybe last week and one reboot ago, what was it? Undo until I find it, copy it, paste it into the current version. Or, a bit more commonly: I had this working, then I tidied it up ready to commit, now it isn’t working, what did I do? Vim has kept this working across major version upgrades over a period of about 20 years. I don’t even think about it, it’s just part of Raskin’s First Law: A program may not harm a user’s data or, through inaction, allow a user’s data to come to harm. If vim or the computer crash, or if I close a file and come back to it six months later, my undo history is still there. NeoVim is a fork of vim (in news for other reasons ): So I tried NeoVim when it was quite new. Vim that you are familiar with, but better? Great! The first thing I noticed in NeoVim was that undo didn’t work. I tried opening the file in vim and undo didn’t work there either . Neovim had changed the format of the undo files. It hadn’t upgraded the old one. It hadn’t used a different name for its undo files. It had just noticed the existence of a vim undo file, deleted it (losing all of the data in it) and replaced it with one that vim couldn’t read. I raised an issue about this and was told that the persistent undo format was unstable and users should not rely on data being preserved in a feature explicitly called persistent undo. It had changed once and would probably change again. And that ended my experience with NeoVim. The authors showed immediately that they absolutely could not be trusted with any of my data. Breaking persistent undo is something I could forgive as a bug, but the attitude that just because something is a persistent file on your filesystem that contains data that you might want is no reason for their program not to delete it meant they had no concept of a duty of care to their users. I liked this post (which I quoted almost in its entirety), because it covers a few important things: I also loved it for the appearance of Raskin’s First Law. Jef Raskin, of Macintosh and Canon Cat fame, put together the three laws in his 2000 book The Humane Interface , and they go as follows: It was a very important and formative book for me to encounter as a young designer. I have no idea how these laws haven’t made it to Unsung before today. #principles #text editing #undo I have never heard of the persistent undo like that, and it seems kind of amazing. People do remember when software loses their hard work or disrespects them. I can see how “It had changed once and would probably change again” can be such a powerful feeling. A computer shall not harm your work or, through inaction, allow your work to come to harm. A computer shall not waste your time or require you to do more work than is strictly necessary. An interface is humane if it is responsive to human needs and considerate of human frailties.

1 views
マリウス 2 days ago

Hyperuplink: Discuss like it's 1998

Between the screeching handshake of a 2400-baud modem, the steady hum of a computer in Turbo mode, and the pale flicker of a CRT glowing at three in the morning, the internet seemed to have had something that, over the past two decades or so, it has completely lost: A soul . Communities made up of actual humans gathered around bulletin boards that loaded in a single request, where regular hardware could bring an entire world to your screen without borrowing half your CPU just to show you a list of topics. If you happened to have lived through that brief and peculiar moment in history, you probably still carry a fondness for those days, and you remember the strange little corners of the internet that you discovered, the hours you spent exploring them, and perhaps the friends you made along the way. Hyperuplink is my attempt at bottling those memories and the feelings attached to them, and pouring them into something that makes sense in 2026. It is a modern internet bulletin board that doesn’t require Telnet and works inside your browser, that renders proper, modern HTML5 and CSS on the server-side, that runs 100% JavaScript-free, and that ships as a single statically linked binary with no external runtime, no interpreter, no FastCGI , no , no , and not a single loose file spilled across your disk. It can talk to a PostgreSQL server or an entire cluster, it makes use of any Redis -compatible cache, and it comes with a whole set of nostalgic as well as modern themes. More importantly, Hyperuplink is fun, it doesn’t take itself too seriously, and it is the forum software for everyone who’s done wrangling with phpBB ’s runtime environment or Discourse ’s broken JavaScript UI. And yes, Hyperuplink , too, really whips the llama’s ass. The short version, for anyone who has been idling in the community channel or who already read the back-story in an earlier status update , is that back at the time I wanted a community discussion forum for the people who use any of the tools, programs and services I’m building , and I could not find a single piece of software that I actually wanted to put up with. I was looking for an internet forum that would resemble the good old BBS systems from back in the day, but that would allow people to use it from the comfort of their modern-day browsers. I also wanted something that … phpBB was the obvious first stop, because it has been around for decades and, unlike Discourse and Lemmy , it does not force JavaScript down the visitor’s throat. But phpBB is a monster that carries far too many features, takes time to install and configure, and, once you account for its extensions and runtime dependencies, demands a recurring maintenance ritual that I frankly do not have time for. Discourse and Lemmy , on the other hand, I won’t even consider to begin with, because they simply don’t work without JavaScript enabled. Everything else I looked at either missed something I needed, would have brought similar runtime headaches, or would have required me to fork it and maintain that fork forever, just to get a few of the features that I needed. So I did the reasonable, well-adjusted thing and started writing my own bulletin board software at the end of last year . Before writing a single line I sat down and weighed the usual suspects, PHP with Laravel , Python with Django , Elixir with Phoenix , Go with Fiber , and Zig with Jetzig . I did not consider server-side TypeScript for even a second, because the Node.js and NPM ecosystem is a dumpster fire of outright malware that I refuse to pick for even something as deliberately absurd as Hyperuplink . The scripting stacks make web development pleasant, and they abstract away the tedious HTTP, session and form handling so you can focus on the thing you are building, but every one of them drags a runtime and a maintenance burden behind it. My one goal for Hyperuplink was for a hobbyist admin to be able to run a board without babysitting a whole stack. From an administrative perspective I wanted to be able to update one or two binaries every once in a while and be good, without having to subscribe to e.g. the PHP security announcements, and its mailing lists, and its security advisories on GitHub, and the NVD only to make sure I’m not missing a critical CVE of one of the gazillion of dependencies. Go sits in the sweet spot between the low-level compiled languages like C , C++ and Zig that hand you performance at the cost of development speed, and the interpreted languages like PHP and Python , that make data structures a joy to work with but that are expensive to run. The deciding factor was that Go compiles down to one statically linked binary that you copy to any VPS and simply launch. The one downside is that Go is not exactly a “web native” language and has nothing like Django or Laravel to accelerate the boring parts, so I built my own small web application framework on top of the Fiber v3 framework and went from there. Hyperuplink is a single static binary, compiled with CGO disabled, cross-compiled to Linux, macOS, FreeBSD, NetBSD, OpenBSD and a long tail of architectures, so that deployment is nothing more than copying that binary into place. It is PostgreSQL-native and cluster-friendly and it uses materialised views to optimize read performance. Schema migrations are embedded and run automatically on startup, which means there are no external migration files and upgrades should be as easy as simply firing up the new version. In addition, a Redis -compatible service is used for caching, and sessions, and the async job queue. Profile pictures, attachments, and custom assets can either be uploaded to the local disk or into any S3-compatible object store (such as MinIO ), which is useful when scaling the service horizontally. Hyperuplink runs zero client-side JavaScript, meaning that every page is server-rendered HTML5 and CSS, and there is nothing that logs how your cursor drifts back towards that thread about whether pineapple belongs on pizza, just to collect data on your account. Speaking of which, accounts can sign-up/-in via local password, with optional TOTP two-factor authentication, but Hyperuplink also supports login through OAuth providers for the friends you are trying to lure over from other platforms. And for anyone who finds email too boomer , sign-ups and notifications work over XMPP , too. As for authorization, accounts can be assigned to groups with per-category permissions so the good stuff stays available only to the good people . Hyperuplink features a range of pre-built themes, some of which feature beautiful retro aesthetics that it owes to the classic-stylesheets project. There are also slightly more modern looks available, and with every theme the colour schemes are interchangeable, so a Gruvbox -tinted macOS 9 board is very much a thing. The bulletin board supports Markdown in posts, it offers uploading profile pictures and attachments, it comes with reporting and moderation features for the admins, and the UI supports i18n. Hyperuplink also ships with a REST API that I believe is friendlier to work with than what Lemmy or Discourse expose, and it even has its own TUI client with the official Hyperuplink integration into Neon Modem Overdrive . Hyperuplink is developed on tty.fail and mirrored to GitHub , the mirror being where the pre-built binaries and container images are built and hosted ( thanks for the free CPU cycles! ). Regardless of how you decide to run your bulletin board, you will need a PostgreSQL and a Redis -compatible server reachable, plus, optionally, an S3-compatible store if you would rather keep uploads off the local disk. The official repositories contain all the required documentation and configurations to get you started as quickly as possible, but the basic idea is that you can simply grab the binary for your platform from the releases page , drop it wherever you please, and run it: There’s a whole Docker and Podman (rootless!) setup available, if you would rather bring the entire stack up at once. The repository ships a / with PostgreSQL and Valkey and an optional MinIO profile: The Podman setup does everything the Docker setup does but rootless, and alongside the there is even a set of Quadlet units if you prefer using systemd . Kubernetes works, too, and a minimal with a couple of replicas and the config handed in through a is pretty much all you need. Since the uploads can use S3 the pods stay stateless. Ebuilds are available in the repository so you can compile it on your own Gentoo… server… I guess. I tried including all that’s necessary for Nix but to be fair, I’m not actively using it anywhere so consider this more of a proof-of-concept rather than an actively maintained thing. If you feel like actively maintaining the Nix part of things I’d be happy for you to reach out. The repository also contain the required init scripts for FreeBSD, OpenBSD, OpenRC and even a Service definitions for systemd . If you want to build Hyperuplink yourself, for which you need Go , it’s equally easy to do: The self-contained binary lands in , ready to be moved wherever you please. Note: Alright, alright, I lied, and you got me. No runtime dependencies was not 100% accurate and you’re probably staring at a forum that refuses to allow its users to upload profile pictures. The reason for that is that Hyperuplink has one runtime dependency, which is ImageMagick ’s command. The service must be able to find that binary in its in order for profile pictures to work. As for why, it’s a long story, but the tl;dr is that image processing is hard and there aren’t many people, me included, who feel like reinventing the wheel for, let’s say, the WebP compression algorithm, by building a native Go implementation. Because I explicitly didn’t want to use for the sake of preserving Go ’s easy cross-compiling, I decided that calling the binary was the most reasonable approach. After all, you’re likely to have ImageMagick on your system if you’ve ever hosted anything web related on there. However you end up running the process, put it behind a reverse proxy that terminates TLS, because in the session cookie is HTTPS-only and you most definitely wouldn’t want to run the forum in mode. Also, if you want the service supervised there are the aforementioned service files for systemd , OpenRC on Gentoo and Alpine, and rc.d on FreeBSD and OpenBSD waiting in the repository. Hyperuplink is open source under the SEGV License , the code is available on tty.fail with the mirror and the pre-built binaries over on GitHub , and everything else you could want to read about it is either available at hyperup.link , or within its own embedded manual that you can find under Help -> Manual . If all of this sounds like your kind of thing, come and say hello in the chatroom , show off your board once you have it set up, and if you would like to lend a hand with development or testing then reach out , because the best communities were always the ones with real people, putting in real work. can either use an existing database table to authenticate users, and/or… supports simple signups, ideally with XMPP JIDs instead of email addresses supports notifications and replies via email and ideally via XMPP is lightweight and does not drag along a mountain of runtime dependencies does not require users to have JavaScript enabled does not bury me under administrative features I will likely never touch is reasonably easy to theme and, more importantly, to maintain long term

0 views

Two Alleged ‘TeamPCP’ Hackers Arrested in Australia

Authorities in Australia have arrested two men believed to be members of TeamPCP , a prolific cybercrime and data extortion group blamed for perpetrating the longest running spree of software supply chain attacks ever. In a statement released today, the Australian Federal Police (AFP) said two men from Western Australia, aged 21 and 23, were arrested in connection with a “sophisticated cybercrime syndicate that allegedly created malicious open-source software to rob thousands of global businesses.” The AFP did not name the defendants, but KrebsOnSecurity learned the 21-year-old suspect’s real identity in June, and has been communicating with him ever since. This story includes interviews with TeamPCP’s self-described spokesperson, and examines clues left behind by the TeamPCP leader that likely led to his undoing. TeamPCP vaulted onto the cybercrime scene in late 2025, embedding malicious code in hundreds of open source software tools and extorting victims for profit. Members of the group made headlines by compromising corporate cloud environments using a self-propagating worm dubbed  Shai-Hulud , which added malicious code to open source programs maintained by developers whose credentials at public code repositories like GitHub or NPM were phished or stolen. Writing for Wired , journalist Andy Greenberg described TeamPCP’s core tactic as a kind of cyclical exploitation of software developers. “The hackers gain access to a network where an open source tool commonly used by coders is being developed,” Greenberg wrote in May . “The hackers plant malware in the tool that ends up on other software developers’ machines, including some who are writing other tools intended to be used by coders. The malware allows TeamPCP’s hackers to steal credentials that let them publish malicious versions of those software development tools, too. The cycle repeats, and TeamPCP’s collection of breached networks grows.” TeamPCP also has practiced something akin to cyclical recruitment. In May, the source code for the third iteration of Shai-Hulud was published online, and TeamPCP soon after launched a contest offering $1,000 in virtual currency to whichever participant could conduct the largest supply chain operation using the worm’s code. According to the contest rules, participants were scored based on the number of weekly and monthly downloads of packages they compromised — directly incentivizing them to target the most popular code libraries. A screenshot of a message from TeamPCP’s Telegram account, announcing the supply chain hacking contest. Image: dataminr.com. “TeamPCP has stated the competition is a recruiting opportunity and they intend to purchase all meaningful access harvested from participants’ campaigns,” the security firm Dataminr wrote . “The $1,000 XMR (Monero) prize is a recruitment floor and has been dismissed by the actor as ‘just like participation trophy,’ adding ‘if you find something good you will be paid way more,’ confirming the contest’s true function as talent identification and malicious access acquisition at scale.” In March, TeamPCP executed a supply chain attack targeting AI infrastructure by compromising the code for LiteLLM , an open source AI gateway that connects users to more than 100 different large language models. A recent analysis by the security firm CloudSEK found TeamPCPs attack on LiteLLM harvested cloud service keys and other secrets from more than 2,500 organizations, including many of the world’s top technology companies. In May, TeamPCP claimed credit for compromising at least 3,800 code repositories at the Microsoft-owned GitHub , after a GitHub developer installed a code extension that was compromised by TeamPCP’s malware. Security experts say TeamPCP is less of a hacker group than an amalgamation of threat actors from multiple cybercriminal gangs who sometimes work together toward similar goals. “It is not a structured criminal crew with a single operator,” said Austin Larsen , a principal threat analyst with the Google Threat Intelligence Group . “It is a peer community of individually-skilled actors, with one clear center of gravity.” That center of gravity is George Prepakis , an accomplished security researcher and self-described exploit developer who operates the Twitter/X profile @kernelstub . Earlier this year, @kernelstub tweeted a public invite link to a Matrix chat server he created and dubbed “Cybercats,” and TeamPCP and several other cybercrime entities have been using this server to communicate daily for the past several months. A screenshot of the Matrix chat server “Cybercats,” whose members used hacker handles associated with multiple distinct cybercrime groups that have occasionally collaborated on a series of supply chain and data ransom attacks over the past nine months. Kernelstub, like other administrators in the Cybercats chat, has been using his Twitter/X profile name as his handle in these Matrix communications, frequently tweeting references to other members and to conversations taking place in the Cybercats chat. In a number of cases, the corresponding X accounts for members of the Cybercats chat taunted cybercrime victims publicly before the incidents were reported in the news media. The Cybercats administrator listed at the top of the screenshot above — “ Boxturtle ” — is a close associate of TeamPCP who has been tweeting about the group’s conquests under the name @xpl0itrsturtle . This handle corresponds to a data breach broker active on Breachforums and Darkforums who has been selling data stolen in a wave of recent breaches at automobile manufacturers, including BMW Group , Audi , Honda , Mercedes-Benz , Volvo and Toyota , as well as data allegedly taken from Snapchat and SportRadar . The data leak site for the extortion group or handle “xpl0itrs.” The Cybercats administrator “ SeesawSec ” in the screenshot above is the alias of whoever is behind the cybercrime group known as Fulcrumsec , which recently claimed credit for data extortion attacks against the pharmaceutical giant Novo Nordisk , the data broker LexisNexis , and Avnet , a Fortune 500 distributor of electronic components. The data leak site of Fulcrum Security, a.k.a. Fulcrumsec. The Cybercats administrator “ @pcpcasper ” also has been using a similar name on X to discuss TeamPCP’s attacks and victims. This person has an extensive message history on Telegram, where their messages and shared videos show @pcpcasper is an active and vocal member of the National Socialist Network, a neo-Nazi political organization based in Australia. At one point in these chats, @pcpcasper shared videos and images of what they claimed was their cat, and several of those videos place this user in Western Australia. One source close to the investigation told KrebsOnSecurity that @pcpcasper was one of the two arrested, a claim supported by messages that @kernelstub posted online this morning. The Cybercats member roster pictured above also features an administrator with the username “ T ,” which is short for the now-banned Twitter/X profile @pcpcats , the account operated by the self-described TeamPCP spokesperson who was arrested today. As we’ll see in a moment, @pcpcats also is from Western Australia. By the time @kernelstub tweeted a public invite link to the Cybercats Matrix server, T/@pcpcats was posting only infrequently to the group chat, with other members often inquiring as to his whereabouts and well-being. The group’s collective concern related to @pcpcats’s tendency to blame his increasingly extended absences on the use of hallucinogens and other narcotics that kept him awake for days on end, but also caused him to crash in bed for several days after the highs wore off. The Cybercats member @pcpcats has used multiple nicknames on the cybercrime forums, including EllisD25/LSD on Darkforums, BulkDMT on Breachstars, and Express on Breachforums. These accounts are linked because they all advertised the same Tox ID and/or Session ID as instant message contact handles in their cybercrime forum posts. BulkDMT was also known on the forums as DMT Host , which was a virtual private server (VPS) hosting service that was peddled on Darkforums and Breachstars. DMT Host/EllisD25, posting on the English-language cybercrime community DarkForums in September 2025. Image: ke-la.com. According to the cyber intelligence firm Intel 471 , Express registered on Breachforums using the email address [email protected] . Intel 471 finds Express posted on Breachforums across a two-month period in 2025 using four different Internet addresses located in South Africa . On July 30, 2025, Express announced on Breachforums they were selling access to 14 gigabytes of data stolen from South Africa’s State Information Technology Agency. The threat intelligence platform Flashpoint recorded more than a year’s worth of messages from the TeamPCP leader’s alter ego on Telegram — Persy_PCP —  who claimed they split their life living between two countries [full disclosure: Flashpoint is an advertiser on this blog]. “I have these [files] as well, problem is these are in another country,” Persy_PCP explained to another user inquiring about a stolen data set in November 2025. Later that month, Persy_PCP complained, “My whole country is racist and they want people like me dead.” Flashpoint records show BulkDMT shared in September 2025 that “this country is going to fucking starve when they take the farmers land,” a likely reference to white landowners in South Africa who claim to be targeted by an ongoing genocide campaign . This tracks with public reporting on TeamPCP. Cyberscoop reported in June that Google had traced TeamPCP’s residential and mobile Internet address connections to South Africa, “indicating the primary operator was located there during at least some of its attacks.” BulkDMT also shared on the group chat at Breachforums that they were recovering from an addiction to methamphetamine. “My life is kinda fucked rn [right now], but that’s fine and there isn’t really a point in pouring so much emotional energy into that fact, my parents had money but I unfortunately got really addicted to some things so I don’t get to benefit from that. As long as I continue to survive, stay sober, and move closer towards my goals that’s enough drive and meaning.” The identity threat protection company SpyCloud finds [email protected] shows up in the registration of an account called ChristmasSnow on the cybercrime community Raidforums in 2022. Nearly all of the Internet addresses used to access that account came from ISPs in Perth, Australia, SpyCloud found. KrebsOnSecurity looked up all of those Perth IP addresses in passive DNS records maintained by DomainTools.com , and found one of them — 211.27.196.111 — for several years was used as a private file server by a family in Perth with the last name of Thomson . Those records show at least three hosts — ithomson.direct.quickconnect.to (a remote Synology server), kthomson0061.direct.quickconnect.to, and joshuawthomson39.myqnapcloud.com (a QNAP network storage device) — persisted at that address between 2022 and 2025. Searching on “ joshuathomson39 ” in the breach tracking service Constella Intelligence reveals an account at the freight forwarding company kwe.com created in the name of Joshua Thomson from Perth, Australia. The open source intelligence platform Epieos finds the phone number attached to that kwe.com account was used to register a Facebook profile for Josh Thomson, which says his family includes a brother named Ruben , his father Ian , and his mom Cindy. That Facebook profile also says Josh and his family are originally from Pietermaritzburg , in KwaZulu-Natal, South Africa, but currently living in Cottesloe , a beach-side suburb of Perth. A search in DomainTools for Ian Thomson and Australia unearthed five domains by the same registrant, including securecomputing.au , thomson.org.au , and thomsonfamily.net.au . Ian Thomson is a dentist in Cottesloe, and a biography says he graduated from The University of the Witwatersrand in Johannesburg, South Africa. Constella finds a [email protected] registered a number of accounts online, but Josh doesn’t seem to have much of a connection to dodgy cybercrime forums. His brother Ruben, on the other hand, has quite the presence on these communities, dating back to at least 2018. Constella reports [email protected] frequently reused the password “joshuathomson1,” and Constella further finds that password was used by just a handful of accounts, including [email protected] and [email protected] . According to Intel 471, [email protected] was used to register the user Yolosolo17 on the crime forum Altenen in 2018, and that user account was registered from the Perth address 110.141.230.15 . On Altenen, Yolosolo17 advertised free web proxies, as well as the domain rubenthomson.com, which was at one point used to sell steeply discounted iPhones. DomainTools says rubenthomson.com was hosted at 110.141.230.15 and registered to [email protected]. A cached copy of the domain rubenthomson.com from 2017 shows a login page underneath a banded stack of money. Image: archive.org. SpyCloud reports 10.141.230.15 was used by the email address [email protected] on Raidforums and [email protected] on Nulled, and that the same IP was used by the email addresses [email protected], [email protected], and [email protected]. SpyCloud also shows that sheepstealing Gmail address is tied to the accounts Sheep420 , YoloSolo117 and Yakuza.cc on Raidforums, and to the account “Sheep Stealing” on Hackforums. Intel 471 says [email protected] was used to register the account DingoFlour on Breachforums in October 2023, as well Sheepx on Altenen. Epieos reports that [email protected] is tied to an Airbnb account for Ruben, who described himself as a Web developer who went to school at the University of Western Australia and was living outside the country. “Hey, I’m Ruben, my friends call me Ellis . I’m a Perth creative who occasionally books rooms when visiting family and for photography.” Epieos also finds [email protected] registered an upwork.com profile under the name Ruben, who said his main skills are setting up secure server hosting solutions and PHP full-stack Web development. “I’m familiar with Linux, working with relational databases (SQL),” the Upwork profile reads. “I also script in Python mainly for writing social media bots.” The Upwork profile for Ruben Thomson in Cottesloe, Australia. Epieos further discovered [email protected] is connected to a Microsoft account for Ruben Thomson, and to a now-defunct GitHub account called XmasSnow/XmasSnowisBack that scammed people on the forums in 2022 by claiming to sell exclusive exploits for recently-released software patches (recall that [email protected] was used to register a forum account named ChristmasSnow). This same sheepstealing email address registered a Twitter/X account in 2026 called “Gone Fishing” that lists its location as South Africa. That Gmail account also left several reviews for businesses listed on Google Maps over the past seven years, but all of those establishments are located on the west coast of Australia. Business reviews in Western Australia left by the Google account sheepstealing at gmail.com. The people search service Pipl finds a 21-year-old Ruben Thomson in Western Australia who has a phone number ending in 979. A lookup on that number at Epieos reveals it is connected to a TikTok account under the name Ellis, and to a PayPal account in the name of Ruben Thomson. Finally, a search on the name Ruben Thomson from Cottesloe at the Australian government’s record of registered businesses finds he has incorporated or served as an official in multiple companies created since 2024, including Secure Computing Solutions , Tensor Industries , and another entity ironically named OPSEC Express . Recall that Express was BulkDMT’s nickname on Breachforums. Australian companies connected to Ruben Thomson. Image: abr.business.gov.au. It’s ironic because OPSEC is short for the term “operational security,” which refers to techniques and behaviors used to obfuscate and compartmentalize one’s real-life identity online, and using your cybercrime handle as part of your own company name is very much the antithesis of that practice. There is at least one other major opsec failure by Ruben that exposed a link to TeamPCP. In June 2025, someone using the name Ruben Thomson registered on HackerOne , a popular “bug bounty” program that seeks to reward and recognize researchers who agree to work with affected software vendors to help fix the flaws before publishing about their findings. What was Ruben Thomson’s chosen HackerOne username? Deadcatx3 , a nickname that has been flagged by multiple security firms as an alias used by TeamPCP. The HackerOne profile for “Ruben Thomson” uses the nickname Deadcatx3, which multiple security firms have concluded is an alias used by TeamPCP. Image credit: flare.io. In early July 2026, not long after having discovered clues about Ellis’s real life identity, KrebsOnSecurity interviewed the TeamPCP leader via Signal, where he was remarkably open about his activities and personal struggles [for the sake of simplicity, the TeamPCP spokesperson will be referred to from here on as Ellis]. Ellis claims he stopped doing cybercrime for TeamPCP in March 2026 — just before the attacks that compromised LiteLLM — and that at least one other individual has taken over the group’s leadership since then. Ellis shared that a year earlier he had just completed the latest in a series of detox and sobriety programs, and was two months sober when he reconnected with some old friends from the malware development scene. “One year ago I needed help monetizing some [GitHub credentials], I was two months sober and needed a distraction and something to keep busy as well as people to speak to,” Ellis said. “I had largely disconnected from my old circle, they had become very toxic and I needed to get away from the substances. Previously I had done some mass exploitation campaigns and grew up doing [malware development] and [capture the flag] contests. There were some friends who were also vending but had stopped a while, and one of them introduced me to some chats where I posted access for sale.” Prior to that, Ellis said, he was homeless and hopping between “some very unstable places.” “Blackhatting is fun,” he said. “There are actual rewards and incentives to learn and you grow with your team. Without qualifications, no employer will even take the time to hear you out.” Ellis claims he’s earned a grand total of about $20,000 for his activities with TeamPCP, and that it was never about the money or fame for him. Asked whether his experiences with TeamPCP might prepare him for gainful employment in a legitimate IT job, Ellis said he doubted it. “I am nowhere close to a skill level where I am comfortable, and this would take maybe half a decade of further experience,” he said. “I no longer have to choose between rent and food for that I’m grateful and so are the team members.” Ellis expressed no remorse over his cybercrime activities, and said he was grateful for the friendships and relationships built throughout his engagement with TeamPCP. The young hacker also seemed resigned to his fate, and told KrebsOnSecurity that he’ll accept the consequences if he’s ever arrested. “If I’ve already been found out then its out of my control, I’ll make peace with that,” he said. “Honestly, I think someone like me needs a lot of help that prison just can’t provide. If I had the funds to study different parts of the field and closer guidance, this would have turned out differently. But that’s a pipe dream and we both know this.” It is clear from reading Ellis’s posts to the group’s Matrix server chats that his struggles with sobriety are ongoing. On Thursday, June 25, Ellis told @kernelstub he was about to “trip” with his “homie.” “What kind,” @kernelstub inquired. “Ketty and some DMT,” Ellis replied, referring to the dissociative anesthetic ketamine and dimethyltryptamine (DMT), a powerful psychedelic compound that is found naturally in some plants but is also synthetically produced in underground lab environments. “There’s a little 2cb so we might throw that in the mix,” he continued, referring to another psychedelic compound by its chemical shorthand. Roughly two weeks before his arrest, Ellis told KrebsOnSecurity he was ready to leave his life of crime behind and was prepared to turn himself in, but that in the meantime he was making plans to tie up loose ends. Less than 24 hours later, the TeamPCP leader posted an image on Telegram showing a yellowish powdered substance in a baggie and on a scale, possibly synthetic DMT. The image shows the powder being weighed next to a series of small vape cartridges, two of which are open on the table in front of the photographer. An image posted by the TeamPCP leader to Telegram, advertising his acquisition of some type of psychoactive substance, most likely a synthetic version of the powerful hallucinogen known as DMT. The two defendants were arrested Wednesday morning. The AFP said the men face a combined 14 cybercrime offenses and are scheduled to appear in Perth Magistrates Court today. Charlie Eriksen is a security researcher at Aikido Security who has closely followed TeamPCP’s cybercrime campaigns. Eriksen said TeamPCP are a good example of a new kind of threat actor that does not fit neatly into the usual categories. “They are not a state actor, not quite organized cybercrime, and not purely ideological,” he said. “Their motivations seem to mix money, disruption, attention, and ideology.” Eriksen said that historically there has always been a meaningful gap between reading about an attack technique and being able to reliably turn it into an operational campaign, but that large language models (LLMs) and artificial intelligence increasingly are helping threat actors to bypass that knowledge gap. “You had to understand the research, adapt the code, troubleshoot it, build infrastructure around it, and then repeat that process across different targets,” he said. “LLMs have compressed that gap significantly.” According to Eriksen, this creates an environment where threat actors suddenly have the ability to operate at significant scale without having developed the operational discipline that traditionally accompanies that level of capability. Put another way, it sets the stage for cybercriminals who are capable enough to cause significant damage, but not necessarily careful enough to understand or care about the consequences. “They can be noisy, they can make mistakes,” he said. “They can leave evidence everywhere. They can take risks that a professional criminal group or intelligence service would consider completely unacceptable. But that does not necessarily make them less dangerous. In some ways, it can make them more dangerous.” In a recent blog post , Eriksen called TeamPCP’s Shai-Hulud worm the “best thing to happen to supply chain security,” because it forced GitHub and other public coding platforms to erect new security safeguards. In direct response to TeamPCP’s broad success at pushing poisoned versions of popular software packages, GitHub in late July introduced a three-day “cooldown” mechanism for Dependabot, the platform’s tool for auto-fetching newly shipped updates for any package dependencies. Cooldown periods are designed to help buy time for security tools and package maintainers to identify and remove any compromised versions. Other coding ecosystems like Python and various JavaScript platforms also added support for cooldown periods this year amid growing calls from security experts about the need for more widespread adoption of the safety feature. Eriksen said TeamPCP’s legacy is that they achieved in the span of a few months what the supply chain security community has been unable to do for years. “They managed to wake up Microsoft to the fact that they had become negligent in terms of security,” Eriksen said. “By compromising GitHub and stealing their source code, they humiliated Microsoft into action, making them finally act on what we had been asking them to do and take seriously for a while now.” Update, 10:08 a.m. ET: A story this morning from ABC News in Australia confirms Ruben Ian Thomson of Cottesloe was one of the two arrested. The 23-year-old suspect thought to be @pcpcasper, Michael Gaebler, also was arrested in Perth. ABC News reports that Thomson was denied bail (Mr. Gaebler’s attorney reportedly did not request bail for his client), and that both men will be held in custody until their next court appearance on September 18.

0 views

Forgejo hack: How to set a starting issue and pull request number

I'm currently working on migrating my open source projects from GitHub to a self-hosted Forgejo instance. As part of this effort I often end up looking through the Forgejo source code to figure out if there are hidden ways to configure certain things to my liking when I can't do it on the administration UI. I thought I'd start putting my discoveries in writing here, in case they can help others. So here goes the first one. One of the aspects of the migration that is tricky is how to transition issues and pull requests. What makes the most sense to me is to only use Forgejo to track issues and pull requests going forward, leaving all the issues and pull requests created up to the migration point on GitHub. Of course whether this is a good or bad idea is debatable, but considering all the options I have decided that this is the solution that is going to inflict the least pain on me. The one problem with this approach is that I would end up having duplicate issue numbers, because Forgejo would start creating issues and pull requests all the way back from , and all those low numbers have been used on the GitHub side. So I wanted to hack my Forgejo instance so that issues start from, say, 10000. That way when anyone references an issue by its number I would know that numbers below 10000 are on GitHub and only those above are on my own instance.

0 views
iDiallo 5 days ago

Foot Guns for Sale

I don't think it's going to work out the way everybody thinks it will. The current narrative, at least the one pushed by the companies selling the shovels, is that AI will become centralized. Anthropic and OpenAI will offer safe, vetted AI. Developers will become mere prompt-engineers, submitting requests to these benevolent gatekeepers. They have tamed the dragon. We will benefit only if we become tenants to their API-driven fiefdoms, paying by the token for the privilege of renting intelligence. “They didn’t care that they’d seen it work in practice because they already knew it couldn’t work in theory” - Clay Shirky I complain about AI frequently on this blog, but not because I don’t think it is useful. I use it quite frequently on my day to day. But what I hate is hype and fake narratives. In fact, I believe that the opposite of their narrative will come true. Technology will continue to improve whether Moore’s Law becomes a relic of the past or not. It’s been called dead, yet CPUs and GPUs are becoming faster than ever. I do not believe for one second that developers with pre-LLM experience will end up on the losing side. And if technology continues to improve, then we won’t need OpenAI or Antropic in the future. We will be able to load large open models right into our powerful personal computers with more than acceptable inference speeds. Unless you believe that computers have reached their zeniths and it is all stagnation going forward. The gap between frontier models and open-source alternatives is already all about specialty, and it will continue to narrow. And when developers ubiquitously have access to local models, they will have access to everything. Right now, companies are hoping that developers will use their AI and remain within their ecosystems. They're building guardrails, imposing limits, and designing their models to serve corporate interests first. They see developers as customers, not as threats to their business model. But I’ve seen how easy it is to switch from one frontier model to the next. In fact, some developers in my team accidentally switched by selecting the “auto” mode on their IDE. They didn’t realise that every subsequent request was from a different model. It’s the developer who will end up benefiting from this far more than the corporations will. One developer recently spent his evenings using an AI agent to reverse engineer every peripheral within arm's reach. From those devices, I’ve come away with a full plaintext command shell inside my microphone, a webcam whose activity LED I can switch off while it records, and a key light that hands out memory writes to anyone on the WiFi. He documented the entire process, implemented his own firmware update utilities, and completely enumerated the functionality of devices that were never designed to be user-serviceable. AI gave him the ability to fully control hardware that he has paid for in his own terms. Another developer created OpenLogi , a local-first alternative to Logitech’s software used to remap your own mouse button. The manufacturer was forcing users to create an online account to have access to the hardware they had paid for. OpenLogi gives full control back to the user. This is what happens when developers have the tools and the motivation to bypass corporate control. And AI is about to make this kind of reverse engineering and alternative-building dramatically more accessible. At the speed of large language models, an activist could create a brand new rotating messaging platform every week to avoid the prying eyes of an oppressive government. Someone else can review his own model and add some self improving features. In fact, you could explore new AI paradigms. Most developers I know have a side project they don't have time to work on. With AI, they will have the ability to execute on those ideas. These days, I'm able to run Deepseek on my own $400 local machine at a much slower speed, but I'm not in a hurry. Give it a couple years, I could run something even more powerful. I don't need a project management tool where I have to pay monthly. My actual needs are much simpler. For example, I actually like Jira, despite how much I complain about it on this blog. But I don't need to have Jira for myself on my personal projects. I can build a tool that works solely for my needs. I can easily build applications in environments I am not too familiar with but that are more appropriate for the task. I can build prototypes in a couple hours now and throw them away if they don't match my initial expectation. I can do so much more. Dario Amodei and others are trying to scare us with the capability of AI. They sell the fear of superintelligent systems that will render human developers obsolete or dependent. But this is not what's going to happen. What's going to happen is we will not need Anthropic anymore. Yes, they will have the high-end hardware. But we don't need high-end hardware the same way most people did not need a professional camera. All they needed was a crappy camera with a good filter to post on Instagram. Flickr was superior to Instagram, but the superior technology lost to the one that was "good enough" and in everyone's hands. The funny thing in all this is that by making everything AI-dependent, by building their moats and their guardrails and their API toll booths, companies like OpenAI and Anthropic are selling foot guns. They are building the dark fibers of our era, the infrastructure that developers will eventually subvert, repurpose or simply bypass. Because eventually, we won’t ask for permission. We can just do whatever we want with tools freely available to anyone. We own our devices. We own our data. And soon, we'll own the intelligence on our own terms, without a subscription fee and without a corporate overlord. At the very least, we will get free GPUs .

0 views
Hugo 6 days ago

My software stack to build products

I would like to list here the software that I use or recommend to create products. I have an approach that deliberately aims to support the European market, so I prioritize, in order: (Despite this, you will see that I have 28% US software) Here I only talk about the basic building blocks for creating a company, support software, observability, emailing, payment, etc… ::toc{open="true"} :: Criteria: Merchant of record, metered billing, coupons, connect Alternatives that I couldn't test: alternative often cited: Crisp (France) Alternatives currently being evaluated to replace Sentry: European Alternatives: upvoty (FR) , LiteFeedback (FR) , Feedfast (RO) , No European actor allows me to send marketing emails at AWS prices. It's on average x10 on prices, which is totally prohibitive. Well yes, of course I use my own product, what did you think I was going to put? :) European software Non-US open source software (self hosting) US open source software Non-US software US and Chinese software.

0 views

distributed identity

Sorry but the law doesn't care about your merkle trees We've all heard the horror stories of dealing with names and technology , and yet, we must persist. In this story, we journey through the thorny brambles of git commit history and life events, and ultimately manage to tame them using ATProto. Say that you have a big 'ol git repo. Thousands of commits, hundreds of issues, dozens of PRs. And now let's say one of your contributors—not a maintainer, mind you, just someone who helps out once in a while—is named Andrea P. Researcher <[email protected]>. Andrea gets a new job at Greenfield & Co and changes her email. She come to you with a request: actually, i didn't like my old job very much, could you update the commit history to the new email? Now you have a small problem: Git is a merkle tree that preserves all past commits in amber. You can't change any past commit without "force-pushing" to the default branch, invalidating every single commit hash, distributed checkout, and open PR. Not to worry, you say! The authors of git predicted this. You have the perfect tool: git mailmap . Andrea says perfect, perfect, and adds an entry: Now, Andrea gets married and changes her maiden name to Locksmith. She's still working at Greenfield & Co, though, so she has the same email. She comes back and asks: can you change the commits since I got married to Andrea Locksmith, but keep the old ones as Andrea Researcher? And you say, no, mailmap doesn't really work that way ... git identifies you by your (name, email) tuple, it doesn't have any concept of a date. She grumbles a bit, but well, it's not such a big deal. She uses mailmap to change her commits to consistently use Andrea Locksmith for all the changes (it's close enough) and leaves the ones be. Andrea meets some friends and goes to some movies and shows and reads some books and has a few revelations about himself. He comes back and says, hey i have some news, um, my new name is Bobby. Can you update all my commits? And you point him to mailmap and he says no no, that keeps my deadname around right at the top of the repo. Can't you change the actual data somehow? Look, man, this is important to me. 1 And you apologize, and you really do feel bad; but you look at the 300 open PRs, and the hard-coded commits in , and the merge tooling you wrote that can't handle force-pushes, and you just ... you just don't want to think about how much effort it would be to fix all those. And Bobby gets it, he does, and he makes a mailmap entry instead. But all the same, he contributes a bit less now. Bobby moves to Germany and learns they have this neat thing called GDPR . And one of his friends tells him, look man, you have a right to be called the name you chose, you know? An honest-to-god, enshrined-in-law legal right. And now Bobby comes back to you and say "I want you to rip my name out of the repository because it's personal data of an individual." Well, you're not quite sure that's how GDPR works (maybe you have a "legitimate interest"? are we really sure you were offering a "product or service" to Bobby?). But all the same, lawyers are expensive, and you'd rather not go through the hassle, especially since, well, Bobby really does have a good reason here. And anyway, it would be bad PR, and this isn't the thing you want to lose contributors over. So you figure out how to use and update and force-push to and write a blog post telling everyone how to rebase their PRs and realize you hard-coded commit hashes in your docs so you go back and fix those too and realize you hard-coded them even in some blog posts so now you have to update those and ugh. ok. that's probably most of it now. And Bobby is happy and you're happy he's happy and you put on a half-hearted smile. And then his mate Charlie comes by and says actually that was neat, can you do that for me too? Bobby asked for three things: Git can give us 1, but not 2 or 3. Git is making your life a right-old pain here! If this happens two or three more times, you might even be willing to switch to a different tool, one that supports this better. And—what's this?—there's something called ! It says this: The censor command instructs Mercurial to erase all content of a file at a given revision without updating the changeset hash. This allows existing history to remain valid while preventing future clones/pulls from receiving the erased data. Typical uses for censor are due to security or legal requirements, including: Perfect, perfect, except wait that said content of a file . The author of a commit is actually not the content of a file. It's metadata attached to the commit itself. Damn. So close. Well, how does work anyway? Censored nodes can interrupt mercurial's typical operation whenever the excised data needs to be materialized. Some commands, like hg cat/hg revert, simply fail when asked to produce censored data. Others, like hg verify and hg update, must be capable of tolerating censored data to continue to function in a meaningful way. Such commands only tolerate censored file revisions if they are allowed by the "censor.policy=ignore" config option. Oh. Uh. They're destroying the "cryptographic hashes" part of the merkle tree. That's fine? Probably? We don't really need to work. For complicated reasons related to "filelogs" , this doesn't let us get up to much mischief anyway; we can corrupt but not much more. If we extended this same scheme to metadata though, things would get worse, we might be able to corrupt itself to point to a malicious history. Does it need to work that way? Let's consider the properties we want by comparing to how changes usually work online: gets us 1, kinda. It's still traceable pretty easily. gets us 3. Nothing currently out there gets us 2 or 2 4. 4 is probably not something we care about too much here. "Delete all traces of this commit, even the fact it existed" doesn't seem particularly necessary. But better support for 1 and 2 would be very nice. I have good news for you: there is already an online identity service that does this! (No, it's not OpenID Connect.) It's called ATProto and it's the protocol powering Bluesky . Exactly how ATProto works is a bit out of scope for this post (for more on that see The Hitchhiker's Guide to the Atmosphere ), but what is relevant is how ATProto handles identity . It does this with a decentralized identifier (DID) . For example, my Bluesky handle is , but my ATProto DID 3 is . Because the two are different, that allowed me to change my handle from to when I first joined Bluesky 4 . What's interesting about this is it allows you to control where your data lives. ATProto has a concept of a Personal Data Server (PDS) : by default, when you join Bluesky, your data lives on their servers, but you can migrate your PDS and self-host your own data. This means, for example, that Bluesky can't ban you; you can always migrate to Blacksky 5 . Ok, so, let's put this together and use it in our Git identity alternative. We now have portability, modification, revocability, and—oh? what's that? a primary source? The full history of DID operations and updates, including timestamps, is permanently publicly accessible. This is true even after DID deactivation. It is important to recognize (and communicate to account holders) that any personally identifiable information (PII) encoded in alsoKnownAs URIs will be publicly visible even after DID deactivation, and can not be redacted or purged. In the context of atproto, this includes the full history of handle updates and PDS locations (URLs) over time. To be explicit, it does not include any other account metadata such as email addresses or IP addresses. Handle history could potentially de-anonymize account holders if they switch handles between a known identity and an anonymous or pseudonymous identity. Does it need to work this way? This is talking specifically about bluesky handles . But ATProto has a bunch of other kinds of data . We could just. You know. Build our own. With blackjack, and hookers. Here's an example of a custom ATPRoto record: This is a chess game played between me and , on checkmate.blue , a multiplayer chess app built fully-client side on top of ATProto. Unlike records, normal ATProto records have no permanent history and can be deleted. So, one way we could fix Bobby's problem is something like this: This gets us all the properties we want! One possible UI that could be built around this: Bobby is happy because from his perspective he just commits like normal, maybe with one extra if he wants to tie his identity to the repo immediately. The maintainer is happy because they NEVER EVER EVER have to think about GDPR for commits again. Bobby's ex is unhappy that he moved to Germany, but that's a different story. You could imagine an extension of this idea to commit bodies that allows building on the same mechanism, although it's more complicated because you probably want that to be under the control of the repo owner, not the person who originally submitted the change. Now, this doesn't solve literally every problem—archive.org is a thing—but it sure does solve "all people have to do to deanonymize you is run ". if this doesn't sound important to you, imagine that idk, Bobby is going into a witness protection program or something, or getting a divorce from an abusive ex. also, get the hell off my site. ↩ 522 came up with an alternate way to allow deletions and renames by having a mutable mailmap that's not on the main branch . This isn't quite as flexible as the proposal here, because it only allows the maintainer to change your identity, not you yourself, but it's much much simpler, and works today with normal config. ↩ technically DIDs aren't specific to ATProto , but they weren't widely used before Bluesky started using them. ↩ actually, tangled cheats and lets you write only your email in the commit, then looks for a ATProto DID with that email and uses that to find your Bluesky handle and display name. it also lets you use a DID directly rather than an email. wild shit. doesn't help with our goal of hiding names and emails though. ↩ they can ban you from Bluesky, but not from ATProto as a whole. see the creator of Blacksky's post about this for more information. ↩ you want this per-repo so that you can delete your association with one project without having to delete all of them. ↩ live fetches would be expensive, but you can make them cheaper with an appview , which you can think of as a giant cache with structured database-like queries. this is similar to the idea behind trustfall . ↩ If you want, you can imagine the public/private keypair to be a literal SSH key, which makes it work out of the box with most existing VCS'. This also lets you do fancy things with ssh-agent and SSH forwarding. ↩ real crypto, not that web3 bullshit. "crypto means cryptographers". ↩ Changing names and emails after the fact. Changing names after the fact, in a way that's time-based instead of identity-based. Changing names after the fact, in such a way that the previous name isn't detectable. Passwords, private keys, cryptographic material Licensed data/code/libraries for which the license has expired Personally Identifiable Information or other private data People can rename themselves and change their emails. People can delete their accounts. This usually shows up as a post by a user, or a username. People can (usually) delete the contents of their posts; sometimes admins retain edit history. People can (rarely) delete the post itself, in such a way that you can't distinguish "used to be a post here" from "never was a post here". Just build a new VCS data model from scratch. Look, if we make it a jj backend, it can't be that much work, right? holds a list of mumble mumble unique public key per repo , not a list of names/emails 6 . When you create a commit, instead of having a name/email pair in metadata, embed a private key signature of the commit. Create a new ATProto schema that has an optional current name and email, optional past emails, optional github link using OAuth, etc. Embed the public key and mumble mumble per-repo private key signature of the DID . When you run , it fetches your identity from ATProto. 7 You can edit any identity after the fact. You can add custom fields to the identity record that say to use certain names before or after a given date. You can delete your identity by removing the signature of the DID from your ATProto record. Because the signature is per-repo, deleting one signature doesn't affect the others. The mumble mumble asymmetric key pair make sure that only you can claim that DID corresponds to that commit. Probably. I'm not a cryptographer. Bobby runs , which gives him a private key he puts in 1password. The public key is automatically set up for him. Bobby, optionally, sets up commit signing. 8 If he doesn't set up signing, just embeds the public key as the identity. Bobby visits a website that has a pretty GUI setup for letting him edit his identity record. It can't exfiltrate his key because it runs fully client-side, which Bobby can test by turning off WiFi on his laptop, generating the new record (with only the signature, not the key), and then turning WiFi back on to copy-paste it into a fresh page of the app. Git preserves all data forever , in amber. Trying to change it is a goddamn nightmare. This is a problem for credentials, identities, and copyrighted material. makes a good-faith attempt to fix this, but only works for commit contents, not commit metadata This post proposes a way to fix this for identities, not just commit contents, using ATProto's distributed identities and personally-owned data storage, as well as a completely off-the-cuff unreviewed crypto 9 scheme. The scheme allows you to change your identity without having to rely on a second- or third-party. if this doesn't sound important to you, imagine that idk, Bobby is going into a witness protection program or something, or getting a divorce from an abusive ex. also, get the hell off my site. ↩ 522 came up with an alternate way to allow deletions and renames by having a mutable mailmap that's not on the main branch . This isn't quite as flexible as the proposal here, because it only allows the maintainer to change your identity, not you yourself, but it's much much simpler, and works today with normal config. ↩ technically DIDs aren't specific to ATProto , but they weren't widely used before Bluesky started using them. ↩ actually, tangled cheats and lets you write only your email in the commit, then looks for a ATProto DID with that email and uses that to find your Bluesky handle and display name. it also lets you use a DID directly rather than an email. wild shit. doesn't help with our goal of hiding names and emails though. ↩ they can ban you from Bluesky, but not from ATProto as a whole. see the creator of Blacksky's post about this for more information. ↩ you want this per-repo so that you can delete your association with one project without having to delete all of them. ↩ live fetches would be expensive, but you can make them cheaper with an appview , which you can think of as a giant cache with structured database-like queries. this is similar to the idea behind trustfall . ↩ If you want, you can imagine the public/private keypair to be a literal SSH key, which makes it work out of the box with most existing VCS'. This also lets you do fancy things with ssh-agent and SSH forwarding. ↩ real crypto, not that web3 bullshit. "crypto means cryptographers". ↩

0 views
Evan Hahn 1 weeks ago

Vim's UserGettingBored autocmd

In short: Vim has a joke autocmd called that doesn’t do anything. Vim’s automatic commands feature, usually shortened to “ autocmd ”, lets you run code when various events occur. For example, you could implement an auto-save feature by binding the event to the command. Vim has over 100 events, from “buffer was created” to “file was saved”. But one of them sticks out to me: . Here’s the documentation: : When the user presses the same key 42 times. Just kidding! :-) When I saw this, I was busy doing something else and it completely derailed me. “I must know more,” I thought. Here’s what I found: Unfortunately, it doesn’t do anything. It only exists in the documentation (and some tests). If you try to use it with somethig like , you’ll get a “no such group or event” error. It’s present in Vim , Neovim , and Vim Classic . It was first added by Bram Moolenaar in July 2000 , over a year before Vim 6.0 was released. The original description was, “When the user hits CTRL-C. Just kidding!” And it didn’t do anything back then, so I don’t think it’s ever been real. In August 2001, he added the smiley face to the documentation . It then read, “When the user hits CTRL-C. Just kidding! :-)” Twelve years later, in 2013, the description changed to its current iteration: “When the user presses the same key 42 times. Just kidding! :-)” In 2022, developer Mike Smith created an unofficial plugin inspired by this joke autocmd . If you press the same key 42 times in Insert mode, a picture of Samuel L. Jackson appears. 22 years later, it’s finally real. Unfortunately, it doesn’t do anything. It only exists in the documentation (and some tests). If you try to use it with somethig like , you’ll get a “no such group or event” error. It’s present in Vim , Neovim , and Vim Classic . It was first added by Bram Moolenaar in July 2000 , over a year before Vim 6.0 was released. The original description was, “When the user hits CTRL-C. Just kidding!” And it didn’t do anything back then, so I don’t think it’s ever been real. In August 2001, he added the smiley face to the documentation . It then read, “When the user hits CTRL-C. Just kidding! :-)” Twelve years later, in 2013, the description changed to its current iteration: “When the user presses the same key 42 times. Just kidding! :-)”

0 views
matklad 1 weeks ago

Rust Glancer

Rust Glancer , a functional LSP server for Rust which uses two orders of magnitude less RAM, is incredibly cool. Go check it out! This post started as a comment on lobste.rs, but I figured it out that it’s better to publish it somewhat more prominently. Don’t expect polished writing though! Some thoughts: rust-analyzer uses rowan for syntax tree representation Yeah, rowan is garbage :P I was really thinking about And Rowan is pretty good for that. But that’s 1% use case. The 99% use case is all the code in your 6666 dependencies which you won’t ever look at, but which needs to be at least shallowly analyzed. Even for incremental tool whose main goal is refactoring, the primary AST structure should be just a list of arrays. There might be a real post about that at some point, see https://youtu.be/G93oYL1ry70 as a teaser. Rust workspaces genuinely have a lot of information that must be indexed: thousands of functions, structures, traits, relationships between these, function bodies and statements in them, etc. Each of these needs to be analyzed and remembered, and you can’t really cheat if you want to have things like “find all references to this structure”. If I understand correctly, Rust Glancer wants to process each function body. I think that part can perhaps be made lazy (but not incremental!) with little overhead? Index all items, but, for functions, do only the currently opened file? This might combine some of the better parts of both worlds. Would be interesting to compare memory usage with Rust Rover. Net of the IDE GUI itself, I would expect RR to be more compact. Some features are unlikely to be supported though, such as build scripts / proc macros support via proc macro invocation I might be rationalizing/misremembering things, but IIRC it’s exactly around adding proc macros that the thing began to feel unreasonably bulky. Expanding proc macros is slow as we are running real code, we can’t really do normal IDE cheats. And proc macros generate a lot of code. At one point I measured, it was like 30% of rust-analyzer binary size was attributed to JSON parsing code. If no one sees the code, it can’t harm anybody, right? One potential approach here is to pull the Sorbet trick, where you don’t run meta programming at all, and instead have a plugin interface to “explain” the effects of what that would have done. Instead of running serde, we just add a shim that injects with an empty body. I’m not sure why, but in rust-analyzer I’ve observed that when agents edit the code, inlay hints can get out of place Rust analyzer’s core data model is very pedantic about always observing consistent snapshots of the code, and does its best to ensure that the language client and server have a shared, strictly serializable view of the world. It’s a shame that LSP doesn’t allow that to be correct , only heuristically right , unlike the older Dart Analyzer protocol, which has sound data synchronization. However our implementation of file watching is sketchy! First, there are two backends: we can ask the editor to do watching for us, or we can use server side watching. Try changing this option and see if it helps? But then, yeah, my recollection is that our native watcher’s API was fundamentally racy, and I didn’t do the messy platform-specific work of making it correct. But the main thing I want to write, and why I moved from the cozy lobste.rs text area to the luxurious comforts of an Emacs buffer, is that right now rust-analyzer is a bit like that half-drawn horse meme, except that it’s only the head half of the horse. One Big Idea of IntelliJ is that it’s PSI API (essentially AST with resolved types) is really an interface, and there are multiple provides. And in a typical usage, there’s at least three backends in play: This is how I think such things should work. rust analyzer shouldn’t use salsa for all those 6666 dependencies you still haven’t looked at. It should just use rustc’s .rmeta files, switching to salsa, transparently, only when the user starts messing around their folder. The prerequisite for that is defining the abstract API for accessing Rust code. That was always the plan, and we did start on that at some point: https://hackmd.io/ytd82QNiT_Ku2XFr1EAtiQ rmeta-transparent – source code might not be available for some crates, the API should support pre-compiled rmeta files as inputs. But I don’t think that work was ever completed. This still seems to me to be the lowest-hanging watermelon here — split the world into arcy-pointy incremental tip of the iceberg, and mostly read-only, on disk, compact, dark, moist breeding ground for supply chain attacks. Such glance analyzer architecture would be great, imo! incremental parsing, incremental, DOM-mutation style refactorings, For the files opened in the editor, actively modified by the user, the PSI is backed by the concrete syntax trees. For the rest of the project files, the PSI is backed by the so called Stub Tree, a compact on disk representation storing only the “externally visible” parts of the file (so, without function bodies). If the user navigates to a new file, its PSI transparently switches from stubs to syntax tree. For dependencies, the PSI is often backed by the compiled .class files, produced by javac. If you navigate there, the IDE just decompiles stuff four you! Super cool!

0 views
Matthias Endler 2 weeks ago

GitHub Was Never About the Source Code

With GitHub down once again, I decided to write this long-overdue blog post. This was the top comment on Hacker News when I started writing this: For a variety of reasons, we had a centralized place where everyone of a particular set of persuasions could connect, and this had outside benefits for the community as a whole. That place is becoming untenable, and with the loss of goodwill and stability, the community’s started to dissipate. […] But those emergent features like a core community and default expectation of where you can find someone will fade . And that is a very real loss for all of us. That resonates sooo much with me. GitHub has never been about the source code. It’s where I hang out with friends, see what they’ve built, and discover all the cool projects they star. It’s the only “Facebook” I still visit. There is no other place like it anymore. GitHub is on its way out, and the community around it is starting to crumble. My feed basically consists of the same two people starring repos and being really “active.” The rest is just noise. (I wish I could show you a screenshot, but, well, GitHub is down.) “AI” has only accelerated this trend. These days, you’re lucky to get a pull request from a real human being. Discussing changes is no longer fun when you’re talking to a bot 90% of the time. I hope someone builds a community for developers; we need one now more than ever. The way I see it, our code could live anywhere. We could share stars, comments, and project updates through a common network, similar to Keybase or Mastodon . Send me an email if you like the idea.

0 views
Farid Zakaria 2 weeks ago

DEFCON34 wrap-up

I recently came back from DEFCON34 and the nix.vegas community. The talks I gave are now online if you are interested in watching them. 🙌 Many thanks to all the organizers of DEFCON34 and nix.vegas. This is our, the Nix community and mine specifically, second year at DEFCON34 and it was a blast. To be honest, I barely interacted with the rest of DEFCON because I was so busy with the Nix community. The talks, the hallway conversations, and the in-chance encounters were all amazing. One particular story, was that Carl Dong happen to be walking by the Nix Vegas village as I was giving my talk on Guix by Nix . He was a Bitcoin core developer and was one of the contributors responsible for the Bitcoin Core reproducible builds project that leverges Guix . 1 For those that don’t know: nix.vegas is the Nix community that runs within DEF CON in Las Vegas, hosted by the SoCal NixOS User Group and Distractions, Inc. This was its second year: DEF CON 33 ran under the banner “Rebuild the World” , and this year’s theme was “Escape Your Fate” . The full playlist is on YouTube . Note If the sound is a bit off or weird, this year DEF CON experimented with “silent” talks. Each talk was broadcasted and attendees had to wear headphones to listen. It was a bit weird giving talks to a quiet room. 🤷 Summary : Nix’s absolute paths buy us reproducibility, but costs us the ability to put the store anywhere else. You can change the store prefix today, but it changes the hash of every single derivation in the closure down to , so you get to rebuild the world before you get to run . How can we circumvent this? The talk walks through in and upstreaming support in the Linux kernel via a eBPF-based solution. Further reading: Linux kernel will support $ORIGIN, sort of . Summary : What was meant to be a lightning talk on guix-transfer and GuixPkgs but went a little over. This is our project on rewriting Guix derivations into Nix derivations so that every Guix package becomes buildable by Nix. This lets us include their source-bootstrapped JDK for instance, which nixpkgs does not have. Further reading: Guix by Nix and GuixPkgs: every Guix package, as a Nix flake Summary : This talk is a bit of a rant, but it is given in good faith with a dose of humor. The core claim is that we optimize Nix and nixpkgs for social comfort and broad appeal, and we pay for it in technical ambition. Further reading: How to piss off your Nix friends . Looking forward to next year. Three talks in two days was a little ambitious, but I would do it again. Everything lives on my talks page alongside their slides and the rest of my talks. He was pleasantly surprised and happy to hear that Nix also has reproducible builds that start from stage0 .  ↩ He was pleasantly surprised and happy to hear that Nix also has reproducible builds that start from stage0 .  ↩

0 views
Kev Quirk 2 weeks ago

2026-08-16 14:35: Working great 👍🏻

Working great 👍🏻 Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment .

0 views
Phil Eaton 2 weeks ago

The road to ACID transactions in Cassandra 6

This is an external post of mine. Click here if you are not redirected.

0 views
xenodium 2 weeks ago

Focus windows across Emacs frames

Historically, I've only ever known two great ways of focusing Emacs windows: the built-in command, which cycles through all the available windows, and the ace-window package, offering random window access. It's not that there aren't more or better ways, I just didn't look any further as these were enough for my needs. While I really wanted to make my default choice, for whatever reason, it never stuck. with a custom binding always felt like the smoother fit for my limited needs. You see, I hardly ever have more than two visible windows, so whenever kicked into action (for 3 or more windows), it often took me by surprise. The one area didn't fit my needs revolved around focus requiring visual feedback, but I eventually solved that with winpulse (a little package I wrote). As you can see, I'm a simple man using few Emacs windows, and when it comes to frames, I almost never use more than one. That is until somewhat recently, when I built ytr , a tiny YouTube radio player that sits in the corner of your frame. While it all feels fairly integrated into your frame, under the hood, renders in a separate frame. This broke my trusty flow. I couldn't just focus my radio window using my well-internalized binding. Turns out, actually caters for focusing windows across frames, but only when invoked programmatically. Sure, I can wrap it with my own custom command, but Emacs already had me covered. I found the built-in command. All I had to do was bind it to and Bob's your uncle . I can now switch between my current window and my YouTube radio, with my dear binding. Balance restored.

0 views
Pete Warden 2 weeks ago

Why I ported Moonshine to Javascript

One of the most common requests I’ve heard from developers is an in-browser version of Moonshine that can run on a web page. In theory this should be straightforward – we already built MoonshineJS for the previous generation of models, and the core library is written in portable C++, so emscripten can compile it into WASM. There have even been some interesting community porting projects but I held off on official support until I had time to do it justice. I knew that porting the C++ core was just the beginning. Building something that would be straightforward for web developers to use required a lot more: After a lot of work, I finally have a version ready for feedback. The easiest way to try it is on the new moonshine.ai home page, where you can now see everything from a minimal transcription example to a full-blown Granola-style meeting note taker . As an open-source project, all the code for these is available and the simple examples include code snippets in-line too. Here’s one that shows how to run speech to text on a web page, to give you a flavor of the API: You may still be asking yourself why I made supporting Javascript in the browser such a priority? A lot of “X ported to WASM” stories end up being Hacker News bait without having any practical uses. The evidence that drove me was: I’m excited to get feedback on how to improve the initial version, and I’m looking forward to hearing about what people build with it, so please come by our Discord channel if you’d like to join our community. High-level APIs that were both idiomatic for browser Javascript and consistent with the other Moonshine language bindings. Infrastructure for testing from units to full web pages. Integration with the existing CI and deployment process. Examples that were interactive and showed the key capabilities of the library, with interactive inline code. Larger applications that demonstrated and tested how the framework runs in real-world conditions. Improved support for in-memory models and data files. This was involved a lot of changes to the core library, because while there had always been some methods that took memory buffers, coverage was patchy compared to loading from files. Clear developer demand . It came up frequently as a wishlist item when talking to users. Javascript’s dominance . Python rules machine learning, but JS is the most common language for applications, web and server-side. Advantages over alternatives . Voice interfaces are clearly only going to grow in importance over the next few years, but browser APIs are neglected and server-based alternatives are slow and costly compared to our on-client framework. Obvious applications . Dictation and meeting note taking are popular use cases for speech technology already, and talking with AI bots is becoming a lot more common too. Technical alignment . Deep in my bones I know that voice interfaces want to run on the client. The current status quo of streaming audio data to a server just to get text and intent back only exists because models used to be too large to run on consumer hardware. Today even household appliances have enough compute horsepower for local voice agents . It offends my engineering sensibilities to see old approaches linger on purely out of inertia. Speech wants to be free to use and local, just like all our other input devices like keyboards, mice, touchscreens, and cameras. Porting makes that possible on the web. Options . These days a lot of us have to frequently switch between languages and operating systems, and the power of AI coding assistants only increases the pressure to rapidly port applications. A library dependency is a big commitment, and knowing that it will be available anywhere you’re likely to run in the future makes the risk of betting on a framework much lower, even if you don’t need the option in the end.

0 views
xenodium 2 weeks ago

agent-shell 0.73 updates

Another month, another agent-shell update. If you missed the last one, have a look at the 0.63 update . While this post showcases the latest highlights, the full list of changes is far chunkier than what we'll cover. agent-shell is a native Emacs mode to interact with AI agents powered by ACP ( Agent Client Protocol ). Since inception in September last year (yikes nearly a year), has featured a shell-like experience, powered by comint mode . There's also viewport mode (via ), if you prefer a more focused experience, and now we have . Chat mode fuses mode with a more traditional chat-like labelling experience. We're living on the edge here, so chat mode is now enabled by default. Ok not really that edgy, it's fairly safe (powered overlays ) and can be disabled entirely via . Chat mode itself is a minor mode, so you can always toggle it on and off via . In the last post, we talked about making less chatty with more grouping for the likes of tool calls and agent thinking, all collapsed by default. If that's far too quiet, you can expand by default via . If you found these two settings either too quiet or too chatty, we now have a third alternative via . When set, only the latest grouped activity is expanded by default, and automatically collapsed when the agent moves on to something else. I've become quite fond of this feature (thank you @nhojb for the PR ), so yes. It's also enabled by default. Queuing received some improvements. The related commands have been consolidated under . You can queue prompts while the agent is busy, then view, resume, or drop pending prompts via , , and . The pending queue is now shown after each new submission. opens a dedicated buffer for crafting a prompt, and it's now more independent of the shell. You can invoke it from any buffer (it resolves to the right shell), sends and returns you to whatever you were doing (fire and forget), and submits the prompt and immediately lets you craft another queued prompt. Shell initialization may take a second or two, depending on what agent you're using, which meant you had to wait for initialization before you could start typing into your new shell. Unnecessary, so that's no longer the case. Shell prompts are now offered as soon as possible, so one can get typing. While Markdown lists are easily digestible without special rendering, we can do better than that, so we now give them a better treatment with normalized padding, indentation, and of course, civilized bullets. TAB navigation made it into fairly early on. I love being able to TAB my way into any section in the buffer and press RET to toggle folding. That's great and all, but we can make the navigation experience richer by welcoming the likes of Markdown source blocks, links, and images to the navigation party. Why these in particular? They are all actionable by RET too, of course. While you'd rightly assume RET opens links to local text files in Emacs and delegates to browsers when needed, Markdown source blocks and images get a similar treatment. While their respective RET actions may not be as obvious, we can certainly make them much more discoverable, so we now add hints. Landing point on an actionable item now echoes a hint of what you can do and which key does it (say, "Press RET to copy" on a source block, "Press + to enlarge" on an image, and so on). Hints are also shown on mouse over events. While offers for customizing image sizes, it's fairly restrictive. Not all images are the same, so why force them all to fit within the same constraint? continues to offer a preferred default, but you can now scale images differently by getting the agent to annotate Markdown images with Pandoc-style link attributes . The attribute block goes right after the image, taking a and/or in pixels or percentages: I may be sweating the small stuff here, but this was really grinding my gears. We have lovely Markdown table rendering, which I'm glad we do as LLMs aren't always great at producing perfectly aligned tables. In the best of cases, the LLMs align the table perfectly, but it's just too wide for our Emacs window. Luckily, our lovely rendering also wraps cells to make them fit into our window. The thing is, all that lovely rendering goes out the door the moment you either resize your Emacs frame or merely split your window, resulting in a monstrosity like this: I know. I'm sweating the small stuff here, but hey we don't have to live like this. Emacs has all the hooks in the world, so let's track window changes and rejoin the civilized world. On a much smaller scale, I also wanted auto resize for images, so here you have it… While we can influence image size at render time, this can still generate undesirable image dimensions, so we can now rescale all images in buffer on demand. Sometimes a hammer really isn't the right tool, so we can now also rescale an image at point. Opening a local file (from a link, image, or mention) now routes through , a standard action you can customize. The default reuses a window already showing the file, or takes over the current one. If you'd like a different window arrangement, you could do something like: Following a local file link now pushes your origin onto xref 's marker stack, so ( ) brings you right back to where you were in your , just like other Emacs jumps. Foldable fragments now use and bind . If you're not a fan of the RET binding to toggle folding, you can now use your preferred binding. If is your jam, you can do something like: If you prefer styling agent thoughts differently, a new face lets you do just that. Streaming performance also received some love. Thanks to @suhail-singh for the profiling and improvements , and to @Scott-Guest and @claytharrison for the trace analysis and benchmarking in #757 . The "Available config options" section now displays possible values. can now be set to a function, letting you compute the available agent configurations dynamically rather than hard-coding a static list. The function is called on every access, so it stays current across code reloads. Maybe you'd like to list only available agents. Here's a rough snippet. now broadcasts an event, handy for external integrations that want to observe streamed output. now works correctly on remote hosts ( #742 by @CeleritasCelery ), smoothing out TRAMP-driven remote agents. agent-shell-hq joins the family, offering an interface for managing multiple sessions. If you peeked at the commit logs , you'll notice I've been working daily on , keeping up with project inflow. Since last month, 27 issues have been closed and 13 pull requests merged. As of today, the backlog sits at 11 open issues and 5 open PRs (versus 13 and 4 last time around). If there's something you'd like me to prioritize, feel free to ping. Vendor-neutral tooling matters more than ever, and there are a couple of ways to help keep going. Some cost money, others just a click. All are appreciated ;) is just me, an indie dev, while the tools it competes with have well-funded teams behind them. Time spent on is time away from work that pays the bills, so if it's useful to you, please consider sponsoring the project. And if your employer benefits from your use, nudge them to chip in too, they can typically contribute at a scale individuals can't. GitHub stars help with exposure, attracting new users and potential sponsors. Starring agent-shell costs nothing and can potentially help bring in more funding, so if you don't mind a couple of clicks, the project can really use another GitHub star . Thank you to all contributors for these improvements! Liking ? Would like to see it evolve? Consider sponsoring the effort. #730 : Queue requests sent during session/push and submit them when it ends ( @catern ) #737 : Add a workaround for goose ( @bergmannf ) #740 : Add a option to ( @nhojb ) #742 : Fix executable-find on remote host ( @CeleritasCelery ) #743 : Preserve "claimed" regions when rendering bold / italic / strikethrough ( @alberti42 ) #746 : Avoid repeated system-sleep load attempts ( @liaowang11 ) #748 : Preserve properties on escaped Markdown punctuation ( @Scott-Guest ) #752 : Guard group member walk against non-advancing block range ( @hamza-m-masood ) #756 : Preserve point during table rendering ( @Lenbok ) #762 : Document viewport workflow ( @KarimAziev ) #763 : Render raw tool output ( @mrychlik ) #765 : Prevent syntax highlighting from delaying other buffers' mode hooks ( @Scott-Guest ) #766 : Refresh viewport header even with nil agent-shell-prefer-viewport-interaction ( @catern )

0 views
Farid Zakaria 2 weeks ago

nixpkgs-multiverse: fast mode

“The fastest evaluation is the one that never happens.” – Sun Tzu, The Art of Evaluation nixpkgs-multiverse gives you every version of every package that ever shipped in Nixpkgs from a single flake input. Note It continues to blow my mind that this is even possible. It feels like it suddenly unlocks a new dimension of Nixpkgs, and I am still trying to understand what it means. I think this capability is a fundamental change to the way we think about Nixpkgs, and it is not just a new feature. It is a new way of thinking about the entire ecosystem. There was always a penalty at the center of it. Asking for a specific version of , such as , meant fetching the whole ~378 MB Nixpkgs tree from 2021 and evaluating it to determine the . What if we could skip that evaluation? What if we could just ask for the path directly, and have Nix fetch it from the cache if it is there? This is a common idiom if you have ever used . That requires knowing the store path upfront. nixpkgs-multiverse now has a attribute that does exactly that: it gives you the store path for every indexed version of every package. This lets you skip the download and evaluation of Nixpkgs and get the store path straight from the cache. No Nixpkgs is fetched. Nothing is evaluated. No experimental features and no needed for this to work. The complete Nix API, except for releases, works with this fast path. If you want to learn more read the docs about the feature. Every channel bump published a listing of every path Hydra built for it: , or a for back in the pre-2017 era. These files are still available, and they are the source of the multiverse index. The listing is a map from derivation name to store path. The multiverse index is a map from to the revision that shipped it. By joining the two, every historical version gets a concrete address: Knowing the path is not enough, especially in the Nix language. We need to convince Nix that a string that looks like a store path actually is a store path. exists but it is an impure function and requires to work. How do we get around this? We attach “context” to the String. Context is the invisible baggage a String carries in Nix. When you interpolate a derivation into a String, the result remembers where it came from, and that is what makes realise the dependency instead of writing a dangling path into a script. lets you attach it by hand. The identifies that “this String names a store path that must exist,” which is exactly what produces for a path already in your store, except this works for a path that is not in your store yet and is not in this evaluation’s input closure either. Loopole! 👿 We then wrap that in an attrset that resembles like a derivation and the Nix CLI is satisfied: This is tomberek ’s trick from fastpkgs , and it is an amazing trick to circumvent needing . Everything about this remains pure evaluation, and the resulting graph is gauranteed to be bit-for-bit identical to what Nixpkgs would have produced if it had been evaluated. The only difference is that we skip the evaluation of Nixpkgs itself, and instead use the store path directly. The eval path derives the address, the fast path remembers it. A “fake” ( ) derivation has no , because there is no behind it. Nothing can build it and it can only be substituted. The CLI often wants a though when you hand it a derivation attrset, so we must make sure to append the output (i.e. ): and need a real derivation. Every fake derivation carries a lazy that is the real, revision-exact derivation: In the spirit of trying to keep my index small, is empty, so there is not additional information about the package. You can still get the from the real derivation by using as well. This scheme rests on cache.nixos.org still serving thirteen-year-old paths, which thankfully it does and with the same signing key. To demonstrate that the cache is offering nearly every path that Nixpkgs ever built, I ran a census of every indexed version of every package and asked the cache if it was still alive. As of August 14 2026, all 271,187 of them are alive . All of them, down to every NAR payload file. That is 14.8 TB of unpacked software from 2013 onward, one fast command away. 1 There’s some other data on nixmultiverse.com about the census, dependency graphs and additional features. Check it out! All of this is also available via the mvs command line tool as well for offline use. I guess now there is a caveat: there is now a trick in the multiverse. It remains mostly an index, some JSON, and a behind a memo table. The clever trick is tomberek ’s, and it is three important lines. The NixOS infrastructure has never garbage collected the binary cache. It is an S3 bucket that only grows, and the bill is paid by the NixOS Foundation and its sponsors.  ↩ The NixOS infrastructure has never garbage collected the binary cache. It is an S3 bucket that only grows, and the bill is paid by the NixOS Foundation and its sponsors.  ↩

0 views
daniel.haxx.se 2 weeks ago

curl performance

tldr: the live version is here: https://curl.se/perf/ How fast is “fast” and is it good enough? Does it run as fast now as it did before or was there a regression? What exactly needs to be fast? How fast is it? These are questions that many projects and products face, and in curl we are no different. Yet, performance testing and comparisons are hard and full of landmines and time-wasting efforts. For many years we have occasionally brought up the idea of a performance test suite for curl only to shut it down again because the challenges seemed hard and no one was volunteering to do this. This week it changed. I started out trying to find existing projects that host performance results for Open Source projects so that we could just feed our results something else and get great visualizations and data management. I did not find any such. I then took a look at what existing tools there are for this purpose, and most pointers seemed to suggest that Grafana is a popular and maybe even a good solution to build something like this with. But man, that is a complicated machine and it felt more than a little overwhelming just figure out where or how to start with it. I decided to postpone that take as well. I decided that instead of trying to do this the best and optimal way – I shouldn’t let perfect be the enemy of good – I would start out by doing the things I know how to do and take it as far as I can one step at a time. Something should be better than nothing . Performance testing needs decently stable system conditions so that repeated runs produce reasonably similar results, when all involved factors remain identical. This is basically impossibly to accomplish using most cloud infrastructure since those are almost always shared with countless other users. At least on the cheap and free tiers we use. We probably need our own dedicated hardware for this, but instead of trying to figure out where to get that and arrange for that, I would start by running performance tests on my own local development machine. I am a single user on this and it has many cores and runs decently fast. It should be good enough to get this going on. I created a first shell script that updates the curl source code from git, it configures and builds it. Then it runs a bunch of tests, outputs a bunch of data and logs all the output in a single log file. I started out with a few simple tests. How fast does curl download a 100 GB file from localhost, how many allocations and how big allocations does it need for a single HTTP download? My second script parses all the test log files from the previous builds and generates summaries and graphs for them. To make it possible for humans to see how the performance changes between builds and ideally to automatically detect when something changes more than what should be tolerated. As I am a graph addict already since before , and that journey has taught me a little gnuplot , I decided that even while there probably are much better tools and fancy JavaScript things that could be used, I don’t know them and learning them now is an endeavor I rather avoid. So I stick to what I know and can get results with quickly. A third script is invoked from a crontab every twenty minutes, sets up some variables and invokes the runner script. Once the basics started to work, I showed my curl friends the early versions and I soon created a new git repository for the code . After a little more poking, I soon made my locally produced performance test summary get packaged and automatically transferred to the curl website after each build, and voila, the first public curl performance tests were live and public. Getting this data available immediate triggered curl developers. It only took hours until we had the first proposed changes to improve some numbers, and soon we had a few merges to that affect. Visibility really helps! The performance numbers we get are still varying to a certain degree, partially of course because I still use my machine for my daily development things, but also because most of them do real (localhost) networking and that is by its nature a little… varying . The system builds and runs a new round every twenty minutes and it does that using the latest commits from git. This setup makes it sometimes run many rounds on the same commit and it might also mean that it sometimes updates and get several new commits at once, so it might skip a round for some commits. I might reconsider this design later, but since it is still a twenty minute time window, the number of commits is still limited. When the script makes multiple build rounds on the same commit, it accumulates the numbers and for the graph it stores the maximum, the median and the minimum value. It helps show the variation per commit and allows us to cram more into the graphs. It is still early days, but there will be a maximum limit to how many commits that can be displayed in a single graph and still be helpful. HTTP/2 parallel download speed through 261 builds spread over 31 build rounds Distribution To help visualize the distribution and data spread per test, I created a separate illustration that shows the minimum, maximum, P25, P75, medium and mean values in a Box-and-Whisker Plot . A Box-and-Whisker Plot showing the HTTP/2 parallel download speed data distribution. Changing conditions An obvious downside with me just storing build logs in files, is that it will not scale up to the millions. I did however decide that I’m not designing this system for that. At least not now. Performance tests are highly specific and dependent on the exact machine it runs on, the exact third party libraries and their versions that are used, the other components involved in the tests, such as the servers, and more. I expect that we will change conditions for the tests every once in a while that makes it hard to compare the current numbers with past numbers. Therefore I think the performance test numbers and values are primarily useful in the short term. To help us spot if we land something that subtly and unintentionally degrades something. To detect extremely slow and long-term changes in performance and even making sure we can better survive wiping all the existing build logs etc, I introduced a concept I call stakes . As in a stake pole. A marker. An arbitrary threshold set manually for each specific test. This value can be used to measure performance test results against, now and later. As conditions change and maybe something makes the results go up or down and we are fine with those changes because they are motivated and expected, then we just change the stakes. If it works out, I might try to have the system automatically detect and maybe highlight tests that deviate too much from its set stake (at least if done in the wrong direction) . It could be a signal that something bad was merged. As with everything in life, things are often balanced out. We already ran into this when we eagerly merged several changes to reduce the number of allocations done for a single HTTP download, only to realize that one of the optimizations we did had the side-effect that it expanded the size one of the main structs maybe a little too much… Improvements in one area might come at an expense in another. With sufficient tests and data we can improve curl for users, and at the same time make sure that our changes don’t come with a cost we are not prepared to pay. Exactly how to make the balance is of course a question we need to deal with, discuss and decide. Possibly for every change we do! As I write this, we have 24 tests and a full test round completes in about six minutes on my machine. We can of course do multiple builds using different hardware, different operating systems, different build options, different third party libraries and different test servers to check more angles of performance, and I am certainly open for and prepared to do that going forward. I will however first let this single-flavor run for a while so that we get more data, get a change to tweak it and make it as usable as possible for curl developers. As with everything there is no end to what we can make this do. This is a start. I sure we can take it further as we move along. In particular if people join in and help out. Both with ideas and proposals for visualizations, graphs and new tests to add, but also with actual pull-requests and code. Over the last year, we have merged, on average, about 10 commits per day. If we keep this pace up and this performance test setup can show 100 build rounds conveniently into a single graph, that is just ten days of development. Probably not enough. Once we reach one hundred builds or so in the first graphs I need to consider adding separate long term graphs that use select data-points to display data development over a longer time. Some googling told me the Largest-Triangle-Three-Buckets, or LTTB for short, is a fine algorithm to use for this. I now do a separate “long term” graph that “downsamples” the full range down to something that can be shown in a reasonable way. I suppose we will see properly in the future how this works. The stake thing I mentioned is one way to help us spot gradual performance changes over time. Another googling told me that there’s a Mann-Kendall Test + Sen’s Slope algorithm to use to identify trends in graphs like this and it can be used to plot a trend. It might work as a helper to better identify… yeah, the data trend for each test. The HTTP/2 parallel download speed trend at a specific moment Developing This setup has only existed for a few days. There is lots to do, lots to learn and much more to experiment with. Your comments, help and pull-requests will be appreciated!

0 views
matduggan.com 2 weeks ago

OTel Isn't Going Well (And I Made A Spreadsheet About It)

For years now one of the most reliable complaints I hear when I try to drag a team off their vendor specific SDK and onto OpenTelemetry is some variation of: "why does it seem like this isn't done yet?" Vendor SDKs for observability are, to put it charitably, idiot-proof. You install the thing, dashboards just load data, someone else worries about how all those pieces fit together, and you get on with your life. OpenTelemetry, by contrast, greets you at the door with a lot of "experimental" stamps and roughly six different ways to accomplish any given task. In OpenTelemetry's defense this was never what they were going for as a project. I've always respect that they stuck to their guns by attempting to build a truly vendor agnostic system that really doesn't care what you do with the data. I have never gotten a sense of a vendor being strongly preferred with OTel, which is quite the feat considering how lucrative and contentious the observability ecosystem was. Also considering that the maintainers of this project are largely employed by exclusively those companies. As the years wore on, I started to get nervous. Conversations in the semantic-conventions repo drag on and on and on. Different languages had dramatically different stories. Golang and Dotnet were first class citizens, but other languages lagged years behind the others. I started asking a lot of probing questions before recommending OpenTelemetry to smaller teams who didn't have the time, budget, or emotional bandwidth for it. Auto-instrumentation was genuinely magical, but the cliff between "auto-instrument works" and "now I have to manually instrument something" was steep enough that you owed people a warning before you pushed them off it. This narrative has been going on for awhile in the observability space, a vague sense of "something is wrong in Otel-land". But let's try to generate some actual data here. Is there an actual problem, or is this something where the perception by the community of slow progress is imaginary? Is the problem not enough maintainers, too big of a scope, or something in-between? My guess when I started was "oh this is your classic open-source bit off more than they can chew". Not enough maintainers, not enough budget. Now there is some of that, but there's also something else going on. The actual problem happening inside of OpenTelemetry is a three way crash. You have a binary stability gate which, when combined with a very small bench of actual maintainers means there is understandable worry about marking a feature not experimental then add on just a massive scope of languages and frameworks they are attempting to cover. This creates a perfect storm where there is an incentive to argue about potential problems a feature might create since once it is locked in and shipped as stable you can never change them. So OpenTelemetry currently is attempting to support a dizzying number of languages and frameworks. OpenTelemetry is a g iant project. It spans dozens of languages, hundreds of libraries, and countless backends. To keep things sane, the project splits work into two buckets: There exists the otel-collector, the thing that runs along the thing so that you can ship logs metrics and traces. That copies the same rough pattern. But for the languages when we're talking about core vs contrib this is what we're talking about. Stuff that breaks goes in contrib, stuff that doesn't break goes into core. Now the reason this causes a conflict. is massive overkill for most projects. You don't want 300 exporters to add the one you typically need. On the language side, this isn't that big of a problem. gives you the stuff you need for flask. However on the collector side you end up having to do the OpenTelemetry Collector Builder to make your own collector (or just kinda ride the wave and hope it works out). While cool that this exists, it's a lot of scope to ask a team to take on. So I believe I have captured the workflow of adding a new feature to OTel. You can check my homework here: Things I'm not really clear on So because OpenTelemetry is a CNCF project, I figured it made the most sense to compare them to other CNCF projects. My basis for comparison is Envoy and Prometheus. I have used a hacky Python script I've used before for measuring the "health" of open-source projects, which is probably not the best. However I'll include a link to the raw data without the charts so folks can review it and (more than likely) find a problem in what I generated. So we look at 24 months of activity for Envoy and what we see is a pretty healthy project. There's good distribution of authors, mergers, issue closers. is obviously pretty important to the project but in general there's a good bench of people to step in if needed. I've attempted to filter out all the known bot traffic. Let's compare that to one of the OpenTelemetry languages. The ones I have the most professional experience with are Golang and Python, but I hear from a lot of folks in the community that the Ruby and PHP ones struggle a lot. This is the PHP one for the same period. So we see pretty clearly that there's way too much concentrated on 2 people. This is not a healthy open-source project and they clearly don't have enough people to cover the kind of scope OTel needs to cover. Same story with Ruby. In comparison the "strongest" OpenTelemetry SDKs in my opinion, Golang and Dotnet (although Python is also no slouch) look more healthy. So the first issue is maybe the least surprising. There's too much concentration among too few maintainers. Your authors shouldn't also be your mergers and your issue closers. Ideally these tasks should be distributed out more evenly. For what its worth I think the maintainers have done a good job of attempting to keep their discussions public. It was very easy for me to find the public meeting notes of the different groups of maintainers, read through them and see what was going on. I don't get the sense that these maintainers are trying to stop people from getting involved as much as the expectations of stability have, more or less, frozen the project in place. The issue is more a classic case of "someone has to pay the maintainers". The project is too complex for someone to realistically do this as a hobby. I think any project signing on for such long stability contracts cannot turn to the community of hobbyists expecting assistance. I can't join calls and do the things I would be expected to do for a project of this size and importance for free. But it also means that the people doing this critical work have expectations placed on them by their parent organizations. So these SDKs have too few maintainers. But that doesn't fully explain why it seems to take so long for new features to get through the stack. My guess for that was that somewhere in the process between submission of the new idea and the formalization of the idea was a long discussion that took a million years. So with this level of surface area across different frameworks and languages, it makes sense to concentrate the conversation about conventions in one place. That lives here: https://github.com/open-telemetry/semantic-conventions If vendor debate is causing the slowdown, we should (in theory) see this slowdown in PRs here. Then you should see the slowdown basically propagate out. Spoiler alert, I was wrong about this. Big thanks to the OpenTelemetry people for having good conventions on labeling their PRs which made this much easier. So if is the slowdown, let's look at the slowest PRs there. Yeah some of them are pretty slow, but there are some complex topics being discussed. However interestingly this slowdown doesn't really trickle into the SDK/API space, suggesting that OpenTelemetry is going a good job of keeping these conversations siloed off. If we look at Python we see that their slowest PRs aren't related. In reality the slowdown for these are the extra required check imposed by the which requires another maintainer. But that seems appropriate and takes us back to the initial problem of "not enough maintainers". So after looking at all of this, the pattern becomes clear. A new feature takes a very long time to make it to the end user in OpenTelemetry because they take stability very seriously, combined with a relatively limited bench of talent to pull from. Once things make it through the entire stack, implementing the API and getting that API change through to the end user falls on an overworked maintainer pool. So what do we do? I think one idea worth exploring is adding some sort of time-bound beta tier. Basically between the "Experimental" and the "Stable" in the following diagram. The problem is that for end users, due to the extra steps to use Experimental features, they might as well not exist. 99% of us have no idea when an experimental feature is added and we would never engage with it. But if I knew the feature would stick around for at least 12 months without a removal and was more accessible to me as an end user, it could actually help the project get more actionable feedback. Basically a feature would go Experimental (pretty low usage) -> Beta (more exposed to the end user than Experimental) -> 12 months -> Removal or Stable. Now confusingly Beta exists for Otel but is used for SDKs, not for components. Like Rust is a Beta but it seems like Profiles cannot be a Beta. Honestly it's nearly impossible for me to figure out like what labels should apply to what things. I suspect nobody really knows. Here's the explanation of Beta that I think only applies to SDKs. In addition it is, respectfully, misleading to imply that Go and Ruby are being maintained at the same standard. This isn't a shot at the Ruby folks — they are doing heroic work with what they have. But pretending parity exists when it doesn't just creates confusion and quiet resentment when a user shows up expecting one experience and gets another. Being honest about maintenance tiers would let people make informed choices and might attract more help to the other tiers by naming the problem out loud. Finally I would try to surface these problems more openly for OpenTelemetry from the perspective of "we need more maintainers". I feel like the people doing this work probably knew there was a problem, but it seems like the community at large has no idea that there is a need for frankly more engaged ideally independent maintainers and contributors. OpenTelemetry is a great project that is doing great work. It's doing, frankly, heroic work at this scale with this few people. But I think in order to actually replace the vendor specific SDKs we need to start getting a bit more pragmatic about what is realistic to do in terms of stability contracts and number of languages. I don't think breaking changes are as devastating to the community as these promises imply as long as they are communicated well and I think with this thin of a bench of maintainers, something has to give. Anyway feel free to check my data for accuracy and let me know if you find problems! Core → Maintained directly by the OTel project. Small, stable, vendor-neutral, and tightly reviewed. This is the "spec-defining" surface. Contrib → Community- and vendor-contributed. Broader, faster-moving, and covers the long tail of integrations. OpenTelemetry Enhancement Proposal (OTEP) ( https://github.com/open-telemetry/opentelemetry-specification/tree/main/oteps/ ) Once the OTEP is accepted, the text goes into the Specification directory in the same repo. After that it seems to go to Semantic conventions. This seems to be where we get down to the specific details and where most of the long discussions seem to live. At this point we're talking about more or less a permanent commitment to this design and where the lock-in process becomes very hard to change. Each of the SDKs implements the API surface that is defined in the specification. Now some of the SDKs have done 2.0 breaking changes, so it does seem like the earlier "please no 2.0 at all costs" sentiment has been abandoned (which I think is smart and good). Contrib / instrumentation. This is slightly more mushy. Looks like they should track latest API/SDK but each contrib package may version independently so its more flexible as a design. Collector + OTLP. The data has to actually go somewhere. OTLP (wire protocol) has its own stability lifecycle and specification ( here ). Collector components have their own stability in their READMEs and as far as I can tell that's kinda all over the place. It's unclear how long the OTEP -> Specification process takes. I've looked through the Git history but there doesn't seem to be any predictable number or cycle. I don't fully understand what is the relationship between all these stability commitments. Does Collector + OTLP group work in lockstep? Can a language "fall out of scope" if you lag too far behind?

0 views
Farid Zakaria 2 weeks ago

nixpkgs-multiverse is audacitymaxxing

Every package manager on earth picks one version for you. Nixpkgs picked one too. It never had to. I shared nixpkgs-multiverse recently: one flake input that hands you every version of every package that ever shipped in Nixpkgs. I love how unbelievable audacious Nix lets me be, audacitymaxxing . As of this writing, you have access to 31,783 packages and 304,484 distinct package version pairs pulled from 1,537 revisions. 🤯 The fact most distributions only give you one version of each package is not a bug. It is often considered a feature: a single self-consistent set of software that boots and runs together. It falls directly out of a shared global filesystem, the filesystem hierarchy standard (FHS), like , and . The purpose and existence of Nix is to eschew from that convention and allow multiple versions of the same package to coexist. Nixpkgs is a distribution built on that capability, and yet, it has been doing the same thing as every other distribution: picking one version of everything. nixpkgs-multiverse only supports, at the moment , top-level attributes that are packages but already the sheer volume of installable software dwarfs . 1 Nix’s answer to the FHS was audacious in 2003 and is still audacious now. A package lives at , where the hash is derived from every input that went into building it: the intensional model . How audacious are we? How about 246 distinct CPython versions, from 2.6.8 forward, all installable side by side, all built and cached, all addressable by version number instead of commit hash. 2 To re-iterate, these are distinct versions of CPython, including their transitive dependencies. There is no or or that is shared between them. 3 They work just as reliably as when they were first released, and they are all still installable today and can be substituted from the cache. People want to pin to a version. Upgrading software can be disruptive, and some people have to stay on a particular version but that should not impede the rest of the world from moving forward. The nixpkgs-multiverse helped solve one of the oldest devenv.sh issues, cachix/devenv#16 , the desire to pin a specific package. “It is not really practical to pin a separate version of nixpkgs for every different version of a tool needed in a dev environment. Normally we have at least 20-30 different tools all with a specific pinned version that we would want to specify.” – itpropro The issue, “Pinning a specific package”, was opened on 2022-11-10 and is now closed. devenv now documents the multiverse as the solution. 💪 The audacity of the multiverse is not technical. Nix took care of that. There is no clever trick in here; it’s 5 MB of JSON, about 200 lines of Nix and a behind a memo table. The audacity is in the premise. Two smaller things landed that I like and which was driven by feedback from the community. A soak period. gives you the whole of as it stood N days before an anchor, a cooldown window, in the spirit of Determinate Systems’ cooldowns , except the anchor can be any selector takes. Provenance. Every package set carries where it came from, so a you were handed can be interrogated rather than guessed at. The data was fetched from the Repology repository size map.  ↩ The data for other distributions was fetched from Repology .  ↩ Unless they happen to dedupe due to their hash.  ↩ The data was fetched from the Repology repository size map.  ↩ The data for other distributions was fetched from Repology .  ↩ Unless they happen to dedupe due to their hash.  ↩

0 views