Latest Posts (20 found)

My Monitor Was Cutting My Download Speed by 75%

Somehow I am one of those few very lucky people to have the opportunity of owning a 2.5 gig symmetrical line for my internet. But one day, when I wanted to brag to my equally nerdy friends, shock and horror: 600 Mbps down? Luckily my upload speed was still 2.4 Gbps, so I didn’t lose all my bragging rights, but that’s weird, right? If anything, you’d expect the opposite! My ISP is pretty cool. It’s run by this one guy (Hey Keith!), and I know I can always message them and they respond promptly, as they did this time. We tried to run through a few debugging steps, they checked on their end, and everything was normal. I tried my work laptop too, same speed.

0 views

A Town Called Discovery

Author: R.R. Haywood Genre: Sci-fi Released: 09 April 2020 Rating: ★★★★☆ A man falls from the sky. He has no memory. He has no sense of self. What lies ahead are a series of tests. Each more brutal than the last, and if he gets through them all, he might just reach A Town Called Discovery. Learn more on Goodreads ➡ RR Haywood continues to be one of my favourite authors and A Town Called Discovery was another really fun book. It's all set in the same world as the DELIO books, and the Extracted series , although I failed to detect any direct crossovers within this book there is an AI at the middle of it all (like DELIO), as well as time travel (like Extracted). The book starts really dark and weird - it felt like a combination of Battle Royale and Minority Report. True to form, early on I had very little idea of what was going on, which immediately pulled me into the story as I wanted to understand what was happening. The story ends in a very open way, and I really hope that Haywood is planning to write a sequel, although I can't find any mention of this on his site . The Code Trilogy is what I'm going to go onto next, as it's also part of the same world apparently. Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment .

0 views

Burns on cybersecurity

In Scotland, your health data is far more at risk from the NHS than from AI or China.

0 views

My Emacs 31 contributions

For me, landing changes in Emacs upstream always felt somewhat out of reach, so I kinda postponed it for a long while. Perhaps the biggest hurdle was getting over the process to assign copyright to the Free Software Foundation , which does feel a bit archaic (download, sign, and return a PDF) and slow (mostly waiting for an email response), though hey, I have nothing but sympathy. Open source is unsustainably tricky. As consumers, we often like free as in freedom and free as in beer, but neither pay the bills. We hope some company steps in to fund these projects, but that rarely happens. As a result, most open source projects are starved (even popular ones), run by volunteers working for free. All that is to say, if the upstream Emacs contribution/onboarding process seems less than ideal, cut it some slack. The project as a whole primarily runs on the blood, sweat, and tears of unpaid contributors. If you can, fund the maintainers you rely on (or better yet, get your employer to do so). If you've been thinking of contributing to Emacs upstream, don't procrastinate like me and just give it a try. Compared to the typical GitHub pull-request process, your first Emacs contribution may feel a little daunting and dated, but there's a lovely fuzzy light at the end of the tunnel. You see, Emacs has been around for 50 years . Very few software projects get to live this long. Landing contributions on a project with such a rich history is a real treat! But I digress… Let's get moving. Once I got over procrastinating, I finally got a handful of small contributions landed on the master branch. Thank you, Eli Zaretskii, Stefan Kangas, and Gerd Möllmann, for your help. From then on, I eagerly waited for them to formally make it to a stable release. Emacs 31 shipped the lot. I do enjoy a visual (and functional) tweak here and there , and a Dired tweak felt small enough to be the perfect candidate for my very first Emacs contribution. With that in mind, I tweaked by adding the user option, which hides the directory's absolute path, typically displayed at the top of Dired buffers ( bug#72272 , commit ). More on this in my Hide another detail post. At some point, I ran into a snag trying to run macosrec from my beloved . The process just crashed with a cryptic , so the issue wasn't super obvious. It turned out the utility required granting an OS permission that can only be requested by the host app (Emacs) if it includes the relevant description in its file. As of version 31, Emacs includes in its ( bug#72827 , commit ). now includes a "Send to…" item, enabling sending the current file(s) or region text to external apps or services. You can also invoke directly. On macOS, it's powered by the native sharing picker (I use it frequently to AirDrop things to my iPhone). Elsewhere, it falls back to ( bug#76120 , commit , follow-up fix ). These two implementations plug into new generic plumbing ( ), so you could, for instance, add a handler sharing via Android's native share sheet ( can do the heavy lifting) or KDE Connect on Linux ( ). More in my Emacs send-to (aka macOS sharing) merged upstream post. Emacs 30 lost macOS dictation when migrating to . Implementing in brought it back ( bug#79070 , commit ). With LLMs/agents being a thing these days, dictating into your Emacs buffer may be just the thing to complement your AI workflow. I covered this in my macOS dictation returns to Emacs (fix merged) post. Are you getting value from any of my upstreamed changes or my Emacs packages? Please consider sponsoring to keep the work going. While my patches may seem relatively straightforward, some took quite a bit of effort to land upstream . My Emacs work not your cup of tea? No worries. As mentioned earlier, sponsor the maintainers you rely on instead.

0 views

The Epistemic Jerk

In physics, jerk is the rate of change in acceleration. It’s the feeling in your body when a driver slowly accelerates before slamming on the gas. While thinking about AI, I keep returning to this word: jerk . It captures the feeling of suddenness and surprise at the staggering rate of change. Within roughly two years, many programmers have gone from writing code “by hand” to using AI to code nearly everything. And it isn’t slop anymore. Coding agents can convert mockups into websites, write bug patches for the Linux kernel, and reverse-engineer compiled binaries. Overnight, the experience of programming has changed. Many developers point to Anthropic’s release of Opus 4.5 as the moment agentic coding started to work. The model was released less than a year ago. Similarly, a few years ago LLMs would routinely fumble basic arithmetic. Then Google’s Deep Think achieved gold-level performance at the 2025 IMO. Then OpenAI said its internal model had produced a Lean proof resolving the Navier–Stokes Millennium Prize problem. Then OpenAI dumped over 700 papers on GitHub, with results ranging from the unique games conjecture to quantum magnets. The progress is so rapid that the math community is having a collective and public reckoning. For myself, this has been humbling. I started my PhD in machine learning in 2016, a year before the transformer. I’ve published a paper using deep learning to extract latent representations from histology slides. I’ve written about LLMs and hosted vibe-coding hackathons for friends. And I write code and solve technical problems every single day. I guess my point is this: I’ve had my eye on the puck for a while now, and I am still shocked at the rate of progress. What happens next? AI safety researchers warn about misalignment and existential risks. Tech workers worry about job losses and a shock to knowledge work. Accelerationists argue we should move even faster to maximize technological progress. And market bears say that AI capex debt is a house of cards that will eventually crash the economy. Meanwhile, most people’s view of AI is shaped by outdated chatbots. It’s a disorienting time. As I’ve tried to make sense of the moment, I keep circling the same inchoate thoughts. Here are some of them. We are all fitting superlinear functions to a handful of data points. In Bayesian inference, your posterior distribution can widen (more information doesn’t always make you more certain). In a world in which specialized skill becomes commoditized, people with problems to solve will thrive relative to people who want to be told what to do. It is an insidious failure mode to solve problems you don’t actually care about simply because AI has made them easier to solve. I struggle to tell the difference between cognitive surrender and a higher level of abstraction. I derive a lot of existential pleasure from deep understanding, and I don’t want to lose that. One kid is using AI to do their math homework, while another is using AI to teach themselves calculus. The transistor isn’t even a hundred years old. I have never made a value-aligned decision from a place of fear. Do not mistake the shock or uncertainty of this moment for fatalism or powerlessness. While my thoughts are incomplete, my conviction is not. I keep telling friends about this time I watched Hikaru Nakamura play bullet chess on his live stream. He was chatting casually to the audience while moving at lightning speed when he suddenly paused and burnt most of his remaining time deciding his next move. After he won, he talked about the importance of that move. I forget the details, but he had sensed a shift in the game, a pattern that felt off. He paused because the next move was critical. That is what this moment feels like to me. So this is a time capsule for myself, a record of this feeling. Genuinely, I don’t know what happens next, but I can feel in my shoulder blades that the game has changed.

0 views
ava's blog Yesterday

chronic illness and personal time

When I have a random flare up of my chronic illnesses (or my treatment stops working), my free time is no longer something I have much control over. It's constrained and controlled by illness management. Most of it is dedicated to doing the bare minimum, getting myself through the day, distracting myself, managing pain, and resting - it's what I have to do to either endure it or not worsen it. That's a very common experience for basically all chronically ill people. From the outside it looks like self care, like some really good me-time: Taking baths, napping, scrolling, getting massages, immersing yourself in something easy you can handle right now, for hours a day, weeks, taking it slow and not doing much. Like retreating into your own cocoon. People in your life are waiting on you to reemerge better and ready to go; you got your fill of lonesome me-time after all. From the inside, it's different: You feel limited in what you can do and how much of it, so the things you do are the only things you can do, but not necessarily want to do. Lying in bed (usually a nice activity) suddenly isn't a voluntary choice, it's forced upon you, and maybe it's in a hospital, too. It doesn't feel particularly replenishing when it's most of what you can do. Maybe you wanted to do this other thing for four hours, but only got to do it for one. This flare up broke your routine, and you wanna get back to it when you finally can. This or that hobby has to pause until you feel better, and no hot bath can change that. You're falling behind in the things you need to do. It's not really feeling like self care, it feels more like a prison sentence, and things keep piling up, both physically and emotionally. In other people's eyes, once this silly flare up is over, they expect that you are finally available again and have time for them - to go out, to engage in hobbies together, to spend more time actively doing something instead of them just taking care of you or waiting for you. They too may have kept a backlog in their head for what they wanna do with you once you're well again. For the chronically ill person, the list of things they want to do has been piling up over weeks as well; and once they feel better, they're invested in getting their time back and finally getting to do the things they enjoy or that they were prevented from doing. Maybe that means doing more things more intensely to make up for the weeks they couldn't. Yes, that may include other people too, like being happy that you are finally well enough to go to that meetup or make it to the concert with your friends, but more often than not, it also includes very isolated hobbies that can be done independently, often at home - they're most suitable for chronic illness, after all. In my case, that may be things like exercising, or studying for more hours and catching up on the study load I had to drop, writing blog posts for a couple hours, finally taking on more cases for my volunteering position again, working away on my reading backlog, playing games for longer, and more. This results in possibly not getting my attention for basically the entire day, if I don't catch myself and make a dedicated effort to reach out to people, take breaks, and actively choose to spend time with my wife even if I have a list of other things in my head, for example. It's kinda terrible, because for quite a while, you might not be able to do much of anything at all while the people in your life are patiently waiting. Then when you can, you feel like you have to use these precious good illness days and first pour into yourself, while possibly neglecting others who had hoped that finally, you'd be open to more time and activities together... with the full you, not the brain-fogged, dimmed version of you. Being aware of this is the first step, but it's hard to manage the different expectations and find a good balance, especially if the hobby can't just be shared, needs quiet or focus, or can't easily be done in just small chunks. It's difficult for me because I really need to immerse myself and end an activity when I feel like it, not because of external constraints. Stopping prematurely or being interrupted makes it hard for me to emotionally regulate or focus on anything else. I also have enough stuff to do to just fill an entire day and more. I'd just keep going if unchecked, and my hobbies and interests are really independent. Chronic illness can be weirdly selfish... and maybe, that is somewhat unavoidable. One week it's about what feels good to you right now during all the pain, another week is about what you can or cannot do in this limited state and full of no's, the next week is all about what you want to do now that you're feeling good. It's just us, us, us and our needs. It looks like we spend our newfound energy running away from everyone. That can be especially painful to people that actively take care of us. Some weeks, our own plate is so full with illness-related things that we have a hard time checking in with anyone about their lives and problems or helping them with anything, and then the other weeks, we're too busy by ourselves playing catchup with our goals and desires that fell by the wayside that we once again don't do it. That cycle may repeat. It's easy to be almost completely unaware of this recurring pattern, but it deserves to be addressed and talked about. There needs to be some understanding by the a chronically ill person's surroudings that once things are going well again, there is a strong desire to finally make up for lost time, even alone, and that the sick time was not fulfilling me-time at all; but there also needs to be awareness on the side of the chronically ill person that it's easy for them to lose sight of others when constantly managing something so personal that puts their own needs front and center. Published 14 Nov, 2026

0 views
Farid Zakaria Yesterday

beer.drv.drv.drv

Have you heard about dynamic derivations ? They are an experimental-feature in Nix that lets a derivation produce other derivations as its output. Why ? This lets you dynamically create your build graph, instead of having to know it all at evaluation time. It’s Nix’s answer to try and match the build speed of other build systems like Bazel or buck2 . There is not much written about dynamic derivations, and the documentation is sparse. I have written a few posts about them, so I sometimes get questions about them. Recently, I got a DM from Evan Porter over LinkedIn , the ultimate bug triaging network, about whether it is possible to query the deriver of a dynamic derivation to get the initial derivation. For dynamic derivations let’s say I have a .drv.drv file. This file outputs a derivation. Is it usually possible to query deriver the output derivation to obtain the initial derivation? They had tried it and got this: That should work. 🤨 Every path a build registers carries a , the that produced it, and a that came out of a build is still a path a build registered. When I tried it, it worked. 🤷 You will need in your to drive the samples in this post. Turns out, there is a case where you do get if you set on the producer. If the producer is content-addressed, its output path is not known until after the build, and Nix does not fill in the of a path that already exists. I filed NixOS/nix#16823 . The question though made me wonder how far back you can walk the chain of dynamic derivations. I had to try it. 😈 Let’s pick a fun example to see how far we can go. Here is a derivation that imitates the classic song 99 bottles of beer by producing a chain of derivations, each one outputting the next verse. 1 Tip A derivation whose output is a must set . In our example we are using a content-addressed derivation, but the same idea works with an input-addressed one, but it’s more work to compute the output path in the shell, unless you are using to do it for you with recursive Nix. The Nix side builds the first derivation and then follows once per bottle. While working through this example, I hit a bug in Nix that made me set on the top derivation. That is not necessary for a derivation that outputs a , but it works around the bug. I filed NixOS/nix#16830 . You might notice that we limit the number of beers to 50. Why 50 and not 99? Well, 50 is the most that fits in a store path. The maximum length of a store path is 211 characters . A derivation’s store path must include the suffix, and every in a chain adds four characters. The top derivation is named and if you add 51 s we get 208 characters. No room for any more beer. 🍻 Let’s build sing it. Fifty-one builds, one after another 2 . Evaluation knows about only the first one. Nix discovers each of the other 50 by building the one before it. 🍺 Back to the original question: can we query the deriver of the output derivation to get the initial derivation? Yes, we can. Let’s put in a loop that walks from the result all the way back to the top. We have the result plus 51 derivations. 💪 Since every derivation in the chain was built, every one has a build log, and you can ask any of them to sing its verse again. I’m very excited for the future of dynamic derivations. tomberek is leading a working-group to finalize the feature and are currently meeting weekly. With dynamic derivations, Nix will be able to compete with other build systems in terms of speed and flexibility and take over the world . Nix the world. The builder writes the next derivation by hand in ATerm , so there is no recursive Nix and no involved.  ↩ A single trivial build takes about 1.5 seconds on my laptop, so the cost of building 51 derivations is about 75 seconds largely due to substituter lookups. If you disable substituters , the builds take about 1.3 seconds in total.  ↩ The builder writes the next derivation by hand in ATerm , so there is no recursive Nix and no involved.  ↩ A single trivial build takes about 1.5 seconds on my laptop, so the cost of building 51 derivations is about 75 seconds largely due to substituter lookups. If you disable substituters , the builds take about 1.3 seconds in total.  ↩

0 views
ava's blog Yesterday

what i read this week - week 46 2026

Haven't done these posts since August! I've also generated quite the backlog since end of September probably, now I'm working that off :) [EN] EDPB Statement on Smart Glasses - 8 page statement of the European Data Protection Board on smart glasses. Among the usual known issues around them like consent, data minimization and transparency obligations, they also point out the difficulties around identifying the controllers (in the sense of Article 4 Nr. 7 GDPR) of the data processing - depending on the process and context, it couldn't just be the owner or wearer of the glasses, it could also be the producer of the device (like Meta), or the provider of a linked application or service (like OpenAI, if ChatGPT was used), which means joint controllership is possible. The document also discusses the GDPR's household exemption and when it stops applying to the recording via smart glasses. [EN] EDPB Guidelines 03/2026 on web scraping in the context of generative AI - 23 page guidelines from the European Data Protection Board on web scraping done by bots to train AI, first version from July this year. As far as personal data is concerned, the GDPR also applies here, yet its requirements are easily circumvented, with almost no way to catch it or hold scrapers accountable. It's clear that scraping has the goal of maximizing data acquisition (quite the opposite of the data minimization principle) and it's usually not feasible to notify affected all parties of the processing of their data, so they don't have to. Scrapers tend to not discriminate very strongly what kinds of data they scrape (unless it is for a very specific project and on limited websites only), so within it all, there could be data that falls under special categories like Article 9 GDPR, so scraping it is a clear violation. Incidental and accidental collection of this data might be permissible if enough technical- and organizational measures have been taken to avoid this as much as possible, but this directly clashes with the interest to get as much data as possible for AI, so this is sadly an unrealistic standard to expect. It's also questionable whether the purposes can be defined in a way that satisfies legal requirements and doesn't cause a purpose change later on. The guidelines suggest considering using synthetic data instead of personal data, defining precise collection criteria, applying filters to exclude the collection of certain categories of data, and excluding websites which structurally contain these data and websites which clearly oppose the scraping of their content. [EN] From a Right-Based Approach to Competitiveness? The EU Digital Omnibus Reform - 18 pages published in the Journal of Risk Regulation by Giovanni De Gregorio and Hannah Ruschemeier, the latter who used to be my professor and who motivated me to make the jump into data protection law. The paper discusses the standard criticisms of the digital omnibus, mainly the increasing risks for fundamental rights and legal certainty in favor of competitiveness, and the lack of enforcement. It also mentions an underdiscussed point in the current reworks: The changes and relaxing of certain requirements, definitions and standards actively rewards companies who either operate in bad faith or have simply never invested in complying while flying under the radar, while it's penalising those that have invested in rigorous legal compliance who now have to change processes, documents and internal rules all over again. [EN] Growing Up With Generative AI - 44 slides about a YouGov online survey on behalf of interface in France from June 2026 about the youth experience of growing up with AI. 1,138 parents of children aged 11 to 17 and 970 teenagers aged 11 to 17 were asked. Key takeaways are that AI is widely used among children in those ages, and parents tend to underestimate the use of their children quite significantly unless it is AI companions. Regular use of generative AI increases with the child’s age, even though confiding in AI is more common among prepubescent children aged 11-12 (concerning). [EN] Working Paper on Extended Reality 28 pages from the International Working Group on Data Protection in Technology exploring what Extended Reality is (umbrella term for Augmented Reality, Virtual Reality, and Mixed Reality (“MR”)) and what privacy risks apply. This sort of tech usually needs a lot of cameras and sensors to function and may collect biometric data, and infers a lot of things via AI and other means. Nothing really new or mind-boggling here, the standard risks apply, more or less, especially when it comes to wearable tech (VR headsets), third parties being captured during use, and sharing location to use (like Pokemon GO). [DE] KI-Taskforce Bericht: Deutschland als KI-Nation 2039 - 39 page PDF about Germany's AI future that was written exclusively by ministries, no citizens, no NGOs or companies. Much of it reads more like vague advertising than concrete steps and plans. What's especially disappointing is that while they can't stop talking about sovereignty and acknowledging that if you don't start early, you're always gonna be reliant on others who are far ahead, they're still planning on being almost completely reliant on foreign tech, especially US giants. EU models shall only serve as fallback options, except for their pipe dream of some highly specialized AI tools that are exclusive to German knowledge/commercial niches that is supposed to catapult us forward. The social and ecological questions don't really get addressed aside from making sure everyone can have access to AI freely and equally. Not much to be found on how the disruption of the job market and possible mass unemployment will be dealt with specifically, as well as deepfakes and misinformation and its effects on society and elections; just that they are problems and shouldn't be. The document also has very arbitrary, hype-based goals whose numbers seem very random and not grounded in any actual need or progress. All in all, a document and plan that's kinda like the bad and late kid in class who saw the other groups present first, reworked parts of their presentation on a whim after seeing those, and now pretends they had this stuff all along, but doesn't know what they're talking about, pretending to be one of the top students in class. [DE] Stellungnahme der DVD e. V.und des Netzwerks Datenschutzexpertise zum Gesetzentwurf zum Einsatz von KI in der Migrationsverwaltung (KIMVG) - 21 page statement of two privacy groups in Germany regarding the planned use of AI in migration management and edits of AsylG and AufenthG to facilitate this data processing. The statement endorses the plan (yikes!) based on the amount of work in that area making it difficult to decide the applications within the required time frames, but thankfully nonetheless corrects some false understandings and points out some risks and necessary standards as well (especially around the automation and the classification of high risk AI). I really don't want to entrust AI with making automated decisions on something so serious, life or death in many cases, and it's not good enough for me if there is a human in the loop somewhere just nodding about the AI assessment. I also consider it disrespectful if one party creates a convoluted process to apply for asylum or other forms of immigration which demands a ton of documents, just to whine about being overwhelmed by the documents and deciding they no longer really want to see them personally. There are different ways German administrations could thin out and quicken the process without making someone jump through hoops and then avoid looking at what was asked for. I don't want extremely vulnerable groups to have to agree to this sort of invasive processing of their personal data just for a chance at safety and a new life. [DE] Digitale Grundrechte sichern – Datenschutz einfach, wirksam und föderal stärken - 5 page motion from several politicians and Grüne at the Bundestag, calling for more data protection. They're calling out the open assault on our digital human rights and the practice of blaming data protection and privacy concerns for anything from IT failures to commercial flops. They call, among other things, for a modernization of our federal DPA system instead of centralization and the continued support and further development of the GDPR, and are against sweeping data privileges for AI. [DE] Nein, Europa ist keine Kolonie von Big Tech - Arguments on why the EU is not victim of digital colonialism by the US. We voluntarily bought into it out of convenience, we were not forced into it, and we still profit from colonialism today. [EN] Who is reducing your privacy? - informative site from noyb about who votes against your privacy interests in the EU. [EN] We can’t just change the definition of ‘recording’ - Article by The Verge about the newest attempt of tech companies to make you comfortable with always-on recording if it isn't saved and highlights are immediately processed by AI during use, as if just saving recordings was the issue and not everything else around it (like chilling effects, or the fact that devices now do the recording that seemingly shouldn't be capable of it). [EN] You’ve Changed. Shouldn’t Your Algorithm? - Tech Policy Press article on interesting considerations and problems behind your interests and needs changing while your algorithm keeps you boxed in, plus conflicting behavior between what you say you want (let's say, cat videos) and then engaging with what you don't want (because it is shocking, enraging, etc.) even though you know you shouldn't, training your algorithm to show you more of the things you say you don't want. Fitting quote from the article: " A crisis can command our attention without defining what we want to understand next month. A reader may want to keep following a local newspaper while receiving fewer unsolicited disaster videos. " More granular controls around what you want to see and don't wanna see, with agreed upon endpoints, are required. [EN] Humans Have Rights. Machines Do Not. - Tech Policy Press article covering the attempts of the AI industry to argue that their chat bots' outputs are their "own opinions", therefore protected speech, and that their models deserve human rights such as the freedom of speech, all to avoid liability for its outputs and consequences on the users. The result is " agency without accountability ". [EN] Tech Companies Are Trying to Shift the Entire Concept of Privacy (Again) - Electronic Frontier Foundation post on a topic I have also touched on : Privacy is a team effort that should extend to everyone, yet tech companies are trying to offload responsibility onto users and seem to only keep the person using the device a bit more private (if at all), and see everyone else the user is surrounded by as fair game. It requires you to become an expert and dodge an assault of things while being at the mercy of everyone around you. That's by design! Published 14 Nov, 2026 [EN] EDPB Statement on Smart Glasses - 8 page statement of the European Data Protection Board on smart glasses. Among the usual known issues around them like consent, data minimization and transparency obligations, they also point out the difficulties around identifying the controllers (in the sense of Article 4 Nr. 7 GDPR) of the data processing - depending on the process and context, it couldn't just be the owner or wearer of the glasses, it could also be the producer of the device (like Meta), or the provider of a linked application or service (like OpenAI, if ChatGPT was used), which means joint controllership is possible. The document also discusses the GDPR's household exemption and when it stops applying to the recording via smart glasses. [EN] EDPB Guidelines 03/2026 on web scraping in the context of generative AI - 23 page guidelines from the European Data Protection Board on web scraping done by bots to train AI, first version from July this year. As far as personal data is concerned, the GDPR also applies here, yet its requirements are easily circumvented, with almost no way to catch it or hold scrapers accountable. It's clear that scraping has the goal of maximizing data acquisition (quite the opposite of the data minimization principle) and it's usually not feasible to notify affected all parties of the processing of their data, so they don't have to. Scrapers tend to not discriminate very strongly what kinds of data they scrape (unless it is for a very specific project and on limited websites only), so within it all, there could be data that falls under special categories like Article 9 GDPR, so scraping it is a clear violation. Incidental and accidental collection of this data might be permissible if enough technical- and organizational measures have been taken to avoid this as much as possible, but this directly clashes with the interest to get as much data as possible for AI, so this is sadly an unrealistic standard to expect. It's also questionable whether the purposes can be defined in a way that satisfies legal requirements and doesn't cause a purpose change later on. The guidelines suggest considering using synthetic data instead of personal data, defining precise collection criteria, applying filters to exclude the collection of certain categories of data, and excluding websites which structurally contain these data and websites which clearly oppose the scraping of their content. [EN] From a Right-Based Approach to Competitiveness? The EU Digital Omnibus Reform - 18 pages published in the Journal of Risk Regulation by Giovanni De Gregorio and Hannah Ruschemeier, the latter who used to be my professor and who motivated me to make the jump into data protection law. The paper discusses the standard criticisms of the digital omnibus, mainly the increasing risks for fundamental rights and legal certainty in favor of competitiveness, and the lack of enforcement. It also mentions an underdiscussed point in the current reworks: The changes and relaxing of certain requirements, definitions and standards actively rewards companies who either operate in bad faith or have simply never invested in complying while flying under the radar, while it's penalising those that have invested in rigorous legal compliance who now have to change processes, documents and internal rules all over again. [EN] Growing Up With Generative AI - 44 slides about a YouGov online survey on behalf of interface in France from June 2026 about the youth experience of growing up with AI. 1,138 parents of children aged 11 to 17 and 970 teenagers aged 11 to 17 were asked. Key takeaways are that AI is widely used among children in those ages, and parents tend to underestimate the use of their children quite significantly unless it is AI companions. Regular use of generative AI increases with the child’s age, even though confiding in AI is more common among prepubescent children aged 11-12 (concerning). [EN] Working Paper on Extended Reality 28 pages from the International Working Group on Data Protection in Technology exploring what Extended Reality is (umbrella term for Augmented Reality, Virtual Reality, and Mixed Reality (“MR”)) and what privacy risks apply. This sort of tech usually needs a lot of cameras and sensors to function and may collect biometric data, and infers a lot of things via AI and other means. Nothing really new or mind-boggling here, the standard risks apply, more or less, especially when it comes to wearable tech (VR headsets), third parties being captured during use, and sharing location to use (like Pokemon GO). [DE] KI-Taskforce Bericht: Deutschland als KI-Nation 2039 - 39 page PDF about Germany's AI future that was written exclusively by ministries, no citizens, no NGOs or companies. Much of it reads more like vague advertising than concrete steps and plans. What's especially disappointing is that while they can't stop talking about sovereignty and acknowledging that if you don't start early, you're always gonna be reliant on others who are far ahead, they're still planning on being almost completely reliant on foreign tech, especially US giants. EU models shall only serve as fallback options, except for their pipe dream of some highly specialized AI tools that are exclusive to German knowledge/commercial niches that is supposed to catapult us forward. The social and ecological questions don't really get addressed aside from making sure everyone can have access to AI freely and equally. Not much to be found on how the disruption of the job market and possible mass unemployment will be dealt with specifically, as well as deepfakes and misinformation and its effects on society and elections; just that they are problems and shouldn't be. The document also has very arbitrary, hype-based goals whose numbers seem very random and not grounded in any actual need or progress. All in all, a document and plan that's kinda like the bad and late kid in class who saw the other groups present first, reworked parts of their presentation on a whim after seeing those, and now pretends they had this stuff all along, but doesn't know what they're talking about, pretending to be one of the top students in class. [DE] Stellungnahme der DVD e. V.und des Netzwerks Datenschutzexpertise zum Gesetzentwurf zum Einsatz von KI in der Migrationsverwaltung (KIMVG) - 21 page statement of two privacy groups in Germany regarding the planned use of AI in migration management and edits of AsylG and AufenthG to facilitate this data processing. The statement endorses the plan (yikes!) based on the amount of work in that area making it difficult to decide the applications within the required time frames, but thankfully nonetheless corrects some false understandings and points out some risks and necessary standards as well (especially around the automation and the classification of high risk AI). I really don't want to entrust AI with making automated decisions on something so serious, life or death in many cases, and it's not good enough for me if there is a human in the loop somewhere just nodding about the AI assessment. I also consider it disrespectful if one party creates a convoluted process to apply for asylum or other forms of immigration which demands a ton of documents, just to whine about being overwhelmed by the documents and deciding they no longer really want to see them personally. There are different ways German administrations could thin out and quicken the process without making someone jump through hoops and then avoid looking at what was asked for. I don't want extremely vulnerable groups to have to agree to this sort of invasive processing of their personal data just for a chance at safety and a new life. [DE] Digitale Grundrechte sichern – Datenschutz einfach, wirksam und föderal stärken - 5 page motion from several politicians and Grüne at the Bundestag, calling for more data protection. They're calling out the open assault on our digital human rights and the practice of blaming data protection and privacy concerns for anything from IT failures to commercial flops. They call, among other things, for a modernization of our federal DPA system instead of centralization and the continued support and further development of the GDPR, and are against sweeping data privileges for AI. [DE] Nein, Europa ist keine Kolonie von Big Tech - Arguments on why the EU is not victim of digital colonialism by the US. We voluntarily bought into it out of convenience, we were not forced into it, and we still profit from colonialism today. [EN] Who is reducing your privacy? - informative site from noyb about who votes against your privacy interests in the EU. [EN] We can’t just change the definition of ‘recording’ - Article by The Verge about the newest attempt of tech companies to make you comfortable with always-on recording if it isn't saved and highlights are immediately processed by AI during use, as if just saving recordings was the issue and not everything else around it (like chilling effects, or the fact that devices now do the recording that seemingly shouldn't be capable of it). [EN] You’ve Changed. Shouldn’t Your Algorithm? - Tech Policy Press article on interesting considerations and problems behind your interests and needs changing while your algorithm keeps you boxed in, plus conflicting behavior between what you say you want (let's say, cat videos) and then engaging with what you don't want (because it is shocking, enraging, etc.) even though you know you shouldn't, training your algorithm to show you more of the things you say you don't want. Fitting quote from the article: " A crisis can command our attention without defining what we want to understand next month. A reader may want to keep following a local newspaper while receiving fewer unsolicited disaster videos. " More granular controls around what you want to see and don't wanna see, with agreed upon endpoints, are required. [EN] Humans Have Rights. Machines Do Not. - Tech Policy Press article covering the attempts of the AI industry to argue that their chat bots' outputs are their "own opinions", therefore protected speech, and that their models deserve human rights such as the freedom of speech, all to avoid liability for its outputs and consequences on the users. The result is " agency without accountability ". [EN] Tech Companies Are Trying to Shift the Entire Concept of Privacy (Again) - Electronic Frontier Foundation post on a topic I have also touched on : Privacy is a team effort that should extend to everyone, yet tech companies are trying to offload responsibility onto users and seem to only keep the person using the device a bit more private (if at all), and see everyone else the user is surrounded by as fair game. It requires you to become an expert and dodge an assault of things while being at the mercy of everyone around you. That's by design!

0 views
Unsung Yesterday

“And honestly, that feels like magic.”

This is a really fun video by 3Blue1Brown about the cleverness and elegance of Hamming codes, simple algorithms that help recover data when some of it gets mangled – for example, if you scratch your CD disc: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/and-honestly-that-feels-like-magic/yt1-play.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/and-honestly-that-feels-like-magic/yt1-play.1600w.avif" type="image/avif"> The default assumption for complete data protection might be to have a second, backup copy to be sure. But Hamming codes not only promise the same kind of safety in a fraction of the space needed – they also can point to where exactly the data was damaged, and in many cases they can even fix the damage, for free. And the code required is trivial, too . It reminded me quite a bit of this video about Unicode , another example of beautiful and smart encoding.

0 views
Unsung Yesterday

A few more style guides

A while back I shared a bunch of interface guidelines and style guides , and I found a few more since then. I will update the previous link so it can stay a reference, but here they are as well: GEM User Interface Guidelines (1980s?) and GEM/TOS style guidelines (1980s? maybe?) One Laptop Per Child Human Interface Guidelines (2006) Windows Phone 7 UI Design and Interaction Guide – pre-release (early 2010) and version 2.0 (July 2010) Android Design (2012) Samsung One UI Design Guidelines (2018?)

0 views
Robin Moffatt Yesterday

Itches I Have Scratched (November 2026)

My background is databases, SQL, ETL, all that kind of thing. Not coding. Well, maybe a hack (and a dirty one) at best - BBC Basic in the old days, VBA in Excel a long time ago, and more recently even trying to learn Go . All enough to just about get by, none to seriously trouble anyone for whom this is actually their job. And I was quite content with continuing my next decades as I’d spent my previous ones, thinking of fun software, occasionally finding the time to hack at something trivial, but nothing more. And then agentic coding came along, and turned my world upside down :)

0 views
Kev Quirk Yesterday

Microsoft's Outage Showed Me Just How Distracting Email Can Be

I spend a lot of time in email at work - to the detriment of a lot of other stuff, actually. My default setup at work is Outlook on 2/3 of the screen, with Teams on the other 1/3. That's my "background", then I have anything I'm currently working on over the top, like a document or VS Code. During yesterday's Microsoft outage the company I work for had no email, and Teams was working sporadically. So instead of my default behaviour of immediately checking my emails, I opened my to do list and started working through that instead. I was so fucking productive! Outlook is a staple on my machine. I pride myself on having a very organised inbox and calendar. This means I'm constantly checking my emails at work. Which is good, as I'm usually quite responsive, but if I have other tasks to do, they end up taking a backseat. This is usually what happens: I'm a very organised person, so my shit always gets squared away - I rarely "drop a ball", but the constant context switching can be very tiring. There's been a tonne of research 1 done about context switching and how bad it can be for your productivity, and by Microsoft kindly removing the main culprit for my context switching yesterday, it really helped. I pretty much cleared out my entire to do list in a single day! I think I'm going to change up how I do things. I'll still keep Teams open, as it's how people get hold of me when they need me urgently, but I'm pretty good at ignoring Teams until I'm ready to chat. Outlook is trickier though, as I use it so much for my calendar too. Teams has my calendar, but it doesn't display category colours (told you I was organised!) so I'll need to work around that. But my plan is to keep Outlook closed and only open it a couple times per day to check email. I'm hoping that this will keep me more focused on the task I'm currently working on. I'm not sure how long this will last, or if it will work at all, as my method of working with Outlook is pretty much muscle memory at this point, but we shall see. It'll be an interesting experiment. Do you have manage your inbox in a similar way? If so, let me know using the reply button below - I'm always interested in learning new productivity tips. Studies from the APA , ScienceDirect , UC Irvine , and Information Age .  ↩ Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment . I have a break between meetings in my calendar, so I start something from my to do list. An email comes in that I need to take action on, so I park the thing I'm working on and get stuck into the latest fire drill. My to do item gets put back to another day when I have more time. Studies from the APA , ScienceDirect , UC Irvine , and Information Age .  ↩

0 views

Building Kakuro Zone

I built Kakuro Zone ! When I go on work trips, I often use hotel business centers to print out some paper puzzles to do if I’m bored in the hotel room or on the airplane. I almost always do sudoku puzzles, but wanted to mix it up. As I started exploring different puzzles, I discovered kakuro ! They’re a type of logic puzzle very similar to a crossword puzzle, except you add numbers across and down to fulfill rows and columns. It’s super fun to do with paper and pencil! There’s quite a few kakuro puzzle websites out there, so I didn’t have to make a new website for them but: With the power of AI, I actually got pretty far a couple months ago! I set it up to be a React and Vite PWA, and got my printable puzzles all done in time for the WeAreDevelopers conference! I had printed puzzles for the week, and was able to play a bit on my phone on the plane as well. …and then I promptly ignored the repo until this week, because it solved my needs, ha. After the hubbub of GitHub Universe died down, I remembered that I should open source the game and clean it up so that others can play it too. This is, admittedly, a fairly vibe-coded project. Or should I say, agentically engineered . I did some touching of code, mostly in the CSS. I really cared about the what things should look like on different screen sizes, and how navigation should feel with a keyboard or on mobile, but besides that, I let GitHub Copilot fly here. It felt weird letting go… but it was also nice to just get the puzzle experience the way I wanted, without overthinking implementation logic. Here’s the repository , and here’s the game again for you to try out! I wanted to print them in a certain way that would fold well in my notebook I liked the idea of using the date as a “seed” for a puzzle, so I could have a friend do the same puzzle as I did on a given day I wanted to have the game on my phone if I didn’t print it out (but felt a hankering to solve one) I wanted to play with Neobrutalism.dev

0 views
Sean Goedecke Yesterday

How to find meaning in large tech companies

Why do many people hate working in large tech companies? The obvious answer is “because organizational dysfunction stops them from doing the things they want to do”. Engineers like to write code and ship things, and if their company’s dysfunction is preventing that, it’ll make them unhappy. Right? I don’t think so, actually. If that were the case, those people would be interested in understanding the dysfunction and learning how to work around it, so they can do more of the things they want to do 1 . However, many people seem to actively avoid that. They find even contemplating the dysfunction to be as unpleasant as actually dealing with it. I think the reason for that is similar to why most people don’t like working in a messy room. People — particularly engineers — have a compulsion towards neatness. As I’ve written about before , it’s often our weird internal compulsions that drive our professional selves. If you feel a visceral disgust towards organizational dysfunction, spending any time thinking about it probably feels like being tainted. In Anathem , Neal Stephenson writes about a strange punishment where initiate scholars are forced to study “The Book”: a text written to be both complex and entirely meaningless. It takes a lot of intellectual effort to master The Book, but there’s absolutely zero useful content there. Studying it is thus a kind of self-damage. In Stephenson’s words, “the punishment lay in knowing that you were putting all that effort into letting a kind of intellectual poison infiltrate your brain”. Even if you’re not instinctively repulsed by inefficient organizations, working in one can take its toll. There are many, many things that a large tech company treats as important that are hard to take seriously: OKRs, some deadlines, internal training, attempts to build “hype” among employees, and so on. Acting emotionally invested in all of this can be alienating 2 . Getting good at persuading yourself to act like unimportant things are important is a lucrative but dangerous skill. If you do it long enough, it can feel like importance itself is an illusion. Before I became a software engineer, I did a master’s degree in philosophy 3 . When you tell people you’re studying philosophy, many people will ask if philosophy can solve their existential angst: they feel that the universe is very big and uncaring, and that humans make up only a tiny pin-prick of it, and that therefore nothing matters. Philosophers must have written about that, right? What’s the solution? The solution is to stop doing philosophy . If working in a dysfunctional organization has convinced you that nothing matters, you should refute that by coming into direct contact with things that matter. Go hang out with your kids, or take a walk in the park, or read a good book, or take your medication 4 . In other words, spend less time thinking about work 5 . I think this is good advice. But I also think there is quite a lot of real value and meaning you can get from working in a large tech company. Where is it? At large tech companies, millions of people will benefit from the work you do . If you make an amazing video conferencing app that a handful of people use, you will have to derive your satisfaction from the quality of your work. But if you can make Teams a tiny bit better, you’re making a tiny improvement in the lives of millions of people who use it. It isn’t easy to make enterprise pieces of software better, for lots of reasons . It sometimes requires jumping through awkward hoops or playing demeaning political games. However, it’s such a valuable thing to do. Every bug you fix or feature you add saves cumulative months or years of users’ time. This is still true if the software you’re working on is steadily getting worse, as most pieces of enterprise software do. Slowing that process down is important! Software engineering is really hard. Getting anything done in large organizations is also really hard. The intersection of these two hard problems is a nightmare morass of difficulty. It is not for the faint of heart 6 . Engineers from smaller organizations will mock you and invent fantasy stories about what your work is like. But if you can handle it — if you’re tolerant of mess, and if you have a rich enough personal life to be able to treat work politics with a healthy detachment — it is as rich a source of meaning as you could ask for. More or less the mission statement for this blog. I wrote about this in much more detail in Selling out . My specific subfield was “moral epistemology”: how we can know what’s right and wrong. If you think it’s a bit anachronistic for philosophers to believe in right or wrong at all, I suggest reading this Reddit post I wrote a long time ago: moral realism is surprisingly popular among academic philosophers. I mean this seriously. There are probably 1% of people who are so philosophy-brained that their existential angst does in fact derive from how seriously they take the arguments. These people might benefit from doing more philosophy. Everyone else is just depressed. At the risk of cultural stereotype, I think it is kind of distinctively American to try and derive your primary meaning in life from your workplace. On the other hand, you do spend a huge chunk of your life working, so it had better not be completely meaningless. I’m being hyperbolic here: in fact it is a relatively chill, well-paid job. More or less the mission statement for this blog. ↩ I wrote about this in much more detail in Selling out . ↩ My specific subfield was “moral epistemology”: how we can know what’s right and wrong. If you think it’s a bit anachronistic for philosophers to believe in right or wrong at all, I suggest reading this Reddit post I wrote a long time ago: moral realism is surprisingly popular among academic philosophers. ↩ I mean this seriously. There are probably 1% of people who are so philosophy-brained that their existential angst does in fact derive from how seriously they take the arguments. These people might benefit from doing more philosophy. Everyone else is just depressed. ↩ At the risk of cultural stereotype, I think it is kind of distinctively American to try and derive your primary meaning in life from your workplace. On the other hand, you do spend a huge chunk of your life working, so it had better not be completely meaningless. ↩ I’m being hyperbolic here: in fact it is a relatively chill, well-paid job. ↩

0 views
codedge 2 days ago

Pihole in HA mode

Pihole is the central point of resolving internal and external domains in my homelab. Running this on only one device is a classical SPOF so I run two separate instances on two physical devices in HA mode. In my homelab I decided to use two spare Raspberry Pis doing all DNS resolving. Even my Fritzbox uses the Pihole devices as custom DNS server. The problem is, that the Fritzbox can only take one IP as DNS server server. There is no builtin logic running Pihole in a typical high availability (HA) setup like one Pihole goes down and the other one kicks in. If your router takes more than one IP already, there is no need for the solution I describe here. Just put in the IPs of your two devices and you are done. So even having two devices running Pihole, Fritzbox would only use one for resolving names, which is a single point of failure (SPOF). The solution would be having a floating IP that jumps to the device, which can handle the current request. If both are up, there is a primary instance, and one or multiple secondary instances. Fortunately there is keepalived which brings load balancing and high availability to Linux. […] keepalived runs a set of health checkers that watch each server pool and decide, in real time, whether a real server stays in the topology or drops out according to its health. keepalived ships in most Linux distributions. I use Raspberry Pi OS Lite , the headless version without any desktop environment. My current Pihole setup is: On the Raspberry Pis I rolled out this configuration, with a different value, to define the primary and seconday instance. In the healthcheck script you define how often a check can fail or should succeed to trigger failed or healthy states. In the example above, a check each 2 seconds with means a failover after 4 seconds The only IP address that Fritzbox hands out to all my devices is which is a floating, load balanced IP what either one of my Raspberry Pis answers. I can update one Pi while the other one automatically serves all DNS requests. Pihole 1: x.x.x. 18 Pihole 2: x.x.x. 19 Floating IP: x.x.x. 53

0 views
Kev Quirk 2 days ago

2026 Self-Host User Survey Results

by Ethan Sholly Visualized results across 6,200+ responses to 2026's annual user survey. Read post ➡ Fascinating look at how people (like me) self-host their tech stack across the Internet. There were some surprises for me, like Debian being above Ubuntu, and Android being far more popular that iOS. Interesting reading if you like data. Thanks for reading this post via RSS. RSS is ace, and so are you. ❤️ You can reply to this post by email , or leave a comment .

0 views

Premium: The Hater’s Guide To Megaspeed

I’m not gonna lie — it’s been one of the most enjoyable weeks of writing in my entire life. As I discussed in this week’s free newsletter , Jensen Huang's tryst with an alleged spy running a front for the Chinese to buy GPUs is just the tip of the iceberg of an NVIDIA-backed smuggling ring, one that may, per Culper Research , account for more than 20% (over $43 billion) of NVIDIA’s Fiscal Year 2026 revenue. And now Jensen Huang is no longer CEO of NVIDIA , replaced by CFO Colette Kress. I also cannot express how many notes, connections, and sidequests got left on the cutting room floor as I wrote this week’s free piece piece, which may seem surprising given that it was over 7800 god damn words. Megaspeed — and its connected entities — represents a sprawling, multinational smuggling operation that goes far beyond NVIDIA, Jensen, Alice, or even China. And, more fundamentally, it’s just weird .  Up until as recently as last week , talking about NVIDIA’s illicit shipments of GPUs would have seen you branded a conspiracy theorist or a “doomer,” mostly because the allegations are so severe, but also because NVIDIA’s bull run was so impressive and rapid, the idea that any meaningful part of it was the result of anything underhand simply couldn’t register with the most dented cheerleaders for the AI bubble.  And yet, none of this should have been a surprise. Culper’s research — which I referenced heavily in the free piece — was published in May 2026, and it formed part of the basis for its short position. And it needs to be stressed how significant this is, and how much peril Culper would find itself in if it turned out to be wrong — and, therefore, how confident it was in its findings. If Culper was wrong, it didn’t open itself up to potential civil action (US defamation law sets a higher bar than in other countries, like the UK or Australia, but it still exists), but also, if you’re wrong when making a short, the losses are theoretically infinite .   The Wall Street Journal’s piece ( that ran on Wednesday ) fails to square the circle of how the largest media outlets in the world ignored the story entirely, yet the most damning part is that even in accepting that Culper might be onto something, the Journal fails to mention that 20% of NVIDIA’s Fiscal Year 2026 revenue might have come from China, nor does it make any attempts to discuss the contents of the report. In short, it treats Culper’s research — and the Jensen Huang infidelity — as a sort-of turbocharged tabloid scuttlebut, rather than looking at the bigger picture, which is that the (intermittently) biggest company in the world had lied about how and where it makes its money, and whether a significant chunk of its revenues came from illicit sources.  That’s because (like so much in the AI bubble) once you actually consider the contents — the scale of the smuggling, the massive amounts of “sanctioned” silicon that still makes it to China, and the amount of American counterparties (like Super Micro) involved — it becomes clear that NVIDIA has repeatedly and unashamedly profited from breaking the law.  Nobody wants to talk about the elephant in the room — that NVIDIA is ultimately a dodgy company that misleads its customers, investors and the general public, and I believe that the business and tech journalism ecosystem has helped it varnish its reputation again and again.  People are so quick to forget that NVIDIA was fined by the SEC less than four years ago for misleading investors in 2017 (Fiscal Year 2018, beginning in February 2017), when it claimed that its surge of revenue growth came from gamers buying graphics cards rather than cryptocurrency mining operators. Per the SEC: And just like with the AI bubble, some members of the media fell over themselves to sell the narrative to investors, unrepentantly repeating Jensen’s fictitious narrative about gaming driving its GPU sales , with GamesBeat printing one of the most damning quotes in history : It turned out this was a bag of bollocks. This isn’t the only time the SEC has slapped NVIDIA around. Back in 2002 , NVIDIA settled — albeit without paying a fine — with the SEC after it was revealed that its April 2000 quarter artificially inflated its gross profit and gross margins by showing a $3.3 million cost reduction from a supplier…but had also agreed to increase what it was paying to that supplier in August 2000…by $3.299 million: NVIDIA escaped any fine or real punishment, though its CFO was forced to pay a $672,000 settlement for accounting fraud . Then there’s the time in the middle of 2025 when NVIDIA started using multiple underhanded techniques to ( and I quote The Verge ) “delay, derail, and manipulate reviews of its $299 GeForce RTX 5060 graphics card,” deliberately launching it while most reviewers would be in Taiwan for Computex, refusing to provide drivers for the card until it went on sale, and only allowing “cherry-picked” reviewers access to drivers if they “agreed to a borderline unethical deal…where they’d test specific games at 1080p resolution with fixed graphics settings against two weaker GPUs (the 3060 and 2060 Super) where the new card would be sure to win.” They then made the mistake of pissing off GamersNexus , one of the best-known hardware reviewers in the industry:  I mention all of this because I need you to know exactly what kind of company NVIDIA is — a corporate bully that uses its outsized influence to manipulate investors and the media to pump up its sales.  And a few days ago, we found out that Mr. Huang is far more loose with his hips and his lips than we thought, with the New York Times reporting that he had an affair with NVIDIA healthcare VP Kimberly Powell… and the company attempted to cover it up: The next day, it came out that Jensen had another affair with another NVIDIA employee .  All of this is important context for today’s newsletter, where I’ll dig into the vast, ungainly smuggling network underpinning one of the biggest scandals in the history of the tech industry, and why it’s time to accept the truth: we can’t trust a damn word that Jensen Huang or NVIDIA says.

0 views
Stratechery 2 days ago

2026.46: The End of Jensen

Welcome back to This Week in Stratechery! As a reminder, each week, every Friday, we’re sending out this overview of content in the Stratechery bundle; highlighted links are free for everyone . Additionally, you have complete control over what we send to you. If you don’t want to receive This Week in Stratechery emails (there is no podcast), please uncheck the box in your delivery settings . On that note, here were a few of our favorites this week. Presented by Agentic commerce purchases with Link, Stripe’s digital wallet that works surprisingly well for agents, grew 38x in September. The catalyst was consumer agents like Muse and Instinct. Filling out a web form is not a value-add activity for anyone. No one chooses to fill out web forms in their spare time. That’s why these agents have been particularly useful.  I’ve written about how these platforms are giving ordinary people an AI agent with a computer, enabling agents to do genuinely useful things on our behalf.  When one of those things is spending money, it needs a wallet that knows what it is, and isn’t, allowed to do. Stripe has learned that’s really about deciding how much freedom to give it. You might approve one purchase, or give it $500 for flights this week, and let it figure out the rest. The wallet’s job is to keep it within those lines while handling everything between “go buy this” and the purchase actually happening. That’s what Stripe is building with Link: You approve what it spends, it can come back to you if the final price changes, and eligible purchases can even come with protection if the agent makes a mistake. You can use Link on Muse, Instinct, as well as many other of the agents you might use already. This week’s Stratechery video is on AI on the Ballot . The Shocking Demise of Jensen Huang. This week’s coverage was dominated, unsurprisingly, by the shocking and sudden end of Jensen Huang’s 32 year tenure as CEO of Nvidia. The end came on Thursday morning, and just a few hours later, we recorded a Sharp Tech episode recounting Huang’s legendary history, figuring out what happened, and looked forward to what this might mean for Nvidia, the tech industry, and America going forward. Also see Andrew’s view from DC, Jensen Huang Was Always Going to Lose , and listen to Wednesday’s episode of Sharp China about the geopolitical angle, and Thursday’s Dithering about both the parallels and differences between Steve Jobs and Apple. — Ben Thompson Enterprise, Consumer and Intel.  The more things change, the more they stay the same: Intel remains on the back foot in the AI era, and the foundry dreams continue to flounder. On the heels of disappointing earnings and 2027 guidance that promises yet more letdowns, Ben’s Update on Wednesday examined the state of America’s would-be national champion by drawing comparisons to AMD, an Intel peer that’s red-hot right now. To understand why one company is capitalizing on skyrocketing enterprise demand while the other confronts flatlining consumer sales, the answer starts with chips chiplets and leads to more profound questions about culture, Intel’s nature, and its bifurcated strategy for the long-term (and, for subscribers, look forward to more on this topic in Sunday’s Sharp Tech mailbag episode).  — Andrew Sharp And Then There’s Professional Basketball. As a palate cleanser after a week of Jensen drama, this week’s Greatest of All Talk was a look at the top and bottom of the NBA after the first three weeks. On the euphoric side of the ledger we have Warriors fans (pre-Steph injury), Lakers fans, Spurs fans, and on the other end spectrum there is nothing positive to say in Brooklyn, a familiar malaise in Cleveland, and precisely one silver lining in Salt Lake City. All of it was a great time; we also hit Adam’s experience with the NBA in Mexico City last weekend, the hardest working players in NBA history, and an anxiety-inducing conversation about the number one pick, AJ Dybantsa.  — AS Jensen Huang’s Affair, The Culper Report, Huang’s Righteousness — Nvidia CEO Jensen Huang admitted to an affair with a Chinese spy, and that might not be the end of his troubles. I hope he survives, but his influence will never be the same. The Jensen Huang Story Gets Worse, Washington vs. New York, Silicon Valley — The Jensen Huang story got worse for him on Monday; the real risk for Huang is governance, even more than national security. Who benefits? Intel Earnings, AMD Earnings, Intel and AMD’s Divergence — Intel and AMD’s diverging earnings point to fundamental historical differences, and make me wonder if Intel needs to find a new (old) path. An Interview With Roblox CEO David Baszucki About Aggregating Play — An interview with Roblox CEO David Baszucki about how AI is transforming game creation, and why that opens up huge opportunities to both enable developers and make Roblox both an Aggregator and a platform. Jensen Huang Was Always Going to Lose — Jensen Huang spent four years substituting his own judgment for that of the U.S. government. His downfall was inevitable, and probably isn’t over. The Jensen Huang Affair Jensen Fired How Cisco Got Bullwhipped Indonesia’s Mega Rice Failure Jensen Huang Likely Out at Nvidia; The Megaspeed Maelstrom in D.C.; APEC in Seven Days; Legislation That May Come Next Agony and Ecstasy After Three Weeks, The “Earn Their Ice” All-Stars, Difficult Conversations About AJ and Season Ticket Holder Decorum Subscriber Mail: Twitter, Microsoft, Bending Spoons, Movie Reviews Nvidia After Jensen Huang, What Huang Meant to the AI Ecosystem, Washington Gets Its Revenge

0 views

I moved 39 repos into a monorepo to boost agentic coding

Hey, what's up? It's Takuya here. I recently moved my projects into a monorepo to speed up my agentic coding. I came across a post by Mitchell Hashimoto . He recently started a new company, Superlogical, which builds Rex , a terminal app designed for working with AI agents. His team is small, and everyone on it is trusted to push commits straight to main. They skip the traditional flow of opening a pull request, waiting for someone to review and approve it, and then merging it. Instead, AI reviews the code before the push, and humans and AI review it again after it lands. Their workflow looks super fast. I'm a big fan of Mitchell, and I admire the quality of his work, like Ghostty. So I trust his technical decisions, including how his company uses AI. His post inspired me a lot and made me rethink my own workflow, starting from how my repositories are structured. Before I started coding with agents, each of my private modules lived in its own repository and was published to npm from there. Development was slower before AI, so I usually focused on one module at a time: make the changes, publish a new version, then update the apps. With semver, a change in one module couldn't unexpectedly break the apps. Many of these packages are shared across multiple projects, so this workflow was crucial for me. When I started using Claude Code, I kept the same structure, and at first it simply sped up my traditional workflow. But the more I relied on agents, the more friction I felt. AI makes a lot of changes, blazingly fast, so my workflow became the bottleneck. I had set up trusted publishing on npm, so GitHub Actions published new versions automatically. Still, after every change to a package, I had to wait for the publish run to finish before an app could use the new version. Agents also make many changes that span multiple modules, and keeping the module structure in shape became painful. Because each task was spread across several repositories, agents often missed shared constants, functions, and APIs. I think my job now is to help AI make fewer mistakes. One way to do that is to give the project a shape where AI can quickly understand the whole picture: where to look and where to add things. That's why I decided to move to a monorepo. The transition wasn't that simple. I have an Electron app, a React Native app, and a web app, each with its own tech stack and package manager (my server-side code stays in its own repos). Most packages used pnpm, but the React Native app used Yarn, so I had AI switch it to pnpm. Some of my repositories are public, as contributions to the open source community: for example, React Native modules and tools that help developers build Inkdrop plugins. Those can't move into a private monorepo. Everything else moved in, and the repository is now a single pnpm workspace. I used to bring each repository in with its full git history. For each repo, it moved every file into its new folder (for example, or ) across all past commits. It also renamed the tags to the format, like , since every repo had its own . And it rewrote bare issue references like in old commit messages to , so they still point to the right repository. Then I merged each rewritten repo into the monorepo with . For the plugin-related repositories, I made a second, public monorepo ( inkdropapp/plugins ). So now there are two monorepos, and the release flow looks like this: With everything in one repo, I could also unify the toolchain. Some modules still used ESLint and Prettier, so I migrated them all to Oxlint and Oxfmt, which now run once from the root. I also tried Vite+ , a unified toolchain that bundles Vite, Vitest, Oxlint and Oxfmt behind a single command. I was already using most of these tools in my modules, so switching looked easy, and its task caching promised a faster loop with agents. But in my case, its type-check cache reported a broken TypeScript 7 project as passing. With agents, that's risky: they see a green check and move on. There also wasn't much build time to save in the first place, so I decided not to go with it. I'll try it again in the future. This work took two days to finish, and I'm happy with the result. Now, my agent and I can quickly update modules and keep them aligned in a single change, and I can comfortably create a git worktree for each task. Before the move, a task that touched several repositories needed a worktree in each of them, and that was painful. So, this transition has already paid off. Inkdrop's codebase is over 10 years old. If you also have an old project split across many repositories, I hope this helps you rethink how you work with agents. I have so many ideas to improve my app, and I'm so excited to work on them starting next week. That's it! Thank you for reading.

0 views
Anton Zhiyanov 2 days ago

Probabilistic data structures in Go

Probabilistic data structures give you approximate answers to questions about the data they handle. Does the set contain a certain item? How many unique items are in the collection? Which items appear most often? Unlike deterministic structures like sets and frequency tables, the answers might not be exact, but in return, you can drastically reduce memory usage. In this post, we'll talk about what probabilistic structures are, how they work, and what kinds of guarantees they provide. I'll also introduce unsure — a package that implements these structures in Go. It's simple (the core logic for each data structure is 100–300 LOC), fast (usually several times faster than other implementations), and has no dependencies except for the standard library. Bloom filter • Count-min sketch • HyperLogLog • T-digest • Top-k A Bloom filter is a set that trades certainty for space. It can tell you that an item is definitely not in the set, or that it probably is (which might be a false positive). The filter doesn't actually store the items, so it's much more compact than a regular set. For example, a regular set with 1,000,000 distinct 8-byte strings takes about 64 MB. A Bloom filter for the same strings, with a 1% false positive rate, uses only 1.24 MB (about 10 bits per item) — that's 50 times smaller. The size of the filter doesn't depend on the item size, so the difference gets even bigger with longer strings. A Bloom filter often resides in memory and helps minimize expensive I/O operations such as disk reads or network calls. You check if an item is in the filter, and if the answer is "no", you skip the costly lookup. If the answer is "yes", you still have to perform the lookup because it might be a false positive. The filter is sized for the given number of items with a given false positive rate. You can add more, but the false positive rate will go up. You can add an item to a Bloom filter and check if an item is in the filter, but you cannot delete an item, count the number of items, or iterate over them. Bloom filter 1 is a large array of m bits, all set to zero at first. To add an item, you compute k hash values for it and set the k bits at those positions: To check if an item is in the filter, you compute the same k hashes and check if all k bits are set: The k bits are randomly distributed throughout the entire array. When the filter is larger than the CPU cache, each bit access could result in a cache miss, so a lookup may wait on memory k times. Unsure uses a blocked Bloom filter 2 . The bit array is divided into 512-bit blocks, each matching the size of a CPU cache line. When you add or check an item, it is hashed once: part of the hash selects a block, and the rest selects k bits within that block. Each add or check only accesses one cache line, so the filter stays fast even when it's very large. In a blocked Bloom filter, some blocks end up with more items than others, which increases the false positive rate. To make up for this, the filter needs to be a bit larger. Compared to a standard Bloom filter, it uses about 3% more memory at a 1% false positive rate, 8% more at 0.1%, and 15% more at 0.01%. The filter uses the local wyhash 3 implementation for hashing. It performs roughly the same as xxh3 4 but has a much simpler implementation, without SIMD or assembly. Here's an excerpt from the full benchmarks 5 , comparing unsure to bits-and-blooms 6 (a classic Bloom filter) and gloom 7 (a blocked one) (ns/op, lower is better): Memory, bits per item at 10M items (lower is better): Unsure is about 4× faster than bits-and-blooms and 2× faster than gloom , using 1–8% more memory than bits-and-blooms and 4–10% less than gloom . A count-min sketch is a frequency table that trades accuracy for space. It tells you approximately how many times each item has appeared. The estimate is never less than the actual count, but it can be higher . The sketch doesn't actually store the items, so it's much more compact than a regular map of per-item counts. For example, a regular map counting 1,000,000 distinct 8-byte strings takes about 64 MB. A count-min sketch for the same strings, with estimates within 0.01% of the total count (with 99% probability), uses only 0.55 MB — that's 116 times smaller. The size of the sketch doesn't depend on the number of distinct items or their size, only on the accuracy you choose. So the difference gets even bigger with more or longer strings. A count-min sketch is useful when you want to track how many times an item shows up in a large stream of events. It's often used to check if something has happened a certain number of times, like rate-limiting an IP after X requests or sending an alert after an error happens Y times. A sketch can also help decide if a key is worth keeping in the cache given how often it was requested. The sketch uses two settings: epsilon ε and delta δ . If the actual count for any given item is X and the total of all counts is N, then the estimated count for this item will usually be between X and X+ε·N. However, rarely (with a probability of δ) the estimate could be even higher. For example, if ε is 0.001, δ is 0.01, and N is 1 million, then the estimate could be off by up to 1,000 (ε·N) with at least 99% probability (1−δ), and by more than 1,000 with at most 1% probability (δ). The counting error in a sketch depends on the total of all counts, not on each individual item. Therefore, frequent items are estimated more or less accurately relative to their count, but rare items are not — the absolute error is too high for them. You can increase the per-item counter in a sketch and get the estimated count for an item, but you cannot decrease a counter or iterate over the items. Count-min sketch 8 is a table of counters with d rows and w columns, all starting at zero. Each row uses its own hash function that maps an item to one of the w counters. To add an item, you increment its counter in each row: To estimate an item's count, you take the smallest value from its d counters: With w = e/ε columns, each counter's error is at most ε·N, with probability at least 1 − 1/e. If you use d = ln(1/δ) rows, the chance that all d of the item's counters exceed its true count by more than ε·N is ≤ δ. Unsure uses a conservative update . When you add n to an item, it first finds the current estimate E (the smallest counter), then increases only the counters that are below E+n, leaving the larger ones unchanged. This keeps the same accuracy guarantee, but the actual error is about 2× lower. The tradeoff is that counts can't be decreased. Counters are 32-bit instead of the usual 64-bit, so the sketch uses 2× less memory. The item is hashed once, and each row gets its own position from that hash by salting it (multiplying by a random per-row constant). The sketch uses wyhash for hashing. Here's an excerpt from the full benchmarks 9 , comparing unsure to BoomFilters 10 (a classic count-min sketch with 64-bit counters) (ns/op, lower is better): Accuracy at ε=0.0001, N=10M additions over 1M distinct items, mean overestimate (lower is better): Unsure is about 1.5× faster for Add and 4× faster for Count, using 2× less memory with 2–2.7× lower error. HyperLogLog is a unique items counter that trades accuracy for space. It tells you approximately how many distinct items it has seen, no matter how many times each one appeared. HyperLogLog (also called a sketch from here on) doesn't actually store the items, so it's much more compact than a regular set. For example, a regular set of 1,000,000 distinct 8-byte strings takes about 64 MB. A HyperLogLog sketch counting the same strings with about 0.8% error uses only 16 KB — that's almost 4,000 times smaller. The size of the sketch doesn't depend on the number of items or their size, only on the accuracy you choose. A 16 KB sketch counts a billion items as accurately as a thousand. HyperLogLog is useful when you need to count unique things in a large stream: unique visitors to a website, unique search queries per day, or distinct values in a database column. You can merge sketches to gather data from multiple machines or segments (like by hour, page, or region). When you merge two sketches, you get the total number of unique items from both, without counting shared items twice. The sketch has one setting: the precision p , which gives a relative standard error (stdErr) of 1.04/√(2^p). A common default is p=14, which gives 0.81% error. With p=14 and 1 million distinct items: The error is relative to the count, so the standard error is the same 0.81% for a thousand items or a billion. You can add items to a sketch, get the estimated count, and merge sketches. You cannot check if a specific item was added, remove items, or list them. HyperLogLog 11 is based on a simple observation. If you hash items to random bit strings, about half of them start with 1, a quarter with 01, an eighth with 001, and so on. Seeing a hash with k leading zeros suggests there were about 2^(k+1) distinct items. Duplicates hash to the same value, so they don't affect the result. A single observation is unreliable, so the sketch divides the items into m = 2^p groups, where p is the selected precision. The first p bits of the hash pick a group (a register ), and the register keeps the highest rank (leading zeros plus one) it has seen in the rest of the hash: The count is estimated from all the registers together, using a harmonic mean of 2^register. The standard error is about 1.04/√m. Merging two sketches is just taking the maximum value from each pair of registers. The original estimator is biased when dealing with small counts (up to a few times m), so in these cases it uses a different formula called linear counting . This switch creates a sudden change in error, and some implementations (like HyperLogLog++ 12 ) address it by using large tables of bias corrections based on real data. Unsure uses the improved estimator by Otmar Ertl 13 . Instead of a harmonic mean of the registers, it works from a histogram : how many registers have value 0, value 1, and so on. A formula applied to the histogram gives an accurate estimate for any number of items, from zero to billions, so there's no need for a separate formula for small counts or for correction tables. The sketch updates the histogram whenever registers change. Most adds don't change any registers, so this is almost free. Counting always takes the same amount of time, no matter how big the sketch is. Other implementations scan all the registers on every count, which takes from microseconds to milliseconds. Registers are one byte each, and the histogram has 64 four-byte counters (256 bytes). The item is hashed once with wyhash , and the 64-bit hash is split into the register index and the rank. Here's an excerpt from the full benchmarks 14 , comparing unsure to axiom 15 (HyperLogLog with LogLog-Beta estimator and sparse mode) (ns/op, lower is better): Accuracy at p=14 (0.81% standard error), relative RMSE over 100 sketches (lower is better): Memory at p=14 after adding the given number of distinct items (lower is better): Unsure adds about as fast as axiom , counts 270–83,000× faster, and merges 2.3–2.9× faster, with the same accuracy and memory for counts above the number of registers. When there are fewer items than registers, axiom 's sparse mode makes it more accurate and compact. A t-digest is a summary of a stream of numbers that trades accuracy for space. It can give you an approximate median, 99th percentile, or any other quantile of the data. It can also estimate what fraction of the numbers are below a certain value (the CDF). The digest doesn't actually store the numbers, so it's much more compact than a sorted list. For example, to find the exact 99th percentile of 1,000,000 numbers, you'd have to keep all of them. With 64-bit floats, that takes 8 MB. A t-digest estimates the same percentile within about 0.03% of the rank (between the 98.97th and the 99.03rd percentile) using only 8 KB — 1,000 times smaller. The size of the digest doesn't depend on the number of values, only on the accuracy you choose. An 8 KB digest summarizes a billion values as well as a million. A t-digest is useful when you need quantiles over a large stream: latency percentiles for a service, the distribution of request sizes or prices, or the outlier thresholds in sensor data. You can merge digests to gather data from multiple machines or segments (like by hour, endpoint, or region). A merged digest estimates the percentiles of all values, but with less accuracy than individual digests do. The digest has one setting: the compression δ , which limits the number of centroids (see How it works below) to δ+2. A common default is δ=100, which takes 8 KB. Increasing δ uses more memory but reduces estimation error. The error is measured in ranks: a rank error of 0.03% at quantile q=0.99 means the value returned by is somewhere between the 98.97th and the 99.03rd percentile of the actual values. Tail estimates, such as q=0.99 or q=0.999, are more accurate than the median (q=0.5). You can add numbers to a digest, estimate quantiles and the CDF, and merge digests. You cannot remove numbers or list them. T-digest 16 summarizes the numbers as a sorted list of centroids . Each centroid has a mean and a weight — the number of values it represents. To estimate a quantile, the digest finds the centroids around the requested rank and interpolates between their means. The trick is in how big the centroids are allowed to grow. A scale function k(q) maps a quantile q to a number from -δ/4 to δ/4, and each centroid may only span 1 unit of k: if a centroid covers the quantiles from q₁ to q₂, then k(q₂) − k(q₁) ≤ 1. The arcsine scale function, k(q) = δ/(2π)·asin(2q-1), is steep near q=0 and q=1 and flat near the median. This means the centroids it produces are larger in the center and smaller at the ends: With 1M values and δ=100, the centroids around the median hold about 27,000 values each, those around the 99th percentile about 5,000, and the outermost ones about 500. This keeps the error near the tails small. Unsure is the merging t-digest from the Dunning and Ertl paper. New values go to a buffer of 5·δ numbers. When the buffer is full, the digest sorts it and merges it with the centroids (already sorted) in a single pass, building new centroids from left to right as long as each one fits within its span of k. Every other pass goes right to left, so that neither tail ends up with fuller centroids. , and flush the buffer first. The centroids are kept in two fixed arrays of δ+2 entries (the current list and the next one being built), along with the cumulative weights for a binary search in and . With δ=100, that's 16·102·2 + 8·103 bytes for the centroids, and 8·500 bytes for the buffer, about 8 KB in total. Centroids with weight 1 are treated as exact values rather than spread around their mean, and the exact minimum and maximum are used at the ends. So digests with few values give exact results, and and are always exact. Here's an excerpt from the full benchmarks 17 , comparing unsure to influxdata 18 (merging t-digest with the same scale function) (ns/op, lower is better): Accuracy at δ=100, mean rank error over 1M normal values (lower is better): Memory at δ=100 (lower is better): unsure 8.5 KB, influxdata 14.4 KB. Unsure adds values about 1.8× faster than influxdata , queries at the same speed, merges 7–9× faster, and uses 40% less memory at about the same accuracy. A Top-k sketch finds the k most frequent items in a stream (the "heavy hitters"), along with their approximate counts. It uses a fixed amount of memory based only on k, no matter how many different items are in the stream. To find the exact top items, you'd have to count every unique item, for example with a map of per-item counts. With millions of distinct items, that takes tens or hundreds of megabytes. A Top-k sketch for the top 1000 items takes about 360 KB, and it usually finds all of them. A Top-k sketch is useful when you want to identify the "hottest" data: the most requested URLs, the most active users, or the most common search queries. Unlike a count-min sketch, it keeps the actual items, so it can list them. returns up to k items, from the most frequent to the least frequent. Counts are estimates: they're usually exact for the most frequent items, but may be slightly lower than the actual count for less common items. The sketch works best when the stream is skewed, meaning a few items appear much more often than the others. This is common in real situations. If all items are about equally frequent, there is no clear top k, and the results become less accurate. You can add items to the sketch and get the top items with their counts, but you can't decrease counts or check items that aren't in the top k. HeavyKeeper 19 is a table of buckets that estimates item counts. It has d rows and w columns, and each item maps to one bucket in each row. Each bucket holds a fingerprint of the item and a count. To add an item, you update its bucket in each row: The item's estimated count is the largest count among its buckets. Frequent items quickly reach counts that make decay very unlikely, so they keep their buckets. Rare items keep replacing each other without affecting the frequent ones. An item that becomes frequent later in the stream can still outnumber a less frequent owner and take over its bucket. Next to the table, the sketch keeps a min-heap with the k items that have the largest estimates so far. After an item's buckets are updated, if its estimate is larger than the smallest count in the heap, the item is added to the heap, and the smallest item is removed. If the item is already in the heap, its count is increased. Unsure uses d = 2 rows of max(16k, 256) buckets, and a decay base of b = 0.8. The probabilities b^count are precomputed as 64-bit thresholds, so decay is cheap: the count c decreases if a random number is below the c-th threshold. The original HeavyKeeper uses b = 1.08⁻¹ ≈ 0.93, and more rows. A smaller b lets frequent items lock their buckets faster, which makes the sketch more precise on flat streams. However, if b is too small (like 0.5), the first items to claim buckets keep them permanently, so items that become frequent later are never counted, even if they end up being the most frequent. Using b = 0.8 keeps most of the precision and still catches most late-arriving frequent items. As for the rows, two are enough to protect frequent items from collisions in a single row, and adding more rows only makes the sketch slower. The item is hashed once. The hash is mapped to a position in each row by salting it (multiplying by a random per-row constant). The fingerprint is the top 32 bits of the hash. The heap uses the 64-bit hash to identify items, so it doesn't compare the actual item bytes. Each bucket is 8 bytes (a 32-bit fingerprint and a 32-bit count), so the table takes 256·k bytes. The heap takes about 100 bytes per item plus the item itself. The sketch uses wyhash for hashing and wyrand for random numbers. Here's an excerpt from the full benchmarks 20 , comparing unsure to keiler 21 (HeavyKeeper with b = 0.9 and d = ln k) and dgryski 22 (Filtered Space-Saving 23 ) on the Zipf s=0.8 stream (time/op, lower is better): Precision is the fraction of the reported top k that is in the true top k (higher is better): Count error is the mean of |estimate − true| / true over the reported items (lower is better): Unsure is 2–4× faster than the others for Add and 1.8–4× faster for Top. On flat streams (s=0.6), it finds the true top k with 90–97% precision, while keiler drops to 38–70% precision with a count error of 62–94%, and dgryski to 19–52% with counts 18× or more too high. On skewed streams (s=1.0), all three are about equally accurate. As engineers, we usually want exact answers from our data. But sometimes, allowing a bit of uncertainty can cut memory usage by tens, hundreds, or even thousands of times. That's where probabilistic data structures come in: To use them in Go, try the nalgeon/unsure package: Space/time trade-offs in hash coding with allowable errors by Burton H. Bloom, 1970.  ↩︎ Cache-, hash-, and space-efficient Bloom filters by Felix Putze et al., 2009.  ↩︎ Reference wyhash implementation by Wang Yi.  ↩︎ Reference xxHash implementation by Yann Collet.  ↩︎ Bloom filter benchmarks .  ↩︎ bits-and-blooms/bloom by Daniel Lemire.  ↩︎ jcalabro/gloom by Jim Calabro.  ↩︎ An improved data stream summary: the count-min sketch and its applications by Graham Cormode and S. Muthukrishnan, 2005.  ↩︎ Count-min sketch benchmarks .  ↩︎ tylertreat/BoomFilters by Tyler Treat.  ↩︎ HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm by Philippe Flajolet et al., 2007.  ↩︎ HyperLogLog in practice by Stefan Heuleet et al., 2013.  ↩︎ New cardinality estimation algorithms for HyperLogLog sketches by Otmar Ertl, 2017.  ↩︎ HyperLogLog benchmarks .  ↩︎ axiomhq/hyperloglog by Axiom.  ↩︎ Computing extremely accurate quantiles using t-digests by Ted Dunning and Otmar Ertl, 2019.  ↩︎ T-digest benchmarks .  ↩︎ influxdata/tdigest by InfluxData.  ↩︎ HeavyKeeper: An Accurate Algorithm for Finding Top-k Elephant Flows by Junzhi Gong et al., 2018.  ↩︎ Top-k benchmarks .  ↩︎ keilerkonzept/topk by KEILERKONZEPT.  ↩︎ dgryski/go-topk by Damian Gryski.  ↩︎ Efficient Computation of Frequent and Top-k Elements in Data Streams by Ahmed Metwally et al., 2005.  ↩︎ If any of the bits is not set, the item definitely was never added. If all the bits are set, the item was probably added, but it's not guaranteed (the bits could have been set by other items). Other items might hash to the same counter, so each counter value is ≥ true count. The smallest counter is the one with the fewest collisions, so it's the best estimate. 68% of the time the estimate is within 0.81% (8,100) of the true count, 95% within 1.6% (2·stdErr), 99.7% within 2.4% (3·stdErr). If the bucket is empty or owned by the item, increase its count. If the bucket is owned by another item, decay it: decrease its count by 1 with probability b^count (b < 1). When the count reaches 0, the new item takes over the bucket with a count of 1. Bloom filter checks if an item might be in a set. Count-min sketch estimates how many times each item has occurred. HyperLogLog estimates the number of distinct items. T-digest estimates quantiles of a stream of numbers. Top-k sketch identifies the most frequent items in a stream. Space/time trade-offs in hash coding with allowable errors by Burton H. Bloom, 1970.  ↩︎ Cache-, hash-, and space-efficient Bloom filters by Felix Putze et al., 2009.  ↩︎ Reference wyhash implementation by Wang Yi.  ↩︎ Reference xxHash implementation by Yann Collet.  ↩︎ Bloom filter benchmarks .  ↩︎ bits-and-blooms/bloom by Daniel Lemire.  ↩︎ jcalabro/gloom by Jim Calabro.  ↩︎ An improved data stream summary: the count-min sketch and its applications by Graham Cormode and S. Muthukrishnan, 2005.  ↩︎ Count-min sketch benchmarks .  ↩︎ tylertreat/BoomFilters by Tyler Treat.  ↩︎ HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm by Philippe Flajolet et al., 2007.  ↩︎ HyperLogLog in practice by Stefan Heuleet et al., 2013.  ↩︎ New cardinality estimation algorithms for HyperLogLog sketches by Otmar Ertl, 2017.  ↩︎ HyperLogLog benchmarks .  ↩︎ axiomhq/hyperloglog by Axiom.  ↩︎ Computing extremely accurate quantiles using t-digests by Ted Dunning and Otmar Ertl, 2019.  ↩︎ T-digest benchmarks .  ↩︎ influxdata/tdigest by InfluxData.  ↩︎ HeavyKeeper: An Accurate Algorithm for Finding Top-k Elephant Flows by Junzhi Gong et al., 2018.  ↩︎ Top-k benchmarks .  ↩︎ keilerkonzept/topk by KEILERKONZEPT.  ↩︎ dgryski/go-topk by Damian Gryski.  ↩︎ Efficient Computation of Frequent and Top-k Elements in Data Streams by Ahmed Metwally et al., 2005.  ↩︎

0 views