daniel.haxx.se 2 days ago

Workshop Basel day one

On this hot summer’s day in Basel, Switzerland, the seventh HTTP workshop started. These events tend to work roughly the same way and the people in the room are also to large extent familiar and known since previous editions. Forty people in a meeting room, where we take turns in doing short talks on HTTP and networking topics, with the following question and discussion session. The rules for the meetings are explicitly Chatham rules, which means that everything I write about the meeting will be sufficiently fuzzy and without many company or personal names. This is not the kind of meeting that can be easily summed up in a short blog post anyway. You really should be here. Present in the room were representatives from all the world’s most prominent and used HTTP deployments: clients, browsers, CDNs, proxies and servers. I’m happy to say that there were also several first-timers. We like fresh blood. (If you think I’m being overly brief or vague about specifics in this post; that is partially on purpose but primarily because I’m a lousy note-taker and mostly write this up after a busy day that also may have involved beer.) After a round of introductions, we started. REST is a set of constraints, and in this presentation it was argued that it can or maybe even should be extended to do more. A number of recent applications like Mastodon/ActivityPub, Bluesky/AT, Matrix, Nostr, IndieWeb, all currently use HTTP to do state synchronization but they all do it differently in their own unique ways. Can REST and maybe HTTP be adjusted to help this for improved interoperability? Looking at the Common Crawl data and comparing data over time, it was observed that responses use the Last-Modified header field more now than they did in the past, and there were great follow-up speculations on why this is so. Data also shows that a large share of these headers present dates that are almost identical to the time the requests were issued. With the cc-lint tool , data was gathered on how HTTP is actually used today, proving that there is work to be done: deprecated headers are used, some headers are done wrong, and many are overly big. This indicates that there are well used both servers and clients out there that would benefit from cleanup. It probably also shows that doing HTTP correctly and all the correct headers is far from an easy task. Another presentation showed data, this time from a well-known CDN, on the impact the existing AI scraper bots have on the Internet from their point of view. It showed that roughly half of the requests and half of the bandwidth are spent by scraper bots. A long discussion followed where the numbers were questioned as maybe the numbers look like this because a sufficiently large number of the “bad AI scrapers” appear as regular users to the classifiers. Speculations of different kinds were made.  As a follow-up from a presentation from a previous HTTP workshop we got to learn how the journey on developing their new HTTP stack has progressed and several fun adventures and lessons from that were shared with the audience. A look into new HTTP API development at Apple . Some discussions and lessons learned from creating new APIs for both servers and clients. We got an excellent walk-through of some details and internals of the Android networking stack. Emphasis was perhaps especially put on ECH and QUIC connection migration, and the final “don’t tell us when your connection closed” led to a long new discussion on how we really should fix the problem: when connection has been left idle for a long time and it is closed by the server, the client (mobile phones) don’t want to be told. This, because getting that RST and more, just wakes up the radio and more on the phone only to tell it to go back to sleep. It was theorized that if we could get rid of this unnecessary battery waste, the accumulated gain across billions of devices would make a serious dent. Several additional HTTP related problems were of course also subsequently solved as we then wandered into the city for dinner and maybe a beer. Of course yours truly returned back to his hotel room in good time to be able to write up this blog post. The best part of these workshops might be the (no pun intended) networking and discussions had completely outside of the agenda. End of day one. Two more to come,

Web Development Data Analysis API

Dangling Pointers 2 weeks ago

F3: The Open-Source Data File Format for the Future

F3: The Open-Source Data File Format for the Future Xinyu Zeng, Ruijun Meng, Martin Prammer, Wes McKinney, Jignesh M. Patel, Andrew Pavlo, and Huanchen Zhang SIGMOD'26 F3 is a file format for columnar data (e.g., Parquet ) that is designed to be efficient and extensible. The optimizations make sense, the extensibility mechanism is ingenious , dangerous , fascinating. The key assumption made by this paper is that the hardware and software will continue to improve. It is hard to argue with that. The trouble is that interoperable formats like Parquet take a snapshot of the state-of-the-art and freeze it in a specification. Some innovations that are invented after the format is frozen are incompatible with existing formats because they require a different data layout. Section 1 of the paper refers to many examples related to compression, indexing, and filtering. The goal of F3 is to be general enough to allow seamless incorporation of future innovations without changing the F3 spec nor F3 decoder implementations. Fig. 2 illustrates an F3 file: Source: https://dl.acm.org/doi/10.1145/3749163 A file consists of a metadata and a set of row groups. A specific row group contains data for all columns and a subset of rows. F3 contains incremental improvements over existing columnar formats, for example: F3 metadata supports random access, which is important for operations that only need to access a smaller percentage of all columns. F3 decouples file I/O from a row group storage. The rows associated with a given column in a row group are further subdivided into , which are actually stored. This allows row groups to be sized for efficient row-group level filtering, while the IO unit size is tuned to minimize working set while also amortizing the fixed costs associated with file I/O. F3 allows flexible . Each IO unit can contain a dedicated dictionary, or multiple IO units can share a dictionary. Columns with low cardinality will benefit from smaller dictionary scopes, whereas columns with high cardinality do better with larger dictionary scopes. The stand-out feature for F3 is the yellow block in the block. The idea is that an F3 file can contain within it the WebAssembly code needed to decode the encoded values in an IO unit. If someone invents a brilliant new encoding method that works well with some data sets, they can ship the decoder right along with the data set. Storage of the WASM code shouldn’t be too much of an issue, because the storage cost is amortized across all row groups. The big questions are performance and security. Section 6.2 has some comments on this. In theory, the WASM specification is air-tight, and a bug-free implementation should be able to securely run arbitrary WASM code in-process. WASM also supports performance optimizations like parallel compilation and SIMD instructions. Something I don’t see in the paper is a discussion about how filtering interacts with WASM decoding. I suppose the extensibility could only be used for decoding, and filtering could be hard coded into F3, but that seems against the extensible spirit of F3. Fig. 11 shows the working set reduction from decoupling IOUnit size from row group size: Source: https://dl.acm.org/doi/10.1145/3749163 Table 3 shows how flexible dictionary scopes allow one to trade encoding time for compression ratio (lower relative CR numbers mean smaller files on disk): Source: https://dl.acm.org/doi/10.1145/3749163 Fig. 15 quantifies WASM overhead by comparing decode time for hard coded F3 decoder implementations vs the same algorithms expressed in WASM: Source: https://dl.acm.org/doi/10.1145/3749163 Fig. 16 shows potential savings associated with using WASM extensibility to implement a custom decoder from the literature. Source: https://dl.acm.org/doi/10.1145/3749163 Dangling Pointers I wonder how well WASM decoders can be implemented on other hardware architectures. Is WASM the ideal language for expressing this, or convenient standard that already exists? Thanks for reading Dangling Pointers! Subscribe for free to receive new posts. Source: https://dl.acm.org/doi/10.1145/3749163 A file consists of a metadata and a set of row groups. A specific row group contains data for all columns and a subset of rows. F3 contains incremental improvements over existing columnar formats, for example: F3 metadata supports random access, which is important for operations that only need to access a smaller percentage of all columns. F3 decouples file I/O from a row group storage. The rows associated with a given column in a row group are further subdivided into , which are actually stored. This allows row groups to be sized for efficient row-group level filtering, while the IO unit size is tuned to minimize working set while also amortizing the fixed costs associated with file I/O. F3 allows flexible . Each IO unit can contain a dedicated dictionary, or multiple IO units can share a dictionary. Columns with low cardinality will benefit from smaller dictionary scopes, whereas columns with high cardinality do better with larger dictionary scopes.

Data Analysis Open Source Database

Ankur Sethi 2 weeks ago

Your analytics are lying to you

Alistair Davidson writes about migrating a form-heavy web application from a React SPA to a traditional server-rendered HTML-first website . The entire article is worth reading, but I want to draw attention to this bit about analytics (emphasis mine): The results? When we launched, the number of people completing the form doubled. The analytics people didn’t even know where these users were coming from. Of course, your javascript-based analytics package doesn’t see the users you are bouncing because of javascript failures. It was a flood! We also saw my “keep a backend session, never lose user data” approach pay off. In one case, someone completed a form a month after starting it. Web analytics are fragile. They fail in so many ways that making product decisions based wholly on your Google Analytics or Plausible data is folly of the highest degree. Here's a subset of all the reasons your analytics package undercounts or miscounts visitors: Web analytics can only give you an approximation of what your web traffic looks like. Even when they work correctly, they paint an incomplete picture. As I said in my post about share buttons , the number one referrer for pages on this website is "Direct/none". It's impossible for Plausible to figure out where those users are coming from. Further, my server logs report three times as much traffic as my Plausible dashboard over a seven day window. Some of this might be bot traffic and thus irrelevant, but I know for a fact that a large chunk of this traffic comes from RSS readers. Plausible will never have insight into these users. My point is, if you rely on your analytics dashboard to make product decisions, you're excluding a large chunk of potential users who simply don't show up in your graphs. You might be missing out on serving thousands of potential users because you can't see them in your data. These are users who want to sign up for your newsletter, buy your app, subscribe to your service. These are human beings you could help, whose lives you could improve. I'm not saying that analytics are completely useless. They can and should have a place in your decision-making process. Just don't treat analytics data as gospel, because there will always be massive blind spots in what it tells you. To get a real understanding of how users experience your products, test them on real devices under real conditions as much as possible. And as always, get out there and talk to your users. Network errors prevent your analytics script from loading. Ad-blockers and tracking prevention block your script from loading (enabled by default on many browsers today). A JavaScript error in an unrelated part of the page prevents the analytics script from working correctly. The user loses network connectivity before the analytics script can send data to the server. The user gets impatient and bounces off your website before the page can load fully and start collecting data. Too much JavaScript on the page causes the browser tab to crash (a common issue on low-end devices). The analytics script is blocked by a DNS rule, corporate proxy, firewall, or VPN. The user has disabled JavaScript. The user's browser has limited or no support for JavaScript (Opera Mini still has more than half a million downloads on Android, and it's still widely-used in Africa ). The user is accessing your content using a service that strips JavaScript (e.g. an RSS reader, a web archiving tool, Telegram Instant View, AMP, a read-later service, or a bookmarking service). You only test your app in Chrome, so you don't realize that your website is entirely broken in Firefox and Safari.

Data Analysis HTML Web Development Analytics JavaScript

Gabriel Weinberg 1 months ago

No, everyone is not using AI for everything.

Last year around this time The New York Times Magazine ran an A.I. issue with an introduction titled “ Everyone Is Using A.I. for Everything. Is That Bad? ” It’s an edited transcript from the Hard Fork podcast , which I think assumes two things are true that are turning out to be false. Once you’ve tried AI, you use it “for everything.” No, in fact most people who’ve tried it are just occasional AI users. AI has gotten so good that despite any misgivings, “everyone is using A.I.” No, in fact large chunks of the population aren’t using AI at all. (It isn’t really strictly defined in the article, but I’m taking AI to mean generative AI accessible via a chat interface.) Take Gen Z, where AI awareness is the highest : in the last year, even though AI has supposedly gotten a lot better, Gen Z AI adoption has all but stalled, with a meaningful percentage of the Gen Z population still using AI rarely, if at all. Here’s Gallup’s year-over-year (2025/2026) breakdown: 79/81% use AI at least rarely 41/42% are anxious about AI 32/31% use AI only monthly/every few months 22/31% are angry about AI 21/19% never use AI This tracks with Microsoft’s new United States AI Diffusion site , based on “anonymized, aggregated Microsoft telemetry.” Their associated blog reports “more than 30 percent of the US working-age population is using AI [meaning about 70% isn’t], an increase of 3 percentage points from the end of 2025 . ” The underlying academic paper specifies that usage is defined as “engagement with major AI services including ChatGPT, Google Gemini, Anthropic Claude, Microsoft Copilot, and others….with at least 90 minutes of usage time in a given month.” The Microsoft data is brand new, and it mirrors another usage study from Datos from last year, also based on real-world usage data. The Datos study found similarly that, as of last June, only 21% of desktop devices visited “AI Tools” 10 or more times a month, 62% visited 0 times, and the remaining 17% in between . Back on the survey side, a recent Searchlight Institute study found “ 58% report using or trying AI, specifically tools like ChatGPT or Claude, divided evenly between fairly regular users (30% use at least a few times a month) [roughly matching the Microsoft/Datos data] and more infrequent users (29% have used AI, but only once a month or less). ” And finally a new survey from The Argument finds “ most Americans use AI once a week or less. ” All of this triangulates to AI use in America at approximately one third actively using AI, one third occasionally using AI, and one third never using AI , with some movement depending on how you define those terms. In any case, this split is a far cry from “everyone is using AI for everything;” it’s much closer to “some people are using AI for some things.” AI use also hasn’t shifted that much in the past six months to a year. In fact, the only thing that has substantially changed is negative sentiment about AI has gone significantly up, for example the Gallup’s Gen Z poll reporting anger about AI jumping about 40% relative year over year. I think it is a reasonable conclusion to draw from all of this data that a significant percentage of the population is actively limiting their AI usage. The Searchlight study examines a big reason why: real concerns people have with AI. The top three concerns found are “AI will replace jobs and cause unemployment” (42%), "AI will violate people’s privacy” (35%), and “AI will spread misinformation and lies” (33%). This sentiment also matches a strong desire for safety/privacy AI regulation. A solid majority thinks “the government should prioritize creating safety/privacy rules for AI, even if that means the U.S. develops AI more slowly than countries like China.” Another big reason is skepticism in AI usefulness. SearchLight asked about a range of technologies and to say “whether you believe the overall impact of each technology on society is positive or negative.” AI only has an +8% net positive rating right now, right next to +7% for social media, which were only greater than crypto at -17%. Meanwhile cell phones, the internet, and solar energy are at +68%, +67%, and +65%, respectively. The Argument study broke this down further, asking about specific societal benefits from AI, finding broad skepticism and concluding “people aren’t really buying the bullish case for AI that CEOs and boosters alike are selling. In other words, the skepticism about AI’s effects is real and deep-running. And given how many people use it daily, this is not just an ill-informed set of opinions on something respondents have never seen before (like tariffs were before 2025).” It is possible for people to have one view at a societal level and then act differently at an individual level, but that does not seem to be what we’re seeing here. The plurality occasional usage and large percentage of complete avoidance speaks to the fact that a lot of people seemingly aren’t yet finding enough individual value net of their concerns to justify daily or even weekly usage. The gap in media narrative (that everyone is using AI for everything ) relative to the reality (that some people are using AI for some things ) perhaps reflects a bubble around early-adopting knowledge workers that includes much of the tech press (and me for that matter, though I’m trying really hard to stay connected to reality). It’s a mistake for companies, pundits, and policy makers to ignore how people are really feeling and acting about AI. It’s not all sunshine and rainbows. It’s also clearly not binary (all use or no use), but instead a continuum of AI opinions and use, with a lot of people in the middle. I think there is an apt analogy to be made here to preferences around meat consumption. Another thing that seems to be everywhere right now is protein . Telling us how important protein is in our diet is analogous to telling us how useful AI is for productivity. And, meat being a primary source of protein is analogous to AI chat tools being a primary source of generative AI. And yet here’s how Americans break down in terms of their meat consumption preferences, based on a handful of U.S. studies from this decade: 95% eat meat ( Gallup , 2023) 70% report reducing red meat consumption ( Rutgers , 2024) 30% eat (all) meat only rarely/occasionally ( Gallup , 2020) 12% don’t eat red meat ( Nature , 2026) 4% don’t eat any meat, that is are vegetarian ( Gallup , 2023) 1% don’t eat any animal products, that is are vegan ( Gallup , 2023) That is, not everyone eats meat, a majority actively curbs their consumption of red meat, and a significant percentage don’t eat it at all. Different people have different (not mutually exclusive) reasons for limiting their meat consumption, including health, cost, environment, and ethics. Those are all also primary concerns for AI consumption! The analogy also highlights market opportunities to appeal to people across the continuum, speaking to their feelings on AI and addressing their particular AI concerns. For example, we (at DuckDuckGo) make all AI features optional and one of those features, duck.ai , is a private chatbot alternative that helps address AI privacy concerns. To extend the analogy in this way, we’re a restaurant with a variety of options on the menu, from healthy meat dishes (private AI) to vegetarian (turn down AI) to vegan dishes (turn off AI), which most eaters across the spectrum can appreciate. Does this mean about one third of the population is bound to use AI only rarely/occasionally forever? No. Unlike with meat, the AI technology landscape is changing so rapidly that it is very unclear both where AI products and regulations will end up. Product evolution could make AI more useful to the average person, and regulations could reduce concerns. However, we can say that, as of right now, a meaningful percentage of the population has tried the current state of AI and has decided to actively limit their use of it. Thanks for reading! Subscribe for free to receive new posts or get the audio version . Once you’ve tried AI, you use it “for everything.” No, in fact most people who’ve tried it are just occasional AI users. AI has gotten so good that despite any misgivings, “everyone is using A.I.” No, in fact large chunks of the population aren’t using AI at all. 79/81% use AI at least rarely 41/42% are anxious about AI 32/31% use AI only monthly/every few months 22/31% are angry about AI 21/19% never use AI All of this triangulates to AI use in America at approximately one third actively using AI, one third occasionally using AI, and one third never using AI , with some movement depending on how you define those terms. In any case, this split is a far cry from “everyone is using AI for everything;” it’s much closer to “some people are using AI for some things.” AI use also hasn’t shifted that much in the past six months to a year. In fact, the only thing that has substantially changed is negative sentiment about AI has gone significantly up, for example the Gallup’s Gen Z poll reporting anger about AI jumping about 40% relative year over year. Many people are holding back AI use because of real AI concerns and lack of perceived AI value I think it is a reasonable conclusion to draw from all of this data that a significant percentage of the population is actively limiting their AI usage. The Searchlight study examines a big reason why: real concerns people have with AI. The top three concerns found are “AI will replace jobs and cause unemployment” (42%), "AI will violate people’s privacy” (35%), and “AI will spread misinformation and lies” (33%). This sentiment also matches a strong desire for safety/privacy AI regulation. A solid majority thinks “the government should prioritize creating safety/privacy rules for AI, even if that means the U.S. develops AI more slowly than countries like China.” Another big reason is skepticism in AI usefulness. SearchLight asked about a range of technologies and to say “whether you believe the overall impact of each technology on society is positive or negative.” AI only has an +8% net positive rating right now, right next to +7% for social media, which were only greater than crypto at -17%. Meanwhile cell phones, the internet, and solar energy are at +68%, +67%, and +65%, respectively. The Argument study broke this down further, asking about specific societal benefits from AI, finding broad skepticism and concluding “people aren’t really buying the bullish case for AI that CEOs and boosters alike are selling. In other words, the skepticism about AI’s effects is real and deep-running. And given how many people use it daily, this is not just an ill-informed set of opinions on something respondents have never seen before (like tariffs were before 2025).” It is possible for people to have one view at a societal level and then act differently at an individual level, but that does not seem to be what we’re seeing here. The plurality occasional usage and large percentage of complete avoidance speaks to the fact that a lot of people seemingly aren’t yet finding enough individual value net of their concerns to justify daily or even weekly usage. The gap in media narrative (that everyone is using AI for everything ) relative to the reality (that some people are using AI for some things ) perhaps reflects a bubble around early-adopting knowledge workers that includes much of the tech press (and me for that matter, though I’m trying really hard to stay connected to reality). It’s a mistake for companies, pundits, and policy makers to ignore how people are really feeling and acting about AI. It’s not all sunshine and rainbows. It’s also clearly not binary (all use or no use), but instead a continuum of AI opinions and use, with a lot of people in the middle. The meat of it all I think there is an apt analogy to be made here to preferences around meat consumption. Another thing that seems to be everywhere right now is protein . Telling us how important protein is in our diet is analogous to telling us how useful AI is for productivity. And, meat being a primary source of protein is analogous to AI chat tools being a primary source of generative AI. And yet here’s how Americans break down in terms of their meat consumption preferences, based on a handful of U.S. studies from this decade: 95% eat meat ( Gallup , 2023) 70% report reducing red meat consumption ( Rutgers , 2024) 30% eat (all) meat only rarely/occasionally ( Gallup , 2020) 12% don’t eat red meat ( Nature , 2026) 4% don’t eat any meat, that is are vegetarian ( Gallup , 2023) 1% don’t eat any animal products, that is are vegan ( Gallup , 2023)

Data Analysis AI

DYNOMIGHT 1 months ago

Is “colorectal cancer” rising in “young people”?

(Yes, but.) Over the past few years, I’ve seen many articles about mysterious rise in colorectal cancer (CRC) in young people. There are various stories for why this might be happening: General health. Maybe modern people are unhealthy (obesity, low physical activity, diabetes, poor sleep), leading to insulin resistance and chronic inflammation, meaning faster epithelial cell proliferation and a miscalibrated immune system that fails to stop early cancers? Ultra-processed food. Maybe people are eating more ultra-processed foods that contain additives (like emulsifiers) that degrade colon mucus, allowing bacteria to contact epithelial cells and drive inflammation? Or maybe ultra-processed food has low fiber and glycemic load, leading to insulin resistance and chronic inflammation, with the problems mentioned above? Bad meat. Maybe people are eating more red and/or processed meats, which expose the colon to nitrites and secondary bile acids, which inflame the epithelium and promote chronic inflammation? The microbiome. Maybe it’s the microbiome. For example, maybe people’s guts are getting colonized by strains of E. coli that produce genotoxic colibactin . Or maybe overuse of antibiotics in early life depletes protective bacteria in the gut, allowing harmful strains to expand, e.g. strains of B. fragilis that cause inflammation, or strains of F. nucleatum that can survive in the gut and drive tumor growth ? Environmental exposures. Maybe people are getting exposed to bad stuff in the environment (microplastics, forever chemicals , pesticides, endocrine disruptors, air pollution) that does bad stuff (damages gut barrier, screws up the microbiome, disrupts hormonal signaling)? Maternal health. Maybe poor maternal health (obesity, diabetes) exposes the fetus to elevated glucose / insulin / inflammation, and these in turn program the child for a lifetime of metabolic issues and inflammation? Whatever. Maybe alcohol / smoking / painkillers / calcium / vitamin D / inflammatory bowel disease / hereditary syndromes / screening bias? None of the experts seem to agree on which of these is the culprit, so I figured that I (person with blog) should help. If you poke at these stories, most of them are individually pretty weak. It can’t all be detection bias since CRC deaths are also going up in younger people. And several proposed causes (air pollution, tobacco) have actually fallen in rich countries. Other explanation, like E. coli producing colibactin, seem biologically real, but there’s no evidence that they’re increasing over time. Still other suggested causes (microplastics, forever chemicals) are mostly mechanistic speculation at this point. Obesity, inactivity, and chronic inflammation also all seem biologically real, and they are likely increasing, but why should they specifically cause colorectal cancer in young people ? A plausible answer to that last question is that they aren’t. They’re doing it, but not specifically . This will sound pedantic, but bear with me: If you say that CRC is increasing in younger people, what exactly does that mean? After all, the set of people who qualify as young changes over time. (Ever notice that you keep getting older?) Siegel et al. (2026) plot how often CRC was found in different age groups in 1995 and in 2022. They also provide this plot of how common different types of CRC are in different age groups. At a glance, this doesn’t look so bad. If you’re young, you might think, “OK, my current risk is higher than previous generations faced at the same age, but I can look forward to decreasing rates when I’m old.” You could easily think this is good news: While there’s a relative increase when you’re young, it’s tiny compared to the absolute decrease while you’re old. Unfortunately that’s the wrong way to think about it. Downham et al. (2026) plot CRC rates in different age groups across the Anglosphere over time. Everyone I’ve shown this plot to has said it’s confusing, so let me explain: The different lines track age-bands as people born in different years move in and out of those bands. For example, in the US plot in the bottom right, the “20-25” line starts with the left-most dot showing the CRC rate for people born between 1965 and 1970 when they were 20 to 24 years old (around 1990). The next dot shows the rate for people born between 1970 and 1975 when they were 20 to 24 years old (around 1995), and so on. That figure is weird, because the lines connect different groups of people. I wanted a plot where there are lines for different birth cohorts as they age. For unknown reasons, no one seems to make such plots, and the data isn’t trivial to access. So I used a plot digitizer to click on every damned point that US figure above and then replotted it: Now the individual lines show specific groups of people tracked through time. For example, the “1932.5” line shows CRC rates for people born between 1930 and 1935, when those people were at different ages. If you look closely, you’ll notice that these rates are higher those for people born between 1940 and 1945 for all ages (where we have data). That was the pattern for a long time: Between 1920 and 1950, later generations enjoyed lower CRC rates across all phases of their lives. But between 1950 and 1960, that pattern reversed and since then later generations have had higher CRC rates at all ages . We don’t know for sure what will happen in the future. But I think it’s likely this trend will continue. Yes, if you are currently young, you face higher CRC risk than previous generations did when they were young. That’s the bad news. The other bad news is that when you are old, you may also face higher CRC risk than previous generations did when they were old. The other other bad news is that CRC isn’t the only type of cancer that’s rising in later generations. Sung et al. (2019) give this plot: These are again the confusing graphs where individual lines show age bands as different people move in and out of them. But you get the point: Lots of cancers are going up in younger people later generations, including uterine, gallbladder, kidney, liver, pancreas, and thyroid. (Their additional material contains plots for 18 other cancers.) Note that these plots have a logarithmic y-axis, meaning the changes are larger than they might appear. Moving up a quarter of the way between two vertical ticks corresponds to an increase of a factor of ≈ 1.78. If lots of cancers are becoming more common in later generations, then why is everyone talking about CRC? I think that’s because CRC in unique in that it is: For example, thyroid cancer diagnoses have skyrocketed in recent decades. But that’s partly because of more detection, and thyroid cancer is highly treatable, without clear benefits from early detection. Pancreatic cancer also seems to be increasing, but we don’t have good ways to screen for it and even if we did, we don’t have good ways to treat it. CRC is really unique in that you can save lives by telling people, “Hey! CRC is going up! You should get screened!” If you’re interested in public health, that’s the most important thing. But if you’re interested in unraveling the mystery of CRC going up, it’s important to note that CRC isn’t really unique at all. Colorectal cancer is going up in young people. Various kinds of cancer are going up in later generations. (Definitely at younger ages, possibly at all ages.) This blog endorses colorectal cancer screening . We don’t yet know if colonoscopies are better than other methods of screening (sigmoidoscopy, stool tests), but we do know that screening is better than not screening. When caught early, CRC is highly treatable, often with only surgery (no chemotherapy or radiation) and a return to normal activities within a couple weeks. increasing in later generations treatable if caught early detectable via screening

Science Data Analysis

Entropic Thoughts 1 months ago

90 % of the t distribution

William Sealy Gosset was great. He improved beer at Guinness by using the statistics that existed at the time. Not happy with that, he invented new statistics to brew even better beer. The things he invented are used all over the place now, but Guinness wanted to keep him a secret weapon, so they made him publish his results under the fake name Student . One thing Gosset realised is that it is wrong to compute 90 % confidence intervals for the mean by taking the standard deviation of the sample, and assume a normal distribution , like-a-so: \[\hat{\mu} \pm 1.645 \hat{\sigma}\] (Continue reading the full article on the web.)

Statistics Science Data Analysis

Lalit Maganti 2 months ago

Which country voted the best at Eurovision?

Eurovision was on yesterday. I’ve never been interested much in the musical side but the weird political dynamics of Eurovision voting have always fascinated me; I tune in each year just for them and somewhat snarky commentary of Graham Norton, the UK commentator. As I was watching the jury votes come in, a question popped into my head: Which country has voted the best in Eurovision? That is, which country was best at picking the eventual top 10 and in the right order? Strangely enough, while there’s plenty of work on voting blocs and bilateral biases at Eurovision, most of it asks who votes for whom ; I wanted to ask who votes accurately . I couldn’t find anyone asking the question that way, so I decided to do some data analysis myself. To begin to answer this question, I first needed to formalize what “best” even means. That is, some mathematical notion of “good” and “bad”.

Data Analysis

Entropic Thoughts 2 months ago

Regatta Starting Stations – Chi-squared Continued

In the Henley Royal Regatta two teams at a time propel their boats up a river and compete to be first to go a distance. Teams get assigned to their starting stations – Berkshire or Buckinghamshire – at random. From there, it is a straight shot up the river, with the lane from each starting station being seemingly identical. I didn’t know any of this, but a reader reached out some time ago because they had noticed something odd about this, and they wanted to borrow me as a sounding board. Here’s the odd thing: the team that starts from the Berkshire station has won 53.5 % of the 7555 races in the historic data this reader looked at. This is highly unexpected. If teams are assigned at random, and the starting stations are practically equal, then the starting station of the winning team should be a coin flip. If we flip 7555 coins, we would never have as many as 53.5 % come up heads. (Continue reading the full article on the web.)

Statistics Data Analysis

Armin Ronacher 2 months ago

Content for Content’s Sake

Language is constantly evolving, particularly in some communities. Not everybody is ready for it at all times. I, for instance, cannot stand that my community is now constantly “cooking” or “cooked”, that people in it are “locked in” or “cracked.” I don’t like it, because the use of the words primarily signals membership of a group rather than one’s individuality. But some of the changes to that language might now be coming from … machines? Or maybe not. I don’t know. I, like many others, noticed that some words keep showing up more than before, and the obvious assumption is that LLMs are at fault. What I did was take 90 days’ worth of my local coding sessions and look for medium-frequency words where their use is inflated compared to what wordfreq would assume their frequency should be. Then I looked for the more common of these words and did a Google Trends search (filtered to the US). Note that some words like “capability” are more likely going to show up in coding sessions just because of the nature of the problem, so the actual increase is much more pronounced than you would expect. You can click through it; this is what the change over time looks like. Note that these are all words from agent output in my coding sessions that are inflated compared to historical norms: The interactive word trend chart requires JavaScript. Something is going on for sure. Google Trends, in theory, reflects words that people search for. In theory, maybe agents are doing some of the Googling, but it might just be humans Googling for stuff that is LLM-generated; I don’t know. This data set might be a complete fabrication, but for all the words I checked and selected, I also saw an increase on Google Trends. So how did I select the words to check in the first place? First, I looked for the highest-frequency words. They were, as you would expect, things like “add”, “commit”, “patch”, etc. Then I had an LLM generate a word list of words that it thought were engineering-related, and I excluded them entirely from the list. Then I also removed the most common words to begin with. In the end, I ended up with the list above, plus some other ones that are internal project names. For instance, habitat and absurd , as well as some other internal code names, were heavily over-represented, and I had to remove those. As you can see, not entirely scientific. But of the resulting list of words with a high divergence compared to wordfreq, they all also showed spikes on Google Trends. There might also be explanations other than LLM generation for what is going on, but I at least found it interesting that my coding session spikes also show up as spikes on Google Trends. The choice of words is one thing; the way in which LLMs form sentences is another. It’s not hard to spot LLM-generated text, but I’m increasingly worried that I’m starting to write like an LLM because I just read so much more LLM text. The first time I became aware of this was that I used the word “substrate” in a talk I gave earlier this year. I am not sure where I picked it up, but I really liked it for what I wanted to express and I did not want to use the word “foundation”. Since then, however, I am reading this word everywhere. This, in itself, might be a case of the Baader–Meinhof phenomenon , but you can also see from the selection above that my coding agent loves substrate more than it should, and that Google Trends shows an increase. We have all been exposed to LLM-generated text now, but I feel like this is getting worse recently. A lot of the tweet replies I get and some of the Hacker News comments I see read like they are LLM-generated, and that includes people I know are real humans. It’s really messing with my brain because, on the one hand, I really want to tell people off for talking and writing like LLMs; on the other hand, maybe we all are increasingly actually writing and speaking like LLMs? I was listening to a talk recording recently (which I intentionally will not link) where the speaker used the same sentence structure that is over-represented in LLM-generated text. Yes, the speaker might have used an LLM to help him generate the talk, but at the same time, the talk sounded natural. So either it was super well-rehearsed, or it was natural. At least on Twitter, LinkedIn, and elsewhere, there is a huge desire among people to write content and be read. Shutting up is no longer an option and, as a result, people try to get reach and build their profile by engaging with anything that is popular or trending. In the same way that everybody has gazillions of Open Source projects all of a sudden, everybody has takes on everything. My inbox is a disaster of companies sending me AI-generated nonsense and I now routinely see AI-generated blog posts (or at least ones that look like they are AI-generated) being discussed in earnest on Hacker News and elsewhere. Genuine human discourse had already been an issue because of social media algorithms before, but now it has become incredibly toxic. As more and more people discover that they can use LLMs to optimize their following, they are entering an arms race with the algorithms and real genuine human signal is losing out quickly. There are entire companies now that just exist to automate sending LLM-generated shit and people evidently pay money for it. If we take into account the idea that the highest-quality content should win out, then the speed element would not matter. If a human-generated comment comes in 15 minutes after a clanker-generated one, but outperforms it by being better, then this whole LLM nonsense would show up less. But I think that LLM-generated noise actually performs really well. We see this plenty with Open Source now. Someone builds an interesting project, puts it on GitHub and within hours, there are “remixes” and “reimplementations” of that codebase. Not only that, many of those forks come with sloppy marketing websites, paid-for domains, and a whole story on socials about why this is the path to take. I have complained before that Open Source is quickly deteriorating because people now see the opportunity to build products on top of useful Open Source projects, but the underlying mechanics are the same as why we see so much LLM slop. Someone has a formed opinion (hopefully) at lunch, and then has a clanker-made post 3 minutes later. It just does not take that much time to build it. For the tweets, I think it’s worse because I suspect that some people have scripts running to mostly automate the engagement. And surely, we should hate all of this. These low-effort posts, tweets, and Open Source projects should not make it anywhere. But they do! Whatever they play into, whether in the algorithms or with human engagement, they are not punished enough for how little effort goes into them. That increases in speed and ease of access can turn into problems is a long-understood issue. ID cards are a very unpopular thing in the UK because the British are suspicious of misuse of a central database after what happened in Nazi Germany. Likewise the US has the Firearm Owners Protection Act from 1986, which also bans the US from creating a central database of gun owners. The gun-tracing methodologies that result from not having such a database look like something out of a Wes Anderson movie . We have known for a long time that certain things should not be easy, because of the misuse that happens. We know it in engineering; we know it when it comes to governmental overreach. Now we are probably going to learn the same lesson in many more situations because LLMs make almost anything that involves human text much easier. This is hitting existing text-based systems quickly. Take, for instance, the EU complaints system, which is now buckling under the pressure of AI . Or take any AI-adjacent project’s issue tracker. Pi is routinely getting AI-generated issue requests, sometimes even without the knowledge of the author . I know that’s a lot of complaining for “I am getting too many emails, shitty Twitter mentions, and GitHub issues.” I really think, though, that now that we know that it’s happening, we have to change how we interact with people who are increasingly automating themselves. Not only do they produce a lot of shitty slop that we all have to sit through; they are also influencing the world in much more insidious ways, in that they are influencing our interactions with each other. The moment I start distrusting people I otherwise trust, because they have started picking up LLM phrasing, it erodes trust all over society. You also can’t completely ban people for bad behavior, because some of this increasingly happens accidentally. You sending Polsia spam to me? You’re dead to me. You sending me an AI-generated issue request and following up with an apology five minutes later? Well, I guess mistakes happen. Yet, in many ways, what is going on and will continue to go on is unsettling. I recently talked with my friend Ben who said he forced someone to call him to continue a conversation because he was no longer convinced he was talking to a human. Not all of us have been exposed to the extreme cases of this yet, but I had a handful of interactions in which I questioned reality due to the behavior of the person on the other side. I struggle with this, and I consider myself to be pretty open to new technologies and AI in particular. But how will my children react to stuff like this? My mother? I have strong doubts that technology is going to solve this for us. The reason I don’t think technology is going to solve this for us is that while it can hide some spam and label some generated text, it won’t fix us humans. What is being damaged here are social interactions across the board: the assumption that when someone writes to you, there is a person on the other side who has put some care into the interaction. I would rather have someone ghost me or reject me than send me back some AI-generated slop. Change has to start with awareness and an unfortunate developmend is that LLMs don’t just influence the text we rea and influence the text we write, even when we don’t use htem. Given the resulting ambiguity, we need to become more aware of how easily we can turn into energy vampires when we use agents to back us up in interactions with others. Consider that every time someone reads text coming from you, they will have to increasingly have to make a judgement call if it was you, or an LLM or you and an LLM that produced the interaction. Transparency in either direction, when there is ambiguity, can help great lengths. When someone sends us undeclared slop, we need to change how we engage with them. If we care about them, we should tell them. If we don’t care about them, we should not give them visibility and not engage. When it comes to creating platforms and interfaces where text can be submitted, we need to throw more wrenches in. The fact that it was cheap for you to produce does not make it cheap for someone else to receive, and we need to find more creative ways to increase the backpressure. GitHub or whatever wants to replace it, will have a lot to improve here and some of which might be going against it’s core KPIs. More engagement is increasingly the wrong thing to look at if you want a long term healthy platform. Whatever we can do to rate-limit social interactions is something we should try: more in-person meetings, more platforms where trust has to be earned, and maybe more acceptance that sometimes the right response is no response at all. And as for AI assistence on this blog, I have an AI transparency disclaimer for a while. In this particular blog post I used Pi as an agent to help me generate the dynamic visualization and I use the agent to write the code to analyze and scrape Google Trends.

Data Analysis JavaScript AI

Robin Moffatt 2 months ago

Interesting links - April 2026

A bit of a streamlined edition, this month. Lots of interesting links still, but less commentary. You can put that down to me prevaricating on getting my previous blog about Materialized Tables in Apache Flink finished, and leaving myself little time to work on this one :) Not including the detailed narration actually knocks a bunch of time off the preparation—I’d be interested in your feedback as to how much the absence of narration impacts (if at all) your enjoyment of reading it. Let me know in the comments below! Something that I’m slowly changing is how I categorise links to do with AI. A few months back anything "AI" got its own section. It wasn’t much more than a novelty really; certainly not something worth distracting the regular link sections with. But now AI is just part-and-parcel of many people’s workflows, a regular component in their toolbox. So where an article is about credibly using AI as part of an existing topic (such as data engineering), I’ll file it in that section. (And if this news makes you cross because you abhor anything AI, well, I’ve got news for you ) .

Data Data Analysis AI

iDiallo 2 months ago

Have You Seen the New Excel?

Stop coding. Stop hiring. Stop building. While the tech world obsesses over large language models and neural networks, I discovered the real disruptor that has been hiding in plain sight. Mine was originally installed on my desktop in 1992. And now, it's about to change everything in the world. We are talking about Microsoft Excel, of course. If you haven't looked at a spreadsheet lately, you are missing the most significant leap in enterprise capability since the invention of the corporation itself. We are entering an era of No-Code where the code was never needed in the first place. My own job as a software engineer is not safe, and I'm looking forward to the future. Developers from every walk of life are afraid, and for good reasons. You hear the complaints constantly: "How can I ensure the code works? I can't possibly review a PR with a thousand files. It's unmaintainable." This is a crisis of confidence in the software engineering sector. This specific anxiety has never existed in the Excel ecosystem. Code is called code for a reason, it is meant for the machine to read, not people. In Excel, we don't worry about "reviewing pull requests." We worry about results. The spreadsheet handles the logic and you handle the business outcome. It abstracts away the complexity so you don't have to pretend to understand it. And let's talk about the intimidation factor. Have you ever opened a modern codebase? It's a labyrinth of directories, dependencies, and config files. Where do you even start? It's paralyzing. How do you get started with Excel? You double-click an icon. It opens. It is a file. It is a grid. You type. It works. The barrier to entry is non-existent, yet the ceiling is infinite. If you are getting paid a high salary, and are watching how efficient excel is, you will be terrified. Companies are realizing they don't need distinct software solutions for distinct problems. They just need a grid. We are seeing enterprises replace entire departments with a single file. That is not an exaggeration. The HR department? Replaced by an org chart linked to a payroll calculator. The supply chain team? Replaced by a real-time inventory tracker. The marketing department? Replaced by a pie chart and a mailing list. Why pay for Salesforce? A well-formatted sheet with conditional formatting is a Customer Relationship Manager (CRM). Who even knows how to write SQL? SQL is legacy. A workbook with 1 million rows is a database. Jira is redundant when you have Gantt charts generated from cell dependencies. On top of it all, it has AI. It comes equipped with Microsoft Copilot for 365 apps, not to be confused with Windows Copilot, Microsoft Copilot, Copilot for Teams, Copilot+, Copilot Chat, or Copilot Web. This is the Copilot. It sits inside your grid, ready to extrapolate trends from column D and write your VLOOKUPs for you. While other AI startups are fighting for funding rounds, this integration is already live, embedded directly into the tool that runs the global economy. You aren't hearing much about Venture Capital funding or Series A rounds when it comes to Excel. Why? Because it is already profitable. It doesn't need a roadmap to profitability because it is the roadmap. While other platforms burn cash to acquire users, Excel is the default operating system of business. It requires no adoption curve. It requires no evangelists. It requires only that you open it and have a Microsoft 365 apps subscription. Total Vertical Integration Excel is versatile. It is a text editor; you can write your novel in cell B2. It is a design tool; pixel-perfect layouts can be achieved by merging cells and removing gridlines. It is an IDE; you can write and execute VBA code directly within the environment. It handles the visual and the logical simultaneously. You can present a quarterly report to the board while the underlying formulas are calculating the ROI of the lunch break. It creates a seamless workflow where the input and the output exist in the same plane. Privacy, Scalability, and The Cloud For the enterprise client, Excel offers the ultimate flexibility. Are you concerned about data sovereignty? Run your entire global operation locally on a ThinkPad from 2012. The file sits on your hard drive, unbreachable by the cloud. Do you need to scale? Push it to the cloud. Collaborate in real-time. Ten thousand employees can edit the same cell, creating a hive mind of productivity that traditional management structures cannot compete with. Oh, if you want to add support for crypto, just add a new worksheet. Batteries are included. The Future is a Cell The economy is shifting. We are moving away from specialized labor and toward generalized grid management. If your job involves inputting data, processing data, or presenting data, Excel has already automated you. It doesn't sleep, it doesn't ask for a raise, and it doesn't make calculation errors unless you tell it to. Best of all, it doesn't hallucinate. The grid is absolute, it is infinite and the grid is the future. Learn Excel now, or get left behind. That’s what AI Hype sounds like to my ears. Yes, it’s a great tool. But I don’t think we are all gonna die and lose our jobs. The same way we didn’t die and or lose our jobs to Excel. None of these things are jokes about Excel by the way, you can run entire companies from it. I'm tempted to just start hyping it everyday until everyone gets annoyed.

Business Data Analysis Sql

Entropic Thoughts 3 months ago

Readership maths skills

Statistics Data Analysis Analytics

Carlos Becker 3 months ago

Mapping Brazilian Cell Towers

I was curious about how many cell towers were around me - so I built the tool I wanted.

Data Analysis Data Visualization

Michael Lynch 4 months ago

Refactoring English: Month 15

Hi, I’m Michael. I’m a software developer and founder of small, indie tech businesses. I’m currently working on a book called Refactoring English: Effective Writing for Software Developers . Every month, I publish a retrospective like this one to share how things are going with my book and my professional life overall. At the start of each month, I declare what I’d like to accomplish. Here’s how I did against those goals: Visits and orders are down, but mainly because January was such an outlier due to “The Most Popular Blogs of Hacker News in 2025.” I got another lucky bump from the HN moderators putting “My Eighth Year as a Bootstrapped Founder” on the front page. I mentioned in January that I added regional pricing for my book. I wasn’t tracking data carefully, but just based on order notifications, it seemed like most of my orders were coming from countries outside the US, so I took a closer look at the data. The first question was: is it really true that the majority of orders use regional pricing now? It’s true. The majority of Refactoring English customers are now outside of the US. The US accounts for only 28% of orders by volume and 40% by revenue. I was also surprised to see how many customers purchase from countries like India and Brazil, where English is not the primary language, so I checked English vs. non-English primary countries: Surprisingly, the majority of orders for Refactoring English come from countries where English is not the primary language, though English-speaking countries are a small majority revenue-wise. Next question: Do readers from certain countries purchase at a higher rate than others relative to total website visitors? Wow! One out of every six readers in Kazakhstan purchases the book! I need to start advertising in Kazakhstan. Okay, the extreme Kazakhstan result is based on a single customer, so that’s probably an outlier. And I bet my website analytics undercount visitors from Kazakhstan. What if I focus on the top countries based on website visitors? The US is my top country for website visitors, but a relatively low share (0.5%) purchase the book. Indian readers purchase at the highest rate, with 2.5% of website visitors purchasing the book. Canadian readers purchase the most by revenue, with every Canadian reader giving me about $0.47 in additional book sales. Clearly, I need to start pandering more to India and Canada in the book. I could change all the Docker examples to cricket examples and look for more opportunities to praise Shopify. After the US, most website visitors come from China (5.9% of total), but I’ve had zero sales in China. At first, I thought buying ebooks was not so popular in China, but I just checked what regional discount I was offering in China and was surprised to find it was zero. I wasn’t offering a regional discount in China at all. I made two mistakes in my price generation scripts that excluded a huge number of countries: The local currency thing is silly in retrospect because I can still offer a discount and just accept payment in USD. And I’m not sure how I ended up missing so many Stripe-supported countries. I even missed Kazakhstan, my new favorite country! I was only offering regional discounts in about 39 countries. After my fixes, the list grew to 156. And within 12 hours, I got a new order from Kazakhstan. With the majority of Refactoring English readers coming from countries where English is a second language, should I adjust the book to better serve non-native speakers? A few readers have asked about English tips for non-native speakers. I’d like to tackle the subject, but I have no experience writing as a non-native speaker. I want everything in the book to be techniques I personally use rather than things I’ve heard secondhand . My best idea is to find editing clients who are non-native speakers and look for patterns in their writing to include in the book. But right now, I’d like to get the v1 finished. The beauty of an ebook is that you can keep iterating on it and find ways to improve it even after official release. I’ve been using AI for software development for about a year and a half, but there have been two major inflection points: Since December, I’ve been spending more and more time doing AI-assisted coding. It’s become an ever-increasing part of my workday and non-work time. I used to have a bad habit of checking email and social media excessively. During the past month, I’ve repeatedly had the experience of noticing that it’s 4pm, but I haven’t checked email or social media. Except it’s because I’ve fallen into an AI vortex and forgot everything else. Every month, I think, “Is this a problem?” And in the past few weeks, I’ve had to face the fact that, yes, it’s a problem. I generally start each workday by writing a schedule on a little notepad on my desk. I break the day into 30-minute blocks and write down how I’ll spend that block. Historically, I stick to the schedule when I’m disciplined. When I have less will power, I let fun tasks exceed their budgets by a block or two. With AI-assisted coding, I was getting to the point where I’d make a schedule and then completely ignore it and play with AI all day. I wouldn’t say that I have an “addiction” to AI in the way people develop addictions to drugs or alcohol, but I am letting AI-assisted coding distract me from work that I recognize is more important, like finishing my book. There are a few factors that make AI especially compelling and easy for me to get sucked into: I feel like I can integrate any technology, write in any programming language, install any tool. There used to be an annoying level of friction in using any new software, but now I can mostly just hand it to AI and ask it to figure out how to install it or debug it, and it just works. In the 90s, Bill Gates published a book called Business @ the Speed of Thought . I’ve never read it, but I keep thinking back to that book title as I use AI. It’s not literally at the speed of thought, but it’s closer than anything I ever imagined. I can have an idea for a feature, give a brief explanation to an AI agent, and see the feature materialize in minutes. Even before AI, I’d often intend to spend an hour coding and instead spent three. But there were natural limits to how long I could code. A few hours of intense dev work fries my brain, and work becomes unpleasant, unproductive, or both. With AI, you can build for hours without doing any deep thought. And even when something does require thought, AI makes it easier than ever to take on tech debt. When I’m coding myself, I don’t want to do something the ugly way because then I’m the one who has to maintain that hack. But if I’m making AI do everything, I don’t feel the pain of hacky, ugly code. One of the things that makes gambling addictive is variable rewards . Our brains are more captivated by a system that gives you $10 at random intervals than one that delivers you money on a fixed, predictable schedule. Whether intentional or not, my experience with AI agents varies wildly. Sometimes, I point it at a 2,000 line log file and it diagnoses the issue before I’ve even asked a question. Other times, I give it a simple task, and it spends the next 20 minutes aimlessly roaming my codebase. Because I don’t know if the wait will be 5 seconds or 20 minutes, I sit there staring at the agent for a minute, then compulsively check it every few minutes, then start some other AI task while I’m waiting. And then I’m cycling between multiple agents and don’t even remember what they’re all doing. One of the most maddening experiences I have with AI is when I’ve set up the AI agent to complete a long task, and I come back hours later to find the AI paused its work a few minutes after I left and asked, “Okay, the next step is to try a full build, but that will take 30-60 minutes. Would you like me to continue?” Yes! That’s why I left the task to you! It’s hard to predict exactly what effect AI will have on the software industry, but I feel confident that it will completely upend the ecosystem. We’re in the early stages of a massive shake-up. Depending on how things turn out, there are paths forward for me as a software developer, but I also think there’s at least a 20% chance that we’re in the last year or two of “software developer” being a job that requires any special knowledge or skill. It could be like what happened to elevator operators . Right now, there are a few factors that make AI-assisted development especially attractive for developers in my position: The current situation with AI can’t last. The AI bubble could burst, and I’ll have to start paying the non-subsidized, metered rate. Or AI will continue to improve to the point where I have no advantage over junior engineers or even people with no software experience. I’ve found a few techniques for getting my AI usage back to a manageable place: It turns out that most of Refactoring English ’s readers come from outside the US. I’m using AI-assisted coding too much. Result : Published “Why Improve Your Writing?” and “Improve Your Grammar Incrementally” Result : Scheduled a discussion about design reviews I only included countries where Stripe supports the local currency. Even with this filter, I accidentally omitted a lot of countries where Stripe supports the local currency. In February 2025, I started using an integrated AI agent in my code editor In December 2025, I started running AI agents with full permissions (within isolated environments) AI is helpful for junior engineers, but senior engineers are the ones who can use it best There are multiple AI companies competing heavily on price and using VC money to subsidize costs. I use flat-rate plans, but I consume the equivalent of about $4k/month in API costs, and even those rates are probably VC-subsidized. Don’t start the day with an AI project If I start with AI and then work on my book, then I’m switching from an exciting, easy task to a hard, unsexy task. If I instead start the day with an hour of writing , I’ve done my hard task for the day and don’t have to move uphill. This is challenging because I often set up long AI tasks overnight, and I’m always curious in the morning to see how they turned out. Reduce parallel AI-driven projects. Parallel work feels appealing because I can cycle between agents. In practice, I find it sucks me in too much because there’s a spinning plates mentality of some agent always needing attention. Published two new book chapters Published “Eversource EV Rebate Program Exposed Massachusetts Customer Data” and complained to the MA Department of Public Utilities Don’t start the day with an AI coding project. It’s too distracting and too hard to switch to something harder but more important. Finish Refactoring English It won’t be fully polished and edited, but I want to complete all the chapters.

Data Analysis Writing Business

Rado's Radical Reflections 4 months ago

Human Intuition, AI Formalization: A Real Analysis Case Study

Disclaimer - I wrote the core ideas; Claude helped flesh out and polish the article. See appendix for more on this. This is a follow up to my previous post on leaning on Claude for Lean. I’ve now worked up to chapter 8.3 in Tao’s companion. The speed is great and Claude’s capabilities continue to impress (autoformalization is possible, but not my goal). I haven’t been stuck on anything so far. I’ve also upstreamed many typos to the companion repo .

Data Analysis AI

Manuel Moreale 4 months ago

Step aside, phone: week 2

Halfway through this enjoyable life experiment, and overall, I’m very pleased with the results. As I mentioned last week, I was expecting week two usage to be a bit higher compared to week one, where I went full phone-rejection mode, but I’m still pleased with how low my usage was, even though it felt like I was using the phone a lot. No huge spikes this week, didn’t need to use Google Maps a lot, so the time distribution is a lot more even, as you can see. The first three days of the week were pretty similar to the previous week. I moved my chats back on the phone, and that’s most of the time spent on screen since “social” is just the combination of Telegram, WhatsApp, and iMessage. Usage went up a bit in the second part of the week, but I consider that a “healthy” use of the phone. On Thursday, I spent 20 or so minutes setting up an app, one that I’d categorise as a life utility app, like banking or insurance apps. They do have a site, but you’re required to use the phone anyway to take pictures and other crap, so it was faster to do it on the phone. Then on Saturday, I had to use Maps as well as AllTrails to find a place out in the wild. I was trying to find a bunker that’s hidden somewhere in a forest not too far from where I live (this is a story for another time), and that’s why screen time was a bit higher than normal on that particular day. Overall, I’m very happy with how the week went. A thing I’m particularly pleased with is the fact that I have yet to consume a single piece of media on my phone since we started this experiment. So far, I have only opened the browser a couple of times, and it was always to look up something very specific, and never to mindlessly scroll through news, videos or anything like that. My content consumption on the phone is down to essentially zero. One fun side effect of this experiment is how infrequently I now charge my phone. I took this screenshot this morning before plugging it in, and apparently, the last time it was fully charged was Wednesday afternoon. I’m now charging it once every 3 or 4 days, which is pretty neat. Thank you for keeping RSS alive. You're awesome. Email me :: Sign my guestbook :: Support for 1$/month :: See my generous supporters :: Subscribe to People and Blogs

Media Data Analysis Culture

Ankur Sethi 5 months ago

I used a local LLM to analyze my journal entries

In 2025, I wrote 162 journal entries totaling 193,761 words. In December, as the year came to a close and I found myself in a reflective mood, I wondered if I could use an LLM to comb through these entries and extract useful insights. I’d had good luck extracting structured data from web pages using Claude, so I knew this was a task LLMs were good at. But there was a problem: I write about sensitive topics in my journal entries, and I don’t want to share them with the big LLM providers. Most of them have at least a thirty-day data retention policy, even if you call their models using their APIs, and that makes me uncomfortable. Worse, all of them have safety and abuse detection systems that get triggered if you talk about certain mental health issues. This can lead to account bans or human review of your conversations. I didn’t want my account to get banned, and the very idea of a stranger across the world reading my journal mortifies me. So I decided to use a local LLM running on my MacBook for this experiment. Writing the code was surprisingly easy. It took me a few evenings of work—and a lot of yelling at Claude Code—to build a pipeline of Python scripts that would extract structured JSON from my journal entries. I then turned that data into boring-but-serviceable visualizations. This was a fun side-project, but the data I extracted didn’t quite lead me to any new insights. That’s why I consider this a failed experiment. The output of my pipeline only confirmed what I already knew about my year. Besides, I didn’t have the hardware to run the larger models, so some of the more interesting analyses I wanted to run were plagued with hallucinations. Despite how it turned out, I’m writing about this experiment because I want to try it again in December 2026. I’m hoping I won’t repeat my mistakes again. Selfishly, I’m also hoping that somebody who knows how to use LLMs for data extraction tasks will find this article and suggest improvements to my workflow. I’ve pushed my data extraction and visualization scripts to GitHub. It’s mostly LLM-generated slop, but it works. The most interesting and useful parts are probably the prompts . Now let’s look at some graphs. I ran 12 different analyses on my journal, but I’m only including the output from 6 of them here. Most of the others produced nonsensical results or were difficult to visualize. For privacy, I’m not using any real names in these graphs. Here’s how I divided time between my hobbies through the year: Here are my most mentioned hobbies: This one is media I engaged with. There isn’t a lot of data for this one: How many mental health issues I complained about each day across the year: How many physical health issues I complained about each day across the year: The big events of 2025: The communities I spent most of my time with: Top mentioned people throughout the year: I ran all these analyses on my MacBook Pro with an M4 Pro and 48GB RAM. This hardware can just barely manage to run some of the more useful open-weights models, as long as I don’t run anything else. For running the models, I used Apple’s package . Picking a model took me longer than putting together the data extraction scripts. People on /r/LocalLlama had a lot of strong opinions, but there was no clear “best” model when I ran this experiment. I just had to try out a bunch of them and evaluate their outputs myself. If I had more time and faster hardware, I might have looked into building a small-scale LLM eval for this task. But for this scenario, I picked a few popular models, ran them on a subset of my journal entries, and picked one based on vibes. This project finally gave me an excuse to learn all the technical terms around LLMs. What’s quantization ? What does the number of parameters do? What does it mean when a model has , , , or in its name? What is a reasoning model ? What’s MoE ? What are active parameters? This was fun, even if my knowledge will be obsolete in six months. In the beginning, I ran all my scripts with Qwen 2.5 Instruct 32b at 8-bit quantization as the model. This fit in my RAM with just enough room left over for a browser, text editor, and terminal. But Qwen 2.5 didn’t produce the best output and hallucinated quite a bit, so I ran my final analyses using Llama-3.3 70B Instruct at 3bit quantization. This could just about fit in my RAM if I quit every other app and increased the amount of GPU RAM a process was allowed to use . While quickly iterating on my Python code, I used a tiny model: Qwen 3 4b Instruct quantized to 4bits. A major reason this experiment didn’t yield useful insights was that I didn’t know what questions to ask the LLM. I couldn’t do a qualitative analysis of my writing—the kind of analysis a therapist might be able to do—because I’m not a trained psychologist. Even if I could figure out the right prompts, I wouldn’t want to do this kind of work with an LLM. The potential for harm is too great, and the cost of mistakes is too high. With a few exceptions, I limited myself to extracting quantitative data only. From each journal entry, I extracted the following information: None of the models was as accurate as I had hoped at extracting this data. In many cases, I noticed hallucinations and examples from my system prompt leaking into the output, which I had to clean up afterwards. Qwen 2.5 was particularly susceptible to this. Some of the analyses (e.g. list of new people I met) produced nonsensical results, but that wasn’t really the fault of the models. They were all operating on a single journal entry at a time, so they had no sense of the larger context of my life. I couldn’t run all my journal entries through the LLM at once. I didn’t have that kind of RAM and the models didn’t have that kind of context window. I had to run the analysis one journal entry at a time. Even then, my computer choked on some of the larger entries, and I had to write my scripts in a way that I could run partial analyses or continue failed analyses. Trying to extract all the information listed above in one pass produced low-quality output. I had to split my analysis into multiple prompts and run them one at a time. Surprisingly, none of the models I tried had an issue with the instruction . Even the really tiny models had no problems following the instruction. Some of them occasionally threw in a Markdown fenced code block, but it was easy enough to strip using a regex. My prompts were divided into two parts: The task-specific prompts included detailed instructions and examples that made the structure of the JSON output clear. Every model followed the JSON schema mentioned in the prompt, and I rarely ever ran into JSON parsing issues. But the one issue I never managed to fix was the examples from the prompts leaking into the extracted output. Every model insisted that I had “dinner with Sarah” several times last year, even though I don’t know anybody by that name. This name came from an example that formed part of one of my prompts. I just had to make sure the examples I used stood out—e.g., using names of people I didn’t know at all or movies I hadn’t watched—so I could filter them out using plain old Python code afterwards. Here’s what my prompt looked like: To this prompt, I appended task-specific prompts. Here’s the prompt for extracting health issues mentioned in an entry: You can find all the prompts in the GitHub repository . The collected output from all the entries looked something like this: Since my model could only look at one journal entry at a time, it would sometimes refer to the same health issue, gratitude item, location, or travel destination using different synonyms. For example, “exhaustion” and “fatigue” should refer to the same health issue, but they would appear in the output as two different issues. My first attempt at de-duplicating these synonyms was to keep a running tally of unique terms discovered during each analysis and append them to the end of the prompt for each subsequent entry. Something like this: But this quickly led to some really strange hallucinations. I still don’t understand why. This list of terms wasn’t even that long, maybe 15-20 unique terms for each analysis. My second attempt at solving this was a separate normalization pass for each analysis. After an analysis finished running, I extracted a unique list of terms from its output file and collected them into a prompt. Then asked the LLM to produce a mapping to de-duplicate the terms. This is what the prompt looked like: There were better ways to do this than using an LLM. But you know what happens when all you have is a hammer? Yep, exactly. The normalization step was inefficient, but it did its job. This was the last piece of the puzzle. With all the extraction scripts and their normalization passes working correctly, I left my MacBook running the pipeline of scripts all day. I’ve never seen an M-series MacBook get this hot. I was worried that I’d damage my hardware somehow, but it all worked out fine. There was nothing special about this step. I just decided on a list of visualizations for the data I’d extracted, then asked Claude to write some code to generate them for me. Tweak, rinse, repeat until done. I’m underwhelmed by the results of this experiment. I didn’t quite learn anything new or interesting from the output, at least nothing I didn’t already know. This was only partly because of LLM limitations. I believe I didn’t quite know what questions to ask in the first place. What was I hoping to discover? What kinds of patterns was I looking for? What was the goal of the experiment besides producing pretty graphs? I went into the project with a cool new piece of tech to try out, but skipped the important up-front human-powered thinking work required to extract good insights from data. I neglected to sit down and design a set of initial questions I wanted to answer and assumptions I wanted to test before writing the code. Just goes to show that no amount of generative AI magic will produce good results unless you can define what success looks like. Maybe this year I’ll learn more about data analysis and visualization and run this experiment again in December to see if I can go any further. I did learn one thing from all of this: if you have access to state-of-the-art language models and know the right set of questions to ask, you can process your unstructured data to find needles in some truly massive haystacks. This allows you analyze datasets that would take human reviewers months to comb through. A great example is how the NYT monitors hundreds of podcasts every day using LLMs. For now, I’m putting a pin in this experiment. Let’s try again in December. List of things I was grateful for, if any List of hobbies or side-projects mentioned List of locations mentioned List of media mentioned (including books, movies, games, or music) A boolean answer to whether it was a good or bad day for my mental health List of mental health issues mentioned, if any A boolean answer to whether it was a good or bad day for my physical health List of physical health issues mentioned, if any List of things I was proud of, if any List of social activities mentioned Travel destinations mentioned, if any List of friends, family members, or acquaintances mentioned List of new people I met that day, if any A “core” prompt that was common across analyses Task-specific prompts for each analysis

JSON Python AI Data Analysis

James Stanley 5 months ago

Evidence of absence

"Absence of evidence is not evidence of absence", they say. They're wrong. In this post we'll work through a scenario and show that absence of evidence is in fact evidence of absence. You can see Yudkowsky for more on this topic. You have a box with 100 bags in it. Each bag has 100 balls in it. 99 of the bags have 100 white balls, the final bag has 99 white balls and 1 black ball. (Assume all of the bags are indistinguishable from the outside, and all of the balls are the same size and weight etc., and the black ball is at a random position within its bag). What is the probability that a bag, selected uniformly at random, contains the black ball? You'll give your first probability estimate before the bag is opened, and then balls will be removed from the bag one by one and you can revise your estimate after seeing each ball. What is your strategy? Your lab assistant rummages around the box and selects a bag uniformly at random. You should agree that this bag contains the black ball with 1% probability, because you know that 1 in 100 bags contain the black ball. Your lab assistant takes out the first 99 balls. You observe that they're all white. This observation is merely an absence of evidence of the black ball, and, so they say, it is therefore not evidence of absence of the black ball, and your probability estimate should be unchanged, at a 1% chance that the bag contains the black ball. But while the 99 bags that only contain white balls would provide this observation every time, the 1 bag that contains the black ball would provide this observation only 1 time out of 100. So now you ought to agree that the probability that the bag contains the black ball is now only about 0.01%, not 1%, and we see that the ongoing "absence of evidence" of the black ball was in fact evidence of its absence after all. Your lab assistant pulls out the final ball. It's white .

Data Analysis Statistics

Entropic Thoughts 5 months ago

Wilks' Tolerance Intervals

Data Analysis Statistics

Gabriel Weinberg 5 months ago

Simulating likely 2026 World Cup matchups (for all matches)

I’ve been using Cursor for coding for some time, but I finally gave Claude Code a try for this short side project: simulating the 2026 World Cup bracket to predict likely matchups for all matches, which is useful when considering which matches to potentially go to. Methodology: Start with the official World Cup tournament schedule (including yet-to-be played playoff matches) Blend Elo rankings with FIFA rankings (50/50) Use the Elo formulas to probabilistically predict winners (assuming no draws, even in group stage) Run one million individual simulations of the full tournament (it reaches diminishing returns around 50K, but hey, why not!) Run again with a home field advantage boost (+180 Elo) for the U.S., Canada, and Mexico based on prior World Cup outcomes Count up who participated in each match Some interesting findings (at least to me as a U.S. fan) are below, followed by a rundown for every match (in reverse order). Big Disclaimer 1 : The above is of course a gross simplification of the actual tournament. For example, it doesn’t take into account team matchup histories, game models, etc. etc. I do think, however, it is useful enough for the designed purpose of generally predicting likely match participants. Big Disclaimer 2 : I did a lot of output validation so I think the results are largely accurate (to the extent they can be given Big Disclaimer 1). However, I didn’t write or review every line of code, so it is likely there are still some bugs in there. If you think you see anything that seems off, let me know and I’ll try to track it down (and update anything if necessary). Aside on Claude code: Like many others, I found this process both productive and frustrating. It was definitely faster than I could have done it alone, but Claude kept forgetting basic context, and was way overconfident in the accuracy of the results. That is, many rounds of validation at every stage of output was absolutely necessary despite Claude saying things were good. I couldn’t trust its word at all. HA+ = with home field advantage (anytime this comes into play there is a + next to the team name) HA- = without home field advantage Here’s a visualization of the above made by a reader (thanks!) Start with the official World Cup tournament schedule (including yet-to-be played playoff matches) Blend Elo rankings with FIFA rankings (50/50) Use the Elo formulas to probabilistically predict winners (assuming no draws, even in group stage) Run one million individual simulations of the full tournament (it reaches diminishing returns around 50K, but hey, why not!) Run again with a home field advantage boost (+180 Elo) for the U.S., Canada, and Mexico based on prior World Cup outcomes Count up who participated in each match

Sports Data Analysis Programming