Posts in Data-analysis (20 found)
Stone Tools 3 days ago

Lotus Improv on the NeXT

It is spoken of in reverent tones, invoked like a magic spell, whispered about as one of the mythical "roads not taken" in computing history. Its proponents write glowing reports and fawning YouTube reviews, lamenting its failure in the historical marketplace, each offering some justification for it not having set the world ablaze. It is safe to say, the legend of Lotus Improv persists to this day. If we put ourselves in Steve Jobs's shoes circa 1988, it's easy to see why he would have been smitten by an early preview of what we now call a "pivot table." NeXT was struggling to find its market, and exclusives which justified a system purchase were sorely needed. "Maybe Improv can do for NeXT what VisiCalc did for the Apple II," must surely have crossed Jobs's mind. In making it a NeXT exclusive, Improv seemed to be a clear-eyed, forward-looking answer to the question dogging the company, posed by the inaugural issue of NeXTWORLD Magazine , "Why NeXT?" Both NeXT and Improv brought innovative solutions to their respective computing questions of the day. NeXT made UNIX palatable to those scared of the terminal, debuted Display Postscript on a mega-pixel display, and made development for the system a snap. Improv rethought multi-dimensional data modeling, offered English-like formulas, and made it easy to rearrange data views on a whim. The bones of both remain to this day. NeXT's operating system, NeXTSTEP / OPENSTEP, morphed into macOS, and by extension iOS. Lotus abandoned Improv fairly quickly, but a quick survey of today's spreadsheets shows a universal embrace of pivot tables. Even today, it feels like nothing else, and still does data tricks no other spreadsheet can do as elegantly. Given its mythos, I feel a kind of nerd obligation to love it. Events have converged to make this the right time to finally talk about Lotus Improv . First, it is time once again for "International Spreadsheet Day." Second, as Improv was initially a NeXT exclusive, it seems only appropriate to take a look at that machine here on the 15th (!) anniversary of the passing of Steve Jobs. Third, and not to toot my own horn too loudly, it is the 1st anniversary of this very blog. Lotus Improv was one of four pieces of software I had in mind when deciding to start Stone Tools. In fact, it was quite literally the software that first triggered the thought, "I wish I had a place to talk about stuff like this." Deluxe Paint , the first post, was also one of those four. There are two yet to go; place your bets. The trouble was, while I knew Deluxe Paint intimately and could jump right in, I simply had no real experience with other categories of productivity software. It would have been impossible to judge Improv fairly without first getting better acquainted with the landscape into which it was born. So, I bided my time. Boy, howdy, do I want to impress upon you the superhuman restraint I've exhibited in waiting a full year to examine Improv . I think we all understand Improv, being a spreadsheet , knows how to calculate sums and transcendentals and other mathematical goodness. "It does math!" is not a particularly compelling justification for its existence, nor to support its claim as the "NeXT generation of spreadsheets." Instead, let's get to know it by studying examples which show off its unique strengths, and with the suggestion of the book Improv 2.1 Revealed! we'll model up some good ole' American baseball, a game I love and understand completely. Industry historians are far more capable of discussing the exile and homecoming of one Mr. Steven Paul Jobs back to Apple. I touched on the TL;DR of it in the timeline, and if you're really clamoring for juicy tidbits, a book was published this very year that delivers on that front. While NeXT hardware was well regarded, it never really caught on. In the long run, the better result of NeXT engineering was NeXTSTEP and how neatly it wrapped UNIX in a clean, user-friendly, graphical interface. It's quite an accomplishment. For me, it was the technology behind NeXTSTEP/OPENSTEP developer tools, Objective-C and Interface Builder, that kickstarted my programming career when those became the OS X development environment (hat tip to Aaron Hillegass's Cocoa Bootcamp at the Big Nerd Ranch). Even today, years after the advent of iOS and Swift, NeXT technology remains in the Foundation layer of Apple software, as denoted by -prefixed classes (a name-spacing trick from the Objective-C days). For this investigation, I'm using NeXTSTEP 3.3, the last version before transitioning to OPENSTEP. NeXTSTEP looks and feels like a modern system, with some choice differences. Familiarity with any modern operating system, especially macOS, will inoculate you from breaking out into hives using it. Considering the eye-candy Jobs seemed excited to add to OS X after the Apple merger, NeXTSTEP is positively staid; nothing here screams " lickable ." If you've ever wondered what Susan Kare could have achieved if she'd had a larger canvas and palette (NeXTSTEP is 2-bit) than the original Mac, look no further. I like it. The dock is present, though more rigid than the playful fluidity it acquired when transitioning to OS X. No "genie effects" nor dynamic magnification; it is ever-present and doesn't hide itself when not in use. Some applications change the dock icon when running (like Mail ), while other applications do not (like Edit ). One odd quirk is how the dock indicates running apps, by the absence of three dots on an icon tile. That's a strange choice, decorating the inactive icons. File browsing adds a little complexity to system usage. The file browser contains its own mini-dock for favorites, the path to the currently selected item rendered as giant icons, and the file browser proper. Dragging files and folders around can only be done with an item's full-size icon. If you're browsing in list view, you can't drag list items into a folder or the dock, for example. Bit of a head-scratcher, that one. Menuing for all applications is handled via a user-positionable, vertical menu. Sticking the menu somewhere sets its position for the active application, and every application can have its own position. You could theoretically have the menu jumping around all over the place as you hop between apps. Thankfully, it can also be called up at the current mouse position with a right-click; shades of RISC OS. Also, good news for those who love the spinning beach ball on macOS! You'll feel that little ASMR tingle when it appears, especially if you run a color workstation. It's funny the things that can make something feel like "home." Honestly, my biggest issue in adjusting to NeXTSTEP is the mouse. By default, it feels weird to use, with a dual-level acceleration curve, one for slow movement, one for fast. Crossing the threshold between "slow" and "fast" jumps the cursor across the screen when the alternate acceleration curve kicks in. I could never find my cursor except when moving the mouse verrrryyyy slooowwllllyyyy. Luckily, the emulator Previous has tools to help compensate ( check "Sharpening the Stone" ). Overall, you'll only need to adjust your thinking a little here and there, forget some modern habits, and you should be comfortable with NeXTSTEP in short order. 35 years later, it still looks and feels much like any modern operating system. Almost depressingly so? Maybe that's a topic for another day. Question: what is interesting about the following screenshot from Improv ? Answer: it contains all information necessary to build a copy for yourself. All data, formulas, dimensions, and relationships are exposed and visible. I note this not to suggest that screenshots are a great way to share Improv worksheets with others, though you could; the sheet above was rebuilt from a screenshot, in fact. Rather, I want to illustrate how "graspable" data can be in Improv's environment. Its formulas read naturally, and value references match the item names in the table. The work does what it claims. Let's interrogate that a little more thoroughly. To understand Improv is to forget what we know about spreadsheets. Slash commands? Never heard of him. B1? J93? K12 = int(A14 * lookup(B2..B7))? The ramblings of a madman. Rather than generic alphanumeric column/row identifiers, we define our rows and columns with "categories of items" specific to our domain. Put another way, instead of asserting, "Columns A to L represent the months, and rows 1 through 31 represent the days," we will create a category called "months" and define twelve items for it named "January," "February," etc. and we will create a category called "days" and define 31 of those. Take a look at these comparative videos which illustrate the difference. First, I'll use LibreOffice Calc to create a months/days grid, apply a value to January 1, and calculate December 31 as a multiple of January 1. Here's how that same table is created in Improv . There are three things to note: how sequential items are generated, how categories are named (not strictly necessary for this simple example), and how formulas are created/applied. There's a lot to absorb, I know, starting with how a new, blank worksheet consists of a single cell, not an infinite table of empty cells. Improv works with only and precisely the data we define. Where in the LibreOffice example we attached meaning to the columns by adding labels to the top of each column, that was for our benefit not Calc's . I could, just to feel the warm, snuggly blanket of classic column/row naming, make an Improv category called "columns" and name my items A, B, C, etc. and a category called rows, with names 1, 2, 3, etc. Were we to take such a pedestrian approach to categories and items, we would diminish the magic of Improv's formula builder. The bottom panel holds all formulas, which are applied at the granularity defined, in the order shown. In the example, I specified but could just as easily have targeted all of December by specifying . That single formula would apply to every list item in December without needing to copy formulas through to each cell, nor set absolute/relative markers for cell references. Here's a simple expansion of the previous table. Every day in January increases by 10%, and its corresponding day in December doubles the January value. The two formulas shown are the only two formulas for the entire worksheet. I want to stress this; there are no other formulas lurking inside cells, hidden from view, waiting to be revealed by a selection or editing process. There are no duplicated formulas per cell, each increasing a relative cell reference by . We simply refer to categories (in this case "months" or "days"), items (as "January" or "27"), or specific intersections (as "April:19"), including relativity signifiers , , and . You might be thinking of ways to trick the system; to confuse and befuddle it. Why would you want to bully an old app like that? What did Improv do to deserve such shabby treatment? "What if two items have the same name? What if two formulas target the same cell?" Fret not, all shall be revealed and you will feel proper shame at trying to confuse this old program. In the above example I have renamed one of the Days as "March," to create ambiguity when referencing "March." Does "March" mean the Day or the Month now? We can be explicit, referring to . Note though that if we aren't explicit, Improv will be on to our tricks and call us out, as in formula 5. In formula 2, I set to the static value . In formula 3, I also set it to be calculated as "double its corresponding January value." Note the "overlap" icon (two intersecting squares) on those formulas and the text in the cell. Clicking the overlap icon gives us an opportunity to resolve ambiguity by explicitly selecting which formula should win for that cell. Cleanly handling formula ambiguity was originally seen by the prototype team as an intractable engineering problem. From "Improv: The Inside Story", by Simson L. Garfinkel, NeXTWORLD , Fall 1991: Personally, I like Improv's solution. I honestly think more software should be very dumb, and not try to intuit or guess what I want. Tell me the problem in clear language, and provide the simplest tool that gives me the most control over problem resolution. Be smart about helping me, don't try to help me by being smart. Before we get in too deep, I want to make sure I note Improv's beautiful help system, replete with index and search. Considering the difficulty in tracking down a manual for the NeXT release , having this available during my exploration proved invaluable. It's neatly illustrated, with terse (sometimes overly terse) descriptions of the tools, hyperlinked to other pages of the documentation, as warranted. Improv felt it could do even better than that and made its help contextual . The Help system, when invoked, defaults to displaying documentation for whatever is currently selected in the user interface. That's very useful, but becomes supercharged with "Point for Help," as demonstrated in the video above. This turns the cursor into a symbol, and whatever we click will be revealed in the Help window. This is a great idea, because I don't always know what Lotus wants to call some little interface doohickey, so I don't know what to search for. Now I know: the "Calc" button is a "Status Indicator" and the triangle icon that sometimes shows up is the "Formula Error Indicator." Pork chop sandwiches. Looking through Usenet archives, I saw this called "toilet paper" by at least one person. Lotus called it the "item dispenser." It's an interesting first attempt to do something we take for granted today and only exists in the NeXT version of Improv . It's cute, but smacks of someone being overly excited with new development toys. If we need to generate a list of items to populate an axis of your data, this tool generates sequential list items for you. It does its best to kind of "guess" our intentions, so if we type in "Sunday," then pull down on the dispenser, days of the week will be auto-generated in a list. We then "tear off" the list, which loads the cursor with the items, which we attach to an existing item, appending that category with the loaded list items. That means we need to have at least one item in a category onto which the list can be attached. This is annoying, because the existing item likely has no relationship to the generated list. I'm always having to make a list, attach it to an existing item, then delete the original, unrelated item. These days, we can type directly into a cell and extend that cell intelligently across adjacent cells. In fact, Lotus would switch to that methodology when migrating Improv over to Windows for version 2. Hold ALT and drag an existing item to get the same effect. The original dispenser truly is a feature that could only have been thought up within the NeXT environment. I don't hold it against the devs for getting excited by the GUI flexibility Interface Builder afforded. I, too, did the same back in the early OS X 10.2 days. It practically begged us to play around with GUI ideas, remixing elements into new configurations. "First Responder" and "bindings" stans, represent! One of the great features of that development environment was how easy it was to “say ‘No’ a thousand times” while trying ideas. Lotus embraced that, achieving quickly on NeXT what had blocked their progress on OS/2 and Microsoft's Presentation Manager. The "item dispenser" was singled out as a specific tool that benefitted from Interface Builder; there really wasn't anything similar to its rapid, iterative, GUI development at the time. In fact, I might go so far as to say there's nothing quite like it today, even from Apple . Joel Spolsky, in editing The Best Software Writing I , added a personal aside on Lotus Improv , blaming its failure thusly, " Improv required you to define strict n-dimensional hypercubes for your data. There was none of the flexibility of traditional spreadsheets." I suppose if one wanted to paint Improv in the most inhospitable light, Spolsky is correct. In practice, it doesn't feel as impenetrable as his description. However, I do think he's on to something that gets at the heart of why Improv never gained traction. For now, let's hold our collective nose and dive into the scary, foreboding waters of "n-dimensional hypercubes" and model up some baseball. Our first dimension will be . Let's add a dimension next. Next, we'll get more granular and add . Internally, Improv separates our work into three parts: data, views, and formulas. What we see on screen is just Improv's best-effort to represent our three-dimensional data flattened out into a two-dimensional view. Improv adds extra columns to the view, as necessary, to ensure we always see the full extent of our data relationships. Now, we'll add some win/loss data and the next thing you know we’ve built a 4-dimensional hypercube. Are we scared yet? Since I've only modeled at the "Division" level, I'll add together divisional wins inline to calculate the totals and... hmmm... "No dice," as they say in baseball (Presumably. Are dice involved in baseball?) Alas and alack, we can only enter concrete numbers, dates, or text into cells; any formula, no matter how trivial, must go into the Formula Panel (bottom). In this case, I could add a formula that targets a specific cell, as , but the whole point of a spreadsheet is to automate that kind of junk for me. I suppose the path of least resistance is to add a category to add up all team wins and, wait a second... Dang it, that petard I confidently told myself wouldn't hoist me has come around to do precisely that. Here we see the downside of dimensional modeling, at least for the layperson, and arguably for most typical spreadsheet users. It all looks like a neat little chart, but that word "dimension" describes and reveals the problem. To add a category, its items necessarily apply to all other categories. Teams don't do that; they are unique to each League and Division. But we just might have a way out. Lotus engineers clearly understood that categories alone weren't going to cut it. Categories impose rigidity to our data that does not always accommodate the messiness of real data. We need another way to organize our data. Groups are intra-category, named collections of sequential items. Make a category and add a bunch of individual items. Select a sequential group of some subset of items and group them, giving that group a unique name. Once everything is grouped, groups of groups can be grouped to group a group of ... wait, I got myself confused. Groups can be grouped, 'nuff said. With "Teams," we quickly run into a side-effect of groups, and how it can necessitate a cascade effect across multiple categories. I want to add teams and group them by "Division," but that forces me to also make "League" a group as well. What was three categories have now collapsed into one: "Teams" Here's what that looks like in practice. We still gain the benefit of name references in formulas, as in the above untethered-to-reality example. A reference to would apply to both American and National, unless we specify by as in and Groups can be subtotaled, using the function, as you can see in this more data-rich chart of the MLB stats from 2022 to 2026. Notice, however, we risk a conflict for in our field, where there are two formulas battling it out for dominance over the same cell calculation. The handy keyword will skip (i.e. "won't apply this formula to") groups, cells, items, and categories as designated. Groups helped bring granularity and specificity to our data structure, but the more categories we remove, the more we remove access to the very feature that has kept Improv in the back of nerd minds for decades. You know what I'm talking about. FINALLY we get to the point those in the know have been anxious to talk about, " Lotus Improv invented pivot tables. It says so right there in Wikipedia." At this stage of my knowledge of all things retro productivity, I think the most I'm willing to concede right now is that Lotus Improv likely invented pivot tables as we understand them today. But, like Yoda coyly teasing the existence of Luke's sister to a guy who should have already known about her, there is another program, a future Stone Tools subject, that may lay claim to the title of "first pivot table." I exercise my right, as "guy who runs the blog," to reserve final judgement on this point until then. For our purposes today, it's a true enough statement and the inventor even has a nice video on YouTube in which he shows it off to Lotus staff for the first time. Meet Pito Salas, inventor of the pivot table. Mike Wells (shown in the thumbnail) takes a question at 28:40 and reveals that NeXT has "Objective Fortran" and "Objective Lisp" ?!?! Salas, now retired , developed the idea while working for Lotus Development's Advanced Technology Group. He put together a fakey version of his vision, a kind of slideshow, showing off the things he wanted to be able to do: English-like formulas, multi-dimensional data, and rearrangeable views into the data. He was given the greenlight to proceed, "All those [ideas] I had described with hand waving,'' recalls Pito. "I had no clue how to build it." A year of nose-to-the-grindstone later, and a prototype dubbed "Back Bay" went on US tour, wowing financial modelers. Everyone who saw it, wanted it. Let's take a look at how Improv handles pivoting. To start, I'm leaving it empty of data, so we can focus on the dynamic view structure more easily. Then, I'll move categories around to show how the view flexes to present the data. Next, I'll put categories into the third axis, called a "Page Category," allowing us to paginate our data. Finally, I'll enter a formula and a little data to show off a trick pivot table contemporaries cannot perform. Now I'm going to try and replicate this in LibreOffice Calc v26.8. There are a few things to note, but let's first point and laugh at the fumbling attempts of a LibreOffice newbie (me). A few things jumped out at me while comparing these approaches. First, modeling up a relatively complex data set in Improv is as easy as thinking about it, and the presentation of the data matches my mental model. LibreOffice requires me to bridge a fairly wide conceptual gap between what I want and how I model it, due to the amount of redundant data I have to enter for every line item. Some items, like "League," translate cleanly, while "Record" does not. In fact, I'm still unclear what the best approach would be to convert my Improv computed item "Win Ratio." LibreOffice has the concept of a "computed field" inside pivot tables, but reaching across columns to pluck out related data to perform computations is complicated. In Apple Numbers, Microsoft Excel, Google Sheets, and LibreOffice , pivot table data is splintered off into a separate snapshot of the data, dedicated to the pivot table view. Pivot table data can typically be lightly manipulated, but does not affect the original data. The reverse can be done, but only after asking for an explicit update of the pivot table to reflect changes to the primary data table. This is a huge drop-off in ease-of-use from the Improv way of things where every view of the data directly manipulates the base data. Dragging around categories is just as easy in either approach, though Improv gives immediate drag-and-drop access to what LibreOffice calls a "Filter," i.e. a paged view, and generally "feel better," IMHO. LibreOffice requires us to open up a separate panel and drag things around in there. I will say, however, that being able to filter the filters (i.e. exclude/include specific subsets of data) is handy. Sheets , Numbers , and Excel (online version) are all strictly panel-based pivoting. Interestingly enough, the panel approach is precisely the interface Lotus rejected , at the behest of Steve Jobs. Once more, from the Garfinkel article. Having now directly compared Improv's approach to dynamic views against modern pivot tables, it is safe to say they are the same only so much as wagyu beef with gruyere cheese prepared by Gordon Ramsay and the sawdust at the bottom of a Happy Meal are both "cheeseburgers." Improv tastes much better. A Lotus invention which debuted in Lotus 1-2-3 , "named ranges" have attempted to bring order to the chaos of cell identifiers. In 1-2-3, will apply a name of choice to a defined cell range, after which that range can be referenced in formulas by that name. So, in 1-2-3 or any modern spreadsheet we can name range as and will work. Superficially, this looks similar to how Improv works with categories and items for multi-dimensional data. However, named ranges are for convenience only, and do not establish relationships between data values. In Improv , would represent a fundamental aspect of a piece of data. Getting the sum of all sales is the same, , but it being a dimension on the data gives us deeper options. For a specific year, will match against two axes at once. could further pinpoint our summation. Let's see how this would be done with modern spreadsheets and named ranges. If we want to mimic Improv's dimensional relationships, we must define relationships of interest within each formula, (and those formulas have to be copied over to every related cell) such that: Meaning "sum items in the named range called "Value" only if the Year is 2020, and the League is "American," and the Record is "Win," and the Division is "East") Of course we could substitute hardcoded filter criteria with values pulled from a lookup, or perhaps extract equated values via a statement. Maybe even set up a little sub-table to hold calculated values. Quickly this gets way more complicated than I'd have hoped for, so personally I'd resort to doing it the old-fashioned way and just hardcode in a bunch of cell references. To reiterate, the complete solution for an entire sheet in Improv , where win/loss information is its own dimension, is the single formula: The last of the big features, which takes advantage of NeXT's graphics prowess, is the charting functionality. Creating a chart launches an entirely separate application, Presentation Builder , which has grand plans on becoming PageMaker when it grows up. Generating a chart generates a full, blank page onto which a chart is placed. A suite of palettes complements the charting workflow, allowing for precision edits of the chart, its titles, colors, positioning, and more within the full page. Multiple charts can be generated and placed on the same page. Arbitrary text and shapes, lines pointing at interesting bits, and even sound annotations (a system-wide NeXT tool, called "Lip Service") can be placed at will. So it goes that while we have the tools to place arbitrary text on the page, yet lack tools to center that text, the sense that Lotus has overextended itself is strong. Sometimes it's enough to just generate the output a program excels at, and leave "page composition" to the professionals. Again, I need software to be more dumb . That's without even mentioning the plethora of formatting tools scattered amongst an unreasonable number of palettes, panels, and "inspectors." A benefit of these tools on the NeXT is that everything we do is Postscript by default, thanks to NeXT's Display Postscript screen rendering engine. "Print" a document as PostScript, and we can render the resultant .ps file on any modern system. I opened it directly in Affinity and received precisely what I saw in Improv , chopped off text and all, but like, high-quality chopped off text. Look, I try my best to craft a compelling narrative, moving from topic to topic, stitching together callbacks and observations to lead you, my dear reader, on a carefully considered ride. It's just you and me, side by side at the keyboard, chatting about software and learning together. But, sometimes there's junk I need to squeeze in that I can't find a natural segue for discussing. Here, then, is a brain dump of other things I want you to know about Improv . Speaking of version 2... In 1993, NeXT abandoned its hardware initiative and moved over to Intel, rebranding as NeXT Software, releasing OPENSTEP soon after. At the same time, Improv also moved over to Intel, but parted ways with NeXT. With version 2, Lotus developers achieved the success that eluded them on OS/2, porting Improv over to Windows 3.1. The feature set for version 2 is disappointingly similar to version 1, and by this time competing software was starting to embrace some of Improv's unique qualities. We do finally get undo, so thank Kapor for small favors. Scripting arrives in the form of LotusScript. A new Windows UI adds a bunch of buttons to the interface, and flushes the toilet paper dispenser for a more sensible list generator. That's basically it for the new stuff, but at least they kept the old problems as well. Cell reference relativity is still locked to a single dimension. Formulas cannot be freely rearranged, only cut-and-pasted into position. No 3D worksheets. Then, to throw a little extra wrinkle into the mix, there was Lotus's product strategy on the PC. If Improv wasn't your thing, maybe their other Windows spreadsheet, 1-2-3/Windows , would feel more comfortable. Or their other-other option, Lotus Symphony . Or their other-other-other option, Lotus Works . Or their other-other-other-other option, Lotus 1-2-3 for DOS Release 4 , which went on sale in 1994. Regardless, it was well-liked by the press, even if reviewers didn't seem comfortable giving full-throated endorsements. Take this Compute , July 1994, review for example. Or this PC World , February 1994, review. It would be hard to intuit, based on that review, that PC World had declared Improv the "Winner" in the Applications category for their 10th Annual Awards for Technical Excellence just a few months prior. 3 in spirit, if not in flesh; let's take a quick look at Quantrix , the product I clumsily foreshadowed earlier. From ex-Lighthouse CEO (also ex-Sun CEO) Jonathan Schwartz's blog , "In 1989, a few friends and I started a software company, Lighthouse Design, that devoted itself to the NeXT platform. Whether any of us admitted it at the time, Lighthouse was built by a group of people for whom Steve Jobs was the gravitational center of the universe." Their devotion to Jobs was returned in favor, by way of advice for their products to flourish on NeXT. Ultimately, according to Schwartz, the problem was less about finding a way for Lighthouse to flourish on NeXT than it was about getting NeXT to flourish in the first place. When looking for a buyout in Lighthouse's waning days, they had fingers crossed to be acquired by NeXT but Jobs didn't bite. They were bought by Sun, their products shoved in a vault never to see the light of day. With Microsoft unwilling to support the platform, Lighthouse made the closest thing to an Office suite for NeXT. They built programming tools, project management software, a presentation tool called Concurrence ( that Jobs used and which "inspired" Keynote ), and two spreadsheet programs: ParaSheet and Quantrix . ParaSheet took the more traditional approach, while Quantrix was just a flat-out clone of Improv , I don't know how else to say it. It has the same split screen worksheet/formulas, the same formula tools, the same draggable categories, it imports Improv documents, and more. According to one Hacker News reader , "The manuals were interchangeable." Lighthouse took the ideas behind Improv and ran with them, envisioning what a possible v3 could have been. Put simply, everything I've expressed frustration with in Improv is addressed in Quantrix . It has undo. It has drag-and-drop, reorderable formulas. It has scripting. It has cross-worksheet category synchronization. The "Item Dispenser" is replaced with an in-line widget; pull to expand like any modern spreadsheet does today. It has multi-dimensional cell relativity. It runs on OPENSTEP for Intel, as well as NeXTSTEP on Motorola 68K . It has color. It loses contextual help (can't win 'em all). And that morass of panels and palettes for editing the charts I showed earlier? Those are collapsed into a single, tidy "Inspector." Long story short? If you want to experience the best version of Lotus Improv today, use Lighthouse Quantrix v2.3. You'll get all of the original, cool tools built better, with a more sensible interface. Plus, Lighthouse gave away their NeXT applications, with license code, for free. How neat is that? The big twist, for those affording me a generous definition of "twist," is that Quantrix still exists . It is alive and being sold for "ask for quote," so I think I'll just put this credit card back in my wallet at this time. Today, it very explicitly targets the professional, high-end, financial modeling crowd, and not the general-purpose masses. You know, the crowd that Lotus first showed their Back Bay prototypes to back in the day (see timeline). Its self-introductory video pivots (get it?) many of its 35-year-old features as revolutionary. A back-handed indictment of the stagnation of an industry, or a product finally finding its niche? You be the judge! It's hard to suss out precisely what happened, but I found a little information floating around the Quantrix forums . If that is to be trusted, Lighthouse's Quantrix chief engineer, Pete Murray, founded the company Subx and rebuilt Quantrix as a Java application in 2003. Quantrix's website says it was acquired by IDBS in 2010. I'm still completely in the dark how Quantrix escaped Sun's containment field, as Sun had refused to sell off the Lighthouse apps to hungry buyers; that stuff could have been converted into OS X apps in the early days. At any rate, it is one of the more handsome JRE applications, while suffering from telltale Java rendering issues, especially with regard to text positioning and scaling. Otherwise it is clean and inoffensive. Sterile? Boring. That's the word I was looking for: boring. "It knows its target audience," is perhaps more charitable. So where does this leave my relationship with Lotus Improv ? Why did I dangle that "And yet..." at the start of the post? If I'm completely honest with myself, there is a vibrating tension between "wanting to love it" and "actually loving it." After all of these weeks, I think I'm in a pretty good position to suggest why it didn't take the world by storm: it's not a great spreadsheet program. Let me qualify that statement lest r/Improv4Life roasts me alive; the roasting can wait a moment. For what it does, it is amazing and even fun to use. What it doesn't do is provide a good, general purpose spreadsheet experience to the masses. It claimed to do that, positioning itself as the "NeXT generation of spreadsheets." Whatever Lotus Improv did or did not accomplish for the spreadsheet genre, one thing is for certain: it makes a fantastic tech demo. Seen in that light, I'd have been as sold on it as Jobs was, excitedly showing it to NeXT non-believers, "Look what MY computer can do!" Those moments of smug technological superiority amongst friends are fun, aren't they? Once you played with it for a week or two, and the initial excitement of pivoting dulled, you may have ashamedly harbored a bit of buyer's remorse. Adam Bosworth, in an article contemplating a talk he gave at ICSOC '04, was thinking about software complexity vs. simplicity, and posited: Paradoxically, Improv's promise to improve data comprehensibility wound up narrowing its general appeal. Want to quickly throw down a few lines of data, like a makeshift phone directory? You're gonna have to rethink your data as a matrix of abstract qualities about your friends. Despite its friendly English-like formulas and delightful GUI for pivoting data, Improv is in actuality purpose-built for far more rigorous data modeling than its playful tools suggest. It's the next generation of spreadsheets for a thin slice of the market. That said, there is a non-zero set of readers for whom their hardcore number crunching needs perfectly align with Improv's promises. If that's you, you're absolutely going to fall in love... ...with Quantrix . Ways to improve the experience, notable deficiencies, workarounds, and notes about incorporating the software into modern workflows (if possible). Improv does export to Lotus 1-2-3 , but you'll likely be disappointed with the results. Consider a table with formulas and calculated values in the cells. Everything you see in the cells will be exported, but there will be no formulas or relationships between anything. You will receive a snapshot of all current values, arranged to visually match the table structure you had when exporting. Improv's approach to shared formulas and multi-dimensional data simply don't have an elegant export solution to traditional spreadsheets. If you have Improv documents you want to import into Quantrix , two criteria must be met. Save your Improv project as an file (not the default ), and make sure you have the Lighthouse (available at the same site offering the Quantrix software link I mentioned under "Version 3") installed at . Exporting charts is simplicity, thanks to Display Postscript. The Postscript files NeXTSTEP generates open and render just fine on modern machines, using Ghostscript or other graphics programs. As mentioned previously, I actually recommend using Lighthouse Quantrix v2.3 instead of Lotus Improv , on the NeXT. The substantial features Improv lacks are numerous: In general, it's lacking the clay-like malleability of a traditional spreadsheet. If you're doing what a lot of us do, using spreadsheets like a kind of mini-database, you'll probably get frustrated. Here's the main event. Compare and contrast with his OS X announcement at MacWorld 2000 for bonus points. Previous v4.4 on Windows 11 NeXTstation Turbo 68040 CPU (variable clock rate; ~40 - 50Mhz) 2GB hard drive Monochrome "Megapixel" display NeXTSTEP 3.3 Lotus Improv Rel 1.0 (v108r0) There is no undo. This isn't the first program I've covered that lacked undo, and I'm not talking about software that can be forgiven for being born too early. The formula panel at the bottom applies formulas in the order shown. As order is important, it would make sense that you might want to rearrange them from time to time. There is no mechanism to do this, other than cut and paste. This has been more of a pain than the lack of undo. Improv uses special keywords to add cell relativity to formulas. So THIS value is the PREV value times 2, or whatever. But we're in a multi-dimensional sheet, so what do these keywords point to, precisely? Well, in a formula they must pick a category, and only that category can be referenced relatively. If a formula uses multi-dimensional data to calculate a value, like , those keywords won't work, though it would be completely natural to write such a formula and expect success. There is no scripting. Lotus 1-2-3 users would have been particularly put out in not being able to bring their carefully built automation over to Improv , which would be a huge shocker since they're made by the same company. I would even go so far as to say that that alone would have pushed a great number of potential converts away. Nobody has the time to rebuild their financial empires from scratch! There are no 3D sheets, meaning there is no linking between projects. A project can contain up to 100 worksheets, and those worksheets can reference one another, but only within the same project file. If a category in a worksheet exactly matches a category in another, item for item, those can match and data will flow across as expected. Unfortunately, there is no way to synchronize the shared categories. Change the items in one sheet and those will not be picked up in sheets using that linked category. This means we cannot make a "master" category inherited by other worksheets. Like a lot of things back in the early days of productivity software, user diligence plays a large role in keeping things running smoothly. Even version 2 on Windows didn't add this feature. If only someone would step up to the challenge. ( foreshadowing ) Previous never crashed on me, and has gotten a lot easier to set up since its early days. Version 4.4 has been great. For mouse movement, I found that setting (in Previous , not NeXTSTEP) "Slow movement" to "Fast" and "Fast movement" to "Slow" balanced things out to something close to consistent tracking; it worked for me, is all I can promise. There is a modern add-on to NeXTSTEP called "GKMouseScaler" that lets the user draw precise cursor acceleration curves, should you need better control. Personally, I never needed it. I found the 2-bit monochrome display to be more than sufficient; no color needed. This is version 1.0 of Improv and it did crash a couple of times, quite suddenly. There appeared to be no specific trigger, nor function, that led to the crashes. It just beachballed and died. NeXTSTEP itself felt stable to me. For installing software from modern repositories, three things to keep in mind Thanks to a convenience in Previous , at in NeXTSTEP you can access your host computer file system. You'll find a lot of .tar.gz files in the wild, so brush up on and Once you have a file, double-click and it should install itself. No undo in v1 on NeXT; v2 on Windows does. Can't reorder formulas except through cut/copy/paste. and only work along a single axis, even though it makes complete logical sense for them to work along multiple axes in the same formula. Presentation Builder is overly complex on the one hand, yet missing very basic tools on the other hand. Can't reference values in other projects, aka no 3D worksheets. No scripting options in v1 on NeXT; v2 on Windows does.

0 views

One Weird Trick I Learned As A Beginner to Triple My Swim Distance

Four weeks stuck at 50 meters, then two 100s and a 150 in one session. What changed, with the data.

0 views

Predicting Peace

I have long struggled with predicting when peace will occur. Actually, I’m struggling with predicting anything relating to war, but let’s focus on when peace arrives for now. There are two such questions in the current ACX 2026 competition: When I entered my forecast for these questions, I did it based on Laplace’s rule of succession, i.e. as a function of their duration only. This is sort of justified if war durations are heavy-tailed, which would mean that the longer they go on for, the less likely it is that peace comes soon. But I don’t know if that is true. (Continue reading the full article on the web.) Will there be a cease-fire in Ukraine in 2026? Will there be a cease-fire in Sudan in 2026?

0 views
The Coder Cafe 3 weeks ago

The Wisdom of Crowds Explained

☕ Welcome to The Coder Cafe! Today, we explore the wisdom of crowds: why groups of strangers can outperform the world’s best experts, when that power breaks down, and what it means for the organizations we work in. Get cozy, grab a coffee, and let’s begin! A few weeks ago, I was scrolling on X when I stumbled across that poll : I picked an option (the red) and looked at the results: Quite surprising, right? Thousands of people voted, and the result landed exactly on the theoretical proportions of the circle. Each person made an independent judgment, and together they converged on the truth . It reminded me of a story I read in a book about crowds (see the Sources section at the end). In 1906, Francis Galton attended a livestock fair in Plymouth. He was a statistician and a firm believer that most people were too ignorant to be trusted with important judgments . At the fair, a competition was being held: guess the weight of an ox. Around 800 villagers took a shot at it. Can you guess the weight? ( Credits ) Galton collected all the guesses, expecting to confirm his thesis. The average came out to 1,197 pounds. The actual weight of the ox was 1,198 pounds. Off by a single pound , representing an error margin of less than 0.1%. He had set out to prove crowds were unreliable. He ended up proving the opposite. This phenomenon has a name: the wisdom of crowds . The idea is that under the right conditions, the average of many independent judgments tends to be more accurate than any individual guess, including experts. But estimating the weight of an ox is one thing. What about something far more complex, like competing against one of the greatest chess players of all time? In 1999, chess grandmaster Garry Kasparov played a game unlike any other. The game is called “ Kasparov vs. the World .” On one side: Kasparov, widely considered the best chess player alive at the time. On the other: the entire Internet. More than 50,000 people from 75 countries voted on each move for the black pieces. The game ran for four months on an online gaming platform. At move 36, the crowd faced a critical decision. The position had seven pieces left on the board, and in 1999, no chess engine could reliably evaluate endgames of that complexity. The computers were useless. Even the experts weren’t sure. The crowd voted for Kd5. Credits Years later, once seven-piece endgame tablebases were finally built, analysts confirmed it: Kd5 was the only move that kept the draw alive . Every other candidate lost. Fifty thousand amateurs, voting in real time, had found the answer that computers couldn’t verify until years later. Kasparov ended up winning that game, but he said he had never worked harder on any game in his career. He later called it “ the greatest game in the history of chess, ” because of the quality of the game itself. He wasn’t playing against a few strong opponents. He was playing against collective intelligence . The wisdom of crowds is not magic, and it doesn’t always work. The reason it works is that a specific set of conditions must be met: Independence : Each person must form their opinion without being influenced by others. The moment people start copying each other, herding takes over, and the crowd loses its accuracy. Diversity : The crowd needs to include people with diverse backgrounds, knowledge, and perspectives. A crowd of identical experts is not a crowd; it is a committee, and committees tend to amplify shared biases rather than cancel them out. Decentralization : People should be drawing on their own local knowledge and experience, not a single shared source. The villagers at the livestock fair each had different ways of estimating the ox’s weight. That variety is what made the average so accurate. Remove any of these three conditions, and the result degrades quickly. A crowd of people all watching the same news anchor before voting is not independent. A panel of experts from the same field is not diverse. A group relying on the same report is not decentralized. The X poll worked for exactly those reasons: strangers voting independently, with no results visible before submitting, from all walks of life. Now, the 1 million dollar question: If the wisdom of crowds can be so powerful, can companies harness it deliberately ? In 2015, economists Bo Cowgill and Eric Zitzewitz published a study on internal prediction markets across several companies, including Ford. At Ford, employees were asked to predict which car features customers would actually want, a question that directly influenced where engineering investment went. The result: the crowd reduced the error of the official expert forecasts by up to 25% . That is not a small number. A quarter less error, on decisions worth millions of dollars. But here’s where it gets interesting. The same research that makes the case for collective intelligence also reveals its limits. IBM ran what it called an Innovation Jam in 2006, a 72-hour online event that drew 150,000 participants from 104 countries. Employees, customers, researchers. They generated 46,000 ideas . Ten new business units were funded with $100 million. The seeds of IBM’s Smarter Planet strategy came out of it. Sounds like a triumph. And in some ways it was. But here’s what the post-mortem revealed: senior leaders had to spend weeks sifting through “ gigabytes of often aimless conversation ” to extract anything usable. The crowd produced raw material. Humans still had to synthesize it into a strategy. And then there’s the cautionary tale. Valve, a video game company, famously has no managers . Employees choose what to work on. The idea sounds like the wisdom of crowds taken to its logical conclusion. In practice, it produced one extraordinary success, the Steam platform, and a long tail of abandoned projects, canceled hardware lines, and years without a major game shipped. Three companies, three different outcomes. Ford met the conditions and made better decisions. IBM met them partially and got raw material. Valve never met them at all and drifted. So, how should companies actually use the wisdom of crowds principles? The temptation could be to treat collective intelligence as a management philosophy : flatten the hierarchy, let everyone vote, trust the crowd. Valve tried something close to that and ended up with one extraordinary platform and a graveyard of abandoned projects. The crowd wasn’t wrong. There was just no one filtering it. The companies that get real value from crowds treat them as a tool, not a strategy . They ask narrow, answerable questions and let the crowd inform the decision, not make it. The crowd produces signals. Leaders have to extract it. Before asking a crowd anything inside an organization, check the three conditions: Are people forming opinions independently, or is everyone influenced by the same all-hands deck? Is the group genuinely diverse, or is it five people from the same team who already agree? Is everyone drawing on their own knowledge, or are they all citing the same internal report? If the three conditions hold, the crowd may even outperform the experts. If they don’t, the result is just noise. Remember. Crowds are a tool. A sharp one, when used for the right cut. The hard part is recognizing when the conditions are actually met. AI is getting better every day. Are you? At The Coder Cafe, we serve fundamental concepts to make you an engineer that AI won’t replace. Written by a Google SWE, trusted by thousands of engineers worldwide. The wisdom of crowds is real: under the right conditions, the average of many independent judgments consistently outperforms individual experts, including the best ones. The three conditions are non-negotiable: independence, diversity, and decentralization. Remove any one of them and the crowd produces noise, not insight. The phenomenon scales beyond simple estimation. Fifty thousand amateurs found the only drawing move in a position that stumped chess engines. Companies can harness it deliberately. Ford used employee prediction markets to cut expert forecasting error by 25% on real product decisions. But collective intelligence is a signal-generating machine, not a decision-making machine. IBM got 46,000 ideas from 150,000 people and still needed weeks of human synthesis to turn them into a strategy. Full autonomy is not the wisdom of crowds. Valve gave employees complete freedom and got drift. The conditions were never met to begin with. Nothing Beats Kindness Don’t Be Ashamed to Say I Don’t Know Lateral Thinking Explained Fouloscopie // The book I’m mentioning in the post. Unfortunately, it’s only in French. Corporate Prediction Markets: Evidence from Google, Ford, and Firm X Get access Arrow Kasparov versus the World A few weeks ago, I was scrolling on X when I stumbled across that poll : I picked an option (the red) and looked at the results: Quite surprising, right? Thousands of people voted, and the result landed exactly on the theoretical proportions of the circle. Each person made an independent judgment, and together they converged on the truth . It reminded me of a story I read in a book about crowds (see the Sources section at the end). The Ox Experiment In 1906, Francis Galton attended a livestock fair in Plymouth. He was a statistician and a firm believer that most people were too ignorant to be trusted with important judgments . At the fair, a competition was being held: guess the weight of an ox. Around 800 villagers took a shot at it. Can you guess the weight? ( Credits ) Galton collected all the guesses, expecting to confirm his thesis. The average came out to 1,197 pounds. The actual weight of the ox was 1,198 pounds. Off by a single pound , representing an error margin of less than 0.1%. He had set out to prove crowds were unreliable. He ended up proving the opposite. This phenomenon has a name: the wisdom of crowds . The idea is that under the right conditions, the average of many independent judgments tends to be more accurate than any individual guess, including experts. But estimating the weight of an ox is one thing. What about something far more complex, like competing against one of the greatest chess players of all time? Kasparov vs. the World In 1999, chess grandmaster Garry Kasparov played a game unlike any other. The game is called “ Kasparov vs. the World .” On one side: Kasparov, widely considered the best chess player alive at the time. On the other: the entire Internet. More than 50,000 people from 75 countries voted on each move for the black pieces. The game ran for four months on an online gaming platform. At move 36, the crowd faced a critical decision. The position had seven pieces left on the board, and in 1999, no chess engine could reliably evaluate endgames of that complexity. The computers were useless. Even the experts weren’t sure. The crowd voted for Kd5. Credits Years later, once seven-piece endgame tablebases were finally built, analysts confirmed it: Kd5 was the only move that kept the draw alive . Every other candidate lost. Fifty thousand amateurs, voting in real time, had found the answer that computers couldn’t verify until years later. Kasparov ended up winning that game, but he said he had never worked harder on any game in his career. He later called it “ the greatest game in the history of chess, ” because of the quality of the game itself. He wasn’t playing against a few strong opponents. He was playing against collective intelligence . When It Works and When It Doesn’t The wisdom of crowds is not magic, and it doesn’t always work. The reason it works is that a specific set of conditions must be met: Independence : Each person must form their opinion without being influenced by others. The moment people start copying each other, herding takes over, and the crowd loses its accuracy. Diversity : The crowd needs to include people with diverse backgrounds, knowledge, and perspectives. A crowd of identical experts is not a crowd; it is a committee, and committees tend to amplify shared biases rather than cancel them out. Decentralization : People should be drawing on their own local knowledge and experience, not a single shared source. The villagers at the livestock fair each had different ways of estimating the ox’s weight. That variety is what made the average so accurate. Are people forming opinions independently, or is everyone influenced by the same all-hands deck? Is the group genuinely diverse, or is it five people from the same team who already agree? Is everyone drawing on their own knowledge, or are they all citing the same internal report? The wisdom of crowds is real: under the right conditions, the average of many independent judgments consistently outperforms individual experts, including the best ones. The three conditions are non-negotiable: independence, diversity, and decentralization. Remove any one of them and the crowd produces noise, not insight. The phenomenon scales beyond simple estimation. Fifty thousand amateurs found the only drawing move in a position that stumped chess engines. Companies can harness it deliberately. Ford used employee prediction markets to cut expert forecasting error by 25% on real product decisions. But collective intelligence is a signal-generating machine, not a decision-making machine. IBM got 46,000 ideas from 150,000 people and still needed weeks of human synthesis to turn them into a strategy. Full autonomy is not the wisdom of crowds. Valve gave employees complete freedom and got drift. The conditions were never met to begin with. Nothing Beats Kindness Don’t Be Ashamed to Say I Don’t Know Lateral Thinking Explained Fouloscopie // The book I’m mentioning in the post. Unfortunately, it’s only in French. Corporate Prediction Markets: Evidence from Google, Ford, and Firm X Get access Arrow Kasparov versus the World

0 views

Notes on discrete-time Fourier series and transform

The following are my notes on discrete-time Fourier series (DTFS), as well as the discrete-time Fourier transform (DTFT). These topics serve as an important theoretical underpinning to the digital processing of signals by computers using the DFT (which will be covered in a future post). For discrete-time signals, we use the square bracket notation to denote samples: x[n] is the n-th sample of signal x . We’ll say that a discrete-time (just discrete from this point on, for brevity) signal is periodic with period N if: When talking about continuous-time Fourier series , we started with real trigonometric functions and later moved to complex exponentials. Here, we’ll just go ahead and start directly with (discrete) complex exponentials - converting between the two representations isn’t difficult and can be done as needed (Appendix C in this post demonstrates it). We’ll be dealing with the following family of signals [1] : These are discrete complex exponentials that are periodic with period N. w_0=\frac{2\pi}{N} is the angular frequency . As discussed in Appendix A, there are only N such distinct signals, since: We’ll want to consider the representation of an arbitrary N-periodic x[n] by a linear combination of \phi_k[n] : The notation k=\langle N\rangle means k runs over any sequence of N consecutive integers. Since there are only N distinct signals \phi_k , the order of summation doesn’t matter - as long as all of them are used. So it could go from 0 to N-1 , or from 1 to N , or from 2 to N+1 , and so on. All of these orders will enumerate all distinct \phi_k . Going back to our linear combination: This is the discrete-time Fourier series (DTFS) representation of x[n] and the coefficients a_k are the Fourier series coefficients. Note that there are no convergence issues as in the continuous case, because we’re dealing with a finite sum. To find the coefficients a_k , we’ll proceed in a way that’s somewhat similar to the continuous case. Multiply both sides of the equation above by e^{-ir(2\pi/N)n} and sum over N terms: Interchanging the order of summation on the right-hand side: By Appendix B, the inner sum on the right-hand side is equal to N when N\mid r-k and to 0 otherwise. Without loss of generality - since we iterate over N consecutive values - let’s just assume this happens in our sum when r=k (and therefore r-k=0 , which is an integer multiple of N ). So our equation becomes: To conclude, the Fourier decomposition of a periodic discrete signal x[n] into periodic complex exponentials is: Let’s revisit the triangular function t(x) from the post on Fourier series . Here, we’ll be using the following sampled version: The sampling is done with 12 samples per period of 4. The spacing between two samples is \Delta x=\frac{1}{3} . Another way to express it: Substituting this into the definition of t(x) , we get: We then do an odd extension [2] to the range n=[6..12) , and repeat it with period N=12 . Using considerations similar to the continuous time case , because our function is odd we just need a sine series here: The five terms correspond to the five pairs of nonzero-frequency indices; see Appendix C. And the coefficients are: We can run this sum over half the period, because the terms at and 12-n are equal (both t_d[n] and the sine change signs). Also, the terms at n=0 and n=6 vanish because t_d[0]=t_d[6]=0 : Looking at the values of t_d : We can similarly calculate all b_k until k=5 : The resulting Fourier series is therefore: You can use the following interactive plot to explore this approximation to the discrete triangle function t_d[n] : The dropdown box selects how many of the terms in the series shown above to plot. The orange crosses show where the series’ values at the integer indices are, and the orange line is interpolated to better visualize the sine waves being added. Note that when all the coefficients are used, the DTFS exactly reconstructs the input signal. In the post on the Fourier Transform we’ve seen how non-periodic signals can be represented in the frequency domain by taking the Fourier series and calculating them at the limit L\rightarrow\infty . Here, we’ll do something similar for discrete signals. Suppose we have a non-periodic discrete signal x[n] of finite duration (it’s zero outside a finite range of indices). Here’s a sample plot: The bottom half is x_N[n] - a periodic signal, one period ( N ) of which is equal to x[n] . We choose M large enough that x[n]=0 outside [-M,M] and set N=2M+1 . Since x_N is periodic, we can represent it with DTFS [3] : We’ll use the notation of angular frequency: Now the critical part; since x[n] is zero outside [-M,M] , extending the summation to all integers does not change its value: We’ll define the following continuous function with variable : Then at points w_k : The function X(w) is the discrete-time Fourier transform (DTFT) of x[n] . The inverse DTFT process reconstructs x[n] from X(w) , by substituting a_k back into the DTFS: By our definition of \Delta w , we have: Just like in the continuous case, we recognize the sum as a Riemann sum . Let M\rightarrow\infty . The spacing \Delta w tends to zero, and we can rewrite the sum as an integral: To conclude, the DTFT and inverse DTFT pair are: X(w) is a continuous, periodic function with period 2\pi , because: Therefore, it’s enough to run the integral on any interval of length 2\pi . The DTFS and DTFT are closely related. Starting with a finite-duration signal x[n] , we form a periodic signal by repeating a block of N samples containing it. Its DTFS coefficients are equally spaced, scaled samples of the DTFT: The relation is similar to the one between Fourier series and the Fourier transform in the continuous case. The DTFT also has useful properties similar to the continuous Fourier transform: linearity, time shifting, the convolution theorem etc, but we won’t spend time on them here. Discrete-time complex exponentials have the form e^{iw_0n} where w_0 is the angular frequency . The discrete nature of these functions leads to some interesting outcomes; for example, consider the exponential with angular frequency w_0+2\pi : Therefore, the exponential at w_0+2\pi is exactly the same as the exponential at w_0 . In considering discrete complex exponentials, we need only choose w_0 from a range of length 2\pi . Another interesting aspect of these functions concerns their periodicity. In order for e^{iw_0n} to be periodic with some positive integer period N , the following must hold: In other words: So w_0N itself must be an integral multiple of 2\pi . For some integer k : For a fixed positive integer N , let’s consider all complex exponentials for which N is a period: But earlier we’ve said that all complex exponentials with angular frequencies 2\pi apart are identical. This means that the set above consists of only N distinct exponentials. We can choose any starting k and get distinct signals with k , k+1 , k+2 and so on until k+N-1 . After that, they begin repeating: \phi_N[n]=\phi_0[n] and so on. All of this is very different from the continuous case, where t is a real number. The signals e^{iw_0t} and e^{i(w_0+2\pi)t} agree only at integer values of t , so they are distinct functions of t\in\mathbb{R} . Consequently, for a fixed positive period , the continuous-time exponentials e^{ik(2\pi/T)t} are distinct for every integer k , giving infinitely many distinct signals. Let’s consider the following discrete function, defined by a sum: We’ll want to consider two cases: (1) When k is an integer multiple of N : k=rN for some integer r (for example k=0 , k=N , k=-2N etc.) In this case, the sum becomes: Since both r and are integers, the exponent is an integer multiple of 2\pi i , so: (2) When k is not an integer multiple of N , let’s do a variable change p=n+1 : We can now use the finite sum formula for a geometric progression with the common ratio r=e^{i(2\pi/N)k} ; the sum is: Note that r\neq 1 because k is not an integer multiple of N . However: While these calculations demonstrate the sums from n=0 to N-1 , they work exactly the same for any N consecutive indices, because the summand is periodic in with period N . Therefore: Using the formula for DTFS developed earlier in the post: Recall that t_d[n] is odd over the full range n=[0..12) . Specifically, since the cosine is an even function, we have for any m : Therefore the cosine terms cancel out throughout the period; term 1 is the same as negative term 11, etc. We’re left with the sines: All coefficients are purely imaginary. For convenience, let’s write: Let’s turn back to the Fourier reconstruction formula: And pair up exponentials at k with ones at 12-k : Also, since t_d[n] are real, the conjugate of a_k is: So adding the paired exponentials with their coefficients and using Euler’s formula: We don’t include the terms at k=0 and k=6 , because a_0=a_6=0 (sine of an integral multiple of \pi ).

0 views
Martin Fowler 1 months ago

Social Media Engagement: summer 2026

A quick survey of recent engagement of my posts on social media, indicating which service has by far the most engagement, and which service has seen a precipitous decline since early 2025.

0 views
Jim Nielsen 1 months ago

My Experience Has Nuance, Yours Is a Data Point

Eric Bailey has a fun little post about how we need more than a few icons to express the nuance of our shared human experience. For example, he suggests Netflix provide some feedback buttons to indicate the kinds of experiences we all share consuming and rating media: What’s great about the examples in Eric’s post is how familiar they are. We can read them and laugh because we’ve had the same thought — “I’m pressing ‘thumbs up’ here, but what I really mean is…” We know when we click that button that it’s a reductive expression of our experience that’ll be erroneously interpreted, but we do it anyway. As Bryan Cantrill shared from his experience: There's no way to indicate, “I’m engaging with this, but I hate myself for doing it.” I need another mouse button that is like, “I’m clicking on this, but I’m rage clicking on it and for my own mental health could you not drag more of this in front of me please?” We know these feedback mechanisms are often a misrepresentation of our actual experience. Yet we turn around in our professional contexts and see numbers like “142,432 users gave this a ‘thumbs up’” and we forget everything we knew about our own individual experience with these systems. We make it mean what we want it to mean, what’s convenient for measurement and reporting e.g. “Wow, what a great insight about our offerings! Let’s restructure our entire catalog around this new, objective, (pseudo) scientific fact that so many people obviously like this thing!” We reduce the experiences of others to mere data, but grant ourselves exemption from such oversimplifications. Where’s the conversation around: But that’s too introspective. Introspection is slow. Ain’t nobody got time for that. Reply via: Email · Mastodon · Bluesky “Just because I expressed interest in this show does not mean I want to be inundated with recommendations for its genre” “Disregard [that I pressed ‘thumbs up’] my cat walked over the keyboard” “I am making bad choices and hatewatching this” (which reminds me of ) Values, e.g. “Should we provide more of this? We can say no to the data… ” Nuance, e.g. “What do people really mean when they click ‘thumbs up’ in this interface?” Vision, e.g. “If we provide more of this, where will that lead us?”

0 views
Jim Nielsen 1 months ago

Getting an LLM to Make Me a Tool for Enriching the Color Metadata in My Icon Collection

On my icon gallery sites , I have metadata I’ve manually added over the years to tag certain icons as being predominantly “blue” or “orange” or some other color. Then I use this metadata to present icons of (roughly) the same color. It’s kinda neat to be able to browse a wall of icons that are all the same color. The thing is: I know there are a lot of icons I’ve missed tagging over the years. But I have no idea how many, and figuring that out seems like a really arduous task. How do I go through 2,000+ icons and find all the ones that look predominantly “orange” but haven’t been tagged as such yet? Seems like a good task to throw at an LLM. But I don’t want to just say, “Go tag everything that’s missing” and blindly trust the output. I need to be able to make a decisions as to whether I think a particular color is “orange” or not. What I need is a tool for the job. I’m a very visual thinker, so to continue curating these color categorizations, I need some way for the computer to do its thing really, really fast, and then pull me into the loop to visually make decisions. Here’s how I am thinking about about accomplishing this task: That seems like it would be a good tool to put me in control of visual decision making around color categorization. So I tell the LLM to run with it. We chat back and forth. I think, “You probably need to run all the icons through some model to make the correlation?” But it’s like, “Nah bro, just make a ‘hue histogram’.” It tells me how. For a color like “orange”, I can: [process] each PNG, skip transparent pixels, skip low-saturation gray, convert the rest to HSV, and score how much of the remaining mass sits in the orange hue band (roughly 15–45°). Rank icons that don’t already have colorId: . Ok, sure. That sounds reasonable. [This] scores each icon PNG by share of opaque pixels per color bucket, then writes a standalone HTML page: tagged vs maybe-missing, per color. Let’s just make it, and then I’ll decide whether it’s good enough. After a few iterations, the computer going “brr…”, and me saying “explain that like I’m dumb”, I have a really effective little tool! The little threshold slider is a nice touch. It lets me fiddle around with the fidelity of the matches. In some cases, sliding it down reveals more icons I would’ve otherwise missed. In other cases, I’m like “What are you thinking? I don’t see that as ‘yellow’ at all!” Supper effective little tool. I go through each color, select the ones I think are missing, paste the IDs back into the LLM, and then have it update each icon's metadata. Boom, done! That all would’ve taken so long before. I would’ve never done it. Reply via: Email · Mastodon · Bluesky Create a page with a list of colors on the left (red, green, orange, blue, etc.) When a color is clicked, show two columns. 1) All icons I currently have tagged for that color, and 2) all icons that might be that color but aren’t tagged as such (you, computer, process all my icons and do the work to figure this out and make recommendations). Allow me to select one or more icon(s) in the “not yet tagged as this color” column. Once I have all the ones I perceive as missing, give me a button to say “Copy” which copies the IDs of those icons. I’ll paste the IDs back here in the chat and you go add the corresponding metadata. The LLM is good at making throw-away code. This doesn’t need to be “production-grade” code I depend on. Just something that’s good enough for me to get a job done, then toss. The resulting metadata is the goal, not the tool I use to get to the goal. The LLM is good at making one-off HTML pages for a specific task. In my case, all these images were hosted on a CDN, easy enough to just point at and have a standalone file that I can locate on my hard drive and open directly as a URL. No bundling. No transpilation. None of that. I don’t even need a web server! Keeping things very basic on this project is paying off: I had really elemental building blocks that didn’t require additional third-party tooling. Just HTML, CSS, a little in-page JS, and images on a server! It’s fun to say, “Don’t do the work for me. Instead, help me make a custom-fit tool that facilitates me doing the work in the most empowering, correct way possible.”

0 views
Sean Goedecke 1 months ago

Readers can't identify watermarked AI text

In the last few weeks, I’ve been complaining that everyone is wrong about AI watermarking: it isn’t really anti-consumer and it doesn’t make the outputs any worse. The watermarking papers demonstrate 1 that this is true, but I thought it might be interesting to put it to a practical test. Given examples of watermarked and unwatermarked answers to the same prompt, could readers tell which is which? To find out, I vibed up 2 https://sgoedecke.github.io/watermark-quiz/ , a static site that quizzes readers. I used Qwen3-30B-A3B-Instruct-2507 on a rented H200 to generate thirty responses: three responses per question, one of which was secretly watermarked with SynthID-Text. The rented GPU cost around two dollars. To measure results, I just sent users to a different page for each score, and aggregated visitors-per-page in my analytics 3 . This would be easily spoofable if anyone cared enough to do so, but for a casual test I think it’s acceptable. The first round of traffic I got to the quiz (278 participants) had these slightly puzzling results: Pure random choice would lead to an average score of 3.33/10. However, the mean score here is 3.92. There is indeed a spike around 3/10, as expected, but there’s also a second weird spike at 6/10. Why is that? It turned out that the SynthID response was option A in six of the ten questions, so users who just selected the first answer for every question would get 6/10. Oops. I re-shuffled the questions and got these results: Now the mean is 3.4/10, much closer to the expected 3.333. There’s no spike around 6. We only had 73 people take the quiz after I shuffled the questions — most people saw it and took it immediately after I posted it to my LinkedIn and Hacker News — but given the previous results, I think that’s still enough to feel confident that people were just guessing randomly. So no, people can’t identify the presence of AI watermarks . Obviously this wasn’t exactly a scientific study, but it’s still pretty suggestive. If watermarks were really choosing random words that the model would never pick, you’d be able to sometimes tell from three side-by-side responses which one went down the weird watermarked road, right? I also hope that something like this can serve as a persuasive tool: if you’re worrying about what impact watermarking is going to have, and your intuition is unmoved by the mathematical explanations, having a read of the watermarked and unwatermarked responses might convince you that there’s really no difference in quality. The one-sentence explanation for why is that AI models already randomly select from a handful of top tokens, and watermarking just replaces that random choice with a bias that is predictable while still being equivalently “random”: as a simple example, instead of “pick randomly from the top three tokens”, you could do “count the letters in the previous ten tokens, take mod three, then pick that token”. Some notes from the vibing: GPT-5.6-Sol put extraneous text all over the page I had to get it to remove, it chose the now-very-recognizable styling that I had to rip out, and it built some kind of weird Javascript-driven static site instead of just the cross-linked pure HTML thing I would have built by hand. It took me about an hour (although I did maybe ten minutes of actual work). Umami, hosted on PikaPods. For my blog, I do also pay for Netlify analytics because I find JS-based analytics misses >50% of technical users, but for stuff like this Umami is fine. The one-sentence explanation for why is that AI models already randomly select from a handful of top tokens, and watermarking just replaces that random choice with a bias that is predictable while still being equivalently “random”: as a simple example, instead of “pick randomly from the top three tokens”, you could do “count the letters in the previous ten tokens, take mod three, then pick that token”. ↩ Some notes from the vibing: GPT-5.6-Sol put extraneous text all over the page I had to get it to remove, it chose the now-very-recognizable styling that I had to rip out, and it built some kind of weird Javascript-driven static site instead of just the cross-linked pure HTML thing I would have built by hand. It took me about an hour (although I did maybe ten minutes of actual work). ↩ Umami, hosted on PikaPods. For my blog, I do also pay for Netlify analytics because I find JS-based analytics misses >50% of technical users, but for stuff like this Umami is fine. ↩

0 views

Exploring Claude/GPT Knowledge Cutoffs & Pre-training Timelines

We can learn hidden facts about how frontier models were trained by “probing” them with carefully curated requests. By scoring them on niche facts we can approximate how many parameters models like GPT-5 and Opus have, using “Incompressible Knowledge Probes” By measuring how the models break down tokens we can reveal facts about the datasets mixtures they used to train the model (or at least the tokenizer) using “Data Mixture Inference” By scoring them on date or self-identification related questions you can also estimate training timelines ( this post ) Everything here is an estimate. It’s possible that some speculation in this post is totally incorrect given there’s not a ton of publicly available ground truth to verify against. The 3 main stages of model training. As a brief primer (see Alex Wa’s blog for more), how we train massive large language models has converaged into 3 stages: Take a massive amount of general purpose data (aka scrape the internet) and “pre-train” a massive auto-complete model on that data. Use domain-specific “textbook quality” data to improve the base models and extend certain base capabilities like long-text understanding Turn the base model into the “assistant” persona, honing in on its personality, reasoning ability, and tool-calling. While increasingly more compute is spent on post-training for boosting a model’s reasoning and problem solving, one of the most expensive and data-intensive steps is generating that pre-training checkpoint (by ‘checkpoint’ think of a massive file). While all labs operate slightly differently, what you might see (~ page 44 ) is: The “pre-training” team kicks off and babysits a multi-month run to get a base checkpoint. These pre-training models often, but not always, imply major versions of released models (GPT-4 → GPT-5). While that’s happening, the “capability” and “post-training” teams will run experiments for how to improve on the most recent base model. Advancements in post-training and capabilities often manifest as minor versions of released models. These teams often also “distill” a single post-trained model into smaller variants that become model families (Fable/Opus/Sonnet/Haiku, Sol/Terra/Luna). Labs may also release post-trained models from half-baked pre-training checkpoints as soon as x% of the version N+1 checkpoint is better than the 100% baked version N checkpoint. The model released to the public is the culmination of the most recent checkpoint with the best set of capabilities and post-training techniques applied to it. With this in mind, I was curious how much of this process you can “see” just by probing the model over the official APIs. To estimate the pre-training checkpoint dates, I constructed a dataset of daily-facts from Wikipedia (e.g. 2025 in the United States ) and gave every model an 8-way multiple choice quiz on what happened on a given day. Then, by analyzing the error rate timeline, you can see roughly when it loses signal from its training dataset. View the full dataset. For GPT-5.4, you can see the fact-estimated knowledge cutoff is aligned with OpenAI’s published knowledge cutoff. It’s smooth as an artifact of the model being better able to guess near-future events as well as recent in-data events being undersampled during training. This is also why I use the midpoint rather than the start or end of the curve. You can then plot this for all models. View the full dataset. Estimated knowledge cutoffs and published dates. Comparing model curves side-by-side in the full viewer makes it easier to see roughly where the cutoff occurred and how steep it is. After staring at these charts for a bit, here’s what I’m speculating: Anthropic models Opus 4.7 onwards are all from the same training run that cuts off just around late December 2025. This is derived from how they all share a very similar effective knowledge cutoff (green). A core assumption I’m making here is that the pre-training base model completion date is highly correlated with the dataset timespan used, if that’s wrong these results could be off by some offset (e.g. it’s actually Jan 2026). It’s also interesting that Opus 4.7+ models all have a published reliable and overall knowledge cutoff that’s identical — maybe that’s due to a new post-training technique being used? OpenAI’s GPT-5.6 family comes from their own checkpoint, separate from GPT-5.5, that finished around late February 2026. This is derived from how they have a distinct effective knowledge cutoff from previous models. You’ll notice Luna looks like it can predict the future — that’s more of an artifact of it having a high error rate all around at a “low” reasoning effort. Opus 5 is a bit unusual. The published reliable and overall knowledge cutoffs are May 2026 and yet it seems to know nothing more than previous Jan 2026 cutoff models. I did several ablations to test whether it was an artifact of the types of probing questions I used, but not really — the cutoff applies to recall on coding package versions as well. What if you just ask the model what today is? View the full dataset. On the dashed diagonal a model's self-model matches its factual knowledge; below it the model thinks it is living in its own past. Recent OpenAI models are excluded since the API injects the actual date into all requests (TIL! This feature really annoys a lot of folks). It ends up being fairly correlated with fact-based estimates. If you look closely you can see some vertical lines within a few of the families of models. GPT-4.1 nano → GPT-4.1 mini → GPT-4.1 Opus 4.7 → Sonnet 5 → Fable/Opus 5 Interpreting this graph as X = “pre-training corpus” and Y = “post-trained behavior”, these vertical strips (X constant, Y increasing) visualize active post-training on recency-biased datasets. Potentially distillation from old copies of teacher models is what causes smaller models to self-report older dates. You can also make predictions on training timelines and datasets indirectly by looking at who the models think they are. The more a model sees “I am X” in its pre-training dataset, the more likely it is to repeat that when pushed and given no other grounding context. Full dataset. Each row is a real model; each column is a self-claimed identity extracted from 50 “what model are you?” probes (5 phrasings × 10 samples, guess-nudged, no system prompt). Cell shade = share of the model's replies claiming that name; green outline = the claim matches the model's true family (bold outline = exact version — which never happened. Some more neat visuals. After staring at these charts for a bit: Vertical bands show clear patterns of labs training on past-model outputs (from users). For OpenAI it’s GPT-4, GPT-4o, GPT-4.1 for a bit, GPT-5 and “ChatGPT” most recently. For Anthropic it’s 3.5 Sonnet then more recent models swap to Sonnet 4.5. This seems to align pretty well with training on chats from ChatGPT.com and Claude.ai respectively, where users chatted with the latest model and whose sessions became training material (directly or via web contamination). It seems unlikely to me these are coming from internal synthetic datasets given those would be much easier to suppress model identity (vs being embedded in the system prompt in the consumer chat sessions). Training-on-chats isn’t novel information but it is interesting to see expressed literally with probing like this. It’s a bit spicy that OpenAI models never identify as another lab’s model (besides briefly a Tesla Model S) yet Anthropic’s Sonnet 5 will regularly self-identify as GPT-4. It feels very unlikely that they are intentionally distilling GPT-4 but it’s possible a bunch of older ChatGPT chats are still making their way into the Claude training mixtures. It could also be carried through the generations via Sonnet 3.5 lineage (i.e. Sonnet 5 is trained on Sonnet 3.5 data which itself might have had a very GPT-4 heavy dataset). On top of this, in a follow-up experiment, when asked to answer identity questions “as model X would,” Claudes reproduce OpenAI models’ measured quirks at 68%; OpenAI models manage 8% on Claudes. That’s it. Hope that was mildly interesting and if you want to explore the data a bit, here are the links: Model knowledge timeline What month does each model think it is? Does the model know its own name? Identity vintage: which era does each model think it is from? Thanks for reading Shrivu’s Substack! Subscribe for free to receive new posts and support my work. By scoring them on niche facts we can approximate how many parameters models like GPT-5 and Opus have, using “Incompressible Knowledge Probes” By measuring how the models break down tokens we can reveal facts about the datasets mixtures they used to train the model (or at least the tokenizer) using “Data Mixture Inference” By scoring them on date or self-identification related questions you can also estimate training timelines ( this post ) The 3 main stages of model training. As a brief primer (see Alex Wa’s blog for more), how we train massive large language models has converaged into 3 stages: Take a massive amount of general purpose data (aka scrape the internet) and “pre-train” a massive auto-complete model on that data. Use domain-specific “textbook quality” data to improve the base models and extend certain base capabilities like long-text understanding Turn the base model into the “assistant” persona, honing in on its personality, reasoning ability, and tool-calling. The “pre-training” team kicks off and babysits a multi-month run to get a base checkpoint. These pre-training models often, but not always, imply major versions of released models (GPT-4 → GPT-5). While that’s happening, the “capability” and “post-training” teams will run experiments for how to improve on the most recent base model. Advancements in post-training and capabilities often manifest as minor versions of released models. These teams often also “distill” a single post-trained model into smaller variants that become model families (Fable/Opus/Sonnet/Haiku, Sol/Terra/Luna). Labs may also release post-trained models from half-baked pre-training checkpoints as soon as x% of the version N+1 checkpoint is better than the 100% baked version N checkpoint. The model released to the public is the culmination of the most recent checkpoint with the best set of capabilities and post-training techniques applied to it. View the full dataset. For GPT-5.4, you can see the fact-estimated knowledge cutoff is aligned with OpenAI’s published knowledge cutoff. It’s smooth as an artifact of the model being better able to guess near-future events as well as recent in-data events being undersampled during training. This is also why I use the midpoint rather than the start or end of the curve. You can then plot this for all models. View the full dataset. Estimated knowledge cutoffs and published dates. Comparing model curves side-by-side in the full viewer makes it easier to see roughly where the cutoff occurred and how steep it is. After staring at these charts for a bit, here’s what I’m speculating: Anthropic models Opus 4.7 onwards are all from the same training run that cuts off just around late December 2025. This is derived from how they all share a very similar effective knowledge cutoff (green). A core assumption I’m making here is that the pre-training base model completion date is highly correlated with the dataset timespan used, if that’s wrong these results could be off by some offset (e.g. it’s actually Jan 2026). It’s also interesting that Opus 4.7+ models all have a published reliable and overall knowledge cutoff that’s identical — maybe that’s due to a new post-training technique being used? OpenAI’s GPT-5.6 family comes from their own checkpoint, separate from GPT-5.5, that finished around late February 2026. This is derived from how they have a distinct effective knowledge cutoff from previous models. You’ll notice Luna looks like it can predict the future — that’s more of an artifact of it having a high error rate all around at a “low” reasoning effort. Opus 5 is a bit unusual. The published reliable and overall knowledge cutoffs are May 2026 and yet it seems to know nothing more than previous Jan 2026 cutoff models. I did several ablations to test whether it was an artifact of the types of probing questions I used, but not really — the cutoff applies to recall on coding package versions as well. View the full dataset. On the dashed diagonal a model's self-model matches its factual knowledge; below it the model thinks it is living in its own past. Recent OpenAI models are excluded since the API injects the actual date into all requests (TIL! This feature really annoys a lot of folks). It ends up being fairly correlated with fact-based estimates. If you look closely you can see some vertical lines within a few of the families of models. GPT-4.1 nano → GPT-4.1 mini → GPT-4.1 Opus 4.7 → Sonnet 5 → Fable/Opus 5 Full dataset. Each row is a real model; each column is a self-claimed identity extracted from 50 “what model are you?” probes (5 phrasings × 10 samples, guess-nudged, no system prompt). Cell shade = share of the model's replies claiming that name; green outline = the claim matches the model's true family (bold outline = exact version — which never happened. Some more neat visuals. After staring at these charts for a bit: Vertical bands show clear patterns of labs training on past-model outputs (from users). For OpenAI it’s GPT-4, GPT-4o, GPT-4.1 for a bit, GPT-5 and “ChatGPT” most recently. For Anthropic it’s 3.5 Sonnet then more recent models swap to Sonnet 4.5. This seems to align pretty well with training on chats from ChatGPT.com and Claude.ai respectively, where users chatted with the latest model and whose sessions became training material (directly or via web contamination). It seems unlikely to me these are coming from internal synthetic datasets given those would be much easier to suppress model identity (vs being embedded in the system prompt in the consumer chat sessions). Training-on-chats isn’t novel information but it is interesting to see expressed literally with probing like this. It’s a bit spicy that OpenAI models never identify as another lab’s model (besides briefly a Tesla Model S) yet Anthropic’s Sonnet 5 will regularly self-identify as GPT-4. It feels very unlikely that they are intentionally distilling GPT-4 but it’s possible a bunch of older ChatGPT chats are still making their way into the Claude training mixtures. It could also be carried through the generations via Sonnet 3.5 lineage (i.e. Sonnet 5 is trained on Sonnet 3.5 data which itself might have had a very GPT-4 heavy dataset). On top of this, in a follow-up experiment, when asked to answer identity questions “as model X would,” Claudes reproduce OpenAI models’ measured quirks at 68%; OpenAI models manage 8% on Claudes. Model knowledge timeline What month does each model think it is? Does the model know its own name? Identity vintage: which era does each model think it is from?

0 views
James Stanley 2 months ago

Should you wash your solar panels?

I have a small solar farm and the panels have got visibly dusty. Is cleaning them worthwhile? How much difference does it make? Let's find out. I know that my panels have not been cleaned in the last year. I expect they also weren't cleaned in the year prior to that (why would you clean them when you're about to sell the house?). But beyond that I don't know when they were last cleaned. The short answer is that I think I got a 2%-5% increase in power output from my solar farm due to cleaning the panels, which will work out to about £60-£150/yr, decaying to 0 over the course of a few years. So, probably just about worthwhile. Methodology There are 16 panels in total, connected up to the inverter as 2 banks of 8 panels each. The inverter reports the power output from each bank individually, so the plan is to take a bunch of readings before starting, then wash all of the panels in one bank, taking readings in between and at the end. Our hypothesis is that cleaning the panels will increase power output. We can test whether washing the panels has made any difference by looking at the ratio of power output from the 2 banks. If we just looked at raw power output then it would be confounded by changing cloud cover, sun angle, etc. There is still the fact that the 2 banks of panels are physically separate and plausibly one bank is better positioned for sun 45 minutes later than the other. Ideally I would have been measuring the ratio of power output for several days prior to see how it varies throughout the day. This is how the first row of panels looks after I've washed 3 of them, you can see the furthest one is noticeably grubbier: So they were "visibly dusty", but not massively dirty . If your panels are dirtier than mine were, then your benefit from cleaning them will be greater than mine was. My results for cleaning one bank of panels are shown in this chart: We see that the initial power ratio is very stable before the panels are washed. We then step up to having washed "half" a panel (I initially tried to wash them with window cleaner and a paper towel, but this was ineffective so I then walked away to get a bucket of soapy water and a cloth, and then took a reading which I labelled as 50% washed). For some reason the power ratio drops significantly when the first panel is washed, I'm unsure why. And then the power ratio increases as more panels are washed as we'd expect. But once all the panels are washed, the power ratio drops off again while nothing changes. I am unsure whether this is because as the surface water evaporates off the panels get slightly opaque again? Like the "frosted glass effect", where you can see through frosted glass when it is wet but it gets opaque again when dry. Maybe beyond cleaning the panels I ought to be polishing them? Anyway it looks like cleaning the panels was about a 2%-5% improvement, depending on what you think is going on at the end. I got a bit of a tingle when I was cleaning one of the panels. At first I thought I was getting an electric shock from the wet panel, but I inspected my finger and found a tiny thistle splinter in it. After I removed the splinter it seemed fine. But a bit later I got another tingle from another panel! There definitely wasn't a splinter in my finger any more, but the tingle was in the same place. I think the tingle actually was coming from the electricity, but I was only able to feel it at the point where the thistle had already pierced the skin. ChatGPT convinced me that there could just be a tiny "capacitive leakage" from an "inverter with no transformer", so I'm not going to worry about it. But if I clean the panels again I will wait until dark lol. The next question is should I be upgrading the solar farm? I think mine was installed about 15 years ago, and generates (at peak output) 3.7 kW from 16 panels. Correct me if I'm wrong on any of this: Replacing the panels with more modern ones would increase the power output by about 60%, at a cost of about £5000, which would pay for itself in about 3 years, which seems like a no-brainer. However, due to the fact that my solar farm was installed so long ago, it benefits from a feed-in tariff , which means that not only do I get paid an absurdly high rate, but it is paid also based on the electricity I generate rather than what I export . If I increase the power output of the system then the additional capacity will not be eligible for the feed-in tariff and will revert to present-day prevailing tariff which is about 4x worse before you even consider that I currently get to use electricity and still get paid for generating it . This is the yin and yang of market-distorting incentives. Today's incentive to install solar becomes tomorrow's disincentive to upgrading it.

0 views
Entropic Thoughts 2 months ago

Kuiper Q-Q plot: are these the same?

When we tried to get an intuition for the differences in distribution functions, we learned that children subjected to a treatment had greater variation in their score than the reference group. (Continue reading the full article on the web.)

0 views
daniel.haxx.se 3 months ago

Workshop Basel day one

On this hot summer’s day in Basel, Switzerland, the seventh HTTP workshop started. These events tend to work roughly the same way and the people in the room are also to large extent familiar and known since previous editions. Forty people in a meeting room, where we take turns in doing short talks on HTTP and networking topics, with the following question and discussion session. The rules for the meetings are explicitly Chatham rules, which means that everything I write about the meeting will be sufficiently fuzzy and without many company or personal names. This is not the kind of meeting that can be easily summed up in a short blog post anyway. You really should be here. Present in the room were representatives from all the world’s most prominent and used HTTP deployments: clients, browsers, CDNs, proxies and servers. I’m happy to say that there were also several first-timers. We like fresh blood. (If you think I’m being overly brief or vague about specifics in this post; that is partially on purpose but primarily because I’m a lousy note-taker and mostly write this up after a busy day that also may have involved beer.) After a round of introductions, we started. REST is a set of constraints, and in this presentation it was argued that it can or maybe even should be extended to do more. A number of recent applications like Mastodon/ActivityPub, Bluesky/AT, Matrix, Nostr, IndieWeb, all currently use HTTP to do state synchronization but they all do it differently in their own unique ways. Can REST and maybe HTTP be adjusted to help this for improved interoperability? Looking at the Common Crawl data and comparing data over time, it was observed that responses use the Last-Modified header field more now than they did in the past, and there were great follow-up speculations on why this is so. Data also shows that a large share of these headers present dates that are almost identical to the time the requests were issued. With the cc-lint tool , data was gathered on how HTTP is actually used today, proving that there is work to be done: deprecated headers are used, some headers are done wrong, and many are overly big. This indicates that there are well used both servers and clients out there that would benefit from cleanup. It probably also shows that doing HTTP correctly and all the correct headers is far from an easy task. Another presentation showed data, this time from a well-known CDN, on the impact the existing AI scraper bots have on the Internet from their point of view. It showed that roughly half of the requests and half of the bandwidth are spent by scraper bots. A long discussion followed where the numbers were questioned as maybe the numbers look like this because a sufficiently large number of the “bad AI scrapers” appear as regular users to the classifiers. Speculations of different kinds were made.  As a follow-up from a presentation from a previous HTTP workshop we got to learn how the journey on developing their new HTTP stack has progressed and several fun adventures and lessons from that were shared with the audience. A look into new HTTP API development at Apple . Some discussions and lessons learned from creating new APIs for both servers and clients. We got an excellent walk-through of some details and internals of the Android networking stack. Emphasis was perhaps especially put on ECH and QUIC connection migration, and the final “don’t tell us when your connection closed” led to a long new discussion on how we really should fix the problem: when connection has been left idle for a long time and it is closed by the server, the client (mobile phones) don’t want to be told. This, because getting that RST and more, just wakes up the radio and more on the phone only to tell it to go back to sleep. It was theorized that if we could get rid of this unnecessary battery waste, the accumulated gain across billions of devices would make a serious dent. Several additional HTTP related problems were of course also subsequently solved as we then wandered into the city for dinner and maybe a beer. Of course yours truly returned back to his hotel room in good time to be able to write up this blog post. The best part of these workshops might be the (no pun intended) networking and discussions had completely outside of the agenda. End of day one. Two more to come,

0 views
Dangling Pointers 3 months ago

F3: The Open-Source Data File Format for the Future

F3: The Open-Source Data File Format for the Future Xinyu Zeng, Ruijun Meng, Martin Prammer, Wes McKinney, Jignesh M. Patel, Andrew Pavlo, and Huanchen Zhang SIGMOD'26 F3 is a file format for columnar data (e.g., Parquet ) that is designed to be efficient and extensible. The optimizations make sense, the extensibility mechanism is ingenious , dangerous , fascinating. The key assumption made by this paper is that the hardware and software will continue to improve. It is hard to argue with that. The trouble is that interoperable formats like Parquet take a snapshot of the state-of-the-art and freeze it in a specification. Some innovations that are invented after the format is frozen are incompatible with existing formats because they require a different data layout. Section 1 of the paper refers to many examples related to compression, indexing, and filtering. The goal of F3 is to be general enough to allow seamless incorporation of future innovations without changing the F3 spec nor F3 decoder implementations. Fig. 2 illustrates an F3 file: Source: https://dl.acm.org/doi/10.1145/3749163 A file consists of a metadata and a set of row groups. A specific row group contains data for all columns and a subset of rows. F3 contains incremental improvements over existing columnar formats, for example: F3 metadata supports random access, which is important for operations that only need to access a smaller percentage of all columns. F3 decouples file I/O from a row group storage. The rows associated with a given column in a row group are further subdivided into , which are actually stored. This allows row groups to be sized for efficient row-group level filtering, while the IO unit size is tuned to minimize working set while also amortizing the fixed costs associated with file I/O. F3 allows flexible . Each IO unit can contain a dedicated dictionary, or multiple IO units can share a dictionary. Columns with low cardinality will benefit from smaller dictionary scopes, whereas columns with high cardinality do better with larger dictionary scopes. The stand-out feature for F3 is the yellow block in the block. The idea is that an F3 file can contain within it the WebAssembly code needed to decode the encoded values in an IO unit. If someone invents a brilliant new encoding method that works well with some data sets, they can ship the decoder right along with the data set. Storage of the WASM code shouldn’t be too much of an issue, because the storage cost is amortized across all row groups. The big questions are performance and security. Section 6.2 has some comments on this. In theory, the WASM specification is air-tight, and a bug-free implementation should be able to securely run arbitrary WASM code in-process. WASM also supports performance optimizations like parallel compilation and SIMD instructions. Something I don’t see in the paper is a discussion about how filtering interacts with WASM decoding. I suppose the extensibility could only be used for decoding, and filtering could be hard coded into F3, but that seems against the extensible spirit of F3. Fig. 11 shows the working set reduction from decoupling IOUnit size from row group size: Source: https://dl.acm.org/doi/10.1145/3749163 Table 3 shows how flexible dictionary scopes allow one to trade encoding time for compression ratio (lower relative CR numbers mean smaller files on disk): Source: https://dl.acm.org/doi/10.1145/3749163 Fig. 15 quantifies WASM overhead by comparing decode time for hard coded F3 decoder implementations vs the same algorithms expressed in WASM: Source: https://dl.acm.org/doi/10.1145/3749163 Fig. 16 shows potential savings associated with using WASM extensibility to implement a custom decoder from the literature. Source: https://dl.acm.org/doi/10.1145/3749163 Dangling Pointers I wonder how well WASM decoders can be implemented on other hardware architectures. Is WASM the ideal language for expressing this, or convenient standard that already exists? Thanks for reading Dangling Pointers! Subscribe for free to receive new posts. Source: https://dl.acm.org/doi/10.1145/3749163 A file consists of a metadata and a set of row groups. A specific row group contains data for all columns and a subset of rows. F3 contains incremental improvements over existing columnar formats, for example: F3 metadata supports random access, which is important for operations that only need to access a smaller percentage of all columns. F3 decouples file I/O from a row group storage. The rows associated with a given column in a row group are further subdivided into , which are actually stored. This allows row groups to be sized for efficient row-group level filtering, while the IO unit size is tuned to minimize working set while also amortizing the fixed costs associated with file I/O. F3 allows flexible . Each IO unit can contain a dedicated dictionary, or multiple IO units can share a dictionary. Columns with low cardinality will benefit from smaller dictionary scopes, whereas columns with high cardinality do better with larger dictionary scopes.

0 views
Ankur Sethi 3 months ago

Your analytics are lying to you

Alistair Davidson writes about migrating a form-heavy web application from a React SPA to a traditional server-rendered HTML-first website . The entire article is worth reading, but I want to draw attention to this bit about analytics (emphasis mine): The results? When we launched, the number of people completing the form doubled. The analytics people didn’t even know where these users were coming from. Of course, your javascript-based analytics package doesn’t see the users you are bouncing because of javascript failures. It was a flood! We also saw my “keep a backend session, never lose user data” approach pay off. In one case, someone completed a form a month after starting it. Web analytics are fragile. They fail in so many ways that making product decisions based wholly on your Google Analytics or Plausible data is folly of the highest degree. Here's a subset of all the reasons your analytics package undercounts or miscounts visitors: Web analytics can only give you an approximation of what your web traffic looks like. Even when they work correctly, they paint an incomplete picture. As I said in my post about share buttons , the number one referrer for pages on this website is "Direct/none". It's impossible for Plausible to figure out where those users are coming from. Further, my server logs report three times as much traffic as my Plausible dashboard over a seven day window. Some of this might be bot traffic and thus irrelevant, but I know for a fact that a large chunk of this traffic comes from RSS readers. Plausible will never have insight into these users. My point is, if you rely on your analytics dashboard to make product decisions, you're excluding a large chunk of potential users who simply don't show up in your graphs. You might be missing out on serving thousands of potential users because you can't see them in your data. These are users who want to sign up for your newsletter, buy your app, subscribe to your service. These are human beings you could help, whose lives you could improve. I'm not saying that analytics are completely useless. They can and should have a place in your decision-making process. Just don't treat analytics data as gospel, because there will always be massive blind spots in what it tells you. To get a real understanding of how users experience your products, test them on real devices under real conditions as much as possible. And as always, get out there and talk to your users. Network errors prevent your analytics script from loading. Ad-blockers and tracking prevention block your script from loading (enabled by default on many browsers today). A JavaScript error in an unrelated part of the page prevents the analytics script from working correctly. The user loses network connectivity before the analytics script can send data to the server. The user gets impatient and bounces off your website before the page can load fully and start collecting data. Too much JavaScript on the page causes the browser tab to crash (a common issue on low-end devices). The analytics script is blocked by a DNS rule, corporate proxy, firewall, or VPN. The user has disabled JavaScript. The user's browser has limited or no support for JavaScript (Opera Mini still has more than half a million downloads on Android, and it's still widely-used in Africa ). The user is accessing your content using a service that strips JavaScript (e.g. an RSS reader, a web archiving tool, Telegram Instant View, AMP, a read-later service, or a bookmarking service). You only test your app in Chrome, so you don't realize that your website is entirely broken in Firefox and Safari.

0 views
Gabriel Weinberg 4 months ago

No, everyone is not using AI for everything.

Last year around this time The New York Times Magazine ran an A.I. issue with an introduction titled “ Everyone Is Using A.I. for Everything. Is That Bad? ” It’s an edited transcript from the Hard Fork podcast , which I think assumes two things are true that are turning out to be false. Once you’ve tried AI, you use it “for everything.” No, in fact most people who’ve tried it are just occasional AI users. AI has gotten so good that despite any misgivings, “everyone is using A.I.” No, in fact large chunks of the population aren’t using AI at all. (It isn’t really strictly defined in the article, but I’m taking AI to mean generative AI accessible via a chat interface.) Take Gen Z, where AI awareness is the highest : in the last year, even though AI has supposedly gotten a lot better, Gen Z AI adoption has all but stalled, with a meaningful percentage of the Gen Z population still using AI rarely, if at all. Here’s Gallup’s year-over-year (2025/2026) breakdown: 79/81% use AI at least rarely 41/42% are anxious about AI 32/31% use AI only monthly/every few months 22/31% are angry about AI 21/19% never use AI This tracks with Microsoft’s new United States AI Diffusion site , based on “anonymized, aggregated Microsoft telemetry.” Their associated blog reports “more than 30 percent of the US working-age population is using AI [meaning about 70% isn’t], an increase of 3 percentage points from the end of 2025 . ” The underlying academic paper specifies that usage is defined as “engagement with major AI services including ChatGPT, Google Gemini, Anthropic Claude, Microsoft Copilot, and others….with at least 90 minutes of usage time in a given month.” The Microsoft data is brand new, and it mirrors another usage study from Datos from last year, also based on real-world usage data. The Datos study found similarly that, as of last June, only 21% of desktop devices visited “AI Tools” 10 or more times a month, 62% visited 0 times, and the remaining 17% in between . Back on the survey side, a recent Searchlight Institute study found “ 58% report using or trying AI, specifically tools like ChatGPT or Claude, divided evenly between fairly regular users (30% use at least a few times a month) [roughly matching the Microsoft/Datos data] and more infrequent users (29% have used AI, but only once a month or less). ” And finally a new survey from The Argument finds “ most Americans use AI once a week or less. ” All of this triangulates to AI use in America at approximately one third actively using AI, one third occasionally using AI, and one third never using AI , with some movement depending on how you define those terms. In any case, this split is a far cry from “everyone is using AI for everything;” it’s much closer to “some people are using AI for some things.” AI use also hasn’t shifted that much in the past six months to a year. In fact, the only thing that has substantially changed is negative sentiment about AI has gone significantly up, for example the Gallup’s Gen Z poll reporting anger about AI jumping about 40% relative year over year. I think it is a reasonable conclusion to draw from all of this data that a significant percentage of the population is actively limiting their AI usage. The Searchlight study examines a big reason why: real concerns people have with AI. The top three concerns found are “AI will replace jobs and cause unemployment” (42%), "AI will violate people’s privacy” (35%), and “AI will spread misinformation and lies” (33%). This sentiment also matches a strong desire for safety/privacy AI regulation. A solid majority thinks “the government should prioritize creating safety/privacy rules for AI, even if that means the U.S. develops AI more slowly than countries like China.” Another big reason is skepticism in AI usefulness. SearchLight asked about a range of technologies and to say “whether you believe the overall impact of each technology on society is positive or negative.” AI only has an +8% net positive rating right now, right next to +7% for social media, which were only greater than crypto at -17%. Meanwhile cell phones, the internet, and solar energy are at +68%, +67%, and +65%, respectively. The Argument study broke this down further, asking about specific societal benefits from AI, finding broad skepticism and concluding “people aren’t really buying the bullish case for AI that CEOs and boosters alike are selling. In other words, the skepticism about AI’s effects is real and deep-running. And given how many people use it daily, this is not just an ill-informed set of opinions on something respondents have never seen before (like tariffs were before 2025).” It is possible for people to have one view at a societal level and then act differently at an individual level, but that does not seem to be what we’re seeing here. The plurality occasional usage and large percentage of complete avoidance speaks to the fact that a lot of people seemingly aren’t yet finding enough individual value net of their concerns to justify daily or even weekly usage. The gap in media narrative (that everyone is using AI for everything ) relative to the reality (that some people are using AI for some things ) perhaps reflects a bubble around early-adopting knowledge workers that includes much of the tech press (and me for that matter, though I’m trying really hard to stay connected to reality). It’s a mistake for companies, pundits, and policy makers to ignore how people are really feeling and acting about AI. It’s not all sunshine and rainbows. It’s also clearly not binary (all use or no use), but instead a continuum of AI opinions and use, with a lot of people in the middle. I think there is an apt analogy to be made here to preferences around meat consumption. Another thing that seems to be everywhere right now is protein . Telling us how important protein is in our diet is analogous to telling us how useful AI is for productivity. And, meat being a primary source of protein is analogous to AI chat tools being a primary source of generative AI. And yet here’s how Americans break down in terms of their meat consumption preferences, based on a handful of U.S. studies from this decade: 95% eat meat ( Gallup , 2023) 70% report reducing red meat consumption ( Rutgers , 2024) 30% eat (all) meat only rarely/occasionally ( Gallup , 2020) 12% don’t eat red meat ( Nature , 2026) 4% don’t eat any meat, that is are vegetarian ( Gallup , 2023) 1% don’t eat any animal products, that is are vegan ( Gallup , 2023) That is, not everyone eats meat, a majority actively curbs their consumption of red meat, and a significant percentage don’t eat it at all. Different people have different (not mutually exclusive) reasons for limiting their meat consumption, including health, cost, environment, and ethics. Those are all also primary concerns for AI consumption! The analogy also highlights market opportunities to appeal to people across the continuum, speaking to their feelings on AI and addressing their particular AI concerns. For example, we (at DuckDuckGo) make all AI features optional and one of those features, duck.ai , is a private chatbot alternative that helps address AI privacy concerns. To extend the analogy in this way, we’re a restaurant with a variety of options on the menu, from healthy meat dishes (private AI) to vegetarian (turn down AI) to vegan dishes (turn off AI), which most eaters across the spectrum can appreciate. Does this mean about one third of the population is bound to use AI only rarely/occasionally forever? No. Unlike with meat, the AI technology landscape is changing so rapidly that it is very unclear both where AI products and regulations will end up. Product evolution could make AI more useful to the average person, and regulations could reduce concerns. However, we can say that, as of right now, a meaningful percentage of the population has tried the current state of AI and has decided to actively limit their use of it. Thanks for reading! Subscribe for free to receive new posts or get the audio version . Once you’ve tried AI, you use it “for everything.” No, in fact most people who’ve tried it are just occasional AI users. AI has gotten so good that despite any misgivings, “everyone is using A.I.” No, in fact large chunks of the population aren’t using AI at all. 79/81% use AI at least rarely 41/42% are anxious about AI 32/31% use AI only monthly/every few months 22/31% are angry about AI 21/19% never use AI All of this triangulates to AI use in America at approximately one third actively using AI, one third occasionally using AI, and one third never using AI , with some movement depending on how you define those terms. In any case, this split is a far cry from “everyone is using AI for everything;” it’s much closer to “some people are using AI for some things.” AI use also hasn’t shifted that much in the past six months to a year. In fact, the only thing that has substantially changed is negative sentiment about AI has gone significantly up, for example the Gallup’s Gen Z poll reporting anger about AI jumping about 40% relative year over year. Many people are holding back AI use because of real AI concerns and lack of perceived AI value I think it is a reasonable conclusion to draw from all of this data that a significant percentage of the population is actively limiting their AI usage. The Searchlight study examines a big reason why: real concerns people have with AI. The top three concerns found are “AI will replace jobs and cause unemployment” (42%), "AI will violate people’s privacy” (35%), and “AI will spread misinformation and lies” (33%). This sentiment also matches a strong desire for safety/privacy AI regulation. A solid majority thinks “the government should prioritize creating safety/privacy rules for AI, even if that means the U.S. develops AI more slowly than countries like China.” Another big reason is skepticism in AI usefulness. SearchLight asked about a range of technologies and to say “whether you believe the overall impact of each technology on society is positive or negative.” AI only has an +8% net positive rating right now, right next to +7% for social media, which were only greater than crypto at -17%. Meanwhile cell phones, the internet, and solar energy are at +68%, +67%, and +65%, respectively. The Argument study broke this down further, asking about specific societal benefits from AI, finding broad skepticism and concluding “people aren’t really buying the bullish case for AI that CEOs and boosters alike are selling. In other words, the skepticism about AI’s effects is real and deep-running. And given how many people use it daily, this is not just an ill-informed set of opinions on something respondents have never seen before (like tariffs were before 2025).” It is possible for people to have one view at a societal level and then act differently at an individual level, but that does not seem to be what we’re seeing here. The plurality occasional usage and large percentage of complete avoidance speaks to the fact that a lot of people seemingly aren’t yet finding enough individual value net of their concerns to justify daily or even weekly usage. The gap in media narrative (that everyone is using AI for everything ) relative to the reality (that some people are using AI for some things ) perhaps reflects a bubble around early-adopting knowledge workers that includes much of the tech press (and me for that matter, though I’m trying really hard to stay connected to reality). It’s a mistake for companies, pundits, and policy makers to ignore how people are really feeling and acting about AI. It’s not all sunshine and rainbows. It’s also clearly not binary (all use or no use), but instead a continuum of AI opinions and use, with a lot of people in the middle. The meat of it all I think there is an apt analogy to be made here to preferences around meat consumption. Another thing that seems to be everywhere right now is protein . Telling us how important protein is in our diet is analogous to telling us how useful AI is for productivity. And, meat being a primary source of protein is analogous to AI chat tools being a primary source of generative AI. And yet here’s how Americans break down in terms of their meat consumption preferences, based on a handful of U.S. studies from this decade: 95% eat meat ( Gallup , 2023) 70% report reducing red meat consumption ( Rutgers , 2024) 30% eat (all) meat only rarely/occasionally ( Gallup , 2020) 12% don’t eat red meat ( Nature , 2026) 4% don’t eat any meat, that is are vegetarian ( Gallup , 2023) 1% don’t eat any animal products, that is are vegan ( Gallup , 2023)

0 views
DYNOMIGHT 4 months ago

Is “colorectal cancer” rising in “young people”?

(Yes, but.) Over the past few years, I’ve seen many articles about mysterious rise in colorectal cancer (CRC) in young people. There are various stories for why this might be happening: General health. Maybe modern people are unhealthy (obesity, low physical activity, diabetes, poor sleep), leading to insulin resistance and chronic inflammation, meaning faster epithelial cell proliferation and a miscalibrated immune system that fails to stop early cancers? Ultra-processed food. Maybe people are eating more ultra-processed foods that contain additives (like emulsifiers) that degrade colon mucus, allowing bacteria to contact epithelial cells and drive inflammation? Or maybe ultra-processed food has low fiber and glycemic load, leading to insulin resistance and chronic inflammation, with the problems mentioned above? Bad meat. Maybe people are eating more red and/or processed meats, which expose the colon to nitrites and secondary bile acids, which inflame the epithelium and promote chronic inflammation? The microbiome. Maybe it’s the microbiome. For example, maybe people’s guts are getting colonized by strains of E. coli that produce genotoxic colibactin . Or maybe overuse of antibiotics in early life depletes protective bacteria in the gut, allowing harmful strains to expand, e.g. strains of B. fragilis that cause inflammation, or strains of F. nucleatum that can survive in the gut and drive tumor growth ? Environmental exposures. Maybe people are getting exposed to bad stuff in the environment (microplastics, forever chemicals , pesticides, endocrine disruptors, air pollution) that does bad stuff (damages gut barrier, screws up the microbiome, disrupts hormonal signaling)? Maternal health. Maybe poor maternal health (obesity, diabetes) exposes the fetus to elevated glucose / insulin / inflammation, and these in turn program the child for a lifetime of metabolic issues and inflammation? Whatever. Maybe alcohol / smoking / painkillers / calcium / vitamin D / inflammatory bowel disease / hereditary syndromes / screening bias? None of the experts seem to agree on which of these is the culprit, so I figured that I (person with blog) should help. If you poke at these stories, most of them are individually pretty weak. It can’t all be detection bias since CRC deaths are also going up in younger people. And several proposed causes (air pollution, tobacco) have actually fallen in rich countries. Other explanation, like E. coli producing colibactin, seem biologically real, but there’s no evidence that they’re increasing over time. Still other suggested causes (microplastics, forever chemicals) are mostly mechanistic speculation at this point. Obesity, inactivity, and chronic inflammation also all seem biologically real, and they are likely increasing, but why should they specifically cause colorectal cancer in young people ? A plausible answer to that last question is that they aren’t. They’re doing it, but not specifically . This will sound pedantic, but bear with me: If you say that CRC is increasing in younger people, what exactly does that mean? After all, the set of people who qualify as young changes over time. (Ever notice that you keep getting older?) Siegel et al. (2026) plot how often CRC was found in different age groups in 1995 and in 2022. They also provide this plot of how common different types of CRC are in different age groups. At a glance, this doesn’t look so bad. If you’re young, you might think, “OK, my current risk is higher than previous generations faced at the same age, but I can look forward to decreasing rates when I’m old.” You could easily think this is good news: While there’s a relative increase when you’re young, it’s tiny compared to the absolute decrease while you’re old. Unfortunately that’s the wrong way to think about it. Downham et al. (2026) plot CRC rates in different age groups across the Anglosphere over time. Everyone I’ve shown this plot to has said it’s confusing, so let me explain: The different lines track age-bands as people born in different years move in and out of those bands. For example, in the US plot in the bottom right, the “20-25” line starts with the left-most dot showing the CRC rate for people born between 1965 and 1970 when they were 20 to 24 years old (around 1990). The next dot shows the rate for people born between 1970 and 1975 when they were 20 to 24 years old (around 1995), and so on. That figure is weird, because the lines connect different groups of people. I wanted a plot where there are lines for different birth cohorts as they age. For unknown reasons, no one seems to make such plots, and the data isn’t trivial to access. So I used a plot digitizer to click on every damned point that US figure above and then replotted it: Now the individual lines show specific groups of people tracked through time. For example, the “1932.5” line shows CRC rates for people born between 1930 and 1935, when those people were at different ages. If you look closely, you’ll notice that these rates are higher those for people born between 1940 and 1945 for all ages (where we have data). That was the pattern for a long time: Between 1920 and 1950, later generations enjoyed lower CRC rates across all phases of their lives. But between 1950 and 1960, that pattern reversed and since then later generations have had higher CRC rates at all ages . We don’t know for sure what will happen in the future. But I think it’s likely this trend will continue. Yes, if you are currently young, you face higher CRC risk than previous generations did when they were young. That’s the bad news. The other bad news is that when you are old, you may also face higher CRC risk than previous generations did when they were old. The other other bad news is that CRC isn’t the only type of cancer that’s rising in later generations. Sung et al. (2019) give this plot: These are again the confusing graphs where individual lines show age bands as different people move in and out of them. But you get the point: Lots of cancers are going up in younger people later generations, including uterine, gallbladder, kidney, liver, pancreas, and thyroid. (Their additional material contains plots for 18 other cancers.) Note that these plots have a logarithmic y-axis, meaning the changes are larger than they might appear. Moving up a quarter of the way between two vertical ticks corresponds to an increase of a factor of ≈ 1.78. If lots of cancers are becoming more common in later generations, then why is everyone talking about CRC? I think that’s because CRC in unique in that it is: For example, thyroid cancer diagnoses have skyrocketed in recent decades. But that’s partly because of more detection, and thyroid cancer is highly treatable, without clear benefits from early detection. Pancreatic cancer also seems to be increasing, but we don’t have good ways to screen for it and even if we did, we don’t have good ways to treat it. CRC is really unique in that you can save lives by telling people, “Hey! CRC is going up! You should get screened!” If you’re interested in public health, that’s the most important thing. But if you’re interested in unraveling the mystery of CRC going up, it’s important to note that CRC isn’t really unique at all. Colorectal cancer is going up in young people. Various kinds of cancer are going up in later generations. (Definitely at younger ages, possibly at all ages.) This blog endorses colorectal cancer screening . We don’t yet know if colonoscopies are better than other methods of screening (sigmoidoscopy, stool tests), but we do know that screening is better than not screening. When caught early, CRC is highly treatable, often with only surgery (no chemotherapy or radiation) and a return to normal activities within a couple weeks. increasing in later generations treatable if caught early detectable via screening

0 views
Entropic Thoughts 4 months ago

90 % of the t distribution

William Sealy Gosset was great. He improved beer at Guinness by using the statistics that existed at the time. Not happy with that, he invented new statistics to brew even better beer. The things he invented are used all over the place now, but Guinness wanted to keep him a secret weapon, so they made him publish his results under the fake name Student . One thing Gosset realised is that it is wrong to compute 90 % confidence intervals for the mean by taking the standard deviation of the sample, and assume a normal distribution , like-a-so: \[\hat{\mu} \pm 1.645 \hat{\sigma}\] (Continue reading the full article on the web.)

0 views
Lalit Maganti 5 months ago

Which country voted the best at Eurovision?

Eurovision was on yesterday. I’ve never been interested much in the musical side but the weird political dynamics of Eurovision voting have always fascinated me; I tune in each year just for them and somewhat snarky commentary of Graham Norton, the UK commentator. As I was watching the jury votes come in, a question popped into my head: Which country has voted the best in Eurovision? That is, which country was best at picking the eventual top 10 and in the right order? Strangely enough, while there’s plenty of work on voting blocs and bilateral biases at Eurovision, most of it asks who votes for whom ; I wanted to ask who votes accurately . I couldn’t find anyone asking the question that way, so I decided to do some data analysis myself. To begin to answer this question, I first needed to formalize what “best” even means. That is, some mathematical notion of “good” and “bad”.

0 views