Martin Alderson 3 days ago

A little tool to visualise MoE expert routing

I've been curious for a while about what's actually happening inside Mixture of Experts models when they generate tokens. Nearly every frontier model these days (Qwen 3.5, DeepSeek, Kimi, and almost certainly Opus and GPT-5.x) is a MoE - but it's hard to get an intuition for what "expert routing" actually looks like in practice. So I built a small tool to visualise it: moe-viz.martinalderson.com You can pick between a few different prompts, watch the generation animate out, and see exactly which experts fire at each layer for each token. The top panel shows routing as the token is generated, the bottom panel builds up a cumulative heatmap across the whole generation. I built this by modifying the llama.cpp codebase to output more profiling data, with Claude Code's help. So it may have serious mistakes, but it was a really fun weekend project. The thing that really surprised me: for any given (albeit short) prompt, ~25% of experts never activate at all. But it's always a different 25% - run a different prompt and a different set of experts goes dormant. That's a much more interesting result than I expected. Interestingly Gemma 26BA4 runs really well with the "CPU MoE" feature - 4b params is not a lot to run on a fairly fast CPU and having KV cache on GPU really helps. I think there's a lot of performance improvements that could be done with MoE inference locally as well - eg caching certain experts on GPU vs CPU. If you're interested in learning more about LLM inference internals I'd certainly recommend pointing your favourite coding agent at the llama.cpp codebase and getting it to explain the various parts - it really helped me learn a lot.

AI C++ Data Visualization

Carlos Becker 2 weeks ago

Mapping Brazilian Cell Towers

I was curious about how many cell towers were around me - so I built the tool I wanted.

Data Analysis Data Visualization

Rik Huijzer 2 weeks ago

Biblical Earth

This is an image by ChatGPT showing roughly how the Bible describes the earth:  Unfortunately, it's not showing the four corners. This image is based on, Isaiah 40:22 “It is he that sitteth upon the circle of the earth, and the inhabitants thereof are as grasshoppers…” Job 26:10 “He hath compassed the waters with bounds, until the day and night come to an end.” Job 26:7 “He stretcheth out the north over the empty place, and hangeth the earth upon nothing.” Proverbs 8:27 “When he prepared the heavens, I was there: when he set a...

Science Data Visualization

Artyom's Chaotic Blog 3 weeks ago

Kaktovik Numerals

Read on the website: Kaktovik numerals are a surprisingly good counting system. It allows many arithmetic operations to be done visually and effortlessly. Though it takes some getting used to. Thus this page!

Data Visualization Science

daniel.haxx.se 1 months ago

One hundred curl graphs

In the spring of 2020 I decided to finally do something about the lack of visualizations for how the curl project is performing, development wise. How does the line of code growth look like? How many command line options have we had over time and how many people have done more than 10 commits per year over time? I wanted to have something that visually would show me how the project is doing, from different angles, viewpoints and probes. In my mind it would be something like a complicated medical device monitoring a patient that a competent doctor could take a glance at and assess the state of the patient’s health and welfare. This patient is curl, and the doctors would be fellow developers like myself. GitHub offers some rudimentary graphs but I found (and still find) them far too limited. We also ran gitstats on the repository so there were some basic graphs to get ideas from. I did a look-around to see what existing frameworks and setups that existed that I should base this one, as I was convinced I would have to do quite some customizing myself. Nothing I saw was close enough to what I was looking for. I decided to make my own, at least for a start. I decided to generate static images for this, not add some JavaScript framework that I don’t know how to use to the website. Static daily images are excellent for both load speed and CDN caching. As we already deny running JavaScript on the site that saved me from having to work against that. SVG images are still vector based and should scale nicely. SVG is also a better format from a download size perspective, as PNG almost always generate much larger images for this kind of images. When this started, I imagined that it would be a small number of graphs mostly showing timelines with plots growing from lower left to upper right. It would turn out to be a little naive. I knew some basics about gnuplot from before as I had seen images and graphs generated by others in the past. Since gitstats already used it I decided to just dive in deeper and use this. To learn it. gnuplot is a 40 year old (!) command line tool that can generate advanced graphs and data visualizations. It is a powerful tool, which also means that not everything is simple to understand and use at once, but there is almost nothing in terms of graphs, plots and curves that it cannot handle in one way or another. I happened to meet Lee Phillips online who graciously gave me a PDF version of his book aptly named gnuplot . That really helped! I decided that for every graph I want to generate, I first gather and format the data with one script, then render an image in a separate independent step using gnuplot. It made it easy to work on them in separate steps and also subsequently tune them individually and to make it easy to view the data behind every graph if I ever think there’s a problem in one etc. It took me about about two weeks of on and off working in the background to get a first set of graphs visualizing curl development status. I then created the glue scripting necessary to add a first dashboard with the existing graphs to the curl website. Static HTML showing static SVG images. On March 20, 2020 the first version of the dashboard showed no less than twenty separate graphs. I refer to “a graph” as a separate image, possibly showing more than one plot/line/curve. That first dashboard version had twenty graphs using 23 individual plots. Since then, we display daily updated graphs there . All data used for populating the graphs is open and available, and I happily use whatever is available: Open and transparent as always. Every once in a while since then I get to think of something else in the project, the code, development, the git history, community, emails etc that could be fun or interesting to visualize and I add a graph or two more to the dashboard. Six years after its creation, the initial twenty images have grown to one hundred graphs including almost 300 individual plots. Most of them show something relevant, while a few of them are in the more silly and fun category. It’s a mix. The 100th graph was added on March 15, 2026 when I brought back the “vulnerable releases” graph (appearing on the site on March 16 for the first time). It shows the number of known vulnerabilities each past release has. I removed it previously because it became unreadable, but in this new edition I made it only show the label for every 4th release which makes it slightly less crowded than otherwise. vulnerabilities in releases This day we also introduce a new 8-column display mode. Many of the graphs are internal and curl specific of course. The scripts for this, and the entire dashboard, remain written specifically for curl and curl’s circumstances and data. They would need some massaging and tweaking in order to work for someone else. All the scripts are of course open and available for everyone. I used to also offer all the CSV files generated to render the graphs in an easy accessible form on the site, but this turned out to be work done for virtually no audience, so I removed that again. If you replace the .svg extension with .csv, you can still get most of the data – if you know. The graphs and illustrations are not only silly and fun. They also help us see development from different angles and views, and they help us draw conclusions or at least try to. As an established and old project that makes an effort to do right, some of what we learn from this curl data might be possible to learn from and use even in other projects. Maybe even use as basis when we decide what to do next. I personally have used these graphs in countless blog posts, Mastodon threads and public curl presentations. They help communicate curl development progress. On Mastodon I keep joking about me being a graphaholic and often when I have presented yet another graph added the collection, someone has asked the almost mandatory question: how about a graph over number of graphs on the dashboard? Early on I wrote up such a script as well, to immediately fulfill that request. On March 14 2026, I decided to add it it as a permanent graph on the dashboard. Graphs in the curl dashboard The next-level joke (although some would argue that this is not fun anymore) is then to ask me for a graph showing the number of graphs for graphs. As I aim to please, I have that as well. Although this is not on the dashboard: Number of graphs on the dashboard showing number of graphs on the dashboard More graphs I am certain I (we?) will add more graphs over time. If you have good ideas for what source code or development details we should and could illustrate, please let me know. The git repository: https://github.com/curl/stats/ Daily updated curl dashboard: https://curl.se/dashboard.html curl gitstats: https://curl.se/gitstats/ git repository (source, tags, etc) GitHub issues mailing list archives curl vulnerability data hackerone reports historic details from the curl past

Java Open Source Data Visualization HTML Programming

Harper Reed 1 months ago

Note #728

I posted a fun post about how I made my transcripts into really neat knowledge graphs. Hope it is helpful harper.blog/2026/03/1… Thank you for using RSS. I appreciate you. Email me

Data Visualization

Stone Tools 1 months ago

Lotus 1-2-3 on the PC w/DOS

What would a piece of software have to do today to make you cheer and applaud upon seeing a demo? I don't mean the "I'm attending a keynote and this is expected, please don't glower at me Mr. Pichai," polite-company type of applause. I mean the "Everything's different now." kind. For that, the bar is pretty high these days. "Photorealistic" fight scenes between Brad Pitt and Tom Cruise against an apocalyptic cityscape are generated out of nothing but a wish, and social media, smelling the cynical desperation, can offer no more than a clenched-teeth grimace. Within 48 hours the cold light of the epic battle has faded, leaving no residual heat. A sense of awe was easier to elicit back in the golden era. Bill Atkinson scrubbed out some pixels with an eraser in MacPaint to thunderous applause. Andy Warhol did a flood fill on an image capture of Debbie Harry, leaving an audience enraptured. Perhaps miracles work best when they're minor. Mitch Kapor has been on the receiving end of the adulation. As CEO of newly-formed Lotus Corporation, demos of their flagship product 1-2-3 generated significant light and heat with the crowds. In a 2004 interview with the Computer History Museum, Kapor said, "You could with one-click see the graph from your spreadsheet. You could not do that before. That was the killer feature when we demo’d it. I mean, literally, people used to applaud – as hard as it is to believe." He knew all too well the struggles of the VisiCalc crowd, having previously built VisiPlot and VisiTrend for VisiCorp. Those programs worked with VisiCalc data to draw graphs, but required a lot of disk swapping to move in and out of the various programs when fine-tuning charts and graphs. 48K on the Apple 2 made it essentially impossible to fit all of the software into memory at once, but they could at least put everything onto the same diskette, Kapor reasoned. Eliminating that song and dance would be useful to the customers. Depicted as a literal song-and-dance in their advertising. In an interview in Founders at Work, Kapor said, "At various times I raised a number of ideas with the publisher about combining ( VisiCalc and VisiPlot onto one disk) and they weren't interested at all. I don't think they really saw me as an equal. They saw me, when I was there as a product manager, as an annoyance—as a marginal person without experience or credentials who was kind of a pest. And I suppose I was kind of a pest." He said the feeling was mutual, and that was basically it for his employment with Personal Software and the VisiCalc team. He let them buy him out (i.e. the juicy royalties he was receiving for VisiPlot and VisiTrend ) for $1.2M, then took that money and went off to build the better mousetrap he had tried to pitch. Lotus 1-2-3 would quickly become the "killer app" for the nascent IBM-PC, doing for that system what VisiCalc had done earlier for Apple. 1-2-3 's success (and corporate in-fighting between Personal Software and VisiCorp) drove VisiCalc sales into the ground almost immediately. Two years later, Lotus would buy out Personal Software. One year later, Lotus would kill VisiCalc . Today, Microsoft Excel documentation still references Lotus 1-2-3 , not VisiCalc . I have no 1-2-3 experience going into this. I always thought "1-2-3" referred to its relationship to numbers. "1, 2, 3. Row numbers. Numbers in a spreadsheet. Mathy number stuff. I get it." I honestly had no idea "1-2-3" indicated something more. I'm learning that VisiCalc walked so 1-2-3 could run (over VisiCalc's ashes in a Sherman tank) . I have one goal in learning Lotus 1-2-3 . I want to understand what it did that was so superior to my beloved VisiCalc that it practically wiped them out in the first year of launch. Kapor had projected first year 1-2-3 sales of US$1M, but did US$53M instead. That's not just a little better than VisiCalc, that's " VisiWho ?" dominance. VisiCalc is a spreadsheet and 1-2-3 is a spreadsheet, so what's the big fuss? First, the platform of choice, the IBM-PC running PC-DOS (MS-DOS, to those buying it separately), affords two big wins right off the bat. 80-column text mode makes the Apple 2's 40-columns feel claustrophobic (and perhaps a bit un-business-like?). The greatly expanded memory of the 16-bit PC, max 640K vs. the 8-bit Apple 2's 48K, lets far more complex worksheets fill out those roomy 80-columns. As Lotus Corporation and magazines and Wikipedia pages and other blogs love to point out, the true game-changer is contained in the program's very name. "1-2-3" refers to the three components of this "integrated software" package. "1" is the spreadsheet capability, which surpassed most contemporaries handily in speed, being written in x86 assembly (until Release 3). "2" is for those graphing tools which had Kapor's audiences applauding. "3" was intended to be a word processor, but according to programmer Jonathan Sachs, "I was a few weeks into working on the word processing part, and I was getting bogged down. That's about when Context MBA came out, and I got a look at what they had done." "What they had done" was integrate a word processor, communications, and database, along with the spreadsheet and graphics components. Context 1-2-3-4-5 , as it were. When Sachs saw the database, that felt to him like a more natural fit and "3" was re-implemented as a database. "It would be a heck of a lot easier to implement," he noted. Woz bless our lazy programmers. The upshot is 1-2-3 plays nicely with last post's focus, dBase , which feels like a particularly powerful combination. I feel a tingle when skills picked up on a previous exploration pay dividends later. Deluxe Paint + Scala paid off similarly. Is this what it feels like to "level up?" Obtaining literature on Lotus 1-2-3 is only difficult in the " overchoice " sense. I expected to find a lot of books, but perhaps not the "What have I gotten myself into?" existential dread of 1,000 hits on archive.org. It wasn't just books, that period had an interesting side phenomenon of "software vendor published enthusiast magazines." Companies like Aldus, Corel and Oracle all had self-titled publications on newsstands. Lotus Corporation did as well with LOTUS Magazine . Published monthly by Lotus Corporation, it debuted with the May 1985 issue (probably on newsstands late March, early April). The tagline, "Computing for Managers and Professionals," oriented itself toward the decision makers, the ones with purchasing power. A poll of Lotus software users revealed, "Most of you see the computer primarily as a tool and are not interested in computing, per se." Toward that end, the magazine took a different tack than the BYTE s and PC Magazine s of the time. It was to be no-nonsense, non-techno-babble, short, easy-to-digest articles about computing from the manager's perspective. "What's all this I keep hearing about 'floopy disks' and 'rams' and 'memories' and such and so on? It's enough to drive a reasonable business computerist straight to distraction!" says the frazzled corporate executive trope. There there, fret not! LOTUS Magazine feels your pain and addresses it with the cover story of issue 1. "The world of computer memory has enough complexity and high-tech jargon to drive the most reasonable business computerist straight to distraction," leads in to "An Inside Look at Computer Memory" by T.R. Reid. The article explains the differences between RAM and ROM, floppies and hard disks, and so on, unfurrowing the knitted brows of befuddled mid-80's business executives. When it got into the 1-2-3 of it all, LOTUS Magazine didn't pull its punches. Articles were short, around four pages, and assumed a higher level of analytical aptitude than IT aptitude. Lots of charts of formulas, macro definitions with explanations, tips and tricks for faster data entry, and so on fill out the pages. That ran for about seven years, until the December 1992 issue, when publishing duties transferred to PC Magazine as PC Magazine: LOTUS Edition . It was PC Magazine with a mini-magazine's worth of Lotus-specific content appended each month, as a special imprint. That ran until August 1995 , marking a 10-year publication run which would have exceeded my prediction by about eight years. After judging books entirely by their covers, I've chosen the official Lotus manuals for 1.0A, 2.2, and 3.4, and two compilations of tips and tricks previously published in LOTUS Magazine . I flip through other stuff as well, but honestly nothing is holding my attention this time around; they all read the same, "dry and boring." 1,000 pages or more for some of those books and they didn't have room for even one joke? I promise at least seven in this post alone. See if you can spot them all! Launching into the program proper brings me to the expected "I'm a spreadsheet!" grid layout, with column and row labels, arrow-key controllable cell cursor, and a blank area at the top for VisiCalc -y stuff. Let's go. As an intermediate level VisiCalc user, I am delighted my menu muscle memory pays immediate dividends. Clearly Lotus welcomes defectors and even makes life easier on everyone by taking advantage of the 80-column display. VisiCalc 's single-letter menu mnemonics are enhanced in 1-2-3 by simply spelling it all out on-screen. Full menu item names are always visible, yet still accessible by single-letter commands. From the jump, 1-2-3 makes a strong case for itself, providing improved usability and discoverable tools. Before digging in too deeply, I should note that 1-2-3 does all of the VisiCalc things. A1-style cell references, slash menu, fixed and relative cell references, @ functions including transcendentals, range specifier, prefix for values, and on and on. It adds, it subtracts, it calculates interest. 1-2-3 "Yes, and..."s VisiCalc from there. We gain a lot, but there is a notable absence: the upper-right status check. VisiCalc shows calculation order, arrow-key toggle, and free memory in that spot. Those are all gone in 1-2-3 and good riddance, frankly. On the PC I have full arrow keys and more RAM than Woz; 1-2-3 sees my full 16MB of DOS Extended memory. There is no stopping me. 1-2-3 also says nuts to VisiCalc 's "calculation order" (by row or by column) hoo-hah and introduces "minimal recalculation." From the almost comically-straightforward named book Lotus 1-2-3, Release 2.3 , "When 1-2-3 recalculates a worksheet, only those formulas directly affected by a change in the data are recalculated." I am living large here in 1989, or 1991, or whatever year I'm pretending it is this week. Even VisiCalc 's gets a glow up. You know it today as and , both of which were present in 1-2-3 Release 1 back in 1983. At this rate, 1-2-3 is flirting dangerously close to "expected spreadsheet behavior in 2026." Don't get my hopes up, Lotus. There's only down from there. The more I encounter this, the more I wonder if we gave up on it too soon. This could be "blogger overly immersed in their subject matter" brain, but I'm growing to oftentimes prefer two-line horizontal menus over modern GUI menus. I find the left-right, up-down, left-right, up-down, scanning through GUI menus kind of tiring. With the two-line menu, I can step through top-level options with the left/right arrow keys, eyes focused on line two as I scan sub-menu items. It also provides something GUI menus don't: an immediate explanation of a menu item before committing its action to the document. If a menu item is not a sub-menu, line two describes it. It's easy to audit features in an unknown program. Also, every menu item has a keyboard shortcut; just type the first letter. This requires creativity by the developer when naming menu items such that each has a unique first letter, but it also creates a de-facto mnemonic for the user. Don't discount muscle memory! There's one "drawback," but I'll try to make a case for it. Specifically, it is probably impossible to fit everything in a modern GUI menu into a two-line scheme. There's just too much! I suggest the horizontal menu-bar solves this precisely because of that design constraint. If there's too much, the menu needs to be simplified. "Problem solved," the author asserted. This has to be one of 1-2-3 's greatest contributions to modern spreadsheets. It still exists, just open up your modern spreadsheet of choice and try it. Enter 1 through 5 down the A column. Starting with B2, enter the formula and copy it down a few rows. Old hands know that a symbol in a cell reference fixes that row or column of the reference, otherwise references are relative. That's a huge step up from VisiCalc 's "all or nothing" approach to cell references. Put in a formula and copy it through to other cells. For every cell reference, in every copy of the formula, VisiCalc prompts the user for "relative or fixed?" It is a complete drag, and Woz help you the day that formula needs updating. The approach is superior, allowing us to embed relativity into the formula itself. Then, copying a formula across cells copies our intent as a natural course. It's simple to understand and hard to mess up: my favorite combination. While it can't load non- 1-2-3 documents natively, Lotus does provide a nice translation tool for helping us get data out of the heavy hitters of the day. From a Stone Tools perspective, this handles everything I need so far, as VisiCalc and dBase are both accounted for and work as advertised. Translation works both ways, so bringing in dBase data, messing around with it in 1-2-3 , and going back out to dBase is possible, though there are cautions in doing so. One notable thing to watch out for is "deleted" records. dBase only "marks for deletion" (until a .PACK command), and that flag won't survive transit. A small inconvenience, all things considered. In the top-level menu is the shiny new option, the "2" in "1-2-3." I know exactly what I want: a pie chart of game software genres imported from dBase II . The options for are straightforward, and the limitations are self-evident. Notably, look at the "Ranges" settings. Range sets value labels which will appear along the X-axis. Ranges through define six, and only six, ranges of data to plot on the graph. That's it. Everything else you see is "make it pretty." Within the confines of my self-imposed time capsule, my only point of reference thus far is VisiCalc and its clones. Through that lens, I'm blown away by Lotus 1-2-3 . I mean, come on, 3-D bar charts ?! Am I living in the world of TRON right now?! The applause is well-earned, Mitch. Bravo! Encore, even! Now, Mr. Kapor, if you'll excuse me a moment, I need to have a quick, private chat with my readers. Yes, sorry, I'll only be a moment. Hello dear readers. Mitch can't hear us, yeah? We're safe? OK, between you and me, that graphing tool is a little underwhelming, huh? There's a lot we can do to make a graph look as pretty as possible for screens and printers of the time, but the core graphing options themselves are kind of anemic. Here's Google Sheets making the pie chat I'd hoped 1-2-3 could generate. However, 1-2-3 cannot do this because it can only graph strict numeric values; strings, like "genre" types, return blank charts. 1-2-3 also can't coalesce data, like we see Sheets doing above. To achieve my goal, I'll need to figure out a different approach. (Plus, maybe I've discovered a DOSBox-X bug ?) It's not fair to judge past tools as being "inferior" just because they don't live up to 2026 standards. Still, what I'm trying to do must have been one of the first things many business owners wanted to do, right? Am I storing my data in a style that hadn't been popularized yet? Is my 2026 brain making life more difficult for my 1991 doppelgänger unnecessarily? How does one graph out the count of each unique genre? Alright, this is going to get complicated, so I think a diagram is in order. This actually explains a lot about the Lotus 1-2-3 approach to data in general, how to manipulate it, how to query it, and generally how to interface with the more complex functions of the program. Having imported the dBase list of CP/M games from the dBase article, let's extract a list of all titles that are of genre "Simulation." I'll use a subset of the total data so everything fits on screen for demonstration purposes and perform (aka , aka The Notorious DQU, aka Query's L'il Helper) A worksheet is not just rows and columns of data. It also serves as a control mechanism for defining interactions with the data. A worksheet has columns up to IV (256) and rows up to 8192. What do we do with 2,000,000+ cells? In true Dwarf Fortress fashion, we section off areas ("ranges" in 1-2-3 speak) and designate functions to those areas. First, I have my data as the main table, field names at top. Then, I need to set up my query criteria. This is a separate portion of the worksheet, with the fields I want to query against and room below to accept the criteria definition. Think of it like building a little query request form. Then, Lotus needs a place to spit out the results. Again, I set up a little "form" to receive the data. Put in whichever field names are of interest in the final data capture. Now, what if there are multiple queries I want to re-use from time to time? Painful as it sounds, I must set up multiple query forms, one for each query I expect to re-use. So, re-copy all of the field headers of interest into a new portion of the worksheet. Re-copy the field headers for the output range. Put in the new query criteria. Do another extraction. Keep dividing the worksheet up into all of the various queries one might need to reuse. Each lives in its own little area of the worksheet, so maybe now's a good time to start labeling things? Maybe mentally divide the worksheet into "my queries live over here, in Q-Town" and "my results live over there, in Resultsville" and so on. For my stated goal, I need the unique list of genres for my game list and the count of each genre within the data set. From the previous section, I know how to extract a list of unique genres. To count them, can count all non-empty records which match my criteria. Lemme draw up another diagram here. After extracting the list of unique values for "Genre", I get a column of results as seen at in the image above. Notice the criteria at is empty? By not specifying anything, that equates to matching any "Genre". Next, I need to reformat that column into countable criteria for . Just like in a query, criteria consists of two vertically contiguous cells, the top of which is the field name and the bottom holds the parameter. The field name must be physically, immediately above each and every genre I want to count. will transpose a range of vertical or horizontal cells into their mirror universe opposite. That's how I generated the horizontal list at . A of the field name across row 15 generated nice pairings, perfect for use with . The cell formula outlined in yellow is essentially the same across , each lightly modified to point to a different criteria range. That calculates the count for each genre in column , and column holds my titles. Now I have what I need to generate the chart I wanted (aforementioned pie chart drawing bug notwithstanding). Here it is in glorious 3-D from the future (of the past)! Frustratingly, figuring all of that out took the better part of a day. But now I know! If only there were some way to make it easier. There are issues with my solution thus far, many of which boil down to the physical spaces assigned to hold queries and results and transformations and data. If I bring in new data with new genres, new result lists could physically lengthen and overlap one another. Planning a physical map for the worksheet is a priority. Building out the sheet, especially keeping cell references flexible to changes in data, is a drag. I'd also like to generate a graph from the new sheet arrangement, with just a simple hot-key. Like all great developers, I want to be lazy. The first step toward the promised land of laziness is "hard work," unfortunately. Hard work can be captured and reused, luckily, as Lotus 1-2-3 features "Friend of the Blog": macros. VisiCalc didn't have it, and 1-2-3 's implementation is robust enough that many books were devoted to understanding and taming it. Here's a simple macro, which hints at its latent power. 0:00 / 0:07 1× Custom menus are easy to build. Selecting an option could trigger a longer automation task, simplifying a multi-step process, or something as simple as a help menu. Macros are stored... ( say it with me now ) ...in the worksheet. Yep, whatever map you had in mind for dividing up the worksheet into query-related fiefdoms, redistrict once more to hold macro definitions. Custom menus are an easy way to illustrate macro structure. Here's a dumb example. The text in column A is mostly comments to organize our worksheet and thoughts. represents the keyboard shortcut assigned to the macro, accessed by . is a reference to a named cell range. Named ranges are an important improvement over VisiCalc . Once defined, a range can be invoked by name anywhere a range is expected. Assuming a cell range as has been assigned a name like , is totally valid. is a range defined as . is a range defined as . Notice a range only needs to define the first start of a macro definition. Macro execution will read each cell in order down a given column until the first empty cell. range names are interpreted by 1-2-3 as macro keyboard shortcuts automatically. The convention shown, of a human-readable label to the immediate left of a range by the same name is so common it has its own menu shortcut. applied to column A will auto-assign column B cells to the names in A. To a certain extent, a named range can function like a programming "goto". In the macro case, its saying "Goto the range named and continue executing the macro from there." Programmers in the readership are salivating at the deviously complex ways this "goto labeling" could be abused. Combine it with decision making through and iteration through and the possibility space opens wide. After doing dBase work last post, I noted that I had accidentally become a dBase developer without even trying; the dBase scripting language was precisely equivalent to the commands issued at the dot prompt. I'm not so lucky with 1-2-3 . Setting up a macro which issues a simple string of commands is easy enough, and reads (mostly) like how I'd type it at the menu, akin to Bank Street Writer 's approach to macros. For example, will issue to bring up the slash menu, access the ( W )orksheet menu, then the ( C )olumn sub-menu, and finally ( H )ide a column. ~ issues "enter", which at this point in the menu navigation will commit the prompt default, i.e. the current position of the cursor. Just like that, hiding the current column just became a single keystroke. There is also a menu tool which is "record every keystroke I do from now." That recording will be output into the worksheet. Apply a range name to that and it transforms into a macro. Very nice! That said, 1-2-3 macros go from zero to 100 pretty quickly and are visually difficult to parse and reason out. One must be super-duper intimately familiar with every command in the slash menu, plus the macro-specific vocabulary. Lotus understood things could get hairy pretty quickly and added a debugging tool to help make sense of things. enters mode, which executes macros one line at a time. The status bar at the bottom of the screen explains what is being run, so when something goes wrong I know who to blame. OK , are you ready to dig in and implement macros which simplify the queries and procedure discussed earlier? < cracking knuckles> Well, I'm not. < uncracks knuckles back to stiffness > The macro system has proven too complicated to feel any sense of control or mastery beyond Baby's First Macro™. With a couple of more weeks' study I think I could achieve my goal. Unfortunately, for this post, I am defeated. The "3" in "1-2-3", 1-2-3 can function as a database. A very simple, limited, one-row-equals-one-record, 8192 record max, 256 field max, flat database. Let's be honest, oftentimes that's more than enough. I showed examples of querying earlier, and that's as fancy as it gets for this. We can sort records ascending/descending by up to two keys, find and replace values, find records which match a search query, and extract those records into another area of the spreadsheet. And nothing else (at least for Releases 2.x). 0:00 / 0:52 1× Sorting dBase II data by genre. It may seem I'm giving this aspect of the program short-shrift, but so did Lotus. In their own manual for Release 2.2, macros have 300 pages devoted to them. Database functionality has 50, and the first 20 of those are instructions for typing in dummy data. Sorting, querying, finding, and extracting, the meat and potatoes of database-ing, warrant a mere 20 pages total. It's a useful feature and I'm glad it's here. It's enough to handle most of my meager needs. Beyond that, there's not much to say, except to note its legacy. It was an obvious idea to anyone who touched VisiCalc for more than five minutes, so its development feels inevitable. Do some database work in Excel tonight and light a candle for 1-2-3 . A very nice feature of 1-2-3 that fits right in with its "integrated" approach, is what we would call today "plug-ins" or "extensions," but which Lotus calls "add-ins." 1-2-3 shipped with a few. For example, one expanded macros by letting them live in-memory, for use across worksheets. Normally the only macros accessible to a worksheet are those defined within itself. Man, VisiCalc is just getting lapped by 1-2-3 's ingenuity, huh? According to a PC Magazine article about the state of add-ins, many business-people lived inside 1-2-3 all day long and wanted to do everything from within its confines . The 3rd party add-in after-market happily commodified those desires. In addition to obvious ideas, like automated save/backup utilities, or industry-specific analysis tools, add-ins could mold 1-2-3 into almost anything. Complete word processors, entire graphic subsystem replacements for complicated graphing needs, expert system logic, and non-linear function solvers were injected into the program. Oracle offered a way to connect to their external SQL databases from within the snugly confines of 1-2-3 's security blanket. The Lotus approach, being a product of lower-memory days, is both annoying and useful. Add-ins can be, though are not by default, loaded at app startup. Add-ins must be "activated" one-by-one to gain access to their extended powers, or "deactivated" to make room for other add-ins or a larger worksheet. I have enough memory, so I'm not in trouble here, though I'm sure it's easy to imagine on a 512K system that manual memory management was a real thing. Between macros and add-ins, 1-2-3 becomes an ecosystem unto itself, like dBase or HyperCard . One thing I don't like about Lotus's approach is how it can bifurcate the user experience. That's seen clearly with their own WYSIWYG add-in. With Release 2.3, Lotus included this add-in to help a world transitioning from textual interfaces into the flash and sizzle of OS/2, Windows, and Mac GUI interfaces. It's DOS for the GUI envious and frankly, I'm cold on it. It's not integrated elegantly, feels sluggish, and makes the program more difficult to use. Activating WYSIWYG switches the application from terminal mode to graphics mode, so already as a DOSBox-X user I'm annoyed at losing my lovely TrueType text. That's not Lotus's fault, but a blogger's gotta have his standards. The big usability problem is how the functionality of the program now splits in two. The menu works as before, but we also have a new menu for all things WYSIWYG. So, when you want to use a menu command, you must remember which menu holds that command. Many options appear at first blush to be the same as their counterparts, but they control WYSIWYG-specific parameters of those functions. Usually. That's not to say the add-in isn't useful for cell styling, or placing graphs into a worksheet directly. Making documents look nice is important after all. The boss needs to be impressed with those Q3 projection charts, even when they forecast doom. Especially then, probably! Release 3 embraced WYSIWYG as its main and only interface, no add-in required, which is probably why I keep gravitating to the 2.x releases. I'd chalk it up to being a stubborn old man, but the recent embrace of TUI interfaces by the Hacker News crowd seems to have me in good company. I'm writing this part on February 22. Two days prior, a project called "Pi for Excel: AI sidebar add-in for Excel" released and got good traction on Hacker News. As I noted in the XPER column , our current "AI" boom is the biggest, but not the first. English language interactions, first by keyboard and fingers-crossed-one-day-by-voice-if-AI-technology-continues-along-our-projected-path-of-wishes-and-dreams, were available as add-ins to various programs. Databases in particular were a notable target for those experiments. Consider how English-like dBase 's user interface is, and it doesn't take a huge leap to understand why developers felt something closer to true English was within reach. Symantec's Q&A had its natural language "Intelligent Assistant" built right in. R:BASE tried it with their CLOUT add-in, promising a user could query, "Which warehouses shipped more red and green argyle socks than planned?" The spreadsheet Silk promised built-in English language control over its tools. Like those self-published magazines at the start of this article, Lotus didn't want to miss out on this English parser party either. (For this exploration I must drop down into R2.01) Released for US$150 in late 1986, HAL is a memory-resident wrapper to 1-2-3 . We launch HAL directly, which in turn launches 1-2-3 . Its advertising explains the gimmick well enough. "Lotus HAL gives you the ability to perform 1-2-3 tasks using simple English phrases." What I've seen in my early time with it can honestly feel kind of magical. Look at how easily it generates monthly column headers. 0:00 / 0:22 1× That's pretty slick, I can't deny it. Similarly tedious actions are promised to be eased greatly by "requesting" HAL to do the heavy lifting. Here, I'm stepping through a quick tutorial to have HAL build an entire spreadsheet. I never touch the formula; I only describe it by intent. 0:00 / 1:14 1× HAL only recognizes the first three letters of anything. "Name" and "Names" and "Namaste" are all the same to well-meaning, but a bit dimwitted, HAL. As is the case for all such English-like languages for the time, it's English only within a generous definition of the word. Ultimately, we're learning to speak 1-2-3 's specific dialect and vocabulary. PC Magazine , February 1987, their HAL review was the cover story, " HAL comes with a 250-page manual. It is as important to read this manual as it is to read the 1-2-3 manual. All the commands are described as rigidly as the syntax of any command-line interface." That it takes a 250 page manual to explain how to speak "English" with HAL perhaps makes an argument against its own existence? The base 640K of DOS must hold both programs in memory at the same time, so this is a nice piece of corroborating history for those who think software today is too bloated. An industry-defining spreadsheet with graphing and database capabilities close to modern expectations, an online help system, plus a natural language interface, all run together in less than 1MB of RAM . There's the retro-computing dopamine hit I've been hoping for! HAL doesn't just provide an English-language interface to 1-2-3 's native tools, it brings its own unique toys to the Release 2.01 sandbox. I do need to emphasize the release version here, because some of these tools were later worked into the product proper over time. That said, HAL worked hard to be your friend. Even though HAL controls 1-2-3 , interfacing with it still feels bolted on. brings up the HAL dialog box, which isn't hard to remember, but never feels natural. Even after setting the HAL request dialog to remain on screen, it feels tenuous. Sometimes it toggles off after navigating a menu option, or the request box will intercept commands I wanted to do through the normal slash menu. It's in the way more than I expected, and I couldn't find a balance between "when I want it" and "when I don't." PC Magazine also felt that HAL is a bit of a kludge. Charles Petzold wrote in his review, "Is HAL really a natural-language interface for 1-2-3 ? Is it useful? Will it revolutionize the computer industry? Are menus dead? My answers are: Not really. Often. Give me a break. No way." This is all academic, because Lotus killed HAL . It has been difficult to find sales figures, though in a Raymond Chen post we catch a glimpse of the Softsel Hot List for December 1986. HAL hit the top 10 (along with other, future blog subjects), moving up the charts over the previous three weeks. On the other hand, it was only available for Releases 1A through 2.01, the pre-WYSIWYG releases, and never returned. Earlier I poked at macros, hoping to make charting "count by genre" easier, and failed. Then I got to ponderin' if HAL might be able to do it for me. Shockingly, HAL can, through its special vocabulary word "tabulate." It makes those previously complex actions, the ones I diagrammed earlier, so simple to perform I don't really need a macro (though I could make one). Check out this 80's magic . 0:00 / 0:22 1× We are supposed to be able to execute HAL requests via to have the system output the 1-2-3 commands HAL puts together to get the job done. It's a peek inside HAL 's brain, basically. If I watch HAL think, maybe it can teach me a better way to do all of the busywork I slogged through earlier? In 1962's Diffusion of Innovations , author Everett Rogers described five characteristics individuals consider when adopting new solutions to existing problems. If VisiCalc was the "existing problem," how well did Lotus 1-2-3 make its case as the "new solution?" In the VisiCalc post I talked about how much of its DNA is seen in modern spreadsheets. I see now that an equal case can be made for Lotus 1-2-3 . I'd phrase it as VisiCalc contributed the "look," and 1-2-3 contributed the "feel" we've come to expect. Where VisiCalc was life-changing for number crunchers, 1-2-3 positioned itself as an engine for business and executed that vision almost perfectly. Having gotten to know 1-2-3 over the past weeks, I can now say, "I get it." I see what the fuss was about and, truth be told, I'm a convert. Sorry, VisiCalc , you know I love you! But the next time I reach for a spreadsheet, I'm reaching for 1-2-3 . Ways to improve the experience, notable deficiencies, workarounds, and notes about incorporating the software into modern workflows (if possible). Obviously, it depends on what you're trying to do. For business work, it doesn't play well in groups unless you're the CEO and can dictate, "OK people, we're all switching to DOS now." For personal projects, it meets many common needs and doesn't feel too much like compromise, aside from the graphing. Heck, the DOS version supports mouse control, and you can always turn on WYSIWYG mode to approximate modernity. We're also in luck with Y2K compatibility. Even Release 1.0 supports dates up to the year 2099. Let's take a moment of silent appreciation for yet another 1-2-3 foresight which keeps its spirit alive and kicking here in the 21st century. DOSBox-X 2026.01.02, Windows x64 build. I updated from the 2025.12 build mid-investigation. CPU set to 286 DOS reports as v6.22 Windows folder mounted as drive C:\ holds multiple Lotus installations 2x (forced) scaling; 80 columns x 25 lines I flipped back and forth with TrueType text mode (this is moot for 1-2-3 's WYSIWYG mode) Lotus 1-2-3 Releases 2.01, 2.2, 2.3, 2.4, and 3.4 all get exercised to some extent; you'll see that reflected in the screenshots. I mostly gravitate toward R2.3; it does what I need without bogging me down in feature creep. "Sharpening the Stone" explains getting DOSBox-X to work with R3.x. dBase III Plus for compatibility testing with 1-2-3 . Undoing your last action. It's almost worth installing HAL just for this, though it is a little dangerous that is the keyboard shortcut. Entering a sequential list of days, months, letters, or numbers automatically, though I wonder if macros could duplicate this to a certain degree. Linking a cell in one worksheet to data in another. Release 2.3 has this. Referring to columns and rows by name is a very neat trick. In fact, it's so neat I'm going to ask you to remember this fact for a later article. Just keep it tucked away in the part of your mind devoted to spreadsheet history, as we all have. The cell-row-bellum, I think its called? (I refuse to apologize.) Worksheet "auditing" can identify cell relationships/dependencies, or list out all formulas in use by a table in natural English. Auditing would become an add-in in later 2.x releases. Find and replace; change all instances of a product name, for example. Macros can mix HAL English with native 1-2-3 macro commands. "Relative advantage  is the degree to which an innovation is perceived as better than the idea it supersedes." 1-2-3 received applause for one-button graphing. Check. "Compatibility  is the degree to which an innovation is perceived as being consistent with...past experiences, and needs of potential adopters." 1-2-3 shipped with a VisiCalc translation tool and its interface is clearly built to make VisiCalc users comfortable. Check. " Complexity  is the degree to which an innovation is perceived as difficult to understand and use." 1-2-3 was initially praised for the simplicity with which a user could get up to speed. Its adoption of high-level VisiCalc concepts, like the slash menu, @ functions, and A1 cell references, helped. Check. "Trialability  is the degree to which an innovation may be experimented with on a limited basis." Trial disks for software during the 80's and 90's wasn't so prevalent; there was a lot of "blind faith" in software purchasing. I can't find any widespread cases of 1-2-3 demo disks circulating. No check. " Observability  is the degree to which the results of an innovation are visible to others." If the live demos, prevalent advertising, and magazine write-ups didn't convince you, 1-2-3 made it clear in the product name itself that you're getting 3x what VisiCalc delivers. Check. As with ThinkTank , DOSBox-X provided a simple, pain-free experience to get Lotus running. Multi-disk installs are handled well, but could be improved. Specifically, the "Swap Disk" option when loading up a stack of disks into the A: drive could use a selector and/or indicator of which disk is currently loaded. in autoexec.bat to auto-mount at launch. Revision 3.4 would not run until I explicitly set in DOSBox-X. I noted the pie graph bug in Release 2.x. I suspect, but cannot prove, that some x86 assembly call is being mangled by DOSBox-X. 86Box, which strives to be as pedantically accurate a simulation of real-world hardware as possible, does not exhibit this issue. However, setting up 86Box comes with a whole day of learning about the parts and pieces of assembling one's own raw DOS system from virtual components, installing from diskettes, and all of the old-school troubleshooting that entails. It's a commitment, is what I'm saying. I found that DOSBox-X would run the for Release 2.2, but failed to run it for Releases 2.3 and 2.4. can launch and run without issue. is a front-end utility to launch auxiliary programs like GraphPrint . If you're mounting a system folder as a "hard drive" in DOSBox-X, it is trivial to extract your data files. The Lotus utility "Translate" is handy for moving data between formats. I found that native .wk1 files open in LibreOffice , as-is. From there, you have any number of modern exporting options, though you might find some quirks from time to time. Check your formulas, just in case! I'd recommend checking out Travis Ormandy 's site. He's smarter than me and performs magic I didn't think possible, like pulling live stock data as JSON into 1-2-3 . He also got the Unix build to work natively in Linux.

Data Visualization Business Scala Culture JSON

xenodium 1 months ago

Bending Emacs - Episode 13: agent-shell charting

Time for a new Bending Emacs episode. This one is a follow-up to Episode 12 , where we explored Claude Skills as emacs-skills . Bending Emacs Episode 13: agent-shell + Claude Skills + Charts This time around, we look at inline image rendering in agent-shell and how it opens the door to charting. I added a handful of new charting skills to emacs-skills : /gnuplot , /mermaid , /d2 , and /plantuml . The agent extracts or fetches data from context, generates the charting code, saves it as a PNG, and agent-shell renders it inline. Cherry on top: the generated charts match your Emacs theme colors by querying them via . Hope you enjoyed the video! Liked the video? Please let me know. Got feedback? Leave me some comments . Please like my video , share with others, and subscribe to my channel . As an indie dev, I now have a lot more flexibility to build Emacs tools and share knowledge, but it comes at the cost of not focusing on other activities that help pay the bills. If you benefit or enjoy my work please consider sponsoring .

Data Visualization Tutorial Shell

Ginger Bill 3 months ago

Mitigating the Billion Dollar Mistake

This article is continuation to: Was it really a Billion Dollar Mistake? . After reading a lot of the comments on numerous social media sites on the original article , I think I need to clarify a lot more. The main points I wanted to clarify: A lot of commentors based their complaints in their experience with languages like Java/C#/Python/etc, and the issues with null-pointer-exceptions (NPEs) in them. What I think a lot of people seemed to forget is that in those languages, virtually everything is a pointer, unlike in a language like C/Go/Odin which has explicit pointers. When everything is a pointer, it is exponentially more likely that you will hit a pointer that is invalid. And in the case of a managed (garbage collected) language, that invalid pointer will most definitely be a null pointer. This is why I can understand the problem of having pointers in such languages. But I think this still missed the point of what I trying to state, that the reason even exists in those languages is because you can declare a variable without an explicit initialization value: Because you can declare such a thing in a language like Java, then there are three options to try and mitigate this design flaw: Unfortunately existing languages like Java cannot have these problems solved, but newer languages that want to stylize themselves similar to that could solve them. One of the issues is that languages like Java added maybe/option/optional types too late AND it is not the default behaviour. The first approach is the current status quo, the second approach keeps the implicit value declarations but adds more checks, whilst the third approach requires doing explicit value declarations. The enforcement of maybe types as the default pointer/reference type leads to two possibilities: Version 1 would be something like this: but because of the ergonomic pains, can also lead to unwrapping cases, which are practically equivalent to NPEs: At least with an , it is a little clearer that a panic could happen. But it can also just be an early-out too like with Odin’s : Version 2 is a bit weirder, since it doesn’t remove the concept of but propagates further up the expression tree. The first approach is unergonomic to use, especially in a language where virtually everything is a pointer/reference, and with the addition of unwrapping which just panics on , it’s practically reinvented NPEs with more steps. As for the second approach, I’d argue is very bug prone if it was the default, since you cannot trivially know where the arose from since it was just passed up the stack 2 . Therefore most people think the third approach to mitigating pointers is the “obvious” and “trivial” approach: explicit individual initialization of every value/element everywhere . One thing which I commonly saw was people saying was that I “missed the point” that null safety is not about protecting from common invalid memory access but rather it’s about clarifying the states that a pointer can be in the type system itself, whether it cannot be null or maybe it could be null. I already knew this, and I find it bizarre 3 that people did not understand that from the article. The point I was trying to get across which most people seemed to either ignore or not understand was that the approach of requiring explicit initialization of every element everywhere comes with a cost and trade-offs. Most people who bring this up as “the solution” think there was either no cost or they think the cost is worth it. The former group are just wrong, and the latter group are the point I was focusing the article at in the first place: you don’t actually understand the costs fully if you are answering the way that you do. I understand this sounds “condescending” to some people, but I am not trying to be. The point I am arguing is far from the common view/wisdom, and thus I tried to explain my position. Why would a person listen to someone with a “fringe” view? “Fringe” views are typically wrong in other areas of life, so it makes sense to apply that heuristic to the domain of programming too. I don’t care if people agree with me or not, rather I wish people actually understand it and then comment. But as a systems programmer, I deal with memory all the time, and null pointers are the least common kind of invalid memory that I have to deal with, and the other kinds were not handled by the type system, nor would be handled with solving the problems of null. No, this is not saying “well just because you cannot solve problem X with Y, therefore don’t solve either”, it’s saying that they are different problems, and empirically they are just different with different kinds of severity and ways to mitigate them. I am not saying you shouldn’t try to solve either problem if you are designing your own language, but rather they are both kinds of invalid memory, but solutions to mitigate the problems are completely different in kind 4 . For a managed language like Java, the cost of explicit initialization of every element everywhere is so little in comparison to the rest of the language, that approach is honestly fine. But for a language like the one I have designed and created—Odin—the cost of non-zero initialization is extremely costly as things scale. This simple/naïve approach looks like this in a pseudo-C: But if you use a lot of pointers everywhere, the initialization becomes a lot more complex, and non-linear too. People argue the need to express non-nullable pointers, and either version 1 of the previous approach or this explicit approach are effectively the only ways of doing this. You could tell the compiler to assume the pointer is never null (e.g. or ), but those are not guarantees in the type system, just you telling the compiler to assume it is never . The non-nullability is not possible outside of those two approaches. This was the entire point I was making between the Individual-Element Mindset and the Group-Element Mindset is that the individual-element mindset lends itself well to thinking about individual elements like this. And as such, it doesn’t really think about the cost of thinking in individual elements as compounding to something expensive. I’ve been in projects where a lot of the time in a program in spent in the destructors/ traits of individual elements, when all they are doing is trivial things which could have been trivially done in bulk. Most people don’t consider these as “costs” nor that there are trade-offs to this approach to programming, rather it’s “just the way it is”. There is the other aspect where if the explicit initialization is applied to every type, not just ones which contains pointers/references, then it can be less ergonomic to type and have visual noise: 5 This constant syntactic noise can be tiring and detracts from what is actually going on. With the implicit zero initialization that I had in Odin, it has worked out really well. Many might expect it to be confusing, but it isn’t and you can rely on it and becomes very natural to use. As the creator and main architect of Odin, a lot of Odin’s design has been to fix a lot of the problems I and many others faced with C, whilst still not veering too far from the general feel of C. Odin does have pointers by default, but in practice they are a very rare problem due numerous features and constructs of the language. One of the reasons for pointers in C is caused to due the lack of a proper array type. Odin has proper array types and does not implicitly demote arrays to pointers. Odin has slices which replace a lot of the needs for pointers and pointer arithmetic, and because array types (including slices) are bounds checked, that already solves many of the problems that would have occurred in C with treating pointers as arrays, which may or may not have an associated length to check against. Odin also has tagged unions and multiple return values. Tagged unions should be “obvious” to the people who had be complaining about the initial article, but the use of tagged unions isn’t necessarily there to solve the pointer problem. Odin’s is an example of a maybe/option type, which is just a built-in discriminated union, with the following definition: And due to the design of Odin’s , if a union only has one variant and that variant is any pointer-like type, no explicit tag is stored. The state of the pointer-like value also represents the state of the . This means that . Another reason why C has problems with pointers is the lack of way to state a parameter to a procedure as being optional. C doesn’t have default values for parameters, nor any way in its type system to express this. C’s type system is just too poor and too weak. This is why people unfortunately use pointers as a way to do thus, since they can write . However, it is rare to see in Odin code be used to indicate pointers except when interfacing with foreign code, or optional parameters to a procedure. This is because the need for a pointer itself is quite rare. There are multiple reasons why: However one of the main reasons why pointers are rarely a problem in Odin is because of multiple return values. Multiple return values when used for this manner, are akin (but not semantically equivalent) to something like a type in other languages 6 . When a procedure returns a pointer, it is either assumed to be never OR accompanied with another value to indicate its validity, commonly in the form of a boolean or : And coupled with the constructs ( , , , ), , and named return values, a lot of those issues never arise: Odin is designed around multiple return values rather than / constructs, but this approach does in practice does solve the same kinds of problems. Before people go “well the assumption is not enforced in the type system”, remember where all of this derives from: Odin allows for implicit declarations of variables without an explicit initialization value. And as the designer of Odin, I think enforcing that is both quite a high cost (see the individual-element vs grouped-elements mindsets) and far from the original approach to programming C. I know this is not going to convince people, but it’s effectively trying to make someone think like another person, which is never easy, let alone always possible to do in the first place. And it’s not a mere “aesthetic preference” either. This very little design decision has MASSIVE architectural consequences which lead to numerous performance problems and maintenance costs as a project grows. Null pointer exceptions (NPEs) are in a category of constructs in a language which I class as “panic/trap on failure”. What I find interesting is that there are numerous other things in this category, but many people will normally take a different approach to those constructs compared to NPEs, due to whatever reason or bias that they have. The canonical example is integer division by zero. Instinctually, what do you think division by zero of an integer should result it? I’d argue most people will say “trap”, even if a lot of modern hardware (e.g. ARM64 and RISC-V) does not trap, and only the more common x86-related architectures do trap. Odin does currently 7 define the behaviour of division by zero to “trap” only because of this assumption, but we have considered changing this default behaviour. Odin does allow the programmer to control this behaviour at a global level or on a per-file level basis if they want a different behaviour for division by zero (and consequentially by zero). But some languages such as Pony , Coq, Isabelle, etc actually define division by zero of an integer to be . This is because it can help a lot of theorem provers . But there is the other question of production code. One of the main arguments against NPEs (especially in languages like Java) is that it causes a crash. So in the case of division by zero, do you want this to happen? Or would you prefer all integer division to be explicitly handled, or default to a predictable/useful value, like ?—which prevents crashing in the first place. Another common example of “panic on failure” is languages with runtime bounds checking. If is out of bounds, most languages just panic. It’s rare to find a language that returns a on every array access to prevent an out of bounds. Not even languages like OCaml do this. NPEs, division by zero (if traps), and runtime bounds checking are all examples of this kind of “panic on failure”, but people rarely treat them as being the same, even if they are of the same kind of problem. Honestly, no. I understand it might be common for beginners to a language like C to have many pointer related problems, but they will also have loads of other problems too. However as you get more competent at programming, that kind of problem is honestly the least of your problems. I honestly think a lot of this discussion is fundamentally a misunderstanding of different perspectives rather than anything technical. A lot of what some people think are their “technical opinions” are merely just aesthetic judgements. And to be clear, aesthetic judgements are not bad, but they are not necessarily technical. But I’d argue most people are not applying their opinions consistently when it comes to the category of “panic on failure”, and NPEs are no different; they only seem more of a problem to them either because of the existence of the name of the “Billion Dollar Mistake” or because they encounter it more. I know a lot of people view the explicit individual initialization of every element everywhere approach as the “obvious solution”, as it seems like low-hanging fruit. As a kid, I was told to not pick low-hanging fruit, especially anything below my waist. Just because it looks easy to pick, a lot of it might not be unpicked for a reason. It does not mean that you should or should not pick that fruit, but rather you need to consider the trade-offs. If you honestly think the costs of explicit individual initialization of every element everywhere are worth it for the language you are working in or developing, then great! But at least know the trade-offs of that approach. For Odin, I thought it was not worth the cost—compared to the alternative ways of mitigating the problem empirically. Most of the bad criticisms just came from people who didn’t read the article or didn’t read past a couple paragraphs. That’s why I wanted to state this comment very clearly.  ↩︎ This is partially why I do not like exceptions as error handling in many languages. It is not obvious where things are thrown/raised from and they encourage the practice of ignoring them until the latest possible space. I discuss that problem in The Value Propagation Experiment Part 2 .  ↩︎ I understand what type systems do and their benefits, and it is a little insulting when people assume my knowledge (or lack of) without doing a modicum of review.  ↩︎ In the case of the other invalid memory addresses, linear/affine substructural type systems with lifetime semantics can help with this (e.g. Rust) but they come at another cost in terms of language ergonomics and restrictions. Language design is hard.  ↩︎ I know typing is never the bottleneck in programming, but the visual noise aspect is a big one when you are trying to scan (not necessarily read ) code. I want to see the pattern and not be swamped with syntactic noise.  ↩︎ I know a result type is a kind of sum type and multiple return values are more akin to a product type, but how different languages want to be used and expressed, this works out fine in practice for the same kinds of problems. Please don’t give me a FP rant.  ↩︎ At the time of writing, I am not sure which approach is the better one: trap or zero by default, but we allow for all four options in the Odin compiler. Division by zero for floats results in “Inf” and that’s not necessarily as much of a problem in practice, so why would division by zero be as bad?  ↩︎ Null pointer dereferences are empirically the easiest class of invalid memory addresses to catch at runtime, and are the least common kind of invalid memory addresses that happen in memory unsafe languages. I do think it was a costly mistake but the “obvious solutions” to the problem are probably just as costly , if not more so, but in very subtle ways which most people neglected to understand in the article 1 . I think that even if Tony Hoare didn’t “invent” pointers, within a couple years someone else would have. I don’t think it’s a “mistake” the programming world was ever going to avoid. I am talking about languages that run on modern systems with virtual memory, not embedded systems where you interact with physical memory directly. Those platforms in my opinion need much different kinds of languages which unfortunately do not exist yet. I was also talking about languages akin to C and Odin, not languages that run on a VM or have “everything be a reference”. Allow for pointers (and just deal with it) All pointers are implicitly maybe types (e.g. in Java) Require all explicit initialization of every element everywhere to assume cannot happen, along with things like maybe types. Requiring each reference to be checked if it is . Check if a value is and propagate that up the expression tree. Odin has slice types Odin has multiple return values to allow for out-only parameters, which could be ignored with Odin isn’t a “everything is a pointer” kind of language: pointers have to be explicit typed to exist. Writing pointer types as value declarations is less common due to type inference e.g. is more much common than: . All bits set ( ) The same value ( ) Most of the bad criticisms just came from people who didn’t read the article or didn’t read past a couple paragraphs. That’s why I wanted to state this comment very clearly.  ↩︎ This is partially why I do not like exceptions as error handling in many languages. It is not obvious where things are thrown/raised from and they encourage the practice of ignoring them until the latest possible space. I discuss that problem in The Value Propagation Experiment Part 2 .  ↩︎ I understand what type systems do and their benefits, and it is a little insulting when people assume my knowledge (or lack of) without doing a modicum of review.  ↩︎ In the case of the other invalid memory addresses, linear/affine substructural type systems with lifetime semantics can help with this (e.g. Rust) but they come at another cost in terms of language ergonomics and restrictions. Language design is hard.  ↩︎ I know typing is never the bottleneck in programming, but the visual noise aspect is a big one when you are trying to scan (not necessarily read ) code. I want to see the pattern and not be swamped with syntactic noise.  ↩︎ I know a result type is a kind of sum type and multiple return values are more akin to a product type, but how different languages want to be used and expressed, this works out fine in practice for the same kinds of problems. Please don’t give me a FP rant.  ↩︎ At the time of writing, I am not sure which approach is the better one: trap or zero by default, but we allow for all four options in the Odin compiler. Division by zero for floats results in “Inf” and that’s not necessarily as much of a problem in practice, so why would division by zero be as bad?  ↩︎

Data Visualization Java Programming Tutorial C#

Alex White's Blog 4 months ago

Babbling About Solutions

I've been in so many meeting where people will discuss a tiny point to death over the course of hours/days. For example, at numerous companies there have been debates around time axis graphs and the current date. There's always someone who thinks users will be confused that the visual for the current day is less than the previous days (because the day is in progress). Additionally, the topic of timezones will come up. "Our data is in X timezone, but the user is in Y, the graph will be confusing". I was reminded of this example as I built out a "daily visitors" graph for my analytics page. I chuckled as I just implemented a solution, without hours of debate and Y salaries * Z hours of money wasted. My solution to the two problems was this: Nothing is ever perfect and every UI will confuse someone, somewhere, somehow. The key is to see if there's a significant amount of data indicating the UI is confusing, not to debate the tiny details to death before even releasing something. Show visitor count as "X visitors in 24 hours". You're not commiting to a day in a timezone, instead it's relative by hours. For the graph, use the wording "Until Now" to represent the partial nature of the value. Give users some credit and know your audience.

Analytics Data Visualization Design

Nicky Reinert 5 months ago

How to create an animated ticker in Excel without VBA

How to create an animated ticker in Excel without VBA? Last night, around 3 am, I woke up and thought: Wouldn’t it be nice to have an animated ticker in Excel? Like text that scrolls from right to left? With just one big formula? Welcome to the second episode of my series “How to create weird …

Tutorial Data Visualization

DYNOMIGHT 5 months ago

Pointing machines, population pyramids, post office scandal, type species, and horse urine

I recently wondered if explainer posts might go extinct. In response, you all assured me that I have nothing to worry about, because you already don’t care about my explanations—you just like it when I point at stuff. Well OK then! How did Michelangelo make this ? What I mean is—marble is unforgiving. If you accidentally remove some material, it’s gone. You can’t fix it by adding another layer of paint. Did Michelangelo somehow plan everything out in advance and then execute everything perfectly the first time, with no mistakes? I learned a few years ago that sculptors have long used a simple but ingenious invention called a pointing machine . This allows you to create a sculpture in clay and, in effect, “copy” it into stone. That sounds magical, but it’s really just an articulated pointer that you move between anchor points attached to the (finished) clay and the (incomplete) stone sculpture. If you position the pointer based on the clay sculpture and then move it to the stone sculpture, anything the pointer hits should be removed. Repeat that thousands of times and the sculpture is copied. I was sad to learn that Michelangelo was a talentless hack, but I dutifully spent the last few years telling everyone that all sculptures were made this way and actually sculpture is extremely easy, etc. Last week I noticed that Michelangelo died in 1564, which was over 200 years before the pointing machine was invented. Except , apparently since ancient times sculptors have used a technique sometimes called the “compass method” which is sort of like a pointing machine except more complex and involving a variety of tools and measurements. This was used by the ancient Romans to make copies of older Greek sculptures. And most people seem to think that Michelangelo probably did use that. I think this is one of the greatest data visualizations ever invented. Sure, it’s basically just a histogram turned on the side. But compare India’s smooth and calm teardrop with China’s jagged chaos . There aren’t many charts that simultaneously tell you so much about the past and the future. It turns out that this visualization was invented by Francis Amasa Walker . He was apparently such an impressive person that this invention doesn’t even merit a mention on his Wikipedia page, but he used it in creating these visualization for the 1874 US atlas : I think those are the first population pyramids ever made. The atlas also contains many other beautiful visualizations, for example this one of church attendance : Or this one on debt and public expenditures : If you haven’t heard about the British Post Office scandal , here’s what happened: In 1999, Fujitsu delivered buggy accounting software to the British Post Office that incorrectly determined that thousands of subpostmasters were stealing. Based on this faulty data, the post office prosecuted and convinced close to a thousand people, of whom 236 went to prison. Many others lost their jobs or were forced to “pay back” the “shortfalls” from their own pockets. Of course, this is infuriating. But beyond that, I notice I am confused. It doesn’t seem like anyone wanted to hurt all those subpostmasters. The cause seems to be only arrogance, stupidity, and negligence. I would have predicted that before you could punish thousands of people based on the same piece of fake evidence, something would happen that would stop you. Obviously, I was wrong. But I find it hard to think of good historical analogies. Maybe negligence in police crime labs or convictions of parents for “shaken baby syndrome”? Neither of these is a good analogy. One theory is that the post office scandal happened because the post office—the “victim”—had the power to itself bring prosecutions. But in hundreds of cases things were done the normal way, with police “investigating” the alleged crimes and then sending the cases to be brought by normal prosecutors. Many cases were also pursued in Scotland and Northern Ireland, where the Post Office lacks this power. Another theory would be: Prosecutors have incredible latitude in choosing who they want to prosecute. Like other humans, some prosecutors are arrogant/stupid/negligent. It’s actually pretty easy for prosecutors to convict an innocent person if they really want to, as long as they have some kind of vaguely-incriminating evidence. Under this theory, similar miscarriages of justice happen frequently. But they only involve a single person, and so they don’t make the news. Type species - Wikipedia I link to this not because it’s interesting but because it’s so impressively incomprehensible. If there’s someone nearby, I challenge you to read this to them without losing composure. In zoological nomenclature, a type species ( species typica ) is the species whose name is considered to be permanently taxonomically associated with the name of a genus or subgenus. In other words, it is the species that contains the biological type specimen or specimens of the genus or subgenus. A similar concept is used for groups ranked above the genus and called a type genus. In botanical nomenclature, these terms have no formal standing under the code of nomenclature, but are sometimes borrowed from zoological nomenclature. In botany, the type of a genus name is a specimen (or, rarely, an illustration) which is also the type of a species name. The species name with that type can also be referred to as the type of the genus name. Names of genus and family ranks, the various subdivisions of those ranks, and some higher-rank names based on genus names, have such types. In bacteriology, a type species is assigned for each genus. Whether or not currently recognized as valid, every named genus or subgenus in zoology is theoretically associated with a type species. In practice, however, there is a backlog of untypified names defined in older publications when it was not required to specify a type. Can such a thing be created unintentionally? I tried to parody this by creating an equally-useless description of an everyday object. But in the end, I don’t think it’s very funny, because it’s almost impossible to create something worse than the above passage. A funnel is a tool first created in antiquity with rudimentary versions fabricated from organic substrates such as cucurbitaceae or broadleaf foliage by early hominid cultures. The etymology of fundibulum (Latin), provides limited insight into its functional parameters, despite its characteristic broad proximal aperture and a constricted distal orifice. Compositionally, funnels may comprise organic polymers or inorganic compounds, including but not limited to, synthetic plastics or metallic alloys and may range in weight from several grams to multiple kilograms. Geometrically, the device exhibits a truncated conical or pyramidal morphology, featuring an internal declination angle generally between 30 and 60 degrees. Within cultural semiotics, funnels frequently manifest in artistic representations, serving as an emblem of domestic ephemerality. The good news is that the Sri Lankan elephant is the type species for the Asian elephant, whatever that is. I previously mentioned that some hormonal medications used to be made from the urine of pregnant mares. But only after reading The History of Estrogen Therapy (h/t SCPantera ) did I realize that it’s right there in the name: Premarin = PREgnant MARe’s urINe If you—like me—struggle to believe that a pharmaceutical company would actually do this, note that was in 1941. Even earlier, the urine of pregnant humans was used. Tragically, this was marketed as “Emmenin” rather than “Prehumin”. Prosecutors have incredible latitude in choosing who they want to prosecute. Like other humans, some prosecutors are arrogant/stupid/negligent. It’s actually pretty easy for prosecutors to convict an innocent person if they really want to, as long as they have some kind of vaguely-incriminating evidence.

Data Visualization

Cassidy Williams 6 months ago

2000 Poops

Flash back to Spring 2020, when we were all confused and uncertain about what the world was going to look like, and unsure of how we would stay connected to each other. One of my cousins texted our cousin group chat mentioning the app Poop Map as a cheeky (heh) way of keeping up with the fam. We started a family league, and it was honestly pretty great. We’d congratulate each other on our 5-star poops, and mourn the 1-stars. Over time I made other leagues with friends online and offline, and it was really fun. I even talked about it on Scott Hanselman’s podcast when he asked about how to maintain social connections online (if you wanna hear about it, listen at the 11 minute mark in the episode). Eventually, people started to drop off the app, because… it’s dumb? Which is fair. It’s pretty dumb. But alas, I pride myself in being consistent, so I kept at it. For years. The last person I know on the app is my sister-in-law’s high school friend, also known by her very apt username, . She and I have pretty much no other contact except for this app, and yet we’ve bonded. 2000 poops feels like a good place to stop. With 12 countries covered around the world and 45 achievements in the app (including “Are you OK?” courtesy of norovirus, and “Punctuate Pooper” for going on the same day for 12 months in a row), I feel good about saying goodbye. My mom is also really happy I’m stopping. Wonder why? Anyway, goodbye, Poop Map, and goodbye to the fun usernames for the friends along the way: (that’s me), , , , , , , , , , , , , , , , , and of course, . Also, before you go, here’s a fun data visualization I made of all my entries ! Smell ya later!

Java Entertainment Data Visualization

Simon Willison 7 months ago

Recreating the Apollo AI adoption rate chart with GPT-5, Python and Pyodide

Apollo Global Management's "Chief Economist" Dr. Torsten Sløk released this interesting chart which appears to show a slowdown in AI adoption rates among large (>250 employees) companies: Here's the full description that accompanied the chart: The US Census Bureau conducts a biweekly survey of 1.2 million firms, and one question is whether a business has used AI tools such as machine learning, natural language processing, virtual agents or voice recognition to help produce goods or services in the past two weeks. Recent data by firm size shows that AI adoption has been declining among companies with more than 250 employees, see chart below. (My first thought on seeing that chart is that I hope it represents the peak of inflated expectations leading into the trough of dissillusionment in the Gartner Hype Cycle (which Wikipedia calls "largely disputed, with studies pointing to it being inconsistently true at best"), since that means we might be reaching the end of the initial hype phase and heading towards the slope of enlightenment .) This is the first I'd heard of the US Census Bureau running a biweekly (that's once every two weeks) survey about AI!

AI Python Data Visualization

Entropic Thoughts 7 months ago

Wiggling Into Correlation

Statistics Data Visualization Data Analysis

Nicole Tietz-Sokolskaya 8 months ago

Visualizing distributions with pepperoni pizza (and javascript)

There's a pizza shop near me that serves a normal pizza. I mean, they distribute the toppings in a normal way. They're not uniform at all. The toppings are random, but not the way I want. The colloquial understanding of "random" is kind of the Platonic ideal of a pizza: slightly chaotic but things are more or less spread out over the whole piece in a regular way. If you take a slice you'll get more of less the same amount of pepperoni as any other slice. And every bite will have roughly the same amount of pepperoni as every other bite. I think it would look something like this. Regenerate this pie! This pizza to me is pretty much the canonical mental pizza. It looks pretty random, but you know what you're gonna get. And it is random! Here's how we made it, with the visualiztion part glossed over. First, we make a helper function, since gives us values from 0 to 1, but we want values from -1 to 1. Then, we make a simple function that gives us the coordinates of where to put a pepperoni piece, from the uniform distribution. And we cap it off with placing 300 fresh pieces of pepperoni on this pie, before we send it into the oven. (It's an outrageous amount of very small pepperoni, chosen in both axes for ease of visualizing the distribution rather than realism.) But it's not what my local pizza shop's pizza's look like. That's because they're not using the same probability distribution. This pizza is using a uniform distribution . That means that for any given pepperoni, every single position on the pizza is equally likely for it to land on. We are using a uniform distribution here, but there are plenty of other distributions we could use as well. One of the other most familiar distributions is normal distribution . This is the distribution that has the normal "bell curve" that we are used to seeing. And this is probably what people are talking about most of the time when they talk about how many standard deviations something is away from something else. So what would it look like if we did a normal distribution on a pizza? The very first thing we need to answer that is a way of getting the values from the normal distribution. This isn't included with JavaScript by default, but we can implement it pretty simply using the Box-Muller transform . This might be a scary name, but it's really easy to use. Is a way of generating numbers in the normal distribution using number sampled from the uniform distribution. We can implement it like this: Then we can make a pretty simple function again which gives us coordinates for where to place pepperoni in this distribution. The only little weird thing here is that I scale the radius down by a factor of 3. Without this, the pizza ends up a little bit indistinguishable from the uniform distribution, but the scaling is arbitrary and you can do whatever you want. And then once again we cap it off with a 300 piece pepperoni pizza. Regenerate this pie! Ouch. It's not my platonic ideal of a pizza, that's for sure. It also looks closer to the pizzas my local shop serves, but it's missing something... See, this one is centered around, you know, the center . Theirs are not that. They're more chaotic with a few handfuls of toppings. What if we did the normal distributions, but multiple times, with different centers? First we have to update our position picking function to accept a center for the cluster. We'll do this by passing in the center and generating coordinates around those, while still checking that we're within the bounds of the circle formed by the crust of the pizza. And then instead of one single loop for all 300 pieces, we can do 3 loops of 100 pieces each, with different (randomly chosen) centers for each. Regenerate this pie! That looks more like it. Well, probably. This one is more chaotic, and sometimes things work out okay, but other times they're weird. Just like the real pizzas. Click that "regenerate" button a few times to see a few examples! So, this is all great. But, when would we want this? I mean, first of all, boring. We don't need a reason except that it's fun! But, there's one valid use case that a medical professional and I came up with [1] : hot honey [2] . The ideal pepperoni pizza just might be one that has uniformly distributed pepperoni with normally distributed hot honey or hot sauce. You'd start with more intense heat, then it would taper off as you go toward the crust, so you maintain the heat without getting overwhelmed by it. The room to play here is endless! We can come up with a lot of other fun distributions and map them in similar ways. Unfortunately, we probably can't make a Poisson pizza, since that's a distribution for discrete variables. I really do talk about weird things with all my medical providers. And everyone else I meet. I don't know, life's too short to go "hey, this is a professional interaction, let's not chatter on and on about whatever irrelevant topic is on our mind." ↩ The pizza topping, not my pet name. ↩

JavaScript Data Visualization

blog.philz.dev 8 months ago

Infrastructure as Code for Grafana Dashboards

This post came about from some work I and others did at Resolve.ai ; check them out for your agentic on-call needs! I'm sharing it with you with their kind permission. Checking dashboards into gets you all the usual "infrastructure as code" advantages: for loops, variables, version control, consistency. The essence of a dashboard is the queries that power the visualizations. More often than not, the visualizations themselves are similar across many queries. Writing the dashboards as code lets you focus on the essence—the queries—and re-use the styling. This post does that with Grafana and TypeScript. I chose to use TypeScript to define the dashboards, so as to embed the dashboards within a language and tooling ecosystem we already know well. (Others may choose to use the Terraform provider or JSONnet ). TypeScript's type system and language server are a real advantage in working with Grafana's APIs, because there are exist good types for the surface area. Grafana's Foundation SDK has types for many Grafana dashboard concepts as well as examples . The JSON model for dashboards is documented as part of Grafana's API documentation . A Grafana dashboard has 3 main components: Rows . These visually separate groups of metrics. Panels . These are the visualizations you place in your dashboard. The Grafana grid is composed of 24 columns, and each "height" unit represents 30 pixels. The grid has negative gravity, which means that a panel slides upwards to empty space, like an upside down game of Tetris. If you want three charts per row, you use a width of 8, and if you want two, use a width of 12. Using "4! = 24" as a basis gives the chart lots of divisors for layout options! We've found that if you just specify a height and width in your panels, Grafana lays them out in order nicely enough. The panels are where all the action is, and there are many, many panel types. This folder has many panel types, including the very popular "timeseries", "text", "piechart", and so forth. For the most part, Grafana's JSON system has sensible defaults, so you don't need to specify all possible properties. This is a big win of using the JS bindings over checking in the "expanded" JSON directly. (We've found that the Cloudwatch panel is pretty picky and doesn't work if you don't specify nearly everything.) Now that we sort of understand's Grafana's nouns, we can build out a dashboard in code. It's very likely that you want a lot of panels of all the same type, so you define something like and invoke it many times. If you need to do advanced things, you can do one manually in the UI, and then find the "Inspect…Panel Json" action on every Grafana Panel to dig in. Most of your dashboards will look something like this. The rest is boilerplate at the per-dashboard and overall layers: We can now look at the end to end example, annotated slightly. You’ll need a Grafana bearer token to run this against your instance. Here are the key files you'll need: This is the dashboard it generates: Here's the TypeScript code that generates this dashboard: Happy monitoring! Coding agents are great at modifying the code above. Give your favorite (I'm partial to Sketch ) agent the Grafana keys, and let it do its thing. A dashboard is, in essence, an array of (title, query) pairs: you can get pretty close to that essence. The styling of the panels within that dashboard is typically common! Use the sample code below to programmatically create Grafana dashboards and alerts. Use your company's common programming language to define dashboards for great developer ergonomics and lower barriers to entry. Grafana dashboard panels play an upside down game of Tetris! Variables . These appear at the top and can be used to drill down to specific instances of your infrastructure. They're used within the individual PromQL queries, and Grafana does a great job of letting you specify a metric to grab the possible values. Rows . These visually separate groups of metrics. Panels . These are the visualizations you place in your dashboard. The Grafana grid is composed of 24 columns, and each "height" unit represents 30 pixels. The grid has negative gravity, which means that a panel slides upwards to empty space, like an upside down game of Tetris. If you want three charts per row, you use a width of 8, and if you want two, use a width of 12. Using "4! = 24" as a basis gives the chart lots of divisors for layout options! We've found that if you just specify a height and width in your panels, Grafana lays them out in order nicely enough. - Dependencies and npm scripts - TypeScript configuration - The main script (shown below) (optional) - ESLint setup for TypeScript

JSON Data Visualization DevOps Programming JavaScript

Weakty 9 months ago

Signs of Positivity in Toronto

Signs of positivity have been hiding in plain sight around a few neighbourhoods in Toronto. I've been seeing these signs for at least a year, perhaps longer. Now that I'm back in the season of running and training for a race in the fall, I've started to re-notice these signs and have decided to document them, snapping photos of them whenever I pass them on my runs. When you get up close the signs depict a series of images that feel very positive to me. Some of them visually sound out their message. Some of the pieces are more abstract and leave me wondering. What are "eyes + ears"? Does it mean, Look and Listen ?. I'm particularly fond of the rendering of the moon (middle photo of the collage above). Look at the moon. Gaze in awe at it. It's a taste of wonder and intrigue, bolted up and waiting for those passing by slowly enough to notice. I extracted the GPS data from the photos and plotted the locations with yellow diamonds on the map below. By the end of my running training I hope to find a few more.

Data Visualization

DYNOMIGHT 9 months ago

My 9-week unprocessed food self-experiment

The idea of “processed food” may simultaneously be the most and least controversial concept in nutrition. So I did a self-experiment alternating between periods of eating whatever and eating only “minimally processed” food, while tracking my blood sugar, blood pressure, pulse, and weight. Carrots and barley and peanuts are “unprocessed” foods. Donuts and cola and country-fried steak are “processed”. It seems like the latter are bad for you. But why? There are several overlapping theories: Maybe unprocessed food contains more “good” things (nutrients, water, fiber, omega-3 fats) and less “bad” things (salt, sugar, trans fat, microplastics). Maybe processing (by grinding everything up and removing fiber, etc.) means your body has less time to extract nutrients and gets more dramatic spikes in blood sugar. Maybe capitalism has engineered processed food to be “hyperpalatable”. Cool Ranch® flavored tortilla chips sort of exploit bugs in our brains and are too rewarding for us to deal with. So we eat a lot and get fat. Maybe we feel full based on the amount of food we eat, rather than the number of calories. Potatoes have around 750 calories per kilogram while Cool Ranch® flavored tortilla chips have around 5350. Maybe when we eat the latter, we eat more calories and get fat. Maybe eliminating highly processed food reduces the variety of food, which in turn reduces how much we eat. If you could eat (1) unlimited burritos (2) unlimited iced cream, or (3) unlimited iced cream and burritos, you’d eat the most in situation (3), right? Even without theory, everyone used to be skinny and now everyone is fat. What changed? Many things, but one is that our “food environment” now contains lots of processed food. There is also some experimental evidence. Hall et al. (2019) had people live in a lab for a month, switching between being offered unprocessed or ultra-processed food. They were told to eat as much as they want. Even though the diets were matched in terms of macronutrients, people still ate less and lost weight with the unprocessed diet. On the other hand, what even is processing? The USDA—uhh—may have deleted their page on the topic. But they used to define it as: washing, cleaning, milling, cutting, chopping, heating, pasteurizing, blanching, cooking, canning, freezing, drying, dehydrating, mixing, or other procedures that alter the food from its natural state. This may include the addition of other ingredients to the food, such as preservatives, flavors, nutrients and other food additives or substances approved for use in food products, such as salt, sugars and fats. It seems crazy to try to avoid a category of things so large that it includes washing , chopping , and flavors . Ultimately, “processing” can’t be the right way to think about diet. It’s just too many unrelated things. Some of them are probably bad and others are probably fine. When we finally figure out how nutrition works, surely we will use more fine-grained concepts. For now, I guess I believe that our fuzzy concept of “processing” is at least correlated with being less healthy. That’s why, even though I think seed oil theorists are confused , I expect that avoiding seed oils is probably good in practice: Avoiding seed oils means avoiding almost all processed food. (For now. The seed oil theorists seem to be busily inventing seed-oil free versions of all the ultra-processed foods.) But what I really want to know is: What benefit would I get from making my diet better? My diet is already fairly healthy. I don’t particularly want or need to lose weight. If I tried to eat in the healthiest way possible, I guess I’d eliminate all white rice and flour, among other things. I really don’t want to do that. (Seriously, this experiment has shown me that flour contributes a non-negligible fraction of my total joy in life.) But if that would make me live 5 years longer or have 20% more energy, I’d do it anyway. So is it worth it? What would be the payoff? As far as I can tell, nobody knows. So I decided to try it. For at least a few weeks, I decided to go hard and see what happens. I alternated between “control” periods and two-week “diet” periods. During the control periods , I ate whatever I wanted. During the diet periods I ate the “most unprocessed” diet I could imagine sticking to long-term. To draw a clear line, I decided that I could eat whatever I want, but it had to start as single ingredients. To emphasize, if something had a list of ingredients and there was more than one item, it was prohibited. In addition, I decided to ban flour, sugar, juice, white rice, rolled oats (steel-cut oats allowed) and dairy (except plain yogurt). Yes, in principle, I was allowed to buy wheat and mill my own flour. But I didn’t. I made no effort to control portions at any time. For reasons unrelated to this experiment, I also did not consume meat, eggs, or alcohol. This diet was hard. In theory, I could eat almost anything. But after two weeks on the diet, I started to have bizarre reactions when I saw someone eating bread. It went beyond envy to something bordering on contempt. Who are you to eat bread? Why do you deserve that? I guess you can interpret that as evidence in favor of the diet (bread is addictive) or against it (life sucks without bread). The struggle was starches. For breakfast, I’d usually eat fruit and steel-cut oats, which was fine. For the rest of the day, I basically replaced white rice and flour with barley, farro, potatoes, and brown basmati rice, which has the lowest GI of all rice. I’d eat these and tell myself they were good. But after this experiment was over, guess how much barley I’ve eaten voluntarily? Aside from starches, it wasn’t bad. I had to cook a lot and I ate a lot of salads and olive oil and nuts. My options were very limited at restaurants. I noticed no obvious difference in sleep, energy levels, or mood, aside from the aforementioned starch-related emotional problems. I measured my blood sugar first thing in the morning using a blood glucose monitor. I abhor the sight of blood, so I decided to sample it from the back of my upper arm. Fingers get more circulation, so blood from there is more “up to date”, but I don’t think it matters much if you’ve been fasting for a few hours. Here are the results, along with a fit , and a 95% confidence interval : Each of those dots represents at least one hole in my arm. The gray regions show the two two-week periods during which I was on the unprocessed food diet. I measured my systolic and diastolic blood pressure twice each day, once right after waking up, and once right before going to bed. Oddly, it looks like my systolic—but not diastolic—pressure was slightly higher in the evening. I also measured my pulse twice a day. ( Cardio .) Apparently it’s common to have a higher pulse at night. Finally, I also measured my weight twice a day. To preserve a small measure of dignity, I guess I’ll show this as a difference from my long-term baseline. Here’s how I score that: Blood sugar. Why was there no change in blood sugar? Perhaps this shouldn’t be surprising. Hall et al.’s experiment also found little difference in blood glucose between the groups eating unprocessed and ultra-processed food. Later, when talking about glucose tolerance they speculate: Another possible explanation is that exercise can prevent changes in insulin sensitivity and glucose tolerance during overfeeding (Walhin et al., 2013). Our subjects performed daily cycle ergometry exercise in three 20-min bouts […] It is intriguing to speculate that perhaps even this modest dose of exercise prevented any differences in glucose tolerance or insulin sensitivity between the ultra-processed and unprocessed diets. I also exercise on most days. On the other hand, Barnard et al. (2006) had a group of people with diabetes follow a low-fat vegan (and thus “unprocessed”?) diet and did see large reductions in blood glucose (-49 mg/dl). But they only give data after 22 weeks, and my baseline levels are already lower than the mean of that group even after the diet. Blood pressure. Why was there no change in blood pressure? I’m not sure. In the DASH trial , subjects with high blood pressure ate a diet rich in fruits and vegetables saw large decreases in blood pressure, almost all within two weeks . One possibility is that my baseline blood pressure isn’t that high. Another is that in this same trial, they got much bigger reductions by limiting fat, which I did not do. Another possibility is that unprocessed food just doesn’t have much impact on blood pressure. The above study from Barnard et al. only saw small decreases in blood pressure (3-5 mm Hg), even after 22 weeks. Pulse. As far as I know, there’s zero reason to think that unprocessed food would change your pulse. I only included it because my blood pressure monitor did it automatically. Weight. Why did I seem to lose weight in the second diet period, but not the first? Well, I may have done something stupid. A few weeks before this experiment, I started taking a small dose of creatine each day, which is well-known to cause an increase in water weight. I assumed that my creatine levels had plateaued before this experiment started, but after reading about creatine pharmacokinetics I’m not so sure. I suspect that during the first diet period, I was losing dry body mass, but my creatine levels were still increasing and so that decrease in mass was masked by a similar increase in water weight. By the second diet period, my creatine levels had finally stabilized, so the decrease in dry body mass was finally visible. Or perhaps water weight has nothing to do with it and for some reason I simply didn’t have an energy deficit during the first period. This experiment gives good evidence that switching from my already-fairly-healthy diet to an extremely non-fun “unprocessed” diet doesn’t have immediate miraculous benefits. If there is any effect on blood sugar, blood pressure, or pulse, they’re probably modest and long-term. This experiment gives decent evidence that the unprocessed diet causes weight loss. But I hated it, so if I wanted to lose weight, I’d do something else. This experiment provides very strong evidence that I like bread. Maybe unprocessed food contains more “good” things (nutrients, water, fiber, omega-3 fats) and less “bad” things (salt, sugar, trans fat, microplastics). Maybe processing (by grinding everything up and removing fiber, etc.) means your body has less time to extract nutrients and gets more dramatic spikes in blood sugar. Maybe capitalism has engineered processed food to be “hyperpalatable”. Cool Ranch® flavored tortilla chips sort of exploit bugs in our brains and are too rewarding for us to deal with. So we eat a lot and get fat. Maybe we feel full based on the amount of food we eat, rather than the number of calories. Potatoes have around 750 calories per kilogram while Cool Ranch® flavored tortilla chips have around 5350. Maybe when we eat the latter, we eat more calories and get fat. Maybe eliminating highly processed food reduces the variety of food, which in turn reduces how much we eat. If you could eat (1) unlimited burritos (2) unlimited iced cream, or (3) unlimited iced cream and burritos, you’d eat the most in situation (3), right?

Data Data Analysis Go Data Visualization

Entropic Thoughts 9 months ago

Various Hill Plots

Data Analysis Statistics Data Visualization