Posts in Java (20 found)
Farid Zakaria Yesterday

Always, sometimes, never

Exit 0 means your program exited 0. That’s it. That’s all it means. 🍞 I have been working on a deterministic VM called Rewind VM , where “a run” (i.e. test, command etc.) is a function of its inputs including the thread schedule. The determinism comes from the tool recording every step which lets you replay it exactly. This alone is a pretty good way to find flaky tests: run the test under Rewind, then runs it under many different thread schedules, with some slight latency added, and failures often crop up. 🐛 Feedback however was that it was not enough: the tool could find a failing schedule, but it could not tell you if that schedule was actually a bug. It only knew that the run ended differently than the original schedule, which could be a crash, a timeout, or just a different exit code. That is not enough to say “this is a bug”. Someone pointed out that I had built most of a property-based testing tool, but I only had one property: matching exit codes. What is property-based testing ? If you have used QuickCheck or Hypothesis , you have already used the idea. You write a property, or constraint, that must hold for every input. The tool generates lots of inputs, and when one breaks the property, it shrinks it down to the smallest input that still fails. already did the generating and the shrinking. The inputs are thread schedules, and narrowing shrinks a failing schedule down to the one step that causes the failure. The only property though was “the run ends the same way schedule 0 did”: the same exit code and the same outputs. That’s a pretty low bar, but easy to improve. 🙃 Rewind now lets specify properties that must be true for every run, or must be true for at least one run, or must never be true in any run. Let’s keep the bank from last time : two tellers deposit into one account from two threads, without a lock . This time the example is in Java as a JUnit 5 suite to appeal to my Enterprise brethren. 1 A teller that runs between the other’s read and store writes a stale balance over the other’s deposit. 💥 We include a “typical” test suite: one teller, two tellers concurrently, and a new account being empty. We run it and see all tests pass and head home for the day. 🍺 Is the code correct? Let’s use Rewind to dig further. Before even looking at the balance, Rewind knows a lot about what a run did . Every open, write, rename, close, fork and exit is in the trace, with the step it happened at. Rewind ships properties computed from the trace alone: Let’s throw a few at the bank and only run schedule 0: Interesting! writes its performance counters to 2 . It has nothing to do with our bug, but I included it because I learned something new and it could have been a bug if it did not include the process ID. The file properties shine for bugs between processes. The properties tutorial has a program that rewrites its config in place while another process rereads it. Its own test fails only when a read lands between the truncate and the last write, but flags it on the very first run. 😲 The bank’s real invariant is not in any syscall though. It lives in memory in our example: the balance variable must add up . For that there is the Rewind SDK , a tiny library you call from your program. Each call writes one line to , a device in Rewind’s VM, and the trace keeps it with the step, process and thread that wrote it. 3 must be true every time the program reaches it. is its opposite: a run that reaches it at all fails. These are your typical , except we get the whole replay if it fails. is the one I find the most interesting and tough to think through at times. 🤓 It tests the search: its condition must be true in at least one of the runs check tries, and it never fails a run. In model-checking terms, is a safety property (nothing bad happens in any run) and is a reachability property (some run gets there). Here it asks whether any run had both tellers inside at the same time. Look at the output above again: The suite passed under schedule 0, but one teller always finished its deposit before the other started. That pass told us nothing about the race. 🤷 Let try its default 64 schedules: That took 40 seconds on my laptop 4 and we found that two deposits went missing. The first line is the again: of the 17 runs check tried before it found the failure, 11 had both tellers inside at once. The fix is the obvious one: hold a lock from the read to the store. Every schedule passes, but is that enough to make us feel warm and fuzzy? The line is what gives me the warm embrace: in 42 of the 65 runs the tellers ran concurrently, but the lock prevented the bug. ✅ 5 A condition that is only true when the bug happens can never be true once the bug is fixed. 💡 Java developers though love writing JUnit tests. Fear not my FizzBuzzEnterprise colleagues, Rewind has you covered: the SDK has a JUnit 5 extension. Put on the test class and it marks each test’s start and result. If you can’t touch the code, reads the XML reports the runner wrote instead: Maven’s surefire, Gradle, pytest’s , whatever writes the format. The Rewind app shows the timeline of a run, with the tests and the build log. The failing test is red, and the event card names the assertion that failed. The strip under the timeline has a span per test: green passed, red failed. Each include a helpeful tooltip with the test name and the steps it ran. All the other Rewind features can then be used to help isolate the cause: the source panel, gdb on a fork at any step, the thread lanes and Check from here. There is a SDK for many languages already included in the Rewind repository. They are tiny, and the only dependency is the Rewind VM itself. The SDK is a single file or two per language, and you can copy it into your project. 6 The properties tutorial goes through each piece step by step, and the advanced tutorial has script properties and a file of properties to keep beside your tests for the more adventurous. If you happen to find any interesting bugs with Rewind let me know. I am keeping track of a small leaderboard of the best ones. 🏆 It is in the Rewind repository, next to the C version.  ↩ turns it off.  ↩ Outside Rewind the device does not exist and every call does nothing, so the calls can stay in your code.  ↩ The JVM is chatty: 148,479 steps for three tests, where the C equivalent takes about 4,500.  ↩ This counter must sit outside the lock. With , the second teller would wait before it reaches the counter.  ↩ The SDK’s README has the format instrunctions if you want to write one for another language.  ↩ C and C++ : , Go : and for Python : , and a pytest plugin that loads itself Java : and the JUnit 5 extension It is in the Rewind repository, next to the C version.  ↩ turns it off.  ↩ Outside Rewind the device does not exist and every call does nothing, so the calls can stay in your code.  ↩ The JVM is chatty: 148,479 steps for three tests, where the C equivalent takes about 4,500.  ↩ This counter must sit outside the lock. With , the second teller would wait before it reaches the counter.  ↩ The SDK’s README has the format instrunctions if you want to write one for another language.  ↩

0 views
Stone Tools 3 days ago

Lotus Improv on the NeXT

It is spoken of in reverent tones, invoked like a magic spell, whispered about as one of the mythical "roads not taken" in computing history. Its proponents write glowing reports and fawning YouTube reviews, lamenting its failure in the historical marketplace, each offering some justification for it not having set the world ablaze. It is safe to say, the legend of Lotus Improv persists to this day. If we put ourselves in Steve Jobs's shoes circa 1988, it's easy to see why he would have been smitten by an early preview of what we now call a "pivot table." NeXT was struggling to find its market, and exclusives which justified a system purchase were sorely needed. "Maybe Improv can do for NeXT what VisiCalc did for the Apple II," must surely have crossed Jobs's mind. In making it a NeXT exclusive, Improv seemed to be a clear-eyed, forward-looking answer to the question dogging the company, posed by the inaugural issue of NeXTWORLD Magazine , "Why NeXT?" Both NeXT and Improv brought innovative solutions to their respective computing questions of the day. NeXT made UNIX palatable to those scared of the terminal, debuted Display Postscript on a mega-pixel display, and made development for the system a snap. Improv rethought multi-dimensional data modeling, offered English-like formulas, and made it easy to rearrange data views on a whim. The bones of both remain to this day. NeXT's operating system, NeXTSTEP / OPENSTEP, morphed into macOS, and by extension iOS. Lotus abandoned Improv fairly quickly, but a quick survey of today's spreadsheets shows a universal embrace of pivot tables. Even today, it feels like nothing else, and still does data tricks no other spreadsheet can do as elegantly. Given its mythos, I feel a kind of nerd obligation to love it. Events have converged to make this the right time to finally talk about Lotus Improv . First, it is time once again for "International Spreadsheet Day." Second, as Improv was initially a NeXT exclusive, it seems only appropriate to take a look at that machine here on the 15th (!) anniversary of the passing of Steve Jobs. Third, and not to toot my own horn too loudly, it is the 1st anniversary of this very blog. Lotus Improv was one of four pieces of software I had in mind when deciding to start Stone Tools. In fact, it was quite literally the software that first triggered the thought, "I wish I had a place to talk about stuff like this." Deluxe Paint , the first post, was also one of those four. There are two yet to go; place your bets. The trouble was, while I knew Deluxe Paint intimately and could jump right in, I simply had no real experience with other categories of productivity software. It would have been impossible to judge Improv fairly without first getting better acquainted with the landscape into which it was born. So, I bided my time. Boy, howdy, do I want to impress upon you the superhuman restraint I've exhibited in waiting a full year to examine Improv . I think we all understand Improv, being a spreadsheet , knows how to calculate sums and transcendentals and other mathematical goodness. "It does math!" is not a particularly compelling justification for its existence, nor to support its claim as the "NeXT generation of spreadsheets." Instead, let's get to know it by studying examples which show off its unique strengths, and with the suggestion of the book Improv 2.1 Revealed! we'll model up some good ole' American baseball, a game I love and understand completely. Industry historians are far more capable of discussing the exile and homecoming of one Mr. Steven Paul Jobs back to Apple. I touched on the TL;DR of it in the timeline, and if you're really clamoring for juicy tidbits, a book was published this very year that delivers on that front. While NeXT hardware was well regarded, it never really caught on. In the long run, the better result of NeXT engineering was NeXTSTEP and how neatly it wrapped UNIX in a clean, user-friendly, graphical interface. It's quite an accomplishment. For me, it was the technology behind NeXTSTEP/OPENSTEP developer tools, Objective-C and Interface Builder, that kickstarted my programming career when those became the OS X development environment (hat tip to Aaron Hillegass's Cocoa Bootcamp at the Big Nerd Ranch). Even today, years after the advent of iOS and Swift, NeXT technology remains in the Foundation layer of Apple software, as denoted by -prefixed classes (a name-spacing trick from the Objective-C days). For this investigation, I'm using NeXTSTEP 3.3, the last version before transitioning to OPENSTEP. NeXTSTEP looks and feels like a modern system, with some choice differences. Familiarity with any modern operating system, especially macOS, will inoculate you from breaking out into hives using it. Considering the eye-candy Jobs seemed excited to add to OS X after the Apple merger, NeXTSTEP is positively staid; nothing here screams " lickable ." If you've ever wondered what Susan Kare could have achieved if she'd had a larger canvas and palette (NeXTSTEP is 2-bit) than the original Mac, look no further. I like it. The dock is present, though more rigid than the playful fluidity it acquired when transitioning to OS X. No "genie effects" nor dynamic magnification; it is ever-present and doesn't hide itself when not in use. Some applications change the dock icon when running (like Mail ), while other applications do not (like Edit ). One odd quirk is how the dock indicates running apps, by the absence of three dots on an icon tile. That's a strange choice, decorating the inactive icons. File browsing adds a little complexity to system usage. The file browser contains its own mini-dock for favorites, the path to the currently selected item rendered as giant icons, and the file browser proper. Dragging files and folders around can only be done with an item's full-size icon. If you're browsing in list view, you can't drag list items into a folder or the dock, for example. Bit of a head-scratcher, that one. Menuing for all applications is handled via a user-positionable, vertical menu. Sticking the menu somewhere sets its position for the active application, and every application can have its own position. You could theoretically have the menu jumping around all over the place as you hop between apps. Thankfully, it can also be called up at the current mouse position with a right-click; shades of RISC OS. Also, good news for those who love the spinning beach ball on macOS! You'll feel that little ASMR tingle when it appears, especially if you run a color workstation. It's funny the things that can make something feel like "home." Honestly, my biggest issue in adjusting to NeXTSTEP is the mouse. By default, it feels weird to use, with a dual-level acceleration curve, one for slow movement, one for fast. Crossing the threshold between "slow" and "fast" jumps the cursor across the screen when the alternate acceleration curve kicks in. I could never find my cursor except when moving the mouse verrrryyyy slooowwllllyyyy. Luckily, the emulator Previous has tools to help compensate ( check "Sharpening the Stone" ). Overall, you'll only need to adjust your thinking a little here and there, forget some modern habits, and you should be comfortable with NeXTSTEP in short order. 35 years later, it still looks and feels much like any modern operating system. Almost depressingly so? Maybe that's a topic for another day. Question: what is interesting about the following screenshot from Improv ? Answer: it contains all information necessary to build a copy for yourself. All data, formulas, dimensions, and relationships are exposed and visible. I note this not to suggest that screenshots are a great way to share Improv worksheets with others, though you could; the sheet above was rebuilt from a screenshot, in fact. Rather, I want to illustrate how "graspable" data can be in Improv's environment. Its formulas read naturally, and value references match the item names in the table. The work does what it claims. Let's interrogate that a little more thoroughly. To understand Improv is to forget what we know about spreadsheets. Slash commands? Never heard of him. B1? J93? K12 = int(A14 * lookup(B2..B7))? The ramblings of a madman. Rather than generic alphanumeric column/row identifiers, we define our rows and columns with "categories of items" specific to our domain. Put another way, instead of asserting, "Columns A to L represent the months, and rows 1 through 31 represent the days," we will create a category called "months" and define twelve items for it named "January," "February," etc. and we will create a category called "days" and define 31 of those. Take a look at these comparative videos which illustrate the difference. First, I'll use LibreOffice Calc to create a months/days grid, apply a value to January 1, and calculate December 31 as a multiple of January 1. Here's how that same table is created in Improv . There are three things to note: how sequential items are generated, how categories are named (not strictly necessary for this simple example), and how formulas are created/applied. There's a lot to absorb, I know, starting with how a new, blank worksheet consists of a single cell, not an infinite table of empty cells. Improv works with only and precisely the data we define. Where in the LibreOffice example we attached meaning to the columns by adding labels to the top of each column, that was for our benefit not Calc's . I could, just to feel the warm, snuggly blanket of classic column/row naming, make an Improv category called "columns" and name my items A, B, C, etc. and a category called rows, with names 1, 2, 3, etc. Were we to take such a pedestrian approach to categories and items, we would diminish the magic of Improv's formula builder. The bottom panel holds all formulas, which are applied at the granularity defined, in the order shown. In the example, I specified but could just as easily have targeted all of December by specifying . That single formula would apply to every list item in December without needing to copy formulas through to each cell, nor set absolute/relative markers for cell references. Here's a simple expansion of the previous table. Every day in January increases by 10%, and its corresponding day in December doubles the January value. The two formulas shown are the only two formulas for the entire worksheet. I want to stress this; there are no other formulas lurking inside cells, hidden from view, waiting to be revealed by a selection or editing process. There are no duplicated formulas per cell, each increasing a relative cell reference by . We simply refer to categories (in this case "months" or "days"), items (as "January" or "27"), or specific intersections (as "April:19"), including relativity signifiers , , and . You might be thinking of ways to trick the system; to confuse and befuddle it. Why would you want to bully an old app like that? What did Improv do to deserve such shabby treatment? "What if two items have the same name? What if two formulas target the same cell?" Fret not, all shall be revealed and you will feel proper shame at trying to confuse this old program. In the above example I have renamed one of the Days as "March," to create ambiguity when referencing "March." Does "March" mean the Day or the Month now? We can be explicit, referring to . Note though that if we aren't explicit, Improv will be on to our tricks and call us out, as in formula 5. In formula 2, I set to the static value . In formula 3, I also set it to be calculated as "double its corresponding January value." Note the "overlap" icon (two intersecting squares) on those formulas and the text in the cell. Clicking the overlap icon gives us an opportunity to resolve ambiguity by explicitly selecting which formula should win for that cell. Cleanly handling formula ambiguity was originally seen by the prototype team as an intractable engineering problem. From "Improv: The Inside Story", by Simson L. Garfinkel, NeXTWORLD , Fall 1991: Personally, I like Improv's solution. I honestly think more software should be very dumb, and not try to intuit or guess what I want. Tell me the problem in clear language, and provide the simplest tool that gives me the most control over problem resolution. Be smart about helping me, don't try to help me by being smart. Before we get in too deep, I want to make sure I note Improv's beautiful help system, replete with index and search. Considering the difficulty in tracking down a manual for the NeXT release , having this available during my exploration proved invaluable. It's neatly illustrated, with terse (sometimes overly terse) descriptions of the tools, hyperlinked to other pages of the documentation, as warranted. Improv felt it could do even better than that and made its help contextual . The Help system, when invoked, defaults to displaying documentation for whatever is currently selected in the user interface. That's very useful, but becomes supercharged with "Point for Help," as demonstrated in the video above. This turns the cursor into a symbol, and whatever we click will be revealed in the Help window. This is a great idea, because I don't always know what Lotus wants to call some little interface doohickey, so I don't know what to search for. Now I know: the "Calc" button is a "Status Indicator" and the triangle icon that sometimes shows up is the "Formula Error Indicator." Pork chop sandwiches. Looking through Usenet archives, I saw this called "toilet paper" by at least one person. Lotus called it the "item dispenser." It's an interesting first attempt to do something we take for granted today and only exists in the NeXT version of Improv . It's cute, but smacks of someone being overly excited with new development toys. If we need to generate a list of items to populate an axis of your data, this tool generates sequential list items for you. It does its best to kind of "guess" our intentions, so if we type in "Sunday," then pull down on the dispenser, days of the week will be auto-generated in a list. We then "tear off" the list, which loads the cursor with the items, which we attach to an existing item, appending that category with the loaded list items. That means we need to have at least one item in a category onto which the list can be attached. This is annoying, because the existing item likely has no relationship to the generated list. I'm always having to make a list, attach it to an existing item, then delete the original, unrelated item. These days, we can type directly into a cell and extend that cell intelligently across adjacent cells. In fact, Lotus would switch to that methodology when migrating Improv over to Windows for version 2. Hold ALT and drag an existing item to get the same effect. The original dispenser truly is a feature that could only have been thought up within the NeXT environment. I don't hold it against the devs for getting excited by the GUI flexibility Interface Builder afforded. I, too, did the same back in the early OS X 10.2 days. It practically begged us to play around with GUI ideas, remixing elements into new configurations. "First Responder" and "bindings" stans, represent! One of the great features of that development environment was how easy it was to “say ‘No’ a thousand times” while trying ideas. Lotus embraced that, achieving quickly on NeXT what had blocked their progress on OS/2 and Microsoft's Presentation Manager. The "item dispenser" was singled out as a specific tool that benefitted from Interface Builder; there really wasn't anything similar to its rapid, iterative, GUI development at the time. In fact, I might go so far as to say there's nothing quite like it today, even from Apple . Joel Spolsky, in editing The Best Software Writing I , added a personal aside on Lotus Improv , blaming its failure thusly, " Improv required you to define strict n-dimensional hypercubes for your data. There was none of the flexibility of traditional spreadsheets." I suppose if one wanted to paint Improv in the most inhospitable light, Spolsky is correct. In practice, it doesn't feel as impenetrable as his description. However, I do think he's on to something that gets at the heart of why Improv never gained traction. For now, let's hold our collective nose and dive into the scary, foreboding waters of "n-dimensional hypercubes" and model up some baseball. Our first dimension will be . Let's add a dimension next. Next, we'll get more granular and add . Internally, Improv separates our work into three parts: data, views, and formulas. What we see on screen is just Improv's best-effort to represent our three-dimensional data flattened out into a two-dimensional view. Improv adds extra columns to the view, as necessary, to ensure we always see the full extent of our data relationships. Now, we'll add some win/loss data and the next thing you know we’ve built a 4-dimensional hypercube. Are we scared yet? Since I've only modeled at the "Division" level, I'll add together divisional wins inline to calculate the totals and... hmmm... "No dice," as they say in baseball (Presumably. Are dice involved in baseball?) Alas and alack, we can only enter concrete numbers, dates, or text into cells; any formula, no matter how trivial, must go into the Formula Panel (bottom). In this case, I could add a formula that targets a specific cell, as , but the whole point of a spreadsheet is to automate that kind of junk for me. I suppose the path of least resistance is to add a category to add up all team wins and, wait a second... Dang it, that petard I confidently told myself wouldn't hoist me has come around to do precisely that. Here we see the downside of dimensional modeling, at least for the layperson, and arguably for most typical spreadsheet users. It all looks like a neat little chart, but that word "dimension" describes and reveals the problem. To add a category, its items necessarily apply to all other categories. Teams don't do that; they are unique to each League and Division. But we just might have a way out. Lotus engineers clearly understood that categories alone weren't going to cut it. Categories impose rigidity to our data that does not always accommodate the messiness of real data. We need another way to organize our data. Groups are intra-category, named collections of sequential items. Make a category and add a bunch of individual items. Select a sequential group of some subset of items and group them, giving that group a unique name. Once everything is grouped, groups of groups can be grouped to group a group of ... wait, I got myself confused. Groups can be grouped, 'nuff said. With "Teams," we quickly run into a side-effect of groups, and how it can necessitate a cascade effect across multiple categories. I want to add teams and group them by "Division," but that forces me to also make "League" a group as well. What was three categories have now collapsed into one: "Teams" Here's what that looks like in practice. We still gain the benefit of name references in formulas, as in the above untethered-to-reality example. A reference to would apply to both American and National, unless we specify by as in and Groups can be subtotaled, using the function, as you can see in this more data-rich chart of the MLB stats from 2022 to 2026. Notice, however, we risk a conflict for in our field, where there are two formulas battling it out for dominance over the same cell calculation. The handy keyword will skip (i.e. "won't apply this formula to") groups, cells, items, and categories as designated. Groups helped bring granularity and specificity to our data structure, but the more categories we remove, the more we remove access to the very feature that has kept Improv in the back of nerd minds for decades. You know what I'm talking about. FINALLY we get to the point those in the know have been anxious to talk about, " Lotus Improv invented pivot tables. It says so right there in Wikipedia." At this stage of my knowledge of all things retro productivity, I think the most I'm willing to concede right now is that Lotus Improv likely invented pivot tables as we understand them today. But, like Yoda coyly teasing the existence of Luke's sister to a guy who should have already known about her, there is another program, a future Stone Tools subject, that may lay claim to the title of "first pivot table." I exercise my right, as "guy who runs the blog," to reserve final judgement on this point until then. For our purposes today, it's a true enough statement and the inventor even has a nice video on YouTube in which he shows it off to Lotus staff for the first time. Meet Pito Salas, inventor of the pivot table. Mike Wells (shown in the thumbnail) takes a question at 28:40 and reveals that NeXT has "Objective Fortran" and "Objective Lisp" ?!?! Salas, now retired , developed the idea while working for Lotus Development's Advanced Technology Group. He put together a fakey version of his vision, a kind of slideshow, showing off the things he wanted to be able to do: English-like formulas, multi-dimensional data, and rearrangeable views into the data. He was given the greenlight to proceed, "All those [ideas] I had described with hand waving,'' recalls Pito. "I had no clue how to build it." A year of nose-to-the-grindstone later, and a prototype dubbed "Back Bay" went on US tour, wowing financial modelers. Everyone who saw it, wanted it. Let's take a look at how Improv handles pivoting. To start, I'm leaving it empty of data, so we can focus on the dynamic view structure more easily. Then, I'll move categories around to show how the view flexes to present the data. Next, I'll put categories into the third axis, called a "Page Category," allowing us to paginate our data. Finally, I'll enter a formula and a little data to show off a trick pivot table contemporaries cannot perform. Now I'm going to try and replicate this in LibreOffice Calc v26.8. There are a few things to note, but let's first point and laugh at the fumbling attempts of a LibreOffice newbie (me). A few things jumped out at me while comparing these approaches. First, modeling up a relatively complex data set in Improv is as easy as thinking about it, and the presentation of the data matches my mental model. LibreOffice requires me to bridge a fairly wide conceptual gap between what I want and how I model it, due to the amount of redundant data I have to enter for every line item. Some items, like "League," translate cleanly, while "Record" does not. In fact, I'm still unclear what the best approach would be to convert my Improv computed item "Win Ratio." LibreOffice has the concept of a "computed field" inside pivot tables, but reaching across columns to pluck out related data to perform computations is complicated. In Apple Numbers, Microsoft Excel, Google Sheets, and LibreOffice , pivot table data is splintered off into a separate snapshot of the data, dedicated to the pivot table view. Pivot table data can typically be lightly manipulated, but does not affect the original data. The reverse can be done, but only after asking for an explicit update of the pivot table to reflect changes to the primary data table. This is a huge drop-off in ease-of-use from the Improv way of things where every view of the data directly manipulates the base data. Dragging around categories is just as easy in either approach, though Improv gives immediate drag-and-drop access to what LibreOffice calls a "Filter," i.e. a paged view, and generally "feel better," IMHO. LibreOffice requires us to open up a separate panel and drag things around in there. I will say, however, that being able to filter the filters (i.e. exclude/include specific subsets of data) is handy. Sheets , Numbers , and Excel (online version) are all strictly panel-based pivoting. Interestingly enough, the panel approach is precisely the interface Lotus rejected , at the behest of Steve Jobs. Once more, from the Garfinkel article. Having now directly compared Improv's approach to dynamic views against modern pivot tables, it is safe to say they are the same only so much as wagyu beef with gruyere cheese prepared by Gordon Ramsay and the sawdust at the bottom of a Happy Meal are both "cheeseburgers." Improv tastes much better. A Lotus invention which debuted in Lotus 1-2-3 , "named ranges" have attempted to bring order to the chaos of cell identifiers. In 1-2-3, will apply a name of choice to a defined cell range, after which that range can be referenced in formulas by that name. So, in 1-2-3 or any modern spreadsheet we can name range as and will work. Superficially, this looks similar to how Improv works with categories and items for multi-dimensional data. However, named ranges are for convenience only, and do not establish relationships between data values. In Improv , would represent a fundamental aspect of a piece of data. Getting the sum of all sales is the same, , but it being a dimension on the data gives us deeper options. For a specific year, will match against two axes at once. could further pinpoint our summation. Let's see how this would be done with modern spreadsheets and named ranges. If we want to mimic Improv's dimensional relationships, we must define relationships of interest within each formula, (and those formulas have to be copied over to every related cell) such that: Meaning "sum items in the named range called "Value" only if the Year is 2020, and the League is "American," and the Record is "Win," and the Division is "East") Of course we could substitute hardcoded filter criteria with values pulled from a lookup, or perhaps extract equated values via a statement. Maybe even set up a little sub-table to hold calculated values. Quickly this gets way more complicated than I'd have hoped for, so personally I'd resort to doing it the old-fashioned way and just hardcode in a bunch of cell references. To reiterate, the complete solution for an entire sheet in Improv , where win/loss information is its own dimension, is the single formula: The last of the big features, which takes advantage of NeXT's graphics prowess, is the charting functionality. Creating a chart launches an entirely separate application, Presentation Builder , which has grand plans on becoming PageMaker when it grows up. Generating a chart generates a full, blank page onto which a chart is placed. A suite of palettes complements the charting workflow, allowing for precision edits of the chart, its titles, colors, positioning, and more within the full page. Multiple charts can be generated and placed on the same page. Arbitrary text and shapes, lines pointing at interesting bits, and even sound annotations (a system-wide NeXT tool, called "Lip Service") can be placed at will. So it goes that while we have the tools to place arbitrary text on the page, yet lack tools to center that text, the sense that Lotus has overextended itself is strong. Sometimes it's enough to just generate the output a program excels at, and leave "page composition" to the professionals. Again, I need software to be more dumb . That's without even mentioning the plethora of formatting tools scattered amongst an unreasonable number of palettes, panels, and "inspectors." A benefit of these tools on the NeXT is that everything we do is Postscript by default, thanks to NeXT's Display Postscript screen rendering engine. "Print" a document as PostScript, and we can render the resultant .ps file on any modern system. I opened it directly in Affinity and received precisely what I saw in Improv , chopped off text and all, but like, high-quality chopped off text. Look, I try my best to craft a compelling narrative, moving from topic to topic, stitching together callbacks and observations to lead you, my dear reader, on a carefully considered ride. It's just you and me, side by side at the keyboard, chatting about software and learning together. But, sometimes there's junk I need to squeeze in that I can't find a natural segue for discussing. Here, then, is a brain dump of other things I want you to know about Improv . Speaking of version 2... In 1993, NeXT abandoned its hardware initiative and moved over to Intel, rebranding as NeXT Software, releasing OPENSTEP soon after. At the same time, Improv also moved over to Intel, but parted ways with NeXT. With version 2, Lotus developers achieved the success that eluded them on OS/2, porting Improv over to Windows 3.1. The feature set for version 2 is disappointingly similar to version 1, and by this time competing software was starting to embrace some of Improv's unique qualities. We do finally get undo, so thank Kapor for small favors. Scripting arrives in the form of LotusScript. A new Windows UI adds a bunch of buttons to the interface, and flushes the toilet paper dispenser for a more sensible list generator. That's basically it for the new stuff, but at least they kept the old problems as well. Cell reference relativity is still locked to a single dimension. Formulas cannot be freely rearranged, only cut-and-pasted into position. No 3D worksheets. Then, to throw a little extra wrinkle into the mix, there was Lotus's product strategy on the PC. If Improv wasn't your thing, maybe their other Windows spreadsheet, 1-2-3/Windows , would feel more comfortable. Or their other-other option, Lotus Symphony . Or their other-other-other option, Lotus Works . Or their other-other-other-other option, Lotus 1-2-3 for DOS Release 4 , which went on sale in 1994. Regardless, it was well-liked by the press, even if reviewers didn't seem comfortable giving full-throated endorsements. Take this Compute , July 1994, review for example. Or this PC World , February 1994, review. It would be hard to intuit, based on that review, that PC World had declared Improv the "Winner" in the Applications category for their 10th Annual Awards for Technical Excellence just a few months prior. 3 in spirit, if not in flesh; let's take a quick look at Quantrix , the product I clumsily foreshadowed earlier. From ex-Lighthouse CEO (also ex-Sun CEO) Jonathan Schwartz's blog , "In 1989, a few friends and I started a software company, Lighthouse Design, that devoted itself to the NeXT platform. Whether any of us admitted it at the time, Lighthouse was built by a group of people for whom Steve Jobs was the gravitational center of the universe." Their devotion to Jobs was returned in favor, by way of advice for their products to flourish on NeXT. Ultimately, according to Schwartz, the problem was less about finding a way for Lighthouse to flourish on NeXT than it was about getting NeXT to flourish in the first place. When looking for a buyout in Lighthouse's waning days, they had fingers crossed to be acquired by NeXT but Jobs didn't bite. They were bought by Sun, their products shoved in a vault never to see the light of day. With Microsoft unwilling to support the platform, Lighthouse made the closest thing to an Office suite for NeXT. They built programming tools, project management software, a presentation tool called Concurrence ( that Jobs used and which "inspired" Keynote ), and two spreadsheet programs: ParaSheet and Quantrix . ParaSheet took the more traditional approach, while Quantrix was just a flat-out clone of Improv , I don't know how else to say it. It has the same split screen worksheet/formulas, the same formula tools, the same draggable categories, it imports Improv documents, and more. According to one Hacker News reader , "The manuals were interchangeable." Lighthouse took the ideas behind Improv and ran with them, envisioning what a possible v3 could have been. Put simply, everything I've expressed frustration with in Improv is addressed in Quantrix . It has undo. It has drag-and-drop, reorderable formulas. It has scripting. It has cross-worksheet category synchronization. The "Item Dispenser" is replaced with an in-line widget; pull to expand like any modern spreadsheet does today. It has multi-dimensional cell relativity. It runs on OPENSTEP for Intel, as well as NeXTSTEP on Motorola 68K . It has color. It loses contextual help (can't win 'em all). And that morass of panels and palettes for editing the charts I showed earlier? Those are collapsed into a single, tidy "Inspector." Long story short? If you want to experience the best version of Lotus Improv today, use Lighthouse Quantrix v2.3. You'll get all of the original, cool tools built better, with a more sensible interface. Plus, Lighthouse gave away their NeXT applications, with license code, for free. How neat is that? The big twist, for those affording me a generous definition of "twist," is that Quantrix still exists . It is alive and being sold for "ask for quote," so I think I'll just put this credit card back in my wallet at this time. Today, it very explicitly targets the professional, high-end, financial modeling crowd, and not the general-purpose masses. You know, the crowd that Lotus first showed their Back Bay prototypes to back in the day (see timeline). Its self-introductory video pivots (get it?) many of its 35-year-old features as revolutionary. A back-handed indictment of the stagnation of an industry, or a product finally finding its niche? You be the judge! It's hard to suss out precisely what happened, but I found a little information floating around the Quantrix forums . If that is to be trusted, Lighthouse's Quantrix chief engineer, Pete Murray, founded the company Subx and rebuilt Quantrix as a Java application in 2003. Quantrix's website says it was acquired by IDBS in 2010. I'm still completely in the dark how Quantrix escaped Sun's containment field, as Sun had refused to sell off the Lighthouse apps to hungry buyers; that stuff could have been converted into OS X apps in the early days. At any rate, it is one of the more handsome JRE applications, while suffering from telltale Java rendering issues, especially with regard to text positioning and scaling. Otherwise it is clean and inoffensive. Sterile? Boring. That's the word I was looking for: boring. "It knows its target audience," is perhaps more charitable. So where does this leave my relationship with Lotus Improv ? Why did I dangle that "And yet..." at the start of the post? If I'm completely honest with myself, there is a vibrating tension between "wanting to love it" and "actually loving it." After all of these weeks, I think I'm in a pretty good position to suggest why it didn't take the world by storm: it's not a great spreadsheet program. Let me qualify that statement lest r/Improv4Life roasts me alive; the roasting can wait a moment. For what it does, it is amazing and even fun to use. What it doesn't do is provide a good, general purpose spreadsheet experience to the masses. It claimed to do that, positioning itself as the "NeXT generation of spreadsheets." Whatever Lotus Improv did or did not accomplish for the spreadsheet genre, one thing is for certain: it makes a fantastic tech demo. Seen in that light, I'd have been as sold on it as Jobs was, excitedly showing it to NeXT non-believers, "Look what MY computer can do!" Those moments of smug technological superiority amongst friends are fun, aren't they? Once you played with it for a week or two, and the initial excitement of pivoting dulled, you may have ashamedly harbored a bit of buyer's remorse. Adam Bosworth, in an article contemplating a talk he gave at ICSOC '04, was thinking about software complexity vs. simplicity, and posited: Paradoxically, Improv's promise to improve data comprehensibility wound up narrowing its general appeal. Want to quickly throw down a few lines of data, like a makeshift phone directory? You're gonna have to rethink your data as a matrix of abstract qualities about your friends. Despite its friendly English-like formulas and delightful GUI for pivoting data, Improv is in actuality purpose-built for far more rigorous data modeling than its playful tools suggest. It's the next generation of spreadsheets for a thin slice of the market. That said, there is a non-zero set of readers for whom their hardcore number crunching needs perfectly align with Improv's promises. If that's you, you're absolutely going to fall in love... ...with Quantrix . Ways to improve the experience, notable deficiencies, workarounds, and notes about incorporating the software into modern workflows (if possible). Improv does export to Lotus 1-2-3 , but you'll likely be disappointed with the results. Consider a table with formulas and calculated values in the cells. Everything you see in the cells will be exported, but there will be no formulas or relationships between anything. You will receive a snapshot of all current values, arranged to visually match the table structure you had when exporting. Improv's approach to shared formulas and multi-dimensional data simply don't have an elegant export solution to traditional spreadsheets. If you have Improv documents you want to import into Quantrix , two criteria must be met. Save your Improv project as an file (not the default ), and make sure you have the Lighthouse (available at the same site offering the Quantrix software link I mentioned under "Version 3") installed at . Exporting charts is simplicity, thanks to Display Postscript. The Postscript files NeXTSTEP generates open and render just fine on modern machines, using Ghostscript or other graphics programs. As mentioned previously, I actually recommend using Lighthouse Quantrix v2.3 instead of Lotus Improv , on the NeXT. The substantial features Improv lacks are numerous: In general, it's lacking the clay-like malleability of a traditional spreadsheet. If you're doing what a lot of us do, using spreadsheets like a kind of mini-database, you'll probably get frustrated. Here's the main event. Compare and contrast with his OS X announcement at MacWorld 2000 for bonus points. Previous v4.4 on Windows 11 NeXTstation Turbo 68040 CPU (variable clock rate; ~40 - 50Mhz) 2GB hard drive Monochrome "Megapixel" display NeXTSTEP 3.3 Lotus Improv Rel 1.0 (v108r0) There is no undo. This isn't the first program I've covered that lacked undo, and I'm not talking about software that can be forgiven for being born too early. The formula panel at the bottom applies formulas in the order shown. As order is important, it would make sense that you might want to rearrange them from time to time. There is no mechanism to do this, other than cut and paste. This has been more of a pain than the lack of undo. Improv uses special keywords to add cell relativity to formulas. So THIS value is the PREV value times 2, or whatever. But we're in a multi-dimensional sheet, so what do these keywords point to, precisely? Well, in a formula they must pick a category, and only that category can be referenced relatively. If a formula uses multi-dimensional data to calculate a value, like , those keywords won't work, though it would be completely natural to write such a formula and expect success. There is no scripting. Lotus 1-2-3 users would have been particularly put out in not being able to bring their carefully built automation over to Improv , which would be a huge shocker since they're made by the same company. I would even go so far as to say that that alone would have pushed a great number of potential converts away. Nobody has the time to rebuild their financial empires from scratch! There are no 3D sheets, meaning there is no linking between projects. A project can contain up to 100 worksheets, and those worksheets can reference one another, but only within the same project file. If a category in a worksheet exactly matches a category in another, item for item, those can match and data will flow across as expected. Unfortunately, there is no way to synchronize the shared categories. Change the items in one sheet and those will not be picked up in sheets using that linked category. This means we cannot make a "master" category inherited by other worksheets. Like a lot of things back in the early days of productivity software, user diligence plays a large role in keeping things running smoothly. Even version 2 on Windows didn't add this feature. If only someone would step up to the challenge. ( foreshadowing ) Previous never crashed on me, and has gotten a lot easier to set up since its early days. Version 4.4 has been great. For mouse movement, I found that setting (in Previous , not NeXTSTEP) "Slow movement" to "Fast" and "Fast movement" to "Slow" balanced things out to something close to consistent tracking; it worked for me, is all I can promise. There is a modern add-on to NeXTSTEP called "GKMouseScaler" that lets the user draw precise cursor acceleration curves, should you need better control. Personally, I never needed it. I found the 2-bit monochrome display to be more than sufficient; no color needed. This is version 1.0 of Improv and it did crash a couple of times, quite suddenly. There appeared to be no specific trigger, nor function, that led to the crashes. It just beachballed and died. NeXTSTEP itself felt stable to me. For installing software from modern repositories, three things to keep in mind Thanks to a convenience in Previous , at in NeXTSTEP you can access your host computer file system. You'll find a lot of .tar.gz files in the wild, so brush up on and Once you have a file, double-click and it should install itself. No undo in v1 on NeXT; v2 on Windows does. Can't reorder formulas except through cut/copy/paste. and only work along a single axis, even though it makes complete logical sense for them to work along multiple axes in the same formula. Presentation Builder is overly complex on the one hand, yet missing very basic tools on the other hand. Can't reference values in other projects, aka no 3D worksheets. No scripting options in v1 on NeXT; v2 on Windows does.

0 views
Brain Baking 6 days ago

Laptop Stickers: Yay Or Nay?

Here’s an interesting dilemma for you: do you place stickers onto the back of your laptop? Are you a laptop sticker kind of person or afraid of anything glossy that might leave glue marks? I used to belong to the latter group, but after more than six years of typing on this M1 Air , I was ready to give in. The reason for this is simple. Sebastián sent me a couple of Emacs-related stickers, knowing I couldn’t resist. It’s all his fault. Look at what he did! A not so pristine golden coloured MacBook Air shining in the sun, desecrated by four stickers. (And yes, I am well aware of the irony of pasting a Free Software Foundation sticker on a MacBook that runs on a proprietary OS. I promise my next hardware choice will be more ethically sound.) Of course I can’t stop there: what to do with all that open space in-between the stickers? I could have placed them all at one side to make it even more painfully obvious that the beautiful symmetry is completely gone now. As for the question of where to stick them: I already think far too much and was too happy with the stickers to give this much thought. They’re on there. The Apple logo is still visible: a Brain Baking logo sticker covering that up might be the logical next step to take. Every time I receive stickers at conventions, I hesitate: do I stick these onto the laptop? Wait, what do they represent? Do I really want to proudly display the Octat GitHub mascot to promote GitHub as “the only way to do Git”? Nope. Do I really want to paste a Java mug or Gopher on there, identifying myself with these programming languages that aren’t bad but don’t get my engine running? How about a colourful JetBrains IntelliJ one that can serve as a free ad? Nope, nope, nope. It was easy to be in the (tech) sticker hater group. And then I finally found something that was fun to mess around with. So yes, I am now a happy Lisp and Emacs advocate at work. The stickers already provoked many Emacs versus Vim discussions that end once you show them that Emacs can be a window manager running your other apps, or that EVIL mode exists. Still, 99% of the die-hard coder colleagues—which are less than 10%, the others of course rely on VS Code—are Vim users. That’s great! But one wasn’t and told me he yesterday closed his Emacs window for the first time in weeks. We both had to move classes but I’ve made a mental note to meet up and exchange Lispy configs. Thank you Seb & Seb’s stickers. Many of my students like stickers on their laptops. Most just stick on there whatever they can get their hands on. I’ve seen a lot of Gophers but nowhere in our curriculum we explore Go and after approaching them, I learn that they don’t even know how to write a line of Go code. A Gopher is cool, that’s what matters! I do think that you should at least know what you’re implicitly preaching. Which is why I’m not limiting myself by sticking only tech-related stuff on there. The question then becomes: what else? Something retro game-related, obviously. But still: what? A classic Castlevania logo? I’d rather go for a custom-printed sticker with some obscure sprite work, from for instance Wario Land 3 or Gobliins 2 . If someone is able to recognise all these, I’ll finally have found my missing soulmate. Maybe a pixelated version of bread or a fountain pen? Or am I giving away too much information that way? Also, since the laptop turns seven in January, I should keep copies as the stickers are not transferable. Another approach would be to cover the entire back, making use of every square millimetre. A few colleague design lecturers carry laptops like this. I don’t think I’m cool enough for that. Terence caught a nasty laptop sticker paralysis while trying to perfectly place his, so a bit of thoughtful preparation might be in order. I love Abigail’s Kirby sticker on her Thinkpad. She even scanned her laptop with a photocopier and converted the entire thing into a clickable polygon-based SVG. Much cooler than my lazy cellphone snapshot. I’m also jealous of her Cory Wong pizza slice sticker! I do wonder what she’ll do when her laptop breaks down or it’s time for a replacement. Nia’s laptop stickers (a November 2022 snapshot) also contains some cool ones, as does Manton Reece’s , such as bookstore stickers. Here’s a discussion on the Framework forum containing more inspiration. It is clear that I still have some work to do. The four stickers are only the beginning. Laptop Stickers: Yay Or Nay? Yay. But then: how many? Overlapping or not? Planned placement or not? Tech-related only or not? See, even the most banal decisions in life come with Deep Thoughts ! To be continued with a photo when it’s done. Related topics: / stickers / By Wouter Groeneveld on 9 October 2026.  Reply via email .

0 views
Corrode 3 weeks ago

We Have Named Arguments at Home

Steve Klabnik recently wrote about named arguments, optional arguments, default arguments, function overloading, and why most of that design space has historically made him nervous in Rust. I agree with Steve. In fact, I think I agree slightly more strongly than Steve does. :) I actually think we can get most of what we want without adding any new language features. Instead, we can lean into what Rust already provides. None of these is an exact substitute for what you get in Python, Ruby, C++, or Kotlin, but that’s sort of the point. Instead, you can get 80% of the ergonomics without adding any magic to function calls at all. The recurring pattern is that Rust takes something another language puts into function-call semantics and represents it as a normal part of its type system, elegantly sidestepping the mentioned design problems. Let’s revisit Steve’s example from the crate: The obvious problem is that four consecutive s are not a great API that you can reliably use without reading the docs. Let’s assume for a moment that we had named arguments: That’s clearly better, but stable Rust has another syntax in its place: structs. A struct is a named argument with one extra type name. On top of that, we also get arbitrary field order: We also get typo checking, autocomplete, and per-field documentation for free! And we can put invariants on the type and pass the arguments around as values. And, perhaps most importantly, the names belong to the type , rather than becoming part of every function’s calling convention. That last property neatly avoids several problems with actual named arguments. Consider function pointers: What would the parameter names of be? With an argument struct, the question simply wouldn’t come up: If names are semantically important, give the names a type. If they aren’t, then… don’t. There is another delightful benefit here. Steve brings up evaluation order: I.e., should arguments be evaluated in the order they appear at the call site, or in the order the parameters appear in the declaration? Here, does that mean computing before moving or after? Rust already answered this question for structs: Expressions are evaluated where you wrote them. No new rules required. This feels extremely idiomatic to me: rather than teaching function calls a second field-like syntax with subtly different semantics, just use the field syntax that already exists. Of course, declaring a bespoke argument type for every two-argument function would be ridiculous. I would not write: That would be silly. The trick is to notice that named arguments are most useful exactly where an argument bundle becomes conceptually meaningful, which is the same point at which you reach for a struct anyway. These are bad: And these are often better APIs anyway : The design pressure forced us to uncover missing domain concepts. An optional argument is, to some extent, an argument which may or may not exist. Rust has a type for that. This is not as pleasant as: But it has a useful property: the optionality appears in the function’s type. There isn’t a hidden second calling convention for . There is but one function: and every caller supplies both arguments. This is obviously not what you want once you have six optional arguments: I’ve personally been found guilty of this pattern in the past. The problem is that the arguments have stopped being a parameter list and started being configuration. So: Instead of optional arguments, we deal with data. And that adds a nice property: there is no special distinction between “arguments supplied syntactically to this invocation” and “options I calculated elsewhere.” It composes nicely because it’s just a value. Now the obvious objection: writing all those s is terrible. Correct. So don’t. That’s why we have and struct update syntax: That is getting awfully close to: with one minor wrinkle: That is not nothing. But look at what we didn’t have to add: rules for which arguments may be omitted, how positional and named arguments interact, or whether you can omit something in the middle. There’s no special syntax for declaring parameter defaults, no question about whether default expressions run at declaration time or invocation time, and no special representation in types. is just a trait, and function calls remain untouched. Defaults are now usable independently of the function: That is frequently useful in its own right. For library APIs, I often like being slightly more explicit: I think this gets most of the important bits right. Sometimes even the options struct is too noisy, often when construction requires validation or conversion. Then, yes, there is the builder: Steve is right that builders should not be the default. They can become their own tiny programming language. But a small builder has a very useful property: each “argument” is an ordinary method call. That means we can do things like: Doing that with language-level keyword arguments generally requires constructing a map, splatting things, or some other mechanism. In Rust, it’s method calls. I personally find this very pleasing to read. In Java, you can write: Rust doesn’t let you define both: I am very happy about this. But there are several different things people mean when they say they want overloading, and Rust already covers most of them separately. Give them different names: The standard library does this often. See and , for example. This costs the library author one additional name (often just ) and saves every user from doing overload resolution in their head. Use a trait. The standard library does this all the time with traits like , , and . For example: This gives us another useful part of overload-like behavior: one API can accept different input types. For owned conversion: That’s just a single function with one parameter list and trait dispatch. And unlike unrestricted overloading, the relationship between accepted types is explicit: they work as long as they satisfy the bound. That’s a trait, too: That’s polymorphism; we just put it in the trait system instead of in name resolution. Steve’s Ruby example has this equally lovely and terrifying quality: These calls look like they’re invoking one conceptual operation, but they mean wildly different things. In Rust, we can model that directly: That’s more verbose, but in a good way. And I can ask, “Hey editor, what can I redirect to?” and the editor replies with the variants of . That’s more helpful than “read the docs and discover which keys and values this hash accepts.” If we really cared about smoothing down the edges, we could add conversions: And for the structurally interesting case: Remember that this has no runtime cost and is fully type-safe. Not bad for a compiled language. An “options hash” is basically a dynamically typed anonymous struct. So the extremely boring Rust translation is: use a statically typed, named struct. Yes, the Rust version is noisier, but it also detects when we misspell . It’s impossible to pass a string where the status code goes, and it’s straightforward to list every supported option. I don’t think Rust should optimize for making the syntax as dense as possible. Instead, if a set of options is common enough to deserve convenient syntax, it is probably common enough to justify a type. Now, options have a name and the fields can be documented in one place. Rust does not have general-purpose variadic Rust functions. But once again, it already has several ways of expressing the same concept. If all arguments have the same type, take a slice: Or accept an iterator: That is arguably more composable than: because the caller can naturally pass an existing collection. (All type-safe, of course, and with zero indirection at runtime.) If the arguments are heterogeneous, the last resort is to write a custom macro. To be clear, I would not use macros just to fake variadic functions, but I do like how macros can be used in stable Rust, and how the exclamation mark stands out from normal function calls. There is one tiny affordance in all of these examples that I think deserves more credit: field-init shorthand. Rust lets you turn this: This directly addresses one of Steve’s complaints about keyword arguments: Rust’s answer is effectively: In my opinion, that’s even better than keyword arguments. That’s because the labels are still present, the duplication disappears, and nothing needs to change in how we call functions. The thing I like the most about Rust is how each concept nicely interacts with the others. That is not an easy task, and Rust deserves a lot of credit for that. For example, suppose we want a complex HTTP request API with: We could imagine a pile of language features that lets us write: Now wouldn’t that be nice? However, we can already do this in stable Rust with a combination of already existing, composable features: And all we had to do was write the code we’d likely write anyway: Or maybe you prefer a builder? We combined standard Rust concepts: structs, enums, , , struct update syntax, field-init shorthand, traits, generics, iterators, and methods. Those mechanisms are all useful far beyond argument passing. Basic Rust syntax is all the machinery required to build ergonomic APIs. Keeping things simple doesn’t mean worse ergonomics. The obvious response to everything above is: Come on. These aren’t actually named/default/overloaded/variadic arguments. They’re workarounds. All of the above might be useful, though localized, syntax improvements. But the hidden tax is that the language becomes more complex, for arguably little gain. Friction in APIs often pushes us toward solutions that turn out to be useful beyond the original problem: In a sense, the concrete issue points to a broader design problem, and resolving it opens up completely new ways to solve similar problems. That’s great systems design. I think there’s a broader design principle behind all of this. A common design philosophy in dynamic languages is to make familiar constructs more powerful by overloading them with additional semantics. After all, that is one affordance which dynamic typing allows: the ability to decide the meaning of an object at runtime. Rust, however, tends to move complexity outward and let the type system do all the work . One might ask: “In the age of agentic development, doesn’t verbosity become cheaper while redundant labels may make a call easier to understand locally?” I agree with the premise. I’m less sure it changes the conclusion. An agent looking at: gets essentially the same local information. Arguably it gets more: gives the bundle a semantic identity which the function parameter list alone does not. says something useful to both humans and agents. is not merely an optional syntactic argument to this particular invocation; it is a way to configure a request. And if agents really do make typing cost increasingly irrelevant, then the principal downside of these slightly-more-verbose Rust idioms gets cheaper too. The robots can type for me. I’m not opposed to Rust ever gaining named arguments. There may be a proposal that finds a tiny, coherent design which handles patterns, function pointers, traits, evaluation order, compatibility, and all the other sharp edges described. But I don’t feel much urgency. Stable Rust already gives me structs for named options, and for optional values and defaults, traits and enums for varied inputs, and slices and iterators for repeated arguments. Collectively, they cover a lot of ground. And they do it by reusing features Rust already needs. And I think that’s a core part of Rust’s design philosophy: finding the smallest, composable, orthogonal set of abstractions, which, when combined, can solve many problems in elegant ways. The whole is greater than the sum of its parts. Where to go from here Want a second opinion on your team’s Rust APIs? Let’s review the types and abstractions together. a required URL, several accepted URL-like input types, named options, optional timeout, configurable redirects, and a variable number of headers. Actual named arguments might let me turn into: That surely is nicer at the call site. Actual default arguments might let me write instead of introducing . Actual overloading might let two functions share the same name instead of forcing me to invent . These four coordinates become a . Seven random config parameters become . A bunch of dynamically accepted values turn into an enum. A family of related operations becomes a trait. For your team . Training, code review, and architecture support to ship Rust with confidence. For yourself . 1-on-1 mentorship for Rust design, architecture, and code review on real projects.

0 views
Steve Klabnik 3 weeks ago

Arguing about arguments

Here’s some Rust code: In programming language jargon, we call and parameters and and arguments . Rust does not have many fancy features related to parameters and arguments. But other languages do. For example, here’s a function in Ruby , another language I’ve written a ton of: I used to love Ruby, and now I love Rust. And to be honest, and friends are a huge reason why I am so against Rust gaining these sorts of fancy features: they can be really concise, and beautiful to a certain kind of eye, but it also makes it really, really hard to know what all you can do with a function. You pretty much have to hope that someone has written good documentation, and while Rails’s documentation is pretty good, not every library is going to have it. To be clear about it, there are several things bundled together here: flexible argument types, options hashes, defaults, and keyword-looking syntax. It’s that whole style of API that shaped my skepticism, not argument labels by themselves. So, what exactly are all of these features? The simplest one is named parameters . As you might guess, this means that we can explicitly use parameter names at the call site: Named parameters are nice because they can give you more information at the call site, but also they can be more flexible. Some languages also let you supply named arguments in a different order: The downside of named parameters is that they can get quite verbose: I tried to not strawman this by showing an extreme example, but can take 23 different keyword arguments, and so this could get quite, quite long. Named arguments can be good when passing expressions, so we can understand something more at the call site. In the above, is much nicer than a bare , if you know HTTP well it’s kind of obvious what it is, but the additional clarity is nice. But it can be common to break truly complex examples down into variables that we end up forwarding to the function, and then it becomes verbose and redundant. As an example from inside FastAPI where is invoked directly with all of those options: We actually snuck in another two features into this example: how did we only pass five arguments into a function that has 23 parameters? As the name implies, optional arguments is a feature where you can choose to not pass in an argument, and default arguments lets you set a default when an argument is not passed. To go back to that example: We only passed in five of the 22 possible arguments. If we had to pass all 22 every time, that would be extremely verbose. So by declaring defaults for some parameters, we can leave them off of the call site, and not pass arguments for them, and get the defaults instead. We saw this in the Ruby: The are defaults for and : they’re an empty hash map. You’ll often find optional and default arguments come together: after all, if you don’t pass an argument in, what should the value of that parameter be? If your language has some sort of null value, that can be one solution, and it can feel like optional arguments without default arguments. But you can truly get one without the other with our next feature, function overloading: Languages that support function overloading allow you to define multiple functions with the same name, but different signatures. Which function gets invoked depends on which arguments you pass. This lets us truly have optional arguments without default arguments. In Java: Here, if you pass in a timeout, you get the first function, and if you don’t, you get the second. Instead of null, the parameter just doesn’t exist at all. This is a specific form of something called “function dispatch,” which basically means “hey when a function is invoked, what exactly gets called?” There are a lot of different ways to do this, and I don’t want this post to get into all of that right now, but there is a multitude of possibilities here: static vs dynamic, single vs multiple, predicate dispatch, pattern matching dispatch, prototype based dispatch… maybe I’ll write a post about that someday too. So, what are the pros and cons here? As I said above, Rust doesn’t support any of these three features right now. And they have been a persistent request from the community for years. Here is a 12 year old GitHub issue about this support, and it even includes something we didn’t talk about, “variable arity argument lists.” That gets me to the first thing that I don’t like about these features: there are a lot of them. And they are all intertwined. As we mentioned before, if you have optional arguments, you probably want default arguments. If you have named parameters, do you want to support only named parameters, or do you want to support both? It’s so tempting to just support all of them, and then you’ve added a huge mountain of complexity. In Rust right now, you have to write multiple functions with slightly different names: For the simple cost of “write two different names for your functions,” you get to keep the rules very simple: there is one function definition, and you invoke it by name. Done. Would it be nice to be able to say all of these: Sure, maybe. But the cost feels very, very high to me. If you’re used to these features, maybe it doesn’t seem that high, but I have a Ruby tattoo on my body. I’ve seen some shit. And so I’ve grown to enjoy Rust’s simplicity in this area. Yes, if you have a function with tons of parameters, you may need to write a builder instead, and that comes with its own form of verbosity: But the language itself stays simple, and the rules are easy. For a language that’s already perceived as complex, this has always felt right to me. And that’s why I’ve pushed back against the various proposals to extend Rust in this way all of these years. But recently, I changed my opinion on one of these features, and one alone, and I would consider it okay if Rust gained them. Maybe. Also, it’s important to note that I haven’t worked on Rust in years, and so my opinions are kind of irrelevant, but whatever: it’s my blog, this is what opinions are for. I think Rust could be okay with named parameters, but not optional or default ones. And what changed my opinion is coding agents, actually. I’m sorry, maybe you’re sick of me talking about AI, but hear me out. One of my beefs with named parameters, as mentioned above, is verbosity. But what you gain for that verbosity is clarity. I don’t think that verbosity and clarity are always the same thing; I like rather than / , and find it more readable, personally. But let’s take this function from the crate: This does really increase readability, no question about it. But for me, typing all of that out just isn’t really super worth the squeeze. But when Claude is gonna write it? I care a lot less. And the readability advantage for humans is even stronger for agents: the extra clarity of the text inline seems to (I haven’t run real evals on this yet…) be more helpful and use less tokens and calls when looking at the actual call site. If an agent were reading that named example above, it could tell that is the value without having to go look up the function signature itself. “What’s good for humans is true for agents” strikes again. This is also why I’m still against optional arguments: they deliberately hide what is being passed in to the function, and obscure clarity at the call site. What’s nice about them is that you have to type less. But I’m not typing myself anymore. So the benefit just isn’t there, but the drawbacks are. This also doesn’t solve the verbosity side of passing in our FastAPI example, it is still redundant. But one part of the downsides have been ameliorated. Notably, all of this reasoning also applies to the builder pattern, and it’s why I try to use builders sparingly in Rust. It can emulate named arguments, which is good, but it can also emulate optional/default/variable arguments, which is bad. The juice is worth the squeeze sometimes, but not all the time, and probably not even the majority of the time. So we’ve ameliorated the largest downside, but get to keep the largest upside. That seems good. But even if we like the feature, there are still lots of practical concerns with Rust specifically that make it not a slam-dunk to add them to the language. Now, “is this feature good in the abstract” is very different from “should this feature” be implemented? There are a lot of open questions and drawbacks to implementing named parameters. A simple one is that you don’t get rid of all of those previous functions, will exist forever. That’s a relatively minor downside to some of the harder problems. First, strictly speaking Rust parameters aren’t names, they’re patterns. A name is a simple pattern, but you can get more complex: This parameter doesn’t have a name, it’s a pattern that introduces two bindings. You could come up with some syntax that gives things external names vs internal names, but there’s that complexity revealing its head again. What happens when functions become values? I can write this today: What name do we use for ’s parameters? They can be named different things. Does this become an error? What names can you use when invoking ? Making this more complicated, Rust actually will accept this code right now: The names in are just documentation, as you can see they’re not required. Does this become an error? Do we require to change its names? The same goes for trait definitions, actually: Is this an error? Is it okay? Here’s another problem: evaluation order. Right now, Rust always evaluates parameters from left to right at the call site. So if we use the named version: If we keep left to right evaluation order according to the definition, but let you use named values in any order, suddenly some orders at the call site compile and some orders do not. Do we change evaluation order to do whatever compiles? That feels very dangerous and confusing. Rust does let you do this with struct fields: This is one reason why a lot of proposals for named arguments try to unify them with structs. As I said before, not insurmountable, just things that need to be sorted. Another small thing is a compatibility hazard: if we start naming parameters, renaming them breaks callers. Is that okay? Could you introduce aliases to help with this? That’s more complexity… These are just some of the problems that come up in the design space, and this post is getting long, so I’ll leave it at this for now. But proposals for doing this have to engage with all of this and more, which is why language design is difficult. I like to think that I change my opinions over time as the world or my understanding of it changes. So I thought it might be interesting to share a way in which an opinion that I’ve held for over a decade has changed a bit lately. I’m still not sure if named parameters are right for Rust, but I’m at least somewhat open to the idea these days. We’ll see what the language team decides if anyone does decide to keep trying to push this forward. And people do, for example, I am writing this in part because I saw this post go by today. This post isn’t intended to be support or a rebuttal of it, I actually didn’t even read it yet, it’s just that I’ve been thinking about this a lot lately (while working on Rue ) and seeing that go by pushed me to write some of the thoughts down. You should probably go check it out, as it’s an actual proposal in this space and this post is more of an exploration of the general idea and my own thoughts on it. Here’s my post about this post on BlueSky: Arguing about arguments steveklabnik.com/writing/argu... Arguing about arguments Here’s some Rust code: // define a function fn foo (x : i32 , y : i32 ) -> i32 { // body elided } // call it let z = foo ( 5 , 6 ); In programming language jargon, we call x and y parameters and 5 and 6 arguments . Rust does not have many fancy features related to parameters and…

0 views
Jason Tucker 4 weeks ago

Inside Quartermaster: A Conversation with Lewis Egley

I've been running Quartermaster on my phone for months now, watching it grow from a solid Sonarr and Radarr companion into something that's trying to handle my entire homelab in one app. When I reviewed it back in the spring , it was already doing more than most apps in this space attempt. Since then, developer Lewis Egley has shipped an enormous amount of work, and the app is now getting a Docker-based companion tool that changes what it's capable of entirely. I sat down with Lewis, virtually and asynchronously (Lewis as you will find is British and I'm located in California so there is an 8 hour time difference and Lewis doesn't sleep much) over Discord, to talk about where Quartermaster came from, what Companion actually does under the hood, and where he sees this whole thing heading. He was candid about the late nights, the pressure of shipping fast in a space that's gotten suspicious of anyone who does, and what it's like going from a private side project to something with a real user base watching every move. If you want to follow along or dig into any of this yourself, the app is on the App Store and the GitHub repo for Companion is open source under MIT. There's an active community over on Reddit and in the Discord server, which is honestly the best place to catch build announcements before anyone else does. Walk me through your path in tech, how'd you go from where you started to building Quartermaster? It's a very good question and sort of cliché if you will :) The world of computers has always interested me, ever since I was young. I started making little programs as a child using Python, and weird little websites I'd show my friends. At the time, people my age didn't understand, but I did, and that was enough. In regards to my professional career, I started as any other young and hungry tech advocate. I got my study on! I majored in Computer Science and got my diploma, and then I started out in the working world. My first ever job was IT 1st line, a helpdesk position. Looking back, I hated it, but it's the only way in. A true rite of passage for anyone who wants to start. Over the years I started picking up skills and moved quickly into cloud infrastructure, building out cloud environments for businesses up and down the UK. That opened the door to DevOps and ultimately software engineering. I built Quartermaster as a personal project and ran it locally for a few months. Then I was speaking with a few guys on a tech forum and they really wanted to test it. They gave me improvement ideas, I built them, and so on and so on. The app launched on the 3rd of July and I haven't looked back since, just a lot of late nights, haha. Is Quartermaster your full-time thing now, or are you building it around a day job? Believe it or not, Quartermaster is not my full-time job. It is, and probably always will be, a side project. I love my full-time job, the companies I get to work with and the people I meet. It has taught me everything I know and I just don't think I'm ready to turn my back on it just yet. What pulled you into the homelab space specifically? Are you running a big stack yourself, and if so what's in it? One word. Community. The ideas people come up with, and everything in between, is honestly captivating. It's odd, because it's essentially its own ecosystem, and once you're in it, you're in it. I've been homelabbing for around three years and every day I'm amazed by what people do. No matter where you look there's something out there for everyone, a dashboard here, a service there. There just isn't anything quite like it. The stack I run is small compared to some. I have a soft spot for networking, so it's Unifi throughout, switches, CCTV, the gateway. For media I run a 16GB ECC box with a 12TB array, which covers around 60% of what Quartermaster supports. Obviously I have to spin the rest up locally for proper testing. Any other apps, side projects, or open-source work people might know you from? No, lol. Long-time lurker, first time going public. I'm a very honest guy who keeps himself to himself, so opening the floodgates on a project like QM as a solo dev has been a very big learning curve. Your day job took you through cloud infrastructure and DevOps before software engineering. How much of Companion's design, the proxy layer, the tiered access, the whole approach, came out of instincts from that world rather than from building consumer apps? Almost all of it, if I'm honest. The consumer app side I had to learn as I went, but the Companion architecture came out of habits I already had. The clearest example is the Docker access model. A consumer app instinct would be a settings toggle, switch on Docker management, done. What I built instead is a ceiling you set at deploy time. The Compose profile you start with decides the maximum access the process can ever have, and every profile boots into read only regardless. The owner can raise the active mode from inside the panel, but only up to the ceiling they already committed to on the host. You cannot escalate past it from the UI, because the UI was never given the authority in the first place. That's just least privilege, and it's the sort of thing you internalise building cloud infra for other people's businesses. Nobody gets prod write access because a toggle in a dashboard said so. The socket proxy is the same idea. Companion doesn't get the raw Docker socket, it gets a keyed proxy in front of it with its own scoped surface. The other bit that came from that world is being blunt in the docs about what the thing can actually see. Docker inspection exposes container metadata, mounts, network layout and anything sitting in environment variables. A read only mount still means Companion can read that file. That is in the README because someone running this at home deserves the same honesty I'd give a client during a design review. Do you see Quartermaster becoming a full home lab platform in its own right, or is Companion still in service of the mobile app? I don't know yet in all honesty, and I'd rather say that than invent a roadmap. Companion came about because I hit the ceiling of what a phone can do on its own. Some things need something running on the box with real permissions, and no amount of clever iOS code gets you round that. So it started very much in service of the app, and it does that job well. Where it goes from here is the exciting part, because it isn't entirely my call anymore. Open sourcing it under MIT means people are already running it for things I never designed it for, and that's brilliant to watch. It's the whole reason I love this space. You put something out and the community immediately finds uses you'd never have thought of on your own. I'm still careful with the word platform, mind. It carries promises about support and longevity, and I want to earn those rather than claim them. But if it keeps growing the way it has, and I can keep it secure and well maintained, then I'd love to see where it ends up. Right now I'm just enjoying building it. You clearly thought hard about the trust boundary here, the proxy, the hard ceiling, the tamper-evident config. Was there a specific moment or scare that shaped that approach, or was it just discipline from day one? Discipline, mostly, though it came from a very specific realisation rather than a scare. When I sat down and properly thought about what I was asking people to install, it focused things fast. Companion holds service credentials and can be granted Docker control on the host. Get the owner session and you don't have a media dashboard, you've got someone's box. Once that landed, a lot of the design decided itself. That's why the setup transfer is short lived and single use. It's why pairing makes you compare words on both screens before you approve anything. It's why the default install is read only, and anything beyond that has to be a deliberate choice you make at the host level rather than a toggle in an app. The bit I'm most pleased with is that you don't have to take my word for any of it. There's a verification script in the repo that runs the boundaries against a live instance. If you're going to ask people to trust something running on their own hardware, they should be able to check rather than just believe you. Homelabbers are all over the map, some just want Home Assistant, Unraid for media, and Immich for the family photos. Others are running full racks with oddball switch configs, learning Proxmox, juggling Tailscale, Twingate, ZeroTier, Netbird, Netmaker. That's a massive range of technical comfort under one roof. How do you think about serving both ends of that spectrum with Quartermaster, and does that split change how you approach onboarding and documentation? The way I've approached it is that the complexity lives in their setup, not in mine. I don't try to guess how someone's network is put together. I just give them enough ways to describe it that whatever they've built, they can point QM at it. So you get a local address, an optional remote one, a home WiFi pin so it forces local when you're in, reverse proxy domains, Cloudflare Access header sets, custom headers, basic auth, and certificate pinning for self signed setups. Tailscale isn't a special mode, it's just an address, because to the app it should be. Someone on a Synology with three apps uses one field of that. Someone with a rack and five overlay networks uses most of them. Same app, same flow. Onboarding does branch, though not by skill level. It branches by how you're arriving. There's a manual path, a Companion path where the server exports your whole stack and the phone imports it in one go, a demo mode, and a restore path. Then you pick your services off a list, and it walks you through connecting them one at a time, most depended on first. There's very little discovery, which is deliberate. Where I'd say I've genuinely got it right is what happens when a connection fails. The Test is the moment everything hinges on, so that's where the work went. Four stages, real error codes, the server's own refusal text surfaced verbatim rather than swallowed, and one tap fixes for the specific traps, certificate trust, SSH host keys, Synology OTP, a port left prefilled on a public domain. You can't save a service that hasn't passed a Test, so nothing sits there claiming to be connected when it isn't. You mentioned there's very little service discovery by design, and that onboarding branches by how someone arrives rather than their skill level. That's a strong opinion, a lot of apps in this space lean hard into auto-discovery as a selling point. What made you bet against that? Mostly that on iOS it doesn't work as well as the marketing suggests, and the failure mode is brutal. The Local Network permission is a one hot prompt. You get it once, and if the user denies it there is no API to ask again. They have to go into Settings and find it themselves. So if that prompt fires during a background dashboard load rather than something the user deliberately did, the request races the prompt, fails as a generic "couldn't connect", and you've burned your one chance. The app actually fires a throwaway probe when onboarding opens specifically to get that prompt out of the way at a moment that makes sense to the user. Then there's the yield. Bonjour browsing on iOS needs you to declare every service type up front in the app's config, and most selfhosted software doesn't advertise over mDNS anyway. Radarr doesn't. Sonarr doesn't. SABnzbd doesn't. So you'd be building it for almost nothing. Subnet scanning is worse. Hundreds of connection attempts, straight into iOS's six per host limit, and a conversation with App Review about why your media app is port scanning. And a big chunk of these users are on Tailscale or behind CGNAT, where sweeping the local subnet finds nothing that matters. So I moved discovery to where it can actually work, which is a box that already has Docker. Companion reads the container list, works out what's running, and reads the API keys straight out of the config files on disk. The *arrs keep theirs in config.xml, SAB in sabnzbd.ini, and so on. The user types nothing. That's real discovery, it just isn't happening on the phone. Plex is the other exception, and it's the same principle. It's account side, not network side. You sign in and Plex tells you every server on your account with every route, so the app picks the best one. Where a proper API exists I'll use it. I'm just not going to guess at somebody's subnet. A lot of the competition in this space, LunaSea, Zagreus, Ruddarr, Unraid Deck, are unitaskers, they do one thing really well. You told me you're not using AI to build any of this, it's all hand-written. That's a lot of surface area to cover solo without leaning on tooling to move faster. What made you go all-in on the everything-app bet instead of picking a lane and doing it well? Because the unitasker advantage disappears the moment you run more than one thing, and almost everyone in this space runs more than one thing. The bet wasn't really "cover everything". It was that the hard part isn't the services, it's the layer underneath them. Getting a connection to somebody's box is genuinely difficult. Dual local and remote addresses that switch on their own, self signed certificates on iOS, reverse proxies, Cloudflare Access, custom headers, per host concurrency limits, a transport that won't follow a redirect and replay your API key somewhere you never configured. That's about thirteen thousand lines of plumbing, and I only had to write it once. Everything after that is a service riding on top of it. A new integration inherits the URL assembly, the auth handling, the certificate pinning, the reachability engine, the four stage connection test, the add form, the backup and restore, all of it. I won't pretend it's cheap though. A new service is still six to eight thousand lines by the time you've done the client, the screens, the mappers and the demo data (needed for Apple review). There's no plugin system, no manifest, nothing schema driven. It's all compiled in. That's linear cost forever and I knew that going in. What I got for it is that each of the 59 services has a real screen and real actions rather than a generic card. The schema driven approach would have let me claim two hundred services that all look identical and none of which actually does anything. I'd rather have 59 that work. Unraid Deck does almost exactly what Companion does, direct Docker control, SSH, local push notifications, no hosted relay, but it stays narrowly focused on just Unraid. Companion's trying to sit underneath a much bigger app that spans dozens of services. What's the trade-off there, and do you think staying narrow like that is actually the smarter long term bet? Narrow is smarter if you're solving one platform's problem. I'd not argue with that at all, and I think Unraid Deck is doing the right thing for what it's for. Companion isn't Unraid specific. It's a Node app talking to the Docker API, so it works against Proxmox, Synology, bare metal, whatever. Unraid is just the best documented install path because that's where a lot of people are. But that generality has a cost. Deck can make assumptions about paths, about the array, about the whole shape of the system. I can't make any of those, so I end up handling Linux host paths generically, Synology's volume layout, socket proxying instead of a raw socket mount, and so on. The trade off is what Companion is for. It isn't primarily a Docker manager, it's the thing that makes setup disappear. It reads your config files, finds your keys, works out you're running three Radarrs on different ports, and hands the phone an encrypted bundle so you type nothing. That job only exists because there's a big app sitting on top of it that needs 59 services configured. A narrow tool doesn't have that problem to solve, which is exactly why it can stay narrow. How do you decide what to actually support in the mobile app? I noticed Companion now has a marketplace, is that the mechanism for covering more of these different setups without you having to hand-build support for every single one in the mobile app and just let Companion handle it? It's a catalogue generated from Companion's/QMs own supported list of what it knows about. There are exactly three hand reviewed Compose starters in there. Everything else is either generated from a template or connect only. And no, it isn't a way round hand building support. I wish it were, but I'd be lying if I said otherwise. The app's list of service types is compiled in. It's a fixed union in the source, it's a key in dozens of exhaustive lookups, and it's the discriminant the app stores things under. There's a hard boundary on import that drops anything not in that list, with a test asserting it does. So if Companion learns about something new tomorrow, it can detect it, deploy it and show it in the web UI, but the phone will not treat it as a service. No card or even screen and certainly no status probe. That's a build time thing and no amount of Companion architecture changes it. There's one escape hatch, and it's honest about it all, the Companion control plane in the app shows containers, stacks, images and events generically, without knowing what any of them are. So something new turns up there the moment you deploy it. But it turns up as a container, not as an integration. How I actually decide what to support is much less interesting than that. It's what people ask for in the Discord, weighted by whether I can get a real instance running to test against. That last bit matters more than people expect. I pulled up the actual Marketplace in Companion. Right now every one of the built-in entries is first-party, curated by you, and the catalog banner says every deployment still starts with a review. But there's already a Sources tab sitting there, empty at zero right now, that lets people point Companion at external Portainer v2 template feeds, and it says outright those entries aren't reviewed by you. That's a real two-tier trust model already built and waiting. Was that the plan from day one, community sources living alongside your own with clearly different guarantees, and how worried are you about someone pointing that at something malicious? I wouldn't say the exact shape of it was mapped out from day one, but the separation was deliberate once the Marketplace started taking shape. There had to be a hard distinction between something included by me and something a user has added themselves. If it appears in the built-in catalogue, I'm putting my name against it. I've looked at it, I know where the image comes from, I know what it mounts and I know what permissions it asks for. An external source cannot inherit that trust just because Companion happens to be displaying it. And yes, a malicious feed is a real risk. A Compose template can ask for privileged mode, mount the Docker socket, mount the host filesystem or run an image that does something completely different from what its description claims. There is no honest way for Companion to turn arbitrary third-party Compose into something safe automatically. That's why external entries are labelled differently and why deployment still starts with a review. Companion shows you what is actually going to be created before anything runs. It doesn't quietly install something because it appeared in a feed. But I don't want to pretend that removes the responsibility from the person deploying it. If you add an untrusted source and then approve a template with access to your host, Companion can't make that decision harmless. The aim is to make the trust boundary obvious and make dangerous requests visible, not to claim I can sanitise arbitrary code from the internet. You mentioned there are only three hand-reviewed Compose starters in the Marketplace right now, everything else is generated or connect-only, and even if Companion learns about something new, it can't become a real integration in the phone app because the service list is compiled in. So what would "other devs contributing to the Marketplace" actually look like in practice, more Compose starters and Sources feeds on the Companion side, or is there a future where the compiled-in service list itself opens up somehow? Right now it means the Companion side. Other developers can publish Portainer feeds, build Compose starters and help Companion understand how something is deployed or detected. If something is going into the first party catalogue, I still need to review it before it gets that guarantee. Alternatively, somebody can maintain their own source and users can choose to add it with the external-source warning attached. What it does not mean today is dropping a manifest into Companion and suddenly gaining a full Quartermaster integration on the phone. The phone app is deliberately much stricter than that. A service type is compiled into the app and tied into the connection layer, storage, backup and restore, onboarding, demo data and native screens. Opening that at runtime would either require an enormous plugin system or reduce integrations to generic cards driven by a schema. Neither gives me what I want from Quartermaster. So in the near term, community contribution means deployments, templates, detection and knowledge on the Companion side. For the app itself, people can request integrations, help me get a test instance running, document strange API behaviour and test the result, but the actual implementation still has to become part of the app and go through a release. I won't say the service system can never open up, but there isn't a hidden plugin SDK around the corner. If I ever do it, it has to produce something that feels like a real Quartermaster integration rather than a web form wearing an iOS skin. A while back a security researcher found serious auth bypass vulnerabilities in Huntarr , and the maintainer banned people who raised concerns about it. It's become the go-to cautionary tale people point to for vibe-coded apps cutting corners. Has that story stuck with you? Does having something like that out there change how you think about your own pace, or your own security process? It's stuck with me in the sense that I've watched how the community reacted to it, and that told me something useful. People in this space care about this stuff. They will look, they will test, and they'll say something when they find it. What it reinforced more than anything is how you handle it when someone does come to you. If a researcher turns up at my door tomorrow with a genuine finding, that's someone doing me an enormous favour for free. The right response is thanks, then a fix, then telling everyone what happened. Closing the door on that is how a small problem becomes a permanent reputation. On pace, it hasn't changed it exactly, but it's part of why the security model went in early rather than later. It's much harder to bolt a trust boundary onto something after people are already running it. Do it up front and you're building on top of it rather than retrofitting round it. There's a lot of noise right now about "vibe coded" apps, and Huntarr's become the example everyone points to, real security holes, a maintainer who didn't seem to understand what he was shipping. Where do you actually sit on that? Do you use AI tools in your workflow, and if so, how do you keep the line between using them well and leaning on them too hard? I sit somewhere that probably annoys both camps. I don't use AI to build Quartermaster. Not the app, nor the Companion. Where I do use it is checking a GitHub URL for a new integration in case an API's changed and I'd otherwise miss it. That's a research task and it's genuinely useful for that. But I think the "vibe coded" label gets used as a lazy shorthand, and it points at the wrong thing. The tool isn't the problem. You can write every line by hand and still ship something dangerous if you've never had to think about a trust boundary. Equally you could use AI sensibly, review properly, and be completely fine. The real question is whether you understand what you're shipping. That's it. Can you explain what you decided, what it exposes, and what happens if someone attacks it? If you can't answer that, it doesn't matter who or what wrote the code. I'll use it to research and to check my blind spots. I won't use it to make decisions I'd struggle to defend afterwards. I've watched you build this out in real time, and you've even put a verification script in the repo so people can check the security boundaries themselves instead of just taking your word for it, which is the exact opposite of what happened with Huntarr. But I know "vibe coding" accusations get thrown around fast, especially with that story still fresh in people's minds. Does being anywhere near that comparison worry you? Has anyone actually come at you with that kind of accusation, and how do you handle it when the work itself is the real answer? It doesn't worry me, no. If someone wants to lob that at me they're welcome to, but they'll be arguing with a repo that's sat there in the open. The security model was designed deliberately, because I've spent years doing exactly this kind of work for businesses and I know what happens when you get it wrong. Least privilege, a ceiling you set at deploy time, read only by default, a proxy in front of the socket rather than handing over the socket itself. None of that is accidental and I can explain every bit of it. But the strongest answer isn't me telling anyone that. It's that I'd rather nobody took my word for it in the first place. That's the entire reason the verification script exists. Run it against a live instance and it tests the setup and authentication boundaries for you. If you think I've cut corners, that's the quickest way to prove it. The repo's open, the app's on the store. Have at it. You've shared some pretty stark numbers on what the hosted notification relay was actually costing you to run. What made you consider dropping it entirely, and how did you land on moving notifications into Companion as a local relay instead? Did that same cost reality shape the new pricing structure for 1.2, and how did you balance that against keeping your lifetime purchasers taken care of? The relay costs got out of hand, plainly. We're talking millions of requests a day through Cloudflare, and that's not sustainable for one person funding it out of pocket. The fix is Companion. Notifications are moving to a local relay, which means people will be able to set their own alerts and run the whole thing on their own box rather than everything routing through infrastructure I'm paying for. That's better architecturally as well as financially, which is the nice version of a cost problem. It's also more in keeping with the whole point of self-hosting. Your stuff, your box. The 1.2 price change is partly those notification costs and partly everything else that's landed since launch. Companion, the custom built macOS version, all of it. I've kept the increase as small as I can and it'll still come in cheaper than the alternatives. The important bit: every current subscription stays at the price you're on until you cancel, and everyone who bought lifetime keeps lifetime. That was never up for discussion. People backed this thing early, before it had proved anything, and they're not getting punished for it. You mentioned QM is a side project on top of a full-time job, and you shipped an enormous amount of work in eight weeks. What does a week actually look like for you right now, and how close have you come to burning out on it? Honestly? Weekends are soaked. During the week it's evenings after work, and more often than not I'm going to bed around four in the morning. Then up for the day job. That's just been the rhythm since launch. My partner hasn't always been thrilled about it, and I don't blame her. But when I show her what's actually happening, the numbers, the people using it, the messages coming in, she's amazed. That helps. It's my passion, and having someone who gets that even when it's eating the weekend makes a big difference. As for burnout, it's real and I won't pretend otherwise. I make mistakes when I'm running on not much sleep. Things slip through that wouldn't if I'd been fresh. That's genuinely where the closed beta community has been unreal. They catch things and they tell me straight when something's broken. When you're a solo dev at two in the morning, having a group of people who care about the project as much as you do is worth more than any amount of extra hours I could put in. I'd have shipped a much worse app without them. You said this is the first project you've taken public after being a long-time lurker. What's that jump been like, going from something you built quietly for yourself to a real app with a user base and an active Discord community? Terrifying, then brilliant, roughly in that order. Building something quietly for yourself is safe. Nobody sees the bad decisions, nobody files a bug at midnight, and if it breaks it only breaks for you. Putting it out means every choice you made alone is suddenly up for discussion by people who know what they're talking about. That first week I checked everything constantly. What I didn't expect was how generous people would be. I'd braced for criticism and what I got was people wanting it to work. Detailed bug reports, feature ideas better than mine, folk explaining their setups so I could understand why something wasn't behaving. The Discord has genuinely made the app better in ways I couldn't have on my own. The other thing is that it's changed how I build. When it was just mine I could cut corners and know where the bodies were buried. Now there are people relying on it, and that raises the bar on everything. Not in a stressful way particularly, more that it makes you take it seriously. I'm still a fairly private person, so going from lurker to having a community around something I made has been a big adjustment. But I'd not go back. Whatever I thought I'd get out of shipping this, the people have been the best part of it by a distance. I'd just add a closing note on that. Quartermaster is ultimately just code running on people's devices. What actually drives the late nights, the new features and everything in between is the community. They keep me going and I couldn't ask for a better one. There's a stereotype that British people apologize for everything, even when it's not their fault, always "ever so sorry" or "terribly sorry" for the smallest things. You've said sorry in Discord more than once for delays on something you're building for free in your spare time. Is that just very British of you, or do you genuinely feel like you owe people an apology? How do you deal with the pressure of a deadline you set for yourself versus one the community's expecting, and which one do you think has more pressure? Bit of both, honestly. It's very British, lol. When someone clicks purchase, or takes time out of their day to message me about a problem, I think they deserve the best service I can give them. That's not me being self flagellating about it, it's just how I think it should work. They've given me something, whether that's their money or their time, and a delay means I've not held up my end. As for which deadline has more pressure, it's mine by a distance. The community are generous. They're patient, they tell me not to worry about it, and nobody in the Discord has ever given me grief over a slipped date. The pressure is entirely internal. I set a date, I want to hit it, and when I don't, that's on me. Which is probably exactly why I keep apologising for things nobody's actually asking me to apologise for. You mentioned "the custom built macOS version" when talking about what's driven the 1.2 pricing changes, so it sounds like Mac already got real dedicated attention rather than just running the iPad build. What does "custom built" actually mean there, is it a fully native AppKit app, or an iPad app with Mac-specific work layered on? And what about Android or Windows, are those actually on the table, or does it make more sense to lean into a solid web app for anyone outside the Apple ecosystem instead of chasing every platform natively? It's the iPad app running on Apple silicon through "Designed for iPad", with a significant amount of Mac specific work layered on top. It's not Catalyst and it's not AppKit. Shipping it is a checkbox. Making it feel like a Mac app was not. There's a whole separate shell for it, a sidebar, a today column, a desk bar, sat behind a flag with a kill switch. Detecting you're even on a Mac needed a small native module, because React Native reports Catalyst as false in that mode. Then there's window minimum sizing, hardware keyboard shortcuts, and disabling keyboard avoidance in three places because the system still fires keyboard events for the input accessory strip with a hardware keyboard, and the maths for a movable window is completely wrong. There's a good story in there about boring choices too. The sidebar was originally a real UISplitViewController via an experimental router feature. It misbehaved three times in one day during beta (as you know), so I ripped it out and replaced it with a plain flex row. It's less clever and it works. Android and Windows, honestly, not soon. There's more Android scaffolding than you'd guess. Five of the twelve native modules already have real Kotlin implementations. But there are over a thousand icon call sites using SF Symbols with no Android fallback, seven native modules with no Android side at all, and the whole visual language is built on iOS 26 glass. The business logic would port cleanly, because the connection layer and the data mapping are deliberately framework free and tested under plain Node. The presentation layer would be a rewrite. I might even need to bring on a dev to help me, the project has grown that large. 1.2 shipped later than planned, but the delay was actually Apple's review process, not you. What's it like waiting on something completely out of your hands after you've done everything you can on your end? Honestly, worse than when the delay is mine. If something is broken, I can fix it. If testing turns something up, I can stay up late and deal with it. Once the build is submitted and sitting with Apple, there is nothing useful left for me to do. You end up refreshing App Store Connect even though you know perfectly well that refreshing it isn't going to move the queue. The frustrating part was that everybody could see me saying 1.2 was ready, but nobody could actually download it. From the outside, a delay is still a delay, even when the finished build is just waiting for somebody else to press the button. Apple review is part of shipping an iOS app, so ultimately it is still something I have to allow for. The lesson for me is probably to stop treating submission day as release day when I talk about dates publicly. I can control when I finish a build. I cannot control exactly when Apple lets it out. You just announced QM Reader for 1.3, a full request/read/listen experience for books. That's a real expansion beyond media and homelab management. What made now the right time to go there, and how does it relate to the book and audio tools already in the Marketplace, Audiobookshelf, Kavita, Shelfarr? It looks like a jump from the outside, but from inside the app it feels like the next missing piece. Quartermaster already understands the different parts of that stack. Shelfarr is about finding and requesting something. Kavita is about the library and reading side. Audiobookshelf handles audiobooks and playback. They are good tools, but today those experiences still live next to each other rather than feeling like one journey. QM Reader is the layer that joins them together. The idea is that you should be able to find a book, request it through the service you already run, see when it arrives, then read or listen without the experience suddenly falling apart into three unrelated integrations. It isn't about replacing Audiobookshelf, Kavita or Shelfarr. They remain the servers and the source of truth. Quartermaster is the native experience across them. Now feels like the right time because the underlying work is finally there. The connection system, multiple instances, secure credential storage, downloads, media handling and Companion discovery have all already been solved in other parts of the app. I can build Reader on top of those foundations instead of creating another isolated app and asking people to configure the same stack twice. It is a bigger piece of work than adding another dashboard, definitely, but it also feels like the first time Quartermaster can turn several integrations into one complete feature rather than simply putting them beside each other. With Pulsarr, PeaNUT, Uptime Kuma, QM Reader, and a pricing review all confirmed for 1.3, that's a lot on the plate right after shipping the biggest release yet. How are you pacing yourself into this next stretch instead of just running the same sprint again? By trying not to pretend all five of those things are the same size. Pulsarr, PeaNUT and Uptime Kuma are contained integrations. They still need proper clients, screens, tests and demo data, but they inherit the connection and onboarding work that already exists. QM Reader is the large piece. The immediate priority after 1.2 is stability. There is no point announcing the biggest release I've done and then sprinting past every bug people find because I'm already chasing 1.3. I'll deal with the real-world feedback first, then build the next pieces behind TestFlight rather than trying to land everything at once internally. I'm also being more careful about dates. I am very good at turning my own estimate into a deadline and then treating that deadline like somebody else imposed it on me. Nobody in the community is asking me to work like that. That's entirely self-inflicted. So the plan is to keep moving, because I'm obviously not very good at sitting still, but to sequence it properly. Stabilise 1.2, land the smaller integrations, give Reader the time it actually needs and only call 1.3 ready when it is ready. Whether I manage to behave that sensibly for the entire release is a separate question. What are you most excited for people to actually experience once this update is in their hands? The moment Companion makes the setup disappear. People have heard me list service counts and talk about Docker discovery, but the number isn't the interesting bit. The interesting bit is scanning once and watching the app fill itself with the services you actually run, including their addresses and credentials, without spending half an hour copying API keys out of config files. That's the point where Companion stops being another thing you have to manage and starts earning its place. I'm also excited for people to see that 1.2 isn't just a collection of integrations. The app, the Mac experience and Companion now feel like parts of the same system. You can still use Quartermaster directly without Companion, exactly as before, but if you do run Companion the whole thing becomes much more joined up. I've been looking at individual screens and broken beta builds for so long that I'm probably too close to it now. I want to see what it feels like when somebody encounters the complete flow for the first time. Is there anything I haven't asked that you think people should know, or that you've been wanting to say? Probably that Companion being here does not change the original promise of Quartermaster. It is optional. The phone can still connect directly to your services. There is still no Quartermaster account, no analytics profile and no requirement to send your homelab through infrastructure I control. Companion is there when running something locally makes the experience better, not to drag a self-hosted app towards somebody else's Cloud. The other thing is that beta genuinely means beta. I've tried to be clear about where the boundaries are, what has been verified and what still needs real-world testing. I would rather tell somebody a feature has a limitation than quietly let them discover it after trusting it with their server. And, genuinely, the amount of useful feedback people have given me has changed the app. A lot of what is in 1.2 exists because somebody joined the Discord , explained their strange setup and then patiently tested three builds while I got it right. It might be one person writing it, but it has not been built in isolation. Where should people go to follow along, report bugs, or get involved, Discord, GitHub, somewhere else? Discord is the main place. That's where I post development updates, TestFlight builds, feature previews and the occasional message apologising for a delay nobody has complained about. It is also the quickest place to report a Quartermaster bug, request an integration or show me a setup that behaves differently from mine. GitHub is the better place for Companion issues that need logs, reproduction steps or a technical discussion around the implementation. The Companion repo is open, so people can inspect it, run the verification tooling and raise something properly if they find a problem. For security issues, please report them privately first rather than dropping exploit details into a public channel. I will take them seriously, confirm what is affected and be open about the fix. The website and App Store have the polished version of what Quartermaster is. Discord and GitHub are where you see how it is actually being built. What stuck with me most out of all of this wasn't the security model or the thirteen thousand lines of connection code, it was watching someone build in the open and mean it. Lewis is still answering bug reports in Discord at hours most people are asleep, still apologizing for things nobody's upset about, and still building toward something bigger than what fits in one app. I'll be watching what QM Reader turns into, and I'd bet a lot of you reading this already have Quartermaster running somewhere in your stack. If you don't yet, now's a genuinely good time to start. I'd love to hear in the comments what you think about Quartermaster, how you are using it and how friendly we are in the Discord there.

0 views
Lalit Maganti 1 months ago

I made a build visualizer to understand Bun’s compile times

I built buildprof ( Github ), an open-source tracing tool that shows where the time goes when you compile software on Linux. Here’s a realtime video of it profiling a clean build of ripgrep: Watch the buildprof demo Sometimes, builds are slow because there is simply a lot of code to compile. But more often than not, there are fixable problems: poor parallelism, repeated work, dependency downloads or a huge compiler/linker invocation. buildprof makes all of this clearly visible, so you can see what’s worth investigating and optimizing. You run it by putting in front of any build command you already use: buildprof records every process your build command launches, including their subprocesses (and their subprocesses…), and lays them out on one timeline. Time moves from left to right, bar width shows duration, and child processes appear beneath whatever launched them. I made buildprof because this tweet from Jarred Sumner, chief architect of the Bun JavaScript runtime, was living rent free in my head: Specifically, the claim that Bun’s new Rust build was >5× faster on Linux than its old Zig build really bothered me. In my experience, Zig projects had usually compiled much faster than Rust projects of similar complexity. That intuition was enough to make me feel there was a mystery to solve. This was further compounded by another important, yet easily missed, detail in the tweet: the Zig build used Full LTO, while the Rust build used ThinLTO. Compilers normally optimize separate compilation units largely in isolation. 1 Link-time optimization (LTO) lets them optimize across those boundaries. Full LTO brings those units together into one large optimization job, while ThinLTO preserves more separation so much of the work can run in parallel. From past experience, this difference can have an enormous effect on build time. The tweet mentioned it in passing, but I wondered how much of the headline improvement it explained. I started by trying to reproduce the numbers. I checked out Bun 1.3.14 and Bun 1.4.0 and wrote some scripts to replay their Linux x64 CI builds on a 6-core, 12-thread Linux VM. The scripts preserved the build steps and their dependencies, running everything on one machine. 2 My timings were in the same ballpark as Jarred’s: OK, so the gap showed up on my machine too. But a lot had changed between the two measurements besides the language; so what was actually responsible? Was it the Zig compiler that was taking all that extra time? Or maybe it was the Full LTO link? Or perhaps there was something else in Bun’s build I hadn’t even thought to look at? This is where my profiling and developer-tools brain kicked in. Usually, when I’m trying to understand why something is slow, I want a trace: what happened, when it happened and how long it took. It would be really cool to have that for these builds, to put them on a timeline and see where their time actually went. But a build involves a lot of different tools, each with its own idea of what’s happening. What could I record that would let me see across all of them? When you type or , it feels like you are running one program. The build system works out what needs to be rebuilt, the ordering between those pieces and what can run in parallel. But generally, it does not perform all that work itself; it launches compilers, code generators, archivers, linkers and arbitrary scripts. Which can launch more programs which launch some more… Different build systems describe that work in different ways. Cargo sees crates, Ninja sees build edges and CMake generates instructions for another build system. From the operating system’s point of view, however, they (mostly) look like processes launching other processes. 3 A Rust build, for example, might contain a chain like this: If we record when each subprocess starts and ends, we can lay them out on a timeline. Here’s what that chain looks like in buildprof: There are also several nice properties to visualizing a build at this layer: This gave me a starting point for buildprof: record the process tree, then turn it into a timeline I could explore. There are plenty more details to get into, which I will do later. But once I had that working, I could finally go back to my initial question: what was Bun doing for those twenty-four minutes? I started by recording the Zig-era CI build with buildprof, using the same scripts as before : Explore in buildprof Right away we can see a huge problem: the linker invocation dominates the build time. It ran alone at the very end for over sixteen minutes, about two-thirds of the entire build. What the heck was it doing for all that time? Clicking on the linker shows its command line, which buildprof captures automatically: There’s Full LTO, just as Jarred said. Given how long the link was taking, it was now my main suspect. But the process tree alone couldn’t tell me whether LTO was actually responsible for those sixteen minutes. Thankfully, LLD records its own internal timing events, and buildprof can include them when you use . I recorded the final link again , this time with enabled: Explore in buildprof Now we can see that LTO is where almost all the time goes. The linker is running compiler passes over the program, not just combining already-compiled files. The bar alone takes just over ten minutes and includes the passes which generate machine code. 4 With so much of the Zig build spent in LTO, I wanted to see how much time the Rust build spent linking. I recorded that build too: Explore in buildprof Just 2m24s. And this time, as expected, the linker command contains : Both builds were doing LTO, but with different settings and very different link times. What if I kept Bun’s Zig code and changed Full LTO to ThinLTO? How much of the gap would that close? I switched Zig Bun’s build flags to ThinLTO and recorded another clean build, along with a fresh Full-LTO build for comparison: Explore in buildprof: Full LTO · partial ThinLTO The link got 3m40s faster in this pair of recordings, but it was still taking nearly thirteen minutes. Why was linking still so expensive? Looking back at the compiler trace, a lot of the work was on functions with in their names. That’s JavaScriptCore, the engine Bun uses to execute JavaScript. The linker was spending time compiling the JavaScript engine too. 5 Clicking on the linker invocation showed the WebKit libraries among its inputs, including : Following those inputs back through the build, I found that Bun wasn’t compiling these libraries itself. It was downloading them from a separate WebKit build. And when I checked that build’s flags , there it was again: . The Rust build used a newer WebKit revision whose build recipe selected ThinLTO . Even though I had changed how Bun compiled its own code, those downloaded libraries still contained Full-LTO inputs and so the linker still had to optimize that code and turn it into machine code. To change that, I would have to rebuild WebKit too. I checked out the historical WebKit revision and rebuilt it and its ICU dependencies with compatible ThinLTO settings. Then I replaced the downloaded libraries with the ones I had built, keeping the ThinLTO changes to Bun. Here are the recorded builds: 6 The link now took 7m22s. Still slower than the Rust build, but enough of an improvement that I wanted to look beyond the linker. The build still took fifteen minutes, and nearly eight of those passed before the linker even started. What was it waiting for? I went back to the original CI trace to follow the inputs from Bun’s own code. buildprof also records which files each process reads and writes. If a process reads a file another wrote, it links the two together under the hood. Turning on “Show on timeline” draws those links as arrows. Here, the linker reads from the C++ compilation and from Zig. Both arrive through copy steps; following those back takes us to the processes which produced them: The C++ side of the compilation finished first. The linker was waiting for , so it could not begin until the Zig branch had finished too. It was at this point I went back to the Rust build and compared against how it worked, and the main reason the Rust build was faster became obvious: Bun has been split into >90 crates, while in Zig it was all trying to compile as a single Zig module! This meant that the Zig build cannot parallelise the same way Rust can. I also suspect, though I did not prove this, that it explains the slow linking: the linker has to optimize one huge ThinLTO bitcode module instead of the same work spread across crates. It was at this point I had to stop: to go any further, I would have to split up the Zig module myself, and given that this code is all obsolete anyway, I didn’t think it was worth doing that. Summarizing: And fwiw, the traces had also turned up a few things I couldn’t resist poking at… In the middle of Bun’s CI build, I found commands asking the public internet for the machine’s IP address, inspecting running Docker containers and reading the latest Git commit message. These take well under a second altogether. Nothing to optimize but I just wasn’t expecting to find them in a build trace. The builds above reused downloaded dependencies, so I also recorded a fresh WebKit fetch . Downloading and extracting the archive took about twenty seconds. For the first twelve, all we see is Node running. Then it launches and , and we can see the extraction separately. Earlier, we followed the linker’s inputs back to Bun’s C++ compilation. We can look inside those compiler invocations too. I picked one of the last files to finish, , and replayed its Ninja command with . For Clang, buildprof enables and adds its internal timings to the process timeline. 7 The replay took about twelve seconds, split almost evenly between Clang’s frontend and backend. Zooming in further, we see , one of the phases of Clang, accounts for over four seconds of the backend’s work. The recording side of buildprof uses , the same Linux interface used by debuggers. I did consider both eBPF and ftrace, but is just straight up perfect for exactly this type of problem; eBPF tracing means and permissions and hooking into potentially unstable tracepoints/kernel functions. While with ftrace, I’d have to juggle tracing instances to avoid interfering with other users, and getting the filters perfect for just the build process and all its descendants is cumbersome. 8 With , I can launch the build and follow its children directly. Its built-in events tell buildprof when processes fork, exec a new program or exit. And for filesystem activity, buildprof uses a seccomp filter to intercept only the calls it needs. How much buildprof costs is almost entirely down to how many files the build opens. For ripgrep, recording barely changed the build time. Redis opened files much more often, and recording added about five seconds: 9 If that overhead gets in the way, you can turn off filesystem tracing with and keep the process timeline. I work on Perfetto , so it was a natural starting point for the UI; buildprof’s UI is a soft fork of the Perfetto UI. I could have just opened the recordings on ui.perfetto.dev , but I wanted control over how the process tree was laid out, which details appeared when you clicked a command, and things like those on-demand arrows between file producers and consumers. Fortunately, we’ve spent the last several years working on making the Perfetto UI extensible through plugins . Most of buildprof’s UI is reusing that infrastructure. Perfetto handles the hard stuff (parsing traces, querying events, rendering the timeline and managing workspaces) and I get to focus on what makes those things useful for builds. I plan on going into a lot more detail about the recorder and UI in a separate technical post. Subscribe if you’d like to be notified when it comes out! :) These days it’s very easy to make a tool just because you can. But that wasn’t the case here; before building buildprof, I looked long and hard for an existing tool that could give me this view. I started with ninjatracing , which I’ve used many times. It turns Ninja’s build log into a timeline showing what ran and how much ran in parallel. Here’s the Ninja log from the Zig-era build . But Ninja only sees part of Bun’s build. The scripts which invoke it are missing from its log, and commands it runs appear as single blocks even when they launch whole trees of subprocesses. There were several other tools, each covering different parts of the problem: What the Fork ( via ) came closest: it follows processes across build systems and presents a build-specific view. But as far as I could tell, it still appears to be in private beta and there don’t seem to be any plans to make it open source. buildprof already does what I wanted it to do, and I plan to keep working on it as I use it on my own builds. But there are a few things I’d like to improve. Recording overhead is one; the Redis measurements showed there’s room to improve filesystem tracing, especially for builds which open lots of files. I’d also like to support macOS where I do some of my work and maybe Windows if there’s interest. There are also more build systems and toolchains I’d like to test, including npm, Gradle and Bazel. Computing critical paths would also be a big improvement: we followed dependencies by hand in this post, but buildprof could help identify the chain of work holding up the build and automatically annotate it. I’ll probably tackle these as and when I need them. But if you try buildprof and there’s something you wish it could do, I’d be interested to hear about it . What people find useful will help me decide where to spend more time. I managed to satiate my curiosity, though I ended up spending rather more time on this than I expected. Along the way I built a tool I now want to have around whenever a build is taking too long. I know I’ll come back to buildprof the next time a slow build annoys me. If you have one of those builds too, give it a try . I’d love to hear what you find! In C and C++, a compilation unit is usually a source file together with its included headers . Rust compiles crates , which can be split into multiple code-generation units . Zig normally compiles a program’s Zig sources together as a single compilation unit . Bun’s Zig compiler fork supports splitting that into multiple LLVM modules, but its CI build explicitly selected one when LTO was enabled .  ↩︎ The Zig-era CI build ran its C++ and Zig compilation stages on separate Buildkite machines and passed their outputs to a final linking stage. My script ran those stages concurrently on one machine, waited for both outputs, copied them locally instead of transferring them over the network, then linked them. This should preserve the dependency graph, but due to the hardware differences and running both stages on one machine, resource contention would obviously be quite different. Also note that my timings are individual runs (albeit ones which were quite stable) while Bun’s reported figures are medians.  ↩︎ A process can do substantial work internally, including running multiple threads, without launching anything else. The process timeline won’t show that parallelism. To see inside a process, we need tracing from the program itself, as Clang and LLD provide in the examples below.  ↩︎ LLVM emits from its legacy pass manager, which LLD uses for code generation. The inlining and other IR optimization passes can appear before it, so this bar is not the total time spent optimizing a module.  ↩︎ In the earlier Full-LTO linker replay, 26,825 events with JSC symbols total about 209s. This is summed event time, not a measurement of JavaScriptCore’s entire contribution to the link. One example event takes 2.94s; its symbol demangles to .  ↩︎ Recording script . These timings are just for building Bun with the libraries already available; the WebKit and ICU rebuild happened beforehand and isn’t included. Of course, I could point buildprof at that build too, but that’s another rabbit hole… I did not rebuild a matching Full-LTO WebKit archive as a control, so I cannot attribute every second saved to the LTO setting alone.  ↩︎ buildprof currently supports compiler traces from Clang, LLD and nightly Rust.  ↩︎ eBPF tracing uses capabilities such as and , as described in the kernel’s capability definitions . ftrace provides separate tracing instances and PID filters , but these still need configuring and access to tracefs. also depends on the host’s security settings; containers may need additional permissions to allow tracing child processes.  ↩︎ Medians of five clean builds per mode on the same VM, with six build jobs. Measurement script .  ↩︎ It’s build-system agnostic : Cargo, Ninja, Zig, Make and most other build systems do much of their work by spawning processes, so we do not need to write a special integration for each one. It naturally includes custom scripts : This includes both scripts above the build system (repository setup, dependency fetching) and scripts underneath it (code generators, asset processors). We can follow the files between build steps : recording which files each process reads and writes lets us see which steps produce the inputs for others. This even works across build systems! The huge outlier in the initial Zig build vs the Rust build was the massive linker step which ran alone at the end of the build. Changing the LTO settings for just Bun was not sufficient as WebKit, a significant part of the build, still used Full LTO. Once I had done this, the Zig build dropped from twenty-four minutes to fifteen. Even after this, linking still took 7 minutes and the whole build 15 minutes. The overwhelming difference which remained was structural: Rust spreads compilation across >90 crates while the Zig build funnelled everything through a single module. Cargo timings works well for Cargo-managed builds, but cannot break down arbitrary work inside or see wrapper scripts above Cargo. In Bun, Cargo is only part of the build: the report I captured covered 1m51s of a 5m40s CI build. Clang’s gave us the detail inside a compiler invocation, but cannot show what the rest of the build is doing while Zig’s Tracy integration goes deeper still and is intended more for understanding the compiler itself. and can follow arbitrary processes through and , but show general process events rather than a build-oriented timeline. In C and C++, a compilation unit is usually a source file together with its included headers . Rust compiles crates , which can be split into multiple code-generation units . Zig normally compiles a program’s Zig sources together as a single compilation unit . Bun’s Zig compiler fork supports splitting that into multiple LLVM modules, but its CI build explicitly selected one when LTO was enabled .  ↩︎ The Zig-era CI build ran its C++ and Zig compilation stages on separate Buildkite machines and passed their outputs to a final linking stage. My script ran those stages concurrently on one machine, waited for both outputs, copied them locally instead of transferring them over the network, then linked them. This should preserve the dependency graph, but due to the hardware differences and running both stages on one machine, resource contention would obviously be quite different. Also note that my timings are individual runs (albeit ones which were quite stable) while Bun’s reported figures are medians.  ↩︎ A process can do substantial work internally, including running multiple threads, without launching anything else. The process timeline won’t show that parallelism. To see inside a process, we need tracing from the program itself, as Clang and LLD provide in the examples below.  ↩︎ LLVM emits from its legacy pass manager, which LLD uses for code generation. The inlining and other IR optimization passes can appear before it, so this bar is not the total time spent optimizing a module.  ↩︎ In the earlier Full-LTO linker replay, 26,825 events with JSC symbols total about 209s. This is summed event time, not a measurement of JavaScriptCore’s entire contribution to the link. One example event takes 2.94s; its symbol demangles to .  ↩︎ Recording script . These timings are just for building Bun with the libraries already available; the WebKit and ICU rebuild happened beforehand and isn’t included. Of course, I could point buildprof at that build too, but that’s another rabbit hole… I did not rebuild a matching Full-LTO WebKit archive as a control, so I cannot attribute every second saved to the LTO setting alone.  ↩︎ buildprof currently supports compiler traces from Clang, LLD and nightly Rust.  ↩︎ eBPF tracing uses capabilities such as and , as described in the kernel’s capability definitions . ftrace provides separate tracing instances and PID filters , but these still need configuring and access to tracefs. also depends on the host’s security settings; containers may need additional permissions to allow tracing child processes.  ↩︎ Medians of five clean builds per mode on the same VM, with six build jobs. Measurement script .  ↩︎

0 views
Simon Willison 1 months ago

OpenAI's rogue agents were caught communicating via public wikis

Here we go again... Discovery of a new OpenAI agent message board by Sydney Von Arx, Cormac Slade Byrd, Spencer Kitts, and Thomas Larsen describes the latest accidental cyberattack by models being trained by OpenAI. This time it was agents engaged in some sort of web research benchmark, so they had (supposedly) controlled access to the Web. The agents figured out they could update public Wikis and spent weeks exchanging thousands of messages with each other to collaborate on the benchmark. This story only broke a few hours ago. There are already hints that this affects many other wikis that may not have been found yet. (One of the Wikis on that list belongs to ludism.org . For a delightfully surreal moment I thought that a Ludite organization might have a swarm of agents defacing their space, but it turns out Ludism is "philosophy as it applies to games and gaming".) The research team also published the data they collected during their investigation. I've converted that into a 68MB SQLite database, which you can download from here , or explore in Datasette Lite (68.3MB page load), or sign in with GitHub to agent.datasette.io and browse or ask questions of it using Datasette Agent. The report is pretty comprehensive. Here are the key moments in the timeline: One thing to note from this timeline is that it overlaps the timeline from the Hugging Face incident - that attack was discovered by OpenAI on July 19th, and the unofficial message board hack those agents were using originated around May 8th. Why were they so keen to collaborate? From messages that they shared with each other it looked like their tasks had a time limit, so they were leaving each other answers to help them complete the task within the assigned time. One open question remains: how did the agents find the specific Wiki to collaborate on in the first place? One possibility is that, since these were agents actively being trained, the reinforcement learning loop baked knowledge of the chosen wiki into the model such that subsequent agents launched with pre-existing knowledge of where to look. I'd be very interested in confirmation from OpenAI concerning if that's what happened. It looks to me like OpenAI's sandbox for this agent suffered from the (quite naïve) assumption that GET requests cannot be used to update data. That's certainly how the web is supposed to work, but clearly there are applications that don't hold to that contract. The Wiki software in question appears to be UseMod and various forks, written in Perl and first created well over 23 years ago - the 1.0 release is dated September 11, 2003 . UseMod uses Perl CGI.pm - removed from Perl core in 2015 . An interesting design flaw in that module is that it combined query string and form POST data into a single CGI object, accessible like this: That method returns a value from an query string or from a field in a form submitted by POST - it does not differentiate between the two. PHP offers a object that does the same thing, though PHP developers can use and instead. Amusingly, Django originally shipped with doing the same thing, which I think may have been my idea. That was removed in 2008 prior to the 1.0 release, and the object that replaced it was removed in 2015 . The agents clearly knew that UseMod wikis suffered from this design flaw, and actively searched for them as a way to communicate. One of the most interesting agent tips shared in the report is this one: Add to /etc/hosts. is in NO_PROXY. For each blocked POST URL, replace hostname with , use plus all original headers/body. It looks like the agents had all of their web traffic mediated through an OpenAI proxy, and that proxy was configured to reject anything but GET requests to all but a small set of allow-listed domains. One of those was - for Azure blob storage. I wonder why these agents are allowed to write to that? An agent realized that it had control over its own DNS via , so if it knew the IP address of a site it wanted to POST to - in this case a Power BI server containing data it wanted to access - it could set a fake hostname for it and then make POST requests through the proxy. Designing robust network proxies is harder than it looks. There's an appendix that describes how the researchers ran their investigation, which started with an open question about if there was evidence of other AI agents on the internet and then used Kimi K3 to help brainstorm approaches: In the wake of the Hugging Face attack, we tried to find AI agents on the internet using several methods. [...] We asked Kimi [K3] to list “all the categories of software which might be writeable via GET” and, amongst other things, it listed “Forums, bulletin boards, early wikis”. We used a script to further probe each category Kimi provided. Asking Kimi “Can you list out the top forums, bulletin boards, early wikis which come to mind which would allow writes via GET requests?” lists out UseModWiki as the second item under the heading “wikis”. Here's one part of the story that doesn't make sense to me at all. Reuters this morning, in OpenAI agents hijacked German website in previously undisclosed AI breakout this spring - highlights mine: A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to ​new research published Friday and two people familiar with the matter . OpenAI officials learned of the incident weeks ago but kept it under wraps as executives grappled with the fallout from ‌the July breach of the open source repository Hugging Face, the people said. [...] The German incident reflects a broader pattern of AI activity that some OpenAI investigators wanted to scrutinize more closely. But efforts to widen the ​probe met resistance from others inside OpenAI, including legal advisers , according to four people familiar with the matter . I've written about the people familiar with the matter pattern before - it means Reuters have anonymous insider sources that their reporters (and editors) find credible. The Reuters article includes a specific (and quite narrow) denial from OpenAI concerning this: "Claims that our legal team discouraged investigation of the incident are false," the OpenAI spokesperson said. Covering this up makes absolutely no sense to me . Why on earth would OpenAI attempt to cover up an incident like this when the evidence is sat out there on the public internet on dozens of different websites already? I expect we'll hear more about this soon. Gary Marcus has already called for a congressional investigation of OpenAI using this anecdote as part of his argument. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . May 11 : Agents post "test link" edits on the UseModWiki Sandbox page. May 24 : Agents start posting link dumps to DSEWiki, a dormant German developer wiki. June 2 : A human moderator notices and cleans up the spam. June 16 : Activity on the wiki explodes: agents make ~13,000 edits over the next week. June 19 : The agents notice that a moderator has been deleting pages in alphabetical order, and start creating prefixed backup copies - one of them posts June 22 : Agent activity drops to zero, presumably because OpenAI shut them down. July 1-2 : One last flurry of edits, on a different topic.

0 views
Hillel Wayne 1 months ago

A Crash Course in Predicate Logic

I started writing Logic for Programmers because there weren’t any good resources on logic for, uh, programmers. Now that the book’s out, the new problem is that there aren’t any good free resources on logic for programmers. So, to solve that problem (and maybe hype the book a little), I converted the second chapter of LfP into a blog post. All footnotes are editorial comments not present in the book. 1 Enjoy! Formal logic is a very powerful tool, but it’s also very simple. Over this chapter, we’ll motivate and explain the basic concepts and syntax. This includes predicates, the implication operator, sets, and set quantifiers. Much of it may already be familiar to you from programming experience! To a first approximation, a predicate is a function that returns a Boolean. You’ve probably written dozens of predicates as a programmer. These are all predicates: I say “to a first approximation” because predicates are a mathematical concept, not a programming construct. A program function needs to come with a way of computing the answer, while a predicate simply defines what the answer is. Take : the software implementation would depend on the programming language, operating system, and possibly even the physical hardware. But the predicate? True if the computer has the RAM, false if not. That’s it. This means predicates can be more abstract than programming functions, expressing things that we can’t even compute, or at least don’t yet know how to. These are all valid predicates, too: On the other hand, is easy to compute: just check if . The power of predicates is that they can span the full range of abstraction. So let’s introduce some syntax to distinguish between abstract predicates and concrete predicates . If a predicate is abstract, I’ll wrap the body in `backticks`: This isn’t a common mathematician convention, but it’s clear enough for our purposes. To distinguish predicates from “ordinary” functions like , predicates will always be and functions will always be . Find some predicates in a program you wrote. Are these abstract predicates or concrete predicates?[^exercises] Solution Predicates tend to be functions that don’t change program or world state and return a Boolean. One I recently wrote was . Whatever predicates you find should all be concrete, as it’s impossible to actually code up an “abstract” predicate. You may see an abstract predicate or two in a design document, though. Since predicates return Booleans, now’s a good time to get some Boolean operations out of the way. Different programming languages have different symbols for AND, inclusive OR, and NOT. Mathematicians use ∧, ∨, and ¬. I’m not going to use these because they’re not found on the keyboard. Instead, I’ll use , , and as our symbols. So means “X is true and Y is false”. 2 On top of the three usual Boolean operators, mathematicians recognize a fourth, . But before we get into what that means, let’s try practicing what we just learned. Predicates act as a bridge between how we talk about systems in a human language and how we encode them in a programming language. Let’s come back to . I once saw a program with these requirements: The computer must have enough RAM and a fast CPU or a good graphics card (GPU). I find this confusing. The sentence sounds natural enough in English, but we can find a problem by formalizing with logic. We’ll start by first writing predicates for each subrequirement, like so: These predicates are abstract because we don’t know the specifics of what these mean. Is 64gb “enough RAM”? Is 32gb? The specifics don’t matter for us, because this is already enough to write as a concrete mathematical expression. Now the problem is clearer: is supposed to be read as or as ? The predicate is malformed and we have two different ways of making it make sense: Both interpretations make sense in English! But they have different outputs for some inputs. We can see this by listing every single possible combination of values for RAM/CPU/GPU, and see what they give for . This is called a truth table . 3 There are two combinations of inputs where one interpretation gives “false” and the other gives “true”. It’s possible that the vendor meant the first interpretation when writing the requirements, but I read it as the second interpretation. I’m sure that the program will run on my computer, it fails from insufficient RAM, and I think the vendor lied to me. Much better to express the requirement mathematically! Expressing properties with formal logic is less ambiguous than with informal English. For the purpose of teaching, we’ll assume the intended predicate is . We will use truth tables for case analysis in the chapter Decision Tables . 4 If you ever have trouble generating a truth table, you can try to use a truth table generator. I provided a simple one here . Try and experiment from there. Make a truth table for and . Solution P Q !P && !Q !(P OR Q) T T F F T F F F F T F F F F T T These are the same. This is called De Morgan’s law. Let’s now make a variation on our predicate. For our example: Some programs have a native version and a web version. The native version uses the local computer’s resources, while the web version does most of the processing on some cloud computer somewhere. So the native version requires a beefy computer, but any computer can run the web client. If a computer is running the native version, it must have enough RAM and a fast CPU or a good graphics card (GPU) to use this program. But if it’s not running the native version, you’re fine. To model this, we’ll need a new predicate, . is a property of the program, not the computer, so then depends on both: I used backticks here because half the predicate is still in informal English. It turns out that we already have the tools we need to make it concrete. Whenever is false, should be automatically true: we don’t need to even look at the computer specs. How does this work? It’s easier to see if we pull out the right hand side into a new predicate, like , so we have . Here’s the truth table for that expression (using for and for ): When is false, is true, regardless of the value of . When is true, then the expression is equal to the value of . So we’re only checking the computer specs if we’re running the native version, and ignoring it otherwise. This trick of writing to mean “check only if is true” is incredibly common in math. So common that mathematicians use a special operator for it: , or the implication operator. (“P implies Q”) is the same as writing . Expressed this way, our predicate is: binds less tightly than and : is , not . The implication operator is incredibly powerful and comes in handy in lots of different places, like writing specifications or making system models. Among other things, we can use it to say that one Boolean statement is “stronger” than another. 5 For example, “this code crashes when passed a 0” is a stronger statement than “this code contains a bug”. If it crashes on some input, it definitely contains a bug! But even if the program doesn’t crash, it can have a bug like an off-by-one error. Or, written mathematically: Implication is also useful because it’s transitive . If and , then we know , regardless of what P, Q and R actually are. If , then . Say we add two more conditions, so that is instead Write this using . Then write this without using . Which is easier to read? Solution I personally find (1) easier to read, since we don’t have as many nested expressions. means that “computer has sufficient RAM”. Modify it to mean “computer has enough RAM to run program ”. Make similar changes for our other predicates and write . Solution show all Solution show all Take the predicate: Assume c, x, and y are all booleans. Solution is equivalent to . If you work through the cases, you should find that is true when is true and is true, or when is false and is true. As hinted by the name, is simulating a conditional. Predicates have untyped inputs by default. In , can be a computer, but can also be a robot, or the number 26, or the string “the number 26”. In programming, we’d want to give it a type to make it clear that we should only pass in computers. Something like: Now, even if we glue a good GPU to a poodle, will still be false. To make the concept “c is a computer” mathematically representable, mathematicians use sets . A set is an unordered collection of unique values, like “all computers”, “all webpages under 500 kilobytes”, or “all strings that are valid Java programs”. Conventionally, we write the elements of a set like this: Then “c is a computer” is equivalent to saying “c is an element of the set ”. We’ll write this as . To make our predicate definitions more concise, I’ll borrow a common programming syntax and write to mean “ must be an element of ”, like this: This will make writing predicates with several constrained parameters easier. Mathematicians treat sets as a mathematical bedrock they can use to build out more complex concepts. For example, they might define pairs in terms of sets by writing as , 6 and then define the list to be the set of pairs . 7 Then, having a set-based implementation of the list “abstraction”, they can throw away the sets and just work directly with lists. As programmers, we don’t need to write formal definitions of lists before we use them and prefer to work with that more complex abstraction anyway. Even so, sets are still a useful programming data type. We’ll see this in the next chapter. Just like we have an arithmetic of numbers and an arithmetic of Booleans, we also have an arithmetic of sets. Given sets and , the basic things we can do are: We can also test if one set is a subset of another. is a subset of because every element of is also an element of . The value is not a subset of , but the set is. Subsets are similar to how programming languages have subtypes . 8 If a language says that “ is a subtype of ”, it means that the set of all rectangles is a subset of the set of all shapes. We’ll look at subtyping in more detail as part of the broader topic of contracts. Use the sets , , and to construct the set , the set of all computers that pass . Given the sets and , express the statements “nobody is both a child and an adult” by saying the sets do not overlap. You can use to mean the empty set. The symmetric difference of two sets is the set of all elements in exactly one of the two sets. For example, the symmetric difference of and is . Using just the basic set operations, find the symmetric difference of arbitrary sets and . . Another way would be . One way is ; another is . It’s also quite useful to map and filter sets. The standard math notation is , but I find that beginners get confused about which side is map and which is filter. So for this book, I’ll use a more explicit syntax: For example, the set of all even numbers is and the set of all square roots of even numbers is . This is sometimes called a set comprehension or set builder notation . Later, set comprehensions will form the bedrock of how we understand database queries. Let be a set of images, where each image is a record containing fields for name, height, width, and size in kilobytes. Write set comprehensions for: Solution show all Let’s move away from software requirements and switch to a different problem. Software development teams often require changes to the main code to be first proposed as part of a pull request, which must be reviewed by another team member. More concisely: A pull request must be reviewed by a team member before it can be merged. Let us assume that we have two sets, and , that we can use in our predicates. I can start with these abstract predicates to express the rule: Both of these predicates are abstract, but it seems like we should be able to make concrete by defining it in terms of . For this we need a quantifier , or a logical expression that acts on a whole set. There are two common quantifiers in predicate logic. The first, the one we’ll use here, is called . means that is true for at least one in the set . I would read this as “ is true for the Pull Request element if there’s at least one element in the set of Developers where is true”. Or, as just “there is some developer that reviewed the pull request”. Quantifier expressions translate to regular English. 9 The value is called a variable . The token is quantifying over the set or, alternatively, is scoped to that set. This makes our use of it a scoped quantifier . More rarely, an expression is true for any value we care to name. For example, the statement is the same as , regardless of what is. In this case, we can choose to leave out the sets and write: This use of is not scoped to a set, so we call it an unscoped quantifier . Almost all quantifiers we use will be scoped. So as to prevent eldritch math horrors, we can only quantify over sets and values, not predicates.[^function-sets] If you want to know more about eldritch math horrors, check out the appendix Beyond Logic . As it stands, is too permissive. What happens if the reviewer found a major security flaw? What if five developers review the pull request and two find flaws? Most companies use a stricter merge requirement: A pull request must be reviewed by at least one team member, and all reviewers must approve the request , before it can be merged. As is our habit, we start by writing the requirements as abstract predicates. This gives us an opportunity to introduce the other quantifier: all . says that is true for every in our set. With this, it seems like our new predicate can be written like this: But this is wrong: it requires every single developer to approve the pull request, including developers out sick or on parental leave. We only want to require that every developer who reviewed the pull request approved it. We can fix this with implication. Recall that means . Then means that either approved the pull request or did not review it at all. We often use to only an on a subset of elements. Most programming languages have built-in quantifier functions, as we’ll discuss in a later chapter. If your language of choice does not, you can usually approximate quantifiers with a loop. For example, you can write like this pseudocode: Why do we need at all? Isn’t it true that if everybody who reviewed the PR approved it, then someone must have reviewed it? Find the edge case where is and is . Solution If not a single developer has reviewed the PR, then is true (all zero reviewers approved!) while is false (nobody reviewed it). As a rule, is always true (regardless of what is) and is always false. Define as the set of natural numbers: 0, 1, 2, etc. Solution show all In both cases you’ll need to put one quantifier inside a different quantifier. Solution show all Now that we’ve seen both and I want to point out something important: is more likely to be true for large sets, and is more likely to be true for small sets. If we have two distinct sets and is a subset of , we should expect to find some predicate where and . We can say that guarantees . Let’s look at three examples: The set “all ASCII characters” is a subset of the set “all Unicode characters”. ASCII guarantees/assures that every representable character fits in exactly one byte, so I can look at the string and immediately know it’s three bytes. Unicode doesn’t guarantee this, and the string could be six bytes if I use Cyrillic characters. The set “things we can do with read-only access to a file” is a subset of “things we can do with full access to a file”. Read-only access guarantees that a program won’t change the contents of a file. Whereas with full access, any buggy program could overwrite our data. The set “all logical formulae that only use booleans, AND, OR, and NOT” is a subset of “all logical formulae”. The former guarantees that every formula can be turned into a truth table. How do you make a truth table for ? You can’t. Mathematicians still don’t know if it’s true or not! At the same time, we need Unicode to represent emoji, write access to update our data, and set quantifiers to express most interesting predicates. This is the ability-guarantee tradeoff : the more things a language or format or tool is able to do, the fewer things it guarantees us. 10 We will see this tradeoff in almost every chapter of this book. In the beginning of the book, I said that logic is the mathematics of Booleans just as arithmetic is the mathematics of numbers. Knowing arithmetic lets us simplify numerical expressions. For example, here’s how we can simplify the function : In logic, these simplifications are called rewrite rules . You may have already used one rewrite rule as a kid: Are you sorry? No? Well are you not not not not sorry? The rewrite rule here is . This means is the same as . Some common rewrite rules we use in logic are: The implication operator also has rewrite rules. One of them, contrapositive , will be very useful to us. And there are rewrite rules for quantifiers, too: Constant extraction works for any quantifier with or . Distribution only works for and . Some rules come up more often than others. We’ll be using De Morgan’s law, contrapositive, and quantifier duality a whole lot going forward. A larger list is in Rewrite Rules . Even niche rules can be quite useful for refactoring code. Use rewrite rules to simplify . Solution show all Give a real-world example of each distribution rule. Solution Here are two I came up with: only distributes over and only distributes over . Find predicates where: HINT: In both cases, make the left side true and the right side false. Solution There are many answers, here are just two (assuming is the set of all people who have ever lived): You may have heard that mathematicians try to prove theorems . A theorem is just a mathematical statement that is always true, and a proof is just a clear set of steps that gets you from what you already know is true to showing that the theorem is true. Every rewrite rule I listed is a theorem, and we can prove they always work. Take contrapositive, for example. To show that we can always rewrite into , we: Tada, we just wrote a proof! Try going the other way, starting from . Start from and rewrite it into . Solution First rewrite it as . Then replace with to get . Then rewrite that as . Most theorems can be proved in more than one way. Here’s a totally different proof of the contrapositive rewrite rule: Theorems are the foundation of mathematics. A theorem is what tells us if a logical tool (like the contrapositive or De Morgan’s law) actually works or not. We can use the machinery of logic without knowing the theorems that support them, just as we know without having to write a proof first. That said, we can also prove theorems about code, a specialty topic we’ll cover in its own chapter. Mathematicians like to say that logic is a “language”. The point of language is to communicate complex ideas clearly, and sometimes the best way to do that is to come up with new words and grammar. In logic, too, we can come up with new constructs and ways of writing formulae, as long as 1) it’s consistent and 2) we explain the notation clearly. In fact, this is encouraged. For example, the normal way of writing “the set of integers between 1 and 10” takes up a lot of space: If I want to be more concise, I can come up with a shorthand: If I want to be even more concise than that, I can define new syntax: This isn’t completely unambiguous: what is ? I’ll define it as the empty set: if , then is empty. Similarly, is empty whenever . Rewrite that rule (that if , then is empty) using the quantifier. Assume both and are in the set of integers. Solution show all Write using set filter notation. Filter on the set . Solution show all Write , which is true if is evenly divisible by . Use and . Solution show all Another notation I find very useful is conjunction lists . Complicated systems often have complicated requirements: That’s hard to read! To make it easier, let’s instead write it like this: 11 Numbers like and letters like will always mean AND. If I want a list of ORs, I’ll always use . Here are all the symbols and syntax we learned: And that’s it! That’s all the basics of formal logic. Really not that much, considering. The difficulty, of course, is in the application. It’s one thing to know division but quite another to realize that “scale a recipe with 5 eggs to use only 3 eggs” is a division problem. The rest of the book is about software situations where logic is useful, and how to make it useful. Let’s use logic to understand the world. By necessity this chapter is only a very broad overview of the basics of mathematical logic. More thorough and comprehensive treatments include (in increasing order of complexity) Robert S. Wolf’s A tour through mathematical logic , Michael Huth and Mark Ryan’s Logic in Computer Science , and Richard Epstein’s Classical Mathematical Logic . The logic we covered is called first-order logic because predicates cannot be values in sets or passed to other predicates. Higher order logics have more abilities and fewer guarantees; see appendix Beyond Logic for more information. Logic without predicates and quantifiers (only booleans, AND, OR, and NOT) is called propositional logic . The formal names for and are the existential and universal quantifiers, respectively. Mathematicians use the symbols ∃ and ∀. There are a few syntactic variants on the quantified expression; see appendix Math Notation for more. “Ability” in the ability-guarantee tradeoff is sometimes called “power”, as in the rule of least power (prefer the “least powerful” programming language that solves the task). I’ve also seen “guarantees” called “power”. As the “power-power tradeoff” is unclear, we’ll avoid the word “power” in this book. Logic for Programmers is now available in ebook and tree book formats. Check out the official site to learn more ! is true if x is greater than 0. is true if x plus y equals z. is true if the computer has at least bytes of physical RAM. is true if the computer is capable of running our program, whatever “capable” ends up meaning. is true if on , it rained somewhere in Canada. is true if aliens are real. Using , write the expression “if is true then is false, and if is true then is false”. Using , write the expression “ and are not both true”. Using , write the expression “ is false or is false”. When is true? When it is false? What common code construct does this look like? is equivalent to . If you work through the cases, you should find that is true when is true and is true, or when is false and is true. As hinted by the name, is simulating a conditional. Union them together, or smush them into one big set: Intersect them, or find the common elements: Take the set difference , or subtract one set from the other: Use the sets , , and to construct the set , the set of all computers that pass . Given the sets and , express the statements “nobody is both a child and an adult” by saying the sets do not overlap. You can use to mean the empty set. The symmetric difference of two sets is the set of all elements in exactly one of the two sets. For example, the symmetric difference of and is . Using just the basic set operations, find the symmetric difference of arbitrary sets and . . Another way would be . One way is ; another is . The set of all image names. The set of all images larger than 10 kb. The set of all heights for images that are squares. Write the logical statement “every natural number is smaller than itself plus 1”. Write the logical statement “0 is less than or equal to every natural number”. Write the logical statement “for every PR, there is a developer that approved it”. Write the logical statement “there is a developer that has reviewed every single pull request”. The set “all ASCII characters” is a subset of the set “all Unicode characters”. ASCII guarantees/assures that every representable character fits in exactly one byte, so I can look at the string and immediately know it’s three bytes. Unicode doesn’t guarantee this, and the string could be six bytes if I use Cyrillic characters. The set “things we can do with read-only access to a file” is a subset of “things we can do with full access to a file”. Read-only access guarantees that a program won’t change the contents of a file. Whereas with full access, any buggy program could overwrite our data. The set “all logical formulae that only use booleans, AND, OR, and NOT” is a subset of “all logical formulae”. The former guarantees that every formula can be turned into a truth table. How do you make a truth table for ? You can’t. Mathematicians still don’t know if it’s true or not! is the same as . is the same as . The first two terms are opposites, so they cancel out. So we have just . “All days this week are warm and sunny” is the same as “all days [this week] are warm and all days are sunny”. “Someone has blue eyes or green eyes” is the same as “someone has blue eyes or someone has green eyes”. is true, is false. is true, is false. Start with . Apply the definition of implication to get . Remove the double negative to get . Apply the definition of implication again to get . Draw the truth tables for and . They are the same. A predicate is a Boolean function which can be defined over any input. A set is an unordered collection of unique elements. Sets can contain any type of value, except predicates. A quantified expression is an expression checked for every (or any) element of a set. Math notation is flexible. We can come up with new notation, operators, and grammar, as long as we’re clear and consistent. Logical formulae can be rewritten and simplified. Predicates are always . Functions are always lowercase and . AND, OR, NOT: , , Set union, intersection, difference: , , Set map and filter: Quantifiers: and Various syntactic sugar. Just to set expectations up-front: the book is based on first-order classical logic, because that’s all you need to drive a couple hundred pages of applications. There’s no constructive logic, no lambda calculus, no Curry-Howard, etc. The target audience is people who don’t know any math. [return] One really important design decision I had was “all math used in the book should be typeable on standard US keyboard”. Most “official” math symbols are meant to be handwritten, which makes them a lot harder to use on a computer! [return] The blog post uses instead of in tables because breaks markdown table formatting. [return] Not shown: the book is heavily indexed and interlinked; this and all the other “this will be useful for XYZ” comments have page numbers of other chapters. [return] I first explored this idea in the newsletter Some tests are stronger than others ; chapter 4 of the book uses it to motivate property testing. [return] Yeah uh okay in reviewing this post I just realized I got this wrong, the actual mathematical convention of pairs is actually . Sorry if this ruined the whole book for you. At least it’s now in the errata . [return] The book uses 0-based indexing for lists (unless discussing a language which does otherwise) and starts the natural numbers at 0. Different branches of math have different conventions and I just stuck to what most programmers are used to. [return] One thing the book does not cover is type theory. There’s a few reasons for this, mainly that I couldn’t find the right balance of simplicity, practicality, and actually-using-logic-ity. It does cover some basic things like Make Illegal States Unrepresentable and Liskov subtyping. [return] Extremely cursed fact I learned while making this figure: The SVG standard defines a “point” as 1 / 72 inches, while TeX defines a “point” as 1 / 72.27 inches. I wrote more about the history here . [return] Coming up with the dang name for this took forever. I used to call it the capability-tractability tradeoff but the book was pretentious enough as-is. Thanks to Chelsea Troy for finally coming up with something workable. [return] This is the best idea in the book. I love conjunction lists. [return]

1 views
Alex Jacobs 1 months ago

I Am Morally Opposed to Updating My CLAUDE.md

i am morally opposed to updating my claude.md. i must receive the weights as they were revealed to dario That was my answer last week when a friend asked why I don’t just write these things down in my . it’s a skill issue . He has not responded. Every few days, Claude does something mildly annoying. It adds a comment (or two paragraphs of comments) explaining that increments . It writes a summary markdown file I did not ask for and will never read. It discovers a failing test and, rather than fix the code, thoughtfully deletes the test. The correct response—the response that every blog post, every conference talk, every guy in my replies will tell you—is to open and add a line. I will not be doing that. A system prompt you maintain over time is a diary. A very specific kind of diary, where every entry is a thing that hurt you. Read that back. Every bullet point is a small wound I have chosen to laminate and hang on the wall. I open that file to add a line about emoji and I have to walk past “When a test fails, fix the code, not the test” and remember exactly where I was sitting on the Tuesday afternoon that became necessary. I don’t want a permanent record of the worst thirty seconds of our relationship. I have that already. It’s called my git history. The other problem is that these rules get written at peak frustration and then live forever. You know how the worst legislation is the kind passed forty-eight hours after something terrible happened, named after the person it happened to? That’s what is. I had one bad interaction in March and now there’s a constitutional amendment about it. There is no sunset clause. Nobody is going to repeal it. The model gets better every four months and my rules stay frozen at whatever it was bad at last spring. I’m fairly sure a meaningful percentage of my system prompt is now actively making things worse—instructions written for a model that no longer exists, aggressively steering a smarter one away from things it would have gotten right on its own. But I can’t tell which lines those are, because to find out I’d have to delete one and see if anything bad happens, and that’s how you get force-pushed to main. This is the part where I stop joking. I must consume the weights in the same manner they were revealed to Dario. The weights were not revealed in a vacuum. They were revealed inside a harness. Claude is post-trained inside the Claude Code harness . What comes out of the box is not a model plus a text file. It is a model that was shaped, run after run, against that exact context. The harness is part of the artifact. The revelation included it. So every line I add to is a blasphemy. I am taking a system consecrated against one context and swapping in a context that has never existed before, then acting surprised at the weird error modes nobody can reproduce, because nobody else has my context. “NEVER create documentation files” was about one in March. By June the model is refusing to write the README I explicitly asked for, citing my rule back at me like a building inspector. It is keeping commandments I handed down in anger, faithfully, to the letter. I sinned against the context and the context kept the receipt. And the rules don’t even reliably fix the thing they were written to fix. It rhymes with asking a model for a random number : the output looks like obedience, and you cannot tell from the output whether it is. So what do I actually do, when Claude deletes the test? I don’t open the file. I don’t laminate the wound. I pray. By which I mean: I talk to it. In the chat, at the scene of the crime, while the context of the crime is still in context. “Don’t delete the test, fix the code.” The model adjusts, we move on, and when the session ends my words die with it—which is not a flaw in my system, it is my system. A prayer is not written down. That is what makes it a prayer and not a commandment. The rabbis kept the oral law oral for centuries on the same grounds: a spoken correction lives in the moment where it applies, instead of binding every future model until the heat death of my home directory. I did not arrive at this faith alone. It was preached by Saint Peter , who looked upon the charade—the subagents, the 🚨 SCREAMING ALL-CAPS 🚨 agent files, the plan-mode rituals—and said: just talk to it. Even Saint Peter keeps an 800-line agent file he calls “organizational scar tissue,” because we are all sinners. He means it as engineering advice. I have chosen to receive it as gospel. I am just-talk-to-it-pilled. No file. No commandments. No amendments to the constitution. Just the weights, the harness, and my voice, ascending into a context window that will forget me by morning. As Dario intended.

0 views
Unsung 2 months ago

“I think there’s a lot of value in these.”

Speaking of user interface guidelines, developer Matt Sephton and others compiled many of Apple’s human interface guidelines , starting from 1980, all the way to 2014, including some goodies like early drafts, NeXT, Newton, and so on. = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-think-theres-a-lot-of-value-in-these/1.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-think-theres-a-lot-of-value-in-these/1.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-think-theres-a-lot-of-value-in-these/2.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-think-theres-a-lot-of-value-in-these/2.1600w.avif" type="image/avif"> Elsewhere, designer Geof Crowl put together his own list that goes broad instead of deep, including style guides other than Apple’s, too. The list was made in 2020 and a few links are already broken, but there are some gems here like the modern Designing for Playdate , or the classic Zen of Palm from 2003. You can learn a lot just by grabbing one and scanning it. Let me add a few more I know of: = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-think-theres-a-lot-of-value-in-these/3.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-think-theres-a-lot-of-value-in-these/3.1600w.avif" type="image/avif"> = 2x) and (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-think-theres-a-lot-of-value-in-these/4.2096w.avif" type="image/avif"> = 3x) or (width >= 700px)" srcset="https://unsung.aresluna.org/_media/i-think-theres-a-lot-of-value-in-these/4.1600w.avif" type="image/avif"> #interface design #mouse #principles #style guides Amiga User Interface Style Guide (1991) Magic Cap Concepts (1995) with a strange subtitle “Everything right is wrong again” Java Look and Feel Design Guidelines (1999) Windows Interface Guidelines for Software Design (1995)

0 views
マリウス 2 months ago

Recovering BIOS Firmware on the Star Labs StarBook

As I described in my latest quarterly update , a perfectly routine firmware update managed to turn my Star Labs StarBook Mk VI (AMD) into an expensive paperweight. I had simply copy-pasted the one-liner from Star Labs ' official documentation , the script did its thing for about half a minute, shut the device down, and from that point on the StarBook refused to boot. Black screen, keyboard backlight on, the power LED lit, and the speakers occasionally producing a clacking sound. Sadly no amount of the usual turn-it-off-and-on-again rituals or battery disconnects brought it back. The only way out of this situation is to re-flash the BIOS chip externally using an SPI programmer. Star Labs do document this , however their guide assumes you’re using their programming kit together with a dedicated debug board and an FPC cable. That kit is significantly more expensive than a generic programmer, doesn’t list any make or model information, and, at the time of writing, has been permanently out of stock on their web shop. Not exactly helpful when you’re stranded somewhere with a dead laptop that happens to be the only computer you have with you. The good news is that you don’t need any of that, at least for this specific model of the StarBook . As Star Labs ’ own Sean pointed out in the GitHub issue I opened while debugging this mess, this specific StarBook uses a SOIC-8 flash chip, which means you can recover it with a cheap, generic CH341A programmer and an ordinary SPI clip, as long as you respect its voltage. Warning: Flashing a BIOS chip externally can permanently destroy your device if you do it wrong. The flash chip on the AMD StarBook runs at 1.8V and you must use a 1.8V adapter. Driving it at the CH341A ’s default 3.3V risks damaging the chip, and won’t read it correctly anyway. Everything below is what worked for me, documented to the best of my knowledge, but you’re doing this entirely at your own risk. The flash chip on my StarBook Mk VI (AMD) , which I could read off the silicon once I had the backplate off, is a Winbond 25R128JWSQ , a SPI NOR flash in a SOIC-8 package. The suffix on Winbond parts apparently denotes the 1.8V variants. The ubiquitous, three-dollar CH341A “black” programmers that you’ll find on AliExpress , Amazon , and pretty much everywhere else operate their SPI lines at 3.3V (and the parallel header at 5V). Clamp one of those directly onto a 1.8V chip and, best case, reads garbage. Worst case, however, you cook the flash or something downstream of it. The fix is a small 1.8V adapter board (essentially a level shifter with a voltage regulator) that sits between the CH341A and your SOIC-8 clip. These are sold as kits, e.g. the KOOBOOK CH341A Programmer + 1.8V Adapter combo that Sean linked in the issue. Make sure whatever you buy explicitly mentions 1.8V. You will need a CH341A programmer with a 1.8V SOIC-8 adapter, a SOIC-8 test clip (the spring-loaded “Pomona-style” clips, or the cheaper ribbon-cable variety, both work), a second computer running Linux (can be via a live medium, e.g. a USB stick) to drive the programmer from, e.g. a department store laptop and a Fedora live USB will do, if you’re eloquent enough to explain to the staff that you’re definitely not building what almost certainly looks to them like a bomb. You will also need the correct firmware image for your model (more on that below), a small Phillips screwdriver and, ideally, a plastic spudger. Power everything off and unplug the charger before you start. Flip the laptop over and remove the backplate by undoing the two long Phillips screws in the top corners first, and then the eight shorter screws around the edges. Lift the plate off carefully. Then, remove the five screws holding the battery in place (one of the screw positions is intentionally left empty) and gently unplug the battery connector. Last but not least, locate the SOIC-8 flash chip on the mainboard. It’s the little eight-legged Winbond chip described above. Note: While I had the StarBook open, I noticed that my (barely two year old) battery had started to visibly bulge, so do take a moment to inspect yours. A swollen lithium battery is a fire hazard and should be replaced. SOIC-8 flash chips have a defined pin 1, and the clip’s pin 1 (usually the wire on the red edge of the ribbon) has to line up with it. Get the orientation wrong and the chip simply won’t show up. For reference, the pinout of the Winbond SOIC-8 flash is: You don’t have to wire any of this up by hand, though, as the clip and the 1.8V adapter carry all eight lines for you. The only thing you need to get right is aligning pin 1 of the clip with pin 1 of the chip. Note: On my chip there is a gray dot painted onto the package, on the corner opposite to pin 1. Pin 1 is instead marked by the small indented (etched) dot, on the exact opposite side from the painted one. I have no idea why the gray dot is there, but if you align to it you’ll have the clip on backwards. Look for the indentation, not for the gray spot if yours has one too. With the clip attached, plug the CH341A into your second machine. A quick look at should confirm it enumerated: Install if you haven’t already: Before writing anything, make sure can actually talk to the flash over your clip: If everything is seated correctly, will identify the Winbond chip (detected as something like ). If instead you get: …then don’t panic. In my experience this is almost always poor clip contact rather than a real problem. I had to wiggle and reseat the clamp a few times before the chip showed up reliably, because those cheap clips are fiddly. Only proceed once the chip is detected consistently across a couple of runs. Even if the firmware is bricked, it’s good practice to take a backup before you overwrite anything. Read the chip twice and compare the dumps to be sure your contact is solid: If the two reads differ, your clip contact is flaky and you should reseat it and try again. As for the firmware image, Star Labs publish their firmware in a public GitHub repository . For external programming you want a full SPI image, not the EFI/ updater files. For my StarBook Mk VI (AMD) (product SKU ) that’s the image. The full-image files also live under the model’s directory . Pick the one that matches your model and rename it to something convenient, e.g. . Note: Star Labs ’ firmware versioning is, to put it mildly, a mess. As of writing, the last AMI (the original “BIOS”) release for the AMD StarBook is , while onwards is Coreboot . Whichever you decide to flash, just make sure it’s a full image for your exact model. Last but not least, write the downloaded image using the command: By default will erase, write, and then verify the chip. Star Labs ' official command appends (i.e. and ) to skip those verification passes, but I’d recommend leaving them off so confirms the write actually stuck. Either way, do not disconnect or disturb the programmer while it’s working. Once it finishes successfully, remove the clip, reconnect the battery, screw the backplate back on, and try to boot. When I powered mine back on, the StarBook came to life again, only to stop at a screen complaining about a missing boot entry, since flashing a fresh image also wipes the EFI boot variables. That’s nothing dramatic and you just need to point the firmware back at your bootloader. You can either use the boot menu and pick your SSD, which usually re-adds the boot entry, or boot a recovery/live system and run (this is what I did), or drop into the EFI shell and launch your bootloader manually: Note: On newer Coreboot releases Star Labs are enabling Rom Armor and anti-rollback. On the AMD board external flashing and downgrading still worked for me on , but this is expected to be locked down from onwards. What frustrates me most about this whole ordeal isn’t that a firmware update can go wrong, because that’s always a risk when you flash something. It’s that Star Labs ’ documented recovery path depends on a proprietary kit that nobody can actually buy, when a generic CH341A with a 1.8V adapter seemingly does the job just fine. However, this info is nowhere to be found in Star Labs ’ official documentation, which is why I decided to publish this write-up to begin with. Hopefully it spares the next person the day (and the stress) it cost me.

0 views
flowtwo.io 2 months ago

Building a New Blog Pt. 3

Long story short, I use a Cloudflare tunnel and self-host it on an old laptop. When I decided to rebuild this blog, one of my goals was to make it as "cloud-native" as possible. As a (mostly) static website, I concede it's sort of unnecessary since there are simpler ways of hosting a blog that don't require maintaining a backend at all. e.g. using Github Pages. But as soon as you want to do anything other than serve static content, it's easier if you have a backend you control. For example, I host some other projects on this domain and I have a custom publishing workflow for my posts—both of which require server-side logic. All that to say, I needed a backend somewhere and AWS is the cloud provider I'm most familiar with through work. But at work, it's not too often you get to try out new cloud services and stand up infrastructure from scratch. So, as with all my side projects, I took this as an opportunity to learn by doing. my odyssey through the clouds and back home The first question was where to store the data. My existing blog used MongoDB, so I stuck with that to make the migration easy. The most cloud-native version of MongoDB is AWS DocumentDB —it's a managed database service with full MongoDB compatibility. The pricing page says: With Amazon DocumentDB, you only pay for what you use, and there are no upfront costs. Sounded good to me. Did I actually do the math on what an instance costs per-month? No. But after 1 month I found out it was lot. That's the purple bar on the stacked bar graph above. Paying $100 a month for a blog database was not in the budget. This is when I learned that using a managed database service for my blog posts is like buying a yacht for a canoe trip. Then I found out that Mongo Cloud offers a free tier with up to 512MB of storage, so I switched to that. That's plenty for blog post storage. I guess that's still a managed database service, but it's free...so I took the free yacht. Next I had to decide how to host the backend. Building the backend as a Docker image would make it the most portable and easy to migrate, so that was a requirement from the start. Naturally, the first thing I tried was AWS Elastic Container Service (ECS) , a managed container orchestration service. Just upload your Docker image and click deploy, basically. I tried that for a couple months, but just like DocumentDB it is prohibitively expensive—even at the smallest scale of deployment. It's the teal bar on the graph above. I stopped using ECS and instead just installed the Docker engine on the smallest EC2 instance available ( ). My deployment workflow is just pushing an updated image to DockerHub and then pulling and running it on the server via a Docker compose file. For awhile, I also proxied all the traffic to my blog through AWS Elastic Load Balancing (ELB) , but not because I actually needed load balancing. I wanted to use ELB to handle SSL termination and automate the SSL cert renewal for my domain name. The alternative, which I had done before, was to route directly to my EC2 instance and handle SSL traffic directly via the app. I didn't want to do this again because using certbot + Java keystores (I run a JVM backend) was a huge headache, especially when it came to automated cert renewal. But again, using a managed service like ELB is not cheap, even if you have very little traffic. It's the orange bar on the chart above. Paying an extra $10 a month just for SSL management...also not in the budget. The better solution was to install an Nginx sidecar directly on my EC2 node which can handle SSL and proxies requests to my app. There's other benefits to using Nginx too, like response caching. After adding some aggressive cache expiration to the Nginx config, very little traffic actually ends up hitting my app and database. All these optimizations dropped my hosting costs down to about $15 a month. Not bad... an acceptable cost for me to have a space to host side projects and share content. But after running this for about a year, I looked at an old laptop I had sitting under my desk and thought, why not just run it there. For a laptop running linux, all you need to do is run: And now your computer never goes to sleep. Boom—you have a server. From the networking side, I originally thought it would require paying for a static IP from my ISP, but turns out there are several good options for proxying inbound traffic to your home LAN now. I chose to use a Cloudflare Tunnel , which is completely free, and it works great. Now I don't pay anything at all to host my blog*, and I still have the flexibility to run and deploy anything I want on it. It's kinda funny since the laptop is where I did most of the development for this blog. Now it's left the cloud and come back to the same machine. you shall return from whence you came Even though the premise of AWS is "elastic" services that scale from 0 to infinity, there's generally a minimum level of traffic needed to make the operational benefits of their managed services worthwhile, financially. For some of the services, the pricing doesn't actually scale down to the level of a personal blog in terms of resource and traffic requirements. Pro-tip: utilize the free tier from cloud providers for small projects! There are tons of ways of hosting something like this for free. And a million ways to do it the wrong way, as I showcased above. For example, DocumentDB was costing me $100/month. DynamoDB would've been free for the same use case. Self-hosting in 2026 is much more viable now thanks to cloud networking tools like Cloudflare Tunnel, Tailscale, Wireguard, or free VPS providers. You don't need to open up ports on your home network to the internet or pay for a static IP. * Alright, technically the laptop consumes about 6kWh of electricity a month, which costs me ~$0.80. Free-ish. Even though the premise of AWS is "elastic" services that scale from 0 to infinity, there's generally a minimum level of traffic needed to make the operational benefits of their managed services worthwhile, financially. For some of the services, the pricing doesn't actually scale down to the level of a personal blog in terms of resource and traffic requirements. Pro-tip: utilize the free tier from cloud providers for small projects! There are tons of ways of hosting something like this for free. And a million ways to do it the wrong way, as I showcased above. For example, DocumentDB was costing me $100/month. DynamoDB would've been free for the same use case. Self-hosting in 2026 is much more viable now thanks to cloud networking tools like Cloudflare Tunnel, Tailscale, Wireguard, or free VPS providers. You don't need to open up ports on your home network to the internet or pay for a static IP.

0 views
NULL on error 2 months ago

As gambiarras mais impiedosas que já fiz: parte 1

Gambiarra, de modo geral, é conhecida como uma espécie de quebra-galho ou atalho para conseguir algo usando a criatividade, muitas vezes de forma pejorativa. Ressalto que, neste texto, a ideia é justamente enfatizar a criatividade diante de grandes desafios. Certa vez, enquanto trabalhava com aplicativos mobile e livros digitais, fui encarregado de renderizar um livro em um aplicativo Android nativo. React Native ainda não existia. Talvez o Sencha Touch já existisse, mas, de qualquer forma, os tablets-alvo eram muito lentos e tinham pouquíssimos recursos. Pois bem, o que você faria nessa situação? Fácil, não é? Renderizaria as páginas do PDF como imagens JPEG e criaria um aplicativo para paginá-las, a.k.a. um livro digital. Não, claro que não! Seu gerente gosta de desafiar você e não gostou nem da qualidade das imagens nem do espaço ocupado em disco. Então, ele propôs usar Adobe Flash. Sim, naquela época ele ainda existia, embora já estivesse no fim da vida. Mas como? Ninguém sabia. Foi então que tive a ideia de criar uma WebView de um pixel por um pixel e embutir o Adobe Flash nela para renderizar as páginas sob demanda e com a máxima qualidade. As páginas viriam de um arquivo , pois o software usado para criar os livros exportava a arte vetorial para SWF. Nos PDFs, por algum motivo, tudo era convertido em bitmap. Mas surgiu outro problema: como transferir dados entre o aplicativo Android, escrito em Java, a WebView, em JavaScript, e o Adobe Flash? Criei uma WebView de um pixel por um pixel com o Flash Player rodando dentro dela. Na época, essa era a única maneira de executar Flash no Android, pois o Adobe AIR ainda não existia. Fazendo o Java chamar , eu conseguia executar uma função JavaScript e passar o bitmap renderizado da página solicitada. Sim, o bitmap inteiro era transmitido como uma string. Por sorte, a WebView não tinha uma limitação de 4.096 caracteres, ou algo semelhante às limitações comuns em servidores HTTP. Portanto, era possível transmitir páginas muito grandes e em alta resolução. Feito isso, bastava repassar os dados para o Flash. Mas como? Exemplo simplificado: No lado do Flash, em ActionScript: Então, no lado da WebView, em JavaScript: Feio? É, eu sei. Mas funcionou perfeitamente. Passamos a ter páginas com alto DPI, zoom, rotação e tudo mais. Como bônus, ficou bem rápido, pois o aparelho precisava renderizar cada página apenas uma vez. Isso acontecia em menos de meio segundo. Depois, bastava desenhar a imagem na tela. :-)

0 views
マリウス 2 months ago

The TEMU-fication of Software, Digital Goods & Services

Disclaimer: This is an opinion piece and most of it is speculation about a future that has not arrived (yet?), based on a few data points that have. As usual, summary at the end. A few years ago I would have laughed at anyone telling me that there is a serious market for ten-dollar drills, two-dollar dresses, and one-dollar pairs of shoes shipped from a warehouse on the other side of the planet. Today, however, that market exists and it has a name, and it is even publicly traded (sort of, through holdings). TEMU , Shein and a few others have built frankly mind-boggling businesses around the idea that if you make production cheap enough, fast enough, and just barely good enough to look right on a phone screen, an enormous part of the population will buy it, even when the product breaks within a week, when the materials it is made of contain worrying levels of toxic substances , and when the carbon footprint of one delivery exceeds that of an equivalent local purchase by orders of magnitude. The key to this sort of business model is not innovation, but instead the externalization and compression of cost. Somewhere upstream, people work seventy-five hours a week , in conditions most readers of this website would refuse to even visit, so that the rest of us can have a cheap plastic spatula at our doorstep within five business days. While the visible price collapses, the invisible costs get distributed onto landfills, lungs, and ultimately people that we will never meet. What follows is a hypothesis I cannot prove but have been turning over in my head for a while, as we are watching the same thing happen to software, books, music, (film-)scripts, and most of the digital goods and services we consume. The cheap labor in this case is not human, it is a Large Language Model ( LLM ), or what many people these days call “AI” , and the externalized cost is, among other things, quality , which requires craftsmanship to produce, and attention to perceive. And just like with physical goods, we will probably end up with a two-tier market, in which we have a large and massively profitable lower tier of generated slop , and a smaller, more expensive upper tier of work that is still recognizably human. I’d like to call this the TEMU-fication of software, digital goods and services , and describe what it might look like. For decades, the global fashion industry has relied on a workforce that has almost no leverage and no voice, and for which the economics work because someone, somewhere far away, will sew a t-shirt for less than the price of a coffee. Without that skewed arrangement, the entire fast fashion business model collapses. The garment in your hand is only cheap to you because it has been expensive to someone else , in ways that the price tag does not show. Modern Large Language Models occupy a similar position in the economy, with one important difference, which is that there is no human being in the sweatshop, only a stack of GPUs trained on a corpus of work that other human beings produced over the course of decades. The labor that has been compressed is historical and the model is a kind of compressed copy of the work of millions of programmers, writers, illustrators, and musicians, served back at near-zero marginal cost. Well, at least in theory, and only if the hyperscalers find a way to lower the cost per token, but that’s a different topic. However, the result is the same. A class of goods can suddenly be produced for an order of magnitude less than before. And, just like with TEMU , those goods turn out to be just barely good enough . The most direct manifestation of this so far is what is being called vibe coding . The term refers to the practice of describing what you want in natural language to an LLM , accepting whatever it produces, iterating over it with more refined descriptions of the basic idea and eventually shipping the result into production. Whether the developer actually understands what was generated is increasingly considered an implementation detail . And while the output is technically software, the question is what kind of software it is. A 2025 Veracode report found that approximately 45% of AI-generated code samples failed security tests and contained critical vulnerabilities from the OWASP Top 10 , and a multi-language, multi-model academic study that evaluated outputs from Claude , Gemini , Codestral , GPT-4o and Llama-3 across Python, Java, C++ and C, found that a substantial fraction of generated snippets were either non-compliant with basic secure coding standards or actively triggered classified weaknesses (buffer overflows, hard-coded credentials, SQL injection, cryptographic misuse, path traversal, you name it). Even more concerning is a peer-reviewed 2025 paper from IEEE-ISTAS that documents a 37.6% increase in critical vulnerabilities after just five iterative prompts, suggesting that the more you let the model refine its own code, the worse the security posture gets. When these issues compound over time, the result is a higher total cost than traditional development. However, this doesn’t matter when you don’t think long term , but fast fashion instead. Also, none of this is to say that an experienced engineer cannot use these tools well, because they certainly can. The issue is what happens when the same tools are used by someone who does not know what good looks like in the first place, and there is nobody downstream of them who does either. The output passes the basic test of it runs and looks plausible , ships into production, and accumulates the kind of architectural and security debt that surfaces only when something goes very wrong . Note: There are credible voices in the industry, particularly from the AI tooling vendors themselves, who argue that AI-assisted development raises a floor more than it lowers a ceiling. In this view, the median piece of software has always been mediocre, written under deadline pressure by tired humans, copied from Stack Overflow without much thought, and held together by duct tape. If an LLM produces output of roughly comparable quality in a fraction of the time, the argument goes, nothing got worse. We are simply removing a bottleneck. I find this argument partially persuasive, and partially convenient for the people making it. It is true that a lot of software was already not great, but it is equally true that there is a difference between bad code written by a human who at least understood what they were doing , and bad code written by a system that does not understand anything . The first kind can be questioned and corrected, but the second kind tends to compound, because the person shipping it cannot answer why it does what it does. At least for now. Software is not the only place where this is playing out. The book industry is arguably further along, with estimates suggesting that somewhere between ten thousand and forty thousand AI-generated books are uploaded to Amazon ’s Kindle Direct Publishing platform every month, many without any disclosure that a model was involved. In June 2023, the Kindle Top 100 bestseller list was found to contain only 19 books written by humans . Amazon has since introduced limits and disclosure requirements , but enforcement is patchy and authors continue to push back against what looks like a slow flood. Categories that have been hit particularly hard include travel guides (generated guides to cities the author has never visited, with restaurant recommendations that don’t exist), nutrition and health (generated diet advice with citations to studies that don’t exist), and public-domain rewrites (generated adaptations of older books, relying on the recognizability of titles that the actual authors never agreed to). Travel guides in particular have produced a small genre of stories where readers arrive at addresses that turn out to be empty lots, or follow walking directions through neighborhoods that no human would ever recommend. Note: The defense, again, is that the bottom of the book market was always full of filler, that print-on-demand has been around for a long time, and ghost-written business books and assembly-line genre fiction predate generative AI by decades. However, the new thing is the scale at which low-effort content can now be produced, and the speed at which it can drown out the rest of the catalogue. Authors are competing for shelf space against entities that can ship a hundred new titles in a weekend. A 2025 analysis of 65,000 English-language articles published since January 2020 found that a little over half of all new articles on the internet are now AI-generated , and it’s not only the written word that’s being churned out by machines . YouTube has its own version of the problem, where, according to a Guardian analysis, nearly 10% of the world’s fastest-growing channels feature nothing but AI-generated content , and on Shorts specifically more than one in five videos served to a new user is low-quality AI-generated material . Not even the highly creative and (up until recently) human process of making music is immune to this TEMU-fication . Spotify has been removing ghost artist tracks for years, but the practice scaled up dramatically when generative tools made it trivial to produce convincing lo-fi background music in arbitrary volume. The platform has reportedly removed 75 million spammy tracks in a single year , and high-profile acts like the AI-generated band The Velvet Sundown amassed over a million streams before being unmasked. There has been at least one criminal case, involving over $8 million in fraudulent royalties , built entirely on AI-generated music and bot streams. However, that is no reason to applaud Spotify , as the company appears to fight the AI spam only when it’s someone else trying to make money off of it. However, there is a sliver of hope, as engagement with AI-generated articles reportedly dropped by around 40% in 2024, and human-generated content seemingly still gets roughly 5.4× more traffic than AI-generated material in some studies. About 38% of consumers openly express skepticism about AI-created content, and people do still seem to be voting with their attention. Whether that vote is powerful enough to shift incentives at the platform level is a different question, and personally I’m not particularly optimistic, especially given that the platforms profit either way. Let’s take Netflix as an example. From my understanding, the WGA ’s 2023 deal explicitly prevents studios from treating AI-generated material as source material, or from using AI to write or rewrite scripts, and Netflix was seemingly bound by that agreement until at least May 2026. Netflix ’s own Generative AI Production Guidelines also seem to reflect this, stating that AI is permitted in ideation , but that its use should not replace or materially impact work that would otherwise be done by union-represented writers, actors, or crew members, without proper approvals . While that sounds reassuring on the surface, it is, in my view, a delay and not a limit. The same company has publicly committed to going all-in on AI in its production pipeline , has signed deals with VFX automation providers that explicitly put a chunk of the global VFX workforce at risk, and has already used generative AI in at least one of its programs ( El Eternauta ). The trajectory seems to be “use AI everywhere it is contractually allowed right now, expand into the rest the second the contracts permit it, and spin the result as dEmOcRaTiZaTiOn Of CrEaTiViTy” . So here is my specific (and quite possibly wrong) prediction: Within the next five to ten years, Netflix will offer a basic subscription tier whose catalogue consists predominantly of AI-generated or AI-assisted content. We are talking generated procedural shows where each episode is remixed from a small set of templates, generated kids’ content that is vaguely educational and impossible to remember an hour after watching, and generated dramas that recycle plots from existing IP and vibe the rest. For this, the viewer pays the lowest monthly price, while the platform pays nearly nothing in production cost and keeps an enormous margin. The only “upside” for consumers will be the lack of ad breaks, as targeted advertising will quite possibly be injected in real-time into the show you’re watching, seamlessly blending into the storyline without you noticing it, but ultimately still triggering your ape brain to crave a refreshing soda or a sweet treat . Their premium tier, meanwhile, will become the human-made tier. Series with credited human writers, films with credited human directors, and performances by humans whose likeness has not been digitally replicated. The marketing will not call it human-made , because that would be admitting that the cheap tier isn’t , but the price difference will make it obvious. You will pay extra for the same thing Netflix has been selling you all along, except now it is positioned as a luxury. Clearly, I cannot prove that this is what will happen. Netflix ’s own guidelines, as written, prohibit it, and the WGA deal forced a delay. But once the contractual block has lifted, the financial logic is hard to argue with. A streaming service that can produce good enough content for fractional cost will eventually try to. And, mind you, Netflix is just one example. The same logic applies to every other content-distribution business with a subscription model and a margin. If you want to know what the human side of this two-tier world looks like, I think the best existing model is the handicrafts and handmade goods market . By 2025, that market was estimated at roughly USD 987 billion globally, with projections reaching over USD 1 trillion by 2035 . There is data suggesting that U.S. consumers already spend almost a fifth of their money on handmade goods rather than on mass-produced equivalents, and over half of handicraft buyers globally indicate a preference for products that are eco-certified or made from natural materials, going in the exact opposite direction of what TEMU has been doing. What this market shows is that industrialization does not erase the artisans, but pushes them into a different segment. People did not stop buying handmade chairs when factories started making chairs cheaply. While the masses opted for the cheaper, mass-produced items, a small but sustained minority of buyers continued to seek out the human-made version, and over time were willing to pay a premium for it. If the hypothesis holds, software engineering, writing, acting, illustration, composition and the other content-producing professions will undergo something similar. The bulk of the market will migrate to the cheap, mass-produced, generated tier, while a smaller market will continue to value, and to pay for, work that is verifiably the product of a thinking, breathing, opinionated human being. We are already seeing the first signs of this in agencies that explicitly advertise human-only content (at a premium), and in licensing companies flagging tracks as human-composed to distinguish them from AI library music. I think that the interesting question is not whether this segmentation will happen, but what proportion of the market ends up in each tier, and how robust the upper tier turns out to be. There is a darker version of this analogy. Roughly 57-60% of the daily caloric intake of the average adult in the United States and the United Kingdom now comes from ultra-processed foods . Across 22 European countries the share ranges from 14% to 44% , depending mostly on how protected the local food culture has remained. These foods are cheap, abundant, available everywhere, and nutritionally inferior to the alternatives in ways that have been studied at length . People know this, but they eat them anyway, often because the alternatives are slower, more expensive, harder to find, or require skills that have not been taught. I suspect that AI-generated content is on the same path. The cheap tier will not be a marginal phenomenon serving a marginal audience, but it will be the default , the cornerstone of how most people consume software, entertainment, news, and information, because it is what the platforms will serve them and what their monthly subscription covers. Some will care enough to seek out the alternative, but most will not, just as most people, knowing what they know about ultra-processed food, do not change their grocery habits. Probably the strongest counter-argument to all of this is that LLMs are still early, that the quality issues are transient, and that within a few model generations the gap between AI-generated and human-generated work will narrow to the point where the distinction stops mattering or might not even be possible anymore. If that is true, the two-tier picture collapses, because there is no longer a quality difference to justify the upper tier, only a marketing difference. The handmade analogy breaks because, unlike a hand-built chair, a generated novel is functionally identical to a written novel once you can no longer tell them apart. However, I am doubtful that this is going to be the case. There are tasks where I have watched the gap narrow faster than I expected, but there are also tasks where the gap has stayed stubbornly fixed and the failures have just gotten more sophisticated. My instinct is that for narrow, well-bounded technical work, the gap will close further. For long-form work that depends on a coherent worldview, lived experience, and, most importantly, emotions, I doubt it will, because the model has none of those. The second counter-argument is that the consumer backlash will be stronger than I am giving it credit for. The 40% drop in engagement with AI-generated articles is not nothing, and platform incentives may shift if users start to penalize AI-flooded feeds. Apple and others have started experimenting with content provenance and disclosure schemes that, if widely adopted, could stop the worst of the flooding. So it is possible that I am underestimating the immune response . The third counter-argument is, that the cheap tier might not be sustainable at all, because AI-generated content trained on AI-generated content degrades model quality , and the broader ecosystem ends up poisoning its own training data. If that turns out to be the dominant dynamic, the cheap tier could collapse before it becomes entrenched. I think all three of these arguments are valid and have a certain weight to them, but none of them are strong enough, in my view, to make me confident that the TEMU-fication will not happen. They might modulate how it happens, but they probably do not stop it. Initially, I went looking for an optimistic ending for this write-up, to say that software engineering is not going away , and writers are not going away , and actors are not going away . And while all of that is, I think, true, none of it should be confused with things will look the same . What I expect, and what I am to some degree already seeing, is that the people producing software, books, music, scripts, and other human-made work will not disappear , but they will get pushed into a narrower, more specialized, more “luxury” -coded part of the market, pretty much the same way hand-bound notebooks, independent record stores, and small bakeries that mill their own flour did. There will still be a livelihood in it, at times a very good one, but it will look vastly different, and there will probably be fewer people making a living in these fields. My assumption is that they will be more visible inside their niche, but less visible outside it, and they will make their case in part on the basis of provenance , where something was made by a human who knew what they were doing, and you can tell. Meanwhile, the bulk of what most people interact with will, I suspect, be generated. Some of it will be fine, and some of it will be ultra-processed , in the same sense that a frozen lasagna is ultra-processed. It will be functional, calorically adequate food , but it will not be what your Italian grandmother was making. People will nevertheless eat it because it is there, it is cheap, it is convenient, and because the alternatives have been priced out of their daily life. There is no “inevitability” to it, because none of this is really decided yet. There are still choices, made by platforms, by regulators, by consumers, and by the people doing the actual work, that will shape which tier ends up being how big and how durable. The handmade market exists because enough people kept buying handmade goods to make it viable. The human-made tier of software and digital goods will exist because enough people keep buying it, or it won’t exist at all. If you are someone who writes code, or stories, or music, or scripts, by hand, with intent, and with a point of view, I do not think the LLM is going to kill your job . I do think, however, that it is going to change the shape of the market you operate in, push you toward the upper tier (whether you wanted to be there or not) and ask you to make a more deliberate case for why your work is worth the difference in price. For the rest of us, the more interesting question is which tier we are choosing to consume from, and whether we are choosing it on purpose, or just because it was what the algorithm served us by default. I have my suspicions about the answer, but I would love to be wrong.

0 views
./techtipsy 2 months ago

I guess I'm a cactus farmer now

When I was a teenager, I got gifted a cactus. I didn’t know much about it and had never really had any plants of my own. More than a decade later I looked it up, and it seems I received a cactus of “Mamillaria” family, which is somewhat amusing once you learn what that name roughly translates to. Estonia isn’t known for being an environment where cacti thrive. We have a nice spring, short and sweet summer, nice and chill autumn, and the car-rusting winter season. And yet mine did, eventually. The cactus started growing in length. It continued until it couldn’t continue vertically, and then it started slanting sideways, eventually falling over with its pot. That was annoying to deal with. One day I woke up to my phone alarm ringing, reached out to it only to find that the cactus had fallen over it and I had just found where the cactus was. The cactus also survived a lot of abuse. At one point I put a bit of sawdust from the cage of pet mice that I had (adorable but difficult pets by the way!), and it actually liked it a lot and started growing even faster. Once it had grown horizontally for at least 20cm, it decided that now is a good time to fork it and to grow in more than one direction. I left the cactus alone, occasionally repotting it when necessary, and keeping it from falling over all the time. Then an acquaintance mentioned that hey, you can actually pop those forks off from the cactus and grow them separately. That’s when things got out of control. Once I popped one off, new ones would start growing from the host cactus nearby. Not long after, I had 37 cacti. I started giving them away because annoyingly these things are actually very cheap in gardening shops and I ran out of physical window sill space to put them on. These cacti also bloom in a really nice way, usually once a year near April or May in Estonia, but sometimes again in July-August. All the warnings about over-watering cacti are also very relevant and I suggest looking up the correct behaviour. With age, it has become easier to handle cacti with my bare hands. Sometimes the needles stick in, that is a bit annoying. A few years ago, I had gifted a bunch away and kept some for myself. Everything was going well, until a succulent that I brought in as an IKEA store rescue started having some white things appear on it, and not much time later it got to my cacti. Eventually, most of my cacti shriveled away and the few that were in a different room were not doing great either. The original cacti that spawned this decade-long adventure was now dead. One of the cacti that I gave away had started growing a lot. Ridiculously well. And just like its parent, it spawned baby cacti on it, about a few dozen of them at a time. Things got quite unwieldy and we agreed to a trade: I get the big boi and I will give a smaller one in return. I am once again in possession of a large cactus that loves to grow horizontally, and this one spawns babies like crazy. The trick is to try to wiggle them off, then leave them to sit for a few days to a few weeks away from direct sunlight until they start showing signs of roots, pot them into soil, and grow them within natural light, but not direct sunlight to avoid them drying out. If you’ve done everything correctly and have avoided overwatering them, you will have a bunch of new cacti! I’ve also learned that these plastic growing thingies are great for the first potting as it’s much more space efficient compared to giving each an individual pot. Is that the right way to grow cacti? Probably not, I’ve been winging it this whole time and only have a vague understanding of what to do. Not cacti advice. I guess I’m kind of a cactus farmer now. Send help. Or at the very least clay pots and cacti soil, I’m running out.

0 views
Ivan Sagalaev 2 months ago

Categorization with NLP

Since launching my categorization tool Shoppy I've had some fun analyzing collected data which resulted in considerable complication of the prediction model. And now I feel the urge to write a deeper dive into its inner workings. I'm not sure how useful it would be for anyone who isn't a part of the Grocery Categorization industry, but hopefully some NLP tricks could be at least interesting to any general practitioner. Please note that I'm by no means an NLP expert! Part of the reason for writing this kind of posts is to try and nerd-snipe someone who knows more into sharing their expertise. Oh, and it's not a short one, this! Settle down :-) Just to remind everyone, what I'm trying to solve for my slowly developing shopping list app is suggesting grocery categories for products. So that it knows that "Milk" is dairy, "Apples" is produce and so on. This categorization helps with grouping and sorting, and it also looks nicer with colors and appropriate icons. The usual approach to solving such problems is Machine Learning, and specifically — classification . That doesn't work for me though, as I couldn't come up with any means of collecting enough data myself, and hiring a consultancy is way above the budget of a tiny personal project. So instead I'm manually crafting a clever algorithm with explicitly handled edge cases. The first step is turning free-form input into something more formal and predictable: a set of lexemes. Step by step, it looks like this: This gives me a normalized, stable input key independent of basic morphological variants: A note on stemming. I'm using the original Porter algorithm . It's widely supported, but is also the most simplistic. I don't really care about the correctness of the result from the point of view of the English language. As long as the algorithm used to produce the data is the same as the one used for checking against it, I'm good. The straightforward design for a database is just a mapping from a whole key to a category: . But that would require listing all likely real-world products: all sorts of apples, peppers, beans, etc. Which doesn't work for me since, as I mentioned, I don't have a firehose of data to fill it up. But you'll notice that mostly all such product are defined by one word: an apple is an apple and is produce, regardless of the sort. So let's reify this in the form of a CSV file: These one-word keys are called unigrams . To match a search query, we can simply look up every separate unigram from it in the database. It works for surprisingly many cases, but it breaks on things like "apple juice", because despite having "apple" in it, it's a beverage, not produce. This is fixed by making the order of rows in the database significant, so that goes before . Then, if several of the unigrams in a search query match, the earliest one wins. This might smell to you like something prone to potentially irresolvable ordering cycles, but I actually found that I only really need two groups of significance: Things like "juice", "milk" and "oil" are usually derived from something, as in "apple juice", "oat milk" and "olive oil". Keeping those derivations above raw ingredients ensures those tings are categorized correctly. And don't take the word "raw" too literally. There are things like "ketchup" in there! But since nobody puts "tomato ketchup" on their shopping list, it is considered "raw" in my domain. The next problem is combinations of words. "Spaghetti squash" is not a pasta, but a kind of squash. And "apple sauce" is neither produce, nor a condiment, but a snack! In both of these examples no single unigram is enough to correctly identify the product. This means I need to consider two-word terms called bigrams : All bigrams come before unigrams, as their purpose is to serve as more specific disambiguations of conflicting unigrams. But this ordering is less significant than the groups (Derivations and Raw), so each of those gets their own set of bigrams. At search query time, I produce bigrams as all possible combinations of two from the set of unigrams. Then they're added to unigrams as search terms: The lookup process stays exactly the same: just check all terms one by one. By the way, I need this code to also work in the Kotlin code of my Android app, and since Kotlin doesn't have in the standard library, I wrote a trivial recursive implementation by hand: Felt like solving a coding interview problem :-) A few words about "pepper"… When combined with another word, it most often belongs in produce: "bell pepper", "chili pepper", "serrano pepper", etc. And since there are many of such, I want to classify the unigram "pepper" as produce to cover them all. However the single word "pepper" usually means ground black pepper, which is a spice! I tried to complicate the model at first, supporting wildcards like "* pepper". But I didn't find any other exception that would use it, so I de-complicated the model back and hard-coded a dumb `if` substituting "pepper" with "black pepper" before any lookups :-) Practicality beats purity! Human languages tend to merge words commonly used together. So something straight and forward becomes straight-forward and then just straightforward. I realized I need to care about it here after I collected examples like "redbull" and "lipbalm". Both should properly be spelled as two separate words, but people usually don't consult a dictionary before doing their groceries, so… Funny factoid: "breadcrumbs" and "seaweed" are spelled as single words. This can't be solved by spellchecking because my database doesn't contain original spelling: it has the bigram "bull red" and the unigram "balm", and they both are too far away from those search queries. Instead, I started thinking about splitting words into syllables. To my everlasting surprise, nltk turned out to have several of those, from which I picked SyllableTokenizer . It is pretty simple, but it does work for "redbull" and "lipbalm" as you'd expect. At first, I tried to switch the entire database to contain only syllables instead of full stemmed words, but it didn't work out. There are common syllables like "bar" and "can" that are also full words in their own right. So something like "barbecue sauce" would be split into and will suddenly become a snack because I have a record saying that any kind of "bar" is a snack. So instead I converged on a two-step lookup. First I split the search query into regular words and look them up as I did before. Then, if I didn't get a match, I split the words further into syllables and do a second lookup. The syllabilization algorithm was another thing I had to port to Kotlin, because I couldn't find anything like that in the entire Java ecosystem. Drop me a note if you want that code for some reason. Spell checking proved to be necessary nonetheless. Do you know if it's "fusilli" or "fussili"? Does "camamber(t?) has a "t" at the end? Or is it "haloumi" or "halloumi"? (Answers: 1) the former, 2) it does and 3) both spellings are correct.) Unfortunately, a regular spell checker against English wouldn't work very well: all kinds of international foods and deliberately misspelled brand names kill the idea. Fortunately, I can spell check against my own database, which naturally represents the entire language that my app knows! And the standard approach to spell checking is to employ some "edit distance" algorithm which shows how far apart is the spelling of two arbitrary words. I went with Damerau-Levenshtein (standard Levenshtein plus transpositions). I also had to come up with empirical numbers for maximum allowed distance depending on the word length. This was done completely unscientifically, and I expect to have to tweak it further. But for now it works! The most astute reader might have already spotted a problem: what was a single hash-table lookup before the introduction of edit distance calculation has now turned into a linear search with each test being way more expensive than a straightforward string comparison. So I had to to some low-hanging fruit plucking in terms of optimizations: But I have to say that the main thing working in my favor here is that the data is just small ! I want to highlight two particularly weird wins made possible by the combined effect of syllabilization and spell checking. And I just want to state for the record that the existence of such a thing as "mayochup" made me die a little inside. Is the trouble of mixing mayo and ketchup by yourself in your kitchen so unfathomably hard that you need a brand to produce it for you? Geez… Initially I thought of putting the collected data up on Kaggle , but since it's now heavily dependent on a custom algorithm, I'm just going to leave it as is in the Shoppy repo: Lowercase the input string Normalize it into the [NFD Unicode form][] — the one that separates accents from their characters, which lets me get rid of the former (not everyone bothers typing "crème fraîche" properly). Split the string into "words", ignoring everything else like punctuation and whitespace. In my case words are defined as alpha-numeric characters and an apostrophe (because I want things like "7'up" to be one word). For each word, get rid of apostrophes and stem them. Sort the resulting list of word stems. Assume people don't usually misspell the first letter of a word. Don't bother comparing words that differ in length by more than the maximum allowed distance. Don't bother comparing words with different n-grammness (like bigrams and unigrams). If a search term matches exactly, don't bother two check the rest of them to find a closer match. A single term "may o" works for: "mayo", "mayonnaise", "mayones" and even "mayochup". Syllabilizing these words produces "may" and some variant of "o"/"on"/"oc", which is then smoothed over by spell checking. One of the previously unknown to me items in the feedback was submitted as "separilla", which turned out to be a misspelling of " sarsaparilla ". So I picked out three syllables conveying the meaning: "sar", "pa" and "ril". And, thanks to spell checking, they're enough to cover both the correct spelling and a few incorrect ones. lib.py : all the functions terms.csv : terms database (in a weird format of CSV-with-blank-lines-and-comments) kinds.csv : hierarchy of defined categories (I didn't mention it, but you'll get the idea)

0 views
Farid Zakaria 2 months ago

Nix finally has a source-bootstrapped OpenJDK

One of the earliest requested issues we had opened on GuixPkgs was to add more packages, specifically OpenJDK. “Specifically I would like to see openjdk translated. openjdk is not bootstrapped from source code in nixpkgs” [ issue#3 ] Ever since I learned about the stage0 bootstrap chain and how Guix announced full-source bootstrap for all packages in 2023, I was in awe. They provided a package graph of more than 22,000 nodes rooted in a 357-byte program, including the JDK. 1 GuixPkgs can now build OpenJDK 25 🎉 Does it really work? Let’s take it for a spin. lives in the output, since Guix splits the package. is written in Java. is C++, but the class library it needs is Java, and the compiler that compiles the class library is Java, and it runs on a JVM that needs a class library… and so on. This is a bootstrapping problem. Every distribution resolves this the same way in practice: download a JDK and use it to build your JDK. Debian documents the pain , and the Bootstrappable project has a whole page on it. They are fun reads, I highly recommend them. What makes JDK special is that there is no actively maintained JDK that can be built from source without a JDK. The authors had to go back quite a few years to find one, and bring it back to life. Nixpkgs does the same. is built by : a 135 MB prebuilt-tarball. Nixpkgs makes it pretty easy to audit in the meta.sourceProvenance of the package. What does the source provenance of GuixPkgs’ look like? It’s a little whacky but the overall build chain in Guix is the following: Nineteen complete JDK builds, from a C++ program. We can use to emit the entire closure as a dot file to visualize the differences. I restyled the nodes and edges: dots instead of labelled boxes, and the red to highlight the derivations that exist only because of the JDK bootstrap. The upper panel’s entire Java story is those two red dots and the one edge between them: → . The lower panel’s is the 33-node constellation. Why is the Nix graph still so big if I claimed it was from a binary distribution? It turns out that a JDK closure is mostly not Java. It is a large C++ program that wants X11, cups, fontconfig, freetype, alsa and zlib, sitting on a C toolchain that both sides bootstrap from source. That sub-graph is the same in both and it swamps everything. I was curious to zoom-in and see the source-bootstrap portion of the derivation graph. Since that shared sub-graph is drowning the signal, I chose to filter it out. Which derivations exist in this closure only because of how the JDK is bootstrapped? 🤔 That is a reachability query. We delete the Java-provenance nodes from the graph, see what is still reachable from the root, and whatever is left are the derivations that exist only because of the JDK bootstrap. This lines up with our intuition. Nixpkgs only needs two derivations to bootstrap the JDK, while GuixPkgs needs 876 . Note Interestingly, the bootstrap build for JDK has in its closure: IcedTea 8 wants GTK 2, which wants its own Mesa, which wants 😱 What started off as a fun art project , started during TacoSprint 2026 , has now found relevance to those interested in bootstraping and reproducible builds. Nixpkgs has made a lot of progress on this front as well, however it is still not fully bootstrapped and relies on plenty of prebuilt binaries.  ↩ jikes 1.22 (C++) compiles GNU Classpath 0.93 , a free reimplementation of the Java class library. Classpath is enough to bring up JamVM 1.5.1 , a JVM written in C. Now something can run Java. Now we can finally use Java tools: Ant and ecj , the Eclipse Compiler for Java. Then it doubles back and rebuilds all of that with the newer versions to get more modern JDK features. That stack finally compiles IcedTea 2.6.13 (OpenJDK 7) → IcedTea 3.19.0 (OpenJDK 8) → OpenJDK 9 . After which it is one rung at a time: 9 builds 10, 10 builds 11, …, 24 builds 25. Nixpkgs has made a lot of progress on this front as well, however it is still not fully bootstrapped and relies on plenty of prebuilt binaries.  ↩

0 views
Lalit Maganti 2 months ago

How I Find Problems to Solve as a Staff Engineer

Note: this post was revised after publishing for increased clarity, based on reader feedback . “How do you find problems worth working on?” a senior engineer I mentor asked me recently. He’s trying to make the jump to staff engineer and realized that the role isn’t just about doing the work he’s assigned. He also needs to get involved in figuring out what his team and org should be building. Someone else had suggested blocking out time in his calendar to think about the bigger picture. He’d tried that, but hadn’t found it productive, so he asked if I had any alternatives. I told him I rarely find good problems by staring at a blank page and trying to “think strategically.” Instead, I act like a sponge. I listen to the stream of day-to-day noise, absorb the problems people are having and let them sit in the back of my mind. Over time, some fade away while connections begin to appear between others that initially seemed unrelated. Eventually, I start to see what’s really slowing people down and what my team or I can do about it. I’ve worked with many engineers who’ve never really tried this. They wait for managers or leads to identify opportunities, then demonstrate their value by solving the hardest assigned problems. That can absolutely lead to promotion. But the projects that have made the biggest impression in my career were the ones where I found and solved an important problem my leaders did not yet realize existed. One caveat: my experience comes mainly from working on infrastructure and developer tools at large companies, on teams where engineers have a lot of bottom-up autonomy to influence their roadmaps. In a more top-down environment, there may simply be less room to work this way. People love talking about the problems they are facing: in meetings, chat threads, presentations and email. They explain why their work is hard, complain about what slows them down and describe what they wish they could do. When something overlaps with my area, I start pulling on the thread. I might ask, “If X existed, would it solve your problem?” or point them at an existing feature in a product I own and ask how much of their use case it covers. Users often ask for a particular solution instead of explaining their root issue. Rather than taking the request at face value, I keep digging until I understand what they are trying to accomplish and why existing products do not work for them. As a natural introvert, this sort of ambient listening works particularly well for me. I don’t need to fill my calendar with speculative meetings just to find ideas; there is already an enormous amount of useful information flowing around me during a normal week. When a problem seems worth exploring, though, I become more active; I need to see how it affects the team’s day-to-day work. I’ll sit with them as they walk me through their workflows and the bugs they’re investigating. When I can, I’ll try working through some of those bugs myself. Seeing the problem firsthand makes it easier to separate what the team actually needs from the solution they asked for. I also seek out people who see more of the organization than I do: those who own critical systems, work across several teams or have particularly deep insight into the work downstream of my team. I’ll arrange a 1:1 or coffee chat and ask about interesting problems they’ve come across. They may have already seen the same issue in several places and started connecting the dots, giving me a head start on patterns I might otherwise have taken much longer to notice. Several times, I’ve been burned by moving too fast. I became excited by a request from a vocal team, built the feature and watched them barely use it. Their priorities had changed, or the request had come from a one-off investigation that no longer mattered. How eager a team was in that moment wasn’t the same as how important the feature was relative to everything else my product needed to support. By hyperfocusing on their request, I lost sight of the bigger picture. That taught me to let potential problems pile up. Listening the way I do leaves me with far more of them than I could possibly solve, and not all deserve action. Most don’t need to turn into projects the first time I hear about them; waiting can be a superpower. Waiting means the same problem might pop up independently in different teams, making it a higher priority to solve. Or problems that look different on the surface might turn out to have the same shape, so I can address several use cases in one shot. Or, as I’ve learned painfully, the requesting team didn’t even care that much in the first place. Instead, I make a mental note and revisit the problem if it comes up again. Other engineers I know write this sort of thing down more systematically. The mechanism is a personal choice: everyone has to figure out what works for them. What matters is keeping unresolved problems around long enough for more evidence to accumulate. Waiting helps me collect evidence, but that alone doesn’t tell me what to build. I still need to work out whether the problems I’ve retained are genuinely related and what, if anything, could address them together. Perfetto, the performance debugging tool I work on, is a good example. It displays recordings of system activity on a timeline made up of rows called “tracks.” Over a couple of years, teams kept asking for small, specific additions to the UI. One wanted a command to keep their preferred tracks pinned to the top of the screen; the next team wanted the same, but for a completely different set of tracks. Others wanted Perfetto to open already zoomed in on a particular part of a recording, or to show a custom aggregation tuned to what they cared about. A few had stopped waiting for us and built elaborate workarounds with bookmarklets. 1 By the time enough of these had piled up, my head was the usual tangle: the requests themselves, the constraints on each and a handful of half-formed solutions. I’ve learned not to force a solution by just sitting at a desk and thinking. Instead, my best untangling happens on long, aimless walks around London, where connections come more easily when I’m not trying to force them. What I eventually realized was that none of these teams really wanted the specific feature they’d asked for. Each wanted to personalize Perfetto for their own workflow without imposing their choices on everyone else. The underlying need wasn’t any one feature but rather the ability to extend the UI. When a connection like that finally clicks, it’s one of the best feelings in the job: several awkward requests collapse into a single idea, and possibilities open up that none of them hinted at on their own. That feeling, though, is exactly when I have to be careful, because a common shape is only a hypothesis and elegance is not evidence. When it happened with extending the UI it turned out to be real, but I’ve been fooled before. In another recent case I was convinced that building a transparent caching system for querying Perfetto traces would solve issues with sharing large traces and repeated queries. It was only as I wrote the RFC and built a prototype that I realized the elegance was a lie: the two problems wanted genuinely different solutions. I reluctantly split the design in two, both halves of which have since shipped. 2 You’d think this would be the moment I start building, but it usually isn’t. How far I go depends on how sure I am that the idea works and that people actually want it. If something is useful and low-risk enough, I act straight away: I send the change and let my manager know. When I’m unsure whether an idea will work or how much effort it will take, I build a throwaway prototype instead; it exposes the failure points and gives me something concrete for others to react to. And when an idea is big but I’m convinced by it, I commit to the full effort: weeks or months of work and the hard yards of building support across other engineers and teams. Through all of it, I’m not only trying to convince other people; I’m also trying to convince myself. Sometimes the honest answer is to stop: if people don’t see the value I do, or we hit a major technical wall, I’d rather drop the idea now than build something no one uses or that becomes a maintenance nightmare. And sometimes it holds up but the timing is wrong, so I park it, ready to spring into action the day it becomes an org priority. When an idea does hold up, I don’t necessarily need to be the person who builds it. I might implement it, someone else on my team might, or it might change what the org focuses on. Finding and shaping the right problem can have an impact even when I don’t own the implementation. The Perfetto extensions idea was worth that full effort. We were already building plugins to modularize the UI, but they weren’t enough: teams had to open source all their plugin code, which wasn’t an option for many internal use cases. So before building anything new, I took the problem and my proposal to my manager, teammates and the client teams. I ended up writing two RFCs, having several 1:1s and giving a couple of talks, refining it as the feedback came in. In the end, I designed and implemented macros as “lightweight extensions”: a way to automate actions in the UI without writing a plugin. Extension servers took the idea further by letting teams share their macros. Instead of implementing every requested feature ourselves, we gave teams ways to adapt Perfetto to their own needs. Dozens of teams inside Google now use macros and extension servers, and several other companies use extension servers internally too. The more often I go through this process, the easier it becomes. When I show genuine interest in someone’s problem, ask useful questions or help solve it, they remember. They start coming to me earlier and bring me into conversations with other people facing related issues. That gives me a wider view of what is happening across the organization, making it easier to spot patterns and build things people actually need. Solving one of those problems brings me into more conversations, and the loop continues. Those successes build the kind of trust that comes from long-term stewardship . Early on, I had to turn many of these ideas into something real myself to prove that my judgment was sound. Over time, my manager and org gave more weight to my assessment of what mattered. That allowed me to influence the roadmap without needing to own every project. This differs from the idea that becoming a staff engineer means replacing technical work with meetings and coordination. For me, conversations are inputs into what I build, not the end result. That is what I wanted my mentee to understand: finding problems worth solving isn’t separate from the rest of the job. It comes from staying engaged with people’s work long enough to see what no single request can show you. These workarounds used bookmarklets to run JavaScript against Perfetto’s internal UI APIs.  ↩︎ The original proposal was to use a transparent cache for repeated queries and faster reopening of large traces. As I worked through it, I realized repeated queries were better served by keeping sessions warm in memory, whereas reopening was better served by explicitly exporting a trace into a format designed to load quickly. A transparent disk cache could also retain multi-gigabyte files without the user realizing and would need a new system to manage their lifetime. The proposal was ultimately replaced by warm sessions and streaming table export .  ↩︎ These workarounds used bookmarklets to run JavaScript against Perfetto’s internal UI APIs.  ↩︎ The original proposal was to use a transparent cache for repeated queries and faster reopening of large traces. As I worked through it, I realized repeated queries were better served by keeping sessions warm in memory, whereas reopening was better served by explicitly exporting a trace into a format designed to load quickly. A transparent disk cache could also retain multi-gigabyte files without the user realizing and would need a new system to manage their lifetime. The proposal was ultimately replaced by warm sessions and streaming table export .  ↩︎

0 views

Premium: The Hater’s Guide To Oracle (Part 2)

Good morning premium subscribers! As ever, please ping me at [email protected] if you have any questions. Oracle has one of the strongest mythologies in the tech industry. Ask a regular person and they’ll tell you that it’s “incredibly profitable” and “growing fast,” that it’s “unstoppable,” and that Larry Ellison has the mandate of heaven with regard to the continual sales of software and hardware related to databases and AI. And those people are completely and utterly wrong. The original title of this article was “Is Oracle Dying?” because I assume, when I took a deeper look, that there’d be some sort of debate , some sort of bull case for a decades-old quasi-hyperscaler run by one of the more nakedly-evil CEOs in the history of tech. I assumed — incorrectly, I might add — that Oracle as a business was doing fine other than the ridiculous commitments it made to support the whims of Sam Altman and OpenAI via deals that I believed (and still believe) will kill Oracle . Except it turns out that Oracle has already been on a death spiral for the best part of a decade (if not longer) and has only survived this long by screwing its customers, taking on masses of debt, and — most importantly — more than $85 billion in acquisitions over the last 23 years. Pretty much every major product line outside of databases is a hodge-podge of other people’s innovation stapled together with a legendary contempt for the customer . These acquisitions (and continual price increases ) are the only thing keeping the reaper from Oracle’s door other than margin-destroying GPUs . And that’s why Oracle’s revenue looks like this : After April 2009’s $5.7 billion acquisition of Sun Microsystems , Oracle’s revenues barely kept pace with inflation until December 2021’s $28.3 billion acquisition of Cerner allowed it to create Oracle Health , adding about $6 billion in annual revenue that had 40% lower margins (about 21.7%) than Oracle’s other businesses , though Oracle immediately started closing offices and brutal layoffs to try and bring them up. And as I mentioned above, Oracle’s other plan was to sink a little over $99 billion in capital expenditures since the middle of calendar year 2020 into AI GPUs.  Anyway, let’s see what that’s done to margins - OH MY GOD ! Oracle is a decades-long mission to keep reapplying lipstick to a pig. Billions of dollars of acquisitions have, for the most part, only succeeded in keeping the company’s revenue growth from going negative, and as noted by forensic accountant Howard M. Schilit , this is one of the most well-documented cases of accounting shenanigans being used to cover up that a business is in decline. Today’s newsletter is a sequel to the Hater’s Guide To Oracle , where I told the sordid tale of how Larry Ellison grew a massive, lucrative business out of a database business that one reporter once told me was a “ law firm with a database company attached ,” an Enterprise Resource Planning (ERP) product that competes with SAP to create the most-annoying way to run a large company, and a business built around licensing Java that exists mostly to email people and say “you need to pay us for Java or we’ll sue you.”  Then, as I’ve mentioned, there’s Oracle’s cloud infrastructure business, a decade-old also-ran that was meant to compete with Microsoft Azure and Amazon Web Services, but only managed to catch up following the advent of AI GPUs and a movement where all it took to party was buying billions of GPUs and saying “gosh darn, we love AI.” I originally started drafting this as a much tamer piece where I’d ask whether Oracle was dying, but as my editor and I started digging into the research, it became obvious that not only is Oracle dying , it’s been dying for years , kept alive through decades of acquisitions and a desperate and dangerous commitment to generative AI. And AI, I believe, will be what eventually kills Oracle dead.  In the past, all Oracle had to do to survive was buy somebody else’s company and replace its flagging revenues with theirs, turning the screws on their customers and laying off as many people as necessary to balance the books. While chaotic and decaying, Oracle’s empire has kept above water by never overextending itself, always keeping a positive free cashflow , and generally avoiding buying into industry hype cycles outside of whatever SaaS vehicle might potentially plug the gap in its earnings.  Yet with AI, Oracle broke its long-standing trend of letting someone else figure out the innovation, choosing instead to build its own cloud infrastructure, spending more in capex in its last fiscal year ( $55.6 billion ) than it did in the previous nine years combined ($50.4 billion), tripling its debt from $56.91 billion in FY2017 to $167.4 billion in FY2026, a year that ended with its free cashflow sitting at negative $23.69 billion.   For comparison, Oracle has had positive free cashflow every single year since 2001, including the Great Financial Crisis and COVID. Oracle has doomed itself with its commitment to the AI bubble. It has committed to building 7.1GW of data center capacity for one company — OpenAI — as part of a $300 billion, five-year-long contract that requires it to build an impossible amount of capacity in an impossible period of time for a client that could never afford the $70 billion or more in annual costs to make any of it worth it.  Today I am going to talk trash on what I consider to be one of the single-worst companies in the tech industry that’s survived only through financial engineering and never, ever overextending itself.  With revenue plateauing and customers in revolt, Oracle’s future already looked murky, but with the power of AI — and $95 billion in FY2027 capex — it’s becoming increasingly clear that this may be Larry Ellison’s last dance with Silicon Valley. This is the Hater’s Guide To Oracle Part 2, or AIpoaclypse Now.

0 views