Posts in Javascript (20 found)
Stone Tools 2 days ago

Bank Street Writer on the Apple II

Stop me if you're heard this one . In 1978, a young man wandered into a Tandy Radio Shack and found himself transfixed by the TRS-80 systems on display. He bought one just to play around with, and it wound up transforming his life from there on. As it went with so many, so too did it go with lawyer Doug Carlston. His brother, Gary, initially unimpressed, warmed up to the machine during a long Maine winter. The two thus smitten mused, "Can we make money off of this?" Together they formed a developer-sales relationship, with Doug developing Galactic Saga and third brother Don developing Tank Command . Gary's sales acumen brought early success and Broderbund was officially underway. Meanwhile in New York, Richard Ruopp, president of Bank Street College of Education, a kind of research center for experimental and progressive education, was thinking about how emerging technology fit into the college's mission. Writing was an important part of their curriculum, but according to Ruopp , "We tested the available word processors and found we couldn’t use any of them." So, experts from Bank Street College worked closely with consultant Franklin Smith and software development firm Intentional Educations Inc. to build a better word processor for kids. The fruit of that labor, Bank Street Writer , was published by Scholastic exclusively to schools at first, with Broderbund taking up the home distribution market a little later. Bank Street Writer would dominate home software sales charts for years and its name would live on as one of the sacred texts, like Lemonade Stand or The Oregon Trail . Let's see what lessons there are to learn from it yet. 1916 Founded by Lucy Sprague Mitchell, Wesley Mitchell, and Harriet Johnson as the “Bureau of Educational Experiments” (BEE) with the goal of understanding in what environment children best learn and develop, and to help adults learn to cultivate that environment. 1930 BEE moves to 69 Bank Street. (Will move to 112th Street in 1971, for space reasons.) 1937 The Writer’s Lab, which connects writers and students, is formed. 1950 BEE is renamed to Bank Street College of Education. 1973 Minnesota Educational Computing Consortium (MECC) is founded. This group would later go on to produce The Oregon Trail . 1983 Bank Street Writer, developed by Intentional Educations Inc., published by Broderbund Software, and “thoroughly tested by the academics at Bank Street College of Education.” Price: $70. 1985 Writer is a success! Time to capitalize! Bank Street Speller $50, Bank Street Filer $50, Bank Street Mailer $50, Bank Street Music Writer $50, Bank Street Prewriter (published by Scholastic) $60. 1986 Bank Street Writer Plus $100. Bank Street Writer III (published by Scholastic) $90. It’s basically Plus with classroom-oriented additions, including a 20-column mode and additional teaching aides. 1987 Bank Street Storybook, $40. 1992 Bank Street Writer for the Macintosh (published by Scholastic) $130. Adds limited page layout options, Hypercard-style hypertext, clip art, punctuation checker, image import with text wrap, full color, sound support, “Classroom Publishing” of fliers and pamphlets, and electronic mail. With word processors, I want to give them a chance to present their best possible experience. I do put a little time into trying the baseline experience many would have had with the software during the height of its popularity. "Does the software still have utility today?" can only be fairly answered by giving the software a fighting chance. To that end, I've gifted myself a top-of-the-line (virtual) Apple //e running the last update to Writer , the Plus edition. You probably already know how to use Bank Street Writer Plus . You don't know you know, but you do know because you have familiarity with GUI menus and basic word processing skills. All you're lacking is an understanding of the vagaries of data storage and retrieval as necessitated by the hardware of the time, but once armed with that knowledge you could start using this program without touching the manual again. It really is as easy as the makers claim. The simplicity is driven by very a subtle, forward-thinking user interface. Of primary interest is the upper prompt area. The top 3 lines of the screen serve as an ever-present, contextual "here's the situation" helper. What's going on? What am I looking at? What options are available? How do I navigate this screen? How do I use this tool? Whatever you're doing, whatever menu option you've chosen, the prompt area is already displaying information about which actions are available right now in the current context . As the manual states, "When in doubt, look for instructions in the prompt area." The manual speaks truth. For some, the constant on-screen prompting could be a touch overbearing, but I personally don't think it's so terrible to know that the program is paying attention to my actions and wants me to succeed. The assistance isn't front-loaded, like so many mobile apps, nor does it interrupt, like Clippy. I simply can't fault the good intentions, nor can I really think of anything in modern software that takes this approach to user-friendliness. The remainder of the screen is devoted to your writing and works like any other word processor you've used. Just type, move the cursor with the arrow keys, and type some more. I think most writers will find it behaves "as expected." There are no Electric Pencil -style over-type surprises, nor VisiCalc -style arrow key manipulations. What seems to have happened is that in making a word processor that is easy for children to use, they accidentally made a word processor that is just plain easy. The basic functionality is drop-dead simple to pick up by just poking around, but there's quite a bit more to learn here. To do so, we have a few options for getting to know Bank Street Writer in more detail. There are two manuals by virtue of the program's educational roots. Bank Street Writer was published by both Broderbund (for the home market) and Scholastic (for schools). Each tailored their own manual to their respective demographic. Broderbund's manual is cleanly designed, easy to understand, and gets right to the point. It is not as "child focused" as reviews at the time might have you believe. Scholastic's is more of a curriculum to teach word processing, part of the 80s push for "computers in the classroom." It's packed with student activities, pages that can be copied and distributed, and (tellingly) information for the teacher explaining "What is a word processor?" Our other option for learning is on side 2 of the main program disk. Quite apart from the program proper, the disk contains an interactive tutorial. I love this commitment to the user's success, though I breezed through it in just a few minutes, being a cultured word processing pro of the 21st century. I am quite familiar with "menus" thank you very much. As I mentioned at the top, the screen is split into two areas: prompt and writing. The prompt area is fixed, and can neither be hidden nor turned off. This means there's no "full screen" option, for example. The writing area runs in high-res graphics mode so as to bless us with the gift of an 80-character wide display. Being a graphics display also means the developer could have put anything on screen, including a ruler which would have been a nice formatting helper. Alas. Bank Street offers limited preference settings; there's not much we can do to customize the program's display or functionality. The upshot is that as I gain confidence with the program, the program doesn't offer to match my ability. There is one notable trick, which I'll discuss later, but overall there is a missed opportunity here for adapting to a user's increasing skill. Kids do grow up, after all. As with Electric Pencil , I'm writing this entirely in Bank Street Writer . Unlike the keyboard/software troubles there, here in 128K Apple //e world I have Markdown luxuries like . The emulator's amber mode is soothing to the eyes and soul. Mouse control is turned on and works perfectly, though it's much easier and faster to navigate by keyboard, as God intended. This is an enjoyable writing experience. Which is not to say the program is without quirks. Perhaps the most unfortunate one is how little writing space 128K RAM buys for a document. At this point in the write-up I'm at about 1,500 words and BSW's memory check function reports I'm already at 40% of capacity. So the largest document one could keep resident in memory at one time would run about 4,000 words max? Put bluntly, that ain't a lot. Splitting documents into multiple files is pretty much forced upon anyone wanting to write anything of length. Given floppy disk fragility, especially with children handling them, perhaps that's not such a bad idea. However, from an editing point of view, it is frustrating to recall which document I need to load to review any given piece of text. Remember also, there's no copy/paste as we understand it today. Moving a block of text between documents is tricky, but possible. BSW can save a selected portion of text to its own file, which can then be "retrieved" (inserted) at the current cursor position in another file. In this way the diskette functions as a memory buffer for cross-document "copy/paste." Hey, at least there is some option available. Flipping through old magazines of the time, it's interesting just how often Bank Street Writer comes up as the comparative reference point for home word processors over the years. If a new program had even the slightest whiff of trying to be "easy to use" it was invariably compared to Bank Street Writer . Likewise, there were any number of writers and readers of those magazines talking about how they continued to use Bank Street Writer , even though so-called "better" options existed. I don't want to oversell its adoption by adults, but it most definitely was not a children-only word processor, by any stretch. I think the release of Plus embraced a more mature audience. In schools it reigned supreme for years, including the Scholastic-branded version of Plus called Bank Street Writer III . There were add-on "packs" of teacher materials for use with it. There was also Bank Street Prewriter , a tool for helping to organize themes and thoughts before committing to the act of writing, including an outliner, as popularized by ThinkTank . (always interesting when influences ripple through the industry like this) Of course, the Scholastic approach was built around the idea of teachers having access to computers in the classroom. And THAT was build on the idea of teachers feeling comfortable enough with computers to seamlessly merge them into a lesson-plan. Sure, the kids needed something simple to learn, but let's be honest, so did the adults. There was a time when attaching a computer to anything meant a fundamental transformation of that thing was assured and imminent. For example, the "office of the future" (as discussed in the Superbase post ) had a counterpart in the "classroom of tomorrow." In 1983, Popular Computing said, "Schools are in the grip of a computer mania." Steve Jobs took advantage of this, skating to where the puck would be, by donating Apple 2s to California schools. In October 1983, Creative Computing did a little math on that plan. $20M in retail donations brought $4M in tax credits against $5M in gross donations. Apple could donate a computer to every elementary, middle, and high school in California for an outlay of only $1M. Jobs lobbied Congress hard to pass a national version of the same "Kids Can't Wait" bill, which would have extended federal tax credits for such donations. That never made it to law, for various political reasons. But the California initiative certainly helped position Apple as the go-to system for computers in education. By 1985, Apple would dominate fully half of the education market. That would continue into the Macintosh era, though Apple's dominance diminished slowly as cheaper, "good enough" alternatives entered the market. Today, Apple is #3 in the education market, behind Windows and Chromebooks . It is a fair question to ask, "How useful could a single donated computer be to a school?" Once it's in place, then what? Does it have function? Does anyone have a plan for it? Come to think of it, does anyone on staff even know how to use it? When Apple put a computer into (almost) every school in California, they did require training. Well, let's say lip-service was paid to the idea of the aspiration of training. One teacher from each school had to receive one day's worth of training to attain a certificate which allowed the school to receive the computer. That teacher was then tasked with training their coworkers. Wait, did I say "one day?" Sorry, I meant about one HOUR of training. It's not too hard to see where Larry Cuban was coming from when he published Oversold & Underused: Computers in the Classroom in 2001. Even of schools with more than a single system, he notes, "Why, then, does a school's high access (to computers) yield limited use? Nationally and in our case studies, teachers... mentioned that training in relevant software and applications was seldom offered... (Teachers) felt that the generic training available was often irrelevant to their specific and immediate needs." From my perspective, and I'm no historian, it seems to me there were four ways computers were introduced into the school setting. The three most obvious were: I personally attended schools of all three types. What I can say the schools had in common was how little attention, if any, was given to the computer and how little my teachers understood them. An impromptu poll of friends aligned with my own experience. Schools didn't integrate computers into classwork, except when classwork was explicitly about computers. I sincerely doubt my time playing Trillium's Shadowkeep during recess was anything close to Apple's vision of a "classroom of tomorrow." The fourth approach to computers into the classroom was significantly more ambitious. Apple tried an experiment in which five public school sites were chosen for a long-term research project. In 1986, the sites were given computers for every child in class and at home. They reasoned that for computers to truly make an impact on children, the computer couldn't just be a fun toy they occasionally interacted with. Rather, it required full integration into their lives. Now, it is darkly funny to me that having achieved this integration today through smartphones, adults work hard to remove computers from school. It is also interesting to me that Apple kind of led the way in making that happen, although in fairness they don't seem to consider the iPhone to be a computer . America wasn't alone in trying to give its children a technological leg up. In England, the BBC spearheaded a major drive to get computers into classrooms via a countrywide computer literacy program. Even in the States, I remember watching episodes of BBC's The Computer Programme on PBS. Regardless of Apple's or the BBC's efforts, the long-term data on the effectiveness of computers in the classroom has been mixed, at best, or even an outright failure. Apple's own assessment of their "Apple Classrooms of Tomorrow" (ACOT) program after a couple of years concluded, "Results showed that ACOT students maintained their performance levels on standard measures of educational achievement in basic skills, and they sustained positive attitudes as judged by measures addressing the traditional activities of schooling." Which is a "we continue to maintain the dream of selling more computers to schools" way of saying, "Nothing changed." In 2001, the BBC reported , "England's schools are beginning to use computers more in teaching - but teachers are making "slow progress" in learning about them." Then in 2015 the results were "disappointing, "Even where computers are used in the classroom, their impact on student performance is mixed at best." Informatique pour tous, France 1985: Pedagogy, Industry and Politics by Clémence Cardon-Quint noted the French attempt at computers in the classroom as being, "an operation that can be considered both as a milestone and a failure." Computers in the Classrooms of an Authoritarian Country: The Case of Soviet Latvia (1980s–1991) by Iveta Kestere, Katrina Elizabete Purina-Bieza shows the introduction of computers to have drawn stark power and social divides, while pushing prescribed gender roles of computers being "for boys." Teachers Translating and Circumventing the Computer in Lower and Upper Secondary Swedish Schools in the 1970s and 1980 s by Rosalía Guerrero Cantarell noted, "the role of teachers as agents of change was crucial. But teachers also acted as opponents, hindering the diffusion of computer use in schools." Now, I should be clear that things were different in the higher education market, as with PLATO in the universities. But in the primary and secondary markets, Bank Street Writer 's primary demographic, nobody really knew what to do with the machines once they had them. The most straightforwardly damning assessment is from Oversold & Underused where Cuban says in the chapter "Are Computers in Schools Worth the Investment?", "Although promoters of new technologies often spout the rhetoric of fundamental change, few have pursued deep and comprehensive changes in the existing system of schooling." Throughout the book he notes how most teachers struggle to integrate computers into their lessons and teaching methodologies. The lack of guidance in developing new ways of teaching means computers will continue to be relegated to occasional auxiliary tools trotted out from time to time, not integral to the teaching process. "Should my conclusions and predictions be accurate, both champions and skeptics will be disappointed. They may conclude, as I have, that the investment of billions of dollars over the last decade has yet to produce worthy outcomes," he concludes. Thanks to my sweet four-drive virtual machine, I can summon both the dictionary and thesaurus immediately. Put the cursor at the start of a word and hit or to get an instant spot check of spelling or synonyms. Without the reality of actual floppy disk access speed, word searches are fast. Spelling can be performed on the full document, which does take noticeable time to finish. One thing I really love is how cancelling an action or moving forward on the next step of a process is responsive and immediate. If you're growing bored of an action taking too long, just cancel it with ; it will stop immediately . The program feels robust and unbreakable in that way. There is a word lookup, which accepts wildcards, for when you kinda-sorta know how to spell a word but need help. Attached to this function is an anagram checker which benefits greatly from a virtual CPU boost. But it can only do its trick on single words, not phrases. Earlier I mentioned how little the program offers a user who has gained confidence and skill. That's not entirely accurate, thanks to its most surprising super power: macros. Yes, you read that right. This word processor designed for children includes macros. They are stored at the application level, not the document level, so do keep that in mind. Twenty can be defined, each consisting of up to 32 keystrokes. Running keystrokes in a macro is functionally identical to typing by hand. Because the program can be driven 100% by keyboard alone, macros can trigger menu selections and step through tedious parts of those commands. For example, to save our document periodically we need to do the following every time: That looks like a job for to me. 0:00 / 0:23 1× Defining a macro to save, with overwrite, the current file. After it is defined, I execute it which happens very quickly in the emulator. Watch carefully. If you can perform an action through a series of discrete keyboard commands, you can make a macro from it. This is freeing, but also works to highlight what you cannot do with the program. For example, there is no concept of an active selection, so a word is the smallest unit you can directly manipulate due to keyboard control limitations. It's not nothin' but it's not quite enough. I started setting up markdown macros, so I could wrap the current word in or for italic and bold. Doing the actions in the writing area and noting the minimal steps necessary to achieve the desired outcome translated into perfect macros. I was even able to make a kind of rudimentary "undo" for when I wrap something in italic but intended to use bold. This reminded me that I haven't touched macro functionality in modern apps since my AppleScript days. Lemme check something real quick. I've popped open LibreOffice and feel immediately put off by its Macros function. It looks super powerful; a full dedicated code editor with watched variables for authoring in its scripting language. Or is it languages? Is it Macros or ScriptForge? What are "Gimmicks?" Just what is going on? Google Docs is about the same, using Javascript for its "Apps Script" functionality. Here's a Stack Overflow post where someone wants to select text and set it to "blue and bold" with a keystroke and is presented with 32 lines of Javascript. Many programs seem to have taken a "make the simple things difficult, and the hard things possible" approach to macros. Microsoft Word reportedly has a "record" function for creating macros, which will watch what you do and let you play back those actions in sequence. (a la Adobe Photoshop's "actions") This sounds like a nice evolution of the BSW method. I say "reportedly" because it is not available in the online version and so I couldn't try it for myself without purchasing Microsoft 365. I certainly don't doubt the sky's the limit with these modern macro systems. I'm sure amazing utilities can be created, with custom dialog boxes, internet data retrieval, and more. The flip-side is that a lot of power has has been stripped from the writer and handed over to the programmer, which I think is unfortunate. Bank Street Writer allows an author to use the same keyboard commands for creating a macro as for writing a document. There is a forgotten lesson in that. Yes, BSW's macros are limited compared to modern tools, but they are immediately accessible and intuitive. They leverage skills the user is already known to possess . The learning curve is a straight, flat line. Like any good word processor, user-definable tab stops are possible. Bringing up the editor for tabs displays a ruler showing tab stops and their type (normal vs. decimal-aligned). Using the same tools for writing, the ruler is similarly editable. Just type a or a anywhere along the ruler. So, the lack of a ruler I noted at the beginning is now doubly-frustrating, because it exists! Perhaps it was determined to be too much visual clutter for younger users? Again, this is where the Options screen could have allowed advanced users to toggle on features as they grow in comfort and ambition. From what I can tell in the product catalogs, the only major revision after this was for the Macintosh which added a whole host of publishing features. If I think about my experience with BSW these past two weeks, and think about what my wish-list for a hypothetical update might be, "desktop publishing" has never crossed my mind. Having said all of that, I've really enjoyed using it to write this post. It has been solid, snappy, and utterly crash free. To be completely frank, when I switched over into LibreOffice , a predominantly native app for Windows, it felt laggy and sluggish. Bank Street Writer feels smooth and purpose-built, even in an emulator. Features are discoverable and the UI always makes it clear what action can be taken next. I never feel lost nor do I worry that an inadvertent action will have unknowable consequences. The impression of it being an assistant to my writing process is strong, probably more so than many modern word processors. This is cleanly illustrated by the prompt area which feels like a "good idea we forgot." (I also noted this in my ThinkTank examination) I cannot lavish such praise upon the original Bank Street Writer , only on this Plus revision. The original is 40-columns only, spell-checking is a completely separate program, no thesaurus, no macros, a kind of bizarre modal switch between writing/editing/transfer modes, no arrow key support, and other quirks of its time and target system (the original Apple 2). Plus is an incredibly smart update to that original, increasing its utility 10-fold, without sacrificing ease of use. In fact, it's actually easier to use, in my opinion than the original and comes just shy of being something I could use on a regular basis. Bank Street Writer is very good! But it's not quite great . Ways to improve the experience, notable deficiencies, workarounds, and notes about incorporating the software into modern workflows (if possible). AppleWin 32bit 1.31.0.0 on Windows 11 Emulating an Enhanced Apple //e Authentic machine speed (enhanced disk access speed) Monochrome (amber) for clean 80-column display Disk II controller in slot 5 (enables four floppies, total) Mouse interface in slot 4 Bank Street Writer Plus At the classroom level there are one or more computers. At the school level there is a "computer lab" with one or more systems. There were no computers. Hit (open the File menu) Hit (select Save File) Hit three times (stepping through default confirmation dialogs) I find that running at 300% CPU speed in AppleWin works great. No repeating key issues and the program is well-behaved. Spell check works quickly enough to not be annoying and I honestly enjoyed watching it work its way through the document. Sometimes there's something to be said about slowing the computer down to swift human-speed, to form a stronger sense of connection between your own work and the computer's work. I did mention that I used a 4-disk setup, but in truth I never really touched the thesaurus. A 3-disk setup is probably sufficient. The application never crashed; the emulator was rock-solid. CiderPress2 works perfectly for opening the files on an Apple ][ disk image. Files are of file extension, which CiderPress2 tries to open as disassembly, not text. Switch "Conversion" to "Plain Text" and you'll be fine. This is a program that would benefit greatly from one more revision. It's very close to being enough for a "minimalist" crowd. There are four, key pieces missing for completeness: Much longer document handling Smarter, expanded dictionary, with definitions Customizable UI, display/hide: prompts, ruler, word count, etc. Extra formatting options, like line spacing, visual centering, and so on. For a modern writer using hyperlinks, this can trip up the spell-checker quite ferociously. It doesn't understand, nor can it be taught, pattern-matching against URLs to skip them.

0 views
Kix Panganiban 2 days ago

Utteranc.es is really neat

It's hard to find privacy-respecting (read: not Disqus) commenting systems out there. A couple of good ones recommended by Bear are Cusdis and Komments -- but I'm not a huge fan of either of them: Then I realized that there's a great alternative that I've used in the past: utteranc.es . Its execution is elegant: you embed a tiny JS file on your blog posts, and it will map every page to Github Issues in a Github repo. In my case, I created this repo specifically for that purpose. Neat! I'm including utteranc.es in all my blog posts moving forward. You can check out how it looks below: Cusdis styling is very limited. You can only set it to dark or light mode, with no control over the specific HTML elements and styling. It's fine but I prefer something that looks a little neater. Komments requires manually creating a new page for every new post that you make. The idea is that wherever you want comments, you create a page in Komments and embed that page into your webpage. So you can have 1 Komments page per blog post, or even 1 Komments page for your entire blog.

0 views
Simon Willison 1 weeks ago

How I automate my Substack newsletter with content from my blog

I sent out my weekly-ish Substack newsletter this morning and took the opportunity to record a YouTube video demonstrating my process and describing the different components that make it work. There's a lot of digital duct tape involved, taking the content from Django+Heroku+PostgreSQL to GitHub Actions to SQLite+Datasette+Fly.io to JavaScript+Observable and finally to Substack. The core process is the same as I described back in 2023 . I have an Observable notebook called blog-to-newsletter which fetches content from my blog's database, filters out anything that has been in the newsletter before, formats what's left as HTML and offers a big "Copy rich text newsletter to clipboard" button. I click that button, paste the result into the Substack editor, tweak a few things and hit send. The whole process usually takes just a few minutes. I make very minor edits: That's the whole process! The most important cell in the Observable notebook is this one: This uses the JavaScript function to pull data from my blog's Datasette instance, using a very complex SQL query that is composed elsewhere in the notebook. Here's a link to see and execute that query directly in Datasette. It's 143 lines of convoluted SQL that assembles most of the HTML for the newsletter using SQLite string concatenation! An illustrative snippet: My blog's URLs look like - this SQL constructs that three letter month abbreviation from the month number using a substring operation. This is a terrible way to assemble HTML, but I've stuck with it because it amuses me. The rest of the Observable notebook takes that data, filters out anything that links to content mentioned in the previous newsletters and composes it into a block of HTML that can be copied using that big button. Here's the recipe it uses to turn HTML into rich text content on a clipboard suitable for Substack. I can't remember how I figured this out but it's very effective: My blog itself is a Django application hosted on Heroku, with data stored in Heroku PostgreSQL. Here's the source code for that Django application . I use the Django admin as my CMS. Datasette provides a JSON API over a SQLite database... which means something needs to convert that PostgreSQL database into a SQLite database that Datasette can use. My system for doing that lives in the simonw/simonwillisonblog-backup GitHub repository. It uses GitHub Actions on a schedule that executes every two hours, fetching the latest data from PostgreSQL and converting that to SQLite. My db-to-sqlite tool is responsible for that conversion. I call it like this : That command uses Heroku credentials in an environment variable to fetch the database connection URL for my blog's PostgreSQL database (and fixes a small difference in the URL scheme). can then export that data and write it to a SQLite database file called . The options specify the tables that should be included in the export. The repository does more than just that conversion: it also exports the resulting data to JSON files that live in the repository, which gives me a commit history of changes I make to my content. This is a cheap way to get a revision history of my blog content without having to mess around with detailed history tracking inside the Django application itself. At the end of my GitHub Actions workflow is this code that publishes the resulting database to Datasette running on Fly.io using the datasette publish fly plugin: As you can see, there are a lot of moving parts! Surprisingly it all mostly just works - I rarely have to intervene in the process, and the cost of those different components is pleasantly low. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . The core process is the same as I described back in 2023 . I have an Observable notebook called blog-to-newsletter which fetches content from my blog's database, filters out anything that has been in the newsletter before, formats what's left as HTML and offers a big "Copy rich text newsletter to clipboard" button. I click that button, paste the result into the Substack editor, tweak a few things and hit send. The whole process usually takes just a few minutes. I make very minor edits: I set the title and the subheading for the newsletter. This is often a direct copy of the title of the featured blog post. Substack turns YouTube URLs into embeds, which often isn't what I want - especially if I have a YouTube URL inside a code example. Blocks of preformatted text often have an extra blank line at the end, which I remove. Occasionally I'll make a content edit - removing a piece of content that doesn't fit the newsletter, or fixing a time reference like "yesterday" that doesn't make sense any more. I pick the featured image for the newsletter and add some tags.

0 views
JSLegendDev 1 weeks ago

Making a Small RPG

I’ve always wanted to try my hand making an RPG but always assumed it would take too much time. However, I didn’t want to give up before trying so I started to think of ways I could still make something compelling in 1-2 months. To help me come up with something, I decided to look into older RPGs as I had a hunch they could teach me a lot about scoping because back in the 80s, games were small because of technical limitations. A game that particularly caught my attention was the first Dragon Quest. This game was very important because it popularized the RPG genre in Japan by simplifying the formula therefore, making it more accessible. It can be considered the father of the JRPG sub-genre. What caught my attention was the simplicity of the game. There were no party members, the battle system was turn based and simple and you were free to just explore around. I was particularly surprised by how the game could give a sense of exploration while the map was technically very small. This was achieved by making the player move on an overworld map with a different scale proportion compared to when navigating towns and points of interest. In the overworld section, the player appeared bigger while the geography was smaller, allowing players to cover large amounts of territory relatively quickly. The advantage of this was that you could switch between biomes quickly without it feeling jarring. You still had the impression of traversing a large world despite being small in reality. This idea of using an overworld map was common in older games but somehow died off as devs had less and less technical limitations and more budget to work with. Seeing its potential, I decided that I would include one in my project even if I didn’t have a clear vision at this point. Playing Dragon Quest 1 also reminded me of how annoying random battle encounters were. You would take a few steps and get assaulted by an enemy of some kind. At the same time, this mechanic was needed, because grinding was necessary to be able to face stronger enemies in further zones of the map. My solution : What if instead of getting assaulted, you were the one doing the assault? As you would move on the map, encounter opportunities signified by a star would appear. Only if you went there and overlapped with one would a battle start. This gave the player agency to determine if they needed to battle or not. This idea seemed so appealing that I knew I needed to include it in my project. While my vision on what I wanted to make started to become clearer, I also started to get a sense of what I didn’t want to make. The idea of including a traditional turn based battle system was unappealing. That wasn’t because I hated this type of gameplay, but ever since I made a 6 hour tutorial on how to build one , I realized how complicated pulling one off is. Sure, you can get something basic quickly, but to actually make it engaging and well balanced is another story. A story that would exceed 1-2 months to deal with. I needed to opt for something more real-time and action based if I wanted to complete this project in a reasonable time frame. Back in 2015, an RPG that would prove to be very influential released and “broke the internet”. It was impossible to avoid seeing the mention of Undertale online. It was absolutely everywhere. The game received praised for a lot of different aspects but what held my attention, was its combat system. It was the first game I was aware of, that included a section of combat dedicated to avoiding projectiles (otherwise known as bullet hell) in a turn based battle system. This made the combat more action oriented which translated into something very engaging and fun. This type of gameplay left a strong impression in my mind and I thought that making something similar would be a better fit for my project as it was simpler to implement. While learning about Dragon Quest 1, I couldn’t help but be reminded me of The Legend of Zelda Breath of The Wild released in 2017. Similarly to Dragon Quest, a lot of freedom was granted to the player in how and when they tackled the game’s objectives. For example, in Breath of The Wild, you could go straight to the final boss after the tutorial section. I wanted to take this aspect of the game and incorporate it into my project. I felt it would be better to have one final boss and every other enemy encounter would be optional preparation you could engage with to get stronger. This felt like something that was achievable in a smaller scope compared to crafting a linear story the player would progress through. Another game that inspired me was Elden Ring, an open world action RPG similar to Breath of The Wild in its world structure but with the DNA of Dark Souls, a trilogy of games made previously by the same developers. What stuck with me regarding Elden Ring, for the purpose of my project, was its unique way it handled experience points. It was the first RPG I played that used them as a currency you could spend to level up different attributes making up your character or to buy items. Taking inspiration from it, I decided that my project would feature individually upgradable stats and that experience points would act as a currency. The idea was that the player would gain an amount of the game’s currency after battle and use that to upgrade different attributes. Like in Elden Ring, if you died in combat you would lose all currency you were currently holding. I needed a system like this for my project to count as an RPG. Since by definition an RPG is stats driven. A system like this would also allow the player to manage difficulty more easily and it would act as the progression system of my game. When I started getting into game development, I quickly came across Pico-8. Pico-8, for those unaware, is a fantasy console with a set of limitations. It’s not a console you buy physically but rather a software program that runs on your computer (or in a web browser) that mimics an older console that never existed. To put it simply, it was like running an emulator for a console that could’ve existed but never actually did. Hence the fantasy aspect of it. Pico-8 includes everything you need to make games. It has a built-in code editor, sprite editor, map editor, sound editor, etc… It uses the approachable Lua programming language which is similar to Python. Since Pico-8 is limited, it’s easier to actually finish making a game rather than being caught in scope creep. One game made in Pico-8 particularly caught my interest. In this game you play as a little character on a grid. Your goal is to fight just one boss. To attack this boss, you need to step on a glowing tile while avoiding taking damage by incoming obstacles and projectiles thrown at you. ( Epilepsy Warning regarding the game footage below due to the usage of flashing bright colors.) This game convinced me to ditch the turned based aspect I envisioned for my project entirely. Rather than having bullet hell sections within a turn based system like in Undertale the whole battle would instead be bullet hell. I could make the player attack without needing to have turns by making attack zones spawn within the battlefield. The player would then need to collide with them for an attack to register. I was now convinced that I had something to stand on. It was now time to see if it would work in practice but I needed to clearly formulate my vision first. The game I had in mind would take place under two main scenes. The first, was the overworld in which the player moved around and could engage in battle encounters, lore encounters, heal or upgrade their stats. The second, being the battle scene, would be were battles would take place. The player would be represented by a cursor and they were expected to move around dodging incoming attacks while seeking to collide with attack zones to deal damage to the enemy. The purpose of the game was to defeat a single final boss named king Donovan who was a tyrant ruling over the land of Hydralia where the game took place. At any point, the player could enter the castle to face the final boss immediately. However, most likely, the boss would be too strong. To prepare, the player would roam around the world engaging in various battle encounters. Depending on where the encounter was triggered, a different enemy would show up that fitted the theme of the location they were in. The enemy’s difficulty and experience reward if beaten would drastically vary depending on the location. Finally, the player could level up and heal in a village. I was now ready to start programming the game and figuring out the details as I went along. For this purpose, I decided to write the game using the JavaScript programming language and the KAPLAY game library. I chose these tools because they were what I was most familiar with. For JavaScript, I knew the language before getting into game dev as I previously worked as a software developer for a company who’s product was a complex web application. While most of the code was in TypeScript, knowing JavaScript was pretty much necessary to work in TypeScript since the language is a superset of JavaScript. As an aside, despite its flaws as a language, JavaScript is an extremely empowering language to know as a solo dev. You can make games, websites, web apps, browser extensions, desktop apps, mobile apps, server side apps, etc… with this one language. It’s like the English of programming languages. Not perfect, but highly useful in today’s world. I’ll just caveat that using JavaScript makes sense for 2D games and light 3D games. For anything more advanced, you’d be better off using Unreal, Unity or Godot. As for the KAPLAY game library, it allows me to make games quickly because it provides a lot of functionality out of the box. It’s also very easy to learn. While it’s relatively easy to package a JavaScript game as an app that can be put on Steam, what about consoles? Well it’s not straightforward at all but at the same time, I don’t really care about consoles unless my game is a smash hit on Steam. If my game does become very successful than it would make sense businesswise to pay a porting company to remake the game for consoles, getting devkits, dealing with optimizations and all the complexity that comes with publishing a game on these platforms. Anyway, to start off the game’s development, I decided to implement the battle scene first with all of its related mechanics as I needed to make sure the battle system I had in mind was fun to play in practice. To also save time later down the line, I figured that I would make the game have a square aspect ratio. This would allow me to save time during asset creation, especially for the map as I wanted the whole map to be visible at once as I wouldn’t use a scrolling camera for this game. After a while, I had a first “bare bones” version of the battle system. You could move around to avoid projectiles and attack the enemy by colliding with red attack zones. Initially, I wanted the player to have many stats they could upgrade. They could upgrade their health (HP), speed, attack power and FP which stood for focus points. However, I had to axe the FP stat as I originally wanted to use it as a way to introduce a cost to using items in battle. However, I gave up on the idea of making items entirely as they would require too much time to create and properly balance. I also had the idea of adding a stamina mechanic similar to the one you see in Elden Ring. Moving around would consume stamina that could only replenish when you stopped moving. I initially though that this would result in fun gameplay as you could upgrade your stamina over time but it ended up being very tedious and useless. Therefore, I also ended up removing it. Now that the battle system was mostly done, I decided to work on the world scene where the player could move around. I first implemented battle encounters that would spawn randomly on the screen as red squares, I then created the upgrade system allowing the player to upgrade between 3 stats : Their health (HP), attack power and speed. In this version of the game, the player could restore their health near where they could upgrade their stats. While working on the world scene was the focus, I also made a tweak to the battle scene. Instead of displaying the current amount of health left as a fraction, I decided a health bar would be necessary because when engaged in a fast paced battle, the player does not have time to interpret fractions to determine the state of their health. A health bar would convey the info faster in this context. However, I quickly noticed an issue with how health was restored in my game. Since the world was constrained to a single screen, it made going back to the center to get healed after every fight the optimal way to play. This resulted in feeling obligated to go back to the center rather than freely roaming around. To fix this issue, I made it so the player needed to pay to heal using the same currency for leveling up. Now you needed to carefully balance between healing or saving your experience currency for an upgrade by continuing to explore/engage in battle. All of this while keeping in mind that you could lose all of your currency if defeated in battle. It’s important to note that you could also heal partially which provided flexibility in how the player managed the currency resource. Now that I was satisfied with the “bare bones” state of the game, I needed to make nice looking graphics. To achieve this, I decided to go with a pixel art style. I could spend a lot of time explaining how to make good pixel art but, I already did so previously. I recommend checking my post on the topic. I started by putting a lot effort drawing the overworld map as the player would spend a lot of time in it. It was a this stage that I decided to make villages the places where you would heal or level up. To make this clearer, I added icons on top of each village to make it obvious what each was for. Now that I was satisfied with how the map turned out, I started designing and implementing the player character. For each distinct zone of the map, I added a collider so that battle encounters could determine which enemy and what background to display during battle. It was at this point that I made encounters appear as flashing stars on the map. Since my work on the overworld was done, I now needed to produce a variety of battle backgrounds to really immerse the player in the world. I sat down and locked in. These were by far one of the most time intensive art assets to make for this project but I’m happy with the results. After finishing making all backgrounds, I implemented the logic to show them in battle according to the zone where the encounter occurred. The next assets to make were enemies. This was another time intensive task but I’m happy with how they turned out. The character at the bottom left is king Donovan the main antagonist of the game. Further Developing The Battle Gameplay While developing the game, I noticed that it took too much time to go from one end of the battle zone to the other. This made the gameplay tedious so I decided to make the battle zone smaller. At this point, I also changed the player cursor to be diamond shaped and red rather than a circle and white. I also decided to use the same flashing star sprite used for encounters on the map but this time, for attack zones. I also decided to change the font used in the game to something better. At this point, the projectiles thrown towards the player didn’t move in a cohesive pattern the player could learn over time. It was also absolutely necessary to create a system in which the attack patterns of the enemy would be progressively shown to the player. This is why I stopped everything to work on the enemy’s attack pattern. I also, by the same token, started to add effects to make the battle more engaging and sprites for the projectiles. While the game was coming along nicely, I started to experience performance issues. I go into more detail in a previous post if you’re interested. To add another layer of depth to my game, I decided that the reward you got from a specific enemy encounter would not only depend on which enemy you were fighting but also how much damage you took. For example, if a basic enemy in the Hydralia field would give you a reward of a 100 after battle, you would actually get less unless you did not take damage during that battle. This was to encourage careful dodging of projectiles and to reward players who learned the enemy pattern thoroughly. This would also add replayability as there was now a purpose to fight the same enemy over and over again. The formula I used to determine the final reward granted can be described as follows : At this point, it wasn’t well communicated to the player how much of the base reward they were granted after battle. That’s why I added the “Excellence” indication. When beating an enemy, if done without taking damage, instead of having the usual “Foe Vanquished” message appearing on the screen, you would get a “Foe Vanquised With Excellence” message in bright Yellow. In addition to being able to enter into battle encounters, I wanted the player to have lore/tips encounters. Using the same system, I would randomly spawn a flashing star of a blueish-white color. If the player overlapped with it, a dialogue box would appear telling them some lore/tips related to the location they were in. Sometimes, these encounters would result in a chest containing exp currency reward. This was to give a reason for the player to pursue these encounters. This is still a work in progress, as I haven’t decided what kind of lore to express through these. One thing I forgot to show earlier was how I revamped the menu to use the new font. That’s all I have to share for now. What do you think? I also think it’s a good time to ask for advice regarding the game’s title. Since the game takes place in a land named Hydralia . I thought about using the same name for the game. However, since your mission is to defeat a tyrant king named Donovan, maybe a title like Hydralia : Donovan’s Demise would be a better fit. If you have any ideas regarding naming, feel free to leave a comment! Anyway, if you want to keep up with the game’s development or are more generally interested in game development, I recommend subscribing to not miss out on future posts. Subscribe now In the meantime, you can read the following :

0 views
Evan Hahn 1 weeks ago

Experiment: making TypeScript immutable-by-default

I like programming languages where variables are immutable by default. For example, in Rust , declares an immutable variable and declares a mutable one. I’ve long wanted this in other languages, like TypeScript, which is mutable by default—the opposite of what I want! I wondered: is it possible to make TypeScript values immutable by default? My goal was to do this purely with TypeScript, without changing TypeScript itself. That meant no lint rules or other tools. I chose this because I wanted this solution to be as “pure” as possible…and it also sounded more fun. I spent an evening trying to do this. I failed but made progress! I made arrays and s immutable by default, but I couldn’t get it working for regular objects. If you figure out how to do this completely, please contact me —I must know! TypeScript has built-in type definitions for JavaScript APIs like and and . If you’ve ever changed the or options in your TSConfig, you’ve tweaked which of these definitions are included. For example, you might add the “ES2024” library if you’re targeting a newer runtime. My goal was to swap the built-in libraries with an immutable-by-default replacement. The first step was to stop using any of the built-in libraries. I set the flag in my TSConfig, like this: Then I wrote a very simple script and put it in : When I ran , it gave a bunch of errors: Progress! I had successfully obliterated any default TypeScript libraries, which I could tell because it couldn’t find core types like or . Time to write the replacement. This project was a prototype. Therefore, I started with a minimal solution that would type-check. I didn’t need it to be good! I created and put the following inside: Now, when I ran , I got no errors! I’d defined all the built-in types that TypeScript needs, and a dummy object. As you can see, this solution is impractical for production. For one, none of these interfaces have any properties! isn’t defined, for example. That’s okay because this is only a prototype. A production-ready version would need to define all of those things—tedious, but should be straightforward. I decided to tackle this with a test-driven development style. I’d write some code that I want to type-check, watch it fail to type-check, then fix it. I updated to contain the following: This tests three things: When I ran , I saw two errors: So I updated the type in with the following: The property accessor—the line—tells TypeScript that you can access array properties by numeric index, but they’re read-only. That should make possible but impossible. The method definition is copied from the TypeScript source code with no changes (other than some auto-formatting). That should make it possible to call . Notice that I did not define . We shouldn’t be calling that on an immutable array! I ran again and…success! No errors! We now have immutable arrays! At this stage, I’ve shown that it’s possible to configure TypeScript to make all arrays immutable with no extra annotations . No need for or ! In other words, we have some immutability by default. This code, like everything in this post, is simplistic. There are lots of other array methods , like and and ! If this were made production-ready, I’d make sure to define all the read-only array methods . But for now, I was ready to move on to mutable arrays. I prefer immutability, but I want to be able to define a mutable array sometimes. So I made another test case: Notice that this requires a little extra work to make the array mutable. In other words, it’s not the default. TypeScript complained that it can’t find , so I defined it: And again, type-checks passed! Now, I had mutable and immutable arrays, with immutability as the default. Again, this is simplistic, but good enough for this proof-of-concept! This was exciting to me. It was possible to configure TypeScript to be immutable by default, for arrays at least. I didn’t have to fork the language or use any other tools. Could I make more things immutable? I wanted to see if I could go beyond arrays. My next target was the type, which is a TypeScript utility type . So I defined another pair of test cases similar to the ones I made for arrays: TypeScript complained that it couldn’t find or . It also complained about an unused , which meant that mutation was allowed. I rolled up my sleeves and fixed those errors like this: Now, we have , which is an immutable key-value pair, and the mutable version too. Just like arrays! You can imagine extending this idea to other built-in types, like and . I think it’d be pretty easy to do this the same way I did arrays and records. I’ll leave that as an exercise to the reader. My final test was to make regular objects (not records or arrays) immutable. Unfortunately for me, I could not figure this out. Here’s the test case I wrote: This stumped me. No matter what I did, I could not write a type that would disallow this mutation. I tried modifying the type every way I could think of, but came up short! There are ways to annotate to make it immutable, but that’s not in the spirit of my goal. I want it to be immutable by default! Alas, this is where I gave up. I wanted to make TypeScript immutable by default. I was able to do this with arrays, s, and other types like and . Unfortunately, I couldn’t make it work for plain object definitions like . There’s probably a way to enforce this with lint rules, either by disallowing mutation operations or by requiring annotations everywhere. I’d like to see what that looks like. If you figure out how to make TypeScript immutable by default with no other tools , I would love to know, and I’ll update my post. I hope my failed attempt will lead someone else to something successful. Again, please contact me if you figure this out, or have any other thoughts. Creating arrays with array literals is possible. Non-mutating operations, like and , are allowed. Operations that mutate the array, like , are disallowed. is allowed. There’s an unused there. doesn’t exist.

0 views
James O'Claire 2 weeks ago

Where are All the AI Generated Native Android & iOS Apps?

I’ve been wanting to keep an eye on AI generated apps to see what kind of SDKs and libraries they end up using for AppGoblin’s massive resource of 100k+ apps and SDKs. So for the past year I’ve clicked/followed a few companies with the goal downloading the actual apps that get created. As I did my most recent rounds checking the “Examples” and “Showcases” I think I can say these apps are just not being delivered in that final native form . To be clear there are lots of web app demos, but when it comes to actual apps landing on the Play and App Stores I’ve found few to no clear examples. Looking for links to native apps in the various company communities/Discords I see a few reasons: As I was writing, this I’m reminded that the ‘native’ term is a bit ambiguous. Most of these companies seem to do Expo/JS related apps into native apps. For this article, I’m just looking for anything that has an app on an app store, as I’d like to later breakdown the use of Expo or other tools. fastshot.ai -> YC Backed, newer so worth giving them some time. Showcase is mostly empty with only one screenshot so far on Discord. a0.dev -> The site has a dozen very high quality examples of web apps. The links to them is to download their own app though. The only examples on the App Stores seem to be from July by the developers. replit.com -> Dozens of real examples of web apps but I couldn’t find any that had Android or iOS apps made by replit. Replit was quite early, launching in February 2024 so I think if there’s any examples this might be the easiest to find. could.ai / dashwave.io / gobuildmy.app -> These ones all have pricing pages but don’t even have examples or demo or showcase pages. I think we’re all coming to see that AI coding tools are good for rapid prototyping but struggling to deliver magic results . That being said, people are using these tools, likely mostly as prototypes and later transitioning to managing the apps directly. If anyone has used these tools and wants to share their apps please reach out. You can also request SDK scans of apps on AppGoblin for free. People are building proof of concepts, and like any project, as you get closer to the reality, the dream gets a bit less rosy. People are getting real value by actually implementing their ideas into apps and thinking through the actual use case. People once finished their app, do not want to publish the fact that it was vibe coded or used a specific company to build. The last mile, from fully working demo to production is hard for AI .

0 views
Sean Goedecke 2 weeks ago

Writing for AIs is a good way to reach more humans

There’s an idea going around right now about “writing for AIs”: writing as if your primary audience is not human readers, but the language models that will be trained on the content of your posts. Why would anyone do this? For the same reason you might want to go on podcasts or engage in SEO: to get your core ideas in front of many more people than would read your posts directly. If you write to make money, writing for AI is counterproductive. Why would anyone buy your writing if they can get a reasonable facsimile of it for free out of ChatGPT? If you write in order to express yourself in poetry, writing for AI might seem repulsive. I certainly find language model attempts at poetry to be off-putting 1 . But if you write to spread ideas, I think writing for AI makes a lot of sense. I don’t write this blog to make money or to express myself in beautiful language. I write because I have specific things I want to say: While it’s nice that some people read about this stuff on my website, I would be just as happy if they read about them elsewhere: via word-of-mouth from people who’ve read my posts, in the Google preview banners, or in my email newsletter (where I include the entire post content so people never have to click through to the website at all). I give blanket permission to anyone who asks to translate and rehost my articles. Likewise, I would be just as happy for people to consume these ideas via talking with an LLM. In 2022, Scott Alexander wrote that the purpose of a book is not to produce the book itself. Instead, a book acts as a “ritual object”: a reason to hold a public relations campaign that aims to “burn a paragraph of text into the public consciousness” via TV interviews and magazine articles. Likewise, I think it’s fair to say that the purpose of a technical blog post is not to be read, but to be a ritual object that gives people a reason to discuss a single idea. Take Jeff Atwood’s well-known post that popularized Foote and Yoder’s original idea of the “big ball of mud”. Or Joel Spolsky’s The Law of Leaky Abstractions , which popularized the idea that “all abstractions leak”. Or Tanya Reilly’s Being Glue , which popularized the term “glue work” to describe the under-rewarded coordination work that holds teams together 2 . Many, many more people are familiar with these ideas and terms than have read the original posts. They have sunk into the public consciousness via repeated discussion in forums, Slack channels, and so on, in the same way that the broad idea of a non-fiction book sinks in via secondary sources 3 . Large language models do read all these books and blog posts. But what they read in much greater quantities is people talking about these books and blog posts (at least via other articles, if they’re not being explicitly trained on Hacker News and Reddit comments). If you write a popular blog, your ideas will thus be over-represented in the training data. For instance, when someone asks about coordination work, GPT-5 immediately calls it “glue work”: When engineers talk to language models about their work, I would like those models to be informed by my posts, either via web search or by inclusion in the training data. As models get better, I anticipate people using them more (for instance, via voice chat). That’s one reason why I’ve written so much this year: I want to get my foothold in the training data as early as possible, so my ideas can be better represented by language models long-term. Of course, there are other reasons why people might want to be represented in the training data. Scott Alexander lists three reasons: teaching AIs what you know, trying to convince AIs of what you believe, and providing enough personal information that the superintelligent future AI will be able to simulate you accurately later on. I’m really only moved by the first reason: teaching AIs what I believe so they can share my ideas with other human readers. I agree with Scott that writing in order to shape the personality of future AIs is pointless. It might just be impossible - for instance, maybe even the most persuasive person in the world can’t argue hard enough to outweigh millennia of training data, or maybe any future superintelligence will be too smart to be influenced by mere humans. If it does turn out to be possible, AI companies will likely take control of the process and deliberately convince their models of whatever set of beliefs are most useful to them 4 . Okay, suppose you’re convinced that it’s worth writing for AIs. What does that mean? Do you need to write any differently? I don’t think so. When I say “writing for AIs”, I mean: The first point is pretty self-explanatory: the more you write, the more of your content will be represented in future AI training data. So even if you aren’t getting a lot of human traffic in the short term, your long term reach will be much greater. Of course, you shouldn’t just put out a high volume of garbage for two reasons: first, because having your work shared and repeated by humans is likely to increase your footprint in the training set; and second, because what would be the point of increasing the reach of garbage? The second point is that putting writing behind a paywall means it’s going to be harder to train on. This is one reason why I’ve never considered having paid subscribers to my blog. Relatedly, I think it’s also worth avoiding fancy Javascript-only presentation which would make it harder to scrape your content - for instance, an infinite-scroll page, or a non-SSR-ed single-page-application. Tyler Cowen has famously suggested being nice to the AIs in an attempt to get them to pay more attention to your work. I don’t think this works, any more than being nice to Google results in your pages getting ranked higher in Google Search. AIs do not make conscious decisions about what to pull from their training data. They are influenced by each piece of data to the extent that (a) it’s represented in the training set, and (b) it aligns with the overall “personality” of the model. Neither of those things is likely to be affected by how pro-AI your writing is. I recommend just writing how you would normally write. Much of the “why write for AIs” discussion is dominated by far-future speculation, but there are much more straightforward reasons to write for AIs: for instance, so that they’ll help spread your ideas more broadly. I think this is a good reason to write more and to make your writing accessible (i.e. not behind a paywall). But I wouldn’t recommend changing your writing style to be more “AI-friendly”: we don’t have much reason to think that works, and if it makes your writing less appealing to humans it’s probably not a worthwhile tradeoff. There may be some particular writing style that’s appealing to AIs. It wouldn’t surprise me if we end up in a war between internet writers and AI labs, in the same way that SEO experts are at war with Google’s search team. I just don’t think we know what that writing style is yet. It’s slop in the truest sense. Technically this was a talk, not a blog post. I also wrote about glue work and why you should be wary of it in Glue work considered harmful . For instance, take my own Seeing like a software company , which expresses the idea of the book Seeing Like a State in the first three lines. There’s also the idea that by contributing to the training data, future AIs will be able to simulate you, providing a form of immortality. I find it tough to take this seriously. Or perhaps I’m just unwilling to pour my innermost heart out into my blog. Any future simulation of me from these posts would only capture a tiny fraction of my personality. That the fundamental nature of tech work has changed since like 2023 and the end of ZIRP That emotional regulation is at least as important as technical skill for engineering performance That large tech companies do not function by their written rules, but instead by complex networks of personal incentives That you should actually read the papers that drive engineering conversations, because they often say the exact opposite of how they’re popularly interpreted Writing more than you normally would, and Publishing your writing where it can be easily scraped for AI training data and discovered by AI web searchers It’s slop in the truest sense. ↩ Technically this was a talk, not a blog post. I also wrote about glue work and why you should be wary of it in Glue work considered harmful . ↩ For instance, take my own Seeing like a software company , which expresses the idea of the book Seeing Like a State in the first three lines. ↩ There’s also the idea that by contributing to the training data, future AIs will be able to simulate you, providing a form of immortality. I find it tough to take this seriously. Or perhaps I’m just unwilling to pour my innermost heart out into my blog. Any future simulation of me from these posts would only capture a tiny fraction of my personality. ↩

0 views
iDiallo 2 weeks ago

What Actually Defines a Stable Software Version?

As a developer, you'll hear these terms often: "stable software," "stable release," or "stable version." Intuitively, it just means you can rely on it. That's not entirely wrong, but when I was new to programming, I didn't truly grasp the technical meaning. For anyone learning, the initial, simple definition of "it works reliably" is a great starting point. But if you're building systems for the long haul, that definition is incomplete. The intuitive definition is: a stable version of software that works and that you can rely on not to crash. The technical definition is: a stable version of software where the API will not change unexpectedly in future updates. A stable version is essentially a guarantee from the developers that the core interface, such as the functions, class names, data structures, and overall architecture you interact with, will remain consistent throughout that version's lifecycle. This means that if your code works with version 1.0.0, it should also work flawlessly with version 1.0.1, 1.0.2, and 1.1.0. Future updates will focus on bug fixes, security patches, and performance improvements, not on introducing breaking changes that force you to rewrite your existing code. My initial misunderstanding was thinking stability was about whether the software was bug-free or not. Similar to how we expect bugs to be present in a beta version. But there was still an upside to this confusion. It helped me avoid the hype cycle, especially with certain JavaScript frameworks. I remember being hesitant to commit to new versions of certain tools (like early versions of React, Angular, though this is true of many fast-moving frameworks and SDKs). Paradigms would shift rapidly from one version to the next. A key concept I'd mastered one month would be deprecated or replaced the next. While those frameworks sit at the cutting edge of innovation, they can also be the antithesis of stability. Stability is about long-term commitment. Rapid shifts force users to constantly evolve with the framework, making it difficult to stay on a single version without continual, large-scale upgrades. A truly stable software version is one you can commit to for a significant amount of time. The classic example of stability is Python 2. Yes, I know many wanted it to die by fire, but it was first released in 2000 and remained active, receiving support and maintenance until its final update in 2020. That's two decades of stability! I really enjoyed being able to pick up old scripts and run them without any fuss. While I'm not advocating that every tool should last that long, I do think that when we're building APIs or stable software, we should adopt the mindset that this is the last version we'll ever make. This forces us to carefully consider the long-term design of our software. Whenever I see LTS (Long-Term Support) next to an application, I know that the maintainers have committed to supporting, maintaining, and keeping it backward compatible for a defined, extended period. That's when I know I'm working with both reliable and stable software.

0 views
Neil Madden 2 weeks ago

Were URLs a bad idea?

When I was writing Rating 26 years of Java changes , I started reflecting on the new HttpClient library in Java 11. The old way of fetching a URL was to use URL.openConnection() . This was intended to be a generic mechanism for retrieving the contents of any URL: files, web resources, FTP servers, etc. It was a pluggable mechanism that could, in theory, support any type of URL at all. This was the sort of thing that was considered a good idea back in the 90s/00s, but has a bunch of downsides: The new HttpClient in Java 11 is much better at doing HTTP, but it’s also specific to HTTP/HTTPS. And that seems like a good thing? In fact, in the vast majority of cases the uniformity of URLs is no longer a desirable aspect. Most apps and libraries are specialised to handle essentially a single type of URL, and are better off because of it. Are there still cases where it is genuinely useful to be able to accept a URL of any (or nearly any) scheme? Fetching different types of URLs can have wildly different security and performance implications, and wildly different failure cases. Do I really want to accept a mailto: URL or a javascript: “URL” ? No, never. The API was forced to be lowest-common-denominator, so if you wanted to set options that are specific to a particular protocol then you had to cast the return URLConnection to a more specific sub-class (and therefore lose generality).

0 views

Is this JS function pure?

In 2019, as functional programming was making the last inroads dethroning OOP, I kept hearing the mantra of “just use pure functions” in JS. Something didn’t sit right with me when talking very deterministically about pure functions in a large unpure language like JS. Especially, after seeing JS tooling perform optimizations based on a pure annotation (like webpack) . While everyone agrees is pure, I felt there are a lot of subtle situations where disagreements might arise. What is worse than terms with imprecise meaning is terms with imprecise meaning where people using them are not aware there is imprecision.

0 views
JSLegendDev 2 weeks ago

What Caused Performance Issues in My Tiny RPG

In a previous post, I mentioned having strange performance issues regarding a tiny RPG I was secretly working on. The crux of the matter was that the game (built using web technologies) would run noticeably less smoothly when wrapped as a desktop app on my machine than when running in Firefox. I initially shared the project’s executables on the KAPLAY Discord server (KAPLAY being the library I used to make the game in JavaScript) and none reported the performance issues I had. Seeing this, I decided to make a Substack post inviting my wider audience to try the game out and report its performance. This led to someone sharing the post on Hacker News which resulted in a large influx of views and feedback. In this post, I would like to explain what went wrong (as best as I understood it myself). First of all, I have mostly fixed the performance of my game. It now runs much more smoothly and I have updated the executables on itch. I would still greatly appreciate feedback regarding how these run on your machines. (Download the ones tagged with -v2) Here’s the link : https://jslegend.itch.io/small-rpg-performance-playtest To build executables for Windows, Mac and Linux, I use either NW.js or GemShell. Similar to Electron but easier to set up. It packages a Chromium instance for rendering. Results in bloated executables but the same rendering engine is used across platforms. Similar to Tauri in how it uses the operating system’s webview to render the app rather than packaging a Chromium instance. This results in leaner executables but different web engines with varying performance differences are used on different platforms. However, contrary to Tauri, GemShell is a one click tool that generate executables for all three platforms. I wrote a post about it recently that delved into more detail. Since I wanted to build executables quickly, I decided to try out GemShell for this project. Executables that I distributed to get feedback were made with this tool. KAPLAY is a library based on the concept of game objects and components. A game object is created from a list of components, many of which, are ready made and offer functionality out of the box. This makes the library very easy to understand and productive. The issue is that it’s by default less performant than other alternatives (Phaser, Excalibur, etc…) I use it because it’s so fast to prototype game ideas in. However, I used it for this project because it was the tool I knew best that would allow me to immediately start working on the game and focus on game design. In hindsight, I should have probably started invested in learning other more performant alternatives to a sufficient level so that I could easily pivot away if the needs of my project demanded it. This was often reported among people for who the game didn’t perform well. In KAPLAY, there is an option to cap the FPS to a maximum amount. It’s used when first initializing the library. The idea behind this option is to enforce a consistent frame rate resulting in more consistently smooth gameplay. However, setting this to 60 in my game resulted in the game eventually slowing down to a crawl all while the debug fps count still displayed 60 fps. I conclude that this happened when the machine running the game wasn’t able to sustain 60fps or more at all times. The fix was simple, remove it. Why did this option behave this way? Probably a bug a in the library. A maintainer is currently working on it. Since GemShell was used to make executables, the underlying web engine used to render the game on a Mac is Webkit and not Chromium. While I knew that Webkit was less performant than Chromium, I didn’t think it would be that noticeable. The apparent solution to this would be to move off of GemShell and use NW.js instead. However, it turns out that MacOS does certain things to not allow Webkit to use its full potential. The dev behind GemShell apparently has a fix for this situation. Below is what they shared on their Discord, which you can join if you’re interested in the tool here . This would be great since it would be nice to have the best of both worlds, more consistent performance across platforms while still having lean executables. Since I’m still not done with the development of the game, I can afford to let the dev cook, as we say! This is the strangest performance issue I experienced. While I could conceive of the game performing better on Firefox VS Safari (since it uses Webkit), I was caught off guard by Chrome performing poorly than Firefox. Chrome is supposed to be more performant due to the Chromium Web engine. The Chrome profiler seemed to indicate that the way I was rendering text in my game probably needed to change. KAPLAY Game Objects and Text Rendering In KAPLAY, if you look at the examples provided in its playground , you’ll find that it’s often shown that to render text you first, create a game object using a text component and then pass the text to that component. However, it turns out that this is inefficient since game objects have a greater cost in terms of performance. For rendering simple text it’s far better to draw it directly in the draw loop. Here’s a simple example. The same logic applies to drawing static images like backgrounds. Instead of doing : I needed to do : Additionally, to not disable batching it was important that all drawSprite calls be placed together in the draw loop before rendering text with drawText calls. Doing this led to better performance on Chrome and Safari. However, there was one last obvious performance improvement I needed to do. In the battle section of my game the player must avoid getting hit by a horde of projectiles thrown at them. A common optimization technique used in this case was to reuse bullets that leave the screen rather than creating new ones. This technique is known as object pooling. I had planned on implementing something like this eventually but didn’t do it since I was developing the game using Firefox as my preview, and the performance was great. There weren’t that many projectiles in the first place so I felt that this would be useful later. However, considering that Chrome and Safari were struggling performance wise probably due to their garbage collector working differently, I resigned myself to implement it now. As expected the game performed much better on both Chrome and Safari. For Chrome, I now had a constant 60fps on my machine (Macbook Air M3, 16 GB RAM) but for Safari it was more around 45-60fps. To stress test my pooling system, I added a bunch of projectiles. Here is footage. I’m happy that I can now resume development and focus more on game design. However, what this adventure taught me is that I should probably invest my time learning other frameworks and game engines. While I eventually ended up fixing the performance issues in my game, I can’t help but think of scenarios where problems could arise later that are unfixable due to limitations of the tools I’m using. In that case, I would have to halt development to learn a new tool at a proficient level on top of having to reimplement the game which would take a lot of time and result in my project getting significantly delayed. If I start learning something else right now, I can at least go faster if I eventually need to reimplement the game. Finally, if you’re interested in keeping up with the game or like technical posts like this one. I recommend subscribing to not miss out when new posts are published. Subscribe now In the meantime, here are few of my previous posts that could interest you!

0 views
maxdeviant.com 2 weeks ago

Head in the Zed Cloud

For the past five months I've been leading the efforts to rebuild Zed 's cloud infrastructure. Our current backend—known as Collab—has been chugging along since basically the beginning of the company. We use Collab every day to work together on Zed in Zed. However, as Zed continues to grow and attracts more users, we knew that we needed a full reboot of our backend infrastructure to set us up for success for our future endeavors. Enter Zed Cloud. Like Zed itself, Zed Cloud is built in Rust 1 . This time around there is a slight twist: all of this is running on Cloudflare Workers , with our Rust code being compiled down to WebAssembly (Wasm). One of our goals with this rebuild was to reduce the amount of operational effort it takes to maintain our hosted services, so that we can focus more of our time and energy on building Zed itself. Cloudflare Workers allow us to easily scale up to meet demand without having to fuss over it too much. Additionally, Cloudflare offers an ever-growing amount of managed services that cover anything you might need for a production web service. Here are some of the Cloudflare services we're using today: Another one of our goals with this rebuild was to build a platform that was easy to test. To achieve this, we built our own platform framework on top of the Cloudflare Workers runtime APIs. At the heart of this framework is the trait: This trait allows us to write our code in a platform-agnostic way while still leveraging all of the functionality that Cloudflare Workers has to offer. Each one of these associated types corresponds to some aspect of the platform that we'll want to have control over in a test environment. For instance, if we have a service that needs to interact with the system clock and a Workers KV store, we would define it like this: There are two implementors of the trait: and . —as the name might suggest—is an implementation of the platform on top of the Cloudflare Workers runtime. This implementation targets Wasm and is what we run when developing locally (using Wrangler ) and in production. We have a crate 2 that contains bindings to the Cloudflare Workers JS runtime. You can think of as the glue between those bindings and the idiomatic Rust APIs exposed by the trait. The is used when running tests, and allows for simulating almost every part of the system in order to effectively test our code. Here's an example of a test for ingesting a webhook from Orb : In this test we're able to test the full end-to-end flow of: The call to advances the test simulator, in this case running the pending queue consumers. At the center of the is the , a crate that powers our in-house async runtime. The scheduler is shared between GPUI —Zed's UI framework—and the used in tests. This shared scheduler enables us to write tests that span the client and the server. So we can have a test that starts in a piece of Zed code, flows through Zed Cloud, and then asserts on the state of something in Zed after it receives the response from the backend. The work being done on Zed Cloud now is laying the foundation to support our future work around collaborative coding with DeltaDB . If you want to work with me on building out Zed Cloud, we are currently hiring for this role. We're looking for engineers with experience building and maintaining web APIs and platforms, solid web fundamentals, and who are excited about Rust. If you end up applying, you can mention this blog post in your application. I look forward to hearing from you! The codebase is currently 70k lines of Rust code and 5.7k lines of TypeScript. This is essentially our own version of . I'd like to switch to using directly, at some point. Hyperdrive for talking to Postgres Workers KV for ephemeral storage Cloudflare Queues for asynchronous job processing Receiving and validating an incoming webhook event to our webhook ingestion endpoint Putting the webhook event into a queue Consuming the webhook event in a background worker and processing it

1 views
Jim Nielsen 3 weeks ago

Leveraging a Web Component For Comparing iOS and macOS Icons

Whenever Apple does a visual refresh in their OS updates, a new wave of icon archiving starts for me. Now that “Liquid Glass” is out, I’ve begun nabbing the latest icons from Apple and other apps and adding them to my gallery. Since I’ve been collecting these icons for so long, one of the more interesting and emerging attributes of my collection is the visual differences in individual app icons over time. For example: what are the differences between the icons I have in my collection for Duolingo? Well, I have a page for that today . That’ll let you see all the different versions I’ve collected for Duolingo — not exhaustive, I’m sure, but still interesting — as well as their different sizes . But what if you want to analyze their differences pixel-by-pixel? Turns out, There’s A Web Component For That™️. Image Compare is exactly what I was envisioning: “A tiny, zero-dependency web component for comparing two images using a slider” from the very fine folks at Cloud Four . It’s super easy to use: some HTML and a link to a script (hosted if you like, or you can vendor it ), e.g. And just like that, boom, I’ve got a widget for comparing two icons. For Duolingo specifically, I have a long history of icons archived in my gallery and they’re all available under the route for your viewing and comparison pleasure . Wanna see some more examples besides Duolingo? Check out the ones for GarageBand , Instagram , and Highlights for starters. Or, just look at the list of iOS apps and find the ones that are interesting to you (or if you’re a fan of macOS icons, check these ones out ). I kinda love how easy it was for my thought process to go from idea to reality: And I’ve written the post, so this chunk of work is now done. Reply via: Email · Mastodon · Bluesky “It would be cool to compare differences in icons by overlaying them…“ “Image diff tools do this, I bet I could find a good one…“ “Hey, Cloud Four makes a web component for this? Surely it’s good…” “Hey look, it’s just HTML: a tag linking to compiled JS along with a custom element? Easy, no build process required…“ “Done. Well that was easy. I guess the hardest part here will be writing the blog post about it.”

1 views
devansh 3 weeks ago

Hitchhiker's Guide to Attack Surface Management

I first heard about the word "ASM" (i.e., Attack Surface Management) probably in late 2018, and I thought it must be some complex infrastructure for tracking assets of an organization. Looking back, I realize I almost had a similar stack for discovering, tracking, and detecting obscure assets of organizations, and I was using it for my bug hunting adventures. I feel my stack was kinda goated, as I was able to find obscure assets of Apple, Facebook, Shopify, Twitter, and many other Fortune 100 companies, and reported hundreds of bugs, all through automation. Back in the day, projects like ProjectDiscovery were not present, so if I had to write an effective port scanner, I had to do it from scratch. (Masscan and nmap were present, but I had my fair share of issues using them, this is a story for another time). I used to write DNS resolvers (massdns had a high error rate), port scanners, web scrapers, directory brute-force utilities, wordlists, lots of JavaScript parsing logic using regex, and a hell of a lot of other things. I used to have up to 50+ self-developed tools for bug-bounty recon stuff and another 60-something helper scripts written in bash. I used to orchestrate (gluing together with duct tape is a better word) and slap together scripts like a workflow, and save the output in text files. Whenever I dealt with a large number of domains, I used to distribute the load over multiple servers (server spin-up + SSH into it + SCP for pushing and pulling files from it). The setup was very fragile and error-prone, and I spent countless nights trying to debug errors in the workflows. But it was all worth it. I learned the art of Attack Surface Management without even trying to learn about it. I was just a teenager trying to make quick bucks through bug hunting, and this fragile, duct-taped system was my edge. Fast forward to today, I have now spent almost a decade in the bug bounty scene. I joined HackerOne in 2020 (to present) as a vulnerability triager, where I have triaged and reviewed tens of thousands of vulnerability submissions. Fair to say, I have seen a lot of things, from doomsday level 0-days, to reports related to leaked credentials which could have led to entire infrastructure compromise, just because some dev pushed an AWS secret key in git logs, to things where some organizations were not even aware they were running Jenkins servers on some obscure subdomain which could have allowed RCE and then lateral movement to other layers of infrastructure. A lot of these issues I have seen were totally avoidable, only if organizations followed some basic attack surface management techniques. If I search "Guide to ASM" on Internet, almost none of the supposed guides are real resources. They funnel you to their own ASM solution, and the guide is just present there to provide you with some surface-level information, and is mostly a marketing gimmick. This is precisely why I decided to write something where I try to cover everything I learned and know about ASM, and how to protect your organization's assets before bad actors could get to them. This is going to be a rough and raw guide, and will not lead you to a funnel where I am trying to sell my own ASM SaaS to you. I have nothing to sell, other than offering what I know. But in case you are an organization who needs help implementing the things I am mentioning below, you can reach out to me via X or email (both available on the homepage of this blog). This guide will provide you with insights into exactly how big your attack surface really is. CISOs can look at it and see if their organizations have all of these covered, security researchers and bug hunters can look at this and maybe find new ideas related to where to look during recon. Devs can look at it and see if they are unintentionally leaving any door open for hackers. If you are into security, it has something to offer you. Attack surface is one of those terms getting thrown around in security circles so much that it's become almost meaningless noise. In theory, it sounds simple enough, right. Attack surface is every single potential entry point, interaction vector, or exploitable interface an attacker could use to compromise your systems, steal your data, or generally wreck your day. But here's the thing, it's the sum total of everything you've exposed to the internet. Every API endpoint you forgot about, every subdomain some dev spun up for "testing purposes" five years ago and then abandoned, every IoT device plugged into your network, every employee laptop connecting from a coffee shop, every third-party vendor with a backdoor into your environment, every cloud storage bucket with permissions that make no sense, every Slack channel, every git commit leaking credentials, every paste on Pastebin containing your database passwords. Most organizations think about attack surface in incredibly narrow terms. They think if they have a website, an email server, and maybe some VPN endpoints, they've got "good visibility" into their assets. That's just plain wrong. Straight up wrong. Your actual attack surface would terrify you if you actually understood it. You run , and is your main domain. You probably know about , , maybe . But what about that your intern from 2015 spun up and just never bothered to delete. It's not documented anywhere. Nobody remembers it exists. Domain attack surface goes way beyond what's sitting in your asset management system. Every subdomain is a potential entry point. Most of these subdomains are completely forgotten. Subdomain enumeration is reconnaissance 101 for attackers and bug hunters. It's not rocket science. Setting up a tool that actively monitors through active and passive sources for new subdomains and generates alerts is honestly an hour's worth of work. You can use tools like Subfinder, Amass, or just mine Certificate Transparency logs to discover every single subdomain connected to your domain. Certificate Transparency logs were designed to increase security by making certificate issuance public, and they've become an absolute reconnaissance goldmine. Every time you get an SSL certificate for , that information is sitting in public logs for anyone to find. Attackers systematically enumerate these subdomains using Certificate Transparency log searches, DNS brute-forcing with massive wordlists, reverse DNS lookups to map IP ranges back to domains, historical DNS data from services like SecurityTrails, and zone transfer exploitation if your DNS is misconfigured. Attackers are looking for old development environments still running vulnerable software, staging servers with production data sitting on them, forgotten admin panels, API endpoints without authentication, internal tools accidentally exposed, and test environments with default credentials nobody changed. Every subdomain is an asset. Every asset is a potential vulnerability. Every vulnerability is an entry point. Domains and subdomains are just the starting point though. Once you've figured out all the subdomains belonging to your organization, the next step is to take a hard look at IP address space, which is another absolutely massive component of your attack surface. Organizations own, sometimes lease, IP ranges, sometimes small /24 blocks, sometimes massive /16 ranges, and every single IP address in those blocks and ranges that responds to external traffic is part of your attack surface. And attackers enumerate them all if you won't. They use WHOIS lookups to identify your IP ranges, port scanning to find what services are running where, service fingerprinting to identify exact software versions, and banner grabbing to extract configuration information. If you have a /24 network with 256 IP addresses and even 10% of those IPs are running services, you've got 25 potential attack vectors. Scale that to a /20 or /16 and you're looking at thousands of potential entry points. And attackers aren't just looking at the IPs you know about. They're looking at adjacent IP ranges you might have acquired through mergers, historical IP allocations that haven't been properly decommissioned, and shared IP ranges where your servers coexist with others. Traditional infrastructure was complicated enough, and now we have cloud. It's literally exploded organizations' attack surfaces in ways that are genuinely difficult to even comprehend. Every cloud service you spin up, be it an EC2 instance, S3 bucket, Lambda function, or API Gateway endpoint, all of this is a new attack vector. In my opinion and experience so far, I think the main issue with cloud infrastructure is that it's ephemeral and distributed. Resources get spun up and torn down constantly. Developers create instances for testing and forget about them. Auto-scaling groups generate new resources dynamically. Containerized workloads spin up massive Kubernetes clusters you have minimal visibility into. Your cloud attack surface could be literally anything. Examples are countless, but I'd categorize them into 8 different categories. Compute instances like EC2, Azure VMs, GCP Compute Engine instances exposed to the internet. Storage buckets like S3, Azure Blob Storage, GCP Cloud Storage with misconfigured permissions. Serverless stuff like Lambda functions with public URLs or overly permissive IAM roles. API endpoints like API Gateway, Azure API Management endpoints without proper authentication. Container registries like Docker images with embedded secrets or vulnerabilities. Kubernetes clusters with exposed API servers, misconfigured network policies, vulnerable ingress controllers. Managed databases like RDS, CosmosDB, Cloud SQL instances with weak access controls. IAM roles and service accounts with overly permissive identities that enable privilege escalation. I've seen instances in the past where a single misconfigured S3 bucket policy exposed terabytes of data. An overly permissive Lambda IAM role enabled lateral movement across an entire AWS account. A publicly accessible Kubernetes API server gave an attacker full cluster control. Honestly, cloud kinda scares me as well. And to top it off, multi-cloud infrastructure makes everything worse. If you're running AWS, Azure, and GCP together, you've just tripled your attack surface management complexity. Each cloud provider has different security models, different configuration profiles, and different attack vectors. Every application now uses APIs, and all applications nowadays are like a constellation of APIs talking to each other. Every API you use in your organization is your attack surface. The problem with APIs is that they're often deployed without the same security scrutiny as traditional web applications. Developers spin up API endpoints for specific features and those endpoints accumulate over time. Some of them are shadow APIs, meaning API endpoints which aren't documented anywhere. These endpoints are the equivalent of forgotten subdomains, and attackers can find them through analyzing JavaScript files for API endpoint references, fuzzing common API path patterns, examining mobile app traffic to discover backend APIs, and mining old documentation or code repositories for deprecated endpoints. Your API attack surface includes REST APIs exposed to the internet, GraphQL endpoints with overly broad query capabilities, WebSocket connections for real-time functionality, gRPC services for inter-service communication, and legacy SOAP APIs that never got decommissioned. If your organization has mobile apps, be it iOS, Android, or both, this is a direct window to your infrastructure and should be part of your attack surface management strategy. Mobile apps communicate with backend APIs and those API endpoints are discoverable by reversing the app. The reversed source of the app could reveal hard-coded API keys, tokens, and credentials. Using JADX plus APKTool plus Dex2jar is all a motivated attacker needs. Web servers often expose directories and files that weren't meant to be publicly accessible. Attackers systematically enumerate these using automated tools like ffuf, dirbuster, gobuster, and wfuzz with massive wordlists to discover hidden endpoints, configuration files, backup files, and administrative interfaces. Common exposed directories include admin panels, backup directories containing database dumps or source code, configuration files with database credentials and API keys, development directories with debug information, documentation directories revealing internal systems, upload directories for file storage, and old or forgotten directories from previous deployments. Your attack surface must include directories which are accidentally left accessible during deployments, staging servers with production data, backup directories with old source code versions, administrative interfaces without authentication, API documentation exposing endpoint details, and test directories with debug output enabled. Even if you've removed a directory from production, old cached versions may still be accessible through web caches or CDNs. Search engines also index these directories, making them discoverable through dorking techniques. If your organization is using IoT devices, and everyone uses these days, this should be part of your attack surface management strategy. They're invisible to traditional security tools. Your EDR solution doesn't protect IoT devices. Your vulnerability scanner can't inventory them. Your patch management system can't update them. Your IoT attack surface could include smart building systems like HVAC, lighting, access control. Security cameras and surveillance systems. Printers and copiers, which are computers with network access. Badge readers and physical access systems. Industrial control systems and SCADA devices. Medical devices in healthcare environments. Employee wearables and fitness trackers. Voice assistants and smart speakers. The problem with IoT devices is that they're often deployed without any security consideration. They have default credentials that never get changed, unpatched firmware with known vulnerabilities, no encryption for data in transit, weak authentication mechanisms, and insecure network configurations. Social media presence is an attack surface component that most organizations completely ignore. Attackers can use social media for reconnaissance by looking at employee profiles on LinkedIn to reveal organizational structure, technologies in use, and current projects. Twitter/X accounts can leak information about deployments, outages, and technology stack. Employee GitHub profiles expose email patterns and development practices. Company blogs can announce new features before security review. It could also be a direct attack vector. Attackers can use information from social media to craft convincing phishing attacks. Hijacked social media accounts can be used to spread malware or phishing links. Employees can accidentally share sensitive information. Fake accounts can impersonate your brand to defraud customers. Your employees' social media presence is part of your attack surface whether you like it or not. Third-party vendors, suppliers, contractors, or partners with access to your systems should be part of your attack surface. Supply chain attacks are becoming more and more common these days. Attackers can compromise a vendor with weaker security and then use that vendor's access to reach your environment. From there, they pivot from the vendor network to your systems. This isn't a hypothetical scenario, it has happened multiple times in the past. You might have heard about the SolarWinds attack, where attackers compromised SolarWinds' build system and distributed malware through software updates to thousands of customers. Another famous case study is the MOVEit vulnerability in MOVEit Transfer software, exploited by the Cl0p ransomware group, which affected over 2,700 organizations. These are examples of some high-profile supply chain security attacks. Your third-party attack surface could include things like VPNs, remote desktop connections, privileged access systems, third-party services with API keys to your systems, login credentials shared with vendors, SaaS applications storing your data, and external IT support with administrative access. It's obvious you can't directly control third-party security. You can audit them, have them pen-test their assets as part of your vendor compliance plan, and include security requirements in contracts, but ultimately their security posture is outside your control. And attackers know this. GitHub, GitLab, Bitbucket, they all are a massive attack surface. Attackers search through code repositories in hopes of finding hard-coded credentials like API keys, database passwords, and tokens. Private keys, SSH keys, TLS certificates, and encryption keys. Internal architecture documentation revealing infrastructure details in code comments. Configuration files with database connection strings and internal URLs. Deprecated code with vulnerabilities that's still in production. Even private repositories aren't safe. Attackers can compromise developer accounts to access private repositories, former employees retain access after leaving, and overly broad repository permissions grant access to too many people. Automated scanners continuously monitor public repositories for secrets. The moment a developer accidentally pushes credentials to a public repository, automated systems detect it within minutes. Attackers have already extracted and weaponized those credentials before the developer realizes the mistake. CI/CD pipelines are massive another attack vector. Especially in recent times, and not many organizations are giving attention to this attack vector. This should totally be part of your attack surface management. Attackers compromise GitHub Actions workflows with malicious code injection, Jenkins servers with weak authentication, GitLab CI/CD variables containing secrets, and build artifacts with embedded malware. The GitHub Actions supply chain attack, CVE-2025-30066, demonstrated this perfectly. Attackers compromised the Action used in over 23,000 repositories, injecting malicious code that leaked secrets from build logs. Jenkins specifically is a goldmine for attackers. An exposed Jenkins instance provides complete control over multiple critical servers, access to hardcoded AWS keys, Redis credentials, and BitBucket tokens, ability to manipulate builds and inject malicious code, and exfiltration of production database credentials containing PII. Modern collaboration tools are massive attack surface components that most organizations underestimate. Slack has hidden security risks despite being invite-only. Slack attack surface could include indefinite data retention where every message, channel, and file is stored forever unless admins configure retention periods. Public channels accessible to all users so one breached account opens the floodgates. Third-party integrations with excessive permissions accessing messages and user data. Former contractor access where individuals retain access long after projects end. Phishing and impersonation where it's easy to change names and pictures to impersonate senior personnel. In 2022, Slack leaked hashed passwords for five years affecting 0.5% of users. Slack channels commonly contain API keys, authentication tokens, database credentials, customer PII, financial data, internal system passwords, and confidential project information. The average cost of a breached record was $164 in 2022. When 1 in 166 messages in Slack contains confidential information, every new message adds another dollar to total risk exposure. With 5,000 employees sending 30 million Slack messages per year, that's substantial exposure. Trello board exposure is a significant attack surface. Trello attack vectors include public boards with sensitive information accidentally shared publicly, default public visibility where boards are created as public by default in some configurations, unsecured REST API allowing unauthenticated access to user data, and scraping attacks where attackers use email lists to enumerate Trello accounts. The 2024 Trello data breach exposed 15 million users' personal information when a threat actor named "emo" exploited an unsecured REST API using 500 million email addresses to compile detailed user profiles. Security researcher David Shear documented hundreds of public Trello boards exposing passwords, credentials, IT support customer access details, website admin logins, and client server management credentials. IT companies were using Trello to troubleshoot client requests and manage infrastructure, storing all credentials on public Trello boards. Jira misconfiguration is a widespread attack surface issue. Common misconfigurations include public dashboards and filters with "Everyone" access actually meaning public internet access, anonymous access enabled allowing unauthenticated users to browse, user picker functionality providing complete lists of usernames and email addresses, and project visibility allowing sensitive projects to be accessible without authentication. Confluence misconfiguration exposes internal documentation. Confluence attack surface components include anonymous access at site level allowing public access, public spaces where space admins grant anonymous permissions, inherited permissions where all content within a space inherits space-level access, and user profile visibility allowing anonymous users to view profiles of logged-in users. When anonymous access is enabled globally and space admins allow anonymous users to access their spaces, anyone on the internet can access that content. Confluence spaces often contain internal documentation with hardcoded credentials, financial information, project details, employee information, and API documentation with authentication details. Cloud storage misconfiguration is epidemic. Google Drive misconfiguration attack surface includes "Anyone with the link" sharing making files accessible without authentication, overly permissive sharing defaults making it easy to accidentally share publicly, inherited folder permissions exposing everything beneath, unmanaged third-party apps with excessive read/write/delete permissions, inactive user accounts where former employees retain access, and external ownership blind spots where externally-owned content is shared into the environment. Metomic's 2023 Google Scanner Report found that of 6.5 million Google Drive files analyzed, 40.2% contained sensitive information, 34.2% were shared externally, and 0.5% were publicly accessible, mostly unintentionally. In December 2023, Japanese game developer Ateam suffered a catastrophic Google Drive misconfiguration that exposed personal data of nearly 1 million people for over six years due to "Anyone with the link" settings. Based on Valence research, 22% of external data shares utilize open links, and 94% of these open link shares are inactive, forgotten files with public URLs floating around the internet. Dropbox, OneDrive, and Box share similar attack surface components including misconfigured sharing permissions, weak or missing password protection, overly broad access grants, third-party app integrations with excessive permissions, and lack of visibility into external sharing. Features that make file sharing convenient create data leakage risks when misconfigured. Pastebin and similar paste sites are both reconnaissance sources and attack vectors. Paste site attack surface includes public data dumps of stolen credentials, API keys, and database dumps posted publicly, malware hosting of obfuscated payloads, C2 communications where malware uses Pastebin for command and control, credential leakage from developers accidentally posting secrets, and bypassing security filters since Pastebin is legitimate so security tools don't block it. For organizations, leaked API keys or database credentials on Pastebin lead to unauthorized access, data exfiltration, and service disruption. Attackers continuously scan Pastebin for mentions of target organizations using automated tools. Security teams must actively monitor Pastebin and similar paste sites for company name mentions, email domain references, and specific keywords related to the organization. Because paste sites don't require registration or authentication and content is rarely removed, they've become permanent archives of leaked secrets. Container registries expose significant attack surface. Container registry attack surface includes secrets embedded in image layers where 30,000 unique secrets were found in 19,000 images, with 10% of scanned Docker images containing secrets, and 1,200 secrets, 4%, being active and valid. Immutable cached layers contain 85% of embedded secrets that can't be removed, exposed registries with 117 Docker registries accessible without authentication, unsecured registries allowing pull, push, and delete operations, and source code exposure where full application code is accessible by pulling images. GitGuardian's analysis of 200,000 publicly available Docker images revealed a staggering secret exposure problem. Even more alarming, 99% of images containing active secrets were pulled in 2024, demonstrating real-world exploitation. Unit 42's research identified 941 Docker registries exposed to the internet, with 117 accessible without authentication containing 2,956 repositories, 15,887 tags, and full source code and historical versions. Out of 117 unsecured registries, 80 allow pull operations to download images, 92 allow push operations to upload malicious images, and 7 allow delete operations for ransomware potential. Sysdig's analysis of over 250,000 Linux images on Docker Hub found 1,652 malicious images including cryptominers, most common, embedded secrets, second most prevalent, SSH keys and public keys for backdoor implants, API keys and authentication tokens, and database credentials. The secrets found in container images included AWS access keys, database passwords, SSH private keys, API tokens for cloud services, GitHub personal access tokens, and TLS certificates. Shadow IT includes unapproved SaaS applications like Dropbox, Google Drive, and personal cloud storage used for work. Personal devices like BYOD laptops, tablets, and smartphones accessing corporate data. Rogue cloud deployments where developers spin up AWS instances without approval. Unauthorized messaging apps like WhatsApp, Telegram, and Signal used for business communication. Unapproved IoT devices like smart speakers, wireless cameras, and fitness trackers on the corporate network. Gartner estimates that shadow IT makes up 30-40% of IT spending in large companies, and 76% of organizations surveyed experienced cyberattacks due to exploitation of unknown, unmanaged, or poorly managed assets. Shadow IT expands your attack surface because it's not protected by your security controls, it's not monitored by your security team, it's not included in your vulnerability scans, it's not patched by your IT department, and it often has weak or default credentials. And you can't secure what you don't know exists. Bring Your Own Device, BYOD, policies sound great for employee flexibility and cost savings. For security teams, they're a nightmare. BYOD expands your attack surface by introducing unmanaged endpoints like personal devices without EDR, antivirus, or encryption. Mixing personal and business use where work data is stored alongside personal apps with unknown security. Connecting from untrusted networks like public Wi-Fi and home networks with compromised routers. Installing unapproved applications with malware or excessive permissions. Lacking consistent security updates with devices running outdated operating systems. Common BYOD security issues include data leakage through personal cloud backup services, malware infections from personal app downloads, lost or stolen devices containing corporate data, family members using devices that access work systems, and lack of IT visibility and control. The 60% of small and mid-sized businesses that close within six months of a major cyberattack often have BYOD-related security gaps as contributing factors. Remote access infrastructure like VPNs and Remote Desktop Protocol, RDP, are among the most exploited attack vectors. SSL VPN appliances from vendors like Fortinet, SonicWall, Check Point, and Palo Alto are under constant attack. VPN attack vectors include authentication bypass vulnerabilities with CVEs allowing attackers to hijack active sessions, credential stuffing through brute-forcing VPN logins with leaked credentials, exploitation of unpatched vulnerabilities with critical CVEs in VPN appliances, and configuration weaknesses like default credentials, weak passwords, and lack of MFA. Real-world attacks demonstrate the risk. Check Point SSL VPN CVE-2024-24919 allowed authentication bypass for session hijacking. Fortinet SSL-VPN vulnerabilities were leveraged for lateral movement and privilege escalation. SonicWall CVE-2024-53704 allowed remote authentication bypass for SSL VPN. Once inside via VPN, attackers conduct network reconnaissance, lateral movement, and privilege escalation. RDP is worse. Sophos found that cybercriminals abused RDP in 90% of attacks they investigated. External remote services like RDP were the initial access vector in 65% of incident response cases. RDP attack vectors include exposed RDP ports with port 3389 open to the internet, weak authentication with simple passwords vulnerable to brute force, lack of MFA with no second factor for authentication, and credential reuse from compromised passwords in data breaches. In one Darktrace case, attackers compromised an organization four times in six months, each time through exposed RDP ports. The attack chain went successful RDP login, internal reconnaissance via WMI, lateral movement via PsExec, and objective achievement. The Palo Alto Unit 42 Incident Response report found RDP was the initial attack vector in 50% of ransomware deployment cases. Email infrastructure remains a primary attack vector. Your email attack surface includes mail servers like Exchange, Office 365, and Gmail with configuration weaknesses, email authentication with misconfigured SPF, DKIM, and DMARC records, phishing-susceptible users targeted through social engineering, email attachments and links as malware delivery mechanisms, and compromised accounts through credential stuffing or password reuse. Email authentication misconfiguration is particularly insidious. If your SPF, DKIM, and DMARC records are wrong or missing, attackers can spoof emails from your domain, your legitimate emails get marked as spam, and phishing emails impersonating your organization succeed. Email servers themselves are also targets. The NSA released guidance on Microsoft Exchange Server security specifically because Exchange servers are so frequently compromised. Container orchestration platforms like Kubernetes introduce massive attack surface complexity. The Kubernetes attack surface includes the Kubernetes API server with exposed or misconfigured API endpoints, container images with vulnerabilities in base images or application layers, container registries like Docker Hub, ECR, and GCR with weak access controls, pod security policies with overly permissive container configurations, network policies with insufficient micro-segmentation between pods, secrets management with hardcoded secrets or weak secret storage, and RBAC misconfigurations with overly broad service account permissions. Container security issues include containers running as root with excessive privileges, exposed Docker daemon sockets allowing container escape, vulnerable dependencies in container images, and lack of runtime security monitoring. The Docker daemon attack surface is particularly concerning. Running containers with privileged access or allowing docker.sock access can enable container escape and host compromise. Serverless computing like AWS Lambda, Azure Functions, and Google Cloud Functions promised to eliminate infrastructure management. Instead, it just created new attack surfaces. Serverless attack surface components include function code vulnerabilities like injection flaws and insecure dependencies, IAM misconfigurations with overly permissive Lambda execution roles, environment variables storing secrets as plain text, function URLs with publicly accessible endpoints without authentication, and event source mappings with untrusted input from various cloud services. The overabundance of event sources expands the attack surface. Lambda functions can be triggered by S3 events, API Gateway requests, DynamoDB streams, SNS topics, EventBridge schedules, IoT events, and dozens more. Each event source is a potential injection point. If function input validation is insufficient, attackers can manipulate event data to exploit the function. Real-world Lambda attacks include credential theft by exfiltrating IAM credentials from environment variables, lateral movement using over-permissioned roles to access other AWS resources, and data exfiltration by invoking functions to query and extract database contents. The Scarlet Eel adversary specifically targeted AWS Lambda for credential theft and lateral movement. Microservices architecture multiplies attack surface by decomposing monolithic applications into dozens or hundreds of independent services. Each microservice has its own attack surface including authentication mechanisms where each service needs to verify requests, authorization rules where each service enforces access controls, API endpoints for service-to-service communication channels, data stores where each service may have its own database, and network interfaces where each service exposes network ports. Microservices security challenges include east-west traffic vulnerabilities with service-to-service communication without encryption or authentication, authentication and authorization complexity from managing auth across 40 plus services multiplied by 3 environments equaling 240 configurations, service-to-service trust where services blindly trust internal traffic, network segmentation failures with flat networks allowing unrestricted pod-to-pod communication, and inconsistent security policies with different services having different security standards. One compromised microservice can enable lateral movement across the entire application. Without proper network segmentation and zero trust architecture, attackers pivot from service to service. How do you measure something this large, right. Attack surface measurement is complex. Attack surface metrics include the total number of assets with all discovered systems, applications, and devices, newly discovered assets found through continuous discovery, the number of exposed assets accessible from the internet, open ports and services with network services listening for connections, vulnerabilities by severity including critical, high, medium, and low CVEs, mean time to detect, MTTD, measuring how quickly new assets are discovered, mean time to remediate, MTTR, measuring how quickly vulnerabilities are fixed, shadow IT assets that are unknown or unmanaged, third-party exposure from vendor and partner access points, and attack surface change rate showing how rapidly the attack surface evolves. Academic research has produced formal attack surface measurement methods. Pratyusa Manadhata's foundational work defines attack surface as a three-tuple, System Attackability, Channel Attackability, Data Attackability. But in practice, most organizations struggle with basic attack surface visibility, let alone quantitative measurement. Your attack surface isn't static. It changes constantly. Changes happen because developers deploy new services and APIs, cloud auto-scaling spins up new instances, shadow IT appears as employees adopt unapproved tools, acquisitions bring new infrastructure into your environment, IoT devices get plugged into your network, and subdomains get created for new projects. Static, point-in-time assessments are obsolete. You need continuous asset discovery and monitoring. Continuous discovery methods include automated network scanning for regular scans to detect new devices, cloud API polling to query cloud provider APIs for resource changes, DNS monitoring to track new subdomains via Certificate Transparency logs, passive traffic analysis to observe network traffic and identify assets, integration with CMDB or ITSM to sync with configuration management databases, and cloud inventory automation using Infrastructure as Code to track deployments. Understanding your attack surface is step one. Reducing it is the goal. Attack surface reduction begins with asset elimination by removing unnecessary assets entirely. This includes decommissioning unused servers and applications, deleting abandoned subdomains and DNS records, shutting down forgotten development environments, disabling unused network services and ports, and removing unused user accounts and service identities. Access control hardening implements least privilege everywhere by enforcing multi-factor authentication, MFA, for all remote access, using role-based access control, RBAC, for cloud resources, implementing zero trust network architecture, restricting network access with micro-segmentation, and applying the principle of least privilege to IAM roles. Exposure minimization reduces what's visible to attackers by moving services behind VPNs or bastion hosts, using private IP ranges for internal services, implementing network address translation, NAT, for outbound access, restricting API endpoints to authorized sources only, and disabling unnecessary features and functionalities. Security hardening strengthens what remains by applying security patches promptly, using security configuration baselines, enabling encryption for data in transit and at rest, implementing Web Application Firewalls, WAF, for web apps, and deploying endpoint detection and response, EDR, on all devices. Monitoring and detection watch for attacks in progress by implementing real-time threat detection, enabling comprehensive logging and SIEM integration, deploying intrusion detection and prevention systems, IDS/IPS, monitoring for anomalous behavior patterns, and using threat intelligence feeds to identify known bad actors. Your attack surface is exponentially larger than you think it is. Every asset you know about probably has three you don't. Every known vulnerability probably has ten undiscovered ones. Every third-party integration probably grants more access than you realize. Every collaboration tool is leaking more data than you imagine. Every paste site contains more of your secrets than you want to admit. And attackers know this. They're not just looking at what you think you've secured. They're systematically enumerating every possible entry point. They're mining Certificate Transparency logs for forgotten subdomains. They're scanning every IP in your address space. They're reverse-engineering your mobile apps. They're buying employee credentials from data breach databases. They're compromising your vendors to reach you. They're scraping Pastebin for your leaked secrets. They're pulling your public Docker images and extracting the embedded credentials. They're accessing your misconfigured S3 buckets and exfiltrating terabytes of data. They're exploiting your exposed Jenkins instances to compromise your entire infrastructure. They're manipulating your AI agents to exfiltrate private Notion data. The asymmetry is brutal. You have to defend every single attack vector. They only need to find one that works. So what do you do. Start by accepting that you don't have complete visibility. Nobody does. But you can work toward better visibility through continuous discovery, automated asset management, and integration of security tools that help map your actual attack surface. Implement attack surface reduction aggressively. Every asset you eliminate is one less thing to defend. Every service you shut down is one less potential vulnerability. Every piece of shadow IT you discover and bring under management is one less blind spot. Every misconfigured cloud storage bucket you fix is terabytes of data no longer exposed. Every leaked secret you rotate is one less credential floating around the internet. Adopt zero trust architecture. Stop assuming that anything, internal services, microservices, authenticated users, collaboration tools, is inherently trustworthy. Verify everything. Monitor paste sites and code repositories. Your secrets are out there. Find them before attackers weaponize them. Secure your collaboration tools. Slack, Trello, Jira, Confluence, Notion, Google Drive, and Airtable are all leaking data. Lock them down. Fix your container security. Scan images for secrets. Use secret managers instead of environment variables. Secure your registries. Harden your CI/CD pipelines. Jenkins, GitHub Actions, and GitLab CI are high-value targets. Protect them. And test your assumptions with red team exercises and continuous security testing. Your attack surface is what an attacker can reach, not what you think you've secured. The attack surface problem isn't getting better. Cloud adoption, DevOps practices, remote work, IoT proliferation, supply chain complexity, collaboration tool sprawl, and container adoption are all expanding organizational attack surfaces faster than security teams can keep up. But understanding the problem is the first step toward managing it. And now you understand exactly how catastrophically large your attack surface actually is.

1 views
JSLegendDev 3 weeks ago

I'm Making a Tiny RPG and I Need Feedback Regarding Performance

This past month, I’ve been working secretly on a small RPG game. While the game is not ready at all and I didn’t plan on talking about it, I’m now kind of forced to. I’ve been using JavaScript + the KAPLAY game library to make this game but I’ve been experiencing performance issues. However, it seems that others aren’t experiencing them so now I wonder, is it just my machine? I’m using a Macbook Air M3 with 16GB of RAM. Normally, things should be smooth and they are when playing the game in the browser via Firefox. However, since I’m making this game with Steam in mind, it’s especially important that the game performs well when wrapped as a desktop app. For this reason, I decided to reach out to you, my audience, for feedback. I have included a build of the unfinished game for Windows, Mac and Linux. It would be very nice if you could try it out on your machine. Additionally, recording gameplay and sharing the link in the comment section of this post would be greatly appreciated. Here is the link to the game in its unfinished current form : https://jslegend.itch.io/small-rpg-performance-playtest Below is a gameplay video to give you an idea of how the game is supposed to be played. You can move around with arrow keys and enter battle by overlapping with a star in the overworld. Performance issues, if any, occur mostly during battles. Thanks in advance! UPDATE 7/11/2025 : It seems that this post was shared on Hacker News and is getting more attention than usual! If you’re new to my Substack and are potentially interested in updates regarding this project, I recommend subscribing. Subscribe now In the meantime, you can read some of my previous posts!

0 views
Simon Willison 3 weeks ago

Video + notes on upgrading a Datasette plugin for the latest 1.0 alpha, with help from uv and OpenAI Codex CLI

I'm upgrading various plugins for compatibility with the new Datasette 1.0a20 alpha release and I decided to record a video of the process. This post accompanies that video with detailed additional notes. I picked a very simple plugin to illustrate the upgrade process (possibly too simple). datasette-checkbox adds just one feature to Datasette: if you are viewing a table with boolean columns (detected as integer columns with names like or or ) and your current user has permission to update rows in that table it adds an inline checkbox UI that looks like this: I built the first version with the help of Claude back in August 2024 - details in this issue comment . Most of the implementation is JavaScript that makes calls to Datasette 1.0's JSON write API . The Python code just checks that the user has the necessary permissions before including the extra JavaScript. The first step in upgrading any plugin is to run its tests against the latest Datasette version. Thankfully makes it easy to run code in scratch virtual environments that include the different code versions you want to test against. I have a test utility called (for "test against development Datasette") which I use for that purpose. I can run it in any plugin directory like this: And it will run the existing plugin tests against whatever version of Datasette I have checked out in my directory. You can see the full implementation of (and its friend described below) in this TIL - the basic version looks like this: I started by running in the directory, and got my first failure... but it wasn't due to permissions, it was because the for the plugin was pinned to a specific mismatched version of Datasette: I fixed this problem by swapping to and ran the tests again... and they passed! Which was a problem because I was expecting permission-related failures. It turns out when I first wrote the plugin I was lazy with the tests - they weren't actually confirming that the table page loaded without errors. I needed to actually run the code myself to see the expected bug. First I created myself a demo database using sqlite-utils create-table : Then I ran it with Datasette against the plugin's code like so: Sure enough, visiting produced a 500 error about the missing method. The next step was to update the test to also trigger this error: And now fails as expected. It this point I could have manually fixed the plugin itself - which would likely have been faster given the small size of the fix - but instead I demonstrated a bash one-liner I've been using to apply these kinds of changes automatically: runs OpenAI Codex in non-interactive mode - it will loop until it has finished the prompt you give it. I tell it to consult the subset of the Datasette upgrade documentation that talks about Datasette permissions and then get the command to pass its tests. This is an example of what I call designing agentic loops - I gave Codex the tools it needed ( ) and a clear goal and let it get to work on my behalf. The remainder of the video covers finishing up the work - testing the fix manually, commiting my work using: Then shipping a 0.1a4 release to PyPI using the pattern described in this TIL . Finally, I demonstrated that the shipped plugin worked in a fresh environment using like this: Executing this command installs and runs a fresh Datasette instance with a fresh copy of the new alpha plugin ( ). It's a neat way of confirming that freshly released software works as expected. This video was shot in a single take using Descript , with no rehearsal and perilously little preparation in advance. I recorded through my AirPods and applied the "Studio Sound" filter to clean up the audio. I pasted in a closing slide from my previous video and exported it locally at 1080p, then uploaded it to YouTube. Something I learned from the Software Carpentry instructor training course is that making mistakes in front of an audience is actively helpful - it helps them see a realistic version of how software development works and they can learn from watching you recover. I see this as a great excuse for not editing out all of my mistakes! I'm trying to build new habits around video content that let me produce useful videos while minimizing the amount of time I spend on production. I plan to iterate more on the format as I get more comfortable with the process. I'm hoping I can find the right balance between production time and value to viewers. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options .

0 views
Simon Willison 3 weeks ago

Code research projects with async coding agents like Claude Code and Codex

I've been experimenting with a pattern for LLM usage recently that's working out really well: asynchronous code research tasks . Pick a research question, spin up an asynchronous coding agent and let it go and run some experiments and report back when it's done. Software development benefits enormously from something I call code research . The great thing about questions about code is that they can often be definitively answered by writing and executing code. I often see questions on forums which hint at a lack of understanding of this skill. "Could Redis work for powering the notifications feed for my app?" is a great example. The answer is always "it depends", but a better answer is that a good programmer already has everything they need to answer that question for themselves. Build a proof-of-concept, simulate the patterns you expect to see in production, then run experiments to see if it's going to work. I've been a keen practitioner of code research for a long time. Many of my most interesting projects started out as a few dozen lines of experimental code to prove to myself that something was possible. It turns out coding agents like Claude Code and Codex are a fantastic fit for this kind of work as well. Give them the right goal and a useful environment and they'll churn through a basic research project without any further supervision. LLMs hallucinate and make mistakes. This is far less important for code research tasks because the code itself doesn't lie: if they write code and execute it and it does the right things then they've demonstrated to both themselves and to you that something really does work. They can't prove something is impossible - just because the coding agent couldn't find a way to do something doesn't mean it can't be done - but they can often demonstrate that something is possible in just a few minutes of crunching. I've used interactive coding agents like Claude Code and Codex CLI for a bunch of these, but today I'm increasingly turning to their asynchronous coding agent family members instead. An asynchronous coding agent is a coding agent that operates on a fire-and-forget basis. You pose it a task, it churns away on a server somewhere and when it's done it files a pull request against your chosen GitHub repository. OpenAI's Codex Cloud , Anthropic's Claude Code for web , Google Gemini's Jules , and GitHub's Copilot coding agent are four prominent examples of this pattern. These are fantastic tools for code research projects. Come up with a clear goal, turn it into a few paragraphs of prompt, set them loose and check back ten minutes later to see what they've come up with. I'm firing off 2-3 code research projects a day right now. My own time commitment is minimal and they frequently come back with useful or interesting results. You can run a code research task against an existing GitHub repository, but I find it's much more liberating to have a separate, dedicated repository for your coding agents to run their projects in. This frees you from being limited to research against just code you've already written, and also means you can be much less cautious about what you let the agents do. I have two repositories that I use for this - one public, one private. I use the public one for research tasks that have no need to be private, and the private one for anything that I'm not yet ready to share with the world. The biggest benefit of a dedicated repository is that you don't need to be cautious about what the agents operating in that repository can do. Both Codex Cloud and Claude Code for web default to running agents in a locked-down environment, with strict restrictions on how they can access the network. This makes total sense if they are running against sensitive repositories - a prompt injection attack of the lethal trifecta variety could easily be used to steal sensitive code or environment variables. If you're running in a fresh, non-sensitive repository you don't need to worry about this at all! I've configured my research repositories for full network access, which means my coding agents can install any dependencies they need, fetch data from the web and generally do anything I'd be able to do on my own computer. Let's dive into some examples. My public research repository is at simonw/research on GitHub. It currently contains 13 folders, each of which is a separate research project. I only created it two weeks ago so I'm already averaging nearly one a day! It also includes a GitHub Workflow which uses GitHub Models to automatically update the README file with a summary of every new project, using Cog , LLM , llm-github-models and this snippet of Python . Here are a some example research projects from the repo. node-pyodide shows an example of a Node.js script that runs the Pyodide WebAssembly distribution of Python inside it - yet another of my ongoing attempts to find a great way of running Python in a WebAssembly sandbox on a server. python-markdown-comparison ( transcript ) provides a detailed performance benchmark of seven different Python Markdown libraries. I fired this one off because I stumbled across cmarkgfm , a Python binding around GitHub's Markdown implementation in C, and wanted to see how it compared to the other options. This one produced some charts! came out on top by a significant margin: Here's the entire prompt I used for that project: Create a performance benchmark and feature comparison report on PyPI cmarkgfm compared to other popular Python markdown libraries - check all of them out from github and read the source to get an idea for features, then design and run a benchmark including generating some charts, then create a report in a new python-markdown-comparison folder (do not create a _summary.md file or edit anywhere outside of that folder). Make sure the performance chart images are directly displayed in the README.md in the folder. Note that I didn't specify any Markdown libraries other than - Claude Code ran a search and found the other six by itself. cmarkgfm-in-pyodide is a lot more fun. A neat thing about having all of my research projects in the same repository is that new projects can build on previous ones. Here I decided to see how hard it would be to get - which has a C extension - working inside Pyodide inside Node.js. Claude successfully compiled a 88.4KB file with the necessary C extension and proved it could be loaded into Pyodide in WebAssembly inside of Node.js. I ran this one using Claude Code on my laptop after an initial attempt failed. The starting prompt was: Figure out how to get the cmarkgfm markdown lover [typo in prompt, this should have been "library" but it figured it out anyway] for Python working in pyodide. This will be hard because it uses C so you will need to compile it to pyodide compatible webassembly somehow. Write a report on your results plus code to a new cmarkgfm-in-pyodide directory. Test it using pytest to exercise a node.js test script that calls pyodide as seen in the existing node.js and pyodide directory There is an existing branch that was an initial attempt at this research, but which failed because it did not have Internet access. You do have Internet access. Use that existing branch to accelerate your work, but do not commit any code unless you are certain that you have successfully executed tests that prove that the pyodide module you created works correctly. This one gave up half way through, complaining that emscripten would take too long. I told it: Complete this project, actually run emscripten, I do not care how long it takes, update the report if it works It churned away for a bit longer and complained that the existing Python library used CFFI which isn't available in Pyodide. I asked it: Can you figure out how to rewrite cmarkgfm to not use FFI and to use a pyodide-friendly way of integrating that C code instead? ... and it did. You can see the full transcript here . blog-tags-scikit-learn . Taking a short break from WebAssembly, I thought it would be fun to put scikit-learn through its paces on a text classification task against my blog: Work in a new folder called blog-tags-scikit-learn Download - a SQLite database. Take a look at the blog_entry table and the associated tags - a lot of the earlier entries do not have tags associated with them, where the later entries do. Design, implement and execute models to suggests tags for those earlier entries based on textual analysis against later ones Use Python scikit learn and try several different strategies Produce JSON of the results for each one, plus scripts for running them and a detailed markdown description Also include an HTML page with a nice visualization of the results that works by loading those JSON files. This resulted in seven files, four results files and a detailed report . (It ignored the bit about an HTML page with a nice visualization for some reason.) Not bad for a few moments of idle curiosity typed into my phone! That's just three of the thirteen projects in the repository so far. The commit history for each one usually links to the prompt and sometimes the transcript if you want to see how they unfolded. More recently I added a short file to the repo with a few extra tips for my research agents. You can read that here . My preferred definition of AI slop is AI-generated content that is published without human review. I've not been reviewing these reports in great detail myself, and I wouldn't usually publish them online without some serious editing and verification. I want to share the pattern I'm using though, so I decided to keep them quarantined in this one public repository. A tiny feature request for GitHub: I'd love to be able to mark a repository as "exclude from search indexes" such that it gets labelled with tags. I still like to keep AI-generated content out of search, to avoid contributing more to the dead internet . It's pretty easy to get started trying out this coding agent research pattern. Create a free GitHub repository (public or private) and let some agents loose on it and see what happens. You can run agents locally but I find the asynchronous agents to be more convenient - especially as I can run them (or trigger them from my phone) without any fear of them damaging my own machine or leaking any of my private data. Claude Code for web offers a free $250 of credits for their $20/month users for a limited time (until November 18, 2025). Gemini Jules has a free tier . There are plenty of other coding agents you can try out as well. Let me know if your research agents come back with anything interesting! You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options . Code research Coding agents Asynchronous coding agents Give them a dedicated GitHub repository Let them rip with unlimited network access My simonw/research collection This is total slop, of course Try it yourself

0 views
neilzone 3 weeks ago

Using LibreOffice and other Free software for documents as a lawyer

I was asked recently about how I get on using LibreOffice for document-related legal work, and I promised to write down some thoughts. The short answer is that I use a mix of LibreOffice and other FOSS tools, and I’m very positive about what I do and how I do it, with no particular concerns. (I’ve written more broadly about how I use Free software for legal work ; this blogpost is more specific.) This is about my experience. Yours might be different. You might not want to, or be able to, use, or try, LibreOffice (or vim, or git, or whatever). And that’s fine. I’m not trying to convert or persuade anyone. I do a lot of work which entails producing and amending, documents, and exchanging documents with others. This includes contracts, policies and procedures, and collaborative report writing. Occasionally, it means filling in other people’s forms. I use LibreOffice’s Writer for this. I use Writer pretty much every day, and have done for several years, with a wide range of clients and counterparties, including large law firms, small companies, and government departments, and I have no concerns, or significant gripes. I have made templates for my most common types of document, and I have styles set up to make formatting easy and consistent. (I don’t know why people produce documents without styles, but that’s just a personal gripe.) I have exchanged complex documents, usually with lots of tracked changes and comments, with many, many recipients, and I have had no problems with tracked changes, or people not being able to open documents or see what I have done. I’ve had a document recently where automatic numbering had gone wrong, and one where formatting was been messed up, but these were both documents which started life 5+ years ago, and I have not been able to identify whether this was a LibreOffice Writer issue, or a Word (or whatever tool others involved have been using) issue, or something else. In both cases, I fixed them rapidly and got on with things. I don’t know what Word is like recently, but when I last used it a few years ago, I found automatic numbering and formatting were mostly fine but occasionally a pain back then too, so perhaps this is just par for the course. I found Writer’s recent change to dark mode / theming a bit of a pain, but I seem to have resolved it now. For version control and documents, I don’t do anything fancy. I have a script which appends a date and timestamp to the beginning of the file’s name, and this works well. I get a directory of drafts, with clear naming / sequencing. I’ve experimented with git and documents, and while it sort of works to a point, it is not the right approach for me at the moment. Factors which might aid my positive experience: I do a lot of advisory work, where I produce reports, advice notes, and briefing notes. I don’t tend to use LibreOffice for this, preferring instead to use vim, writing in Markdown. For instance, this is how I prepared the new terms of service for mastodon.social / mastodon.online , and, on a friendly basis outside work, a draft vendor agreement for postmarketOS . This means none of the cruft of a document filetype, and it means that I can use git for version control in a way that actually works (unlike with documents). It also makes it easy to produce diffs. But it doesn’t work well for things like cross-referencing; it is not the right tool for the job. If the output needs to be a nicely-formatted PDF, I use pandoc and typst to convert the Markdown using a template . This makes producing a formatted document very easy, while letting me focus on the content. Some clients send and receive plain text / .md files (and, yes, you, who likes LaTeX files :)) and share .diffs, others prefer documents. Both are fine with me and I go with whichever works better for each client or each situation. I do not use Impress, the presentation tool, other than for viewing presentations which are sent to me. Instead, I use reveal.js for presentations , writing in markdown and presenting in my browser. I really like reveal.js. I can easily upload my presentations for people to view , and I can convert them to .pdf for distribution. I’ve not had to work on a collaborative presentation in the last 5+ years; I imagine that I’d have to use Impress, or a client’s hosted tool of choice, if someone wanted that. I use the spreadsheet tool, Calc, when I need a spreadsheet, which is not very often. It is mostly basic accountancy. For my limited uses, Calc has been absolutely fine, and I’m certainly not qualified to comment on it in any detail. Some clients want me to use their choice of hosted tools - Microsoft, Google Docs, Cryptpad, Nextcloud, etherpad, and so on. That’s fine; if a client wants to use them, and gives me access, I use them. All the ones that I’ve tried so far work fine in Firefox. I’m also happy to make PRs to, or commit directly into, a client’s git repositories. Over the past few years, I’ve hosted instances of Collabora (via Nextcloud), Cryptpad, and etherpad. All have had their pros and cons, and perhaps that’s something for a different blogpost. Most recently, I hosted etherpad, but right now, I’m not hosting any of these. I just don’t use them enough. I don’t depend on any third party plug-ins or integrations. I imagine that someone whose work depends on that kind of thing, then Writer might not be a good fit. I don’t do litigation, or anything which requires court filings.

0 views
Rik Huijzer 3 weeks ago

AI Crawler Mitigation via a Few Lines of Caddy

To mitigate AI crawlers, simply require a bit of Javascript: ```caddy domain.com { # Match all requests without a "verified" cookie" @unverified not header Cookie *verified* # Serve them a JS page that sets the cookie handle @unverified { header Content-Type text/html respond <<EOF EOF 418 } # Handle all other requests normally reverse_proxy localhost:3001 } ``` This works because these crawlers have disabl...

0 views
./techtipsy 3 weeks ago

Why Nextcloud feels slow to use

Nextcloud. I really want to like it, but it’s making it really difficult. I like what Nextcloud offers with its feature set and how easily it replaces a bunch of services under one roof (files, calendar, contacts, notes, to-do lists, photos etc.), but no matter how hard I try and how much I optimize its resources on my home server, it feels slow to use, even on hardware that is ranging from decent to good. Then I opened developer tools and found the culprit. It’s the Javascript. On a clean page load, you will be downloading about 15-20 MB of Javascript, which does compress down to about 4-5 MB in transit, but that is still a huge amount of Javascript. For context, I consider 1 MB of Javascript to be on the heavy side for a web page/app. Yes, that Javascript will be cached in the browser for a while, but you will still be executing all of that on each visit to your Nextcloud instance, and that will take a long time due to the sheer amount of code your browser now has to execute on the page. A significant contributor to this heft seems to be the bundle, which based on its name seems to provide some common functionality that’s shared across different Nextcloud apps that one can install. It’s coming in at 4.71 MB at the time of writing. Then you want notifications, right? is here to cover you, at 1.06 MB . Then there are the app-specific views. The Calendar app is taking up 5.94 MB to show a basic calendar view. Files app includes a bunch of individual scripts, such as ( 1.77 MB ), ( 1.17 MB ), ( 1.09 MB ), ( 0.9 MB which I’ve never used!) and many smaller ones. Notes app with its basic bare-bones editor? 4.36 MB for the ! This means that even on an iPhone 13 mini, opening the Tasks app (to-do list), will take a ridiculously long time. Imagine opening your shopping list at the store and having to wait 5-10 seconds before you see anything, even with a solid 5G connection. Sounds extremely annoying, right? I suspect that a lot of this is due to how Nextcloud is architected. There’s bound to be some hefty common libraries and tools that allow app developers to provide a unified experience, but even then there is something seriously wrong with the end result, the functionality to bundle size ratio is way off. As a result, I’ve started branching out some things from Nextcloud, such as replacing the Tasks app with using a private Vikunja instance, and Photos to a private Immich instance. Vikunja is not perfect, but its 1.5 MB of Javascript is an order of magnitude smaller compared to Nextcloud, making it feel incredibly fast in comparison. However, with other functionality I have to admit that the convenience of Nextcloud is enough to dissuade me from replacing it elsewhere, due to the available feature set comparing well to alternatives. I’m sure that there are some legitimate reasons behind the current state, and overworked development teams and volunteers are unfortunately the norm in the industry, but it doesn’t take away the fact that the user experience and accessibility suffers as a result. I’d like to thank Alex Russell for writing about web performance and why it matters, with supporting evidence and actionable advice, it has changed how I view websites and web apps and has pushed me to be better in my own work. I highly suggest reading his content, starting with the performance inequality gap series. It’s educational, insightful and incredibly irritating once you learn how crap most things are and how careless a lot of development teams are towards performance and accessibility.

0 views