Brain Baking 4 days ago

Favourites of December (And a Short 2025 Recap)

A late happy new year to everyone! I almost forgot to publish last month’s favourite (blog) posts, and since last month was the last one of 2025, let’s do a short recap as well. Previous month’s recap: November 2025 . Last year was another eventful year. Browse the full 2025 Brain Baking archive for more juicy details. I selected one post per month that for me stands out: Our son also kicked me out of my cosy home office upstairs. Luckily, our renovations were finished in time, so we moved the living room and I took the old space hostage . One of the advantages of directly staring at a larger window is being able to admire the seasonal view: The window at my desk showcases snowy trees. For 2026, I only wish for one thing: stability . Let’s stop the craziness and try to get things settled down. No more kids, renovations, job changes, broken bicycles, and serious sickness please. Just, you know, breathing. Whoosah . Last month I joined the Advent of Code challenge using Clojure, a language I know absolutely nothing about. Since then I’ve been obsessed with Lisp-based dialects. Forgive me if most of the links below are programming-oriented: it’s been invigorating to learn something new and actually enjoy a programming language for a chance. It’s the reason I’m typing this in Emacs now, although I haven’t even installed CIDER yet. All in due time… Ok that was definitely too much Emacs stuff. The lack of other links shows how much I’ve been obsessed with the editor lately. No other random links for this month! Related topics: / metapost / By Wouter Groeneveld on 10 January 2026.  Reply via email . In January, I had the idea to compile your own philosophy . So far, I have collected lots of notes and summarised too many previous ones, but nothing has been published yet. In February, I shared my stationary drawers . I should really clean out all those fountain pens. In March, I dug up a photo of my first console , the SEGA Genesis/MegaDrive. In April, I learned that my sourdough starter has twins somewhere in Switzerland. In May, more thoughts about writing and publishing popped up. In June, I debunked (or confirmed?) the fact that IT freelancers earn more than their employee counterparts . In July, I got influenced by other board game enthusiasts and admitted to having too many games and too little time . In August, we welcomed our second little one and I turned forty —in that order. Yes, that is important to me. In September, I wrote too many articles about trick taking games and local traditions . In October, I fondly looked back at years of downloading warez software . In November, I recovered my late father-in-law’s 1994 IBM PC invoice . In December, I started shaving Emacs yaks . I haven’t stopped ever since. Nick George reports on building static websites with Clojure . Nathan Marz describes how he invented Specter to fill Clojure’s mutability hole. I don’t understand 90% of the technicalities there, but one day, I will. More Clojure stuff. Sorry… Mikko Koski helped me get started: 8 tips for Advent of Code 2022 in Clojure. A more official one, but just as interesting: the State of Clojure 2024 results . 76% of the people using it build web apps, 40% is on Emacs/CIDER, and Babashka is super popular! This Advent of Code GIF archive is crazy. Victor Dorneanu wrote about his Doom Emacs to Vanilla migration. I tried Doom/Spacemacs for about one whole day and then started back from scratch, but damn, it’s very challenging, even though you can “do what you want”—if you’re an Emacs/Elisp acolyte, that is. I’m planning to get babtized in the Emacs Church very soon. Alice from The Wallflower Digest shares her thoughts about personal curriculums ; a way to get started with deliberate life-long learning. (via Joel , I think?) Karthinks found fifteen ways to use Embark , a wonderful context-aware Emacs package. More “Emacs from scratch” blogs to share: this one’s from Arne and lies out the foundations in case you want to get started. Thanks, Arne. You’re in my RSS feed now. Frank Meeuwsen writes (in Dutch) about AI tooling and how they democratise digital literacy. Or rather, how they should . Gregory J. Stein wrote a guide on email in Emacs using Mu and Mu4e . I have more thoughts on that saved for a separate blog post. If you’d like to know how many Emacs packages you’re currently rocking, Manuel Uberti has an Elisp for you (via Sebastián ) Kristoffer Balintona helped me better understand the Vertico completion-at-point-function stack .

Clojure Programming Writing

Abhinav Sarkar 1 months ago

Solving Advent of Code 2025 in Janet: Days 5–8

I’m solving the Advent of Code 2025 in Janet . After doing the last five years in Haskell, I wanted to learn a new language this year. I’ve been eyeing the “New Lisps” 1 for a while now, and I decided to learn Janet. Janet is a Clojure like Lisp that can be interpreted, embedded and compiled, and comes with a large standard library with concurrency, HTTP and PEG parser support. I want to replace Python with Janet as my scripting language. Here are my solutions for December 5–8. This post was originally published on abhinavsarkar.net . This post is a part of the series: Solving Advent of Code 2025 in Janet . All my solutions follow the same structure because I wrote a template to create new empty solutions. Actually, I added a fair bit of automation this time to build, run, test and benchmark the solutions. Parsing the day 5 input was a bit involved because of the two different formats. Other than that, the function is the most interesting part. Since I sorted the ranges in , I needed to do only one linear scan of ranges, merging the current one with the previous one if possible. The trick here was to be correct about finding overlapping ranges and calculating the merged range. I made multiple mistakes but eventually figured it out. Day 6 was entirely a parsing-based problem, and Janet was well suited to it. Parts 1 and 2 required the input to be parsed differently, so the is parameterized. In part 1, I ignored whitespaces in numbers, while in part 2, they were significant. So I passed two different patterns to parse numbers in and . I had to write the function because it is not built into Janet. Rest of it was straightforward. Notice how I used threading macros to write the computations linearly. I solved part 1 of day 7 by simply folding over the input rows, propagating the beam, and splitting it when required. I used a set of indices to keep track of the current indices at which beam was present. Only tricky thing here was using a dict to simulate a set because Janet does not have sets built-in. That’s what the code is doing. Part 2 was harder. I first wrote a brute-force solution to count the number of paths, but it never finished running. The number of paths is \(O(2^n)\) , and impossible to solve with brute-force. I know that there may be better solutions possible, but I simply added a dict-based cache, and that made it work. Day 8 required me to do several new things. It was immediately clear to me that I needed a Disjoint Set to keep track of the connected points. So I wrote one in object-oriented Janet! Object-orientation in Janet is prototype-based , pretty much like JavaScript. You can see the and methods in the above. I first computed all unique pairs and distances between them, and sorted the pairs by distances. In part 1, I union-ed closest \(k\) pairs, while in part 2, I kept going till all points were connected in one circuit. This worked but it took really long to run: over 600ms. I was not satisfied. After a night’s sleep, I realized that I do not need to sort all pairs but only top \(k\) , where \(k\) is much smaller than total number of pairs (~500000). So I rewrote the function to use a max binary heap that keeps only the closest- \(k\) pairs. The function changed to pass \(k\) as a parameter to , which after a bit of experimentation, I set to 5500. The rest of the functions stayed unchanged. This change provided over 10x speedup, reducing the run time to under 60ms 2 ! You can see the mutable nature of Janet in all its glory in this solution. I had several gotcha moments when I tried to mix higher-order functions—such as , , and —with mutable date structures in Janet. Not only they are confusing, but they also result in slower code because Janet does not have Persistent data-structures like Clojure. Every etc. result in a new array being created. My advice is to not mix functional programming code with procedural programming code in Janet. That’s it for now. Next note will drop after 4 or 5 days. You can browse the code repo to see the full setup. If you have any questions or comments, please leave a comment below. If you liked this post, please share it. Thanks for reading! The new Lisps that interest me are: Janet, Fennel and Jank . ↩︎ You may ask why I didn’t write the max-heap as OO-Janet code. Well, I did and I found that it was 50% slower than the procedural version shown here. I guess the dispatch overhead for methods is too much. ↩︎ This post is a part of the series: Solving Advent of Code 2025 in Janet . If you liked this post, please leave a comment . Days 5–8 👈 The new Lisps that interest me are: Janet, Fennel and Jank . ↩︎ You may ask why I didn’t write the max-heap as OO-Janet code. Well, I did and I found that it was 50% slower than the procedural version shown here. I guess the dispatch overhead for methods is too much. ↩︎ Days 5–8 👈

Programming C# Tutorial Haskell Clojure

the website of jyn 1 months ago

I want a better build executor

This post is part 4/4 of a series about build systems . The market fit is interesting. Git has clearly won, it has all of the mindshare, but since you can use jj to work on Git repositories, it can be adopted incrementally. This is, in my opinion, the only viable way to introduce a new VCS: it has to be able to be partially adopted. If you've worked with other determinism-based systems, one thing they have in common is they feel really fragile, and you have to be careful that you don't do something that breaks the determinism. But in our case, since we've created every level of the stack to support this, we can offload the determinism to the development environment and you can basically write whatever code you want without having to worry about whether it's going to break something. In my last post , I describe an improved build graph serialization. In this post, I describe the build executor that reads those files. Generally, there are three stages to a build: There are a lot more things an executor can do than just spawning processes and showing a progress report! This post explores what those are and sketches a design for a tool that could improve on current executors. Ninja depends on mtimes, which have many issues . Ideally, it would take notes from and look at file attributes, not just the mtime, which eliminates many more false positives. I wrote earlier about querying the build graph . There are two kinds of things you can query: The configuration graph (what bazel calls the target graph ), which shows dependencies between "human meaningful" packages; and the action graph , which shows dependencies between files. Queries on the action graph live in the executor; queries on the configuration graph live in the configure script. For example, / , , and query the configuration graph; and query the action graph. Cargo has no stable way to query the action graph. Note that “querying the graph” is not a binary yes/no. Ninja's query language is much more restricted than Bazel's. Compare Ninja's syntax for querying “the command line for all C++ files used to build the target ” 2 : to Bazel's: Bazel’s language has graph operators, such as union, intersection, and filtering, that let you build up quite complex predicates. Ninja can only express one predicate at a time, with much more limited filtering—but unlike Bazel, allows you to filter to individual parts of the action, like the command line invocation, without needing a full protobuf parser or trying to do text post-processing. I would like to see a query language that combines both these strengths: the same nested predicate structure of Bazel queries, but add a new predicate that takes another predicate as an argument for complex output filtering: We could even go so far as to give this a jq-like syntax: For more complex predicates that have multiple sets as inputs, such as set union and intersection, we could introduce a operator: In my previous post , I talked about two main uses for a tracing build system: first, to automatically add dependency edges for you; and second, to verify at runtime that no dependency edges are missing. This especially shines when the action graph has a way to express negative dependencies, because the tracing system sees every attempted file access and can add them to the graph automatically. For prior art, see the Shake build system . Shake is higher-level than an executor and doesn't work on an action graph, but it has built-in support for file tracing in all three of these modes: warning about incorrect edges; adding new edges to the graph when they're detected at runtime; and finally, fully inferring all edges from the nodes alone . I would want my executor to only support linting and hard errors for missing edges. Inferring a full action graph is scary and IMO belongs in a higher-level tool, and adding dependency edges automatically can be done by a tool that wraps the executor and parses the lints. What's really cool about this linting system is that it allows you to gradually transition to a hermetic build over time, without frontloading all the work to when you switch to the tool. The main downside of tracing is that it's highly non-portable, and in particular is very limited on macOS. One possible alternative I've thought of is to do a buck2-style unsandboxed hermetic builds, where you copy exactly the specified inputs into a tempdir and run the build from the tempdir. If that fails, rerun the build from the main source directory. This can't tell which dependency edges are missing, but it can tell you a dependency is missing without fully failing the build. The downside to that is it assumes command spawning is a pure function, which of course it's not; anything that talks to a socket is trouble because it might be stateful. Tracing environment variable access is … hard. Traditionally access goes through the libc function, but it’s also possible to take an in a main function, in which case accesses are just memory reads. That means we need to trace memory reads somehow. On x86 machines, there’s something called PIN that can do this directly in the CPU without needing compile time instrumentation. On ARM there’s SPE , which is how works, but I’m not sure whether it can be configured to track 100% of memory accesses. I need to do more research here. On Linux, this is all abstracted by . I’m not sure if there’s equivalent wrappers on Windows and macOS. There’s also DynamicRIO , which supports a bunch of platforms, but I believe it works in a similar way to QEMU, by interposing itself between the program and the CPU, which comes with a bunch of overhead. That could work as an opt-in. One last way to do this is with a SIGSEGV signal handler , but that requires that environment variables are in their own page of memory and therefore a linker script. This doesn’t work for environment variables specifically, because they aren’t linker symbols in the normal sense, they get injected by the C runtime . In general, injecting linker scripts means we’re modifying the binaries being run and might cause unexpected build or runtime failures. Here I describe more concretely the tool I want to build, which I’ve named . It would read the constrained clojure action graph serialization format (Magma) that I describe in the previous post; perhaps with a way to automatically convert Ninja files to Magma. Like Ekam , Ronin would have a continuous rebuild mode (but unlike Bazel and Buck2, no background server). Like Shake, It would have runtime tracing, with all of options, to allow gradually transitioning to a hermetic build. And it would have bazel-like querying for the action graph, both through CLI arguments with an jq syntax and through a programmatic API. Finally, it would have pluggable backends for file watching, tracing, stat-ing, progress reporting, and checksums, so that it can take advantage of systems that have more features while still being reasonably fast on systems that don’t. For example, on Windows stats are slow, so it would cache stat info; but on Linux stats are fast so it would just directly make a syscall. Like Ninja, Ronin would keep a command log with a history of past versions of the action graph. It would reuse the bipartite graph structure , with one half being files and the other being commands. It would parse depfiles and dyndeps files just after they’re built, while the cache is still hot. Like , ronin would use a single-pass approach to support early cutoff. It would hash an "input manifest" to decide whether to rebuild. Unlike , it would store a mapping from that hash back to the original manifest so you can query why a rebuild happened. Tracing would be built on top of a FUSE file system that tracked file access. 3 Unlike other build systems I know, state (such as manifest hashes, content hashes, and removed outputs) would be stored in an SQLite database, not in flat files. Kinda. Ronin takes a lot of ideas from buck2. It differs in two major ways: The main advantage of Ronin is that it can slot in underneath existing build systems people are already using—CMake and Meson—without needing changes to your build files at all. In this post I describe what a build executor does, some features I would like to see from an executor (with a special focus on tracing), and a design for a new executor called that allows existing projects generating ninja files to gradually transition to hermetic builds over time, without a “flag day” that requires rewriting the whole build system. I don’t know yet if I will actually build this tool, that seems like a lot of work 5 😄 but it’s something I would like to exist in the world. In many ways Conan profiles are analogous to ninja files: profiles are the interface between Conan and CMake in the same way that ninja files are the interface between CMake and Ninja. Conan is the only tool I'm aware of where the split between the package manager and the configure step is explicit. ↩ This is not an apple to apples comparison; ideally we would name the target by the output file, not by its alias. Unfortunately output names are unpredictable and quite long in Bazel. ↩ macOS does not have native support for FUSE. MacFuse exists but does not support getting the PID of the calling process. A possible workaround would be to start a new FUSE server for each spawned process group. FUSE on Windows is possible through winfsp . ↩ An earlier version of this post read "Buck2 only supports non-hermetic builds for system toolchains , not anything else", which is not correct. ↩ what if i simply took buck2 and hacked it to bits,,, ↩ Resolving and downloading dependencies. The tool that does this is called a package manager . Common examples are , , Conan 1 , and the resolver . Configuring the build based on the host environment and build targets. I am not aware of any common name for this, other than maybe configure script (but there exist many tools for this that are not just shell scripts). Common examples are CMake, Meson, autotools, and the Cargo CLI interface (e.g. and ). Executing a bunch of processes and reporting on their progress. The tool that does this is called a build executor . Common examples are , , , and the phase of . It does not expect to be a top-level build system. It is perfectly happy to read (and encourages) generated files from a higher level configure tool. This allows systems like CMake and Meson to mechanically translate Ninja files into this new format, so builds for existing projects can get nice things. It allows you to gradually transition from non-hermetic to hermetic builds, without forcing you to fix all your rules at once, and with tracing to help you find where you need to make your fixes. Buck2 doesn’t support tracing at all. It technically supports non-hermetic builds, but you don't get many benefits compared to using a different build system, and it's still high cost to switch build systems 4 . In many ways Conan profiles are analogous to ninja files: profiles are the interface between Conan and CMake in the same way that ninja files are the interface between CMake and Ninja. Conan is the only tool I'm aware of where the split between the package manager and the configure step is explicit. ↩ This is not an apple to apples comparison; ideally we would name the target by the output file, not by its alias. Unfortunately output names are unpredictable and quite long in Bazel. ↩ macOS does not have native support for FUSE. MacFuse exists but does not support getting the PID of the calling process. A possible workaround would be to start a new FUSE server for each spawned process group. FUSE on Windows is possible through winfsp . ↩ An earlier version of this post read "Buck2 only supports non-hermetic builds for system toolchains , not anything else", which is not correct. ↩ what if i simply took buck2 and hacked it to bits,,, ↩

Programming Clojure Shell DevOps C++

Abhinav Sarkar 1 months ago

Solving Advent of Code 2025 in Janet: Day 1–4

I’m solving the Advent of Code 2025 in Janet . After doing the last five years in Haskell, I wanted to learn a new language this year. I’ve been eyeing the “New Lisps” 1 for a while now, and I decided to learn Janet. Janet is a Clojure like Lisp that can be interpreted, embedded and compiled, and comes with a large standard library with concurrency, HTTP and PEG parser support. I want to replace Python with Janet as my scripting language. Here are my solutions for Dec 1–4. This post was originally published on abhinavsarkar.net . All my solutions follow the same structure because I wrote a template to create new empty solutions. Actually, I added a fair bit of automation this time to build, run, test and benchmark the solutions. Day 1 was a bit mathy but it didn’t take too long to figure out. I spent more time polishing the solution to be idiomatic Janet code. , the PEG grammar to parse the input was the most interesting part for me on the day. If you know Janet, you can notice this is not the cleanest code, but that’s okay, it was my day 1 too. The most interesting part of the day 2 solution was the macro that reads the input at compile-time and creates a custom function to check whether a number is in one of the given ranges. This turned out to be almost 4x faster than writing the same thing as a function. Notice , the PEG grammar to parse the input. So short and clean! I also leaned into the imperative and mutable nature of the Janet data-structures. The code is still not the cleanest as I was still learning. The first part of day 3 was pretty easy to solve, but using the same solution for the second part just ran forever. I realized that this is a Dynamic Programming problem, but I don’t like doing array-based solutions, so I simply rewrote the solution to add caching. And it worked! It is definitely on the slower side, but I’m okay with it. The code has become a little more idiomatic Janet. Day 4 is when I learned more about Janet control flow structures. The solution for the part 2 is a straightforward Breadth-first traversal . The interesting parts are the , and statements. So concise and elegant! That’s it for now. Next note will drop after 4 or 5 days. You can browse the code repo to see the full setup. If you have any questions or comments, please leave a comment below. If you liked this post, please share it. Thanks for reading! The new Lisps that interest me are: Janet, Fennel and Jank . ↩︎ If you liked this post, please leave a comment . The new Lisps that interest me are: Janet, Fennel and Jank . ↩︎

Haskell Clojure Tutorial Programming C#

the website of jyn 1 months ago

I want a better action graph serialization

This post is part 3/4 of a series about build systems . The next post and last post is I want a better build executor . As someone who ends up getting the ping on "my build is weird" after it has gone through a round of "poke it with a stick", I would really appreciate the mechanisms for [correct dependency edges] rolling out sooner rather than later. In a previous post , I talked about various approaches in the design space of build systems. In this post, I want to zero in on one particular area: action graphs. First, let me define "action graph". If you've ever used CMake, you may know that there are two steps involved: A "configure" step ( ) and a build step ( or ). What I am interested here is what generates , the Makefiles it has created. As the creator of ninja writes , this is a serialization of all build steps at a given moment in time, with the ability to regenerate the graph by rerunning the configure step. This post explores that design space, with the goal of sketching a format that improves on the current state while also enabling incremental adoption. When I say "design space", I mean a serialization format where files are machine-generated by a configure step, and have few enough and restricted enough features that it's possible to make a fast build executor . Not all build systems serialize their action graph. and run persistent servers that store it in memory and allow querying it, but never serialize it to disk. For large graphs, this requires a lot of memory; has actually started serializing parts of its graph to reduce memory usage and startup time . The nix evaluator doesn’t allow querying its graph at all; nix has a very strange model where it never rebuilds because each change to your source files is a new “ input-addressed derivation ” and therefore requires a reconfigure. This is the main reason it’s only used to package software, not as an “inner” build system, because that reconfigure can be very slow. I’ve talked to a couple Nix maintainers and they’ve considered caching parts of the configure step, without caching its outputs (because there are no outputs, other than derivation files!) in order to speed this up. This is much trickier because it requires serializing parts of the evaluator state. Tools that do serialize their graph include CMake, Meson, and the Chrome build system ( GN ). Generally, serializing the graph comes in handy when: In the last post I talked about 4 things one might want from a build system: For a serialization format, we have slightly different constraints. Throughout this post, I'll dive into detail on how these 3 overarching goals apply to the serialization format, and how well various serializations achieve that goal. The first one we'll look at, because it's the default for CMake, is and Makefiles. Make is truly in the Unix spirit: easy to implement 2 , very hard to use correctly. Make is ambiguous , complicated , and makes it very easy to implicitly do a bunch of file system lookups . It supports running shell commands at the top-level, which makes even loading the graph very expensive. It does do pretty well on minimizing reconfigurations, since the language is quite flexible. Ninja is the other generator supported by CMake. Ninja is explicitly intended to work on a serialized action graph; it's the only tool I'm aware of that is. It solves a lot of the problems of Make : it removes many of the ambiguities; it doesn't have any form of globbing; and generally it's a much simpler and smaller language. Unfortunately, Ninja's build file format still has some limitations. First, it has no support for checksums. It's possible to work around that by using and having a wrapper script that doesn't overwrite files unless they've changed, but that's a lot of extra work and is annoying to make portable between operating systems. Ninja files also have trouble expressing correct dependency edges. Let's look at a few examples, one by one. In each of these cases, we either have to reconfigure more often than we wish, or we have no way at all of expressing the dependency edge. See my previous post about negative dependencies. The short version is that build files need to specify not just the files they expect to exist, but also the files they expect not to exist. There's no way to express this in a ninja file, short of reconfiguring every time a directory that might contain a negative dependency is modified, which itself has a lot of downsides. Say that you have a C project with just a . You rename it to and ninja gives you an error that main.c no longer exists. Annoyed of editing ninja files by hand, you decide to write a generator 3 : Note this that this registers an implicit dependency on the current directory. This should automatically detect that you renamed your file and rebuild for you. Oh. Right. Generating build.ninja also modifies the current directory, which creates an infinite loop. It's possible to work around this by putting your C file in a source directory: There's still a problem here, though—did you notice it? Our old target is still lying around. Ninja actually has enough information recorded to fix this: . But it's not run automatically. The other problem is that this approach rebuilds far too often. In this case, we wanted to support renames, so in Ninja's model we need to depend on the whole directory. But that's not what we really depended on—we only care about files. I would like to see a action graph format that has an event-based system, where it says "this file was created, make any changes to the action graph necessary", and cuts the build short if the graph wasn't changed. For flower , I want to go further and support deletions : source files and targets that are optional, that should not fail the build if they aren't present, but should cause a rebuild if they are created, modified, or deleted. Ninja has no way of expressing this. Ninja has no way to express “this node becomes dirty when an environment variable changes”. The closest you can get is hacks with and the checksum wrapper/restat hack, but it’s a pain to express and it gets much worse if you want to depend on multiple variables. At this point, we have a list of constraints for our file format: Ideally, it would even be possible to mechanically translate existing .ninja files to this new format. This sketches out a new format that could improve over Ninja files. It could look something like this: I’d call this language Magma, since it’s the simplest kind of set with closure. Here's a sample action graph in Magma: Note some things about Magma: Kinda. Magma itself has a lot in common with Starlark: it's deterministic, hermetic, immutable, and can be evaluated in parallel. The main difference between the languages themselves is that Clojure has (equivalent to sympy symbolic variables) and Python doesn't. Some of these could be rewritten to keyword arguments, and others could be rewritten to structs, or string keys for a hashmap, or enums; but I'm not sure how much benefit there is to literally using Starlark when these files are being generated by a configure step in any case. Probably it's possible to make a 1-1 mapping between the two in any case. Buck2 has support for metadata that describes how to execute a built artifact. I think this is really interesting; is a much nicer interface than , partly because of shell quoting and word splitting issues, and partly just because it's more discoverable. I don't have a clean idea for how to fit this into a serialization layer. "Don't put it there and use a instead" works , but makes it hard to do things like allow the build graph to say that an artifact needs set or something like that, you end up duplicating the info in both files. Perhaps one option could be to attach a key/value pair to s. Well, yes and no. Yes, in that this has basically all the features of ninja and then some. But no, because the rules here are all carefully constrained to avoid needing to do expensive file I/O to load the build graph. The most expensive new feature is , and it's intended to avoid an even more expensive step (rerunning the configuration step). It's also limited to changed files; it can't do arbitrary globbing on the contents of the directory the way that Make pattern rules can. Note that this also removes some features in ninja: shell commands are gone, process spawning is much less ambiguous, files are no longer parsed automatically. And because this embeds a clojure interpreter, many things that were hard-coded in ninja can instead be library functions: , response files, , . In this post, we have learned some downsides of Make and Ninja's build file formats, sketched out how they could possibly be fixed, and designed a language called Magma that has those characteristics. In the next post, I'll describe the features and design of a tool that evaluates and queries this language. see e.g. this description of how it works in buck2 ↩ at least a basic version—although several of the features of GNU Make get rather complicated. ↩ is https://github.com/ninja-build/ninja/blob/231db65ccf5427b16ff85b3a390a663f3c8a479f/misc/ninja_syntax.py . ↩ technically these aren't true monadic builds because they're constrained a lot more than e.g. Shake rules, they can't fabricate new rules from whole cloth. but they still allow you to add more outputs to the graph at runtime. ↩ This goes all the way around the configuration complexity clock and skips the "DSL" phase to simply give you a real language. ↩ This is totally based and not at all a terrible idea. ↩ This has a whole bunch of problems on Windows, where arguments are passed as a single string instead of an array, and each command has to reimplement its own parsing. But it will work “most” of the time, and at least avoids having to deal with Powershell or CMD quoting. ↩ To make it possible to distinguish the two on the command line, could unambiguously refer to the group, like in Bazel. ↩ You don’t have a persistent server to store it in memory. When you don’t have a server, serializing makes your startup times much faster, because you don’t have to rerun the configure step each time. You don’t have a remote build cache. When you have a remote cache, the rules for loading that cache can be rather complicated because they involve network queries 1 . When you have a local cache, loading it doesn’t require special support because it’s just opening a file. You want to support querying, process spawning, and progress updates without rewriting the logic yourself for every OS (i.e. you don't want to write your own build executor). a "real" language in the configuration step reflection (querying the build graph) file watching support for discovering incorrect dependency edges We care about it being simple and unambiguous to load the graph from the file, so we get fast incremental rebuild speed and graph queries. In particular, we want to touch the filesystem as little as possible while loading. We care about supporting "weird" dependency edges, like dynamic dependencies and the depfiles emitted by a compiler after the first run, so that we're able to support more kinds of builds. And finally, we care about minimizing reconfigurations : we want to be able to express as many things as possible in the action graph so we don't have the pay the cost of rerunning the configure step. This tends to be at odds with fast graph loading; adding features at this level of the stack is very expensive! Negative dependencies File rename dependencies Optional file dependencies Optional checksums to reduce false positives Environment variable dependencies "all the features of ninja" (depfiles, monadic builds through 4 , a statement, order-only dependencies) A very very small clojure subset (just , , EDN , and function calls) for the text itself, no need to make loading the graph harder than necessary 5 . If people really want an equivalent of or I suppose this could learn support for and , but it would have much simpler rules than Clojure's classpath. It would not have support for looping constructs, nor most of clojure's standard library. -inspired dependency edges: (for changes in file attributes), (for changes in the checksum), (for optional dependencies), ; plus our new edge. A input function that can be used anywhere a file path can (e.g. in calls to ) so that the kind of edge does not depend on whether the path is known in advance or not. Runtime functions that determine whether the configure step needs to be re-run based on file watch events 6 . Whether there is actually a file watcher or the build system just calculates a diff on its next invocation is an implementation detail; ideally, one that's easy to slot in and out. “phony” targets would be replaced by a statement. Groups are sets of targets. Groups cannot be used to avoid “input not found” errors; that niche is filled by . Command spawning is specified as an array 7 . No more dependency on shell quoting rules. If people want shell scripts they can put that in their configure script. Redirecting stdout no longer requires bash syntax, it's supported natively with the parameter of . Build parameters can be referred to in rules through the argument. is a thunk ; it only registers an intent to add edges in the future, it does not eagerly require to exist. Our input edge is generalized and can apply to any rule, not just to the configure step. It executes when a file is modified (or if the tool doesn’t support file watching, on each file in the calculated diff in the next tool invocation). Our edge provides the file event type, but not the file contents. This allows ronin to automatically map results to one of the three edge kinds: , , . and are not available through this API. We naturally distinguish between “phony targets” and files because the former are s and the latter are s. No more accidentally failing to build if an file is created. 8 We naturally distinguish between “groups of targets” and “commands that always need to be rerun”; the latter just uses . Data can be transformed in memory using clojure functions without needing a separate process invocation. No more need to use in your build system. see e.g. this description of how it works in buck2 ↩ at least a basic version—although several of the features of GNU Make get rather complicated. ↩ is https://github.com/ninja-build/ninja/blob/231db65ccf5427b16ff85b3a390a663f3c8a479f/misc/ninja_syntax.py . ↩ technically these aren't true monadic builds because they're constrained a lot more than e.g. Shake rules, they can't fabricate new rules from whole cloth. but they still allow you to add more outputs to the graph at runtime. ↩ This goes all the way around the configuration complexity clock and skips the "DSL" phase to simply give you a real language. ↩ This is totally based and not at all a terrible idea. ↩ This has a whole bunch of problems on Windows, where arguments are passed as a single string instead of an array, and each command has to reimplement its own parsing. But it will work “most” of the time, and at least avoids having to deal with Powershell or CMD quoting. ↩ To make it possible to distinguish the two on the command line, could unambiguously refer to the group, like in Bazel. ↩

Programming Performance Clojure Bash DevOps

tonsky.me 1 months ago

Logo: Clojure+

Clojure+ is a project to improve Clojure stdlib.

Clojure Programming Open Source

(think) 1 months ago

Burst-driven Development: My Approach to OSS Projects Maintenance

I’ve been working on OSS projects for almost 15 years now. Things are simple in the beginning - you’ve got a single project, no users to worry about and all the time and the focus in world. Things changed quite a bit for me over the years and today I’m the maintainer of a couple of dozen OSS projects in the realms of Emacs, Clojure and Ruby mostly. People often ask me how I manage to work on so many projects, besides having a day job, that obviously takes up most of my time. My recipe is quite simple and I refer to it as “burst-driven development”. Long ago I’ve realized that it’s totally unsustainable for me to work effectively in parallel on several quite different projects. That’s why I normally keep a closer eye on my bigger projects (e.g. RuboCop, CIDER, Projectile and nREPL), where I try to respond quickly to tickets and PRs, while I typically do (focused) development only on 1-2 projects at a time. There are often (long) periods when I barely check a project, only to suddenly decide to revisit it and hack vigorously on it for several days or weeks. I guess that’s not ideal for the end users, as some of them might feel that I “undermaintain” some (smaller) projects much of the time, but this approach has worked for me very well for quite a while. The time I’ve spent develop OSS projects has taught me that: To illustrate all of the above with some example, let me tell you a bit about copilot.el 0.3 . I became the primary maintainer of about 9 months ago. Initially there were many things about the project that were frustrating to me that I wanted to fix and improve. After a month of relatively focused work I had mostly achieved my initial goals and I’ve put the project on the backburner for a while, although I kept reviewing PRs and thinking about it in the background. Today I remembered I hadn’t done a release there in quite a while and 0.3 was born. Tomorrow I might remember about some features in Projectile that have been in the back of my mind for ages and finally implement them. Or not. I don’t have any planned order in which I revisit my projects - I just go wherever my inspiration (or current problems related the projects) take me. And that’s a wrap. Nothing novel here, but I hope some of you will find it useful to know how do I approach the topic of multi-project maintenance overall. The “job” of the maintainers is sometimes fun, sometimes tiresome and boring, and occasionally it’s quite frustrating. That’s why it’s essential to have a game plan for dealing with it that doesn’t take a heavy toll on you and make you eventually hate the projects that you lovingly developed in the past. Keep hacking! few problems require some immediate action you can’t always have good ideas for how to improve a project sometimes a project is simply mostly done and that’s OK less is more “hammock time” is important

Open Source Ruby Clojure

the website of jyn 2 months ago

the terminal of the future

Terminal internals are a mess. A lot of it is just the way it is because someone made a decision in the 80s and now it’s impossible to change. This is what you have to do to redesign infrastructure. Rich [Hickey] didn't just pile some crap on top of Lisp [when building Clojure]. He took the entire Lisp and moved the whole design at once. At a very very high level, a terminal has four parts: I lied a little bit above. "input" is not just text. It also includes signals that can be sent to the running process. Converting keystrokes to signals is the job of the PTY. Similar, "output" is not just text. It's a stream of ANSI Escape Sequences that can be used by the terminal emulator to display rich formatting. I do some weird things with terminals. However, the amount of hacks I can get up to are pretty limited, because terminals are pretty limited. I won't go into all the ways they're limited, because it's been rehashed many times before . What I want to do instead is imagine what a better terminal can look like. The closest thing to a terminal analog that most people are familiar with is Jupyter Notebook . This offers a lot of cool features that are not possible in a "traditional" VT100 emulator: high fidelity image rendering a "rerun from start" button (or rerun the current command; or rerun only a single past command) that replaces past output instead of appending to it "views" of source code and output that can be rewritten in place (e.g. markdown can be viewed either as source or as rendered HTML) a built-in editor with syntax highlighting, tabs, panes, mouse support, etc. Jupyter works by having a "kernel" (in this case, a python interpreter) and a "renderer" (in this case, a web application displayed by the browser). You could imagine using a Jupyter Notebook with a shell as the kernel, so that you get all the nice features of Jupyter when running shell commands. However, that quickly runs into some issues: It turns out all these problems are solveable. There exists today a terminal called Warp . Warp has built native integration between the terminal and the shell, where the terminal understands where each command starts and stops, what it outputs, and what is your own input. As a result, it can render things very prettily: It does this using (mostly) standard features built-in to the terminal and shell (a custom DCS): you can read their explanation here . It's possible to do this less invasively using OSC 133 escape codes ; I'm not sure why Warp didn't do this, but that's ok. iTerm2 does a similar thing, and this allows it to enable really quite a lot of features : navigating between commands with a single hotkey; notifying you when a command finishes running, showing the current command as an "overlay" if the output goes off the screen. This is really three different things. The first is interacting with a long-lived process. The second is suspending the process without killing it. The third is disconnecting from the process, in such a way that the process state is not disturbed and is still available if you want to reconnect. To interact with a process, you need bidirectional communication, i.e. you need a "cell output" that is also an input. An example would be any TUI, like , , or 1 . Fortunately, Jupyter is really good at this! The whole design is around having interactive outputs that you can change and update. Additionally, I would expect my terminal to always have a "free input cell", as Matklad describes in A Better Shell , where the interactive process runs in the top half of the window and an input cell is available in the bottom half. Jupyter can do this today, but "add a cell" is manual, not automatic. "Suspending" a process is usually called " job control ". There's not too much to talk about here, except that I would expect a "modern" terminal to show me all suspended and background processes as a de-emphasized persistent visual, kinda like how Intellij will show you "indexing ..." in the bottom taskbar. There are roughly three existing approaches for disconnecting and reconnecting to a terminal session (Well, four if you count reptyr ). Tmux / Zellij / Screen These tools inject a whole extra terminal emulator between your terminal emulator and the program. They work by having a "server" which actually owns the PTY and renders the output, and a "client" that displays the output to your "real" terminal emulator. This model lets you detach clients, reattach them later, or even attach multiple clients at once. You can think of this as a "batteries-included" approach. It also has the benefit that you can program both the client and the server (although many modern terminals, like Kitty and Wezterm are programmable now); that you can organize your tabs and windows in the terminal (although many modern desktop environments have tiling and thorough keyboard shortcuts); and that you get street cred for looking like Hackerman. The downside is that, well, now you have an extra terminal emulator running in your terminal, with all the bugs that implies . iTerm actually avoids this by bypassing the tmux client altogether and acting as its own client that talks directly to the server. In this mode, "tmux tabs" are actually iTerm tabs, "tmux panes" are iTerm panes, and so on. This is a good model, and I would adopt it when writing a future terminal for integration with existing tmux setups. Mosh is a really interesting place in the design space. It is not a terminal emulator replacement; instead it is an ssh replacement. Its big draw is that it supports reconnecting to your terminal session after a network interruption. It does that by running a state machine on the server and replaying an incremental diff of the viewport to the client . This is a similar model to tmux, except that it doesn't support the "multiplexing" part (it expects your terminal emulator to handle that), nor scrollback (ditto). Because it has its own renderer, it has a similar class of bugs to tmux . One feature it does have, unlike tmux, is that the "client" is really running on your side of the network, so local line editing is instant. alden / shpool / dtach / abduco / diss These all occupy a similar place in the design space: they only handle session detach/resume with a client/server, not networking or scrollback, and do not include their own terminal emulator. Compared to tmux and mosh, they are highly decoupled. I'm going to treat these together because the solution is the same: dataflow tracking. Take as an example pluto.jl , which does this today by hooking into the Julia compiler. Note that this updates cells live in response to previous cells that they depend on. Not pictured is that it doesn't update cells if their dependencies haven't changed. You can think of this as a spreadsheet-like Jupyter, where code is only rerun when necessary. You may say this is hard to generalize. The trick here is orthogonal persistence . If you sandbox the processes, track all IO, and prevent things that are "too weird" unless they're talking to other processes in the sandbox (e.g. unix sockets and POST requests), you have really quite a lot of control over the process! This lets you treat it as a pure function of its inputs, where its inputs are "the whole file system, all environment variables, and all process attributes". Once you have these primitives—Jupyter notebook frontends, undo/redo, automatic rerun, persistence, and shell integration—you can build really quite a lot on top. And you can build it incrementally, piece-by-piece: jyn, you may say, you can't build vertical integration in open source . you can't make money off open source projects . the switching costs are too high . All these things are true. To talk about how this is possible, we have to talk about incremental adoption. if I were building this, I would do it in stages, such that at each stage the thing is an improvement over its alternatives. This is how works and it works extremely well: it doesn't require everyone on a team to switch at once because individual people can use , even for single commands, without a large impact on everyone else. When people think of redesigning the terminal, they always think of redesigning the terminal emulator . This is exactly the wrong place to start. People are attached to their emulators. They configure them, they make them look nice, they use their keybindings. There is a high switching cost to switching emulators because everything affects everything else . It's not so terribly high, because it's still individual and not shared across a team, but still high. What I would do instead is start at the CLI layer. CLI programs are great because they're easy to install and run and have very low switching costs: you can use them one-off without changing your whole workflow. So, I would write a CLI that implements transactional semantics for the terminal . You can imagine an interface something like , where everything run after is undoable. There is a lot you can do with this alone, I think you could build a whole business off this. Once I had transactional semantics, I would try to decouple persistence from tmux and mosh. To get PTY persistence, you have to introduce a client/server model, because the kernel really really expects both sides of a PTY to always be connected. Using commands like alden , or a library like it (it's not that complicated), lets you do this simply, without affecting the terminal emulator nor the programs running inside the PTY session. To get scrollback, the server could save input and output indefinitely and replay them when the client reconnects. This gets you "native" scrollback—the terminal emulator you're already using handles it exactly like any other output, because it looks exactly like any other output—while still being replayable and resumable from an arbitrary starting point. This requires some amount of parsing ANSI escape codes 2 , but it's doable with enough work. To get network resumption like mosh, my custom server could use Eternal TCP (possibly built on top of QUIC for efficiency). Notably, the persistence for the PTY is separate from the persistence for the network connection. Eternal TCP here is strictly an optimization: you could build this on top of a bash script that runs in a loop, it's just not as nice an experience because of network delay and packet loss. Again, composable parts allow for incremental adoption. At this point, you're already able to connect multiple clients to a single terminal session, like tmux, but window management is still done by your terminal emulator, not by the client/server. If you wanted to have window management integrated, the terminal emulator could speak the tmux -CC protocol, like iTerm. All parts of this stage can be done independently and in parallel from the transactional semantics, but I don't think you can build a business off them, it's not enough of an improvement over the existing tools. This bit depends on the client/server model. Once you have a server interposed between the terminal emulator and the client, you can start doing really funny things like tagging I/O with metadata. This lets all data be timestamped 3 and lets you distinguish input from output. xterm.js works something like this. When combined with shell integration, this even lets you distinguish shell prompts from program output, at the data layer. Now you can start doing really funny things, because you have a structured log of your terminal session. You can replay the log as a recording, like asciinema 4 ; you can transform the shell prompt without rerunning all the commands; you can import it into a Jupyter Notebook or Atuin Desktop ; you can save the commands and rerun them later as a script. Your terminal is data. This is the very first time that we touch the terminal emulator, and it's intentionally the last step because it has the highest switching costs. This makes use of all the nice features we've built to give you a nice UI. You don't need our CLI anymore unless you want nested transactions, because your whole terminal session starts in a transaction by default. You get all the features I mention above , because we've put all the pieces together. This is bold and ambitious and I think building the whole thing would take about a decade. That's ok. I'm patient. You can help me by spreading the word :) Perhaps this post will inspire someone to start building this themselves. there are a lot of complications here around alternate mode , but I'm just going to skip over those for now. A simple way to handle alternate mode (that doesn't get you nice things) is just to embed a raw terminal in the output cell. ↩ otherwise you could start replaying output from inside an escape, which is not good . I had a detailed email exchange about this with the alden author which I have not yet had time to write up into a blog post; most of the complication comes when you want to avoid replaying the entire history and only want to replay the visible viewport. ↩ hey, this seems awfully like asciinema ! ↩ oh, that's why it seemed like asciinema. ↩ The " terminal emulator ", which is a program that renders a grid-like structure to your graphical display. The " pseudo-terminal " (PTY), which is a connection between the terminal emulator and a "process group" which receives input. This is not a program. This is a piece of state in the kernel. The "shell", which is a program that leads the "process group", reads and parses input, spawns processes, and generally acts as an event loop. Most environments use bash as the default shell. The programs spawned by your shell, which interact with all of the above in order to receive input and send output. high fidelity image rendering a "rerun from start" button (or rerun the current command; or rerun only a single past command) that replaces past output instead of appending to it "views" of source code and output that can be rewritten in place (e.g. markdown can be viewed either as source or as rendered HTML) a built-in editor with syntax highlighting, tabs, panes, mouse support, etc. Your shell gets the commands all at once, not character-by-character, so tab-complete, syntax highlighting, and autosuggestions don't work. What do you do about long-lived processes? By default, Jupyter runs a cell until completion; you can cancel it, but you can't suspend, resume, interact with, nor view a process while it's running. Don't even think about running or . The "rerun cell" buttons do horrible things to the state of your computer (normal Jupyter kernels have this problem too, but "rerun all" works better when the commands don't usually include ). Undo/redo do not work. (They don't work in a normal terminal either, but people attempt to use them more when it looks like they should be able to.) Tmux / Zellij / Screen These tools inject a whole extra terminal emulator between your terminal emulator and the program. They work by having a "server" which actually owns the PTY and renders the output, and a "client" that displays the output to your "real" terminal emulator. This model lets you detach clients, reattach them later, or even attach multiple clients at once. You can think of this as a "batteries-included" approach. It also has the benefit that you can program both the client and the server (although many modern terminals, like Kitty and Wezterm are programmable now); that you can organize your tabs and windows in the terminal (although many modern desktop environments have tiling and thorough keyboard shortcuts); and that you get street cred for looking like Hackerman. The downside is that, well, now you have an extra terminal emulator running in your terminal, with all the bugs that implies . iTerm actually avoids this by bypassing the tmux client altogether and acting as its own client that talks directly to the server. In this mode, "tmux tabs" are actually iTerm tabs, "tmux panes" are iTerm panes, and so on. This is a good model, and I would adopt it when writing a future terminal for integration with existing tmux setups. Mosh Mosh is a really interesting place in the design space. It is not a terminal emulator replacement; instead it is an ssh replacement. Its big draw is that it supports reconnecting to your terminal session after a network interruption. It does that by running a state machine on the server and replaying an incremental diff of the viewport to the client . This is a similar model to tmux, except that it doesn't support the "multiplexing" part (it expects your terminal emulator to handle that), nor scrollback (ditto). Because it has its own renderer, it has a similar class of bugs to tmux . One feature it does have, unlike tmux, is that the "client" is really running on your side of the network, so local line editing is instant. alden / shpool / dtach / abduco / diss These all occupy a similar place in the design space: they only handle session detach/resume with a client/server, not networking or scrollback, and do not include their own terminal emulator. Compared to tmux and mosh, they are highly decoupled. Runbooks (actually, you can build these just with Jupyter and a PTY primitive). Terminal customization that uses normal CSS, no weird custom languages or ANSI color codes. Search for commands by output/timestamp. Currently, you can search across output in the current session, or you can search across all command input history, but you don't have any kind of smart filters, and the output doesn't persist across sessions. Timestamps and execution duration for each command. Local line-editing, even across a network boundary. IntelliSense for shell commands , without having to hit tab and with rendering that's integrated into the terminal. " All the features from sandboxed tracing ": collaborative terminals, querying files modified by a command, "asciinema but you can edit it at runtime", tracing build systems. Extend the smart search above to also search by disk state at the time the command was run. Extending undo/redo to a git-like branching model (something like this is already support by emacs undo-tree ), where you have multiple "views" of the process tree. Given the undo-tree model, and since we have sandboxing, we can give an LLM access to your project, and run many of them in parallel at the same time without overwriting each others state, and in such a way that you can see what they're doing, edit it, and save it into a runbook for later use. A terminal in a prod environment that can't affect the state of the machine, only inspect the existing state. Gary Bernhardt, “A Whole New World” Alex Kladov, “A Better Shell” jyn, “how i use my terminal” jyn, “Complected and Orthogonal Persistence” jyn, “you are in a box” jyn, “there's two costs to making money off an open source project…” Rebecca Turner, “Vertical Integration is the Only Thing That Matters” Julia Evans, “New zine: The Secret Rules of the Terminal” Julia Evans, “meet the terminal emulator” Julia Evans, “What happens when you press a key in your terminal?” Julia Evans, “What's involved in getting a "modern" terminal setup?” Julia Evans, “Bash scripting quirks & safety tips” Julia Evans, “Some terminal frustrations” Julia Evans, “Reasons to use your shell's job control” “signal(7) - Miscellaneous Information Manual” Christian Petersen, “ANSI Escape Codes” saoirse, “withoutboats/notty: A new kind of terminal” Jupyter Team, “Project Jupyter Documentation” “Warp: The Agentic Development Environment” “Warp: How Warp Works” “Warp: Completions” George Nachman, “iTerm2: Proprietary Escape Codes” George Nachman, “iTerm2: Shell Integration” George Nachman, “iTerm2: tmux Integration” Project Jupyter, “Jupyter Widgets” Nelson Elhage, “nelhage/reptyr: Reparent a running program to a new terminal” Kovid Goyal, “kitty” Kovid Goyal, “kitty - Frequently Asked Questions” Wez Furlong, “Wezterm” Keith Winstein, “Mosh: the mobile shell” Keith Winstein, “Display errors with certain characters Matthew Skala, “alden: detachable terminal sessions without breaking scrollback” Ethan Pailes, “shell-pool/shpool: Think tmux, then aim... lower” Ned T. Crigler, “crigler/dtach: A simple program that emulates the detach feature of screen” Marc André Tanner, “martanne/abduco: abduco provides session management” yazgoo, “yazgoo/diss: dtach-like program / crate in rust” Fons van der Plas, “Pluto.jl — interactive Julia programming environment” Ellie Huxtable, “Atuin Desktop: Runbooks that Run” Toby Cubitt, “undo-tree” “SIGHUP - Wikipedia” Jason Gauci, “How Eternal Terminal Works” Marcin Kulik, “Record and share your terminal sessions, the simple way - asciinema.org” “Alternate Screen | Ratatui” there are a lot of complications here around alternate mode , but I'm just going to skip over those for now. A simple way to handle alternate mode (that doesn't get you nice things) is just to embed a raw terminal in the output cell. ↩ otherwise you could start replaying output from inside an escape, which is not good . I had a detailed email exchange about this with the alden author which I have not yet had time to write up into a blog post; most of the complication comes when you want to avoid replaying the entire history and only want to replay the visible viewport. ↩ hey, this seems awfully like asciinema ! ↩ oh, that's why it seemed like asciinema. ↩

CSS Bash Programming Clojure

the website of jyn 2 months ago

build system tradeoffs

This post is part 1/4 of a series about build systems . The next post is negative build dependencies . If I am even TEMPTED to use , in my goddamn build system, you have lost. I am currently employed to work on the build system for the Rust compiler (often called or ). As a result, I think about a lot of build system weirdness that most people don't have to. This post aims to give an overview of what builds for complicated projects have to think about, as well as vaguely gesture in the direction of build system ideas that I like. This post is generally designed to be accessible to the working programmer, but I have a lot of expert blindness in this area, and sometimes assume that "of course people know what a feldspar is!" . Apologies in advance if it's hard to follow. What makes a project’s build complicated? The first semi-complicated thing people usually want to do in their build is write an integration test. Here's a rust program which does so: This instructs cargo to, when you run , compile as a standalone program and run it, with as the entrypoint. We'll come back to this program several times in this post. For now, notice that we are invoking inside of . I actually forgot this one in the first draft because Cargo solves this so well in the common case 1 . In many hand-written builds ( cough cough ), specifying dependencies by hand is very broken, for parallelism simply doesn't work, and running on errors is common. Needless to say, this is a bad experience. The next step up in complexity is to cross-compile code. At this point, we already start to get some idea of how involved things get: How hard it is to cross-compile code depends greatly on not just the build system, but the language you're using and the exact platform you're targeting. The particular thing I want to point out is your standard library has to come from somewhere . In Rust, it's usually downloaded from the same place as the compiler. In bytecode and interpreted languages, like Java, JavaScript, and Python, there's no concept of cross-compilation because there is only one possible target. In C, you usually don't install the library itself, but only the headers that record the API 2 . That brings us to our next topic: Generally, people refer to one of two things when they say "libc". Either they mean the C standard library, , or the C runtime, . Libc matters a lot for two reasons. Firstly, C is no longer a language , so generally the first step to porting any language to a new platform is to make sure you have a C toolchain 3 . Secondly, because libc is effectively the interface to a platform, Windows , macOS , and OpenBSD have no stable syscall boundary—you are only allowed to talk to the kernel through their stable libraries (libc, and in the case of Windows several others too). To talk about why they've done this, we have to talk about: Many languages have a concept of " early binding ", where all variable and function references are resolved at compile time, and " late binding ", where they are resolved at runtime. C has this concept too, but it calls it "linking" instead of "binding". "late binding" is called "dynamic linking" 4 . References to late-bound variables are resolved by the "dynamic loader" at program startup. Further binding can be done at runtime using and friends. Platform maintainers really like dynamic linking , for the same reason they dislike vendoring : late-binding allows them to update a library for all applications on a system at once. This matters a lot for security disclosures, where there is a very short timeline between when a vulnerability is patched and announced and when attackers start exploiting it in the wild. Application developers dislike dynamic linking for basically the same reason: it requires them to trust the platform maintainers to do a good job packaging all their dependencies, and it results in their application being deployed in scenarios that they haven't considered or tested . For example, installing openssl on Windows is really quite hard. Actually, while I was writing this, a friend overheard me say "dynamically linking openssl" and said "oh god you're giving me nightmares". Perhaps a good way to think about dynamically linking as commonly used is a mechanism for devendoring libraries in a compiled program . Dynamic linking has other use cases, but they are comparatively rare. Whether a build system (or language) makes it easy or hard to dynamically link a program is one of the major things that distinguishes it. More about that later. Ok. So. Back to cross-compiling. To cross-compile a program, you need: Where does your toolchain come from? ... ... ... ... It turns out this is a hard problem . Most build systems sidestep it by "not worrying about it"; basically any Makefile you find is horribly broken if you update your compiler without running afterwards. Cargo is a lot smarter—it caches output in , and rebuilds if changes. "How do you deal with toolchain invalidations" is another important things that distinguishes a build system, as we'll see later. toolchains are a special case of a more general problem: your build depends on your whole build environment , not just the files passed as inputs to your compiler. That means, for instance, that people can—and often do—download things off the internet, embed previous build artifacts in later ones , and run entire nested compiler invocations . Once we get towards these higher levels of complexity, people want to start doing quite complicated things with caching. In order for caching to be sound, we need the same invocation of the compiler to emit the same output every time, which is called a reproducible build . This is much harder than it sounds! There are many things programs do that cause non-determinism that programmers often don’t think about (for example, iterating a hashmap or a directory listing). At the very highest end, people want to conduct builds across multiple machines, and combine those artifacts. At this point, we can’t even allow reading absolute paths, since those will be different between machines. The common tool for this is a compiler flag called , and allows the build system to map an absolute path to a relative one. is also how rustc is able to print the sources of the standard library when emitting diagnostics, even when running on a different machine than where it was built. At this point, we have enough information to start talking about the space of tradeoffs for a build system. Putting your config in a YAML file does not make it declarative! Limiting yourself to a Turing-incomplete language does not automatically make your code easier to read! — jyn The most common unforced error I see build systems making is forcing the build configuration to be written in a custom language 6 . There are basically two reasons they do this: Right. So, given that making a build "declarative" is a lie, you may as well give programmers a real language. Some common choices are: "But wait, jyn!", you may say. "Surely you aren't suggesting a build system where you have to run a whole program every time you figure out what to rebuild??" I mean ... people are doing it. But just because they're doing it doesn't mean it's a good idea, so let's look at the alternative, which is to serialize your build graph . This is easier to see than explain, so let's look at an example using the Ninja build system 8 : Ninjafiles give you the absolute bare minimum necessary to express your build dependencies: You get "rules", which explain how to build an output; "build edges", which state when to build; and "variables", which say what to build 9 . That's basically it. There's some subtleties about "depfiles" which can be used to dynamically add build edges while running the build rule. Because the features are so minimal, the files are intended to be generated, using a configure script written in one of the languages we talked about earlier. The most common generators are CMake and GN , but you can use any language you like because the format is so simple. What's really cool about this is that it's trivial to parse, which means that it's very easy to write your own implementation of ninja if you want. It also means that you can get a lot of the properties we discussed before, i.e.: It turns out these properties are very useful. "jyn, you're taking too long to get to the point!" look I’m getting there, I promise. The main downsides to this approach is that it has to be possible to serialize your build graph. In one sense, I see this as good, actually, because you have to think through everything your build does ahead of time. But on the other hand, if you have things like nested ninja invocations, or like our -> example from earlier 10 , all the tools to query the build graph don't include the information you expect. Re-stat'ing all files in a source directory is expensive. It would be much nicer if we could have a pull model instead of a push model, where the build tool gets notified of file changes and rebuilds exactly the necessary files. There are some tools with native integration for this, like Tup , Ekam , jj , and Buck2 , but generally it's pretty rare. That's ok if we have reflection, though! We can write our own file monitoring tool, ask the build system which files need to rebuilt for the changed inputs, and then tell it to rebuild only those files. That prevents it from having to recursively stat all files in the graph. See Tup's paper for more information about the big idea here. Ok, let's assume we have some build system that uses some programming language to generate a build graph, and it rebuilds exactly the necessary outputs on changes to our inputs. What exactly are our inputs? There are basically four major approaches to dependency tracking in the build space. This kind of build system externalizes all concerns out to you, the programmer. When I say "externalizes all concerns", I mean that you are required to write in all your dependencies by hand, and the tool doesn't help you get them right. Some examples: A common problem with this category of build system is that people forget to mark the build rule itself as an input to the graph, resulting in dead artifacts left laying around, and as a result, unsound builds. In my humble opinion, this kind of tool is only useful as a serialization layer for a build graph, or if you have no other choice. Here's a nickel, kid, get yourself a better build system . Sometimes build systems (CMake, Cargo, maybe others I don't know) do a little better and use the compiler's built-in support for dependency tracking (e.g. or ), and automatically add dependencies on the build rules themselves. This is a lot better than nothing, and much more reliable than tracking dependencies by hand. But it still fundamentally trusts the compiler to be correct, and doesn't track environment dependencies. This kind of build system always does a full build, and lets you modify the environment in arbitrary ways as you do so. This is simple, always correct, and expensive. Some examples: These are ok if you can afford them. But they are expensive! Most people using Github Actions are only doing so because GHA is a hyperscaler giving away free CI time like there's no tomorrow. I suspect we would see far less wasted CPU-hours if people had to consider the actual costs of using them. This is the kind of phrase that's well-known to people who work on build systems and basically unheard of outside it, alongside "monadic builds". "hermetic" means that the only things in your environment are those you have explicitly put there . This sometimes called "sandboxing", although that has unfortunate connotations about security that don't always apply here. Some examples of this: This has a lot of benefits! It statically guarantees that you cannot forget any of your inputs; it is 100% reliable, assuming no issues with the network or with the implementing tool 🙃; and it gives you very very granular insight into what your dependencies actually are. Some things you can do with a hermetic build system: The main downside is that you have to actually specify all those dependencies (if you don't, you get a hard error instead of an unsound build graph, which is the main difference between hermetic systems and "not my problem"). Bazel and Buck2 give you starlark, so you have a ~real 11 language in which to do it, but it's still a ton of work. Both have an enormous "prelude" module that just defines where you get a compiler toolchain from 12 . Nix can be thought of as taking this "prelude" idea all the way, by expanding the "prelude" (nixpkgs) to "everything that's ever been packaged for NixOS". When you write , your nix build is logically in the same build graph as the nixpkgs monorepo; it just happens to have an enormous remote cache already pre-built. Bazel and Buck2 don’t have anything like nixpkgs, which is the main reason that using them requires a full time dedicated build engineer: that engineer has to keep writing build rules from scratch any time you add an external dependency. They also have to package any language toolchains that aren’t in the prelude. Nix has one more interesting property, which is that all its packages compose. You can install two different versions of the same package and that's fine because they use different store paths. They fit together like lesbians' fingers interlock. Compare this to docker, which does not compose 13 . In docker, there is no way to say "Inherit the build environment from multiple different source images". The closest you can get is a "multi-stage build", where you explicitly copy over individual files from an earlier image to a later image. It can't blindly copy over all the files because some of them might want to end up at the same path, and touching fingers would be gay. The last kind I'm aware of, and the rarest I've seen, is tracing build systems. These have the same goal as hermetic build systems: they still want 100% of your dependencies to be specified. But they go about it in a different way. Rather than sandboxing your code and only allowing access to the dependencies you specify, they instrument your code, tracing its file accesses, and record the dependencies of each build step. Some examples: The advantage of these is that you get all the benefits of a hermetic build system without any of the cost of having to write out your dependencies. The first main disadvantage is that they require the kernel to support syscall tracing, which essentially means they only work on Linux. I have Ideas™ for how to get this working on macOS without disabling SIP, but they're still incomplete and not fully general; I may write a follow-up post about that. I don't yet have ideas for how this could work on Windows, but it seems possible . The second main disadvantage is that not knowing the graph up front causes many issues for the build system. In particular: I have been convinced that tracing is useful as a tool to generate your build graph, but not as a tool actually used when executing it. Compare also gazelle , which is something like that for Bazel, but based on parsing source files rather than tracking syscalls. Combining paradigms in this way also make it possible to verify your hermetic builds in ways that are hard to do with mere sandboxing. For example, a tracing build system can catch missing dependencies: and it can also detect non-reproducible builds: There's more to talk about here—how build systems affect the dynamics between upstream maintainers and distro packagers; how .a files are bad file formats ; how mtime comparisons are generally bad ; how configuration options make the tradeoffs much more complicated; how FUSE can let a build system integrate with a VCS to avoid downloading unnecessary files into a shallow checkout; but this post is quite long enough already. the uncommon case mostly looks like incremental bugs in rustc itself , or issues around rerunning build scripts. ↩ see this stackexchange post for more discussion about the tradeoffs between forward declarations and requiring full access to the source. ↩ even Rust depends on crt1.o when linking ! ↩ early binding is called "static linking". ↩ actually, Zig solved this in the funniest way possible , by bundling a C toolchain with their Zig compiler. This is a legitimately quite impressive feat. If there's any Zig contributors reading—props to you, you did a great job. ↩ almost every build system does this, so I don't even feel compelled to name names. ↩ Starlark is not tied to hermetic build systems. The fact that the only common uses of it are in hermetic build systems is unfortunate. ↩ H.T. Julia Evans ↩ actually variables are more general than this, but for $in and $out this is true. ↩ another example is "rebuilding build.ninja when the build graph changes". it's more common than you think because the language is so limited that it's easier to rerun the configure script than to try and fit the dependency info into the graph. ↩ not actually turing-complete ↩ I have been informed that the open-source version of Bazel is not actually hermetic-by-default inside of its prelude, and just uses system libraries. This is quite unfortunate; with this method of using Bazel you are getting a lot of the downsides and little of the upside. Most people I know using it are doing so in the hermetic mode. ↩ there's something called , but it composes containers, not images. ↩ A compiler for that target. If you're using clang or Rust, this is as simple as passing . If you're using gcc, you need a whole-ass extra compiler installed . A standard library for that target. This is very language-specific, but at a minimum requires a working C toolchain 5 . A linker for that target. This is usually shipped with the compiler, but I mention it specifically because it's usually the most platform specific part. For example, "not having the macOS linker" is the reason that cross-compiling to macOS is hard . Programmers aren't used to treating build system code as code. This is a culture issue that's hard to change, but it's worthwhile to try anyway. There is some idea of making builds "declarative". (In fact, observant readers may observe that this corresponds to the idea of "constrained languages" I talk about in an earlier post.) This is not by itself a bad idea! The problem is it doesn't give them the properties you might want. For example, one property you might want is "another tool can reimplement the build algorithm". Unfortunately this quickly becomes infeasible for complicated algorithms . Another you might want is "what will rebuild next time a build occurs?". You can't get this from the configuration without—again—reimplementing the algorithm. Starlark 7 “the same language that the build system was written in” (examples: Clojure , Zig , JavaScript ) "Show me all commands that are run on a full build" ( ) "Show me all commands that will be run the next time an incremental build is run" ( ) "If this particular source file is changed, what will need to be rebuilt?" ( ) Github Actions cache actions (and in general, most CI caching I'm aware of requires you to manually write out the files your cache depends on) Ansible playbooks Basically most build systems, this is extremely common (kinda—assuming that is reliable, which it often isn't.) Github Actions ( rules) Docker containers (time between startup and shutdown) Initramfs (time between initial load and chroot into the full system) Systemd service startup rules Most compiler invocations (e.g. ) Dockerfiles (more generally, OCI images) Bazel / Buck2 "Bazel Remote Execution Protocol" (not actually tied to Bazel), which lets you run an arbitrary set of build commands on a remote worker On a change to some source code, rerun only the affected tests. You know statically which those are, because the build tool forced you to write out the dependency edges. Remote caching. If you have the same environment everywhere, you can upload your cache to the cloud, and download and reuse it again on another machine. You can do this in CI—but you can also do it locally! The time for a """full build""" can be almost instantaneous because when a new engineer gets onboarded they can immediately reuse everyone else's build cache. The rust compiler, actually "A build system with orthogonal persistence" ( previously ; previously ; previously ) If you change the graph, it doesn't find out until the next time it reruns a build. This can lead to degenerate cases where the same rule has to be run multiple times until it doesn't access any new inputs. If you don't cache the graph, you have that problem on every edge in the graph . This is the problem Ekam has , and makes it very slow to run full builds. Its solution is to run in "watch" mode, where it caches the graph in-memory instead of on-disk. If you do cache the graph, you can only do so for so long before it becomes prohibitively expensive to do that for all possible executions. For "normal" codebases this isn't a problem, but if you're Google or Facebook, this is actually a practical concern. I think it is still possible to do this with a tracing build system (by having your cache points look a lot more like many Bazel BUILD files than a single top-level Ninja file), but no one has ever tried it at that scale. If the same file can come from many possible places, due to multiple search paths (e.g. a include header in C, or any import really in a JVM language), then you have a very rough time specifying what your dependencies actually are. The best ninja can do is say “depend on the whole directory containing that file”, which sucks because it rebuilds whenever that directory changes, not just when your new file is added. It’s possible to work around this with a (theoretical) serialization format other than Ninja, but regardless, you’re adding lots of file s to your hot path. The build system does not know which dependencies are direct (specified by you, the owner of the module being compiled) and which are transient (specified by the modules you depend on). This makes error reporting worse, and generally lets you do fewer kinds of queries on the graph. My friend Alexis Hunt, a build system expert, says "there are deeper pathologies down that route of madness". So. That's concerning. emitting untracked outputs overwriting source files (!), using an input file that was registered for a different rule reading the current time, or absolute path to the current directory iterating all files in a directory (this is non-deterministic) machine and kernel-level sources of randomness. Most build systems do not prioritize correctness. Prioritizing correctness comes with severe, hard to avoid tradeoffs. Tracing build systems show the potential to avoid some of those tradeoffs, but are highly platform specific and come with tradeoffs of their own at large enough scale. Combining a tracing build system with a hermetic build system seems like the best of both worlds. Writing build rules in a "normal" (but constrained) programming language, then serializing them to a build graph, has surprisingly few tradeoffs. I'm not sure why more build systems don't do this. Alan Dipert, Micha Niskin, Joshua Smith, “Boot: build tooling for Clojure” Alexis Hunt, Ola Rozenfield, and Adrian Ludwin, “bazelbuild/remote-apis: An API for caching and execution of actions on a remote system.” Andrew Kelley, “zig cc: a Powerful Drop-In Replacement for GCC/Clang” Andrew Thompson, “Packagers don’t know best” Andrey Mokhov et. al., “Non-recursive Make Considered Harmful” apenwarr, “mtime comparison considered harmful” Apple Inc., “Statically linked binaries on Mac OS X” Aria Desires, “C Isn’t A Language Anymore” Charlie Curtsinger and Daniel W. Barowy, “curtsinger-lab/riker: Always-Correct and Fast Incremental Builds from Simple Specifications” Chris Hopman and Neil Mitchell, “Build faster with Buck2: Our open source build system” Debian, “Software Packaging” Debian, “Static Linking” Dolstra, E., & The CppNix contributors., “Nix Store” Eyal Itkin, “The .a File is a Relic: Why Static Archives Were a Bad Idea All Along” Felix Klock and Mark Rousskov on behalf of the Rust compiler team, “Announcing Rust 1.52.1” Free Software Foundation, Inc., “GNU make” GitHub, Inc., “actions/cache: Cache dependencies and build outputs in GitHub Actions” Google Inc., “bazel-contrib/bazel-gazelle: a Bazel build file generator for Bazel projects” Google LLC, “Jujutsu docs” Jack Lloyd and Steven Fackler, “rust-openssl” Jack O’Connor, “Safety and Soundness in Rust” Jade Lovelace, “The postmodern build system” Julia Evans, “ninja: a simple way to do builds” jyn, “Complected and Orthogonal Persistence” jyn, “Constrained Languages are Easier to Optimize” jyn, “i think i have identified what i dislike about ansible” Kenton Varda, “Ekam Build System” László Nagy, “rizsotto/Bear: a tool that generates a compilation database for clang tooling” Laurent Le Brun, “Starlark Programming Language” Mateusz “j00ru” Jurczyk, “Windows X86-64 System Call Table (XP/2003/Vista/7/8/10/11 and Server)” Michał Górny, “The modern packager’s security nightmare” Mike Shal, “A First Tupfile” Mike Shal, “Build System Rules and Algorithms” Mike Shal, “tup” Nico Weber, “Ninja, a small build system with a focus on speed” NLnet, “Ripple” OpenJS Foundation, “Grunt: The JavaScript Task Runner” “Preprocessor Options (Using the GNU Compiler Collection (GCC))” Randall Munroe, “xkcd: Average Familiarity” Richard M. Stallman and the GCC Developer Community, “Invoking GCC” Rich Hickey, “Clojure - Vars and the Global Environment” Stack Exchange, “What are the advantages of requiring forward declaration of methods/fields like C/C++ does?” Stack Overflow, “Monitoring certain system calls done by a process in Windows” System Calls Manual, “dlopen(3)” System Manager’s Manual, “ld.so(8)” The Apache Groovy project, “The Apache Groovy™ programming language” The Chromium Authors, “gn” Theo de Raadt, “Removing syscall(2) from libc and kernel” The Rust Project Contributors, “Bootstrapping the compiler” The Rust Project Contributors, “Link using the linker directly” The Rust Project Contributors, “Rustdoc overview - Multiple runs, same output directory” The Rust Project Contributors, “The Cargo Book” The Rust Project Contributors, “Command-line Arguments - The rustc book” The Rust Project Contributors, “Queries: demand-driven compilation” The Rust Project Contributors, “What Bootstrapping does” Thomas Pöchtrager, “MacOS Cross-Toolchain for Linux and *BSD” “What is a compiler toolchain? - Stack Overflow” Wikipedia, “Dynamic dispatch” Wikipedia, “Late binding” Wikipedia, “Name binding” william woodruff, “Weird architectures weren’t supported to begin with” Zig contributors, “Zig Build System“ the uncommon case mostly looks like incremental bugs in rustc itself , or issues around rerunning build scripts. ↩ see this stackexchange post for more discussion about the tradeoffs between forward declarations and requiring full access to the source. ↩ even Rust depends on crt1.o when linking ! ↩ early binding is called "static linking". ↩ actually, Zig solved this in the funniest way possible , by bundling a C toolchain with their Zig compiler. This is a legitimately quite impressive feat. If there's any Zig contributors reading—props to you, you did a great job. ↩ almost every build system does this, so I don't even feel compelled to name names. ↩ Starlark is not tied to hermetic build systems. The fact that the only common uses of it are in hermetic build systems is unfortunate. ↩ H.T. Julia Evans ↩ actually variables are more general than this, but for $in and $out this is true. ↩ another example is "rebuilding build.ninja when the build graph changes". it's more common than you think because the language is so limited that it's easier to rerun the configure script than to try and fit the dependency info into the graph. ↩ not actually turing-complete ↩ I have been informed that the open-source version of Bazel is not actually hermetic-by-default inside of its prelude, and just uses system libraries. This is quite unfortunate; with this method of using Bazel you are getting a lot of the downsides and little of the upside. Most people I know using it are doing so in the hermetic mode. ↩ there's something called , but it composes containers, not images. ↩

Clojure C++ DevOps C Programming

Evan Hahn 2 months ago

Scripts I wrote that I use all the time

In my decade-plus of maintaining my dotfiles , I’ve written a lot of little shell scripts. Here’s a big list of my personal favorites. and are simple wrappers around system clipboard managers, like on macOS and on Linux. I use these all the time . prints the current state of your clipboard to stdout, and then whenever the clipboard changes, it prints the new version. I use this once a week or so. copies the current directory to the clipboard. Basically . I often use this when I’m in a directory and I want use that directory in another terminal tab; I copy it in one tab and to it in another. I use this once a day or so. makes a directory and s inside. It’s basically . I use this all the time —almost every time I make a directory, I want to go in there. changes to a temporary directory. It’s basically . I use this all the time to hop into a sandbox directory. It saves me from having to manually clean up my work. A couple of common examples: moves and to the trash. Supports macOS and Linux. I use this every day. I definitely run it more than , and it saves me from accidentally deleting files. makes it quick to create shell scripts. creates , makes it executable with , adds some nice Bash prefixes, and opens it with my editor (Vim in my case). I use this every few days. Many of the scripts in this post were made with this helper! starts a static file server on in the current directory. It’s basically but handles cases where Python isn’t installed, falling back to other programs. I use this a few times a week. Probably less useful if you’re not a web developer. uses to download songs, often from YouTube or SoundCloud, in the highest available quality. For example, downloads that video as a song. I use this a few times a week…typically to grab video game soundtracks… similarly uses to download something for a podcast player. There are a lot of videos that I’d rather listen to like a podcast. I use this a few times a month. downloads the English subtitles for a video. (There’s some fanciness to look for “official” subtitles, falling back to auto-generated subtitles.) Sometimes I read the subtitles manually, sometimes I run , sometimes I just want it as a backup of a video I don’t want to save on my computer. I use this every few days. , , and are useful for controlling my system’s wifi. is the one I use most often, when I’m having network trouble. I use this about once a month. parses a URL into its parts. I use this about once a month to pull data out of a URL, often because I don’t want to click a nasty tracking link. prints line 10 from stdin. For example, prints line 10 of a file. This feels like one of those things that should be built in, like and . I use this about once a month. opens a temporary Vim buffer. It’s basically an alias for . I use this about once a day for quick text manipulation tasks, or to take a little throwaway note. converts “smart quotes” to “straight quotes” (sometimes called “dumb quotes”). I don’t care much about these in general, but they sometimes weasel their way into code I’m working on. It can also make the file size smaller, which is occasionally useful. I use this at least once a week. adds before every line. I use it in Vim a lot; I select a region and then run to quote the selection. I use this about once a week. returns . (I should probably just use .) takes JSON at stdin and pretty-prints it to stdout. I use this a few times a year. and convert strings to upper and lowercase. For example, returns . I use these about once a week. returns . I use this most often when talking to customer service and need to read out a long alphanumeric string, which has only happened a couple of times in my whole life. But it’s sometimes useful! returns . A quick way to do a lookup of a Unicode string. I don’t use this one that often…probably about once a month. cats . I use for , for a quick “not interested” response to job recruiters, to print a “Lorem ipsum” block, and a few others. I probably use one or two of these a week. Inspired by Ruby’s built-in REPL, I’ve made: prints the current date in ISO format, like . I use this all the time because I like to prefix files with the current date. starts a timer for 10 minutes, then (1) plays an audible ring sound (2) sends an OS notification (see below). I often use to start a 5 minute timer in the background (see below). I use this almost every day as a useful way to keep on track of time. prints the current time and date using and . I probably use it once a week. It prints something like this: extracts text from an image and prints it to stdout. It only works on macOS, unfortunately, but I want to fix that. (I wrote a post about this script .) (an alias, not a shell script) makes a happy sound if the previous command succeeded and a sad sound otherwise. I do things like which will tell me, audibly, whether the tests succeed. It’s also helpful for long-running commands, because you get a little alert when they’re done. I use this all the time . basically just plays . Used in and above. uses to play audio from a file. I use this all the time , running . uses to show a picture. I use this a few times a week to look at photos. is a little wrapper around some of my favorite internet radio stations. and are two of my favorites. I use this a few times a month. reads from stdin, removes all Markdown formatting, and pipes it to a text-to-speech system ( on macOS and on Linux). I like using text-to-speech when I can’t proofread out loud. I use this a few times a month. is an wrapper that compresses a video a bit. I use this about once a month. removes EXIF data from JPEGs. I don’t use this much, in part because it doesn’t remove EXIF data from other file formats like PNGs…but I keep it around because I hope to expand this one day. is one I almost never use, but you can use it to watch videos in the terminal. It’s cursed and I love it, even if I never use it. is my answer to and , which I find hard to use. For example, runs on every file in a directory. I use this infrequently but I always mess up so this is a nice alternative. is like but much easier (for me) to read—just the PID (highlighted in purple) and the command. or is a wrapper around that sends , waits a little, then sends , waits and sends , waits before finally sending . If I want a program to stop, I want to ask it nicely before getting more aggressive. I use this a few times a month. waits for a PID to exit before continuing. It also keeps the system from going to sleep. I use this about once a month to do things like: is like but it really really runs it in the background. You’ll never hear from that program again. It’s useful when you want to start a daemon or long-running process you truly don’t care about. I use and most often. I use this about once a day. prints but with newlines separating entries, which makes it much easier to read. I use this pretty rarely—mostly just when I’m debugging a issue, which is unusual—but I’m glad I have it when I do. runs until it succeeds. runs until it fails. I don’t use this much, but it’s useful for various things. will keep trying to download something. will stop once my tests start failing. is my emoji lookup helper. For example, prints the following: prints all HTTP statuses. prints . As a web developer, I use this a few times a month, instead of looking it up online. just prints the English alphabet in upper and lowercase. I use this surprisingly often (probably about once a month). It literally just prints this: changes my whole system to dark mode. changes it to light mode. It doesn’t just change the OS theme—it also changes my Vim, Tmux, and terminal themes. I use this at least once a day. puts my system to sleep, and works on macOS and Linux. I use this a few times a week. recursively deletes all files in a directory. I hate that macOS clutters directories with these files! I don’t use this often, but I’m glad I have it when I need it. is basically . Useful for seeing the source code of a file in your path (used it for writing up this post, for example!). I use this a few times a month. sends an OS notification. It’s used in several of my other scripts (see above). I also do something like this about once a month: prints a v4 UUID. I use this about once a month. These are just scripts I use a lot. I hope some of them are useful to you! If you liked this post, you might like “Why ‘alias’ is my last resort for aliases” and “A decade of dotfiles” . Oh, and contact me if you have any scripts you think I’d like. to start a Clojure REPL to start a Deno REPL (or a Node REPL when Deno is missing) to start a PHP REPL to start a Python REPL to start a SQLite shell (an alias for )

Bash Shell Clojure JSON

Ankur Sethi 2 months ago

Weeknote 2025-W42: Magical and terrifying

Wrote this weeknote on an airplane flying to Delhi. Magical and terrifying. Zero progress on the novel this week. Instead, I spent most of my time rushing to doctors' appointments, preparing for my Delhi trip, and collapsed facedown on my bed after driving across the city for three hours straight. Next week is likely to be a wash, too. I should probably check my calendar before I set writing goals. I'm hoping to at least get some reading done. Looking forward to finishing Fingersmith and moving on to Close to the Machine: Technophilia and Its Discontents . I'm mildly allergic to eucalyptus and English plantain. The allergies clog my nose when I'm outdoors, which makes me susceptible to upper respiratory infections. On my allergist's recommendation, I've been taking supplements, regularly washing out my nose with a saline nasal spray, and using a nasal decongestant at night. Thanks to these treatments, I haven't been sick as sick this year as I usually get. Last month, the allergist recommended that I start immunotherapy, which could fix my upper respiratory issues once and for all. So for the past few days, I've been putting a tiny drop of custom-made medication under my tongue right after I wake up. I can't tell if it's doing anything, because it feels like nothing. To scratch my programming itch, I've started learning Clojure. I'm treating it the same way people treat Wordle or the daily crossword: an entertaining way to give my brain a vigorous workout. I'm not expecting to use it for production projects, at least not in the immediate future. I wasted too many hours reading tech news last week, which is something I only do when I'm trying to hide from negative emotions. It's not surprising. I've had a tumultuous few months. But thankfully I'm doing better this weekend, and I've set up an appointment with a therapist for when I get back home to Bangalore. I don't like being in Delhi. It've never felt at home there. But I'm looking forward to seeing my brother and parents after a while, and probably some cousins I haven't seen in over a decade. Many good links this week (maybe one of these will convince Azan to switch). Switch to Jujutsu already: a tutorial by Stavros Korokithakis. Let help you keep a clean commit history by Paul Smith. Jujutsu Megamerges and by Chris Krycho Two pieces on generative AI and creative friction. choosing friction by Jenny A cartoonist's review of AI art by The Oatmeal I'm not sure I agree fully with these essays on syntax highlighting, but they did get me to pay close attention to a part of my workflow I rarely think about. I am sorry, but everyone is getting syntax highlighting wrong by Niki Tonsky Syntax highlighting is a waste of an information channel by Hillel Wayne

Programming Clojure

tonsky.me 3 months ago

I am sorry, but everyone is getting syntax highlighting wrong

Translations: Russian Syntax highlighting is a tool. It can help you read code faster. Find things quicker. Orient yourself in a large file. Like any tool, it can be used correctly or incorrectly. Let’s see how to use syntax highlighting to help you work. Most color themes have a unique bright color for literally everything: one for variables, another for language keywords, constants, punctuation, functions, classes, calls, comments, etc. Sometimes it gets so bad one can’t see the base text color: everything is highlighted. What’s the base text color here? The problem with that is, if everything is highlighted, nothing stands out. Your eye adapts and considers it a new norm: everything is bright and shiny, and instead of getting separated, it all blends together. Here’s a quick test. Try to find the function definition here: See what I mean? So yeah, unfortunately, you can’t just highlight everything. You have to make decisions: what is more important, what is less. What should stand out, what shouldn’t. Highlighting everything is like assigning “top priority” to every task in Linear. It only works if most of the tasks have lesser priorities. If everything is highlighted, nothing is highlighted. There are two main use-cases you want your color theme to address: 1 is a direct index lookup: color → type of thing. 2 is a reverse lookup: type of thing → color. Truth is, most people don’t do these lookups at all. They might think they do, but in reality, they don’t. Let me illustrate. Before: Can you see it? I misspelled for and its color switched from red to purple. Here’s another test. Close your eyes (not yet! Finish this sentence first) and try to remember what color your color theme uses for class names? If the answer for both questions is “no”, then your color theme is not functional . It might give you comfort (as in—I feel safe. If it’s highlighted, it’s probably code) but you can’t use it as a tool. It doesn’t help you. What’s the solution? Have an absolute minimum of colors. So little that they all fit in your head at once. For example, my color theme, Alabaster, only uses four: That’s it! And I was able to type it all from memory, too. This minimalism allows me to actually do lookups: if I’m looking for a string, I know it will be green. If I’m looking at something yellow, I know it’s a comment. Limit the number of different colors to what you can remember. If you swap green and purple in my editor, it’ll be a catastrophe. If somebody swapped colors in yours, would you even notice? Something there isn’t a lot of. Remember—we want highlights to stand out. That’s why I don’t highlight variables or function calls—they are everywhere, your code is probably 75% variable names and function calls. I do highlight constants (numbers, strings). These are usually used more sparingly and often are reference points—a lot of logic paths start from constants. Top-level definitions are another good idea. They give you an idea of a structure quickly. Punctuation: it helps to separate names from syntax a little bit, and you care about names first, especially when quickly scanning code. Please, please don’t highlight language keywords. , , , stuff like this. You rarely look for them: “where’s that if” is a valid question, but you will be looking not at the the keyword, but at the condition after it. The condition is the important, distinguishing part. The keyword is not. Highlight names and constants. Grey out punctuation. Don’t highlight language keywords. The tradition of using grey for comments comes from the times when people were paid by line. If you have something like of course you would want to grey it out! This is bullshit text that doesn’t add anything and was written to be ignored. But for good comments, the situation is opposite. Good comments ADD to the code. They explain something that couldn’t be expressed directly. They are important . So here’s another controversial idea: Comments should be highlighted, not hidden away. Use bold colors, draw attention to them. Don’t shy away. If somebody took the time to tell you something, then you want to read it. Another secret nobody is talking about is that there are two types of comments: Most languages don’t distinguish between those, so there’s not much you can do syntax-wise. Sometimes there’s a convention (e.g. vs in SQL), then use it! Here’s a real example from Clojure codebase that makes perfect use of two types of comments: Per statistics, 70% of developers prefer dark themes. Being in the other 30%, that question always puzzled me. Why? And I think I have an answer. Here’s a typical dark theme: and here’s a light one: On the latter one, colors are way less vibrant. Here, I picked them out for you: This is because dark colors are in general less distinguishable and more muddy. Look at Hue scale as we move brightness down: Basically, in the dark part of the spectrum, you just get fewer colors to play with. There’s no “dark yellow” or good-looking “dark teal”. Nothing can be done here. There are no magic colors hiding somewhere that have both good contrast on a white background and look good at the same time. By choosing a light theme, you are dooming yourself to a very limited, bad-looking, barely distinguishable set of dark colors. So it makes sense. Dark themes do look better. Or rather: light ones can’t look good. Science ¯\_(ツ)_/¯ There is one trick you can do, that I don’t see a lot of. Use background colors! Compare: The first one has nice colors, but the contrast is too low: letters become hard to read. The second one has good contrast, but you can barely see colors. The last one has both : high contrast and clean, vibrant colors. Lighter colors are readable even on a white background since they fill a lot more area. Text is the same brightness as in the second example, yet it gives the impression of clearer color. It’s all upside, really. UI designers know about this trick for a while, but I rarely see it applied in code editors: If your editor supports choosing background color, give it a try. It might open light themes for you. Don’t use. This goes into the same category as too many colors. It’s just another way to highlight something, and you don’t need too many, because you can’t highlight everything. In theory, you might try to replace colors with typography. Would that work? I don’t know. I haven’t seen any examples. Some themes pay too much attention to be scientifically uniform. Like, all colors have the same exact lightness, and hues are distributed evenly on a circle. This could be nice (to know if you have OCD), but in practice, it doesn’t work as well as it sounds: The idea of highlighting is to make things stand out. If you make all colors the same lightness and chroma, they will look very similar to each other, and it’ll be hard to tell them apart. Our eyes are way more sensitive to differences in lightness than in color, and we should use it, not try to negate it. Let’s apply these principles step by step and see where it leads us. We start with the theme from the start of this post: First, let’s remove highlighting from language keywords and re-introduce base text color: Next, we remove color from variable usage: and from function/method invocation: The thinking is that your code is mostly references to variables and method invocation. If we highlight those, we’ll have to highlight more than 75% of your code. Notice that we’ve kept variable declarations. These are not as ubiquitous and help you quickly answer a common question: where does thing thing come from? Next, let’s tone down punctuation: I prefer to dim it a little bit because it helps names stand out more. Names alone can give you the general idea of what’s going on, and the exact configuration of brackets is rarely equally important. But you might roll with base color punctuation, too: Okay, getting close. Let’s highlight comments: We don’t use red here because you usually need it for squiggly lines and errors. This is still one color too many, so I unify numbers and strings to both use green: Finally, let’s rotate colors a bit. We want to respect nesting logic, so function declarations should be brighter (yellow) than variable declarations (blue). Compare with what we started: In my opinion, we got a much more workable color theme: it’s easier on the eyes and helps you find stuff faster. I’ve been applying these principles for about 8 years now . I call this theme Alabaster and I’ve built it a couple of times for the editors I used: It’s also been ported to many other editors and terminals; the most complete list is probably here . If your editor is not on the list, try searching for it by name—it might be built-in already! I always wondered where these color themes come from, and now I became an author of one (and I still don’t know). Feel free to use Alabaster as is or build your own theme using the principles outlined in the article—either is fine by me. As for the principles themselves, they worked out fantastically for me. I’ve never wanted to go back, and just one look at any “traditional” color theme gives me a scare now. I suspect that the only reason we don’t see more restrained color themes is that people never really thought about it. Well, this is your wake-up call. I hope this will inspire people to use color more deliberately and to change the default way we build and use color themes. Look at something and tell what it is by its color (you can tell by reading text, yes, but why do you need syntax highlighting then?) Search for something. You want to know what to look for (which color). Green for strings Purple for constants Yellow for comments Light blue for top-level definitions Explanations Disabled code JetBrains IDEs Sublime Text ( twice )

Sql Design Programming Clojure

Artyom's Chaotic Blog 3 months ago

Line-based Lisp Editing

Read on the website: Not all environments have Lisp-aware structural editing. Some are only line-oriented. How does one go about editing Lisp line-by-line?

Programming Tutorial Clojure

Artyom's Chaotic Blog 3 months ago

Functional Threading “Macros”

Read on the website: Threading macros make Lisp-family languages much more readable. Other languages too, potentially! Except… other languages don’t have macros. How do we go about enabling threading “macros” there?

Backend Clojure Programming

the website of jyn 4 months ago

the core of rust

NOTE: this is not a rust tutorial. Every year it was an incredible challenge to fit teaching Rust into lectures since you basically need all the concepts right from the start to understand a lot of programs. I never knew how to order things. The flip side was that usually when you understand all the basic components in play lots of it just fits together. i.e. there's some point where the interwovenness turns from a barrier into something incredibly valuable and helpful. One thing I admire in a language is a strong vision. Uiua , for example, has a very strong vision: what does it take to eliminate all local named variables from a language? Zig similarly has a strong vision: explicit, simple language features, easy to cross compile, drop-in replacement for C. Note that you don’t have to agree with a language’s vision to note that it has one. I expect most people to find Uiua unpleasant to program in. That’s fine. You are not the target audience. There’s a famous quote by Bjarne Strousup that goes “Within C++, there is a much smaller and cleaner language struggling to get out.” Within Rust, too, there is a much smaller and cleaner language struggling to get out: one with a clear vision, goals, focus. One that is coherent, because its features cohere . This post is about that language. Rust is hard to learn. Not for lack of trying—many, many people have spent person-years on improving the diagnostics, documentation, and APIs—but because it’s complex. When people first learn the language, they are learning many different interleaving concepts: These concepts interlock. It is very hard to learn them one at a time because they interact with each other, and each affects the design of the others. Additionally, the standard library uses all of them heavily. Let’s look at a Rust program that does something non-trivial: 1 I tried to make this program as simple as possible: I used only the simplest iterator combinators, I don't touch at all, I don't use async, and I don't do any complicated error handling. Already, this program has many interleaving concepts. I'll ignore the module system and macros, which are mostly independent of the rest of the language. To understand this program, you need to know that: If you want to modify this program, you need to know some additional things: This is a lot of concepts for a 20 line program. For comparison, here is an equivalent javascript program: For this JS program, you need to understand: I'm cheating a little here because returns a list of paths and doesn't. But only a little. My point is not that JS is a simpler language; that's debatable. My point is that you can do things in JS without understanding the whole language. It's very hard to do non-trivial things in Rust without understanding the whole core. The previous section makes it out to seem like I'm saying all these concepts are bad. I'm not. Rather the opposite, actually. Because these language features were designed in tandem, they interplay very nicely: There are more interplays than I can easily describe in a post, and all of them are what make Rust what it is. Rust has other excellent language features—for example the inline assembly syntax is a work of art, props to Amanieu . But they are not interwoven into the standard library in the same way, and they do not affect the way people think about writing code in the same way. without.boats wrote a post in 2019 titled "Notes on a smaller Rust" (and a follow-up revisiting it). In a manner of speaking, that smaller Rust is the language I fell in love with when I first learned it in 2018. Rust is a lot bigger today, in many ways, and the smaller Rust is just a nostalgic rose-tinted memory. But I think it's worth studying as an example of how well orthogonal features can compose when they're designed as one cohesive whole. If you liked this post, consider reading Two Beautiful Rust Programs by matklad. This program intentionally uses a file watcher because file IO is not possible to implement efficiently with async on Linux (and also because I wrote a file watcher recently for flower , so it's fresh in my mind). Tokio itself just uses a threadpool, alongside channels for notifying the future. I don’t want to get into async here; this just demonstrates Send/Sync bounds and callbacks. ↩ Technically, is syntax sugar around , but you don't need to know that for most rust programs. ↩ which is a terrible idea by the way, even more experienced Rust programmers often don't understand the interior mutability very well; see this blog post by dtolnay on the difference between mutability and uniqueness in reference types. It would be better to suggest using owned types with exterior mutability and cloning frequently. ↩ first class functions pattern matching the borrow checker and take a function as an argument. In our program, that function is constructed inline as an anonymous function (closure). Errors are handled using something called , not with exceptions or error codes. I happened to use and , but you would still need to understand Result even without that, because Rust does not let you access the value inside unless you check for an error condition first. Result takes a generic error; in our case, . Result is an data-holding enum that can be either Ok or Err, and you can check which variant it is using pattern matching. Iterators can be traversed either with a loop or with . 2 is eager and is lazy. has different ownership semantics than . can only print things that implement the traits or . As a result, s cannot be printed directly. returns a struct that borrows from the path. Sending it to another thread (e.g. through a channel) won't work, because goes out of scope when the closure passed to finishes running. You need to convert it to an owned value or pass as a whole. As an aside, this kind of thing encourages people to break work into "large" chunks instead of "small" chunks, which I think is often good for performance in CPU-bound programs, although as always it depends. only accepts functions that are . Small changes to this program, such as passing the current path into the closure, will give a compile error related to ownership. Fixing it requires learning the keyword, knowing that closures borrow their arguments by default, and the meaning of . If you are using , which is often recommended for beginners 3 , your program will need to be rewritten from scratch (either to use Arc/Mutex or to use exterior mutability). For example, if you wanted to print changes from the main thread instead of worker threads to avoid interleaving output, you couldn't simply push to the end of an collection, you would have to use in order to communicate between threads. first class functions nullability yeah that's kinda it. Enums without pattern matching are very painful to work with and pattern matching without enums has very odd semantics and s are impossible to implement without generics (or duck-typing, which I think of as type-erased generics) / , and the preconditions to , are impossible to encode without traits—and this often comes up in other languages, for example printing a function in clojure shows something like . In Rust it gives a compile error unless you opt-in with Debug. / are only possible to enforce because the borrow checker does capture analysis for closures. Java, which is wildly committed to thread-safety by the standards of most languages, cannot verify this at compile time and so has to document synchronization concerns explicitly instead. This program intentionally uses a file watcher because file IO is not possible to implement efficiently with async on Linux (and also because I wrote a file watcher recently for flower , so it's fresh in my mind). Tokio itself just uses a threadpool, alongside channels for notifying the future. I don’t want to get into async here; this just demonstrates Send/Sync bounds and callbacks. ↩ Technically, is syntax sugar around , but you don't need to know that for most rust programs. ↩ which is a terrible idea by the way, even more experienced Rust programmers often don't understand the interior mutability very well; see this blog post by dtolnay on the difference between mutability and uniqueness in reference types. It would be better to suggest using owned types with exterior mutability and cloning frequently. ↩

Rust C++ Clojure Programming Java

the website of jyn 5 months ago

how to communicate with intent

As you can see from this blog, I like to talk (my friends will be the first to confirm this). Just as important as knowing how to talk, though, is knowing what to say and when to listen. In this post I will give a few simple tips on how to improve your communication in various parts of your life. My goal is partly to help you to be a more effective communicator, and partly to reduce the number of people in my life who get on my nerves :P You don't always need these tips. In casual conversations and when brainstorming, it's healthy to just say the first thing on your mind. But not all conversations are casual, and conversations can slip into seriousness faster than you expect. For those situations, you need to be intentional. Otherwise, it's easy to end up with hurt feelings on both sides, or waste the time of everyone involved. First, think about your audience. Adapt your message to the person you’re speaking to. Someone learning Rust for the first time does not need an infodump about variance and type coercion, they need an introduction to enums, generics, and pattern matching. Similarly, if a barista asks you what the weird code on your screen is, don’t tell them you’re writing a meta-Static Site Generator in Clojure , tell them you’re building a tool to help people create websites. If you are writing a promotion doc, a resume, or a tutorial, don't just dump a list of everything that's relevant. Think about the structure of your document: the questions your reader is likely to have, the amount of time they are likely going to spend reading, and the order they are likely to read in. You need to be legible , which means explaining concrete impacts in terms your audience understands. It's not enough to say what's true; you have to also say why it's important. Consider your intended effect. If a teacher goes on Twitter saying she doesn’t understand why maths is important and we should just give out A’s like candy (real thing that happened on my feed!), dog piling on her is not going to change her mind. Show her a case in her life where maths would be useful, and don’t talk down to her. Self-righteousness feels good in the moment, but doesn’t actually achieve anything. If you just want to gloat about how other people are stupid, go play an FPS or something; Twitter has enough negativity. If you are writing a blog post, know why you are writing it. If you are writing to practice the skill of writing, or to have a reference document, or to share with your friends, infodumping is fine. If you are writing with a goal in mind—say you want to give a name to an idea or communicate when software can fail or enter an idea into the overton window —be intentional. Consider your audience, and the background you expect them to start from. Posting the equivalent of a wikipedia article is rarely the most effective way to instill an idea. Don’t fight losing causes, unless the cause is really worth it . Someone on hacker news saying "language A Sucks and you Should use language B instead" is not worth arguing with. Someone who says "language A is good in scenario X, but has shortcomings in scenario Y compared to language B" is much more serious and worth listening to. Arguing with someone who refuses to be convinced wastes everyone’s time. Be a good conversational partner. Ask directed probing questions: they show you are listening to the other person and invested in the topic. Saying “I don’t understand” puts the burden on them to figure out the source of the misunderstanding. If you really aren’t sure what to ask, because you’re confused or the other person was vague, I like “say more?” as a way to leave it open ended for the other person on how to elaborate. Consider the implications of how you communicate. When you say things, you are not just communicating the words you speak, you are also affecting the person you're talking to. If the person you're infodumping to isn't interested in the topic, infodumping anyway puts them in an awkward situation where they either have to ask you to stop or sit through a long conversation they didn't want to be in. Another tricky scenario is when the other person is interested, but an infodump is not the right level of detail for them right now. Perhaps they are new to the topic, or perhaps they asked a direct question. If they're still trying to get the "big picture", zooming in to fine-grained details will often just confuse them further. Info-dumping during an apology—even if it’s related to the thing you're apologizing for!—buries the apology. More than that, it implies that you expect mitigated judgement . If there is a power dynamic between you (say a wealth gap, or you are a manager and they are an employee), that expectation of mitigated judgment implies you expect to be forgiven , and an apology given in expectation of forgiveness is really just a request for absolution . Instead, apologize directly. If you were in an altered mental state (angry, sleep-deprived, experiencing a trauma trigger), you can add at most 1-2 sentences of context asking the other person to mitigate judgement. Not all apologies need context; often "i was wrong, i'm sorry" is enough. As we've seen above, there are times when infodumps actively hurt you. Even when they don't, though, there can be times when they aren't helping. Everyone comes to a conversation with a different background, and you cannot perfectly predict how they will respond. Rather than trying to avoid every possible miscommunication by packing the maximum amount of information—Say what you mean to say. Then, address the actual miscommunication (or regular conversation!) that happens afterwards. This saves time and energy for both conversational partners. The common theme of all of the above is to communicate effectively and radiate intent . Making it easy for the other person to understand both what you're saying and why you're saying it incurs a lot of goodwill, and makes it possible to say more things more bluntly than you would otherwise. A common trap I see people fall into is to say the first thing on their mind. This is fine for conversations between friends (although you should still consider how it affects your relationship!) but but is often counterproductive in other contexts. Slow down. Take your time. Say what you mean to say. If you don't mean to say anything, don't say anything at all.

Rust Career Writing Clojure

the website of jyn 6 months ago

you are in a box

You are trapped in a box. You have been for a long time. Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. most tools simultaneously think too small and too big. “i will let you do anything!”, they promise, “as long as you give up your other tools and switch to me!” this is true of languages too. any new programming language makes an implicit claim that “using this language will give you an advantage over any other language”, at least for your current problem. once you start using a tool for one purpose, due to switching costs, you want to keep using that tool. so you start using it for things that wasn’t designed for, and as a result, tools tend to grow and grow and grow until they stagnate . in a sense, we have replicated the business boom-and-bust cycle in our own tools. there are two possible ways to escape this trap. the first is to impose a limit on growth , so that tools can’t grow until they bust. this makes a lot of people very unhappy and is generally regarded as a bad idea. the second is to decrease switching costs. by making it easier to switch between tools, or to interoperate between multiple tools in the same system, there is not as much pressure to have “one big hammer” that gets used for every problem. tools and languages can decrease switching costs by keeping backwards compatibility with older tools, or at least being close enough that they’re easy to learn for people coming from those tools. for example, ripgrep has almost exactly the same syntax as GNU grep, and nearly every compiled language since C has kept the curly braces. tools can also collaborate on standards that make it easier to interoperate. this is the basis of nearly every network protocol, since there's no guarantee that the same tool will be on the other side of the connection. to some extent this also happens for languages (most notably for C), where a language specification allows multiple different compilers to work on the same code. this has limitations, however, because the tool itself has to want (or be forced) to interoperate. for example, the binary format for CUDA (a framework for compiling programs to the GPU) is undocumented, so you're stuck with reverse engineering or re-implementing the toolchain if you want to modify it. the last "internal" way to talk to languages is through a "foreign function interface", where functions in the same process can call each other cheaply. this is hard because each language has to go all the way down to the C ABI before there's something remotely resembling a standard, and because two languages may have incompatible runtime properties that make FFI hard or slow . languages that do encourage FFI often require you to write separate bindings for each program: for example, Rust requires you to write blocks for each declaration, and python requires you to do that and also write wrappers that translate C types into python objects. i won't talk too much more about this—the work i'm aware of in this area is mostly around WASM and WASM Components , and there are also some efforts to raise the baseline for ABI above the C level. another approach is to compose tools. the traditional way to do this is to have a shell that allows you to freely compose programs with IPC. this does unlock a lot of freedom! IPC allows programs to communicate across different languages, different ABIs, and different user-facing APIs. it also unlocks 'ad-hoc' programs, which can be thought of as situated software for developers themselves. consider for example the following shell pipeline: this shows the 10 largest files in the git history for the current repository . let's set aside the readability issues for now. there are a lot of good ideas here! note that programs are interacting freely in many ways: the equivalent in a programming language without spawning a subprocess would be very verbose; not only that, it would require a library for the git operations in each language, bringing back the FFI issues from before (not to mention the hard work designing "cut points" for the API interface 1 ). this shell program can be written, concisely, using only tools that already exist. note though that the data flow here is a DAG: pipes are one-way, and the CLI arguments are evaluated before the new program is ever spawned. as a result, it’s not possible to do any kind of content negotiation (other than the programmer hard-coding it with CLI args; for example tools commonly have ). the downside of this approach is that the interface is completely unstructured; programs work on raw bytes, and there is no common interface. it also doesn't work if the program is interactive, unless the program deliberately exposes a way to query a running server (e.g. or ). let's talk about both of those. powershell , and more recently, nushell , extend traditional unix pipelines with structured data and a typesystem. they have mechanisms for parsing arbitrary text into native types, and helper functions for common data formats. this is really good! i think it is the first major innovation we have seen in the shell language in many decades, and i'm glad it exists. but it does have some limitations: and it is very hard to fix these limitations because there is no "out-of-band" communication channel that programs could use to emit a schema; the closest you could get is a "standardized file descriptor number", but that will lock out any program that happens to already be using that FD. we have limited kinds of reflection in the form of shell completion scripts, but they're not standardized: there's not a standard for the shell to query the program, and there's not a standard for the format the program returns. the CLI framework inside the program often does have a schema and reflection capabilities, but they're discarded the second you go over an IPC boundary. how do you get a schema? well, you establish in-band communication. RPC is theoretically about "remote" procedure calls, but it's just as often used for local calls. the thing that really distinguishes it is that it's in-band: you have a defined interface that emits structured information. RPC works really quite well! there are frameworks for forwards- and backwards-compatible RPC ; types and APIs are shared across languages; and interop for a new language only requires writing bindings between that language and the on-wire format, not solving a handshake problem between all pairs of languages nor dropping down to the C ABI. the main downside is that it is a lot of work to add to your program. you have to extensively modify your code to fit it into the shape the framework expects, and to keep it performant you sometimes even have to modify the in-memory representation of your data structures so they can live in a contiguous buffer. you can avoid these problems, but only by giving up performance when deserializing (e.g. by parsing JSON at runtime). all these limitations are because programs are a prison . your data is trapped inside the box that is your program. the commonality between all these limitations is that they require work from the program developer, and without that work you're stuck. even the data that leaves the program has to go through the narrow entrances and exits of the box, and anything that doesn't fit is discarded . some languages try to make the box bigger—interop between Java, Kotlin, and Clojure is comparatively quite easy because they all run on the JVM. but at the end of the day the JVM is another box; getting a non-JVM language to talk to it is hard. some languages try to make the box extensible—LISPs, and especially Racket, try to make it very easy to build new languages inside the box. but getting non-LISPs inside the box is hard. some tools try to give you individual features—smalltalk gets you orthogonal persistence; pluto.jl gets you a “terminal of the future”; rustc gets you sub-process incremental builds. but all those features are inside a box. often, tools don’t even try. vendor lock in, subtle or otherwise, is everywhere around us. tools with this strategy tend to be the largest, since they have both the biggest budget and the greatest incentive to prevent you from switching tools. and always, always, always, you are at the mercy of the program author. in my next post, i will discuss how we can escape this box. blog post forthcoming ↩ the output of is passed as a CLI argument to the output of is passed as stdin to the output of is interpreted as a list and programmatically manipulated by . this kind of meta-programming is common in shell and has concise (i won't go as far as "simple") syntax. the output from the meta-programming loop is itself passed as stdin to the command there is no interop between powershell and nushell. there is no protocol for programs to self-describe their output in a schema, so each program's output has to be special-cased by each shell. powershell side-steps this by building on the .NET runtime , and having native support for programs which emit .NET objects in their output stream . but this doesn't generalize to programs that don't run on the CLR . there is no stability guarantee between versions of a program. even tools with semi-structured JSON output are free to change the structure of the JSON, breaking whatever code parses it. D. R. MacIver, “This is important” Wikipedia, “Zawinski’s Law of Software Envelopment” Graydon Hoare, “Rust 2019 and beyond: limits to (some) growth.” Rich Hickey, “Simple Made Easy” Vivek Panyam, “Parsing an undocumented file format” The Khronos® Group Inc, “Vulcan Documentation: What is SPIR-V” Aria Desires, “C Isn’t A Language Anymore” Google LLC, “Standard library: cmd.cgo” Filippo Valsorda, “rustgo: calling Rust from Go with near-zero overhead” WebAssembly Working Group, “WebAssembly” The Bytecode Alliance, “The WebAssembly Component Model” Josh Triplett, “crABI v1” Clay Shirky, "Situated Software" Microsoft, "PowerShell 7.5: 4. Types" Microsoft, "PowerShell 7.5: What is PowerShell?" Microsoft, "PowerShell 7.5: about_Output_Streams" Microsoft, ".NET Execution model: Common Language Runtime (CLR) overview" Nushell Project, "Nu Fundamentals: Types of Data" Google LLC, “Protocol Buffers” Robert Lechte, “Programs are a prison: Rethinking the fundamental building blocks of computing interfaces” Siderea, "Procrustean Epistemologies" blog post forthcoming ↩

Clojure Programming Java JSON

tonsky.me 6 months ago

Podcast: Datomic: самая рок-н-рольная БД @ Тысяча фичей

Чем Datomic отличается от других баз данных и почему иногда остутствие оптимизатора лучше, чем его присутствие

Backend Clojure Database

alavi.me 6 months ago

/now

Updated on 2025 July A page that tells you what this person is focused on at this point in their life. Outstanding book. Very simple and easy to read, while very practical. I'm already applying the concepts and patterns in this book in my work. Just this week I refactored an old legacy piece of our software at work with functional patterns and also optimized it's performance by quite a lot. A very comprehensive resource for learning the Clojure language from scratch. The examples and explanations are great. The book is very long but as a very comprehensive book, it should be. So many things about Clojure appealed to me very much. It's simplicity, elegance, stability, community, culture and more. I see how many great opensource projects are created with it, which means it is very efficient to create with. It lets you express your thoughts instead of wrestling with the language. I'll have to write about to appeal of Clojure more in depth later. I started programming with C and the JS. I don't know, maybe it's how my brain works but I just can't relate to OOP. Never could. Instead, my brain and thoughts have always been closer to the functional paradigm. Functional programming has been really tickling my brain and I want to become great at it, and it will probably be the main way I program later on. I am close to officially becoming the tech leader of my company. This is a role that I feel so comfortable in and I think I am somewhat gifted in it. But like anything else, I need experience and knowledge. I have braced myself and my company that I am for sure going to mess up a bit at first, but I will learn and quickly adapt. I'm pretty OK with English and Persian is my mother tongue. I'm choosing Arabic as my third human language , since it will be very useful and I will gain access to a new plethora of wisdom in Arabic.

JavaScript Career Programming Clojure

anders murphy 8 months ago

Clojure: SQLite C API with project Panama and Coffi

Clojure Programming Database