Latest Posts (20 found)
matklad 1 months ago

CSS: Unavoidable Bad Parts

An ersatz CSS tutorial for people who need to style a web page, but aren’t web developers. I am a wrong person to write this kind of thing, as I have neither the time, nor experience. I’d much rather read a book about this. Alas, I had to learn all this stuff from trawling MDN, so perhaps it is valuable to document what I have so far. CSS, HTML and Web APIs are truly vast, and it takes a career to become a professional. The good news is that modern web has a reasonably-sized, learnable subset which is enough for simple tasks like a programming blog or a simple GUI. I haven’t seen a resource that teaches just this subset, but it’s not too hard to figure this out. The bad news is that there’s also a nasty set of gotchas, which will mess up your page, which you won’t suspect to exist, and which will need days of debugging to figure out. Still, it’s not that bad. I am quite happy with the styling on this site, and it’s only about 200 of readable CSS . Good: HTML5 semantic tag names It’s worth looking through MDN Elements Reference . There aren’t that many elements, and things like , , , make it much easier to structure your page. Less obvious: Bad: Wrappers If you “View Source” on any “real” website, you’ll notice that everything has layers and layers of wrapper elements, so you might be tricked into thinking that wrappers are how you solve layout problems. I can’t really agree or disagree here, as I never wrote “production” CSS, but, in my experience, it’s much easier to understand if you do the opposite — restrict yourself to using only markup-meaningful semantic tags, and then figure out CSS which works with the markup you have. Bad: Layout This one is not an exclusively Web problem, layout is a struggle in every GUI framework I know. Imagine a fixed sized raster image, and a paragraph of text describing it. There are many ways to arrange these two elements on the screen’s rectangle. Generally, for every given width and height, you can do a decent job, as long as the total area is enough. A typical GUI is a hierarchy of such boxes, with a lot of “layout freedom”. The problem though is that layout of each box affects the layouts of all other boxes, as you generally want all boxes to meet exactly, without gaps and overlaps. An important negative realization is that the layout algorithm doesn’t exist. There isn’t a fully general solution to positioning and sizing GUI boxes. Rather, different systems use different sets of heuristics to do the job, from simple RectCut , to fully general constraint solvers , with everything in between . It is hard to get the mental model of how layout works, in general . So, don’t think “how can I do my layout in a given system”, think instead “what possible layouts are allowed by the system”. Bad: Browser defaults Let’s start with a bare (but still semantic) HTML markup of a blog article, without any CSS. If you open it in a browser, it will show something . The content isn’t unstyled — the text is of a certain color, font and size. Headers are bigger than the main text, links are underlined, etc. These are the default styles of your browser. They are helpful! The problem is that these styles differ between the browsers. So, even when you add your own CSS, and the end result looks fine in your browser, I might see something different, because you might rely on a browser default, without knowing it. The last bit is the killer here — the problem is in something you didn’t write. The general solution here is a CSS reset , or normalization — starting your CSS with an explicit set of rules, overriding defaults. Not because defaults are inherently bad, because they are inconsistent. I don’t know which set of rules you need to override in practice, it’s a good idea to compare several existing CSS resets. This touches on the big question: should you style your web page? There are two competing views of the Web platform — some people treat it as a flexible, adaptive, primarily visual medium for expressing design, others would prefer if the Web focused on delivering the content, allowing each user to customize the presentation. My personal answer here is pragmatic — by default, an unstyled page is poorly usable and looks bad. I would have preferred the world where CSS-less pages were readable as is, but, in this world, I think it is helpful to style the content. At the same time, it’s a good idea to allow advanced users to bring their own CSS. Make sure that your HTML markup is reasonable, that you don’t overfit your HTML to CSS (vice-versa is fine), and that your page functions in reader mode. Good: Classless CSS You can’t reset styles to true neutral nothing: if you make the text invisible (white or transparent), it is still a style. So you might as well embrace it: after reset, style common HTML elements directly. For example, to set your favorite font for all code snippets: If you use , , , tags you can set the overall page layout without writing any CSS selectors. This of course requires making assumptions, in CSS, about the structure of your HTML, but, like, this is your HTML and your CSS, you can do whatever, and, if you don’t like the result, you can always change it! Bad: CSS selectors In programming, we collectively came around to distrust inheritance and prefer composition. Default CSS is like supercharged inheritance, each design element on your web page is affected by multiple rules, and you can always “monkey patch” existing elements by appending to your CSS. There’s an unfortunate gap between CSS affordances, and what you actually want to do. The two reasonable approaches are: Conclude that CSS selectors add abstraction capability along the wrong axis, and stick to classless CSS and inline styles, using something like Tailwind to make writing inlines prettier, and something like JSX (or any other templating engine supporting composition) to avoid repetition in HTML. Use CSS nesting to avoid writing “far reaching” selectors and style component-per-component: Bad: box-sizing UIs are recursive rectangles, layout is the process of figuring out where each rectangles goes, and it is determined by the sizes of rectangles themselves. So, understanding what is the size is quite fundamental. Sadly, by default the definition of size in HTML is very unintuitive: element’s width and height do not include element’s border and padding, which leads to surprising results: everything looks perfect at first, but increasing padding somewhere shifts the entire layout unexpectedly. For this reason, deserves to be the first line in your CSS reset. It makes elements encapsulated, such that adding borders is a local-only change. Chaotic Good: margin collapsing Suppose you want to have a gap around an element. You would think that you need to set the padding property. But that would be wrong — if you have two such elements next to each other, the gap between them would be . The paddings would add, creating a visual gap larger than intended. You want something more akin to social distancing, where if one person is more introverted, this person’s bigger radius of exclusion is what defines the distance. And that’s how the property works. Two neighboring margins are combined using rather than . Margin collapsing is very useful, but it can surprise you. E.g. I think child margin can stick beyond parent’s? To be honest, I don’t have a good intuitive understanding of margins, but I know enough to at least identify when it is the problem. Margins are also one of the indirect inspirations for this post. In Moving away from Tailwind, and learning to structure my CSS Julia Evans writes that you generally don’t want to set margin on an element, and should rather let the parent control the inter-element margin of the children, using the so-called owl selector: That is, add margin to all ’s children exempting the first one. I didn’t know that! And, given all the pain that margin gave me so far, I actually get why you want to do this, and why this is a good idea. But it bugs me that you can’t learn that without becoming “professional” web developer, or reverse-engineering someone else’s CSS framework. Bad: Default (flow) layout Layout in general is tricky, because there’s no universal “layout algorithm”, just a bunch of special cases. But what does HTML actually do? The default layout algorithm I think goes back to the origin of HTML as a language for documents, and overfits a use-case of producing papers — mostly text content with some illustrations, where the text can flow around the pictures. That’s actually what you want for the main body of text of your blog, but, as soon as you want to actually control the spatial arrangement of the elements on your page, you want something different, for example… Good: flexbox This is really what separates modern web-development from the olden days, where you’d need a CSS PhD or a full-blown opaque CSS framework to be able to say “this goes to the left, and this goes to the right”. This layout allows you to arrange a series of elements either vertically or horizontally, adapting to the available space. It is rather complex and I can’t use flexbox without referencing MDN all the time, but usually I am able to get things done in the end. Bad: responsive design Modern CSS allows querying screen size, and implementing conditional logic based on that — a design that “responds” to user-agent constraints. This probably what you should use for “real” CSS, but note that HTML is inherently responsive. Unlike PostScript (PDF), it will automatically reflow the paragraphs when you change window size. So, it’s a good idea to avoid writing explicit responsive rules, and just rely on layout to do the reasonable thing. For example, this blog looks OK on mobile, tablet and desktop without any explicit queries. Unconditionally setting on the main column of text is all that it takes. Lawful Evil: pixels does what you want, but not what it says. It’s not a size of one physical pixel on your screen. Rather, it’s a measure of visual angle . That is, should look perceptually the same on any screen, and it is converted to different number of physical pixels, depending on the screen size, its pixel density, and the typical viewing distance. So you can just size everything in pixels, without thinking about different displays’ pixel densities. It gets weirder. CSS allows “real” units like centimeters or inches, but they are also angles, because everything is defined in terms of pixels. Doubleplusungood: font-size Flexbox is a good way to layout UI-elements. Flow layout works ok for laying out paragraphs of text. But what happens on the level of individual lines and glyphs is, in my opinion, a train wreck and a noob trap. Let’s start with the basics: if you write then is the size of what? Sadly, the answer is “nothing in particular” — this is a size of a virtual box around the glyph, but the box isn’t tight, and the size of the glyph varies, depending on the font. Luckily, property can fix it, and make consistent across fonts. See these two posts for details: Though, at the moment seems to be very niche, so, while personally I’d put right next to , few pages do that. The next issue with is a thorny question of defaults. The good news is that it’s one of the properties that is fairly consistent across browsers, with being the overwhelming default. The bad news is that, depending on the font, can be on the smaller size. Not completely illegible, but very close to the lower bound. What’s worse, some default fonts are particularly small. For example, on Apple, looks much smaller than , and is almost uncomfortable to read at 16px. Can you just set or whatever works best for your chosen font? I think the answer is yes, but there are some caveats to keep in mind. Refer to Accessibility: px or rem? for details. The issue is that modern browsers support two ways of making text on a page bigger: Setting in your CSS disables that second approach. Taking everything together: don’t assume that text on your page will be readable by default, check different configurations. Set to reduce the number of degrees of freedom and to pin down the meaning of . If the result looks fine with your chosen (or your user’s default) font and default font-size of , then you are done. Otherwise, set to a bigger number. Afterwards, check that the page is readable in reader mode as well. Bad: Despite the name, doesn’t set the height of a line. It is a height of a run of glyphs, set in the same font . The two coincide when all the text is in the same font. But if you have, e.g., some words set in font, you are in for a surprise. While fixes the size of a glyph inside the box, it still leaves its relative position unspecified. So, when two runs of text in different fonts are aligned vertically to share the baseline, their line-height line-boxes get shifted relative to each other: one sticks below, one sticks above. The line height overall becomes larger that what you’d expect, as it is configured as a union. See Deep dive CSS: font metrics, line-height and vertical-align for a thorough explanation of this effect. Bad: vertical rhythm If you google long enough this cluster of problems, sooner or later you’ll come across the idea of vertical rhythm, that you should make sure that lines are in the same relative position across different paragraphs, even if you have headings, images, and what not. As if there’s invisible lined paper behind your web-page. As far as I can tell, this is pure voodoo and is not useful. If you do two-column layout, then you want lines on opposite sides to align, but it makes no sense to jump through hoops for a single-column layout (hat tip to @chrismorgan ). Bad: The genius of the flow layout is its dynamism. It takes a moment of reflection to appreciate the technical marvel of text breaking itself neatly into lines as the window is resized to be narrower. Getting that to work for the first time ever in the world of durably printed text must have felt incredible. But the magic has its limits — you can only break the line at the whitespace, or at the hyphenation points. And some long spans, like or URLs, might be unbreakable. This leads to overflow annoyance on mobile devices, something you notice only after you publish your work. There’s no one trick to fix it, but some tips are available here: Against Horizontal Scroll for details. And … that’s all I remember so far? I reiterate my request for someone to write a short 100-page book explaining just enough of HTML&CSS to make a simple blog without getting collapsed by the margins! for any kind of list, like site’s sections in . for table-of-contents (check the source of MDN). / for list of pairs. Conclude that CSS selectors add abstraction capability along the wrong axis, and stick to classless CSS and inline styles, using something like Tailwind to make writing inlines prettier, and something like JSX (or any other templating engine supporting composition) to avoid repetition in HTML. Use CSS nesting to avoid writing “far reaching” selectors and style component-per-component: font-size-adjust Is Useful Font size is useless; let’s fix it Zoom, which has a dedicated UI element, shortcuts/gestures, per-page persistence/overrides and a global default. Changing default font-size, a global setting buried deeply in the configuration page.

0 views
matklad 1 months ago

TIL: Symlinking NixOS Dotfiles

The standard answer to managing dotfiles on NixOS is home-manager . I’ve never used it, due to two aesthetic and one practical objection: The approach I like is storing dotfiles in the same repository as / and symlinking them in place. The problem here is that NixOS seemingly doesn’t have a “native” way to say that should be a symlink to . Or has it? If you search options for , you’ll learn about which allows you to configure symlinks, but only for things in , not your . For the latter, you can use gnu stow or some other dotfile link manager, but the complexity of the problem of just managing symlinks doesn’t warrant yet another dependency. It’s fine to do this manually . But wouldn’t it be nice if this framework for declarative configuration of your system allowed you to declaratively configure symlinks? Turns out this is possible, in roundabout way. Inaptly-named systemd-tmpfiles allows creating symlinks from a declarative config, and you can use NixOS to configure itself (thanks, NobbZ !). For example, if I want to symlink to : No opinion at this point how this compares to a bespoke script or something more purpose-built . I avoid dependencies, especially in nix, which rivals Python in the number of approaches to dependency management. home-manager installs packages for the current user only, which makes sense on non-NixOS systems. But on a single-user desktop system, I prefer having just one set of packages. Having a source of truth for dotfiles be in nix store requires rebuilding your system to change config, which gets in the way of Emacs-style direct tinkering.

0 views
matklad 1 months ago

Always Be Blaming

A few tips on 4D-ing your code comprehension skills. I wrote on the importance of reading code before: Look Out For Bugs My default approach to reading is “predictive”: I don’t actually read the code line by line. Rather, I try to understand the problem that it wants to solve, then imagine my own solution, and read the “diff” between what I have in my mind and what I see in the editor. Non-empty “diff” signifies either a bug in my understanding, or an opportunity to improve the code. This is 2D reading, understanding a snapshot of code, frozen in time. This is usually enough to spot “this feels odd” anomalies, worthy of further investigation. Ideal code is memoryless — it precisely solves the problem at hand. Most real code is Markov — the shape of the code at time depends not only on the problem statement, but also on the shape of the code at time . The 3D step is to trace the evolution of code over time, Where Do We Come From? What Are We? Where Are We Going? . The step after that is to understand the why . What were we thinking back then, when we wrote this code? It’s useful to have the “theory of mind” concept ready here. I personally learned the term way too late in my life, so let me give a short intro for today’s lucky 10 000 . Theory of mind is the ability to imagine yourself in someone else’s skin. Not just in their shoes (“I certainly would have acted differently in that situation”), but with their mind (“ I wouldn’t have acted that way, but I get why they did”). This is something people learn. The experimental setup here is to have a child in a room with toys, with a doll sitting near the opposite end of the room, and asking the child “what does the doll see?”. Younger children describe the room from their perspective, older begin to intuit that doll’s perspective is different. So this is the goal of reading code — understanding what the original author was thinking, and why . End of the mumbo-jumbo, some practical advice. First, read Every line of code is always documented , it is very good. Second, make sure it is effortless for you to find out how a given snippet of code evolved. This is harder than it seems! Just isn’t an answer — mind the gap between the problem that’s easy to solve, and the problem in need of solving. answers spatial question of “how each line appeared in this file”, because there’s a relatively straightforward UI for this — annotate each line with a commit hash. But this is not the question you are asking most of the time! You don’t care about the file! There’s a small snippet of code in the middle, and you want a temporal history of that . As much as I don’t like working in the browser GitHub’s web interface for blaming is probably better than what you get locally by default. It starts with the y shortcut, which resolves a symbolic reference like into the one which has a commit hash in the URL: This commit hash is critical, because it anchors the entire repository — if you open a different file from the web UI, it will be shown as of that commit. This enables you to not myopically focus on just the diff in question, but to absorb the entire context at that point in time. So my usual web workflow is: Again, my goal here is not to annotate a diff on a file but rather to get a “virtual checkout” as of the interesting commit. This web approach is what I was using throughout most of my career, but I’ve finally found a way to replicate it locally. The idea is to make blaming “in-place”. Instead of annotating lines of code, I directly switch to a historical commit. I have the following devil hydra of shortcuts: , b l blames line. It notes the the cursor is at, runs to find that introduced the line, and then runs to check it out. I have a dedicated worktree for code archeology, so I don’t worry about trashing my work. There’s also a half-hearted attempt to maintain “logical” cursor position, but it doesn’t work very well. Is there some git command that tells me directly “what’s the equivalent of in for ?” , b p blames parent. Which is just switching to the parent commit of the current , what “blame before this change” does on GitHub (it works slightly differently because it assumes that , b l was the previous command) , b u undoes the last blaming operation, switching to the previous point. I really love that, on the web, I can cmd -click to create an alternative branch of exploration. In theory, this is replicatable locally, but I prefer to destructively mutate a single working tree on disk. A big reason for preferring in-place blame is that LSP, , and the like just work. That’s more important for me than the garden of forking paths, and undo is an acceptable work-around. Finally, , b w copies GitHub link to the current commit and line, which I can paste into the browser. An enormous problem with modern version control landscape is that absolutely critical information in the form of code review comments is not a part of the git repository, and is locked in someone else’s proprietary database. I failed to solve this problem in one weekend, and had to begrudgingly adapt. Opening the commit in a browser links you to the PR and its discussion as well. Implementing this blame workflow required a bit of custom code . Feel free to use it, but beware that it’s somewhat crufty, especially around maintaining current cursor position. Making a production-ready version of this sounds like a fun project ;-) ctrl + f to find the line I am interested in b to toggle blame Click “blame prior to change” a couple of times, repeating ctrl + f to go back to the snippet I am curious about. cmd -click on the commits that are potentially relevant, pinning their commit hashes in the URL in new tabs. Then, from the commit page, “Browse files” button to then go and t to other files. Or, cmd + l to focus browser’s address bar, and (or back!) as needed, to switch between diff and snapshot views.

0 views
matklad 2 months ago

Catch Flakes On Main

A small Mechanical Habit today: When using not rocket science rule / merge queue, continue to redundantly run the full test suite on main. Maintain an easily accessible list of recent main failures — these are the flaky tests to eradicate. For an example, see the “Flakes” link on https://devhub.tigerbeetle.com Flaky tests are tests that fail intermittently, once in a thousand runs. This might be due to a genuine bug (assumptions about scheduling that mostly hold) or due to instability of underlying infrastructure (e.g., inability to download a release from GitHub, or to delete a folder on Windows). In either case, flaky tests are a huge productivity drain — as the size and complexity of test suite grows, more and more CI runs fail spuriously, even as each individual test almost always passes. Flaky tests are challenging to deal with — if you are working on landing a PR and your CI fails due to an obvious flake, the temptation to just re-run the test suite is enormous, especially if there’s a certain background dissatisfaction with infrastructure stability. If you are of a mind to do some flake squashing, then your PRs will be green just to spite you! And working off of others’ PRs would require first to separate flakes from genuine failures. This is why the merge queue is powerful: if there’s a guarantee that every commit on the main branch passes the tests, then every failure on main is a flake, by definition. Collecting all such failures into a single list compresses time, allows to prioritize the most impactful sources of instability, and reveals correlations between failures.

0 views
matklad 2 months ago

Learning Software Architecture

In reply to an email asking about learning software design skills as a researcher physicist: I was attached to a bioinformatics lab early in my career, so I think I understand what you are talking about, the phenomenon of “scientific code”! My thoughts: First meta observation is that “software design” is something best learned by doing. While I had some formal “design” courses at the University, and I was even “an architect” for our course project, that stuff was mostly make-believe, kindergarteners playing fire-fighters. What really taught me how to do stuff was an accident of my career, where my second real project ( IntelliJ Rust ) propelled me to a position of software leadership, and made design my problem. I did make a few mistakes in IJ Rust, but nothing too horrible, and I learned a lot. So that’s good news — software engineering is simple enough that an inquisitive mind can figure it out from first principles (and reading random blog posts). Second meta observation, the bad news: Conway’s law is important. Softwaregenesis repeats the social architecture of the organization producing software. Or, as put eloquently by neugierig , If I were to summarize what I learned in a single sentence, it would be this: we talk about programming like it is about writing code, but the code ends up being less important than the architecture, and the architecture ends up being less important than social issues. I suspect that the difference you perceive between industrial and scientific software is not so much about software-building knowledge, but rather about the field of incentives that compels people to produce the software. Something like “my PhD needs to publish a paper in three months” is perhaps a significant explainer? Two things you can do here. One , at times you get a chance to design or nudge an incentive structure for a project. This happens once in a blue moon, but is very impactful. This is the secret sauce behind TIGER_STYLE , not the set of rules per se, but the social context that makes this set of rules a good idea. Two , you can speedrun the four stages of grief to acceptance. Incentive structure is almost never what you want it to be, but, if you can’t change it, you can adapt to it. This is also true about most industrial software projects — there’s never a time to do a thing properly, you must do the best you can, given constraints. Let me use rust-analyzer as an example. The physical reality of the project is that it’s simultaneously very deep (it’s a compiler! Yay!) and very wide (opposite to an LLM, a classical IDE is a lot of purpose-built special features). The social reality is that “deep compiler” can attract a few brilliant dedicated contributors, and that the “breadth features” can be a good fit for an army of weekend warriors, people who learn Rust, who don’t have sustained capacity to participate in the project, but who can sink an hour or two to scratch their own itch. My insistence that doesn’t require building , that it builds on stable, that it doesn’t have any C dependencies, and that the entire test suite takes seconds, was in the service of the goal of attracting high-impact contributors. I was wrangling the build system to make sure people can work on the borrow checker without thinking about anything else. To attract weekend warriors, the internals of rust-analyzer are split into multiple independent features, where each feature is guarded by at runtime. The thinking was that I explicitly don’t want to care too much about quality there, that the bar for getting a feature PR in is “happy path works & tested”. It’s fine if the code crashes, it will only attract further contributors, provided that: In contrast, when working on the core spine which provided support for features, I was very relatively more pedantic about quality. A word of caution about adapting to, rather than fixing incentive structure — the future is uncertain, and tends to happen in the least convenient manner. The original motivation behind rust-analyzer experiment was to avoid the need to write a parallel compiler (the one in IntelliJ Rust), and to prototype a better architecture for LSP, so that the learnings could be backported to . So, even in core (especially in core), the code was very experimental. Oh well. Stuck with one more compiler now, I guess? I might hazard a guess that something similar happened to uutils project, which started as the primary destination for people learning Rust, and ended up as Ubuntu coreutils implementation. Third , now to some concrete recommendations. Sadly, I don’t know of a single book I can recommend which contains the truths. I suspect one can only find such a book in an apocryphal short story by Borges: practice seems to be an essential element here. But here are some things worth paying attention to: Boundaries talk by Gary Bernhardt is all-time favorite. It contains solid object-level advice, and, for me, it triggered the meta inquiry. How to Test is something I wish I had. I immediately understood the importance of testing, but it took me a long time to grow arrogant enough to admit that most widely-cited testing advice is shamanistic snake-oil, and to conceptualize what actually works. ∅MQ guide and, more generally, writings by Pieter Hintjens introduced me to Conway’s Law thinking. That “feature development” architecture of rust-analyzer? – optimistic merging , applied. Reflections on a decade of coding by Jamii is excellent, goes very meta. It is intentionally the first of my links . Ted Kaminski blog is the closest there is to a coherent theory of software development, appropriately framed as a set of notes to a non-existing book! As for the actual books, Software Engineering at Google and Ousterhout’s The Philosophy of Software Design are often recommended. They are good. SWE, in particular, helped me with a couple of important names . But they weren’t ground breaking for me. the quality is isolated to a feature, and doesn’t spill over, at runtime, the crash is invisible to the user (it’s crucial that rust-analyzer features work with an immutable snapshot, and can’t poison the data).

0 views
matklad 2 months ago

Steering Zig Fmt

Two tips on using effectively. Read this if you are writing Zig, or if you are implementing a code formatter. For me, is better than any other formatter I used: , the one in IntelliJ, . is steerable. For every syntactic construct, it has several variations for how it might be laid out. The variation used is selected by looking at what’s currently in a file. Easier to show a pair of examples: Depending on the trailing comma, function call is formatted on a single line, or with one argument per line. The way this plays out in practice is that you decide how you want to lay out the code, add a couple of , hit the reformat shortcut ( , p is mine ), and does the rest. For me, this works better than the alternative of the formatter guessing. 90% of great formatting are blank lines between logical blocks and tasteful choice of intermediate variables, so you might as well lean into key choices, rather than eliminate them. I know of one non-trivial formatting customization point: columnar layout for arrays: One would think that trailing comma would lead to a number-per-line layout, but, for arrays, also takes note of the first line break. In this case, the line break comes after the first three items, so we get three numbers per line, aligned: How cool is that! Furthermore, with judicious use of ( array concatenation ), you can vary the number of items per line. When I need to pass pairs to subprocess, I often go for formatting like this:

0 views
matklad 2 months ago

Minimal Viable Zig Error Contexts

Out of the box, Zig provides minimal and sufficient facilities for error handling — strongly-typed error codes . Error reporting is left to the user. Idiomatic solution is to pass a out parameter (“sink”) to materialize human-readable strings as needed. Diagnostics pattern works well for “production” code, but for more script-y code it adds too much friction relative to the default option of a plain , which of course gives a less than ideal message on failure: Error trace is helpful, but knowing which file is the problem is even more so. The first attempt at finding a middle ground between fully-fledged diagnostics sink pattern and a plain try is something like this: Unsatisfactory. The friction is high, you need to come up with a reasonably-sounding error message, the “happy path” of the code is obscured, and you need to repeat this for every fallible operation. A worse-is-better version of the above code is That is, just log error context as pairs, guarded by . The result is not pretty, but passable: The friction is reduced a lot: There’s one huge drawback though — the error message is logged, even if the error is subsequently handled. This is especially important in Zig 0.16, where cancelation ( serendipitous-success ) is a possible error for any IO-ing operation, and which is intended to be handled, rather than reported. Generalizing: This does feel like a better error management strategy than decorating errors individually, when they happen. I wonder which language features facilitate this style? No need to come up with any error messages beyond existing variable names. No need to change any of the s. The context is set per-block. If a function does several fallible operations on a file, the path needs to be specified only once. The context is “telescopic” every function in the call-stack can add its own context. Happy path adds context to all operations in-progress. Errors materialize current context.

0 views
matklad 2 months ago

256 Lines or Less: Test Case Minimization

Property Based Testing and fuzzing are a deep and science-intensive topic. There are enough advanced techniques there for a couple of PhDs, a PBT daemon, and a client-server architecture . But I have this weird parlor-trick PBT library, implementable in a couple of hundred lines of code in one sitting. This week I’ve been thinking about a cool variation of a consensus algorithm. I implemented it on the weekend. And it took just a couple of hours to write a PBT library itself first, and then a test, that showed a deep algorithmic flaw in my thinking (after a dozen trivial flaws in my coding). So, I don’t get to write more about consensus yet, but I at least can write about the library. It is very simple, simplistic even. To use an old Soviet joke about Babel and Bebel , it’s Gogol rather than Hegel. But for just 256 lines, it’s one of the highest power-to-weight ratio tools in my toolbox. Read this post if: Zig works well here because it, too, is exceptional in its power-to-weight. The implementation is a single file, , because the core abstraction here is a Finite Random Number Generator — a PRNG where all numbers are pre-generated, and can run out. We start with standard boilerplate: In Zig, files are structs: you obviously need structs, and the language becomes simpler if structs are re-used for what files are. In the above assigns a conventional name to the file struct, and declares instance fields (only one here). and are “static” (container level) declarations. The only field we have is just a slice of raw bytes, our pre-generated random numbers. And the only error condition we can raise is . The simplest thing we can generate is a slice of bytes. Typically, API for this takes a mutable slice as an out parameter: But, due to pre-generated nature of FRNG, we can return the slice directly, provided that we have enough entropy. This is going to be our (sole) basis function, everything else is going to be a convenience helper on top: The next simplest thing is an array (a slice with a fixed size): Notice how Zig goes from runtime-known slice length, to comptime known array type. Because is a constant, slicing with returns a pointer to array, . We can re-interpret a 4-byte array into . But, because this is Zig, we can trivially generalize the function to work for any integer type, by passing in comptime parameter of type : This function is monomorphised for every type, so becomes a compile-time constant we can pass to . Production code would be endian-clean here, but, for simplicity, we encode our endianness assumption as a compile-time assertion. Note how Zig communicates information about endianness to the program. There isn’t any kind of side-channel or extra input to compilation, like flags. Instead, the compiler materializes all information about target CPU as Zig code. There’s a file somewhere in the compiler caches directory that contains This file can be accessed via and all the constants inspected at compile time. We can make an integer, and a boolean is even easier: Strictly speaking, we only need one bit, not one byte, but tracking individual bits is too much of a hassle. From an arbitrary int, we can generate an int in range. As per Random Numbers Included , we use a closed range, which makes the API infailable and is usually more convenient at the call-site: As a bit of PRNG trivia, while this could be implemented as , the result will be biased (not uniform). Consider the case where , and a call like . The numbers in are going to be twice as likely as the numbers in , because the last quarter of 256 range will be aliased with the first one. Generating an unbiased number is tricky and might require drawing arbitrary number of bytes from entropy. Refer to https://www.pcg-random.org/posts/bounded-rands.html for details. I didn’t, and copy-pasted code from the Zig standard library. Use at your own risk! Now we can generate an int bounded from above and below: Another common operation is picking a random element from a slice. If you want to return a pointer to a element, you’ll need a and versions of the function. A simpler and more general solution is to return an index: At the call site, doesn’t look too bad, is appropriately -polymorphic, and is also usable for multiple parallel arrays. So far, we’ve spent about 40% of our line budget implementing a worse random number generator that can fail with at any point in time. What is it good for? We use it to feed our system under test with random inputs, see how it reacts, and check that it does not crash. If we code our system to crash if anything unexpected happens and our random inputs cover the space of all possible inputs, we get a measure of confidence that bugs will be detected in testing. For my consensus simulation, I have a struct that holds a and a set of replicas: has methods like: I then select which method to call at random: Here, is another FRNG helper that selects an action at random, proportional to its weight. This helper needs quite a bit more reflection machinery than we’ve seen so far: is compile-time duck-typing. It means that our function is callable with any type, and each specific type creates a new monomorphised instance of a function. While we don’t explicitly name the type of , we can get it as . is a type-level function that takes a struct type: and turns it into an enum type, with a variant per-field, exactly what we want for the return type: Tip: if you want to quickly learn Zig’s reflection capabilities, study the implementation of and in Zig’s standard library. The built-in function accesses a field given field name. It’s exactly like Python’s / with an extra restriction that it must be evaluated at compile time. To add one more twist here, I always find it hard to figure out which weights are reasonable, and like to generate the weights themselves at random at the start of the test: (If you feel confused here, check out Swarm Testing Data Structures ) Now we have enough machinery to describe the shape of test overall: A test needs an (which ultimately determines the outcome) and an General Purpose Allocator for the . We start by creating a simulated with random action weights. If entropy is very low, we can run out of entropy even at this stage. We assume that the code is innocent until proven guilty — if we don’t have enough entropy to find a bug, this particular test returns success. Don’t worry, we’ll make sure that we have enough entropy elsewhere. We use to peel off error. I find that, whenever I handle errors in Zig, very often I want to discharge just a single error from the error set. I wish I could use parenthesis with a : Anyway, having created the , we step through it while we still have entropy left. If any step detects an internal inconsistency, the entire crashes with an assertion failure. If we got to the end of loop, we know that at least that particular slice of entropy didn’t uncover anything suspicious. Notice what isn’t there. We aren’t generating a complete list of actions up-front. Rather, we make random decisions as we go, and can freely use the current state of the to construct a menu of possible choices (e.g., when sending a message, we can consider only not currently crashed replicas). And here we can finally see the reason why we bothered writing a custom Finite PRNG, rather than using an off-the-shelf one. The amount of entropy in FRNG defines the complexity of the test. The fewer random bytes we start with, the faster we exit the step loop. And this gives us an ability to minimize test cases essentially for free. Suppose you know that a particular entropy slice makes the test fail (cluster enters split brain at the millionth step). Let’s say that the slice was 16KiB. The obvious next step is to see if just 8KiB would be enough to crash it. And, if 8KiB isn’t, than perhaps 12KiB? You can binary search the minimal amount of entropy that’s enough for the test to fail. And this works for any test, it doesn’t have to be a distributed system. If you can write the code to generate your inputs randomly, you can measure complexity of each particular input by measuring how many random bytes were drawn in its construction. And now the hilarious part — of course it seems that the way to minimize entropy is to start with a particular failing slice and apply genetic-algorithm mutations to it. But a much simpler approach seems to work in practice — just generated a fresh, shorter entropy slice. If you found some failure at random, then you should be able to randomly stumble into a smaller failing example, if one exists — there are much fewer small examples, so finding a failing one becomes easier when the goes down! The problem with binary searching for failing entropy is that a tripped assertion crashes the program. There’s no unwinding in Zig. For this reason, we’ll move the search code to a different process. So a single test will be a binary with a function, that takes entropy on . Zig’s new juicy main makes writing this easier than in any previous versions of Zig :D Main gets as an argument, which provides access to things like command line arguments, default allocator and a default implementation. These days, Zig eschews global ambient IO capabilities, and requires threading an Io instance whenever we need to make a syscall. Here, we need Io to read stdin. Now we will implement a harness to call this main. This will be : It will be spawning external processes, so it’ll need an . We also need a path to an executable with a test main function, a System Under Test. And we’ll need a buffer to hold the entropy. This driver will be communicating successes and failures to the users, so we also prepare a for textual output. How we get entropy to feed into ? Because we are only interested in entropy size, we won’t be storing the actual entropy bytes, and instead will generate it from a seed. In other words, just two numbres, entropy size and seed, are needed to reproduce a single run of the test: We use default deterministic PRNG to expand our short seed into entropy slice of the required size. Then we spawn proces, feeding the resulting entropy via stdin. Closing child’s stdin signals the end of entropy. We then return either or depending on child’s exit code. So, both explicit errors and crashes will be recognized as failures. Next, we implement the logic for checking if a particular seed size is sufficient to find a failure. Of course, we won’t be able to say that for sure in a finite amount of time, so we’ll settle for some user-specified amount of retries: The user passes us the number of to make, and we return if they all were successfull, or a specific failing seed if we found one: To generate a real seed we need “true” cryptographic non-deterministic randomness, which is provided by . Finally, the search for the size: Here, we are going to find a smallest entropy size that crashes . If we succeed, we return the seed and the size. The upper bound for the size is the space available in the pre-allocated entropy buffer. The search loop is essentially a binary search, with a twist — rather than using dichotomy on the directly, we will be doubling a we use to change the size between iterations. That is, we start with a small size and step, and, on every iteration, double the step and add it to the size, until we hit a failure (or run out of buffer for the entropy). Once we found a failure, we continue the serach in the other direction — halving the step and subtracting it from the , keeping the smaller if it still fails. On each step, we log the current size and outcome, and report the smallest failing size at the end. Finally, we wrap Driver’s functionality into main that works in two modes — either reproduces a given failure from seed and size, or searches for a minimal failure: Running the search routine looks like this in a terminal: Those final seed&size can then be used for , giving you a minimal reproducible failure for debugging! This … of course doesn’t look too exciting without visualizing a specific bug we can find this way, but the problem there is that interesting examples of systems to test in this way usually take more than 256 lines to implement. So I’ll leave it to your imagination, but you get the idea: if you can make a system fail under a “random” input, you can also systematically search the space of all inputs for the smallest counter-example, without adding knowledge about the system to the searcher. This article also provides a concrete (but somewhat verbose) example. Here’s the full code: https://gist.github.com/matklad/343d13547c8bfe9af310e2ca2fbfe109 You want to stretch your generative testing muscles. You are a do-it-yourself type, and wouldn’t want to pull a ginormous PBT library off the shelf. You would pull a library, but want to have a more informed opinion about available options, about essential and accidental complexity. You want some self-contained real-world Zig examples :P

0 views
matklad 3 months ago

Consensus Board Game

I have an early adulthood trauma from struggling to understand consensus amidst a myriad of poor explanations. I am overcompensating for that by adding my own attempts to the fray. Today, I want to draw a series of pictures which could be helpful. You can see this post as a set of missing illustrations for Notes on Paxos , or, alternatively, you can view that post as a more formal narrative counter-part for the present one. The idea comes from my mathematics of consensus lecture, with versions in English and Russian . I am going to aggressively hand wave the details away, please refer to Notes for filling in the blanks. And, before we begin, I want to stress again that here I am focusing strictly on the mathematics behind the algorithm, on the logical structure of the universe that makes some things impossible, and others doable. Consensus is but a small part of the engineering behind real data management systems, and I might do something about pragmatics of consensus at some point, just not today ;) There’s a committee of five members that tries to choose a color for a bike shed, but the committee members are not entirely reliable. We want to arrive at a decision even if some members of the committee are absent. The fundamental idea underpinning consensus is simple majority vote. If R0, … R4 are the five committee members, we can use the following board to record the votes: A successful vote looks like this: Here, red collected 3 out of 5 votes and wins. Note that R4 hasn’t voted yet. It might, or might not do so eventually, but that won’t affect the outcome. The problem with voting is that it can get stuck like this: Here, we have two votes for red, two votes for blue, but the potential tie-breaker, R4, voted for green, the rascal! To solve split vote, we are going to designate R0 as the committee’s leader, make it choose the color, and allow others only to approve. Note that meaningful voting still takes place, as someone might abstain from voting — you need at least 50% turnout for the vote to be complete: Here, R0, the leader (marked with yellow napoleonic bicorne), choose red, R2 and R3 acquiesced, so the red “won”, even as R1 and R4 abstained (x signifies absence of a vote). The problem with this is that our designated leader might be unavailable itself: Which brings us to the central illustration that I wanted to share. What are we going to do now is to multiply our voting. Instead of conducting just one vote with a designated leader, the committee will conduct a series of concurrent votes, where the leaders rotate in round-robin pattern. This gives rise to the following half-infinite 2D board on which the game of consensus is played: Each column plays independently. If you are a leader in a column, and your cell is blank, you can choose whatever color. If you are a follower, you need to wait until column’s leader decision, and then you can either fill the same color, or you can abstain. After several rounds the board might end up looking like this: The benefit of our 2D setup is that, if any committee member is unavailable, their columns might get stuck, but, as long as the majority is available, some column somewhere might still complete. The drawback is that, while individual column’s decision is clear and unambiguous, the outcome of the board as whole is undefined. In the above example, there’s a column where red wins, and a column where blue wins. So what we are going to do is to scrap the above board as invalid, and instead require that any two columns that achieved majorities must agree on the color. In other words, the outcome of the entire board is the outcome of any of its columns, whichever finishes first, and the safety condition is that no two colors can reach majorities in different columns. Let’s take a few steps back when the board wasn’t yet hosed, and try to think about the choice of the next move from the perspective of R3: As R3 and the leader for your column, you need to pick a color which won’t conflict with any past or future decisions in other columns. Given that there are some greens and blues already, it feels like maybe you shouldn’t pick red… But it could be the case that the three partially filled columns won’t move anywhere in the future, and the first column gets a solid red line! Though choices! You need to worry about the future and the infinite number of columns to your right! Luckily, the problem can be made much easier if we assume that everyone plays by the same rules, in which case it’s enough to only worry about the columns to your left. Suppose that you, and everyone else is carefully choosing their moves to not conflict with the columns to the left. Then, if you chose red, your column wins, and subsequently some buffoon on the right chooses green, it’s their problem, because you are to their left. So let’s just focus on the left part of the board. Again, it seems like blue or green might be good bets, as they are already present on the board, but there’s a chance that the first column will eventually vote for red. To prevent that, what we are going to do is to collect a majority of participants (R0, R2, R3) and require them to commit to not voting in the first columns. Actually, for that matter, let’s prevent them from voting in any column to the left: Here, you asked R0, R2 and R3 to abstain from casting further votes in the first three columns, signified by black x. With this picture, we can now be sure that red can not win in the first column — no color can win there, because only two out of the five votes are available there! Still, we have the choice between green and blue, which one should we pick? The answer is the rightmost. R2, the participant that picked blue in the column to our immediate left, was executing the same algorithm. If they picked blue, they did it because they knew for certain that the second column can’t eventually vote for green. R2 got a different majority of participants to abstain from voting in the second column, and, while we, as R3, don’t know which majority that was, we know that it exists because we know that R2 did pick blue, and we assume fair play. That’s all for today, that’s the trick that makes consensus click, in the abstract. In a full distributed system the situation is more complicated. Each participant only sees its own row, the board as a whole remains concealed. Participants can learn something about others’ state by communicating, but the knowledge isn’t strongly anchored at time. By the time a response is received the answer could as well be obsolete. And yet, the above birds-eye view can be implemented in a few exchanges of messages. Please see the Notes for further details.

0 views
matklad 4 months ago

JJ LSP Follow Up

In Majjit LSP , I described an idea of implementing Magit style UX for jj once and for all, leveraging LSP protocol. I’ve learned today that the upcoming 3.18 version of LSP has a feature to make this massively less hacky: Text Document Content Request LSP can now provide virtual documents, which aren’t actually materialized on disk. So this: can now be such a virtual document, where highlighting is provided by semantic tokens, things like “check out this commit” are code actions, and “goto definition” jumps from the diff in the virtual file to a real file in the working tree.

0 views
matklad 4 months ago

Wrapping Code Comments

I was today years old when I realized that: It’s a good idea to limit line length to about 100 columns. This is a physical limit, the width at which you can still comfortably fit two editors side by side (see Size Matters ). Note an apparent contradiction: the optimal width for readable prose is usually taken to be narrower, 60–70 columns. The contradiction is resolved by noticing that, for code, indentation eats into usable space. Typically, code is much less typographically dense than prose. Still, I find comment blocks easier to read when they are wrapped narrower than the surrounding code. I want lines to be wrapped at 100, and content of comments to be wrapped at 70 (unless that pushes overall line to be longer than 100). That is, I want layout like this (using 20/30 rulers instead of 70/100, for illustrative purposes): This feels obvious in retrospect, but notably isn’t be well-supported by the tools? The VS Code extension I use allows configuring dedicated fill column for comments, but doesn’t make it relative , so indented comment blocks are always narrower than top-level ones. Emacs also doesn’t do relative wrapping out of the box! Aside on hard-wrapping: should we bother with wrapping comments at all? Can’t we rely on our editor to implement soft-wrapping? The problem with soft-wrapping is that you can’t soft-wrap text correctly without understanding its meaning. Consider a markdown list: If the first item is long enough to necessitate wrapping, the wrapped line should also be indented, which requires parsing the text as markdown first: Code and code comments ideally should be wrapped to a different column. For comments, the width should be relative to the start of the comment.

0 views
matklad 5 months ago

CI In a Box

I wrote , a thin wrapper around ssh for running commands on remote machines. I want a box-shaped interface for CI: That is, the controlling CI machine runs a user-supplied script, whose status code will be the ultimate result of a CI run. The script doesn’t run the project’s tests directly. Instead, it shells out to a proxy binary that forwards the command to a runner box with whichever OS, CPU, and other environment required. The hard problems are in the part: CI discourse amuses me — everyone complains about bad YAML, and it is bad, but most of the YAML (and associated reproducibility and debugging problems) is avoidable. Pick an appropriate position on a dial that includes What you can’t just do by writing a smidgen of text is getting the heterogeneous fleet of runners. And you need heterogeneous fleet of runners if some of the software you are building is cross-platform. If you go that way, be mindful that The SSH wire protocol only takes a single string as the command, with the expectation that it should be passed to a shell by the remote end. In other words, while SSH supports syntax like , it just blindly intersperses all arguments with a space. Amusing to think that our entire cloud infrastructure is built on top of shell injection ! This, and the need to ensure no processes are left behind unintentionally after executing a remote command, means that you can’t “just” use SSH here if you are building something solid. One of them is not UNIX. One of them has licensing&hardware constraints that make per-minute billed VMs tricky (but not impossible, as GitHub Actions does that). All of them are moving targets, and require someone to do the OS upgrade work, which might involve pointing and clicking . writing a bash script, writing a script in the language you already use , using a small build system , using a medium-sized one like or , or using a large one like or .

0 views
matklad 5 months ago

Vibecoding #2

I feel like I got substantial value out of Claude today, and want to document it. I am at the tail end of AI adoption, so I don’t expect to say anything particularly useful or novel. However, I am constantly complaining about the lack of boring AI posts, so it’s only proper if I write one. At TigerBeetle, we are big on deterministic simulation testing . We even use it to track performance , to some degree. Still, it is crucial to verify performance numbers on a real cluster in its natural high-altitude habitat. To do that, you need to procure six machines in a cloud, get your custom version of binary on them, connect cluster’s replicas together and hit them with load. It feels like, quarter of a century into the third millennium, “run stuff on six machines” should be a problem just a notch harder than opening a terminal and typing , but I personally don’t know how to solve it without wasting a day. So, I spent a day vibecoding my own square wheel. The general shape of the problem is that I want to spin a fleet of ephemeral machines with given specs on demand and run ad-hoc commands in a SIMD fashion on them. I don’t want to manually type slightly different commands into a six-way terminal split, but I also do want to be able to ssh into a specific box and poke it around. My idea for the solution comes from these three sources: The big idea of is that you can program distributed system in direct style. When programming locally, you do things by issuing syscalls: This API works for doing things on remote machines, if you specify which machine you want to run the syscall on: Direct manipulation is the most natural API, and it pays to extend it over the network boundary. Peter’s post is an application of a similar idea to a narrow, mundane task of developing on Mac and testing on Linux. Peter suggests two scripts: synchronizes a local and remote projects. If you run inside folder, then materializes on the remote machine. does the heavy lifting, and the wrapper script implements behaviors. It is typically followed by , which runs command on the remote machine in the matching directory, forwarding output back to you. So, when I want to test local changes to on my Linux box, I have roughly the following shell session: The killer feature is that shell-completion works. I first type the command I want to run, taking advantage of the fact that local and remote commands are the same, paths and all, then hit and prepend (in reality, I have alias that combines sync&run). The big thing here is not the commands per se, but the shift in the mental model. In a traditional ssh & vim setup, you have to juggle two machines with a separate state, the local one and the remote one. With , the state is the same across the machines, you only choose whether you want to run commands here or there. With just two machines, the difference feels academic. But if you want to run your tests across six machines, the ssh approach fails — you don’t want to re-vim your changes to source files six times, you really do want to separate the place where the code is edited from the place(s) where the code is run. This is a general pattern — if you are not sure about a particular aspect of your design, try increasing the cardinality of the core abstraction from 1 to 2. The third component, library, is pretty mundane — just a JavaScript library for shell scripting. The notable aspects there are: JavaScript’s template literals , which allow implementing command interpolation in a safe by construction way. When processing , a string is never materialized, it’s arrays all the way to the syscall ( more on the topic ). JavaScript’s async/await, which makes managing concurrent processes (local or remote) natural: Additionally, deno specifically valiantly strives to impose process-level structured concurrency, ensuring that no processes spawned by the script outlive the script itself, unless explicitly marked — a sour spot of UNIX. Combining the three ideas, I now have a deno script, called , that provides a multiplexed interface for running ad-hoc code on ad-hoc clusters. A session looks like this: I like this! Haven’t used in anger yet, but this is something I wanted for a long time, and now I have it The problem with implementing above is that I have zero practical experience with modern cloud. I only created my AWS account today, and just looking at the console interface ignited the urge to re-read The Castle. Not my cup of pu-erh. But I had a hypothesis that AI should be good at wrangling baroque cloud API, and it mostly held. I started with a couple of paragraphs of rough, super high-level description of what I want to get. Not a specification at all, just a general gesture towards unknown unknowns. Then I asked ChatGPT to expand those two paragraphs into a more or less complete spec to hand down to an agent for implementation. This phase surfaced a bunch of unknowns for me. For example, I wasn’t thinking at all that I somehow need to identify machines, ChatGPT suggested using random hex numbers, and I realized that I do need 0,1,2 naming scheme to concisely specify batches of machines. While thinking about this, I realized that sequential numbering scheme also has an advantage that I can’t have two concurrent clusters running, which is a desirable property for my use-case. If I forgot to shutdown a machine, I’d rather get an error on trying to re-create a machine with the same name, then to silently avoid the clash. Similarly, turns out the questions of permissions and network access rules are something to think about, as well as what region and what image I need. With the spec document in hand, I turned over to Claude code for actual implementation work. The first step was to further refine the spec, asking Claude if anything is unclear. There were couple of interesting clarifications there. First, the original ChatGPT spec didn’t get what I meant with my “current directory mapping” idea, that I want to materialize a local as remote , even if are different. ChatGPT generated an incorrect description and an incorrect example. I manually corrected example, but wasn’t able to write a concise and correct description. Claude fixed that working from the example. I feel like I need to internalize this more — for current crop of AI, examples seem to be far more valuable than rules. Second, the spec included my desire to auto-shutdown machines once I no longer use them, just to make sure I don’t forget to turn the lights off when leaving the room. Claude grilled me on what precisely I want there, and I asked it to DWIM the thing. The spec ended up being 6KiB of English prose. The final implementation was 14KiB of TypeScript. I wasn’t keeping the spec and the implementation perfectly in sync, but I think they ended up pretty close in the end. Which means that prose specifications are somewhat more compact than code, but not much more compact. My next step was to try to just one-shot this. Ok, this is embarrassing, and I usually avoid swearing in this blog, but I just typoed that as “one-shit”, and, well, that is one flavorful description I won’t be able to improve upon. The result was just not good (more on why later), so I almost immediately decided to throw it away and start a more incremental approach. In my previous vibe-post , I noticed that LLM are good at closing the loop. A variation here is that LLMs are good at producing results, and not necessarily good code. I am pretty sure that, if I had let the agent to iterate on the initial script and actually run it against AWS, I would have gotten something working. I didn’t want to go that way for three reasons: And, as I said, the code didn’t feel good, for these specific reasons: The incremental approach worked much better, Claude is good at filling-in the blanks. The very first thing I did for was manually typing-in: Then I asked Claude to complete the function, and I was happy with the result. Note Show, Don’t Tell I am not asking Claude to avoid throwing an exception and fail fast instead. I just give function, and it code-completes the rest. I can’t say that the code inside is top-notch. I’d probably written something more spartan. But the important part is that, at this level, I don’t care. The abstraction for parsing CLI arguments feel right to me, and the details I can always fix later. This is how this overall vibe-coding session transpired — I was providing structure, Claude was painting by the numbers. In particular, with that CLI parsing structure in place, Claude had little problem adding new subcommands and new arguments in a satisfactory way. The only snag was that, when I asked to add an optional path to , it went with , while I strongly prefer . Obviously, its better to pick your null in JavaScript and stick with it. The fact that is unavoidable predetermines the winner. Given that the argument was added as an incremental small change, course-correcting was trivial. The null vs undefined issue perhaps illustrates my complaint about the code lacking character. is the default non-choice. is an insight, which I personally learned from VS Code LSP implementation. The hand-written skeleton/vibe-coded guts worked not only for the CLI. I wrote and then asked Claude to write the body of a particular function according to the SPEC.md. Unlike with the CLI, Claude wasn’t able to follow this pattern itself. With one example it’s not obvious, but the overall structure is that is the AWS-level operation on a single box, and is the CLI-level control flow that deals with looping and parallelism. When I asked Claude to implement , without myself doing the / split, Claude failed to noticed it and needed a course correction. However , Claude was massively successful with the actual logic. It would have taken me hours to acquire specific, non-reusable knowledge to write: I want to be careful — I can’t vouch for correctness and especially completeness of the above snippet. However, given that the nature of the problem is such that I can just run the code and see the result, I am fine with it. If I were writing this myself, trial-and-error would totally be my approach as well. Then there’s synthesis — with several instance commands implemented, I noticed that many started with querying AWS to resolve symbolic machine name, like “1”, to the AWS name/IP. At that point I realized that resolving symbolic names is a fundamental part of the problem, and that it should only happen once, which resulting in the following refactored shape of the code: Claude was ok with extracting the logic, but messed up the overall code layout, so the final code motions were on me. “Context” arguments go first , not last, common prefix is more valuable than common suffix because of visual alignment. The original “one-shotted” implementation also didn’t do up-front querying. This is an example of a shape of a problem I only discover when working with code closely. Of course, the script didn’t work perfectly the first time and we needed quite a few iterations on the real machines both to fix coding bugs, as well gaps in the spec. That was an interesting experience of speed-running rookie mistakes. Claude made naive bugs, but was also good at fixing them. For example, when I first tried to after , I got an error. Pasting it into Claude immediately showed the problem. Originally, the code was doing and not . The former checks if instance is logically created, the latter waits until the OS is booted. It makes sense that these two exist, and the difference is clear (and its also clear that OS booted != SSH demon started). Claude’s value here is in providing specific names for the concepts I already know to exist. Another fun one was about the disk. I noticed that, while the instance had an SSD, it wasn’t actually used. I asked Claude to mount it as home, but that didn’t work. Claude immediately asked me to run and that log immediately showed the problem. This is remarkable! 50% of my typical Linux debugging day is wasted not knowing that a useful log exists, and the other 50% is for searching for the log I know should exist somewhere . After the fix, I lost the ability to SSH. Pasting the error immediately gave the answer — by mounting over , we were overwriting ssh keys configured prior. There were couple of more iterations like that. Rookie mistakes were made, but they were debugged and fixed much faster than my personal knowledge allows (and again, I feel that is trivia knowledge, rather than deep reusable knowledge, so I am happy to delegate it!). It worked satisfactorily in the end, and, what’s more, I am happy to maintain the code, at least to the extent that I personally need it. Kinda hard to measure productivity boost here, but, given just the sheer number of CLI flags required to make this work, I am pretty confident that time was saved, even factoring the writing of the present article! I’ve recently read The Art of Doing Science and Engineering by Hamming (of distance and code), and one story stuck with me: A psychologist friend at Bell Telephone Laboratories once built a machine with about 12 switches and a red and a green light. You set the switches, pushed a button, and either you got a red or a green light. After the first person tried it 20 times they wrote a theory of how to make the green light come on. The theory was given to the next victim and they had their 20 tries and wrote their theory, and so on endlessly. The stated purpose of the test was to study how theories evolved. But my friend, being the kind of person he was, had connected the lights to a random source! One day he observed to me that no person in all the tests (and they were all high-class Bell Telephone Laboratories scientists) ever said there was no message. I promptly observed to him that not one of them was either a statistician or an information theorist, the two classes of people who are intimately familiar with randomness. A check revealed I was right! https://github.com/catern/rsyscall https://peter.bourgon.org/blog/2011/04/27/remote-development-from-mac-to-linux.html https://github.com/dsherret/dax JavaScript’s template literals , which allow implementing command interpolation in a safe by construction way. When processing , a string is never materialized, it’s arrays all the way to the syscall ( more on the topic ). JavaScript’s async/await, which makes managing concurrent processes (local or remote) natural: Additionally, deno specifically valiantly strives to impose process-level structured concurrency, ensuring that no processes spawned by the script outlive the script itself, unless explicitly marked — a sour spot of UNIX. Spawning VMs takes time, and that significantly reduces the throughput of agentic iteration. No way I let the agent run with a real AWS account, given that AWS doesn’t have a fool-proof way to cap costs. I am fairly confident that this script will be a part of my workflow for at least several years, so I care more about long-term code maintenance, than immediate result. It wasn’t the code that I would have written, it lacked my character, which made it hard for me to understand it at a glance. The code lacked any character whatsoever. It could have worked, it wasn’t “naively bad”, like the first code you write when you are learning programming, but there wasn’t anything good there. I never know what the code should be up-front. I don’t design solutions, I discover them in the process of refactoring. Some of my best work was spending a quiet weekend rewriting large subsystems implemented before me, because, with an implementation at hand, it was possible for me to see the actual, beautiful core of what needs to be done. With a slop-dump, I just don’t get to even see what could be wrong. In particular, while you are working the code (as in “wrought iron”), you often go back to requirements and change them. Remember that ambiguity of my request to “shut down idle cluster”? Claude tried to DWIM and created some horrific mess of bash scripts, timestamp files, PAM policy and systemd units. But the right answer there was “lets maybe not have that feature?” (in contrast, simply shutting the machine down after 8 hours is a one-liner).

2 views
matklad 6 months ago

The Second Great Error Model Convergence

I feel like this has been said before, more than once, but I want to take a moment to note that most modern languages converged to the error management approach described in Joe Duffy’s The Error Model , which is a generational shift from the previous consensus on exception handling. C++, JavaScript, Python, Java, C# all have roughly equivalent , , constructs with roughly similar runtime semantics and typing rules. Even functional languages like Haskell, OCaml, and Scala feature exceptions prominently in their grammar, even if their usage is frowned upon by parts of the community. But the same can be said about Go, Rust, Swift, and Zig! Their error handling is similar to each other, and quite distinct from the previous bunch, with Kotlin and Dart being notable, ahem, exceptions. Here are some commonalities of modern error handling: First , and most notably, functions that can fail are annotated at the call side. While the old way looked like this: the new way is There’s a syntactic marker alerting the reader that a particular operation is fallible, though the verbosity of the marker varies. For the writer, the marker ensures that changing the function contract from infallible to fallible (or vice versa) requires changing not only the function definition itself, but the entire call chain. On the other hand, adding a new error condition to a set of possible errors of a fallible function generally doesn’t require reconsidering rethrowing call-sites. Second , there’s a separate, distinct mechanism that is invoked in case of a detectable bug. In Java, index out of bounds or null pointer dereference (examples of programming errors) use the same language machinery as operational errors. Rust, Go, Swift, and Zig use a separate panic path. In Go and Rust, panics unwind the stack, and they are recoverable via a library function. In Swift and Zig, panic aborts the entire process. Operational error of a lower layer can be classified as a programming error by the layer above, so there’s generally a mechanism to escalate an erroneous result value to a panic. But the opposite is more important: a function which does only “ordinary” computations can be buggy, and can fail, but such failures are considered catastrophic and are invisible in the type system, and sufficiently transparent at runtime. Third , results of fallible computation are first-class values, as in Rust’s . There’s generally little type system machinery dedicated exclusively to errors and expressions are just a little more than syntax sugar for that little Go spell. This isn’t true for Swift, which does treat errors specially. For example, the generic function has to explicitly care about errors, and hard-codes the decision to bail early: Swift does provide first-classifier type for errors. Should you want to handle an exception, rather than propagate it, the handling is localized to a single throwing expression to deal with a single specific errors, rather than with any error from a block of statements: Swift again sticks to more traditional try catch, but, interestingly, Kotlin does have expressions. The largest remaining variance is in what the error value looks like. This still feels like a research area. This is a hard problem due to a fundamental tension: The two extremes are well understood. For exhaustiveness, nothing beats sum types ( s in Rust). This I think is one of the key pieces which explains why the pendulum seemingly swung back on checked exceptions. In Java, a method can throw one of the several exceptions: Critically, you can’t abstract over this pair. The call chain has to either repeat the two cases, or type-erase them into a superclass, losing information. The former has a nasty side-effect that the entire chain needs updating if a third variant is added. Java-style checked exceptions are sensitive to “N to N + 1” transitions. Modern value-oriented error management is only sensitive to “0 to 1” transition. Still, if I am back to writing Java at any point, I’d be very tempted to standardize on coarse-grained signature for all throwing methods. This is exactly the second well understood extreme: there’s a type-erased universal error type, and the “throwableness” of a function contains one bit of information. We only care if the function can throw, and the error itself can be whatever. You still can downcast dynamic error value handle specific conditions, but the downcasting is not checked by the compiler. That is, downcasting is “save” and nothing will panic in the error handling mechanism itself, but you’ll never be sure if the errors you are handling can actually arise, and whether some errors should be handled, but aren’t. Go and Swift provide first-class universal errors, like Midori. Starting with Swift 4, you can also narrow the type down. Rust doesn’t really have super strong conventions about the errors, but it started with mostly enums, and then and shone spotlight on the universal error type. But overall, it feels like “midpoint” error handling is poorly served by either extreme. In larger applications, you sorta care about error kinds, and there are usually a few place where it is pretty important to be exhaustive in your handling, but threading necessary types to those few places infects the rest of the codebases, and ultimately leads to “a bag of everything” error types with many “dead” variants. Zig makes an interesting choice of assuming mostly closed-world compilation model, and relying on cross-function inference to learn who can throw what. What I find the most fascinating about the story is the generational aspect. There really was a strong consensus about exceptions, and then an agreement that checked exceptions are a failure , and now, suddenly, we are back to “checked exceptions” with a twist, in the form of “errors are values” philosophy. What happened between the lull of the naughts and the past decade industrial PLT renaissance? On the one hand, at lower-levels you want to exhaustively enumerate errors to make sure that: internal error handling logic is complete and doesn’t miss a case, public API doesn’t leak any extra surprise error conditions. On the other hand, at higher-levels, you want to string together widely different functionality from many separate subsystems without worrying about specific errors, other than: separating fallible functions from infallible, ensuring that there is some top-level handler to show a 500 error or an equivalent.

0 views
matklad 6 months ago

Parsing Advances

I find myself writing yet another toy parser, as one does during a Christmas break. It roughly follows Resilient LL Parsing Tutorial . Not because I need resilience, but mostly because I find producing a syntax tree and a collection of diagnostics a more natural fit for the problem than bailing out on the first error. One practical pitfall with the approach is infinite loops/recursion. Resilience sometimes means not consuming a token, and, if you do that in a loop or a Pratt recursive call, you’ll get yourself an annoying to debug error: For a concrete example, you might parse function argument list using code like this: The implicit contract here is that consumes at least one token, even if there are errors in the source code. If there’s some token that makes bail without consuming anything, the code loops forever, and you’ll need a debugger to get at the stack trace! The way I solved this issue traditionally is via a combination of two techniques: Fuel: parser has a field, which is decremented even by “readonly” lookahead methods, and topped up every time the parser consumes a token. Fuel is useful to make you parser crash somewhat cleanly, though the crash is typically still removed from problematic function by several stack frames. The second technique is to maintain a mental map of functions which always consume at least one token of input, and functions which might bail without consuming anything. And, whenever you write a loop or a recursive call, consult this map to be sure to call at least one token-consuming function. Hard and error prone! Well, I think I’ve figured something better today! You can assert that parser did advance when you expect it to. The smaller benefit here is that if parser didn’t advance, you get an immediate error. The bigger benefit is that these asserts materialize the mental map of advancing functions in the source code, so it doesn’t have to be mental anymore! This seems like an obvious idea in retrospect, but, well, took me more than one parser to figure it out! Concretely, I came up with the following base parser API: And here is the buggy function that lead to the error at the start of the article: The same function, but with advanced assertions: The new error message: and the fix:

0 views
matklad 6 months ago

Static Allocation For Compilers

TigerBeetle famously uses “static allocation” . Infamously, the use of the term is idiosyncratic: what is meant is not arrays, as found in embedded development, but rather a weaker “no allocation after startup” form. The amount of memory TigerBeetle process uses is not hard-coded into the Elf binary. It depends on the runtime command line arguments. However, all allocation happens at startup, and there’s no deallocation. The long-lived event loop goes round and round happily without . I’ve wondered for years if a similar technique is applicable to compilers. It seemed impossible, but today I’ve managed to extract something actionable from this idea? Static allocation depends on the physics of the underlying problem. And distributed databases have surprisingly simple physics, at least in the case of TigerBeetle. The only inputs and outputs of the system are messages. Each message is finite in size (1MiB). The actual data of the system is stored on disk and can be arbitrarily large. But the diff applied by a single message is finite. And, if your input is finite, and your output is finite, it’s actually quite hard to need to allocate extra memory! This is worth emphasizing — it might seem like doing static allocation is tough and requires constant vigilance and manual accounting for resources. In practice, I learned that it is surprisingly compositional. As long as inputs and outputs of a system are finite, non-allocating processing is easy. And you can put two such systems together without much trouble. routing.zig is a good example of such an isolated subsystem. The only issue here is that there isn’t a physical limit on how many messages can arrive at the same time. Obviously, you can’t process arbitrary many messages simultaneously. But in the context of a distributed system over an unreliable network, a safe move is to drop a message on the floor if the required processing resources are not available. Counter-intuitively, not allocating is simpler than allocating, provided that you can pull it off! Alas, it seems impossible to pull it off for compilers. You could say something like “hey, the largest program will have at most one million functions”, but that will lead to both wasted memory and poor user experience. You could also use a single yolo arena of a fixed size, like I did in Hard Mode Rust , but that isn’t at all similar to “static allocation”. With arenas, the size is fixed explicitly, but you can OOM. With static allocation it is the opposite — no OOM, but you don’t know how much memory you’ll need until startup finishes! The “problem size” for a compiler isn’t fixed — both the input (source code) and the output (executable) can be arbitrarily large. But that is also the case for TigerBeetle — the size of the database is not fixed, it’s just that TigerBeetle gets to cheat and store it on disk, rather than in RAM. And TigerBeetle doesn’t do “static allocation” on disk, it can fail with at runtime, and it includes a dynamic block allocator to avoid that as long as possible by re-using no longer relevant sectors. So what we could say is that a compiler consumes arbitrarily large input, and produces arbitrarily large output, but those “do not count” for the purpose of static memory allocation. At the start, we set aside an “output arena” for storing finished, immutable results of compiler’s work. We then say that this output is accumulated after processing a sequence of chunks, where chunk size is strictly finite. While limiting the total size of the code-base is unreasonable, limiting a single file to, say, 4 MiB (runtime-overridable) is fine. Compiling then essentially becomes a “stream processing” problem, where both inputs and outputs are arbitrary large, but the filter program itself must execute in O(1) memory. With this setup, it is natural to use indexes rather than pointers for “output data”, which then makes it easy to persist it to disk between changes. And it’s also natural to think about “chunks of changes” not only spatially (compiler sees a new file), but also temporally (compiler sees a new version of an old file). Is there any practical benefits here? I don’t know! But seems worth playing around with! I feel that a strict separation between O(N) compiler output and O(1) intermediate processing artifacts can clarify compiler’s architecture, and I won’t be too surprised if O(1) processing in compilers would lead to simpler code the same way it does for databases?

0 views
matklad 6 months ago

Newtype Index Pattern In Zig

In efficiency-minded code, it is idiomatic to use indexes rather than pointers. Indexes have several advantages: First , they save memory. Typically a 32-bit index is enough, a saving of four bytes per pointer on 64-bit architectures. I haven’t seen this measured, but my gut feeling is that this is much more impactful than it might initially seem. On modern architectures, saving memory saves time (and energy) as well, because the computing bottleneck is often the bit pipe between the memory and the CPU, not the computation per se. Dense data structures use CPU cache more efficiently, removing prohibitive latency of memory accesses. Bandwidth savings are even better: smaller item size obviously improves bandwidth utilization, but having more items in cache obviates the need to use the bandwidth in the first place. Best case, the working set fits into the CPU cache! Note well that memory savings are evenly spread out. Using indexes makes every data structure slightly more compact, which improves performance across the board, regardless of hotspot distribution. It’s hard to notice a potential for such saving in a profiler, and even harder to test out. For these two reasons, I would default to indexes for code where speed matters, even when I don’t have the code written yet to profile it! There’s also a more subtle way in which indexes save memory. Using indexes means storing multiple items in an array, but such dense storage contains extra information in relative positions of the items. If you need to store a list of items, you can often avoid materializing the list of indexes by storing a range “pointing” into the shared storage. Occasionally, you can even do UTF-8 trick and use just a single bit to mark the end of a list. The second benefit of indexes is more natural modeling of cyclic and recursive data structures. Creating a cycle fundamentally requires mutability somewhere (“tying the knot” in Haskell relies on mutability of lazy thunks). This means that you need to make some pointers nullable, and that usually gets awkward even without borrow checker behind your back. Even without cycles and just recursion, pointers are problematic, due to a combination of two effects: The combination works fine at small scale, but then it fails with stack overflow in production every single time, requiring awkward work-arounds. For example, serializes error traces from nested macro expansions as a deeply nested tree of JSON objects, which requires using stacker hack when parsing the output (which you’ll learn about only after crashes in the hands of macro connoisseur users). Finally , indexes greatly help serialization, they make it trivial to communicate data structures both through space (sending a network message) and time (saving to disk and reading later). Indexes are naturally relocatable, it doesn’t matter where in memory they are. But this is just a half of serialization benefit. The other is that, because everything is in few arrays, you can do bulk serialization. You don’t need to write the items one by one, you can directly arrays around (but be careful to not leak data via padding, and be sure to checksum the result). The big problem with “naive” indexes is of course using the right index with the wrong array, or vice verse. The standard solution here is to introduce a newtype wrapper around the raw index. @andrewrk recently popularized a nice “happy accident of language design” pattern for this in Zig. The core idea is to define an index via non-exhaustive : In Zig, designates a strongly-typed collection of integer constants, not a Rust-style ADT (there’s for that). By default an backing integer type is chosen by the compiler, but you can manually override it with syntax: Finally, Zig allows making enums non-exhaustive with . In a non-exhaustive enum, any numeric value is valid, and some have symbolic labels: and builtins switch abstraction level between a raw integer and an enum value. So, is a way to spell “ , but a distinct type”. Note that there’s no strong encapsulation boundary here, anyone can . Zig just doesn’t provide language-enforced encapsulation mechanisms. Putting everything together, this is how I would model n-ary tree with parent pointers in Zig: Some points of note: P.S. Apparently I also wrote a Rust version of this post a while back? https://matklad.github.io/2018/06/04/newtype-index-pattern.html pointers encourage recursive functions, and recursive data structures lead to arbitrary long (but finite) chains of pointers. As usual with indexes, you start with defining the collective noun first, a rather than a . In my experience, you usually don’t want suffix in your index types, so is just , not the underlying data. Nested types are good! feels just right. For readability, the order is fields, then nested types, then functions. In , we have a couple of symbolic constants. is for the root node that is stored first, for whenever we want to apply offensive programing and make bad indexes blow up. Here, we use for “null” parent. An alternative would be to use , but that would waste of space, or making the root its own parent. If you care about performance, its a good idea to sizes of structures, not to prevent changes, but as a comment that explains to the reader just how the large the struct is. I don’t know if I like or more for representing ranges, but I use the former just because the names align in length. Both and are reasonable shapes for the API. I don’t know which one I prefer more. I default to the former because it works even if there are several node arguments.

0 views
matklad 7 months ago

TigerBeetle Blog

Continuing the tradition , I’ve been also blogging somewhat regularly on TigerBeetle’s blog, so you might want to check those articles out or even subscribe (my favorite RSS reader is RSSSSR ): Today’s post is a video version of Notes on Paxos ! https://tigerbeetle.com/blog/ https://tigerbeetle.com/blog/atom.xml

0 views
matklad 8 months ago

Readonly Characters Are a Big Deal

I like Emacs UX as exemplified by Magit . I consider it to be a user interface paradigm on the same footing as UNIX pipes: An Engine for an Editor Pipes give you 1D read-only streams of characters which excel at batch processing. Emacs is all about interactive mutable 2D buffers of attributed text. Today I realized that an important feature of Emacs text buffers is read-only characters ( manual ). Like in any editor, you can mark an entire Emacs buffer as read-only. But you also can mark individual substrings read-only, so that you can edit anywhere except specific ranges This is a useful feature for bidirectional interaction. Consider the in-editor terminal I am using currently, which looks like this: The first line is the command to execute, this is typed by me manually, and then I hit a “submit” shortcut to actually run the command. Then goes the status line, which shows how long the command has been running so far and the exit code (when the command terminates). The status line is determined by the “terminal” itself. Finally, there’s output of the command itself, updated live. In this sort of the interface, command is modifiable by the user, but is read-only for the editor. Status is the opposite — the editor updates it every second, but the user should be prevented from touching it. And the output can be CRDT-style edited by both parties (I often find it useful to edit the output in place before pasting it elsewhere). Sadly, in VS Code I can’t prevent the user from editing the status, so my implementation is a bit janky, and this, I think, goes to the core of why I don’t see VS Code as a great platform for the kind of interactive tools I want to write. Read-only ranges are hard to implement! Text editing hates you as is, but this feature requires tracking text attributes in an intelligent way under modifications (see sticky properties ), and also feeds back into modifications themselves! No wonder Monaco, the editor engine underlying VS Code, lost this ability at some point. Still, I feel like “does it support sticky read-only attribute?” is a good litmus test to check if an editor can support interactive applications a-la Magit seamlessly.

0 views
matklad 8 months ago

On Async Mutexes

A short note on contradiction or confusion in my language design beliefs I noticed today. One of the touted benefits of concurrent programming multiplexed over a single thread is that mutexes become unnecessary. With only one function executing at any given moment in time data races are impossible. The standard counter to this argument is that mutual exclusion is a property of the logic itself, not of the runtime. If a certain snippet of code must be executed atomically with respect to everything else that is concurrent, then it must be annotated as such in the source code. You can still introduce logical races by accidentally adding an in the middle of the code that should be atomic. And, while programming, you are adding new s all the time! This argument makes sense to me, as well its as logical conclusion. Given that you want to annotate atomic segments of code anyway, it makes sense to go all the way to Kotlin-style explicit async implicit await. The contradiction I realized today is that for the past few years I’ve been working on a system built around implicit exclusion provided by a single thread — TigerBeetle! Consider compaction, a code that is responsible for rewriting data on disk to make it smaller without changing its logical contents. During compaction, TigerBeetle schedules a lot of concurrent disk reads, disk writes, and CPU-side merges. Here’s an average callback: This is the code ( source ) that runs when a disk read finishes, and it mutates — shared state across all outstanding IO. It’s imperative that no other IO completion mutates compaction concurrently, especially inside that monster of a function . Applying “make exclusion explicit” rule to the code would mean that the entire needs to be wrapped in a mutex, and every callback needs to start with lock/unlock pair. And there’s much more to TigerBeetle than just compaction! While some pairs of callbacks probably can execute concurrently relatively to each other, this changes over time. For example, once we start overlapping compaction and execution, those will be using our GridCache (buffer manager) at the same time. So explicit locking probably gravitates towards having just a single global lock around the entire state, which is acquired for the duration of any callback. At which point, it makes sense to push lock acquisition up to the event loop, and we are back to the implicit locking API! This seems to be another case of two paradigms for structuring concurrent programs . The async/await discussion usually presupposes CSP programming style, where you define a set of concurrent threads of execution, and the threads are mostly independent, sharing a little of data. TigerBeetle is written in a state machine/actor style, where the focal point is the large amount of shared state, which is evolving in discrete steps in reaction to IO events (there’s only one “actor” in TigerBeetle). Additionally, TigerBeetle uses manual callbacks instead of async/await syntax, so inserting an in the middle of critical section doesn’t really happen. Any new concurrency requires introducing an explicit named continuation function, and each continuation (callback) generally starts with a bunch of assertions to pin down the current state and make sure that the ground hasn’t shifted too far since the IO was originally scheduled. Or, as is the case with , sometimes the callback doesn’t assume anything at all about the state of the world and instead carries out an exhaustive case analysis from scratch.

0 views