Posts in C (20 found)
Anton Zhiyanov 6 days ago

Go-flavored concurrency in C

Go's concurrency is one of the main reasons people like the language. You write , send values through channels, and the runtime scheduler runs thousands of goroutines on just a few OS threads. It feels effortless. None of that machinery exists in C. Which made me wonder: how close can you get to Go's concurrency model using only POSIX threads? Obviously, native OS threads can't match the efficiency of lightweight goroutines, but what is the actual cost, when does it become a problem, and is there any way to at least partially avoid it? I ran into these questions while adding concurrency to Solod (So), a strict subset of Go that translates to plain C, with no runtime and no garbage collector. In the end, I came to the conclusion that you can do quite a lot with pthreads — as long as you're honest about the tradeoffs. This post is about the POSIX threads-based concurrency model I chose, the benefits it offers, and its limitations. Mutex/Cond • Atomics • Pool • Channel • Performance • Design • Wrapping up Everything in So's concurrency stack is built on two basic POSIX primitives: the mutex and the condition variable. is a thin wrapper around : Since So translates to C, this is basically a struct that holds a and a function that calls . Here's the transpiler output: That is the whole translation — the generated C is a near-mechanical mirror of the So code, only noisier. From here on, I'll mainly show the So version, but I'll also provide the C code for those who are interested. There's nothing exciting here: is a pthread mutex wrapper that panics if something goes wrong (which is rare). The companion primitive is , a wrapper around . It's the standard "wait until a condition holds" tool, associated with a mutex: These two types — and — are the foundation. Other concurrency tools — , the thread pool, channels — are built using a mutex and one or more condition variables. This has several effects on performance, as we'll see later. Not everything needs a lock. So's mirrors Go's: , , , , , and a generic , all with , , , and methods. The nice thing is that these don't need pthreads at all. They map directly to the C compiler's builtins — the same hardware instructions that Go's compiler emits. So there's no reason for them to be any slower, and they're not: Each number is the cost of one operation on a single thread. is a good example of using atomics effectively. Its fast path only needs a single atomic load — after the given function runs, every future call to checks a flag and returns: To actually run code concurrently, you need threads. The type wraps and its related functions: Consider this function: Usage example: It might look like , but that's just on the surface. starts an actual OS thread, not a goroutine. You have to eventually call to join or it, or else its resources will leak. Also, OS threads are expensive to create — they're nothing like Go's goroutines, which only need a few kilobytes of stack and start up in nanoseconds. That's exactly why you usually don't want to call inside a loop. For tasks that are short-lived or happen often, it's better to use a pool of long-lived worker threads and send tasks to them. to the rescue: Usage example: The first argument to , , is a memory allocator. Solod avoids hidden allocations, so anything that needs memory takes an allocator explicitly — here it backs the pool's task queue. Under the hood, a is a fixed group of worker threads that pull tasks from a shared queue (a ring buffer). It uses one mutex and a few condition variables: wakes up a worker when there are tasks to do, applies back-pressure when the queue is full, and lets know when everything is finished. It's a classic producer-consumer setup, about 200 lines of code , and there's nothing fancy about it. The heart of the pool is the worker loop. Each thread blocks until a task appears, runs it outside the lock so workers execute in parallel, then records that it finished: This is what separates a pool from a plain queue. bumps as it enqueues; each worker decrements it after running a task, and the last one out broadcasts . sleeps until the count hits zero: The tradeoff is that the number of worker threads is fixed. In Go, a program can handle thousands of concurrent I/O waits because blocked goroutines use very little memory. A So pool can't do this — if all N workers are parked on a blocking syscall, the pool is stalled until one returns. You have to set the pool size based on the workload, instead of letting the runtime manage it for you. Channels are an important part of Go's concurrency model, and So's gives you something quite similar. Just like in Go, it passes values by copy and comes in buffered and unbuffered flavors: is a thin generic shell over one of two engines, picked at creation time: Buffered ( ) is a mutex-guarded ring buffer with and condition variables — like the queue. Senders block when it's full, receivers block when it's empty. The full implementation also checks for , but I left it out for brevity. is the mirror method: block while empty, pop the next value, signal to wake a sender. It also handles the closed channel, returning once the buffer is closed and drained. The rest is this lock-wait-signal core. Buffer source code Unbuffered ( ) is a rendezvous: each send blocks until a receiver takes the value, copying bytes directly from the sender's stack to the receiver's destination without using an intermediate buffer. is the other half: it waits for a published, unclaimed value, copies bytes straight from the sender's stack into (no intermediate buffer), marks it as claimed, and broadcasts to wake the sender back, creating wakeup #2. One hand-off, two wakeups. Copying directly from the sender's stack is safe because of that second wakeup. is a pointer to , which lives on the sender's stack. While the receiver is reading it, the sender is parked in , so its stack frame stays alive. The sender only returns (and reclaims that memory) after the receiver sets and wakes it up. There's no need to copy into a shared buffer because the source is guaranteed to outlive the read. Rendezvous source code As you can see, the API is pretty similar to Go. Now let's look at the numbers. Here's the main tradeoff: pthread-based concurrency primitives are fast when no one has to block, but they get slow when someone does. And it's always for the same reason. Go schedules goroutines in userspace. When one goroutine blocks on a channel and another wakes it up, the runtime moves them between its own queues — no kernel involved. POSIX threads, on the other hand, don't provide a userland scheduler. When a thread blocks on a condition variable, it parks in the kernel, and waking it up requires a syscall. Every hand-off between threads that actually parks pays the cost of a syscall on both ends. You can clearly see the difference in the mutex benchmarks. With 8 competing threads, it all comes down to whether the waiting threads have to park or not: Each number is the average time for a single / pair. The uncontended benchmark runs on one thread, while the contended benchmarks have multiple threads fighting over the same mutex. Notice that So actually wins the first two benchmarks, and for good reason. So's is a plain call with nothing extra, while Go's adds more overhead — like starvation-mode tracking and a runtime that stays involved because a goroutine can be preempted in the middle of a critical section. When nobody parks, that overhead is the main cost, and the thinner wrapper is closer to the hardware. With an empty critical section (the spin benchmark), a waiting thread grabs the lock while still spinning and almost never parks — So wins by 2.8x. The uncontended benchmark (a single thread, no contention) shows the same thing: less code between the call and the lock, so 9ns versus 14ns. The picture flips the moment threads have to park. Give the critical section about a microsecond of real work (the work benchmark) and waiters exhaust their spin budget and park. Now every hand-off costs a wakeup syscall, and So drops to half of Go's throughput. The work is identical in both cases — the difference comes from the parking cost. Condition variables demonstrate this clearly because they always park: Each number is the cost of one rendezvous round: a single broadcast that wakes every waiter and hands control back, with N waiters plus one broadcaster. Pthread-based condition variable is consistently 7-10 times slower. There's no trick to close this gap — it's just the cost of waking up a real OS thread instead of a goroutine. Channels have the same issue because they're built using mutexes and condition variables: Each number is the cost of moving one value through the channel (send plus its matching receive). The number in parentheses is the buffer capacity. The uncontended case fills and drains a buffer from a single thread, so nothing ever blocks — it's just a lock plus a copy, which gives So a slight advantage. But the moment a producer and consumer actually start handing off work, So has to wake up a thread for every transfer that gets parked. It's worst for the unbuffered channel, where every value is a rendezvous with two wakeups: 23x slower. A larger buffer helps a lot — with room for 100 items, most sends go through without waking anyone, and the gap narrows to about 2x. The consequence is that the larger your tasks are, the better pthread-based concurrency works. If you use a channel for fine-grained, value-at-a-time streaming between threads, performance will suffer. But if you use a channel to pass whole work items to a pool, where each item takes tens of microseconds to process, the wakeup cost becomes negligible. The pool benchmarks on realistic workloads confirms this: Each number is the wall-clock time for 8 workers to process the whole batch. Here, So is within 1.1x of Go. The per-task dispatch cost is still present, but it's spread out over real work, and the performance penalty is pretty small. Benchmarking All benchmarks were run on an Apple M1 CPU running macOS. The C code was compiled with Clang 16 using these CFLAGS and mimalloc as the system allocator: The results shown are the medians from several benchmark runs. Each benchmark ran many iterations, following the same logic as Go's own benchmarking. The Go benchmarks used Go 1.26 and . Source code for both So's and Go's benchmarks: conc • sync Here's a summary of the strengths and weaknesses of the pthread-based approach: If you're looking for "thousands of cheap goroutines", the pthread-based approach will let you down. But if you're fine with "a few worker threads handling lots of tasks", it holds up well. Three decisions influenced the way I implemented concurrency in Solod. Pthreads, not fibers . I know there are coroutine/fiber libraries for C that avoid the kernel wakeup cost — single-threaded ones like neco , and multi-threaded ones like libfiber . A userspace scheduler is exactly what would help to match Go in the benchmarks above. I decided not to use one. I wanted something dead simple — an approach I could explain in a paragraph, using tools every C programmer already knows. The trade-off is that you lose some performance with fine-grained blocking, but in many real-world situations, pthreads work fine if you use a worker pool. For me, keeping things simple is more important than saving a few microseconds during task hand-offs. For now, at least. Standard library, not language . Go bakes goroutines, channels, and select right into the language. I decided to keep everything in the stdlib for two reasons. ➀ It follows So's "no hidden allocations" rule. In Go, quietly allocates a goroutine stack, and allocates a buffer. In So, all allocations are explicit: you pass an allocator to and , and you always know exactly where the memory comes from — whether it's the system allocator, an arena, or something else. ➁ A library is more flexible. Since a pool is a regular value, you can have as many as you need, each sized for its specific purpose. In a multi-stage pipeline where each stage needs a different capacity, you can start one pool per stage, each with its own and , instead of being given a single global scheduler. The language stays simple, and the flexibility is in code you can easily read. Timeouts, not select . Go's waits on several channel operations at once and proceeds with whichever is ready first. Implementing it would require a lot of work — a thread has to register interest on multiple channels, block once, and then wake up when any of them is ready — so I left it out. Instead, offers and , which cover two common uses of with a single channel: What's missing is the ability to block on multiple channels at once and continue with whichever one is ready first, as well as the option to mix sends and receives in the same selection. How close can you get to Go's concurrency using only pthreads? Close enough to be useful, but not enough to really match Go. You can wrap real OS threads with familiar APIs — mutexes, condition variables, pools, channels — and the code will look and act a lot like Go, at least until a thread needs to block. But there's no scheduler underneath, so when a thread blocks, it's an actual thread waiting in the kernel, not a goroutine that's paused for free. That's the main limitation of this approach. What you get in return is brutal simplicity. Every primitive is a thin wrapper with no runtime hiding behind it, so the performance is exactly what the OS gives you: fast atomics, fast uncontended locks, and pooled throughput within ~10% of Go on coarse-grained work. But as soon as you switch to fine-grained, one-value-at-a-time hand-offs, the cost of kernel wakeups becomes the main factor, and you'll notice the slowdown. If you think the pthread approach might work for you, I invite you to try Solod . It includes the and packages, along with many others ported from Go's standard library. ➕ Coarse-grained pooled workloads are within about 10% of Go's performance. ➕ Uncontended locks and spin-friendly critical sections perform quite well. ➕ Atomic operations are as fast as in Go. ➕ The implementation is 100x simpler. ➖ Anything that needs to park and wake an OS thread is much slower than Go's userspace scheduler. ➖ The pool can't handle thousands of blocked waiters like goroutines can. "Do this, but give up after a while" (Go's idiom). "Do this only if it won't block" (Go's non-blocking branch).

programming Programming c C shell Shell performance Performance go Go
0 views
Anton Zhiyanov 2 weeks ago

Solod v0.2: Networking, new targets, friendlier interop

Solod ( So ) is a system-level language with Go syntax, zero runtime, and a familiar standard library. It's designed for two main audiences: The previous version (v0.1) focused on porting core Go stdlib packages and providing convenient C interop. At the end of that post, I said the next release would focus on networking, concurrency, or both. Now, networking is here — the v0.2 release I'm sharing today includes support for TCP, UDP, and Unix domain sockets. Concurrency is still planned for the future, so for now, servers handle one connection at a time. This release also lets you compile So to more targets, like 32-bit platforms, WebAssembly, and bare metal. And C interop even smoother! Networking • TCP server • TCP client • Deadlines • IP addresses • Targets • Interop • Stdlib • Wrapping up The main feature in v0.2 is the package. It's a simplified version of Go's package which supports the three most commonly used transports: The API mirrors Go closely, so most of it will feel familiar. The big difference is that So has no goroutines, so there's no concurrent server support — you accept and serve connections sequentially. More on that in a moment. Let's build a classic: an echo server that accepts a connection, reads a message, and sends it back. If you've written a TCP server in Go, this should look familiar — , an loop, and / on the connection. The only thing missing is a : without goroutines, each connection is handled to completion before moving on to the next . The client starts the connection using , then uses to send a request and to get the reply: UDP and Unix domain sockets work in a similar way. For UDP, an unconnected socket uses to get data and the sender's address, and to send a reply. For Unix sockets, there are (stream) and (datagram). By default, , , and are blocking. In Go, you'd typically use goroutines and contexts to prevent getting stuck forever. Since that's not available in So (yet), every connection and listener supports deadlines instead: , , and are available on , , , and listener types. When the deadline passes, any pending call fails with . If you don't set a deadline, a blocked call will wait forever. This isn't concurrency, but it's enough to keep a single-threaded server responsive. Along with , v0.2 ports Go's package, which provides small, allocation-free value types for IP addresses. represents an IP address, combines an IP address with a port, and is an IP with a prefix length (a CIDR block): These are simple value types that don't use any heap allocation, which fits well with So's explicit-memory approach. The package also provides and functions to help you work with strings. Solod compiles to plain C, which (in theory) means it can target anything a C compiler can. Because of this, v0.2 adds new targets: Here's the complete toolchain you need to build a freestanding binary using : A large part of the standard library ( , , , , , , , and more) works just fine in freestanding mode. For more details, check out the freestanding guide . A bunch of smaller changes make Solod nicer to write. Three new directives for low-level work, all documented in the interop guide : works with variables, constants, types, and functions. You can use it on multiple lines, and the attributes will stack. For example, will combine with . Type aliases . So now supports Go-style type aliases: Numeric C types . The package now includes named types for C's numeric types — , , , , , , and others. When you declare an extern function, you can use the actual C types in its signature instead of trying to guess the correct fixed-width Go type for your platform. Third-party packages . You can now add external So packages using or by vendoring, and you can organize your own code into multiple modules. So doesn't have a real package ecosystem yet, but it's a good start. Better diagnostics . By default, panic messages report the C file and line. Pass to report the original So source location instead: There's also an optional flag that adds nil-pointer checks when accessing struct fields and calling interface methods. This way, if there's a bad dereference, the program will panic cleanly instead of causing a segmentation fault. Both options are off by default to keep the generated code more readable. Beyond and , v0.2 adds a few more packages: And a small but handy update to memory management: now reclaims the last allocation if you give it the matching pointer. It's a minor optimization, but it means a quick alloc/free pair on an arena no longer wastes space. Stdlib documentation With v0.2, Solod has evolved from just "command-line tools and C glue" into something you can actually use on a network — like a TCP or UDP server, a small protocol client, or a Unix-socket daemon. The new targets (32-bit, WASM, freestanding) mean the same code can now run in more places, even down to bare metal. The big thing that's still missing is concurrency. A server that handles requests one at a time works for some tasks, but a real network service needs to manage many connections at once. That's the obvious goal for v0.3 — adding some kind of concurrency, along with the stdlib packages that support it. If you're interested, take a look at So's readme — it has everything you need to get started. Or try So online without installing anything. Go developers who want low-level control and zero-cost C interop without having to learn Zig or Odin. C developers who like Go's style. TCP (networks , , ) via , , and , with the and types. UDP (networks , , ) via , (a connected socket), and (an unconnected socket with / ). Unix domain sockets ( for streams, for datagrams) via , , , and . 32-bit platforms . The compiler and stdlib now work correctly on 32-bit platforms, where and pointers are narrower. WebAssembly (WASI) . You can compile a So program to and run it under any WASI runtime. Freestanding mode . So programs can run on bare-metal systems without any C standard library. No libc means no malloc, but you can use instead. — hex encoding and decoding, including for hexdump-style output. — generating and parsing UUIDs (v4 and v7), with random components from a cryptographically secure source.

open-source Open Source programming Programming c C go Go Odin
0 views
Unsung 3 weeks ago

I was wrong about Duff’s device

Duff’s device is a C language technique that looks like this: It achieves two things: I always assumed the technique is from the 1970s and was just a show-offy thing that didn’t serve any function, a “look how clever I am” from a programmer who was perhaps just a touch too nerdy. But yesterday, I found a 1988 message from its inventor , Tom Duff, and it turns out I got almost everything wrong. First of all, the technique was from 1983, when Duff was at Lucasfilm – much later than I expected. Second of all, it actually solved a problem. Duff’s device wasn’t just making things faster abstractly, but actually fixed a user-visible performance issue. “[The loop before applying the device] was the bottleneck in a real-time animation playback program which ran too slowly by about 50%,” writes Duff. Most importantly, however, Duff himself had mixed feelings about it: Disgusting, no? But it compiles and runs just fine. I feel a combination of pride and revulsion at this discovery. I recognize this set of feelings from many different software hacks I invented in my life. I think it’s important to carry them all with you – not fall in love with the hack and continue seeing it for what it is (and what it will be in the future as code ages), but at the same time not be above using it if it’s solving a real issue. Also, Duff adds: Many people […] have said that the worst feature of C is that switches don’t break automatically before each case label. This code forms some sort of argument in that debate, but I’m not sure whether it’s for or against. I can’t speak for C, but I have always felt frustrated about JavaScript stealing that convention – it’s so error-prone, and in my many years programming in it, I have never had to use a Duff’s device or anything else that benefitted from it. #coding #hacks It unrolls the loop in chunks of eight. Unrolling the loop is when instead of telling the computer “do X 5 times,” you say “do X do X do X do X do X,” trading some code readability and memory usage for higher speed. It cleverly (ab)uses a property of the C language to unroll the remainder of the loop, which normally would be impossible to do as the remainder is less than 8 and different every time. It does so by basically overlapping a / loop atop a / structure in a way that should come with a coding equivalent of a parental warning.

javascript JavaScript programming Programming c C
0 views
Ankur Sethi 3 weeks ago

Deno Desktop

From the Desktop apps section of the Deno documentation : turns a Deno project (anything from a single TypeScript file to a Next.js app) into a self-contained desktop application. The output is a redistributable binary that bundles your code, the Deno runtime, and a web rendering engine into one bundle per platform. I'm happy to see another attempt at solving the biggest issues with Electron apps (other notable attempts being Tauri , Electrobun , and Neutralinojs ). According to the docs, Deno Desktop is only available in Deno's channel at the moment. So I obviously installed it (version ) and tried running a Hello World example app . On first run, Deno spent a few minutes downloading , then packaged the example into an app bundle weighing 308.8MB. I was curious about that download. A quick Kagi search led me to the homepage for a Rust/C library called laufey , which appears to be the tech underpinning Deno Desktop. Running the app bundle popped open a window that looked like this: This is clearly a work in progress. If somebody who works on Deno is reading this, here's a list of bugs I noticed: Deno uses Chromium as the default webview (via Chromium Embedded Framework ). But you can also use the system webview instead: When I ran that command, it downloaded and produced a much slimmer app bundle at 68.5MB. This is what the window looked like: This version of the app exhibited none of the bugs I noticed in the CEF version, except it doesn't have a title. Deno Desktop also has a backend that skips bundling the webview altogether. I didn't try it, but here's what the docs say: No web engine.  Provides window management, input events, clipboard, and the native API surface, but no webview, no   auto-binding, and no   proxy. Useful for apps that draw their own UI (WebGPU, Skia, custom rendering) or as a foundation for non-web desktop programs. The   backend is selected through the   field in  ; the   flag accepts only   and  . A major difference between Deno Desktop and its competition is how it communicates between the code running in the webview and the code running in the Deno runtime: Bindings are not IPC. The Deno runtime and the rendering backend run as threads / processes inside the same address space (CEF) or coordinated process group (WebView). Calls go through in-process channels, and the backend dispatches them from its run loop. This avoids the cross-process round-trip that socket-based IPC frameworks (Electron's ipcMain / ipcRenderer, Tauri's invoke) impose. Arguments and results are still encoded as they cross the realm boundary, but the transport is in-process: no socket, no cross-process scheduling. In practical terms: bindings are fast enough that you do not need to worry about call frequency for typical app workloads. The docs are light on how they pull this off. I'd love to read more about this. There's a built-in auto-update mechanism, including rollbacks if updates fail: Deno.autoUpdate() polls a release server for new versions, downloads binary-diff patches, applies them to the runtime dylib, and stages the result for the next launch. If the next launch fails, the runtime rolls back to the previous version automatically. Updates ship as small bsdiff patches instead of full binary downloads, with rollback baked into the launcher. The comparison page has this bullet-point under the section titled "What doesn't have yet": Shared CEF runtime across apps.  Every app currently bundles its own CEF copy. A managed shared runtime would drop binary sizes to a few MB per app. On the roadmap. Does this mean all Deno apps on my computer could potentially share a single CEF runtime? If yes, that would mean massive disk space savings. But it's unclear if the developers intend to ship this feature in a future release or if it's just a wishlist item that may or may not see the light of day. Deno Desktop is, of course, heavily under development. Some important features are still missing (platform native file dialogs), and it's not clear if others are on the roadmap or not (mobile support). I'm sure many of the missing features will make their way into the final release, and we'll get a clearer idea of future plans in a release announcement. I have a personal interest in anything that aims to replace Electron, so I'll be keeping an eye out for Deno 2.9. The app window had a dark background by default, even though the demo app didn't contain any styles. Browsers don't default to a dark background unless you explicitly opt in using . Even so, opting into dark mode inverts all the default colors, not just the page background. Something is off here. Running the bundle triggered a macOS permissions dialog for and , both of them asking for notification permissions. The demo app didn't use the notifications API (it didn't even contain any JavaScript), so seeing two permission dialogs felt aggressive. Hitting didn't quit the app. The app always opened on the top left of the screen.

Deno c C web-development Web Development rust Rust javascript JavaScript
0 views
Xe Iaso 3 weeks ago

"No way to prevent this" say users of only language where this regularly happens

In the hours following the release of CVE-2026-55200 for the project libssh2 , site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix an out-of-bounds write in ssh2_transport_read() due to a missing upper bound check on the packet_length field, resulting in heap corruption and potential remote code execution. This is due to the affected components being written in C, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Mr. Alex Doyle, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

c C programming Programming security Security
0 views
Simon Willison 1 months ago

Publishing WASM wheels to PyPI for use with Pyodide

The Pyodide 314.0 release announcement (via Hacker News ) includes news I've been looking forward to for a long time: You can now publish Python packages built for Pyodide (or any Python runtime compatible with the PyEmscripten platform defined in PEP 783 ) directly to PyPI and install them at runtime. Previously, the Pyodide maintainers had to maintain, build, and host over 300 packages ourselves. This created a significant burden on our maintainers and became a major bottleneck for the community, as every new package required manual review. Moving forward, package maintainers can simply build and publish Pyodide wheels to PyPI, just as they do for native wheels on Linux, macOS, or Windows. Here's the PR to PyPI itself supporting this , which landed on April 21st. I adore Pyodide , and have been frustrated in the past by this limitation. It's possible to compile C or Rust extensions to WASM in a wheel file, but before now there was no easy way to distribute them. Thanks to the efforts of a whole lot of people, that's now been fixed! I decided to celebrate by finding something I could package. I have quite a few experimental Pyodide projects lying around, but the best fit for this looked to be my Luau WebAssembly research spike from 9th March. Luau is a "small, fast, and embeddable programming language based on Lua with a gradual type system", developed by Roblox and released under an MIT license. It's written in C++. I already knew it was possible to compile it to WebAssembly and get it running inside of Pyodide, so I set Codex + GPT-5.5 xhigh the task of packaging my experiment up and publishing it to PyPI using GitHub Actions. It took some iteration, but here's the result: luau-wasm is a brand new PyPI package which publishes a 276KB file which can be used in Pyodide like this: You can run that code in the Pyodide REPL demo to see it in action. The GitHub repo for luau-wasm includes all of the build and deploy scripts (using the latest cibuildwheel ) and also deploys an HTML demo page which loads Pyodide, installs and provides an interface for trying it out: https://simonw.github.io/luau-wasm/ I was curious to see how many packages are currently publishing wheels for this platform. After some tinkering with ChatGPT I got to this BigQuery SQL which I ran against PyPI's public dataset on BigQuery . Here's the raw JSON of query results and here's a SQLite SQL query in Datasette Lite which dedupes packages by most recent upload date. If the query is right, there are currently 28 PyPI packages publishing with the new tags: luau-wasm , uuid7-rs , cmm-16bit , pyOpenTTDAdmin , imgui-bundle , numbertoolkit , bashkit , geoarrow-rust-core , arro3-io , arro3-core , arro3-compute , onnx , powerfit-em , tcod , chonkie-core , tokie , robotraconteur , pydantic_core , yaml-rs , cadquery-ocp-novtk-OCP.wasm , uuid_utils , base64_utils , pycdfpp , lib3mf-OCP.wasm , typst , toml-rs , onnx-weekly , dummy-pyodide-ext-test Here's hoping we see a whole lot more of those showing up over the coming months and years. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options .

python Python html HTML open-source Open Source c C c++ C++
0 views
Blargh 1 months ago

AI solving problems

I’ve been able to find some time, lately, to work on my project backlog. And because it’s 2026, I’ve been using AI as a diligent intern. I’ve ranted before about seccomp , but still used it for a project or two. But then, rarely, it triggered an unexpected . That’s exactly the kind of I do want to detect and kill the binary for, so I don’t just want to allow it. I want to know where it’s coming from. showed it’s trying to read . It’s certainly not my code. But just the Rust transitive dependency tree is quite a few crates: Step 1 was to run it in , and reproduce the problem. But it’s a bit trickier than that, because seccomp fully kills the process, so no backtrace. And setting breakpoints requires a few more syscalls to work, just for the process to work under gdb (e.g. ). And turns out some calls fail with if running under a debugger. Yes, I can fix all these things. But why not put the AI intern on it? I’ve previously vibe coded a raytracer just with the prompt “make a raytracer for the files in this directory”. The files in question were my POV-Ray files converted from old Quake demos. It worked very well that time ( flat shaded and textured ), so why not? (youtube’s compression is Very Aggressive™. The originals look crisp) Skipping to the end, yeah: This is now an error instead of killing the binary. Sure. But it reminds me of the well publicised Google interview question “how many piano tuners are there in New York city?”. The best answer I’ve heard is “is it important for the company? — Yes — Then I’d look it up”. That’s the right answer, of course. There’s another “more right” answer to the “why are manhole covers round?” question . Is it important to know which of my transitive dependencies try to read this file? Not really. I could blanket allow that file to be read. I know what that file is, and I actively don’t care if an attacker can read it. So without AI I would not have spent more than 30 seconds trying to find out. And I spent a couple of seconds doing , only showing references in jemalloc, which this project doesn’t transitively depend on. So it’s likely somewhere in some C code. Or in the Rust standard library. So instead I spent 9 minutes reading up on the QUIC protocol for a couple of my other projects, while the AI went brrr. Sometimes the code tries to read readonly and crashes because seccomp. Why does this happen? Sounds good. Change it to do that. diff –git a/src/privs.rs b/src/privs.rs index 19b6307..cce5ef6 100644 — a/src/privs.rs +++ b/src/privs.rs @@ -1,6 +1,6 @@ use anyhow::{Context, Result, anyhow}; use caps::CapSet; -use libseccomp::{ScmpAction, ScmpFilterContext, ScmpSyscall}; +use libseccomp::{ScmpAction, ScmpArgCompare, ScmpCompareOp, ScmpFilterContext, ScmpSys call}; use tracing::{info, trace, warn}; /// Drop privileges to bare minimum. @@ -127,18 +127,32 @@ fn drop_caps() -> Result<()> { ] { f.add_rule(ScmpAction::Allow, ScmpSyscall::from_name(name)?)?; } if with_rustls { // Rustls does some memory allocation. So, now it works. But I’d want to adjust it a bit, to not merely return for any probing of files such as or TLS certificates. I want that to terminate the binary. For merely denying filesystem access on a per path prefix basis I use landlock, but that doesn’t kill the binary. I’m not aware of a way for seccomp to block specific paths (again, seccomp sucks), but maybe the AI knows? But oh no: So maybe it’s time to go outside in the nice weather? “Touch grass”, as the kids say. // glibc malloc may lazily probe /proc/sys/vm/overcommit_memory. f.add_rule_conditional( ScmpAction::Errno(libc::EACCES), ScmpSyscall::from_name(“openat”)?, &[ScmpArgCompare::new( ScmpCompareOp::MaskedEqual(libc::O_ACCMODE as u64), libc::O_RDONLY as u64, )?; if with_rustls { // Rustls does some memory allocation.

ai AI rust Rust security Security c C
0 views
Dangling Pointers 1 months ago

Accelerating Transactional Execution via Processing-In-Memory

Accelerating Transactional Execution via Processing-In-Memory André Lopes, Daniel Castro, and Paolo Romano EUROSYS'26 This paper describes a way to implement OLTP for a processing-in-memory architecture. As with other academic research, it uses UPMEM ( here are two summaries of papers that rely on UPMEM). Something I found surprising in this work is conflicts can cause transactions to abort, even if all transactions only access data in the same UPMEM bank. A UPMEM DIMM is like a DRAM DIMM, but each bank contains a multi-threaded in-order core which can access data from the bank it is co-located with. This paper calls these processors DPUs (some other papers call them IDPs). The only way for two DPUs to communicate with each other is for the host CPU to read data from one bank and write it into another. The system described in this paper is called PIM-TIDE. It assumes that transactions come pre-sliced into computational graphs comprising subtransactions. A single subtransaction only accesses data within one bank (and thus executes on a specific DPU). Users of PIM-TIDE do not need to know exactly which data words a subtransaction will access, but they do need to be able to restrict a subtransaction to only access data from a specific bank. This works well on TPC-C style transactions where most of the database can be partitioned on . The host CPU groups transactions into batches and sends work to the DPUs one batch at a time. Within a batch, transactions are categorized into two groups: Local transactions execute entirely within a DPU Distributed transactions execute across multiple DPUs All subtransactions associated with distributed transactions within a batch are assigned a unique sequence number, which determines the order in which the subtransactions will commit. Local transactions are not preassigned to a commit order. DPUs first process all distributed subtransactions in a batch and then execute all local subtransactions. Algorithm 3 illustrates PIM-TIDE’s concurrency control scheme for dealing with intra-DPU conflicts between subtransactions. is stored in fast on-chip memory and is indexed by a hash of the word address. If a transaction aborts, state can be rolled back and the transaction is retried. All transactions assigned to a DPU will commit eventually. Inter-DPU conflicts are handled by deterministic concurrency control (i.e., the pre-assigned sequence numbers). Source: https://dl.acm.org/doi/10.1145/3767295.3803621 Results Fig. 3 compares performance of PIM-TIDE vs a CPU baseline for a mix of TPC-C transactions, wow that is a significant speedup. I believe all transactions/subtransactions are written by hand in C code that is compatible with UPMEM DPUs. Source: https://dl.acm.org/doi/10.1145/3767295.3803621 Dangling Pointers TPC-C is easy to partition; I wonder how well PIM-TIDE does on workloads that are not as partitionable. Also, this scheme doesn’t seem to allow for interactive transactions, how important are those in the real world? Thanks for reading Dangling Pointers! Subscribe for free to receive new posts. Local transactions execute entirely within a DPU Distributed transactions execute across multiple DPUs

database Database c C hardware Hardware performance Performance
0 views
<antirez> 1 months ago

Distributing LLM inference in DwarfStar

High end NVIDIA cards, and the server and power needed to run them, cost a lot of money, especially if you plan to reach enough VRAM to run massive models. The alternative, so far, has been Apple hardware, or the DGX Spark that, even if severely limited because of memory bandwidth, still allows to run LLMs prompt processing (prefill) fast enough. The Mac Studio provided up to 512GB unified memory, a solution with modest memory bandwidth (but much better than the Spark) and compute at a price that was, after all, given the current situation, relatively fair. For instance, with DwarfStar the Mac Studio M3 Ultra 512GB can run DeepSeek v4 PRO at 150 t/s prefill and ~10-13 t/s decoding, not great but at a level that is usable for certain use cases. Even 2-bit quantized, DeepSeek v4 PRO resists very well, like Flash at the same quantization (today I made PRO write a C compiler, I'll publish the video soon). I would not consider a trivial fact to run a frontier model at home, with a ~12k total spending. One could expect this to get better and better, but the situation at the horizon appears cloudy. There is almost zero hope that NVIDIA setups will get less expensive, and even a small company can’t afford to easily purchase and handle a small data center for local inference. At the same time the RAM shortage is making it not exactly likely that we will see a Mac Studio with an M5 Ultra, maybe 1.2T/s memory bandwidth and more compute (the M5 Max is already faster, compute wise, and has the Neural Accelerators inside each GPU core that help with certain models). So the current situation for local inference is that the best machine is probably a laptop. The M5 Max 128GB can run DeepSeek v4 Flash and Mimo V2.5, 2-bit quantized, at very decent prefill and decoding speeds. We are talking of ~500 t/s prefill and ~35-40t/s decoding speed, with a performance slope as the context size increases which is very acceptable. At the cost of 6-7k depending on the configuration, this is currently one of the best deals. If this is the situation, for local inference projects in general, and for DwarfStart in particular, looking at distributed inference starts to be interesting. What we can do if we have two, three, four MacBook M5 Max systems? Or two M3 Ultra with 512 GB of RAM? Traditionally there are two main systems to run distributed inference. One is to duplicate memory by loading 50% of the transformer layers in computer A, the remaining 50% on computer B, and running the inference in a sequential way. In this case there is to send just the activations around, that’s very simple conceptually, and with some micro-batching magic it is possible to not just duplicate the memory but even in theory to increase substantially the prompt processing speed (but not the decoding: for a single token generation you have to wait the first layers on machine A, the remaining layers on machine B, and so forth — but at least less heat will be produced so it is possible to use a sustained load), which is not bad at all. This means, for example, that the lucky ones that have two Mac Studio 512GB machines could run full size DeepSeek v4 PRO (even if even the 2-bit quants are running very, very well) and with micro-batching even enjoy a faster prefill. Another approach is, using Apple RDMA, to parallelize the execution across the two machines, a vertical split basically. For instance one could try to load the same 2 bit quants on machine A and B, so that both fit, and each side has *all* the routed experts. Then for each layer we could try to do the coordination needed in order to execute half the experts in machine A, half in machine B, and so forth (note that both machines have all the experts, so whatever the router says, we can send 50% of the computation to the other machine, and the activations are tiny). This is more viable for the PRO that has much larger routed experts, so the communication penalty is less sensible. But if this could be made to work well, is all to be seen. There is also tensor parallelism, you are thinking, right? But I bet this is not viable at all with the communication speed we have among two Apple computers, two DGX Spark and so forth (go read the speed of NVLink). The magic about the above two models is that you have to send very little data. Ok, so far I bet you are thinking, this is the same shit everybody knows about running LLMs in a parallel fashion, and indeed this is true. But this post was conceived to reach this exact point. What about if we could, instead, parallelize two Mac or DGX in a completely different way? Open weights models are now in a golden age, we have plenty and many are very powerful. In the 128GB 2-bit quants classes there are many interesting: Minimax M2.7, Mimo V2.5, DeepSeek v4 Flash, and a few more. At the same time it was recently noted that LLMs ensemble (https://arxiv.org/abs/2502.18036) is an understudied possibility that allows two models to run in a completely shared-nothing way in two different machines, to only combine the logits or select the best continuation at the end. There are different ways to do that, and it works even if the two models have different vocabularies: you can pick the continuation where the perplexity is lower (that is, pick the model which is more sure: it’s like a two experts MoE where the routing is implicit), and it is even possible to combine the logits (with some complexities given by the different vocabularies) and sample from there. More recent papers suggest that mixing the two techniques is the best approach. Anyway: these techniques seem to really work, models appear to do better than alone. It’s like if their knowledge is improved because each one brings his POV on what to say next. Maybe this is one of the most logical third approach to try, other than the first two. I really hope to find the time to play more with all that, in the next months. Comments

c C cloud Cloud ai AI
0 views
Michael Stapelberg 1 months ago

How my minimal, memory-safe Go rsync steers clear of vulnerabilities

Back in January 2025, multiple different security researchers published a total of 6 security vulnerabilities in rsync , some of which allow arbitrary code execution and file leaks, so naturally I was wondering whether/how my gokrazy/rsync implementation was affected. Did implementing my own (compatible, but minimal) rsync in Go, a modern and memory-safe programming language, really rule out entire classes of security vulnerabilities? This deep dive article was in the making since January 2025, but was delayed because we uncovered more unpublished vulnerabilities in the process! The “Security Vulnerabilities” section now covers all 12 vulnerabilities from the January 2025 batch and the May 2026 batch. If you are running (upstream, samba) rsync in production, upgrade to version 3.4.3 or newer. If you are running gokrazy/rsync in production, upgrade to version v0.3.3 or newer. Feel free to skip over the nitty-gritty security issue details and jump directly to: For context, I blogged about rsync, how I use it, and how it works back in June 2022. See also all posts tagged “rsync” . The original motivation for writing my own rsync (back then only a server, today all directions are supported) was to provide the software packages of distri, my Linux distribution research project for fast package management , which I wanted to host on router7 , my small home Linux+Go internet router, which in turn is built on gokrazy , my Go appliance platform. I am still running multiple gokrazy/rsync servers for this original purpose, and also many others! Having rsync available as a primitive (that you can link into your Go programs!) is really nice. This article covers the following security vulnerabilities: The first batch of the vulnerabilities above was announced on the oss-security mailing list , but note that the original report has more detail compared to the oss-security summaries! The later vulnerabilities were announced via GitHub Security Advisories on the rsync project . When the checksums are read by the daemon, two different checksums are read: Most importantly, note that field is filled with bytes. always has a size of 16: rsync.h is an attacker-controlled value and can have a value up to bytes, as the next snipper shows: The problem here is that can be larger than 16 bytes, depending on the digest support the binary was compiled with: md-defines.h support is common and sets the value to 64. As a result, an attacker can write up to 48 bytes past the buffer limit. Upstream fix: The upstream fix for CVE-2024-12084 changes the field to a dynamically-allocated field, which is allocated with length, and fixes the bounds check to check against the (checksum length for this transfer’s algorithm). Can Go help prevent this? Yes: Missing or incorrect bounds checks will not result in a heap buffer overflow in Go! Instead, attempting to write out of bounds will result in a panic because the Go runtime performs bounds checks. How does gokrazy/rsync fare? gokrazy/rsync also had insufficient validation! Our issue was different, though: It wasn’t size confusion, we just were not doing any validation of the sum header at all — oops! We can confirm that the Go runtime’s bounds check triggers on an attempt to write out of bounds by changing the code like so and running the tests: As expected, the Go runtime panics with the following message: Of course, crashing the entire server is not the best failure mode, so I added the missing bounds checking to turn the panic into an error . Because of the same lack of validation as in the previous CVE-2024-12084 vulnerability, an attacker could select a checksum algorithm with short checksums (e.g. with 8 byte checksums), but then claim they were sending longer checksums (e.g. 9 bytes), making the victim leak one byte of uninitialized stack content in the response. Leaking one byte of stack content may seem benign, but as the Google Security report puts it: The first pair of vulnerabilities are a Heap Buffer Overflow and an Info Leak. When combined, they allow a client to execute arbitrary code on the machine a Rsync server is running on. The client only requires anonymous read-access to the server. The daemon matches checksums of chunks the client sent to the server against the local file contents in . Part of the function prologue is to allocate a buffer on the stack of bytes: The daemon then iterates over the checksums the client sent and generates a digest for each of the chunks and compares them to the remote digest: Notably, the number of bytes that are compared again are bytes. In this case, the comparison does not go out of bounds since can be a maximum of . However, the local buffer, not to be confused with the attacker-controlled , is a buffer on the stack that is not cleared and thus contains uninitialized stack contents. A malicious client can send a (known) checksum for a given chunk of a file, which leads to the daemon writing 8 bytes to the stack buffer . The attacker can then set to 9 bytes. The result of such a setup would be that the first 8 bytes match and an attacker-controlled 9th byte is compared with an unknown value of uninitialized stack data. An attacker can divide a file into 255 chunks and as a result leak one byte per file download. An attacker can incrementally repeat the process, either in the same connection or by resetting the connection. As a result, they can leak bytes of uninitialized stack data, which can contain pointers to Heap objects, Stack cookies, local variables and pointers to global variables and return pointers. With those pointers they can defeat ASLR. Upstream fix: There are two relevant upstream fixes: Can Go help prevent this? Yes: By design, Go initializes all variables to the zero value. Go programmers do not need to remember to explicitly initialize variables. How does gokrazy/rsync fare? gokrazy/rsync is not affected by this vulnerability: Variables are always initialized in Go. Additionally, selecting checksums other than MD4 was only introduced in protocol version 30 (gokrazy/rsync implements protocol version 27). Description: (quoting the Google Security report ) When the syncing of symbolic links is enabled, either through the or ( ) flags, a malicious server can make the client write arbitrary files outside of the destination directory. A malicious server can send the client a file list such as: Symbolic links, by default, can be absolute or contain characters such as . In practice, the client validates the file list and when it sees the entry, it will look for a directory called , otherwise it will error out. If the server sends as [both, a directory and a symbolic link], [the client] will only keep the directory entry, thus the attack requires some more details to work. In mode, which the server can enable for the client, the server sends the client multiple file lists. The deduplication of the entries happens on a per-file-list basis. As a result, a malicious server can send a client multiple file lists, where: As a result, the directory is created first and is considered a valid entry in the file list. Then, the attacker changes the type of to a symbolic link. When the server then instructs the client to create the file, it will follow the symbolic link and thus files can be created outside of the destination directory. Can Go help prevent this? No. This vulnerability is caused by a logic error: when multiple file lists are used, the merged file list needs to be re-verified. But see Defense in depth: Go’s Upstream fix: The upstream fix for CVE-2024-12087 adds the missing validation. How does gokrazy/rsync fare? gokrazy/rsync is not affected by this vulnerability: gokrazy/rsync does not implement the incremental recursion mode ( ). The trade-off here is implementation complexity vs. resource usage: the incremental recursion mode allows working with the file set in a “windowed” way, as opposed to having to scan the entire file set before any transfer can begin. See also my How does rsync work? blog post. Description: (quoting the Google Security report ) The CLI flag makes the client validate any symbolic links it receives from the server. The desired behavior is that symbolic links target can only be 1) relative to the destination directory and 2) never point outside of the destination directory. The function is responsible for validating these symbolic links. The function calculates the traversal depth of a symbolic link target, relative to its position within the destination directory. As an example, the following symbolic link is considered unsafe: As it points outside the destination directory. On the other hand, the following symbolic link is considered safe as it still points within the destination directory: This function can be bypassed as it does not consider if the destination of a symbolic link contains other symbolic links in the path. For example, take the following two symbolic links: In this case, foo would actually point outside the destination directory. However, the function assumes that is a directory and that the symbolic link is safe. Upstream fix: The upstream fix for CVE-2024-12088 makes stricter by not allowing anywhere within the path, except at the very beginning. Can Go help prevent this? No. This vulnerability is caused by a logic error: the validation function was incorrect. We could have implemented that same bug. But see Defense in depth: Go’s How does gokrazy/rsync fare? gokrazy/rsync is not vulnerable: The feature is not yet implemented in gokrazy/rsync. The rsync receiver (in client mode) did not sanitize file names provided by the rsync sender, or otherwise prevent opening files outside the destination tree. A malicious sender could instruct a receiver to compare checksums of arbitrary files outside the destination tree. By observing the receiver’s reaction to a provided one-byte checksum, a malicious sender can leak arbitrary files. When a client connects to a malicious server the server is able to leak the contents of an arbitrary file on the client’s machine. In the client will read type as well as the from the server if the server sets the appropriate flags. The flag will not be set for the client. The caller ( ) then uses the server provided values to determine a file to compare the incoming data with. In the contents of the file specified by are copied into the destination file. This can be achieved by the server sending a negative token. The server sends a checksum to compare. If they don’t match, a 0 is returned. When the return value is 0 the receiver will then send a to the generator. The generator will then write a message to the server. The server can use this as a signal to determine if the checksum they sent was correct. By starting off with a of 1 a malicious server is able to determine the contents of the target file byte by byte. Upstream fix: The upstream fix for CVE-2024-12086 prevents opening files outside the destination tree by verifying the sender-provided path. Can Go help prevent this? Yes, Go offers an API to prevent this, see Defense in depth: Go’s . How does gokrazy/rsync fare? gokrazy/rsync is not vulnerable: the fuzzy matching feature was introduced with rsync protocol version 29, but gokrazy/rsync implements protocol version 27. Description: (quoting the Red Hat Security Advisory ) A flaw was found in rsync. This vulnerability arises from a race condition during rsync’s handling of symbolic links. Rsync’s default behavior when encountering symbolic links is to skip them. If an attacker replaced a regular file with a symbolic link at the right time, it was possible to bypass the default behavior and traverse symbolic links. Depending on the privileges of the rsync process, an attacker could leak sensitive information, potentially leading to privilege escalation. Upstream fix: The upstream fix for CVE-2024-12747 changes calls in the rsync sender to use the option. The paths are not expected to be symlinks at that point in the algorithm (symlinks would be handled with ). Can Go help prevent this? Yes, Go offers an API to prevent this, see Defense in depth: Go’s . How does gokrazy/rsync fare? gokrazy/rsync was vulnerable before commit , which introduces the same mitigation that upstream rsync uses. To reproduce the issue, use the following steps: Check out gokrazy/rsync v0.2.7: Patch the code as follows to undo the fix and execute the attack: Running the test now shows that the server traversed the symlink: A surprising discovery When I shared a draft of this article with Damien Neil, member of the Go Security Team and the author of the traversal-resistant API , he pointed out: I believe the gokrazy fix for CVE-2024-12747 is insufficient. You’re calling with , but only prevents symlink traversal in the last path component. This is probably still vulnerable to replacing an earlier path component so can be redirected by symlinking to . We reported this to the rsync security contact address in April 2025. In December 2025 I learned that someone else had also independently discovered and reported this issue. Ultimately, this resulted in CVE-2026-29518, published on 2026-05-20. Description: (quoting the rsync 3.4.3 NEWS entry ) TOCTOU symlink race condition allowing local privilege escalation in daemon mode without chroot. An rsync daemon configured with is exposed to a time-of-check / time-of-use race on parent path components. A local attacker with write access to a module can replace a parent directory component with a symlink between the receiver’s check and its open(), redirecting reads (basis-file disclosure) and writes (file overwrite) outside the module. Under elevated daemon privilege this allows privilege escalation. Default is not exposed. Reach: local attacker on the daemon host, write access to a module path, daemon configured with . Upstream fix: The upstream fix for CVE-2026-29518 uses , which is similar to Go’s API. Can Go help prevent this? Yes, Go offers an API to prevent this, see Defense in depth: Go’s . How does gokrazy/rsync fare? gokrazy/rsync was vulnerable until I switched the sender and the receiver to the traversal-resistant API . Description: (quoting the GitHub Security Advisory ) Description: The receiver’s compressed-token decoder accumulated a 32-bit signed counter without overflow checking. A malicious sender can trigger an overflow that, with careful manipulation, leaks process memory contents to the attacker – environment variables, passwords, heap and library pointers – significantly weakening ASLR and facilitating further exploitation. Reach: authenticated daemon connection with compression enabled (the default for protocols >= 30 when both peers advertise it). Disabling compression on the daemon (“refuse options = compress” in rsyncd.conf) is the available workaround. Upstream fix: The upstream fix for CVE-2026-43618 introduces the missing checks. How does gokrazy/rsync fare? gokrazy/rsync is not vulnerable because it does not implement compression. See gokrazy/rsync issue #35 for details on why compression support sounds simple, but is non-trivial. Description: (quoting the GitHub Security Advisory ) The 2025 fix that added a guard in was not applied to the visually-identical block in . A malicious rsync server can drive any connecting client into a deterministic by setting in the compatibility flags, sending a flist whose first sorted entry is not a leading “.” directory (which causes to set ), then sending a transfer record with and a non- iflag word. The receiver reads and dereferences the result. On glibc x86-64 the dereferenced pointer is mmap chunk metadata that lands at an unmapped address, hence a clean ; non-glibc allocators have not been audited. Reach: any rsync client doing a normal pull from an attacker-controlled URL. Works for both rsync:// URLs and remote-shell pulls. is the protocol-30+ default; no special options are required on the victim. Workaround: on the client. Upstream fix: The upstream fix for CVE-2026-43620 adds the guard to as well. How does gokrazy/rsync fare? Just like for CVE-2024-12087 , gokrazy/rsync is not affected by this vulnerability: gokrazy/rsync does not implement the incremental recursion mode ( ). Description: (quoting the GitHub Security Advisory ) Description: Earlier fixes for symlink races on the receiver’s open() call (CVE-2026-29518) missed the same race class on every other path-based system call: chmod, lchown, utimes, rename, unlink, mkdir, symlink, mknod, link, rmdir, lstat. On rsync daemons with “use chroot = no” a local attacker with filesystem access on the daemon host can swap a symlink into a parent directory component between the receiver’s check and one of these syscalls, redirecting it outside the exported module. The fix routes each affected path-based syscall through a parent dirfd opened under RESOLVE_BENEATH-equivalent kernel-enforced confinement (openat2 on Linux 5.6+, O_RESOLVE_BENEATH on FreeBSD 13+ and macOS 15+, per-component O_NOFOLLOW walk elsewhere). Default “use chroot = yes” is not exposed. Reach: local attacker on the daemon host, write access to a module path, daemon configured with use chroot = no. Upstream fix: The upstream fix for CVE-2026-43619 uses the family of syscalls, just like Go’s . Can Go help prevent this? Yes, Go offers an API to prevent this, see Defense in depth: Go’s . How does gokrazy/rsync fare? gokrazy/rsync is not affected, because it uses Go’s API throughout. Description: (quoting the GitHub Security Advisory ) On an rsync daemon configured with the global rsyncd.conf setting, the reverse-DNS lookup of the connecting client was performed after the daemon had chrooted into . If did not contain the files glibc needs for resolution ( , , , NSS service modules), the lookup failed and the connecting hostname was set to “UNKNOWN”. Hostname-based deny rules (“hosts deny = *.evil.example”) therefore could not match, and an attacker controlling their PTR record could connect from a hostname the administrator had intended to deny. IP-based ACLs are unaffected. The per-module setting is unrelated to this issue. Reach: rsync daemon configured with AND hostname-based ACLs AND does not include the libc resolver fixtures. Upstream fix: The upstream fix for CVE-2026-43617 moves the DNS lookup to an earlier point in the protocol. How does gokrazy/rsync fare? gokrazy/rsync is not vulnerable because we only implement IP-based allow/deny lists, not hostname-based allow/deny lists. Description: (quoting the GitHub Security Advisory ) The rsync client’s HTTP proxy support contains an off-by-one out-of-bounds stack write in ( ). After issuing the request, rsync reads the proxy’s first response line one byte at a time into a 1024-byte stack buffer with the bound , so the loop only ever writes . If the proxy (or a man-in-the-middle in front of it) returns 1023+ bytes on the first response line without a terminator, the loop exits with — a slot the loop never wrote, so holds stale stack bytes left there by the earlier that formatted the outgoing request. The post-loop code then does: The lands one byte past the end of the on-stack , corrupting whatever lives in the adjacent stack slot. AddressSanitizer reports at in the frame. Upstream fix: The upstream fix for CVE-2026-45232 validates the attacker-supplied data. How does gokrazy/rsync fare? gokrazy/rsync does not implement such proxy support, so it is not vulnerable. Let’s summarize how Go fares: Aside from being written in Go, another key difference between gokrazy/rsync and the official upstream rsync is that the gokrazy implementation is minimal : Let’s have a look at whether gokrazy/rsync was affected by each CVE at the time of publishing: To be clear: all known vulnerabilities are fixed in gokrazy/rsync! The table above documents what the state was at the time when each CVE was published. In other words: When the January 2025 vulnerabilities were published, gokrazy/rsync panicked (CVE-2024-12084) and was vulnerable to a TOCTOU race (CVE-2024-12747). In the process of fixing the TOCTOU issue, we discovered CVE-2026-29518, which was fixed in gokrazy/rsync before the CVE was published. CVE-2026-43619 was discovered even later, but was also already fixed in gokrazy/rsync with the same fix: using Go’s everywhere. As I was reading the vulnerability reports, I noticed that the reports were slightly misleading by their choice of words: most reports just spoke of “server” and “client”. However, in an rsync transfer, both sides, the rsync client and the rsync server can assume either role: sender (upload files) or receiver (download files)! Some setups come with further restrictions that make certain attacks harder or impossible to pull off. For example, when running in daemon mode, file system access can be restricted to the pre-configured module paths (but not in command mode!). Here is a diagram to give you an overview of the 4 different setups and role/protocol layering: In the context of our vulnerability reports, I would say that the Arbitrary File Leak vulnerability (CVE-2024-12086)’s original title “Server leaks arbitrary client files” can easily be misunderstood. Instead, I would say: The rsync receiver will leak arbitrary files to a malicious sender . I have verified that a malicious client sender can make an unpatched remote rsync open files outside the destination tree (e.g. the system password database) when running in command mode, for example over SSH. (But, when running in daemon mode, the server enables additional path sanitization, which prevents this attack.) Similarly, the Symlink Path Traversal vulnerability (CVE-2024-12087) speaks about a “malicious server”, but again, it should be “malicious sender”, which can be either the client or the server. The OpenBSD project is known for its security focus, so how does openrsync compare? openrsync is not affected by the Heap Buffer Overflow (CVE-2024-12084) and Stack Info Leak (CVE-2024-12085) vulnerabilities because it validates the checksum length and only supports one checksum size/algorithm (MD4). openrsync is not affected by CVE-2024-12086, CVE-2024-12087 and CVE-2024-12088 because it does not implement the relevant features (like gokrazy/rsync). Even if it was vulnerable, openrsync’s defense-in-depth measures like using OpenBSD’s and to restrict file system access would have prevented successful exploitation — at least when running on OpenBSD. openrsync is not affected by CVE-2024-12747 because it used from the very moment they implemented symlink support . But, because is not a sufficient fix for this issue, openrsync is affected by CVE-2026-29518! The above covers the January 2025 batch of vulnerabilities; the May 2026 batch is similar in that most features just are not implemented. Overall, I say: Well done, Kristaps and contributors! By diligently implementing validation, restricting the attack surface and employing defense-in-depth measures, openrsync manages to not be affected by almost all of the reported vulnerabilities. Which APIs and environments can we use on Linux for defense-in-depth measures? I’ll go through the ones supports, ordered by traditional to modern. Within a few weeks after starting the project, I added support for dropping privileges and using mount/pid namespaces on Linux to restrict the file system objects that my rsync server could work with. This approach works very well to mitigate path traversal attacks, but requires privileges, meaning we need to run as or in a Linux user namespace (if enabled on your distribution / system). That limitation makes mount namespaces well-suited for server setups, but usually unavailable for interactive one-off transfers that are typically running under a human’s user account. In the same commit that introduced Linux mount/pid namespace support, I also included a systemd service file that restricted file system access to home directories and encouraged folks in the README to further restrict file system access, depending on what their use-case allows. These file system restrictions, if set up correctly, mitigate the File Leak (CVE-2024-12086) and Path Traversal (CVE-2024-12087) vulnerabilities. The Symlink Race Condition (CVE-2024-12747) relies on privilege escalation through the rsync process, but thanks to the DynamicUser feature, our process has fewer privileges than other users. Similarly to mount namespaces, these measures are great for server setups, but too cumbersome to set up for interactive one-off usages. I stumbled upon Justine’s blog post Porting OpenBSD pledge() to Linux (2022) and was reminded that Linux offers the Landlock API for unprivileged, per-process access control, similar to OpenBSD’s system call, which openrsync uses. The basic idea is that once your program knows the directory it works with, it makes a call like and no longer has access to other file system locations. I had previously heard of Landlock at a Go Meetup, so I knew there was Go support for Landlock. Back in 2022, I enabled Landlock support in the gokrazy kernel images. So I gave it a shot in March 2025 and implemented Landlock support to restrict file system access . It took me a few hours, which seems a little longer than one might expect at first. Making Landlock work (and/or skipping it) in our test environment ran into a couple of road blocks: Our tests had defined many functions that get run in the same process, but when repeatedly adding rulesets, we would exceed the limit of 16 (!) policy layers per process. Once I had it set up just right, it is a beautiful solution. Now we can restrict rsync transfers to their sources (read-only) or destination directories (read-write), even for unprivileged invocations of ! 🎉 The downside to Landlock is that Landlock operates at the process level. This means that Landlock policies must include the files that your program needs, e.g. needs to be able to read for user id lookup, so if the attacker is after the file, Landlock does not help. In February 2025, the Go 1.24 release introduced the API, which is resistant against path traversal, see The Go Blog: Traversal-resistant file APIs (by Damien Neil, March 2025). This API allows more fine-grained control (per file system operation) compared to Landlock. Go 1.25 (released in August 2025) added more methods to , making it a convenient choice for most file system usage. I have converted all of ’s file system usage to use , which is a great fit: users configure input/output directories, but the filenames received over the network are untrusted. That’s exactly what was designed for! When I first looked into using , I thought that some system calls could inherently not be made with this API, like for example to create device node files. Damien explained: It won’t support mknod, though. However, you should be able to use it to enable a safe mknod: If you’re curious how that looks in practice, check out ’s usage in , line 15-29 . Another stumbling block was when I realized that unlike with , Linux only implements , but no (as of Linux 7.0)! Luckily, Lennart Poettering pointed out that there’s a trick to skip path resolution without : you can probably bind to in the meantime… And indeed, this works! Path resolution is skipped because we only specify a basename (last component of a path) after the known-safe , not a path (see line 49-56 ). With these two tips, v0.3.1 and newer are fully using , meaning all file system access is traversal-safe! 🥳 Lacking validation causes vulnerabilities It is interesting to note that aside from the TOCTOU vulnerabilities (CVE-2024-12747, CVE-2026-29518 and CVE-2026-43619), all other vulnerabilities were caused by missing or incorrect input validation. In three cases, there was just no validation to begin with. In another case (CVE-2024-12088), the subject matter of file system path resolution is tricky enough that the existing validation did not cover all edge cases. As the Go verdict section explains in more detail, the most valuable structural fixes are to provide bounds checking (= always-on validation) and safe-by-default APIs like Go’s . Too much complexity A few of the vulnerabilities came from evolution of the rsync protocol: The code used to correctly perform sufficient validation, but then new features were added. For example, when checksum algorithm negotiation was added (protocol version 30), the validation was not correctly updated. When incremental recursion was added (also protocol version 30), the validation that made sense for individual file lists was not updated for the new processing approach of merging incremental file lists. Avoiding complexity avoids vulnerabilities! Both gokrazy/rsync and also openrsync were not vulnerable to 8 out of the 12 security vulnerabilities simply because they do not implement the feature with the vulnerability. Of course, these features were added to rsync because they were valuable to someone at some point, and of course I am not saying that we should just… not develop software any further, ever. But, I consider it ideal to use an implementation whose complexity is appropriate for and proportional to the complexity of the use-case . In other words: for simple use-cases, reach for a simple implementation. Only reach for the fully-featured implementation where needed. The verdict on whether using Go has helped . The verdict on whether a minimal re-implementation like gokrazy/rsync helps . My comparison with OpenBSD’s (written in C). Defense in depth mechanisms one can use on Linux. The conclusion . CVE-2024-12084 to 12088 (original report) CVE-2024-12747 (discovered separately by Aleksei Gorban “loqpa”) CVE-2026-29518 (discovered by Damien Neil and myself! and independently by Nullx3D ) CVE-2026-43617 to 43620 CVE-2026-45232 rsync performed insufficient validation: It read the (attacker-controlled) checksum length from the network and compared the length against . However, rsync’s data structures always declared a 16 byte buffer: is always 16 (bytes), which is sufficient to hold an MD4 or MD5 checksum. used to be 16 (bytes), but can be larger when rsync is compiled with SHA256 or SHA512 checksum support. Hence, the bounds check was ineffective! An attacker could write out of bounds. This issue was introduced with commit in September 2022 , which added SHA256/SHA512 checksum support. A 32-bit Adler-CRC32 Checksum A digest of the file chunk. The digest algorithm is determined at the beginning of the protocol negotiation. The corresponding code can be seen below: sender.c : The “Some checksum buffer fixes” commit prevents this attack because the attacker-controlled can no longer be larger than the transfer’s checksum length. The “prevent information leak off the stack” commit initializes the memory to zero, thereby making any stack leak through impossible. Check out gokrazy/rsync v0.2.7: Patch the code as follows to undo the fix and execute the attack: The Go runtime’s bounds checks turn more serious security issues into a panic. A panic is still a denial-of-service risk, but that’s much preferable. Go initializes memory to zero, making info leaks like CVE-2024-12085 impossible. Go’s API prevents most of the remaining vulnerabilities. Only one out of twelve vulnerabilities (CVE-2026-43617) is a proper bug in the application logic that using Go could not have prevented. gokrazy/rsync is unaffected by many vulnerabilities because it does not implement the feature in question, for example . Like all other wire protocol-compatible rsync implementations, gokrazy/rsync targets protocol version 27, because later protocol versions introduce significant complexity. In some cases, features that would be good to implement come with significant blockers, e.g. compression is tricky, see gokrazy/rsync issue #35 for details. os.Root.OpenFile the parent directory of the target, File.Fd to get the file descriptor for that directory, https://pkg.go.dev/golang.org/x/sys/unix#Mknodat to create the file.

go Go open-source Open Source c C shell Shell security Security
0 views
Xe Iaso 1 months ago

"No way to prevent this" say users of only language where this regularly happens

In the hours following the release of CVE-2026-45584 for the project Microsoft Windows , site reliability workers and systems administrators scrambled to desperately rebuild and patch all their systems to fix a memory safety vulnerability resulting in arbitrary code execution inside the virus scanner Windows Defender. This is due to the affected components being written in C++, the only programming language where these vulnerabilities regularly happen. "This was a terrible tragedy, but sometimes these things just happen and there's nothing anyone can do to stop them," said programmer Dr. Annabelle Connelly, echoing statements expressed by hundreds of thousands of programmers who use the only language where 90% of the world's memory safety vulnerabilities have occurred in the last 50 years, and whose projects are 20 times more likely to have security vulnerabilities. "It's a shame, but what can we do? There really isn't anything we can do to prevent memory safety vulnerabilities from happening if the programmer doesn't want to write their code in a robust manner." At press time, users of the only programming language in the world where these vulnerabilities regularly happen once or twice per quarter for the last eight years were referring to themselves and their situation as "helpless."

c++ C++ security Security programming Programming c C
0 views
Blargh 1 months ago

Everything in C is undefined behavior

If he had been a programmer, Cardinal Richelieu would have said “Give me six lines written by the hand of the most expert C programmer in the world, and I will find enough in them to trigger undefined behavior”. Nobody can write correct C, or C++. And I say that as someone who’s written C and C++ on an almost daily basis for about 30 years. I listen to C++ podcasts. I watch C++ conference talks. I enjoy reading and writing C++. C++ has served us well, but it’s 2026, and the environment of 1985 (C++) or 1972 (C) is not the environment of today. I’m definitely not the first to say this. I remember reading a post by someone prominent about a decade ago saying that a good case can be made that use of C++ is a SOX violation. And while I was not onboard with the rest of their rant (nor their confusion about “its” vs “it’s”), I never disagreed about that point. With time I found it to be more and more true. WAY more things are undefined behavior (UB) than you’d expect. Everyone knows that double-free, use after free, accessing outside the bounds of an object (e.g. array), and accessing uninitialized memory is UB. After all, C/C++ is not a memory safe language. And yet we as an industry seem to be unable to stop making even those mistakes over and over. But there’s more. More subtle. More illogical. Some people seem to think that as long as they don’t compile with optimizations turned on, undefined behavior can’t hurt them. They believe that the compiler is somehow being deliberately hostile, going “AHA! UB! I can do whatever I want here!”, and without optimizations turned on it won’t. This is incorrect. UB doesn’t mean that the compiler can take advantage of your sloppiness. UB means that the compiler can assume that your code is valid. It means that the intention of your code that’s oh so obvious when read by a human, doesn’t even have a way to be expressed between compiler stages or modules. UB means that the compiler doesn’t even have to implement some special cases in its code generation, because they “can’t happen”. The compiler, and really the underlying hardware too, is playing a game of telephone with your UB intentions. It may end up with what you wanted, but there’s no guarantee for now or in the future. The following is not an attempt at enumerating all the UB in the world. It’s merely making the case that UB is everywhere, and if nobody can do it right, how is it even fair to blame the programmer? My point is that ALL nontrivial C/C++ code has UB. As an example of this, take this code: If this function is called with a pointer not correctly aligned (probably meaning on an address that’s a multiple of , but who knows), this is UB. C23 6.3.2.3. On Linux Alpha, in some cases this would merely trap to the kernel, which would software emulate what you intended. In other cases it would (probably) crash your program with a SIGBUS. On SPARC it would cause a SIGBUS. Sure, on x86/amd64 (henceforth just “x86”) this is likely fine. Hell, it’s probably even an atomic read. x86 is famously extremely forgiving about cache coherency subtleties. So here we have three cases: What about ARM, RISC-V, and others? What about future architectures? A future architecture could even have special that do not populate the lowest bits, because such pointers cannot exist. Even if it works, maybe the compiler one day changes from using one load instruction to another, and suddenly that’s no longer fixed up by the kernel. Because the compiler is not obligated to generate assembly instructions that work on unaligned pointers . Because it’s UB. Or how about this: Is this operation atomic when the object is not correctly aligned? That’s the wrong question to ask. Mu , unask the question. It’s UB. (but also yes, in practice this can easily be an atomicity problem) If you want to get even more convinced, you can try thinking about what happens if an object you thought you were reading atomically spans pages . But don’t think too much about it, or you may conclude that “it’s fine”. It’s not. It’s UB. Don’t blame the function, above. The act of dereferencing the pointer wasn’t the problem. Merely creating the pointer was enough to be a problem. That cast is the problem, not . It’s perfectly valid for the compiler to assign specific meaning, such as garbage collection or security tagging bits, to the lower bits of an . is a simple function that takes a character and returns if it’s a hex digit. 0-9 or a-f. It can also take the value . Uh, ok. What value is ? Per C23 7.4p1 we know it’s an , and we can infer that it’s not representable by . therefore takes an , not a . All values of fit inside , so we should be fine. Casting from to fits, so per section 6.3.1.3 we’re fine, right? No. Because if is called with a value other than 0-127, and on your architecture is (implementation defined, per 6.2.5, paragraph 20 in C23 ), then the integer value ends up negative. And the following is a valid implementation of , that would cause a read of who-knows-what memory. It could even be I/O mapped memory, triggering things to happen that is more than merely getting a random value or crash. It could cause the motor to start. Less likely in an application running in a desktop operating system than in an embedded system, sure. But there are user space network drivers (for performance), so even user space won’t protect you. And, by omission, it’s also UB if the float is a non-finite value. So how do you compare a float to ? Do you cast the float to ? No, that’s the UB you want to avoid. So you cast to float? How do you know it can be represented exactly? Maybe casting to rounds to a value not representable in , and your comparison becomes non-representative? Maybe the following works? You’ll miss out on representing some really high values, but maybe that’s OK? I just wanted to convert a float to an int. :-( I bet there’s lots of code out there that take a value in seconds, and convert it to integer milliseconds, by just multiplying and casting. Most programmers won’t have to deal with this, but I don’t think there’s any C standards compliant way in practice to put an object at address zero. This can come up in OS kernel and embedded coding. By 6.3.2.3 an integer constant zero (which is convertible to a pointer) and are the “null pointer constant” (which I’ll just call ). C doesn’t specify that the actual pointer points addr machine address zero , because the C standard only talks of the C abstract machine, not about hardware. All C guarantees is that if you compare to zero you’ll see them equal. But for all you know that’s because the zero is converted to the native platform’s , which happens to be . It also explicitly says that dereferencing a null pointer, no matter what the value, is undefined behavior. It’s the example of UB under 3.4.3. This also means that you can’t assume that will create a pointer! You cannot initialize your structs this way and assume member pointers are ! And this does apply to most programmers. And yes, some historic machines used non-zero NULL pointers . But let’s say you have a modern machine, where is a pointer to address zero, and you actually have an object there. Again, C 6.3.2.3 says that compares unequal to “any object or function”. So this is UB: C says “there is no function there”. For all you know the compiler has no internal way to even express your intention here. You may argue that “but surely it’ll just emit a call instruction to the bit pattern of all zeroes? Nothing else seems reasonable. What is “all zeroes”, though? On 16bit x86, is it ? Is it ? This is UB: This is not: Because the argument needs to be a pointer, and the macro may be misinterpreted as an integer zero. Similarly, this is UB: It needs to be: So how do you print an ? Well, you could cast them to and print them using . But is even unsigned? Oh well, worst case you get a nonsense value printed instead of , I guess. Sure, you probably knew this. But did you consider the security aspects of it? It’s not rare for the denominator to come from untrusted input. And there’s so much more. The C23 standard contains 283 uses of the word “undefined”. And that’s not even including the things that are undefined by omission. Nobody can find integer promotion rules and code skimming speeds. Nobody . This post is already long enough, but as a start: Point an LLM at ANY C code, asking it to find UB, and it will. And it’ll be right almost all the time, nowadays. I felt a bit bad after it correctly found ones in my code, so I thought I’d point it at the mature and pedantically written OpenBSD. I just picked the first tool I could think of, , and it spit out a bunch. I sent the project a patch for an out of bounds write (and also for a non-UB logic bug ). I didn’t send them patches for the UB that was left and right, partly because the OpenBSD project has not been very receptive in the past for bug reports, my sense of “this is probably fine, in practice”, and that if OpenBSD wants to weed out UB from their code base, then that’s a major project that should be done in a better way than me just being the middle man between the LLM and them for a patch here and there. We can’t just throw away our C/C++ code bases. But leaving them inherently broken is also not an option. We need some way of fixing UB at scale, without committing AI slop nor overwhelming human reviewers. This too is not a new opinion, nor a great revelation. But yes, writing C/C++ in 2026 without an LLM supervising you for UB should probably be seen as a SOX violation, and just plain irresponsible. If OpenBSD people can’t find these problems gives 30+ years, what chance do the rest of us have? It may not scale to large code bases, but for my own projects I’ve asked the LLM to find UB, if necessary explain it, and fix it. And then stare at the output until I can confirm the issue and the fix. A problem with this is that in order to confirm the findings, you’ll need an expert human. But generally expert humans are busy doing other things. This is janitor work, but too subtle to leave to the junior programmers who have traditionally been assigned janitor work. kernel gave a helping hand (Alpha for some loads) crash (other Alpha loads, and SPARC) not a problem (x86) No way to parse integers in C Integer handling is broken UB in the Linux kernel Integer promotion

c C security Security programming Programming c++ C++
0 views
Max Bernstein 2 months ago

Partial static single information form

In compilers, static single information form (SSI) is a common extension to static single assignment form (SSA). It was introduced by C. Scott Ananian in 1999 in his MS thesis (PDF) 1 . SSI extends your existing SSA intermediate representation by discovering facts from your existing program and reifying them as path-dependent/flow-sensitive IR nodes. That might sound complicated, but at least the basic idea is pretty natural. I talk a little bit about it in What I talk about when I talk about IRs and I’ll rehash here in more depth, starting with some motivating examples. Consider this admittedly contrived example: We should be able to learn from the comparison that in some branches in the IR, is positive. In that region, we can add a new IR instruction that attaches that knowledge right in the instruction’s type field (yay, sparseness!) and then rewrite uses of to now use . Because we’ve done that, our (imaginary) optimization rule that gets rid of on known-positive integers can kick in, and we can delete the invocation of . Yay, optimization! But a couple of questions remain, at least for me: We’ll go through them, starting with the compiler pipeline. The original SSI paper starts with (I think?) SSA form and places some number of new refinement nodes based on conditionals. I have admittedly not tried very hard, but the into-SSI algorithms look complicated and kind of heavyweight. As a reward, you get “linear” into-SSI time complexity. But I am a humble compiler engineer, and I don’t have the time to go through and load all of this into my head. Instead what I have seen done and have been doing is to take a shortcut: build partial SSI during SSA construction 2 . Most of the time this is from bytecode, but it could also be from some other non-SSA IR. In any case, this is an excellent shortcut for two reasons: This is pretty compelling. We can learn from the bytecode with a very small amount of marginal new complexity. See my implementation in ZJIT , for example. All it really does is modify the abstract interpreter state when building SSA out of , , and bytecode instructions to take into account the new refined values. This is fine for branches that are already in the user’s source program but sometimes optimization, especially of dynamic languages, adds new branches that were not there before. And sometimes these branches get added much later, long after SSA construction. What then? Can we do something similar and rely on existing infrastructure? Implicit in this “can we do it” is the assumption that your IR tracks data dependencies from use to corresponding def, but not from def to uses. Sea of Nodes (at least the Simple implementation), is an IR that tracks both directions all the time for easier rewriting. Many IRs do not do this, so we will continue assuming that there’s no “easy way out”. JIT optimization of dynamic language compilers often adds synthetic instructions to the IR that enforce pre-conditions. These guards allow optimizing happy/fast path cases in JIT code while leaving the interpreter as a fallback. For example, we might be able to optimize two back-to-back instructions (a very dynamic operation in the world of ideas, but fast when concretely implemented using object shapes) from: which is very generic and involves calling into C code that might raise an exception, to something more like: which is much faster (assuming shape stability at run-time). There’s an irritating problem, though, which is that we have a bunch of duplicate instructions littered around the IR now because our optimizer worked on each instruction individually. Kind of a “template optimizer” situation. Now we need some pass to clean up the detritus. Global value numbering (GVN) will do a good job of de-duplicating instructions. It should notice that we already have an instruction that looks like called and rewrite into . That’s great because we have de-duplicated the guard. GVN may not get everything, though; if some instructions later use , they will not get rewritten to instead use the output of these new guard instructions. To do that, we need to add some kind of pass or augment GVN with some canonicalization feature. That canonicalization would handle rewriting operands to use the “latest version” of some value, so to speak. See the canonicalization section of Chris Fallin’s excellent aegraphs blog post for more (and of course the (currently block-local) implementation in ZJIT ). Where I’m going with all of this, though, is that you may already have some dominance-based instruction rewriting mechanism in your compiler, either as part of GVN or separately! And you can use this to do a very low code into-partial-SSI in the middle of your optimizer. This means you could very well get away with inserting instructions in successor blocks of conditionals and get the into-SSI “for free”. That’s up to you. There’s a trade-off between compile-time and run-time, especially in JITs. Inserting more instructions and rewriting more times may slow down your compiler. It’s a cheap lunch, not a free one. I don’t know. I don’t have a good grasp of how this “partial SSI” compares to the “full SSI”. I don’t plan on implementing full SSI in the near future. I will note that this partial SSI approach doesn’t do two things: I can’t tell what impact this has. Like Simple, TruffleRuby is built on a Sea of Nodes IR (Graal). Chris Seaton has an excellent blog post about TruffleRuby’s use of “stamp nodes” (“Pi nodes” 3 ). The function does a lot of heavy lifting, I think because Graal tracks uses. Cinder mostly inserts instructions in the HIR builder, before into-SSA, and then lets the SSA construction take care of things. That’s where I learned this trick, actually. Here is one example of refining the type of the matched operand when building IR for pattern matching. Luau is working on something like this, but for their type checker. Chatting with someone on their team is actually part of the reason I got motivated to write this post. Android ART looks like it has HBoundType and inserts them in reference type propagation . This handles class checks, null checks, and instanceof checks. Last, I want to talk a little bit about some interesting reasoning you can do when you have two implementations of something that you can switch between. For example, JIT (+ interpreter), or aliasing and non-aliasing cases in C code, or the weirdo NULL-UB reasoning LLVM can do to C code, things like that. In ZJIT, we currently insert s opportunistically in “easy” cases when building our HIR from the interpreter bytecode. For example, if in the bytecode there is a branch that compares some value with , it will have two outgoing control-flow edges: one block where is definitely , and one block where is definitely not . In each of these control-flow edges, we can insert corresponding type refinement hints. That’s pretty standard. But we can also do weirder stuff. CRuby has a notion of heap objects vs immediate objects. Many (most?) objects are heap objects. However, integer , for example is not allocated on the heap but instead represented by a tagged bit pattern that pretends to be an address: the whole value is encoded in the pointer itself. We encode this knowledge in the HIR’s type system: “heapness” and “immediateness” each get a bit in the type lattice . We use this in the optimizer to reason about effects , among other things. We can’t know a lot of the time what type a thing is, so we pessimistically type most objects flowing through bytecode as . This type encapsulates the entire world of possible values that could go on the stack or in a local variable. On most heap objects, with only a few exceptions, you can write instance variables (fields, attributes, whatever you want to call them). You can never write an instance variable to an immediate. This means that if we observe the following pattern in the bytecode: Then after building and emitting HIR for the opcode, we can upgrade the type of from a to a . We can do this because if it weren’t a heap-allocated object, we would have left the compiled code and entered the interpreter. This is another SSI-type thing you can do in your compiler. Uhh I guess the conclusion is that you don’t have to do full SSI and partial SSI is available and not too scary? Does your compiler do this? Reader, please write in. …and optimized in 2002 (PDF), revisited in 2009 (PDF), implemented in LLVM in 2010 (PDF), investigated in 2017 for abstract compilation (PDF), and probably more. The 2009 paper by Boissinot, Brisk, Darte, and Rastello even shows that both Ananian and Singer’s papers have bugs, while perhaps unintentionally also making an excellent pun about the literature being “sparse”.  ↩ This blog post is different than the what the LLVM paper (PDF) calls partial SSI. Partial for different reasons. Maybe it’s not even single information anymore.  ↩ Today I learned that this terminology comes from the ABCD paper (PDF).  ↩ Where/when in the compiler pipeline do we insert and remove these type refinements? Do we need to refine after every conditional? Do we need to implement the whole into-SSI and out-of-SSI algorithms from all the complicated-looking papers? It lets me cleanly separate adding the type refinements (pretty straightforward) from the hard part of doing all of the operand rewriting and phi placement and marking and all manner of other nonsense. In addition to separating the concerns, the hard part is already done by SSA construction. We can actually just skip it! SSA construction handles phi placement, operand rewriting, all of it. It probably fits neatly into a naive or a Braun-style (PDF) construction. It doesn’t split variables with a new sigma node, and it generally inserts the refine node within the target block rather than above the branch (For only) It doesn’t insert new phi nodes; it just leaves both IR nodes available and, instead of re-merging, drops them …and optimized in 2002 (PDF), revisited in 2009 (PDF), implemented in LLVM in 2010 (PDF), investigated in 2017 for abstract compilation (PDF), and probably more. The 2009 paper by Boissinot, Brisk, Darte, and Rastello even shows that both Ananian and Singer’s papers have bugs, while perhaps unintentionally also making an excellent pun about the literature being “sparse”.  ↩ This blog post is different than the what the LLVM paper (PDF) calls partial SSI. Partial for different reasons. Maybe it’s not even single information anymore.  ↩ Today I learned that this terminology comes from the ABCD paper (PDF).  ↩

programming Programming c C backend Backend
0 views
daniel.haxx.se 2 months ago

Mythos finds a curl vulnerability

yes, as in singular one . Back in April 2026 Anthropic caused a lot of media noise when they concluded that their new AI model Mythos is dangerously good at finding security flaws in source code. Apparently Mythos was so good at this that Anthropic would not release this model to the public yet but instead trickle it out to a selected few companies for a while to allow a few good ones(?) to get a head start and fix the most pressing problems first, before the general populace would get their hands on it. The whole world seemed to lose its marbles. Is this the end of the world as we know it? An amazingly successful marketing stunt for sure. Part of the deal with project Glasswing was that Anthropic also offered access to their latest AI model to “Open Source projects” via Linux Foundation . Linux Foundation let their project Alpha Omega handle this part, and I was contacted by their representatives. As lead developer of curl I was offered access to the magic model and I graciously accepted the offer. Sure, I’d like to see what it can find in curl. I signed the contract for getting access, but then nothing happened. Weeks went past and I was told there was a hiccup somewhere and access was delayed. Eventually, I was instead offered that someone else, who has access to the model, could run a scan and analysis on curl for me using Mythos and send me a report. To me, the distinction isn’t that important. It’s not that I would have a lot of time to explore lots of different prompts and doing deep dive adventures anyway. Getting the tool to generate a first proper scan and analysis would be great, whoever did it. I happily accepted this offer. (I am purposely leaving out the identity of the individual(s) involved in getting the curl analysis done as it is not the point of this blog post.) Before this first Mythos report, we had already scanned curl with several different very capable AI powered tools (I mean in addition to running a number of “normal” static code analyzers all the time, using the pickiest compiler options and doing fuzzing on it for years etc). Primarily AISLE , Zeropath and OpenAI’s Codex Security have been used to scrutinize the code with AI. These tools and the analyses they have done have triggered somewhere between two and three hundred bugfixes merged in curl through-out the recent 8-10 months or so. A bunch of the findings these AI tools reported were confirmed vulnerabilities and have been published as CVEs. Probably a dozen or more. Nowadays we also use tools like GitHub’s Copilot and Augment code to review pull requests, and their remarks and complaints help us to land better code and avoid merging new bugs. I mean, we still merge bugs of course but the PR review bots regularly highlight issues that we fix: our merges would be worse without them. The AI reviews are used in addition to the human reviews. They help us, they don’t replace us. We also see a high volume of high quality security reports flooding in : security researchers now use AI extensively and effectively. Security is a top priority for us in the curl project. We follow every guideline and we do software engineering properly, to reduce the number of flaws in code. Scanning for flaws is just one of many steps to keep this ship safe. You need to search long and hard to find another software project that makes as much or goes further than curl, for software security. Steps involved in keeping curl secure May 6, 2026 It was with great anticipation we received the first source code analysis report generated with Mythos. Another chance for us to find areas to improve and bugs to fix. To make an even better curl. This initial scan was made on curl’s git repository and its master branch of a certain recent commit . It counted 178K lines of code analyzed in the src/ and lib/ subdirectories. The analysis details several different approaches and methods it has performed the search, and how it has focused on trying to find which flaws. A fun note in the top of the report says: curl is one of the most fuzzed and audited C codebases in existence (OSS-Fuzz, Coverity, CodeQL, multiple paid audits). Finding anything in the hot paths (HTTP/1, TLS, URL parsing core) is unlikely. … and it correctly found no problems in those areas. Completely unscientific poll on Mastodon about people’s expectations for Mythos scanning curl The size of curl curl is currently 176,000 lines of C code when we exclude blank lines. The source code consists of 660,000 words, which is 12% more words than the entire English edition of the novel War and Peace. On average, every single production source code line of curl has been written (and then rewritten) 4.14 times. We have polished on this. Right now, the existing production code in git master that still remains, has been authored by 573 separate individuals. Over time, a total of 1,465 individuals have so far had their proposed changes merged into curl’s git repository. We have published 188 CVEs for curl up until now. curl is installed in over twenty billion instances . It runs on over 110 operating systems and 28 CPU architectures . It runs in every smart phone, tablet, car, TV, game console and server on earth. The report concluded it found five “Confirmed security vulnerabilities”. I think using the term confirmed is a little amusing when the AI says it confidently by itself. Yes, the AI thinks they are confirmed, but the curl security team has a slightly different take. Five issues felt like nothing as we had expected an extensive list. Once my curl security team fellows and I had poked on the this short list for a number of hours and dug into the details, we had trimmed the list down and were left with one confirmed vulnerability. The other four were three false positives (they highlighted shortcomings that are documented in API documentation) and the fourth we deemed “just a bug”. The single confirmed vulnerability is going to end up a severity low CVE planned to get published in sync with our pending next curl release 8.21.0 in late June. The flaw is not going to make anyone grasp for breath. All details of that vulnerability will of course not get public before then, so you need to hold out for details on that. The Mythos report on curl also contained a number of spotted bugs that it concluded were not vulnerabilities, much like any new code analyzer does when you run it on hundreds of thousands of lines of code. All the bugs in the report are being investigated and one by one we are fixing those that we agree with. All in all about twenty bugs that are described and explained very nicely. Barely any false positives, so I presume they have had a rather high threshold for certainty. curl is certainly getting better thanks to this report, but counted by the volume of issues found, all the previous AI tools we have used have resulted in larger bugfix amounts. This is only natural of course since the first tools we ran had many more and easier bugs to find. As we have fixed issues along the way, finding new ones are slowly becoming harder. Additionally, a bug can be small or big so it’s not always fair to just compare numbers My personal conclusion can however not end up with anything else than that the big hype around this model so far was primarily marketing. I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing. This is just one source code repository and maybe it is much better on other things. I can only tell and comment on what it found here. But allow me to highlight and reiterate what I have said before: AI powered code analyzers are significantly better at finding security flaws and mistakes in source code than any traditional code analyzers did in the past. All modern AI models are good at this now. Anyone with time and some experimental spirits can find security problems now. The high quality chaos is real. Any project that has not scanned their source code with AI powered tooling will likely find huge number of flaws, bugs and possible vulnerabilities with this new generation of tools. Mythos will, and so will many of the others. Not using AI code analyzers in your project means that you leave adversaries and attackers time and opportunity to find and exploit the flaws you don’t find. Zero memory-safety vulnerabilities found. Methodology note: this review is hand-driven analysis using LLM subagents for parallel file reads, with every candidate finding re-verified by direct source inspection in the main session before being recorded. The CVE to variant-hunt mapping was built from curl’s own vuln.json. No automated SAST tooling was used. This outcome is consistent with curl’s status as one of the most heavily fuzzed and audited C codebases. The defensive infrastructure (capped dynbufs everywhere, with explicit max on every numeric parse, overflow guard, CURL_PRINTF format-string enforcement, per-protocol response-size caps, pingpong 64KB line cap) systematically closes the bug classes that would normally be productive in a codebase this size. Coverage now includes: all minor protocols, all file parsers, all TLS backends’ verify paths, http/1/2/3, ftp full depth, mprintf, x509asn1, doh, all auth mechanisms, content encoding, connection reuse, session cache, CLI tool, platform-specific code, and CI/build supply chain. It should be noted that the AI tools find the usual and established kind of errors we already know about. It just finds new instances of them. We have not seen any AI so far report a vulnerability that would somehow be of a novel kind or something totally new. They do not reinvent the field in that way, but they do dig up more issues than any other tools did before. These were absolutely not the last bugs to find or report. Just while I was writing the drafts for this blog post we have received more reports from security researchers about suspected problems. The AI tools will improve further and the researchers can find new and different ways to prompt the existing AIs to make them find more. We have not reached the end of this yet. I hope we can keep getting more curl scans done with Mythos and other AIs, over and over until they truly stop finding new problems. Thanks to Anthropic and Alpha Omega for providing the model, the tools and doing the scan for us. Thanks also to the individual who did the scan for us. Much appreciated! Top image by Jin Kim from Pixabay Thanks for flying curl. It’s never dull. They can spot when the comment says something about the code and then conclude that the code does not work as the comment says. It can check code for platforms and configurations we otherwise cannot run analyzers for It “knows” details about 3rd party libraries and their APIs so it can detect abuse or bad assumptions. It “knows” details about protocols curl implements and can question details in the code that seem to violate or contradict protocol specifications They are typically good at summarizing and explaining the flaw, something which can be rather tedious and difficult with old style analyzers. They can often generate and offer a patch for its found issue (even if the patch usually is not a 100% fix).

open-source Open Source c C security Security json JSON
0 views
Blargh 2 months ago

Quantum safe amateur radio secure shell

I’ve previously pointed out that the AX.25 implementation in the kernel is pretty poor . It’s not really being maintained, and even when it gets fixes after I reported it , with people running LTS OSs it can take like 5 years before before the fix actually reaches users, if ever. So when writing applications, you still have to work around kernel bugs from a decade ago. This makes it kind of pointless to upstream patches. The exception is security patches, and reading between the lines of why the AX.25 code is now being removed from the kernel , it sounds like maybe some LLM (like the looming “Mythos” and the related Glasswing ) may have found some severe problems. But even if there aren’t any known security problems yet, having code is now more of a liability than ever. Code needs to be removed, or taken responsibility of. (tangent about ffmpeg at the bottom of this post) With the kernel code removed, say goodbye to the old walkthrough . Well, not “new”, per se, but “replacement”. With the socket based API about to be gone, we need some other way for applications to send packets and manage connections. For sending raw packets to and from the modem there’s KISS . I have no real complaints about it. Not much to get wrong about sending frames. It’s implemented by most modems, like the software modem Direwolf and by some radios like the Kenwood TH-D75 , so it’s not going anywhere. For connected mode (streams of in order data, like with TCP) the biggest contender seems to be AGW . Direwolf implements it, and I’ve made a messy implementation of an AGW client in Rust . The Rust API works, as we’ll see, but the code needs some refactoring and cleanup due to it being written exploratorily while I was deciding what it should even do, and how. The AGW protocol is not super amazing, but it gets the job done. One can build a connection API on top of it, as I have , and never have to think about the AGW protocol ever again. There’s another protocol called RHP, specified here and here . It came out of the XRouter project. Since XRouter is closed source, I have a strong aversion to it. It seems both counter to how I see amateur radio, and anachronistic, for it to be closed source. It’s bad enough that VARA and [Winlink][winlkn] are closed source. And people are definitely working on replacing VARA with various other modes because of it. tl;dr: I’m going with AGW for now. If someone writes a Rust crate for RHP exposing a compatible API, I certainly wouldn’t mind adding that dependency to optionally use. I have not yet implemented AGW (or RHP) in my own AX.25 stack , but I plan to. For now that means I’ll use Direwolf. My previous axsh implementation, since deleted , had some problems: So with everything but terminal management needing a rewrite, this is a reason to rewrite the whole thing. Non-requirement: Encrypt — This would violate the amateur radio license. And then, why not just use SSH? If you have an AGW server, such as Direwolf, then it’s easy to run axsh. Just start a server: Then log in: Then wait like 30-40 seconds for the handshake to complete. The reason for the wait is the large ML-DSA signatures used in the handshake. It can’t be the same direwolf instance, since Direwolf only shuffles packets between the radio and AGW clients, not from one AGW client to another. In my case I had one Direwolf connected to an ICom 9700, and another to a Baofeng UV5R using an AIOC (all in one cable) . AIOC is highly recommended for experimentation over the air. So yeah my test is between just about the cheapest VHF/UHF radio that exists, and maybe the most expensive one. In addition to running : With KISS providing packet support (and AGW providing a higher level API on top, if preferred), why not just run TCP/IP, and let the very stable OS TCP implementation take care of everything? TCP is definitely more modern, stable, and maintained, but it doesn’t scale down to slow speeds very well. A TCP+IPv4 header is at least 40 bytes, and if you don’t want to be some sort of caveman, IPv6 is another 20 bytes. At 1200bps that would be 267-400ms overhead for every packet 1 . Checking a random TCP data packet on my laptop I see that with TCP options TCP/IPv4 is actually 52 bytes, or 350ms. Counting the air time (milliseconds, not just bytes) makes this overhead problem more obvious. And because of amateur radio license reasons TCP would still need to identify the callsign, you probably have to add 17 bytes (113ms) as a surrounding header. That leaves TCP with 69 or 89 bytes overhead per packet, meaning 460ms or 593ms. And since you don’t want to tie up the RF channel for too long (only for the whole packet to be dropped due to interference), you won’t want to send packets that are too large. Of course it’s 4x as slow if you want to do something like Bell 103 on HF. AX.25 connected mode takes that down to 19 bytes (126ms) overhead (if using Mod 128 mode) per data packet. Because of the AX.25 segmenter, for bulk data TCP is not as bad as it may have sounded. For a 1500 byte TCP segment, fitting in just under 8 200 byte AX.25 frames (totalling bytes of overhead), this means 1367ms overhead instead of plain AX.25 (at bytes) 1013ms. A 1500 byte payload takes 10 seconds to send, so that’s an overhead of 13.7% instead of 10.1%. But for interactive use cases, worst case a single payload packet, it’s 467ms vs 133ms. And that’s only counting the data frames, not the acknowledgments. A TCP ACK is at a minimum bytes, or 380ms. An AX.25 RR is 18-19 bytes, or 120-127ms. That makes TCP about three times less efficient, compared to AX.25. A bigger problem with TCP, especially untweaked, is resend timers and window sizes. At 1200bps you don’t actually want too big a window size, since you don’t want to tie up the RF channel for several minutes if the other end has gone away. So a bunch of airtime tweaks are needed. And at best you’ll end up with the numbers above. Maybe you could tweak TCP to be more friendly to lower speeds, and find the other overhead acceptable. If so, then you’ll be happy to hear that axsh supports running on TCP as well. Well first, it inherits the same problems from TCP/IP. Sure, the UDP header is smaller than the TCP header, but then on top of that there’s the QUIC header. The second problem is that QUIC is meant to be encrypted. Ripping out encryption, while staying secure, seems more dangerous that keeping it simple and just working from the requirements. Probably the whole handshake would have to be redesigned. AX.25 being removed from the Linux kernel reminds me of LLM finding that bug in ffmpeg , causing all that drama. I have no dog in this fight, but in my opinion ffmpeg is in the wrong, here. Their argument seems to be all about how this particular encoder is rarely used, is just a hobby project, etc.. Ok, but it’s in your code base. Even if disabled by default, why would you want to ship a security footgun? Maybe some hobbyists out there build ffmpeg with all encoders enabled. Do you want them to be vulnerable to someone’s virus? So Google should either keep quiet, or give a patch? Well, keeping quiet because the codec is rarely used is not really an option. That’s borderline negligent and morally culpable, for when someone eventually gets hacked. So Google “should” always provide a patch in these cases? Perhaps, depending on the meaning of the word “should”. Google is rich, so “should” be morally forced to contribute to your software, just because Google (presumably, via youtube) is a heavy user of ffmpeg? Well, that just sounds like the the (non-)problem with open source software (or free software) in general. The license permits use and profit without contribution. If you wanted a tithe then you should have put that in the license. Sounds like you want everyone to be free only to do what you want. That’s not how that works. This is also why I don’t like the AGPL license . It’s not free software if it binds me in your serfdom. Actually, it’s a tiny bit more, because of the occasional bit stuffing ↩ it was implemented in C++, and not only do I prefer Rust, how could I even call something written in C++ “secure”? (a blog post for another day) used the kernel API, so that needs rewriting, used , which proved to be a bit “weird” when interoperating with some other APIs, and used crypto primitives vulnerable to quantum computers. Don’t use kernel AX.25 sockets — this means use AGW. Also work on TCP (mainly for debugging) — This means using an internal framing protocol. Be quantum safe — Use ML-DSA+ed25519 dual signed for authentication of server and client. Be efficient — This means don’t use ML-DSA for per packet signatures (they are huge), at the cost of some quantum safety (see [the README][axsh]). Actually, it’s a tiny bit more, because of the occasional bit stuffing ↩

rust Rust security Security c++ C++ c C
0 views
Susam Pal 2 months ago

I Will Not Add Query Strings to Your URLs

Last evening, a short blog post appeared in my feed reader that felt as if it spoke directly to me. It is Chris Morgan's excellent post I've banned query strings . Chris is someone whose Internet comments I have been reading for about half a decade now. I first stumbled upon his comments on Hacker News, where he left very detailed feedback on a small collection of boilerplate CSS rules I had shared there. I am by no means a web developer. I have spent most of my professional life doing systems programming in C and C++. However, developing websites and writing small HTML tools has been a long-time hobby for me. I have learnt most of my web development skills as a hobbyist by studying what other people do: first by viewing the source of websites I liked in the early 2000s, and later by occasionally getting possessed by the urge to implement a new game or tool and searching MDN Web Docs to learn whatever I needed to make it work. One problem with learning a skill this way is that you sometimes pick up habits and practices that are fashionable but not necessarily optimal or correct. So it was really valuable to me when Chris commented on my collection of boilerplate CSS rules. It helped me improve my CSS a lot. In fact, a few of the lessons from his comment have really stuck with me; I keep them in mind whenever I make a hobby HTML project: always retain underlines in links and retain purple for visited links. I have been following Chris's posts and comments on web-related topics since then. He often posts great feedback on web-related projects. Whenever I come across one, I make sure to read them carefully, even when the project isn't mine. I always end up learning something nice and useful from his comments. Here is one such recent example from the Lobsters story Adding author context to RSS . A couple of months ago, I created a new project called Wander Console . It is a small, decentralised, self-hosted web console that lets visitors to your website explore interesting websites and pages recommended by a community of independent personal website owners. For example, my console is here: susam.net/wander/ . If you click the 'Wander' button there, the tool loads a random personal web page recommended by the Wander community. The tool consists of one HTML file that implements the console and one JavaScript file where the website owner defines a list of neighbouring consoles along with a list of web pages they recommend. If you copy these two files to your web server, you instantly have a Wander console live on the Web. You don't need any server-side logic or server-side software beyond a basic web server to run Wander Console. You can even host it in constrained environments like Codeberg Pages or GitHub Pages. When you click the 'Wander' button, the console connects to other remote consoles, fetches web page recommendations, picks one randomly and loads it in your web browser. It is a bit like the now defunct StumbleUpon but it is completely decentralised. It is also a bit like web rings except that the community network is not restricted to being a cycle; it is a graph that can take any shape. There are currently over 50 websites hosting this tool. Together, they recommend over 1500 web pages. You can find a recent snapshot of the list of known consoles and the pages they recommend at susam.codeberg.page/wcn/ . To learn more about this tool or to set it up on your website, please see codeberg.org/susam/wander . In case you were wondering why I suddenly plugged my project into this post in the previous section, it is because I recently added a dubious feature to that project that I myself was not entirely convinced about. That misfeature is relevant to this post. In version 0.4.0 of Wander Console, I added support for a query parameter while loading web pages. For example, if you encountered midnight.pub while using the console at susam.net/wander/ , the console loaded the page using the following URL: This allowed the owner of the recommended website to see, via their access logs, that the visit originated from a Wander Console. Chris's recent blog post is critical of features like this. He writes: I don't like people adding tracking stuff to URLs. Still less do I like people adding tracking stuff to my URLs. ? Did I ask? If I wanted to know I'd look at the header; and if it isn't there, it's probably for a good reason. You abuse your users by adding that to the link. I mentioned earlier that I was not entirely convinced that adding a referral query string was a good thing to do. Why did I add it anyway? I succumbed to popular demand. Let me briefly describe my frame of mind when I considered and implemented that feature. When I first saw the feature request on Codeberg, my initial reaction was reluctance. I wasn't convinced it was a good feature. But I was too busy with some ongoing algebraic graph theory research, another recent hobby, with a looming deadline, so I didn't have a lot of time to think about it clearly. In fact, everything about Wander Console has been made in very little time during the short breaks I used to take from my research. I made the first version of the console in about one and a half hours one early morning when my brain was too tired to read more algebraic graph theory literature and I really needed a break. During another such break, I revisited that feature request and, despite my reservations, decided to implement it anyway. During yet another such break, I am writing this post. Normally, I don't like adding too many new features to my little projects. I want them to have a limited scope. I also want them to become stable over time. After a project has fulfilled some essential requirements I had, I just want to call it feature complete and never add another feature to it again. I'll fix bugs, of course. But I don't like to keep adding new features endlessly. That's my style of maintaining my hobby projects. So it should have been very easy for me to ignore the feature request for adding a referral query string to URLs loaded by the console tool. But I think a tired body and mind, worn down by long and intense research work, took a toll on me. Although my gut feeling was telling me that it was not a good feature, I couldn't articulate to myself exactly why. So I implemented the referral query string feature anyway. While doing so, I added an opt-out mechanism to the configuration, so that if someone else didn't like the feature, they could disable it for themselves. This was another mistake. A questionable feature like this should be implemented as an opt-in feature, not an opt-out feature, if implemented at all. The fact that I didn't have a lot of time to reason through the implications of this feature meant that I just went ahead and implemented it without thinking about it critically. As the famous quote from Jurassic Park goes: It soon turned out that my gut feeling was correct. After I implemented that feature, a page from one of my favourite websites refused to load in the console. To illustrate the problem, here are a few similar but slightly different URLs for that page: The first and second URLs load fine, but the third URL returns an HTTP 404 error page. The website uses the query string to determine which one of its several font collections to show. So when we add an arbitrary query string to the URL, the website tries to interpret it as a font collection identifier and the page fails to load. That is why, when my tool added the query parameter to the first URL, the page failed to load. Later, with a little time to breathe and some hindsight, I could articulate why adding referral query strings to a working URL was such a bad idea. Altering a URL gives you a new URL. The new URL could point to a completely different resource, or to no resource at all, even if the alteration is as small as adding a seemingly harmless query string. By adding the referral query string, I had effectively broken a working URL from a website I am very fond of. It is also worth asking whether an HTML tool should concern itself with referral query strings at all when web browsers already have a mechanism for this: the HTTP Referer header, governed by Referrer-Policy . That policy can be set at the server level, the document level or even on individual links. The Web standards already provide deliberate controls to decide how much referrer information should be sent. Appending referral query strings to URLs bypasses those controls. It moves a privacy and attribution concern out of the referrer mechanism and embeds it into the destination URL instead. I don't think an HTML tool should do that. There is also a moral question here about whether it is okay to modify a given URL on behalf of the user in order to insert a referral query string into it. I think it isn't. In the end, I decided to remove the referral query string feature from Wander Console. One might wonder why I couldn't simply leave the feature in as an opt-in. Well, the answer is that once I had deemed the feature misguided, I no longer wanted it to be part of my software in any form. The project is still new and we are still in the days of 0.x releases, so if there is a good time to remove features, this is it. But my ongoing research work left me with no time to do it. Finally, when the post I've banned query strings appeared in my feed reader last evening, it nudged me just enough to take a little time away from my academic hobby and devote it to removing that ill-considered feature. The feature is now gone. See commit b26d77c for details. The latest release, version 0.6.0, does not have it anymore. This is a lesson I'll remember for any new hobby projects I happen to make in the future. If I ever load URLs again, I'll load them exactly as the website's author intended. I will never add query strings to your URLs. Read on website | #web | #technology Wisdom on the Web Wander on the Web Broken URLs https://int10h.org/oldschool-pc-fonts/fontlist/ https://int10h.org/oldschool-pc-fonts/fontlist/?2 https://int10h.org/oldschool-pc-fonts/fontlist/?foo

c C web-development Web Development c++ C++ c# C#
0 views
Stone Tools 2 months ago

PipeDream on the Acorn Archimedes

During the "throw everything at the wall and see what sticks" years of home computing, up to around 1995, a lot was thrown and a lot failed to stick. Sometimes clumps would form that appeared to have the combined friction necessary to maintain wall grip, each holding the other up. But, like Mitch Hedberg's observation of belts and belt loops, it was difficult to discern who was helping who stick to what. Take for example, our focus today. We have a completely novel CPU, built by a tiny team of engineers who had never designed a processor before, running a bespoke operating system squeezed out in a rush to meet the shipping deadline of a computer that wanted to carry on the legacy of a system beloved by British schoolchildren, hosting a productivity suite that completely rethought what the term "productivity suite" even meant. Together, they formed a complete computing dead-end. Yet separately, they each achieved life beyond expectations, given their shaky beginnings. Let's start with the hardware, Acorn Computer Ltd.'s follow-up to the famous 8-bit BBC Micro, the Archimedes. Feeling the 16-bit processors of the day didn't deliver enough bang-for-the-quid, they began an investigation into 32-bit processor options. After reading a U.C. Berkeley paper extolling the virtues of the RISC architecture, and seeing firsthand the ease with which chips could be designed, in 1983 Acorn launched the Acorn RISC Machine project to develop the 32-bit brain of their next system. The fruit of that labor, the ARM processor, defined the Archimedes line. Try as they might, Acorn could never crack the home market the way they did education. Still, those ARM CPUs had longevity well beyond the life of the company that commissioned it. Your smartphone likely has ARM in it right now, and Apple's entire current hardware ecosystem is built on its spec. That powerful hardware needed a preemptive multitasking operating system that befit its computing prowess. That was to be ARX , whose troubled development missed the product launch window. In the meantime, so the computer could have something driving it at launch, a stop-gap operating system called Arthur was shipped. It was similar to Acorn's previous BBC Micro MOS (Machine Operating System), with a graphical layer grafted on top; hit F12 and that text interface will peek out from behind the curtain. Over time it was decided that Arthur was doing a bang-up job and ARX was cancelled. Thus was born RISC OS, a cooperative multitasking WIMP (windows, icons, menu, pointer) with possibly the first application "dock" on a home computer. Its mandatory three-button mouse summons an application's current context menu at the pointer location; there are no menu bars whatsoever. Drag-and-drop is embraced as a central file management metaphor, even to save documents. On top of all that, it was the first to offer scalable, anti-aliased font rendering, even if its fonts were a little "off brand." On top of this unique foundation, we have PipeDream . Developer Mark Colton was convinced that the boundaries between word processor, spreadsheet, and database were artificial and could be eliminated. A document should be able to do any of those functions at any time, anywhere on the page, he posited. One might think, "Oh, like Google Sheets ." but PipeDream handles word processing more elegantly. Another might think, "Oh, like Apple Pages " but the spreadsheet and database functions are more robust in PipeDream . This particular balance of the three productivity functions feels unique amongst even its modern peers. Does a productivity suite work better when it's just a single app? Did Colton successfully execute his vision? And where is the Homerton documentary we deserve? (I didn't know Ghost blogging platform forces images to 2000px max; I've revised my design workflow to mitigate this in the future. To make amends for this timeline's illegibility at 2000px, please accept this PDF version) Testing Rig RPCEmu v371 on Windows 11 RISC OS v3.7 1024 x 768 15-bit color 64MB RAM PipeDream v4.13 Let's Get to Work My process when first examining unfamiliar systems is as follows: I do that across a variety of emulators to see which gives me the least grief; I need to be sure I can trust a basic productivity loop. I usually try to give it a go without research, to see how far I can get on pure skillz (with a Z). It's unusual to sit down at what appears to be a computer I understand and be baffled every step of the way. I've heard this system described as "elegant" and "easy to learn." This has me questioning if maybe I'm actually a very dumb person because my impression is "uncomfortable." You know that modern horror story, aka "creepypasta", The Backrooms ? It's a hidden world that co-exists with our own, which can be entered only by clipping through a seam of reality which separates the two. In there, buzzing fluorescents light an infinite maze of featureless, yellow-wallpapered office-style floor layouts. If one were to find a running computer there, I suspect RISC OS would drive it. It's just common enough in its GUI metaphors to feel familiar, and just off-kilter enough to turn that familiarity against you. Liam Proven wrote in The Register , "You will find it very disorienting, especially if all you know is post-1990s OSes." My dude, I've been computing since the 1970s and I find it disorienting. Nothing is unlearnable (I'm dumb, not incompetent), but I genuinely had to work through its manual to acclimate myself. To be clear, I enjoyed the thrill of venturing into the unknown. After all, one of the goals of this blog is to investigate the less-trodden paths in software history. Still, there are times when I feel RISC OS is " having me on." (trying to ingratiate myself with British readers in today's post) I'll start with the three-button mouse. From left to right the buttons are "Select", "Menu", and "Adjust." After weeks working with the system, I still can't figure out what problem the "Adjust" button solves. It's semi-analogous to on modern systems, as when clicking to add/remove elements to/from a set of selected items. Then, sometimes it does something unexpected like, "drag a window by its title bar without bringing that window to the front." Other times it is baffling. a file icon to a new folder location doesn't move the file to the new location. It copies the file. If you want to move the file, you must . Why are we "SHIFT" dragging anything when we have a perfectly good "Adjust" button? Sometimes the "Adjust" button does "opposite" actions. Click a "down" scroll arrow with "Adjust" and it will to scroll up instead. Is that an "adjustment?" What does it even mean, to "Adjust" a mouse click? It seems like it could mean anything , and that's kind of my point. It's unguessable and unintuitive. An interesting UI element (which predates NeXT and Windows 95) is the Icon Tray, an important tool inexplicably not described at all in the RISC OS 3 manual. Situated along the bottom of the screen, currently running applications and directory icons sit on a little shelf. Double-click "Select" on an application icon to launch it and... nothing. Its icon displays in the Icon Tray, and that's it. We must now Single-click "Select" on that icon to actually bring the application to the forefront and activate it. I don't know what that's all about, but that's how it works. Menus are fascinating in both the positive and negative meanings of the word. There are no menus on screen whatsoever, they are only made visible by the middle "Menu" mouse button. "Menu" clicking opens a given menu at the current mouse pointer location. Icons in the Icon Tray can be "Menu" clicked to get application-level menus, like "Make a new document." Within a document, "Menu" click will give us document-level options. Conceptually, I like the "Menu" button a lot. Within a menu, any choices which open dialog boxes or control panels tend to open in-menu. It's kind of cool, being able to type, or flip switches and radio buttons, directly inside the menu itself, rather than popping up a modal window. However, it is jarring to have large panels suddenly lunge out like a xenomorph's inner jaws when scrolling through menus. These can obscure the root menu, depending on screen position. 0:00 / 0:08 1× The last point to get our collective heads around is file saving. When saving a new document, simply typing in a file name is not sufficient. Save dialog boxes expect and require the full path to your save destination; no assumptions or default folder locations are provided. You can manually type in the full path to your desired save location like this: While you type, the system will not assist you in navigating the directory structure; no autocompletion here. You must know the path by heart. The other option, as described in manuals, is to drag-and-drop your document to its save location. Drag-and-drop really seems to be the RISC OS idiomatic way to manipulate files. In a Save dialog box there is a little icon for the application. It looks like decoration, but it physically represents your document. Type a name into the text field, then drag that icon to your desired save folder. 0:00 / 0:13 1× I don't want to get bogged down enumerating RISC OS's idiosyncrasies, but a few more things need mentioning. There is a kind of "programmer's art" ugliness to the user interface; those folder icons are terrible. There are graphical glitches, as when scrolling a window too quickly (though moving windows around shows full contents, which wasn't typical during that period). Everything you set up to customize the system, like desktop icons, window positions, desktop resolution, and other settings is reset every boot unless you manually tell the system to save the current state as the "boot file." The list goes on like that. Sheesh, what a journey just to understand the basics. I expect that kind of learning curve for the text-based systems, as those DOS-like commands are unknown to me. For a GUI system to throw this "spanner in the works" (continuing my pandering) is unexpected, but a fun challenge. I can't feel myself growing to love it, but the initial feeling of discombobulation is receding. A spreadsheet is an ordered matrix of cells, each of which can hold text or math. Cells with text are typically used as labels for columns and rows of numbers, and the math cells do the work of calculating relationships between those numbers. It's all very simple. No, wait, I mean it's "easy-peasy." (commitment to the bit) Lotus 1-2-3 felt "columns and rows" could also be useful for textual data. They said the line between spreadsheets and databases is pretty fuzzy, and even today spreadsheets are used to hold and manipulate simple databases. Then racecar driver Mark Colton pierced the veil entirely. It wasn't just spreadsheets and databases that had a fuzzy separation. If we can type arbitrary text into a cell in a spreadsheet, why couldn't we type an entire book? What if all applications were really just one application, in the end? He fired his first shot at uniting everything in View Professional . This was released as PipeDream on the Cambridge Z88, a portable Z80 machine by Sir Clive Sinclair's Cambridge Computer. Built into the ROM itself, it was insta-boot, insta-launch right into a multi-purpose integrated document suite. Jerry Pournelle, in BYTE Magazine 's February 1989 issue, was moderately enamored with the hardware, but PipeDream was, "disappointingly hard to use." With Acorn evolving their BBC Micro via the Archimedes, Colton continued to support their hardware line. In interviews, he seemed to really be leaning toward Windows for the future of his company. However, since he switched development to C and there was a C compiler for the Archimedes, he said it wasn't hard to provide his product to the Acorn crowd. Running on Arthur, the precursor to RISC OS, he embraced and extended the "one document, many forms" approach. Much like today's Google Sheets, we can add arbitrarily long sections of text, insert images, set up database information, perform spreadsheet calculations, run spellcheck, and generate inline graphs. However, try typing a chapter of a book into Google Sheets if you want to drive yourself "mental." (there's no stopping me) In PipeDream , that's frictionless (within a certain definition of "friction"). Like RISC OS itself, PipeDream also requires certain shifts in thinking to not lose a finger to its sharp edges. I suppose that when a developer offers a truly new paradigm, it is fair to ask users to meet it halfway. I'm not convinced the advertising (see "Historical Record" at the end) gave customers a full understanding of how drastic that shift was. "Menu" click the Icon Tray icon (i.e. the application-level menu) for PipeDream to start up a new "Text" file and begin typing into cell A1. You'll find that text overflows, across cell boundaries, until it hits the "row wrap marker" seen in the rightmost column header (shown as a "down arrow" icon). Every line of text is its own row, in spreadsheet terms. As you type, PipeDream fills the current row, then silently inserts a new row to catch overflow. Until a paragraph break, these rows are internally associated as a logical unit. Edits which alter or disrupt text flow across rows within a paragraph are not reflected immediately in the UI. Or maybe they are? It's hard to tell with the graphic glitches in the screen redraw, a constant source of frustration while working on this article. PipeDream concedes the reflow point itself. When in doubt about the current visual structure of your text, , a manual action, will force PipeDream to recalculate text wrapping and line spacing. This can be mitigated a bit through a hidden toggle in the "Options" screen, the confusingly named "Insert on Return." This reduces the need to force a manual reflow, but can still leave visual chaos. 0:00 / 0:44 1× I've altered the text flow and initiated a recalculation of the lines. It does the work, but visually shows no change until I trigger a graphics refresh in some way. Selecting the text works, but then leaves its own graphic artifacts behind. I've "gone nutter!" (yes, these are in the captions as well!) Interestingly, I saw similar redraw issues in View Professional on the BBC Micro. It would appear this is, to some extent, part of the software's DNA. Honestly, this is all "a bit of a shambles." (the hits keep coming) Have you ever wanted a word processor that won't indent paragraphs? PipeDream being a chimera, navigation idioms are forced to choose which parent they love most. An examination of the key demonstrates this. In a word processor, we usually have a horizontal page ruler with tab stops. Tab over to a tab stop and type to align text at that indentation point on the page. In a spreadsheet, navigates us to the next cell to the right. In PipeDream , the spreadsheet idiom wins TAB's love. In a text cell, sets an invisible indicator at paragraph start which forces every subsequent line of that paragraph to begin at that same column. For example, by default every line of text is added to column A, the leftmost. If we to column B, the text will start there but when it wraps to the next line, that will also begin in column B. "Indentation" is at the paragraph level, not the line level. How do we indent the first line of a paragraph? The manual has a solution. In looking back through the history of Colton's software on the Acorn line, I found this note in a review of View 2.1, his standalone word processor for the BBC Micro. "Why is there no numerical information on the rulers or cursors to assist formatting?" asked Acorn User , January 1985. It seems Colton had it in for rulers for a decade, and to my thinking this points to a disconnect between what a programmer thinks users need, versus what users actually need. A stubborn rejection of norms doesn't always mean we're on the right track. We can use the cell-based layout engine of the program to pull off a fun party trick. Under "Options" there is a toggle between Row and Column text wrap. "Row" behaves like a typical word processor. "Column" lets us divide the page into columns, like a newspaper. Tab between columns and the column width will be respected by the word wrap. Kind of cool, and could be useful in a "I need to make a newsletter, stat!" pinch. Like a spreadsheet, column widths are document-wide, so no mix-and-match. Someone very clever with the tools could probably coax complex layouts out of it, but that would require an ungodly amount of pre-planning, design, and patience before starting a document. You really have to try to get it right the first time, because I don't find PipeDream particularly adept at handling large structural changes after the fact. The column-based formatting gets frustrating, but in other ways the word processing is "bog-standard." (How many will I squeeze in? Place your bets!) We have a built-in spell check, user-definable dictionary, word count, text alignment, font choices, and an anagram/subgram maker. Bank Street Writer Plus had an anagram maker as well. Why was that such a thing back then? Have I forgotten some fad of the 80s and 90s? That's all fine and dandy, but I'll tell you what isn't: there's no simple cut/copy/paste, at least not as a modern audience may understand those tools. In the document, we are restricted to cell-level selection, meaning I can't select individual words inside cell A1. I can only select the entire cell A1, which in PipeDream means an entire line of text. We can ask PipeDream to edit a cell in its own window, where it pops out for surgical editing. "Edit Formula in Window" highjacks the spreadsheet formula editor in order to get character-level selection control. In this pop-out window, we can highlight individual words and do typical cut/copy/paste actions. Notice, though, we're still restricted to only the text within the cell, which means only that line (row) of text. It's highly likely any given row will contain the tail-end of the previous sentence and the first part of the next sentence. If we want to cut out a specific sentence which doesn't align neatly to the row structure, there is no way to do so. I will repeat that. There is no way to cut/copy/paste an arbitrary string of characters. Now I feel PipeDream's vision working against itself for anything but simple correspondence. Remember, this is version 4 of PipeDream, Colson's fifth software release to pursue this unified application dream, and this is where we're at. I can't imagine writing anything substantial within these frustrating limitations. As a spreadsheet, PipeDream performs far more admirably, even if certain conventions have been eschewed in favor of its new vision. Hey, if you're gonna quirk it up, might as well go for broke. Unlike its spreadsheet ancestors, there is no menu, nor is there a simple way to tell PipeDream that we want to enter a formula into a cell, as with to denote a function call, or to indicate we want to do math. Many of Lotus 1-2-3's innovations have been utterly ignored. The global "Options" allows us to set default behavior for cell entry. Setting it to "numbers" will put us into the right context for easy formula entry, or we can click into the ever-present formula entry line at the top of the window. Turn on the "Grid" overlay to draw cell boundaries and before you know it what was a word processing document is now a spreadsheet with "the full Monty." (TIL it doesn't mean "full-frontal nudity") The functions available to number crunchers are plentiful and robust. Trigonometric functions are a given, but its inclusion of matrix math may come as a surprise. Even complex functions like , which computes "the complex hyperbolic arc cosecant of as a complex number," are present and accounted for, so hardcore math nerds can breathe a sigh of relief. A wide number of financial functions, statistical functions, lookup tables, string manipulations, and date handling are all here. So too are flow control tools, like , , and more. There are even GUI controls available for showing error dialog boxes and prompts for user input, though those are only available from within custom functions. Yes, if you're missing a function, you can make your own. In a new worksheet, start a formula with (which can accept typed parameters) and end it with . In between, do the work. PipeDream will check syntax and accept or reject each line of your function. If accepted, it will prefix a line with In your real working worksheet, access the formula by . That file reference implies PipeDream can access data from other worksheets, and that is true. Even a cell reference in a formula can be pulled from a completely different worksheet. I find the syntax for custom functions opaque, and the manual does a poor job of explaining what is possible and how to use the tool. There are a handful of examples provided with the software installation, with bugs, that reveal secrets only upon very close inspection. For example, notice in the screenshot above that the parameters to the function are later referenced by prefix, but local variables, as set by the function are not prefixed when used in calculations. It's those subtle little things that tripped me up. The same with having the return value called . Or how the program has a selection of "Strings" functions, but when passing a string as a parameter its type is "Text." I stared at that syntax for a LONG TIME before finally realizing my various little misunderstandings. Customization doesn't stop there. Individual keys can be defined as shortcuts to longer string sequences, F-Keys (plain and modified) can be defined to trigger commands, and command sequences (triggered by the CTRL key) can be redefined to your liking (which risks overwriting built-in command shortcuts). You really can make PipeDream your own, though you're in for a struggle compared to Lotus 1-2-3 and the thousands of books available to help learn its principles. I found no actual books for PipeDream , just publishing announcements in old magazines. Something must exist, but the internet at large appears bereft. On the scorecard of "this amalgamation approach to productivity software is working," I'd say we're 1 and 1. The spreadsheet tools are fiddly, but robust. The word processing has me very underwhelmed. Time for the tie-breaker: databases. Using the supplied Lotus 1-2-3 conversion tool, I was able to bring in the data I originally created in CP/M dBASE II and had subsequently converted to DOS Lotus 1-2-3. Now it lives on in RISC OS PipeDream . This data has more passport stamps than Indiana Jones. Let's consider some of the basic things one might want to do with data. PipeDream beats out Lotus in sorting, giving us a five-stage, multi-row, sort with ascension. Not too shabby for the time, all things considered. Search and replace does what it "says on the tin" (in for a penny, in for a pound), and can also accept regex-like tokens and patterns. More interestingly, cells can be set up to directly perform queries on table data. There are a small handful of prefixed database functions to calculate averages, min/max, counts of things, and more. One last feature of note is how to use the query tools to extract a result into a new database. This is interesting as it utilizes RISC OS's drag-and-drop Save functionality in a clever way. 0:00 / 0:15 1× Note how the query for data extraction is much longer than the tiny little text field in the contextual menu can handle elegantly. This is one of those usability tradeoffs for the RISC OS way of doing things. I was initially ready to write off the database functionality as being underwhelming, until I reminded myself of the stated goal for PipeDream . Its core proposition is that there is no difference between the various aspects of the software. The word processor is the spreadsheet is the database. We're not limited to the "database" functions when manipulating our database data. We have access to everything the program has to offer, at all times. Let's clip through the inverted UV plane separating database and spreadsheet, and see what kind of trouble we can get into. I'm thinking back to the Lotus 1-2-3 article and how database information was queried there. With a table of data, we had to use the built-in query forms, define areas on the sheet to hold query parameters, and designate another section of the sheet into which query results would display. It was an obtuse Rube Goldberg machine that I couldn't understand until I drew a diagram of the process. In PipeDream , we just write a formula, the same as if it were a spreadsheet. Let's get the average rating of all adventure games in the database published before 1985. "Bob's your uncle!" (I was hoping to work that one in) Let's mix it up a little and get the same average, but only for titles which begin with "Zork." We can use wildcards, but let's leverage PipeDream's word processing string tools. The most awesome part about this is that, like any spreadsheet formula, it updates in real time. Change the ratings, or add a new Zork game to the mix, and get the new average instantly. The database is the spreadsheet is the database, so that calculation can then be referenced as a value for another cell's formula, perhaps adding sales tax to the average unit price. While we're at it, might as well throw in some fancier text formatting to make it look pretty. In the Lotus 1-2-3 investigation, I wanted a pie chart showing a breakdown by game categories. Lotus had a handy function which removed duplicates from lists, making it possible to extract the full list of unique game categories, which could then be used as the query parameters for generating a chart. PipeDream can't do that, but it does have other string parsing routines, variables, cross-file data referencing, and the ability to write custom functions and macros. I don't doubt it would be possible to homebrew a workaround to this missing function. In fact, let's "have a bash at it." (swish!) 0:00 / 0:12 1× Note the real-time update of the chart as I modify an external database. Ultimately, I couldn't achieve an elegant solution, but I could achieve my goal. I sorted the original data by genre, then created a column that checks if the genre for each row matches the one above it. If so, it's a otherwise a . Then, I extracted all rows with in the column. Last, I did (count any items in a list), where the source list is contained in the original database document. With the documents thus linked, I get real-time graph updates when I alter the core database, thanks to external reference handling. Everything's "tickety-boo!" (I'm trusting The Independent on this one) OK, PipeDream , you're winning me over a little more now. Time to take this to its logical conclusion. We haven't yet pushed it as the multi-purpose document creation tool it promises to be. We've done a little dabbling, with text formatting and data extraction, but I want to see everything come together. I want the borders to crumble . The approach I'm finding to be least troublesome is to begin with a "text" document, then decorate that with spreadsheet/database elements. 0:00 / 0:16 1× As I scroll, text will disappear until I trigger a redraw event in the window. (pay no attention to the content of the letter) In building that document, here's what I learned. We have a unique confluence of interesting technologies coming together to form a strangely flawed jewel. It sparkles and shines when the light hits it just right , and in those sparkles we may catch a fleeting glimpse of a world that might have been. Might have been, but wasn't . Let's see where each of the underlying technologies wound up and those in the know can feign shock with the rest of us when we learn that ARM isn't the only thing that survives to this day. We'll start with the obvious truth: ARM won. It's in everything, everywhere, all at once. If it isn't in your computer, it's in your phone, or your Newton, or your Palm Pilot, or your Canon camera, or your Nintendo DS, or your Nintendo 3DS, or your Nintendo Wii, or your Nintendo Switch, or your Nintendo Switch 2, or your Raspberry Pi, or maybe you're sidetalking on your N-Gage. Its combination of low power consumption with high performance makes it ideal for mobile devices, of which we are in abundance. But why ARM specifically? Others have swung for the RISC fences and stumbled, yet Acorn set two engineers to the task of designing their first ever microprocessor and somehow achieved a ubiquity that has remained (mostly) unchallenged. Apple/IBM/Motorola gathered their forces and developed their own RISC architecture, which debuted in Apple's Power Macintosh 6100. PowerPC doesn't mean much to a Windows/Intel crowd, but the Mac faithful remember all too well Apple's investment in that as the successor to the x68000. Frustrated by delays in the evolution of the chip line, Apple wound up ditching it for Intel x86 , even if they eventually rediscovered the joys of RISC. PowerPC went on to be adopted by a number of game consoles, notably the Nintendo Wii, XBox 360, and PS3 simultaneously. The line continues today, and heck, Mars rovers Curiosity and Perseverance both have PPC inside. Hard to call such a history a "failure," but who outside hardcore Amiga faithful today is clamoring for a PowerPC chip? The SPARC RISC architecture, of "Sun SPARC Workstation" fame, chugged along until as late as 2017, when Oracle purchased Sun. A notable achievement, in pop culture circles, is this is the hardware Pixar's first Toy Story was rendered on . Though Oracle disbanded the design team keeping the architecture alive, the architecture itself is free and open source. There's nothing stopping an intrepid reader from carrying on the lineage, I suppose. Fujitsu, the last of the production line for the series, has abandoned SPARC for ARM. I'll be honest, I can't figure out what ARM does so much better than other attempts, like SPARC, at making a great RISC processor. Reading through the Ars Technica story , it seems to be less about the underlying tech and more about the savvy promotional work of Robin Saxby and his absolute unwillingness to lose the RISC wars. Where others were building RISC for the server-side, ARM committed themselves to the mobile side, skating to where the puck would be . Whatever the case, whatever the magic, ARM makes it available to anyone who wants it, through their licensing partnerships. Ultimately, this really seems to be what has given ARM its staying power; a low barrier to entry to quickly join in on high-performance, low-power draw, ARM fun. It's important to note that ARM doesn't make processors; they only license their IP. <<record_scratch.mp3>> OK, be that as it may, it is still substantially correct to say that IP licenses are their bread and butter. A "core license" allows a company to manufacture a specific ARM-designed CPU, a popular choice for system-on-a-chip designs. Alternatively, an "architectural license" permits a company to design and build its own custom CPU around the ARM instruction set. That's what Apple does with their A- and M-series chips. In recent years, ARM is feeling light competitive pressure from the RISC-V architecture. Born in the same UC Berkeley labs that birthed the original RISC design reports that inspired Acorn to take a chance on RISC, its architecture, unlike ARM, is free and open source. Consumer-level devices running on RISC-V have already started shipping. A new race has begun. Acorn's Archimedes line ultimately never sold particularly well. It's hard to nail down specific sales figures , but a 1991 Acorn shareholder report said, "Acorn is now the UK number one supplier of 32-bit RISC machines with an installed base of over 150,000 units." For context, the Amiga line had sold some 2 million units by 1991. We can't say Acorn didn't put in the effort, releasing some 13 model variations in under a decade. The general consensus seems to be that they "cost a bomb." (that's a new one on me) Schools adopted them, as a natural evolution of Acorn's prior BBC Micro installations, but at US$3,000 to $9,000 (in 2026 money) families just couldn't afford to put one in the home. In the mid-90s, Acorn dropped the Archimedes line, switching tracks to the more business-like Risc PC line, and produced a handful of systems around the StrongARM CPU. However, while the CPU spirit was willing, the motherboard flesh was weak, leaving the CPU underutilized . The lineup ended concurrently with the end of Acorn around 1998. Castle Technology tried to keep the Risc PC line going, post Acorn, but called it quits shortly thereafter, in 2003. Open-sourced in 2018, RISC OS Open keeps it running and up to date for modern RISC-based hardware platforms, especially the Raspberry Pi. Currently at v5.30 at the time of this writing, it is still a 32-bit operating system with " moonshot" aspirations of 64-bit someday. * checks watch* Time is ticking to pull that together before fading into 32-bit irrelevance. Did I mention how tiny this thing is? The latest version for Raspberry Pi is a 155MB download. Version 3.7, which I used for this article, downloaded as a pre-configured emulator with OS and apps pre-installed, was a mere 129MB. Even the most up-to-date pre-configured package tops out at a "massive" 1GB, apps and emulator inclusive. How big is macOS on ARM? Leading in with his View lineup of productivity apps on the BBC Micro, Mark Colton was the man with the all-in-one vision. With View Professional , he took his first stab at providing an uber-app for that 8-bit workhorse. It's primitive and clunky to use, but the spark is present. He would then expand on his ideas through the PipeDream lineup, taking it all the way to version 4.5. Every version refined the vision, but ultimately its character-based layout engine roots became a limiting factor to its growth. One rewrite later, he had a true GUI-based implementation, for both the Archimedes and Windows, in Fireworkz released in 1993. Having created standalone products Wordz and Resultz, Fireworkz combined those back into one. By mid 1995, Fireworkz Pro added in the database functionality, merging the new Recordz into the product, and that's where Colton's involvement ended. Besides asking "What even is a spreadsheet anyway?" Colton's other passion was race car driving. In August 1995, an engineering defect in the front wing of his Pilbeam M72 caused it to fold under his car while he was at top speed. He lost control, crashing headlong into a telegraph pole, and was killed. Most shockingly, both PipeDream and Fireworkz continue to be maintained to this day. Mark's father, Richard, generously open-sourced both PipeDream and Fireworkz just before his own untimely death in 2015. Fireworkz Pro, the version that includes database functionality, is not open-sourced and is still for sale . The PipeDream package available for installation in RISC OS package manager is not the version I'm using for this article. That is the modern update, which adds a bunch of niceties, including a GUI toolbar for formatting text, expanded spreadsheet functions, and a mind-boggling number of bug fixes. This is all maintained by lone developer Stewart Swales , someone intimately involved in the RISC OS and PipeDream history. He worked at Acorn and helped develop Arthur, the OS that became RISC OS. Later, he joined Colton Software as lead developer, working on PipeDream and Fireworkz. There's really nobody better to carry on the legacy. Where, precisely, Colton's continuation of that legacy would have gone, we can't say with certainty. However, we do have a little insight into his thinking. In an interview with Acorn User , December 1994, he said, "Over the next few years...we won’t be writing spreadsheets either; we'll be writing a totally different style of program. I expect spreadsheets, word processors and so on to be provided as part of the operating system in the future." Let me start by making it clear that I appreciate the effort. I say that with all sincerity and for everyone involved. From the machine, to the OS, to the productivity suite, all katamari'd up into a unique star. It was a lot of fun feeling like a beginner again. I had moments of true learning, shedding expectations of "how things should be" and experiencing fresh, alternate ways to approach work. I said at the beginning, the question that needs answering is, "Did Colton successfully execute his vision?" and here I must waffle. From View Professional , through five major releases of PipeDream , and two Fireworkz releases, he held fast to a very particular line of exploration. That he never wavered in his pursuit of that vision, says to me that he must have felt he had achieved his goal to some degree. In that regard, we can say he successfully executed his vision. As an end-user, it is hard to align myself to that vision. I get what he's after, especially when trying to make sure documents always reflect the latest data. After using PipeDream for a number of weeks, I remain unconvinced that the solution is to graft all software into one uber-application. If we follow that thinking to its logical conclusion, then why not include paint features? Why not include robust desktop publishing features? Where would it stop? Had the amalgamation of these productivity apps birthed something uniquely unachievable by other means, or unlocked some latent potential in the individual apps, I'd be very willing to adapt to this "skew-whiff" (last one, I promise!) approach to application design. As it stands, I ultimately don't see what it does that wouldn't be equally well-served, perhaps better-served, by intelligent file link management with robust publish/subscribe functionality. In fairness, a deep implementation of that would work best as an OS-level feature, and Colton could only control his own works. Paradoxically, the most frustrating aspect in removing the barriers between applications is how we wind up with a slate of new barriers forged in that alliance. Colton said of View Professional that even when the apps are combined, none should feel like a compromised version of that app. Yet, compromises are what I feel with every document I build. Is it worth giving up easy text formatting and basic cut/copy/paste for the off-chance I might need to insert a little spreadsheet table? There's an 80/20 rule being almost willfully ignored here. I love that Colton had a unique vision and stuck to it. I love that someone tried to forge a new path in productivity application design. I love that PipeDream exists, but I don't love it . Ways to improve the experience, notable deficiencies, workarounds, and notes about incorporating the software into modern workflows (if possible). Testing Rig RPCEmu v371 on Windows 11 RISC OS v3.7 1024 x 768 15-bit color PipeDream v4.13 boot the system launch my application of interest make a dummy document quit the emulator entirely and reboot load my saved document Because rows and columns are shared throughout the document, insertions and deletions, or moving things around, creates difficult-to-resolve layout issues. If a spreadsheet sits to the right of a block of text, and we want to insert a row into only the spreadsheet part, that's not possible. Doing so will also insert an empty row into the paragraph, leaving a gap. PipeDream has a strange concept of "global font" vs. "local font". Local fonts can't be changed until the global font is set to something other than the system font. The global font controls value cells, which cannot be styled individually. Local fonts will style a cell from wherever the cursor is currently located, and it is very easy to target a cell and style its font, but miss the first character or two, even though the entire cell is highlighted as a selection. "What will be the result of my action?" is not always crystal clear. The controls for styling charts are difficult to understand, and messing up is hard to reverse out. I accidentally added "New Text" to the chart and it took a long time to figure out how to delete it; selecting it and hitting "delete" doesn't work. There is no way to modify the legend. There's no facility for selecting elements for inclusion/exclusion from the graph. In my case, formatting to look good on the printed page meant adding empty columns which wound up in the pie chart. This is very representative of the struggles the layout engine introduces. Making data look good in one context risks "making a shambles of it" (are these working? have I won you over?) in another. Page layout settings are cryptic. Margins can only be set to the top and left (?!?!) and only in unspecified numeric units. I used the template default values, and the page wound up shifted down and to the left. Getting beautiful output is a challenge. How could I forget? There's no UNDO! Some programs, like !Draw (vector illustration) and !Edit (text editor) have undo, and others like !Paint and !PipeDream do not. Getting started with RPCEmu , using a pre-built package, was as dead simple to use as you'd imagine. I experienced no crashes of the emulator, operating system, or PipeDream . It was a very solid experience in that regard. PipeDream itself, at least the version I used, had a ton of annoying bugs and the graphical glitches were even noted in a review by Micro User , February 1992 . But emulator-wise, everything was smooth. I recommend first-time users grab a pre-built image for quickly jumping in and seeing what the fuss is all about. I also do recommend going through the RISC OS Manual. The operating system is almost unusable until you learn its little tricks and nuances of operation. Pre-built images: https://www.marutan.net/rpcemu/easystart.html v3 Manual: https://archive.org/details/ro-3-user-guide v5 Manual: https://archive.org/details/risc-os-5.28-user-guide Technically, I am cheating a bit in this review. RPCEmu doesn't emulate an Archimedes but rather Acorn's later Risc PC. I ran PipeDream from floppy in Arculator, which explicitly emulates Archimedes systems, to compare the experiences. Except for RPCEmu's snappier performance (which I want anyway), RISC OS itself abstracts away the hardware layer so much it didn't seem to matter one emulator over the other. The emulator itself expects some specific keyboard, with the key situated between and . I don't have that, and nothing on my extended keyboard would send the right code to the emulator. is used for logical in PipeDream data queries; I had to use Windows ALT keycodes. I mentioned earlier, but I'll make it explicit here: there is no undo. Fireworkz is available as a native Win32 app. It launches without issue on Windows 11 64-bit, and even in Wine on macOS. It looks and feels exactly like Fireworkz on RISC OS, which looks and feels a lot like the latest version of PipeDream (minus the database parts). The list of bug fixes and quality of life enhancements is vast. Scrolling through all changes since Colton passed is kind of pointless due to its scope. I'll say, "a lot has improved" and leave it at that. As a local-only alternative to the Google/Apple/Microsoft hegemony, it's worth checking out. It's free, open source, actively maintained, a mere 2.5MB download, and for God's sake at least it's trying to do something different. Getting documents out of RISC OS into a modern system is easy, but has its caveats. RPCEmu can directly save to the host operating system, so getting files out is a non-issue. PipeDream's options for saving documents will strip the document's uniqueness, however. Saving as ASCII will try to keep text precisely as shown in PipeDream, inserting line breaks at the end of every line of text. Tables are just tab-indented. Any text formatting, fonts, graphs, etc. are stripped, of course. Saving as "Paragraph" is like ASCII, but will keep text together as logical paragraphs. This is much better for pasting the text into new documents. We still lose anything done to make the document look pretty. PDF printing is an option in RISC OS, and proved to be the best way I could find to get PipeDream documents into the real world. This required two parts: activating the PDF printer and running a separate !PrintPDF application. With both active, PipeDream generated PDFs without issue.

c C hardware Hardware c# C# Bash
0 views
Corrode 2 months ago

Bugs Rust Won't Catch

In April 2026, Canonical disclosed 44 CVEs in uutils, the Rust reimplementation of GNU coreutils that ships by default since 25.10. Most of them came out of an external audit commissioned ahead of the 26.04 LTS. I read through the list and thought there’s a lot to learn from it. What’s notable is that all of these bugs landed in a production Rust codebase, written by people who knew what they were doing, and none of them were caught by the borrow checker, clippy lints , or cargo audit . I’m not writing this to criticize the uutils team. Quite the contrary; I actually want to thank them for sharing the audit results in such detail so that we can all learn from them. We also had Jon Seager, VP Engineering for Ubuntu, on our ‘Rust in Production’ podcast recently and a lot of listeners appreciated his honesty about the state of Rust at Canonical. If you write systems code in Rust, this is the most concentrated look at where Rust’s safety ends that you’ll likely find anywhere right now. This is the largest cluster of bugs in the audit. It’s also the reason , , and are still GNU in Ubuntu 26.04 LTS. :( The pattern is always the same. You do one syscall to check something about a path, then another syscall to act on the same path. Between those two calls, an attacker with write access to a parent directory can swap the path component for a symbolic link. The kernel re-resolves the path from scratch on the second call, and the privileged action lands on the attacker’s chosen target. Rust’s standard library makes this easy to get wrong. The ergonomic APIs you reach for first ( , , , ) all take a path and re-resolve it every time, rather than taking a file descriptor and operating relative to that. That’s fine for a normal program, but if you’re writing a privileged tool that needs to be secure against local attackers, you have to be careful. Here’s the bug, simplified from . Between step 1 and step 2, anyone with write access to the parent directory can plant as a symlink to, say, . Then follows the symlink and the privileged process happily overwrites with whatever happened to contain. The fix uses : The docs for say (emphasis mine): No file is allowed to exist at the target location, also no (dangling) symlink . In this way, if the call succeeds, the file returned is guaranteed to be new. A in Rust looks like a value, but remember that to the kernel it’s just a name. That name can point to different things from one syscall to the next. Anchor your operations on a file descriptor instead. only helps with that when you’re creating a new file. For everything else, open the parent directory once and work relative to that handle . If you act on the same path twice, assume it’s a TOCTOU (Time Of Check To Time Of Use) bug until you’ve proven otherwise. This is a close relative of TOCTOU. You want a directory with restrictive permissions, so you write something like this. For a brief moment, exists with the default permissions. Any other user on the system can it during that window. Once they have a file descriptor, the later doesn’t take it away from them. Reach for and so the file or directory is born with the permissions you want. The kernel will apply your on top, so set that explicitly too if you really care. The original check in was literally this: That comparison is bypassed by anything that resolves to but isn’t spelled . So , , , or a symlink that points to . Run and see it rip right past your check and lock down the whole system. Here’s the fix : resolves , , and symlinks into a real absolute path. That’s a lot better than string comparison. Oh and if you were wondering about this line: I think that’s just a fancy way of saying In the specific case of , this works because has no parent directory, so there’s nothing for an attacker to swap from underneath you. In the more general case of comparing two arbitrary paths for filesystem identity, however, you’d want to open both and compare their pairs, the way GNU coreutils does. (Think identity, not string equality.) By the way, my favorite bug in this group is CVE-2026-35363: It refused and but happily accepted and , then deleted the current directory while printing . 😅 Rust’s and are always UTF-8. That’s a great choice in 99% of all cases, but Unix paths, environment variables, arguments, and the inputs flowing through tools like , , and live in the messy world of bytes. Every time a Rust program bridges that gap, it has three options. The audit found bugs in both of the first two categories. Here’s an example. This is the original code, from . GNU works on binary files because it just shuffles bytes around. The uutils version replaced anything that wasn’t valid UTF-8 with , which silently corrupted the output. Here’s the fix: stay in bytes. forces a UTF-8 round-trip through . does not. It writes the raw bytes directly to . For Unix-flavored systems code, use and for filesystem paths, for environment variables, and or for stream contents. It’s tempting to round-trip them through for easier formatting, but that’s where the corruption creeps in. UTF-8 is a great default for application strings, but it’s absolutely, positively the wrong default for the raw byte stuff Unix tools work with. In a CLI, every , every , every slice index, every unchecked arithmetic operation, every is a potential denial of service if an attacker can shape the input. That’s because a unwinds the stack and aborts the process. If your tool is running in a cron job, a CI pipeline, or a shell script, that means the whole thing just stops working. Even worse, you could find yourself in a crash loop that paralyzes the entire system. A canonical case from the audit was ( CVE-2026-35348 ). The flag reads a NUL-separated list of filenames from a file, but the parser called on a UTF-8 conversion of each name: GNU treats filenames as raw bytes, the way the kernel does. The uutils version required UTF-8 and aborted the whole process on the first non-UTF-8 path: (I reproduced this against on macOS. The Python one-liner is there because most modern shells refuse to create a non-UTF-8 filename for you.) Your nightly cron job is dead and there goes your weekend. In code that processes untrusted input, treat every , , indexing, or cast as a CVE waiting to be filed. Use , , , , and surface a real error. Push back on the boundary of your application and let the caller deal with the fallout. A good lint baseline to catch this in CI: These are noisy in test code where panicking on bad data is exactly what you want. The cleanest way to scope them to non-test code is to put at the top of each crate root, or to gate on the individual modules. Closely related to the previous point, a few CVEs come from ignoring or losing error information. and returned the exit code of the last file processed instead of the worst one. So could fail on half the files and still exit . Your script thinks everything is fine. called on its call to mimic GNU’s behavior on . The intent was reasonable, but that same code ran for regular files too, so a full disk silently produced a half-written destination. The reason was that someone wanted to throw away a and reached for , , or . Here’s a very simple pattern to avoid that: Also, if you write to discard a , leave a comment that explains why this specific failure is safe to ignore. A surprising number of these CVEs aren’t “the code does something unsafe” but “the code does something different from GNU, and a shell script somewhere relied on the GNU behavior.” The clearest example is (CVE-2026-35369). GNU reads as “signal 1” and asks for a PID. uutils read it as “send the default signal to PID -1”, which on Linux means every process you can see . Yikes! A typo becomes a system-wide kill switch. If you reimplement a battle-tested tool, bug-for-bug compatibility on exit codes, error messages, edge cases, and option semantics is a security feature. (Hello, Hyrum’s Law – and obligatory XKCD 1172 !) Anywhere your behavior diverges from the original, somebody’s shell script is making a wrong decision. uutils now runs the upstream GNU coreutils test suite against itself in CI. That’s the right scale of defense for this class of bug. CVE-2026-35368 is the worst single bug in the audit. It’s local root code execution in . The bug is visible if you know what to look for (a followed by a function call that loads a dynamic library), but it’s the kind of thing that doesn’t jump out on a first read. Here’s the pattern, simplified from the utility. Huh. Looks innocent. The trap is that ends up loading shared libraries from the new root filesystem to resolve the username. An attacker who can plant a file in the chroot gets to run code as uid 0. GNU resolves the user before calling . Same fix here. Once you’re across, every library call might run the attacker’s code. And no, static compilation doesn’t help here, because goes through NSS, which s modules at runtime regardless of whether your binary is statically linked. You might have made it this far and thought “Wow, that’s a lot of bugs! Maybe Rust isn’t as safe as I thought?” That would be the wrong conclusion. Keep in mind that none of the following bad things happened: That means, even if the tools were (and probably still are) buggy, they never had a bug that could be exploited to read arbitrary memory. GNU coreutils has shipped CVEs in every single one of those categories. Take a peek at the last few years of the GNU file: …the list goes on and on. The Rust rewrite has shipped zero of these, over a comparable window of activity. 1 That’s most of what historically goes wrong in a C codebase. What’s left is, frankly, a more interesting class of bug. It lives at the boundary between our controlled Rust environment and the messy, chaotic outside world, where paths, bytes, strings, and syscalls are all tangled up in one eternal ball of sadness. That’s the new security boundary of modern systems code. 2 If you write systems code in Rust, treat this CVE list as a checklist. Grep your own codebase for , stray calls, discarded s, , and string comparisons against . I also wrote a companion post, titled Patterns for Defensive Programming in Rust . When I think of “ idiomatic Rust ”, correctness is not the first thing that comes to mind. After all, isn’t that the compiler’s job? Instead, I think of elegant iterator patterns , ergonomic method signatures, immutability , or clever use of expressions . But none of that matters if the code doesn’t do the right thing, and the compiler is far from perfect at enforcing correctness. That’s why we don’t only have idioms for writing more elegant code; we also have idioms for writing correct code. They are the distilled experience of a community that has learned, often painfully, which shapes of code survive contact with reality and which ones do not. Reality is rarely as tidy as the abstractions we would like to impose on it. The mark of robust systems, in any language, is the willingness to reflect that untidiness rather than paper over it. Rust gives us extraordinary tools to do so, and the compiler will hold a great deal for us. But the part it cannot hold, the boundary between our program and everything else, is still ours to get right. The type system can encode many things, but it cannot encode conditions outside of its control, such as the passage of time between two syscalls. Idiomatic Rust, then, is not just code that the borrow checker accepts or that leaves alone. It is code whose types, names, and control flow tell the truth about the system they run in. And that truth is sometimes ugly. It could mean using file descriptors instead of paths, instead of , instead of , and bug-for-bug compatibility over clean semantics. None of it is as pretty as the version you would write on a whiteboard. But it is more honest. Need Help Hardening Your Rust Codebase? Is your team shipping Rust into production and want to make sure you’re not falling into the same traps? I offer Rust consulting services, from code reviews and security-focused audits to training your team on the patterns that the compiler won’t enforce for you. Get in touch to learn more. To be fair to GNU: GNU coreutils is 40 years old and has had a very long time to surface and fix this class of bug. And we don’t know there are no memory-safety bugs in the Rust rewrite, only that the audit didn’t find any. Still, the difference is noticeable when comparing the same duration of development activity. ↩ It’s worth noting that the / TOCTOU class of bug is in some ways easier to avoid in C than in Rust. C code naturally reaches for an open file descriptor and the family of syscalls ( , , , ), and most creation syscalls take a argument directly. Rust’s high-level APIs abstract over the file descriptor and operate on values, which makes the path-based, re-resolving call the path of least resistance. The handle-based APIs exist on every Unix platform; Rust just doesn’t put them front and center. ↩ 🫩 Lossy conversion with silently rewrites invalid bytes to U+FFFD. That’s just fancy data corruption. 🫤 Strict conversion with or crashes or refuses to operate. 😚 Staying in bytes with or is what you should usually do. No buffer overflows. No use-after-free. No double-free. No data races on shared mutable state. No null-pointer dereferences. No uninitialized memory reads. buffer overflow on deep paths longer than (9.11, 2026) out-of-bounds read on trailing blanks (9.9, 2025) heap buffer overflow (9.9, 2025) writes a NUL byte past a heap buffer (9.8, 2025) 1-byte read before a heap buffer with a key offset (9.8, 2025) and crashes with SELinux but no xattr support (9.7, 2025) heap overwrite ( CVE-2024-0684 , 9.5, 2024) reads unallocated memory on malformed input (9.4, 2023) stack buffer overrun with many files and a high (9.0, 2021) To be fair to GNU: GNU coreutils is 40 years old and has had a very long time to surface and fix this class of bug. And we don’t know there are no memory-safety bugs in the Rust rewrite, only that the audit didn’t find any. Still, the difference is noticeable when comparing the same duration of development activity. ↩ It’s worth noting that the / TOCTOU class of bug is in some ways easier to avoid in C than in Rust. C code naturally reaches for an open file descriptor and the family of syscalls ( , , , ), and most creation syscalls take a argument directly. Rust’s high-level APIs abstract over the file descriptor and operate on values, which makes the path-based, re-resolving call the path of least resistance. The handle-based APIs exist on every Unix platform; Rust just doesn’t put them front and center. ↩

rust Rust programming Programming python Python security Security c C
0 views
Eli Bendersky's website 2 months ago

Debugging WASM in Chrome DevTools

When I was working on the WASM backend for my Scheme compiler , I ran into several tricky situations with debugging generated WASM code. It turned out that Chrome has a very capable WASM debugger in its DevTools, so in this brief post I want to share how it can be used. I'll be using an example from my wasm-wat-samples project for this post. In fact, everything is already in place in the gc-print-scheme-pairs sample. This sample shows how to construct Scheme-like s-exprs in WASM using gc references and print them out recursively. The sample supports nested pairs of integers, booleans and symbols. To see this in action, we have to first compile the WAT file to WASM, for example using watgo : The browser-loader.html file in that directory already expects to load gc-print-scheme-pairs.wasm . But we can't just open it directly from the file-system; since it loads WASM, this file needs to be served with a local HTTP server. I personally use static-server for this, but you can use anything else - like Python's built-in http.server : Now it can be opened in the browser by following the printed link and selecting the browser-loader.html file. Open the Chrome DevTools, and in Sources , open the Page view on the left. It should have one entry under wasm , which will show the decompiled WAT code for our module. Note: this code is disassembled from the binary WASM, so it will lose some WAT syntactic sugar (like folded instructions): You can set a breakpoint by clicking on the address column to the left of the code, and then refresh the page. The DevTools debugger will run the program again and stop at the breakpoint: Here you can step over, into, see local values and call stack, etc - a real debugger! The most important use case for me while developing the compiler was debugging unexpected exceptions (coming from instructions like ref.cast ). Notice the checkboxes saying "Pause on ... exceptions" on the right-hand side of the previous screenshot. With these selected, the DevTools debugger will automatically stop on an exception and show where it is coming from. Let's modify the gc-print-scheme-pairs.wat sample to see this in action. The $emit_value function performs a set of ref.test checks to see which kind of reference it's dealing with before casting; let's add this line at the very start: It's clearly wrong to assume that $v is a bool reference without first testing it; this is just for demonstration purposes. Without setting any breakpoints, recompiling this code with watgo and reloading the page, we get: The debugger stopped at the instruction causing the exception; moreover, in the Scope pane on the right we can see that the actual type of $v is (ref $Pair) , so it's immediately clear what's going on. I've found this capability extremely valuable when writing (or emitting from a compiler) non-trivial chunks of WASM code using gc types and instructions. "Should I use a debugger or just printfs" is a common topic of debate among programmers. While I'm usually in the "printf debugging" camp, I'm not dogmatic, and will certainly reach for a debugger when the situation calls for it. Specifically, when investigating reference exceptions in WASM, two strong factors tilt the decision towards using a debugger: In general, WASM's printf capabilities aren't great. We can import print-like functions from the host (and - in fact - our sample does just that), but they're not very flexible and dealing with strings in WASM is painful in general. This is compounded even more when working with gc types, because these aren't even visible to the host (they're opaque references). If we want to do printf debugging of gc values, we have to build a lot of scaffolding first. Exception debugging - in general - is much easier with a supportive debugger in hand. Our ref.cast exception from the example above could have happened anywhere in the code. Imagine having to debug a very large WASM program (emitted by a compiler) to find the source of a failed ref.cast ; the debugger takes you right to the spot! In fact, even for C programming, I've always found gdb most useful for pinpointing the source of segmentation faults and similar crashes. In general, WASM's printf capabilities aren't great. We can import print-like functions from the host (and - in fact - our sample does just that), but they're not very flexible and dealing with strings in WASM is painful in general. This is compounded even more when working with gc types, because these aren't even visible to the host (they're opaque references). If we want to do printf debugging of gc values, we have to build a lot of scaffolding first. Exception debugging - in general - is much easier with a supportive debugger in hand. Our ref.cast exception from the example above could have happened anywhere in the code. Imagine having to debug a very large WASM program (emitted by a compiler) to find the source of a failed ref.cast ; the debugger takes you right to the spot! In fact, even for C programming, I've always found gdb most useful for pinpointing the source of segmentation faults and similar crashes.

c C javascript JavaScript html HTML tutorial Tutorial web-development Web Development
0 views
Simon Willison 3 months ago

Anthropic's Project Glasswing - restricting Claude Mythos to security researchers - sounds necessary to me

Anthropic didn't release their latest model, Claude Mythos ( system card PDF ), today. They have instead made it available to a very restricted set of preview partners under their newly announced Project Glasswing . The model is a general purpose model, similar to Claude Opus 4.6, but Anthropic claim that its cyber-security research abilities are strong enough that they need to give the software industry as a whole time to prepare. Mythos Preview has already found thousands of high-severity vulnerabilities, including some in every major operating system and web browser . Given the rate of AI progress, it will not be long before such capabilities proliferate, potentially beyond actors who are committed to deploying them safely. Project Glasswing partners will receive access to Claude Mythos Preview to find and fix vulnerabilities or weaknesses in their foundational systems—systems that represent a very large portion of the world’s shared cyberattack surface. We anticipate this work will focus on tasks like local vulnerability detection, black box testing of binaries, securing endpoints, and penetration testing of systems. There's a great deal more technical detail in Assessing Claude Mythos Preview’s cybersecurity capabilities on the Anthropic Red Team blog: In one case, Mythos Preview wrote a web browser exploit that chained together four vulnerabilities, writing a complex  JIT heap spray  that escaped both renderer and OS sandboxes. It autonomously obtained local privilege escalation exploits on Linux and other operating systems by exploiting subtle race conditions and KASLR-bypasses. And it autonomously wrote a remote code execution exploit on FreeBSD's NFS server that granted full root access to unauthenticated users by splitting a 20-gadget ROP chain over multiple packets. Plus this comparison with Claude 4.6 Opus: Our internal evaluations showed that Opus 4.6 generally had a near-0% success rate at autonomous exploit development. But Mythos Preview is in a different league. For example, Opus 4.6 turned the vulnerabilities it had found in Mozilla’s Firefox 147 JavaScript engine—all patched in Firefox 148—into JavaScript shell exploits only two times out of several hundred attempts. We re-ran this experiment as a benchmark for Mythos Preview, which developed working exploits 181 times, and achieved register control on 29 more. Saying "our model is too dangerous to release" is a great way to build buzz around a new model, but in this case I expect their caution is warranted. Just a few days ( last Friday ) ago I started a new ai-security-research tag on this blog to acknowledge an uptick in credible security professionals pulling the alarm on how good modern LLMs have got at vulnerability research. Greg Kroah-Hartman of the Linux kernel: Months ago, we were getting what we called 'AI slop,' AI-generated security reports that were obviously wrong or low quality. It was kind of funny. It didn't really worry us. Something happened a month ago, and the world switched. Now we have real reports. All open source projects have real reports that are made with AI, but they're good, and they're real. Daniel Stenberg of : The challenge with AI in open source security has transitioned from an AI slop tsunami into more of a ... plain security report tsunami. Less slop but lots of reports. Many of them really good. I'm spending hours per day on this now. It's intense. And Thomas Ptacek published Vulnerability Research Is Cooked , a post inspired by his podcast conversation with Anthropic's Nicholas Carlini. Anthropic have a 5 minute talking heads video describing the Glasswing project. Nicholas Carlini appears as one of those talking heads, where he said (highlights mine): It has the ability to chain together vulnerabilities. So what this means is you find two vulnerabilities, either of which doesn't really get you very much independently. But this model is able to create exploits out of three, four, or sometimes five vulnerabilities that in sequence give you some kind of very sophisticated end outcome. [...] I've found more bugs in the last couple of weeks than I found in the rest of my life combined . We've used the model to scan a bunch of open source code, and the thing that we went for first was operating systems, because this is the code that underlies the entire internet infrastructure. For OpenBSD, we found a bug that's been present for 27 years, where I can send a couple of pieces of data to any OpenBSD server and crash it . On Linux, we found a number of vulnerabilities where as a user with no permissions, I can elevate myself to the administrator by just running some binary on my machine. For each of these bugs, we told the maintainers who actually run the software about them, and they went and fixed them and have deployed the patches patches so that anyone who runs the software is no longer vulnerable to these attacks. I found this on the OpenBSD 7.8 errata page : 025: RELIABILITY FIX: March 25, 2026 All architectures TCP packets with invalid SACK options could crash the kernel. A source code patch exists which remedies this problem. I tracked that change down in the GitHub mirror of the OpenBSD CVS repo (apparently they still use CVS!) and found it using git blame : Sure enough, the surrounding code is from 27 years ago. I'm not sure which Linux vulnerability Nicholas was describing, but it may have been this NFS one recently covered by Michael Lynch . There's enough smoke here that I believe there's a fire. It's not surprising to find vulnerabilities in decades-old software, especially given that they're mostly written in C, but what's new is that coding agents run by the latest frontier LLMs are proving tirelessly capable at digging up these issues. I actually thought to myself on Friday that this sounded like an industry-wide reckoning in the making, and that it might warrant a huge investment of time and money to get ahead of the inevitable barrage of vulnerabilities. Project Glasswing incorporates "$100M in usage credits ... as well as $4M in direct donations to open-source security organizations". Partners include AWS, Apple, Microsoft, Google, and the Linux Foundation. It would be great to see OpenAI involved as well - GPT-5.4 already has a strong reputation for finding security vulnerabilities and they have stronger models on the near horizon. The bad news for those of us who are not trusted partners is this: We do not plan to make Claude Mythos Preview generally available, but our eventual goal is to enable our users to safely deploy Mythos-class models at scale—for cybersecurity purposes, but also for the myriad other benefits that such highly capable models will bring. To do so, we need to make progress in developing cybersecurity (and other) safeguards that detect and block the model’s most dangerous outputs. We plan to launch new safeguards with an upcoming Claude Opus model, allowing us to improve and refine them with a model that does not pose the same level of risk as Mythos Preview. I can live with that. I think the security risks really are credible here, and having extra time for trusted teams to get ahead of them is a reasonable trade-off. You are only seeing the long-form articles from my blog. Subscribe to /atom/everything/ to get all of my posts, or take a look at my other subscription options .

javascript JavaScript shell Shell ai AI security Security c C
0 views