Posts in C (20 found)

JIT: so you want to be faster than an interpreter on modern CPUs…

Since my previous blog entry about JIT compiler for PostgreSQL, sadly not much happened due to a lack of time, but still some things were done (biggest improvement was the port to ARM64, a few optimizations, implementing more opcodes…). But I am often asking myself how to really beat the interpreter… And on “modern” CPUs, with a well written interpreter, that’s far more complicated than many would imagine. So in order to explain all this and show how I am planning to improve performance (possibly of the interpreter itself too, thus making this endeavor self-defeating), let’s first talk about… If you already know about all the topics mentioned in this title, feel free to jump to the next section . Note that the following section is over-simplified to make the concepts more accessible. I am writing this blog post on a Zen 2+ CPU. If I upgraded to a Zen 3 CPU, same motherboard, same memory, I would get an advertised 25% performance jump in single thread benchmarks while the CPU frequency would be only 2% higher. Why such a discrepancy? Since the 90s and the Pentium-class CPUs, x86 has followed RISC CPUs in the super-scalar era. Instead of running one instruction per cycle, when conditions are right, several instructions can be executed at the same time. Let’s consider the following pseudo-code: X and Y can be calculated at the same time. The CPU can execute these on two integer units, fetch the results and store them. The only issue is the computation of Z: everything must be done before this step, making it impossible for the CPU to go further without waiting for the previous results. But now, what if the code was written as follow: Every step would require waiting for the previous one, slowing down the CPU terribly. Hence the most important technique used to implement superscalar CPUs: out-of-order execution. The CPU will fetch the instructions, dispatch them in several instruction queues, and resolve dependencies to compute Y before computing Z1 in order to have it ready sooner. The CPU is spending less time idling, thus the whole thing is faster. But, alas, what would happen with the following function? Should the CPU wait for X and Y before deciding which Z to compute? Here is the biggest trick: it will try its luck and compute something anyway. This way, if its bet was right, a lot of time will be saved, otherwise the mistake result will be dropped and the proper computation will be done instead. This is called branch prediction, it has been the source of many fun security issues (hello meltdown), but the performance benefits are so huge that one would never consider disabling this. Most interpreters will operate on an intermediate representation, using opcodes instead of directly executing from an AST or similar. So you could use the following main loop for an interpreter. This is how many, many interpreters were written. But this has a terrible drawback at least when compiled that way: it has branches all over the place from a single starting point (most if not all optimizing compilers will generate a jump table to optimize the dispatch, but this will still jump from the same point). The CPU will have a hard time predicting the right jump, and is thus losing a lot of performance. If this was the only way an interpreter could be written, generating a function by stitching the code together would save a lot of time, likely giving a more than 10% performance improvement. If one look at Python, removing this switch made the interpreter 15 to 20% faster. Many project, including PostgreSQL, use this same technique, called “computed gotos”. After a first pass to fill in “label” targets in each step, the execution would be When running a short sequence of operations in a loop, the jumps will be far more predictable, making the branch predictor’s job easier, and thus improving the speed of the interpreter. Now that we have a very basic understanding of modern CPUs and the insane level of optimization they reach, let’s talk about fighting the PostgreSQL interpreter on performance. I will not discuss optimizing the tuple deforming part (aka. going from on-disc structure to the “C” structure used by the code), this will be a topic for a future blog post when I implement it in my compiler. As you may know, PostgreSQL has a very complete type system with operators overloading. Even this simple query ends up being a call to int4eq, a strict function that will perform the comparison. Since it is a strict function, PostgreSQL must check that the arguments are not null, otherwise the function is not called and the result will be null. If you execute a very basic query like the one in the title, PostgreSQL will have the following opcodes: The EEOP_FUNCEXPR_STRICT_2 will perform the null check, and then call the function. If we unroll all the opcodes in real C code, we end up with the following: We can already spot one optimization: why do we check the two arguments, including our constant, against null? It will never change for the entire run of this query and thus each comparison is going to use an ALU, and branch depending on that comparison. But of course the CPU will notice the corresponding branch pattern, and will thus be able to remain active and feed its other units. What is the real cost of such a pointless comparison? For this purpose, I’ve broken a PostgreSQL instance and replaced all FUNCEXPR_STRICT with a check on one argument only, and one with no STRICT check (do not try this at home!). Doing 10 times a simple SELECT * FROM demo WHERE a = 42 on a 100 million rows table, with no index, here are the two perf results: So, even if this is not the optimization of the century, it’s not that expensive to make, so… why not do it? (Patch coming to pgsql-hackers soon) But a better optimization is to go all-in on inlining. Indeed, instead of jumping through a pointer to the int4eq code (again, something that the CPU will optimize a lot), one could have a special opcode for this quite common operation. With this change alone (but keeping the two null checks, so there are still optimizations possible), we end up with the following perf results. Let’s sum up these results. The biggest change comes, quite obviously, from inlining the int4eq call. Why is it that much better? Because it reduces by quite a lot the number of instructions to run, and it removes a call to an address stored in memory. And this is again an optimization I could do on my JIT compiler that can also be done on the interpreter with the same benefits. The biggest issue here is that you must keep the number of opcodes within (unspecified) limits: too many opcodes could make the compiler job far worse. Well. At first, I thought the elimination of null checks could not be implemented easily in the interpreter. The first draft in my compiler was certainly invalid, but gave me interesting numbers (around 5%, as seen above) and made me want to go ahead. And I realized that implementing it cleanly in the interpreter was far easier than implementing it in my JIT compiler … Then I went with optimizing another common case, the call to int4eq, and, well… One could also add an opcode for that in the interpreter, and thus the performance gain of the JIT compiler are going to be minimal compared to the interpreter. Modern CPUs don’t make my job easy here. Most of the cost of an interpreter is taken away by the branch predictor and the other optimizations implemented in silicon. So is all hope lost, am I to declare the interpreter the winner against the limitations of the copy-patch method I have available for my JIT? Of course not, see you in the next post to discuss the biggest interpreter bottleneck! PS: help welcome. Last year I managed to spend some time working on this during my work time. Since then I’ve changed job, and can hardly get some time on this. I also tried to get some sponsoring to work on this and present at future PostgreSQL conferences, to no luck :/ If you can help in any way on this project, feel free to reach me (code contribution, sponsoring, missions, job offers, nudge nudge wink wink). Since I’ve been alone on this, a lot of things are dibbles on scratch paper, I benchmark code and stuff in my head when life gives me some boring time but testing it for real is of course far better. I have some travel planned soon so I hope for next part to be released before next year, with interesting results since my experiences have been as successful as anticipated.

0 views
Corrode 1 weeks ago

Prime Video

Are you one of over 240 million subscribers of Amazon’s Prime Video service? If so, you might be surprised to learn that much of the infrastructure behind Prime Video is built using Rust. They use a single codebase for media players, game consoles, and tablets. In this episode, we sit down with Alexandru Ene, a Principal Engineer at Amazon, to discuss how Rust is used at Prime Video, the challenges they face in building a global streaming service, and the benefits of using Rust for their systems. CodeCrafters helps you become proficient in Rust by building real-world, production-grade projects. Learn hands-on by creating your own shell, HTTP server, Redis, Kafka, Git, SQLite, or DNS service from scratch. Start for free today and enjoy 40% off any paid plan by using this link . Prime Video is a streaming service offered by Amazon that provides a wide range of movies, TV shows, and original content to its subscribers. With over 240 million subscribers worldwide, Prime Video is one of the largest streaming platforms in the world. In addition to its vast content library, Prime Video also offers features such as offline viewing, 4K streaming, and support for multiple devices. On the backend, Prime Video relies on a variety of technologies to deliver its content, including Rust, which is used for building high-performance and reliable systems that can handle the demands of a global audience. Alexandru worked on the transition of Prime Video’s user interface from JavaScript to Rust. He has been with Amazon for over 8 years and previously worked at companies like Ubisoft and EA. He has a background in computer science and is an active open source maintainer. Alexandru lives in London. Ferris Makes Emulators Ep.001 - The Journey Begins - First episode of a famous series where Jake Taylor wrote a Nintendo 64 emulator in Rust from scratch CMake - Very common build system used in C++ applications Conan - C++ Package Manager community project C++ Smart Pointers - Still a footgun Herb Sutter: The Free Lunch Is Over - The seminal 2005 paper that highlights the importance of concurrency, well past C++’s mainstream adoption Rust in Production: cURL - Baseline library used everywhere, written in C, but performant and safe Prime Video Platforms - One app runs on all of these WebAssembly (WASM) - Enabling Rust code with good performance that you can still download and run like JavaScript, avoiding the need for firmware updates on some devices Entity Component System - Used in the UI Rust code for pages in the app Bevy - Game engine written in Rust Leptos - UI framework that makes reactive programming in Rust easier tokio - The de facto standard async runtime for Rust SIMD - A nice feature set some CPUs support WebAssembly Micro Runtime - A tiny WASM runtime well suited for IoT platforms WebAssembly Working Group Amazon Prime Video Rust & WASM for UI: Faster Prime Video on ANY Device - Alexandru Ene, QCon San Francisco 2024 Alexandru Ene on LinkedIn Alexandru’s Blog Alexandru Ene on GitHub

0 views
Matthias Endler 2 weeks ago

On Choosing Rust

Since my professional writing on Rust has moved to the corrode blog , I can be a bit more casual on here and share some of my personal thoughts on the recent debate around using Rust in established software. The two projects in question are git ( kernel thread , Hacker News Discussion ) and the recently rewritten coreutils in Rust , which will ship with Ubuntu 25.10 Quizzical Quokka . What prompted me to write this post is a discussion on Twitter and a blog post titled “Are We Chasing Language Hype Over Solving Real Problems?” . In both cases, the authors speculate about the motivations behind choosing Rust, and as someone who helps teams use Rust in production, I find those takes… hilarious. Back when I started corrode, people always mentioned that Rust wasn’t used for anything serious. I knew about the production use cases from client work, but there was very little public information out there. As a consequence, we started the ‘Rust in Production’ podcast to show that companies indeed choose Rust for real-world applications. However, people don’t like to be proven wrong, so that conspiracy theory has now morphed into “Big Rust” trying to take over the world. 😆 Let’s look at some of the claims made in the blog post and Twitter thread and see how these could be debunked pretty easily. “GNU Core Utils has basically never had any major security vulnerabilities in its entire existence” If only that were true. A quick CVE search shows multiple security issues over the decades, including buffer overflows and path traversal vulnerabilities. Just a few months ago, a heap buffer under-read was found in , which would cause a leak of sensitive data if an attacker sends a specially crafted input stream. The GNU coreutils are one of the most widely used software packages worldwide with billions of installations and hundreds (thousands?) of developers looking at the code. Yes, vulnerabilities still happen. No, it is not easy to write correct, secure C code. No, not even if you’re extra careful and disciplined. is five thousand lines long. (Check out the source code ). That’s a lot of code for printing file names and metadata and a big attack surface! “Rust can only ever match C performance at best and is usually slower” Work by Trifecta shows that it is possible to write Rust code that is faster than C in some cases. Especially in concurrent workloads and with memory safety guarantees. If writing safe C code is too hard, try writing safe concurrent C code! That’s where Rust shines. You can achieve ridiculous levels of parallelization without worrying about security issues. And no, you don’t need to litter your code with blocks. Check out Steve Klabnik’s recent talk about Oxide where he shows that their bootloader and their preemptive multitasking OS, hubris – both pretty core systems code – only contain 5% of code each. You can write large codebases in Rust with no unsafe code at all. As a trivial example, I sat down to rewrite in Rust one day. The result was 3x faster than GNU on my machine. You can read the post here . All I did was use to copy data, which saves one memory copy. Performance is not only dependent on the language but on the algorithms and system calls you use. If you play into Rust’s strengths, you can match C’s performance. At least there is no technical limitation that would prevent this. And I personally feel more willing to aggressively optimize my code in Rust, because I don’t have to worry about introducing memory safety bugs. It feels like I’m not alone . “We reward novelty over necessity in the industry” This ignores that most successful companies (Google, Meta, etc.) primarily use battle-tested tech stacks, not bleeding-edge languages. These companies have massive codebases and cannot afford to rewrite everything in the latest trendy language. But they see the value of using Rust for new components and gradually rewriting existing ones. That’s because 70% of security vulnerabilities are memory safety issues and these issues are extremely costly to fix. If these companies could avoid switching to a new language to do so, they would. Besides, Rust is not exactly new anymore. Rust 1.0 was released 10+ years ago! The industry is moving slowly, but not that slowly. You’d be surprised to find out how many established companies use Rust without even announcing it or thinking of it as “novelty”. “100% orchestrated” Multiple people in the Twitter thread were convinced this is some coordinated master plan rather than developers choosing better tools, while the very maintainers of git and coreutils openly discussed their motivations in public forums for everyone to see. “They’re trying to replace/erase C. It’s not going to happen” They are right. C is not going away anytime soon. There is just so much C/C++ code out there in the wild, and rewriting everything in Rust is not feasible. The good news is that you can incrementally rewrite C/C++ code in Rust, one component at a time. That’s what the git maintainers are planning, by using Rust for new components. “They’re rewriting software with a GNU license into software with an MIT license” Even if you use Rust, you can still license your code under GPL or any other license you want. Git itself remains GPL, and many Rust projects use various licenses, not only MIT. The license fear is often brought up by people who don’t understand how open source licensing works or it might just be FUD. MIT code is still compatible with GPL code and you can use both of them in the same project without issues. It’s just that the end product (the thing you deliver to your users, i.e. binary executables) is now covered by GPL because of its virality. “It’s just developers being bored and wanting to work with shiny new languages” The aging maintainers of C projects are retiring, and there are fewer new developers willing to pick up C just to maintain legacy code in their free time. C developers are essentially going extinct. New developers want to work with modern languages and who can blame them? Or would you want to maintain a 40-year-old COBOL codebase or an old Perl script? We have to move on. “Why not build something completely new instead of rewriting existing tools?” It’s not that easy. The code is only part of the story. The other part is the ecosystem, the tooling, the integrations, the documentation, and the user base. All of that takes years to build. Users don’t want to change their workflows, so they want drop-in replacements. Proven interfaces and APIs, no matter how crude and old-fashioned, have a lot of value. But yes, new tools are being built in Rust as well. “They don’t know how to actually solve problems, just chase trends” Talk about dismissing the technical expertise of maintainers who’ve been working on these projects for years or decades and understand the pain points better than anyone. If they were just chasing trends, they wouldn’t be maintaining these projects in the first place! These people are some of the most experienced developers in the world, and yet people want to tell them how to do their jobs. “It’s part of the woke mind virus infecting software” Imagine thinking memory safety is a political conspiracy. Apparently preventing buffer overflows is now an ideological stance. The closest thing to this is the White House’s technical report which recommends memory-safe languages for government software and mandating memory safety for software receiving federal funding is a pretty reasonable take. Conclusion I could go on, but I think you get my point. People who give Rust an honest chance know that it offers advantages in terms of memory safety, concurrency, and maintainability. It’s not about chasing hype but about long-term investment in software quality. As more companies successfully adopt Rust every day, it increasingly becomes the default choice for many new projects. If you’re interested in learning more about using Rust in production, check out my other blog or listen to the Rust in Production podcast . Oh, and if you know someone who posts such takes, stop arguing and send them a link to this post.

0 views
Dayvster 3 weeks ago

Are We Chasing Language Hype Over Solving Real Problems?

## Intro As you may have heard or seen, there is a bit of controversy around Ubuntu adopting a rewritten version of GNU Core Utils in Rust. This has sparked a lot of debate in the tech community. This decision by Canonical got me thinking about this whole trend or push of rewriting existing software in Rust which seems to be happening a lot lately. To put it bluntly I was confused by the need to replace GNU Core Utils with a new implementation as GNU Core Utils has been around since arguably the 90s and more realistically 2000s and it has been battle tested and proven to be reliable, efficient, effective and most importantly secure as it had basically never had any major security vulnerabilities in its entire existence. So why then would we deem it necessary to replace it with a new implementation in Rust? Why would anyone go through the trouble of rewriting something that already works perfectly fine and has been doing so for decades? When the end result at best is going to be a tool that does the same thing as the original and in the very best case scenario offer the same performance? What bothers me even more is the bigger pattern this points to. Are we as developers more interested in chasing new languages and frameworks than actually solving real problems? I strongly subscribe to the idea that software development 60% problem solving and 40% creative exploration and innovation. But lately it feels like the balance is shifting more and more towards the latter. We seem to be more interested in trying out the latest and greatest languages and frameworks than actually solving real problems. ## The Hype of New Languages and Shiny Object Syndrome We've all been there in the past haven't we? Getting excited about a new programming language or framework that promises to solve all our problems and make our lives easier. It's easy to get caught up in the hype and want to try out the latest and greatest technology, but it's important to remember that just because something is new doesn't mean it's better. We need to be careful not to fall into the trap of "shiny object syndrome" where we chase after the latest trends without considering whether they actually solve our problems or improve our workflows. It's important to evaluate new technologies based on their merits and how they fit into our existing systems and workflows, rather than simply jumping on the bandwagon because everyone else is doing it. Now for the important question: **Do I think the developers of coreutils-rs are doing this just because Rust is the new hotness?** Short and simple: No, no I do not. I believe they have good intentions and are likely trying to improve upon the existing implementation in some way. However, I do not agree with them that there is a need for a rewritten version of GNU Core Utils in Rust. I also do not agree that GNU Core Utils is inherently insecure or unsafe. ### Why do we get Exited About New Languages? It's also important to briefly touch upon the psychological aspect of why we get excited about new languages. New languages often come with new features, syntax, and paradigms that can be appealing to developers. They may also promise to solve problems that existing languages struggle with, such as performance, concurrency, or memory safety. Additionally, new languages can offer a fresh perspective on programming and can inspire creativity and innovation. Not to mention the community aspect, new usually means a changing of the guard, new people, new ideas, new ways of thinking about problems. All of these factors can contribute to the excitement and enthusiasm that developers feel when a new language is introduced. This enthusiasm can sometimes lead to an almost zealous approach of wanting everything and anything to be written only in the new language by this new and fresh community of developers. This can lead to a situation where existing and well-established software is rewritten in the new language, even if there is no real need for it. This can be seen as a form of "language evangelism" where developers are trying to promote their favorite language by rewriting existing software in it. ## The Case of GNU Core Utils As I've briefly touched upon earlier, GNU Core Utils is a collection of basic file, shell and text manipulation utilities that are fundamental to the operation of Unix-like operating systems. These utilities include commands like `ls`, `cp`, `mv`, `rm`, `cat`, `echo`, and many others. They are essential for performing everyday tasks in the command line interface (CLI) and are used by system administrators, developers, and users alike. Some of these can run hundreds of times per second, so performance is absolutely crucial. Even a small reduction in performance to a utility that is run by some OS critical daemon can have a significant impact on the overall performance of the system. GNU core utils has been optimized for this for about 30+ years at this point and is it really worth just tossing all of those lessons and optimizations out the window just to rewrite it in a new language? I've also briefly touched upon that at best in the absolute **best case scenario** a rewritten version of GNU Core Utils in Rust would be able to match the performance of the original implementation. As we know GNU Core Utils are mostly written in C and some C++ mixed in sparingly. So far benchmarks have shown time and time again that at best with a lot of optimizations and tweaks Rust can only ever match the performance of C and in most cases it is actually slower. So the best case outcome of this rewrite is that we get a tool that does the same thing as the original and at best offers the same performance. So what is the actual benefit of this rewrite? Where is the value, what is the actual problem that is being solved here? ## When Hype Overshadows Real Problems This is the crux of the issue, it's very very easy to get swept up in the excitement of a new language and want to use it for everything and anything under the sun. As developers we love novelty and communities with enthusiasm and fresh ideas. It's stimulating it's fun it feels like progress it feels like we are finally building something again instead of just rehashing and maintaining. We all know from personal experience that creating a new project is more fun and enjoyable than maintaining and existing one and this is a natural human tendency. Now do I think this is one of the reasons the developers of coreutils-rs are doing this? Yes, I do. But in the end they are solving a problem that does not exist. ### It's not just about Core Utils Now with how often I've mentioned this specific example of GNU Core Utils you might think I want to single them out or have some sort of grudge or specific issue with this particular project. No, not really... I think this project is indicative of the larger issue we face in the tech community. It's very easy to get caught up in the excitement of new languages and frameworks and want to use them for everything and anything. This can lead to a situation where we are rewriting existing software in new languages without considering whether it actually solves any real problems or improves our workflows. ## Problem Solving Should Be Our North Star At the end of the day, software development is about solving real problems, not about chasing novelty, shiny new languages, or personal curiosity. Every line of code we write, every framework we adopt, every library we integrate should be justified by the problems it helps solve, not by the hype around it. Yet increasingly, it feels like the industry is losing sight of this. We celebrate engineers for building in the “new hot language,” for rewriting tools, or for adopting the latest framework, even when the original solution worked perfectly fine. We reward novelty over necessity, excitement over impact. This is not a phenomenon isolated to just core utils or systems programming, it happens in web development, mobile development, data science, and pretty much every other area of software development. We too often abandon tried and true solutions and ideas of new and exciting shiny ones without considering whether they actually solve any real problems or improve our workflows. For example web development went full circle with React Server Components where we went from separation of concerns straight back to PHP style mixing of HTML and logic, server rendering and delivering interactive components to the client. Or the whole GraphQL craze where traditional REST APIs were abandoned en masse for a new and exciting way of doing things that promised to solve the dreaded problem of "over-fetching" and "under-fetching" of data. Yet in reality, it introduced a whole new set of problems and complexities that were not present in traditional REST APIs. Or perhaps the whole microservices and microfrontend craze where a lot of projects were abandoned or rewritten to be split into smaller and smaller pieces, **Was it all bad? Should we always just stick to what works and only ever maintain legacy projects and systems? Heck no!** There is definitely a place for innovation and new ideas in software development. New languages, frameworks, and tools can bring fresh perspectives and approaches to problem-solving. However, it's important to evaluate these new technologies based on their merits and how they fit into our existing systems and workflows, rather than simply jumping on the bandwagon because everyone else is doing it. We need to be more critical and thoughtful about the technologies we adopt and the projects we undertake. We need to ask ourselves whether a new language or framework actually solves a real problem or improves our workflows, or if we're just chasing novelty for its own sake. ## A Final Thought At the end of the day, it’s not about Rust, React, GraphQL, or the latest microservices fad. It’s about solving real problems. Every line of code, every framework, every rewrite should have a purpose beyond curiosity or hype. We live in a culture that rewards novelty, celebrates “cool” tech, and often mistakes excitement for progress. But progress isn’t measured in how many new languages you touch, or how many shiny rewrites you ship, it’s measured in impact, in the problems you actually solve for users, teams, and systems. So next time you feel the pull of the newest hot language, framework, or tool, pause. Ask yourself: “Am I solving a real problem here, or just chasing excitement?” Because at the end of the day, engineering isn’t about what’s trendy, it’s about what works, what matters, and what actually makes a difference. And that, my friends, is the craft we should all be sharpening.

0 views
Dayvster 4 weeks ago

Why I Still Reach for C for Certain Projects

## The Project That Made Me Choose C Again So a while back I got tired of xserver and decided to give wayland a try. I use Arch (BTW) with a tiling window manager (dwm) and I wanted to try something new, since I had a couple of annoyances with xserver. I've heard some good things about wayland so I thought you know what, why not let's give it a shot. After 30-45min my Arch and Hyprland setup was done and ready to go. I was pretty happy with it, but I was missing some features I've previously had such as notifications for when somebody posts new content to their RSS feed. I quite like RSS feeds I use them to keep up to date with blogs, news, streams, releases etc. So I thought to myself, why not write a small program that checks my RSS feeds and sends me a notification when there's something new. Now the way I was gonna go about this was pretty simple. I would write a simple C daemon that would run in the background, check my RSS feeds on a random interval between 5-15min and if there was something new it would send out a notification using `notify-send`. Then comes the tricky part I wanted `swaync` to allow me to offer two buttons on the notification, one to open the link in my default browser and one to ignore the notification and mark it as ignored in a flat file on my system so that I could see if I have to remove certain feeds that I tend to ignore a lot. Now here's the problem, you can't really do that with `swaync`, I mean it does support buttons but it doesn't really let you handle the button clicks in a way that would allow you to do what I wanted. So I had to come up with a workaround. The workaround was to have another C program run as a daemon that would listen on the DBus for a very specific notfication that would contain an action with a string of `openUrl("url")` or `ignoreUrl("url")` and then handle the action accordingly. This way I could have `swaync` send out a notification with the buttons and when the user(me) clicks on one of the buttons it would send out a DBus notification that my second daemon would pick up and handle. ## Why C Was the Right Choice Here? Now you might be wondering why I chose C for this project and not something like python which would allow me to write this specific program much faster and enjoy the rest of my weekend. Well my answer to you is simple, I don't like python and I didn't feel like wasting multiple 100s of MBs of RAM for something this simple and small. In fact I DID write it in Python and Go first with their respective DBus libraries, both were super easy to work with and I had an initial working prototype within less than an hour. But as I ran a simple `htop` I saw that the python version was using around 150MB of RAM and the Go version was using around 50MB of RAM. Now don't get me wrong, I'm not on an old thinkpad I have RAM to spare 32Gb to be exact. But why waste it on something this small and simple. Plus C is just so much more fun and exciting to work with. So I set up my C project which was just a simple `Makefile` and a couple of `.c` and `.h` files. I imported `dbus/dbus.h` and got to work. Now I'd be lying if I said it took me no time at all, in fact it took me roughly 3-4h which is a lot longer than python or Go. But in the end I had a working prototype that was using around 1-2MB of RAM and was super fast and responsive. I also got to brush up on my DBus skills which were a bit rusty since I hadn't worked with it in a while. Now getting a simple program like that from 150 to 50 to 1-2Mb of RAM usage is a huge performance improvement and it really showcases the power and strength of C as a programming language. Look at this this way I may have spent multiple hours longer writing this program in C but it will just continue to run in the background using a fraction of the resources that python or Go would have used for a long time to come. Additionally this probably won't be the only modification I make to my system now imagine if I were this lackluster with 10-20 small programs that I run in the background and let's make a wild assumption that each of those programs would use about 50-200Mb of RAM. Now we're looking at 500-4000Mb of RAM usage, or precisely one chrome tab! No I don't think that's an acceptable tradeoff for a couple of hours of my time. So in the end I think C was the right choice for this project and I'm pretty happy with the result. ## Why Modern Languages Aren't Always the Best Fit That long rant above is a cute story / anecdote. But I feel like it also highlights a bigger issue that developers face today. A lot of times we just wanna complete a certain task as quickly as possible and have it running before we've had our second coffee. I get it we're all busy people and when resources are as cheap as they are today saving a few Mb of RAM or a few ms of CPU time isn't really a big deal. But there is something so nice and satisfying about writing a small program in C that does exactly what you want it to do and nothing more. It's like a breath of fresh air in a world where everything is becoming more and more bloated and resource hungry. It feels a bit zen taking everything down to the bare minimum and just focusing on the task at hand. I'd liken developing in C to woodworking, you have to be precise and careful with your cuts, but in the end you get a beautiful product that you can be uniquely proud of. Sure you could go pick up a flat pack from your local IKEA and have a nice looking table within an hour. But it will be souless and generic and not really made well, not something you can proudly say to your visitors "Hey I built this myself". I mean you could... but they probably won't really be as impressed with your flat pack assembly as if you were to show them a hand crafted table you made yourself. ## C Excels in Low-Level System Programming and Efficiency So maybe the example project that I described above in the beginning of the article is too rickety or not really high brow enough that's fair I get that. But even if you think my project was stupid or silly or could have been done better or easier there is no denying that C simply is the gold standard for low level systems programming and efficiency. C gives you unparalleled control over system resources, memory management, and performance optimization. This makes it the go-to choice for developing operating systems, embedded systems, and performance-critical applications where every byte of memory and every CPU cycle counts. For example, the Linux kernel, which powers a vast majority of servers, desktops, and mobile devices worldwide, is primarily written in C. This is because C allows developers to write code that can directly interact with hardware and manage system resources efficiently. If you drive any modern vehicle that was created in the past decade or so, chances are that you have multiple microcontrollers in your car that are running C code to manage everything from engine performance, emergency breaking systems (AEB, ABS), distance control systems (ACC) and so on. For these systems you are very limited in the amount of RAM usage you can afford and most importantly you can not afford too much latency or delays in processing. Now how often have you heard about these systems failing and leading to fatal crashes? Not very often I'd wager. Of course safety standards, regulations and certifications play a big role in this but the choice of programming language is also a big factor. C is a mature and well-established language with a long history of use in safety-critical systems, making it a reliable choice for such applications. It does it's job exceptionally well and it does it with a very small footprint and most importantly you can run it in perpetuity on very limited hardware without worry over failure or issues due to resource exhaustion. ## Real World Scenarios Where C Shines Here are a few real-world scenarios where C is the preferred choice: - **Operating Systems**: As mentioned earlier, the Linux kernel is written in C. Other operating systems like Windows and macOS also have significant portions written in C. - **Embedded Systems**: C is widely used in embedded systems development for devices like microcontrollers, IoT devices, and automotive systems due to its efficiency and low-level hardware access. - **Game Development**: Many game engines and performance-critical game components are written in C or C++ to achieve high performance and low latency. - **Database Systems**: Many database management systems, such as MySQL and PostgreSQL, are implemented in C to ensure efficient data handling and query processing. - **Network Programming**: C is often used for developing network protocols and applications due to its ability to handle low-level socket programming and efficient data transmission. - **Compilers and Interpreters**: Many programming language compilers and interpreters are written in C to leverage its performance and low-level capabilities. - **High-Performance Computing**: C is used in scientific computing and simulations where performance is critical, such as in numerical libraries and parallel computing frameworks. - **System Utilities**: Many system utilities and command-line tools in Unix-like operating systems are written in C for efficiency and direct system access. - **Device Drivers**: C is the language of choice for writing device drivers that allow the operating system to communicate with hardware devices. - **Cryptography**: Many cryptographic libraries and algorithms are implemented in C to ensure high performance and security. - **Audio/Video Processing**: Libraries like FFmpeg are written in C to handle multimedia processing efficiently. - **Web Servers**: Popular web servers like Apache and Nginx are written in C to handle high loads and provide fast response times. You get my point, C is everywhere and it excels in scenarios where performance, efficiency, and low-level system access are paramount. ## When C Isn’t the Right Choice ? Now as you can see I'm a pretty big proponent of C and you'll often hear people who exclusively develop in C (sadly not me), say: "Anything and everything under the sun can be written in C". Which is true, if you were to remove all programming languages from existence save one, my choice would always be C as we could potentially rebuild everything from the ground up. However even I have to admit that C may not always be the best choice everything has it's ups and downs. So C may not be your best choice when: - **Rapid Prototyping**: If you need to quickly prototype an idea or concept, higher-level languages like Python or JavaScript may be more suitable due to their ease of use and extensive libraries. - **Web Development**: For building web applications, languages like JavaScript (with frameworks like React, Angular, or Vue) or Python (with frameworks like Django or Flask) are often preferred due to their ease of use and extensive web development libraries. - **Data Science and Machine Learning**: Languages like Python and R are commonly used in data science and machine learning due to their extensive libraries and frameworks (e.g., TensorFlow, PyTorch, Pandas). - **Mobile App Development**: For mobile app development, languages like Swift (for iOS) and Kotlin (for Android) are often preferred due to their platform-specific features and ease of use. - **Memory Safety**: If memory safety is a primary concern, languages like Rust or Go may be more suitable due to their built-in memory safety features and garbage collection(Go). - **Ease of Learning**: If you're new to programming, languages like Python or JavaScript may be easier to learn due to their simpler syntax and extensive learning resources. - **Community and Ecosystem**: If you need access to a large community and ecosystem of libraries and frameworks, languages like JavaScript, Python, or Java may be more suitable due to their extensive ecosystems. ## Conclusion So there you have it why I still reach for C for certain projects. It’s not always the fastest or easiest choice, but for small, efficient programs that need to run lean and mean, it often makes sense. C isn’t flashy or trendy, but it gets the job done, and sometimes that’s all that matters. Curious to hear how others tackle similar projects C, Rust, Go, or something else?

0 views
Dayvster 1 months ago

In Defense of C++

## The Reputation of C++ C++ has often and frequently been criticized for its complexity, steep learning curve, and most of all for its ability to allow the developers using it to not only shoot themselves in the foot, but to blow off their whole leg in the process. But do these criticisms hold up under scrutiny? Well, in this blog post, I aim to tackle some of the most common criticisms of C++ and provide a balanced perspective on its strengths and weaknesses. ## C++ is "Complex" C++ is indeed a complex language, with a vast array of features and capabilities. For any one thing you wish to achieve in C++, there are about a dozen different ways to do it, each with its own trade-offs and implications. So, as a developer, how are you to know which approach is the best one for your specific use case? Surely you have to have a deep understanding of the language to make these decisions, right? **Not really...** I mean, don't get me wrong, it helps, but it's not a hard requirement. Premature optimization is the root of all evil, and in C++, you can write perfectly fine code without ever needing to worry about the more complex features of the language. You can write simple, readable, and maintainable code in C++ without ever needing to use templates, operator overloading, or any of the other more advanced features of the language. There's this idea that for everything you want to do in any programming language, you need to use the most efficient and correct approach possible. Python has this with their pythonic way of doing things, Java has this, C# has this, and Go has this. Heck, even something as simple as painting HTML onto a browser needs to be reinvented every couple of years and argued about ad nauseam. Here's the thing, though, in most cases, there is no one right way to do something. The hallowed "best approach" is often just a matter of personal or team preference. The idea that if you just write your code in the "best" and correct way, you'll never need to worry about maintaining it is just plain wrong. Don't worry so much about using the "best" approach; worry more about writing code that is easy to read and understand. If you do that, you'll be fine. ## C++ is "Outdated" C++ is very old, in fact, it came out in 1985, to put it into perspective, that's 4 years before the first version of Windows was released, and 6 years before the first version of Linux came out, or to drive the point even further home, back when the last 8-bit computer was released. So yes, C++ is quite old by any standard. But does that make it outdated? **Hell no** it's not like C++ has just been sitting around unchanged from its 1985 release. C++ has been actively developed and improved upon for over 40 years now, with new features and capabilities being added all the time. The most recent version of the C++ standard, C++20, was released in 2020 and introduced a number of new features and improvements to the language. C++23 has introduced significant enhancements, particularly in the standard library and constexpr capabilities. Notably, concepts, ranges, and coroutines have been expanded, bringing modern programming paradigms to C++ and making the language more powerful and expressive th an ever before. **But Dave, what we mean by outdated is that other languages have surpassed C++ and provide a better developer experience.** Matter of personal taste, I guess, C++ is still one of the most widely used programming languages with a huge ecosystem of libraries and tools. It's used in a wide range of applications, from game development to high-performance computing to embedded systems. Many of the most popular and widely used software applications in the world are written in C++. I don't think C++ is outdated by any stretch of the imagination; you have to bend the definition of outdated quite a bit to make that claim. ## C++ is "Unsafe" Ah, finally, we get to the big one, and yes, I will draw comparisons to Rust as it's the "memory safe" language that a lot of people claim will or should replace C++. **In fact, let's get the main point out of the way right now.** ### Rewrites of C++ codebases to Rust always yield more memory-safe results than before. Countless companies have cited how they improved their security or the amount of reported bugs or memory leaks by simply rewriting their C++ codebases in Rust. **Now is that because of Rust?** I'd argue in some small part, yes. However, I think the biggest factor is that any rewrite of an existing codebase is going to yield better results than the original codebase. When you rewrite a codebase, you have the opportunity to rethink and redesign the architecture, fix bugs, and improve the overall quality of the code. You get to leverage all the lessons learned from the previous implementation, all the issues that were found and fixed, and you already know about. All the headaches that would be too much of a pain to fix in the existing codebase, you can just fix them in the new one. Imagine if you will that you've built a shed, it was a bit wobbly, and you didn't really understand proper wood joinery when you first built it, so it has a few other issues, like structural integrity and a leaky roof. After a few years, you build a new one, and this time you know all the mistakes you made the first time around, so you build it better, stronger, and more weatherproof. In the process, you decide to replace the materials you've previously used, say for example, instead of using maple, you opt for oak. Is it correct to say that the new shed is better only because you used oak instead of maple? Or is that a really small part of the overall improvement? That's how I feel when I see these companies claim that rewriting their C++ codebases in Rust has made them more memory safe. It's not because of Rust, it's because they took the time to rethink and redesign their codebase and implemented all the lessons learned from the previous implementation. ### But that does not deny the fact that C++ is unsafe. Yes, C++ can be unsafe if you don't know what you're doing. But here's the thing: all programming languages are unsafe if you don't know what you're doing. You can write unsafe code in Rust, you can write unsafe code in Python, you can write unsafe code in JavaScript. Memory safety is just one aspect of safety in programming languages; you can still write unsafe code in memory-safe programming languages. Just using Rust will not magically make your application safe; it will just make it a lot harder to have memory leaks or safety issues. The term "unsafe" is a bit too vague in this context, and I think it's being used as a catch-all term, which to me reeks of marketing speak. ### Can C++ be made safer? Yes, C++ can be made safer; in fact, it can even be made memory safe. There are a number of libraries and tools available that can help make C++ code safer, such as smart pointers, static analysis tools, and memory sanitizers. Heck, if you wish, you can even add a garbage collector to C++ if you really want to(please don't). But the easiest and most straightforward way to make C++ safer is to simply learn about smart pointers and use them wherever necessary. Smart pointers are a way to manage memory in C++ without having to manually allocate and deallocate memory. They automatically handle the memory management for you, making it much harder to have memory leaks or dangling pointers. This is the main criticism of C++ in the first place. ## C++ is Hard to Read Then don't write it that way. C++ is a multi-paradigm programming language; you can write procedural code, object-oriented code, functional code, or a mix of all three. You can write simple and readable code in C++ if you want to. You can also write complex and unreadable code in C++ if you want to. It's all about personal or team preference. Here's a rule of thumb I like to follow for C++: make it look as much like C as you possibly can, and avoid using too many advanced features of the language unless you really need to. Use smart pointers, avoid raw pointers, and use the standard library wherever possible. You can do a heck of a lot of programming by just using C++ as you would C and introducing complexity only when you really need to. ### But doesn't that defeat the whole purpose of C++? Why not just use C then? C++ is a superset of C you can write C code in C++, and it will work just fine. C++ adds a lot of features and capabilities to C. If you were to start with C, then you are locked with C, and that's fine for a lot of cases, don't get me wrong, but C++ gives you the option to use more advanced features of the language when you need them. You can start with C and then gradually introduce C++ features as you need them. You don't have to use all the features of C++ if you don't want to. Again, going back to my shed analogy, if you build a shed out of wood, you can always add a metal roof later if you need to. You don't have to build the whole shed out of metal if you don't want to. ## C++ has a confusing ecosystem C++ has a large ecosystem built over the span of 40 years or so, with a lot of different libraries and tools available. This can make it difficult to know which libraries and tools to use for a specific task. But this is not unique to C++; every programming language has this problem. Again, the simple rule of thumb is to use the standard library wherever possible; it's well-maintained and has a lot of useful features. For other tasks like networking or GUI development, there are a number of well-known libraries that are widely used and well-maintained. Do some research and find out which libraries are best suited for your specific use case. **Avoid boost like the plague.** Boost is a large collection of libraries that are widely used in the C++ community. However, many of the libraries in boost are outdated and no longer maintained. They also tend to be quite complex and difficult to use. If you can avoid using boost, do so. Unless you are writing a large and complex application that requires the specific features provided by Boost, you are better off using other libraries that are more modern and easier to use. Do not add the performance overhead and binary size bloat of Boost to your application unless you really need to. ## C++ is not a good choice for beginners Programming is not a good choice for beginners, woodworking is not a good choice for beginners, and car mechanics is not a good choice for beginners. Programming is hard; it takes time and effort to learn, as all things do. There is no general language that is really good for beginners; everything has its trade-offs. Fact is, if you wanna get into something like **systems programming or game development** then starting with Python or JavaScript won't really help you much. You will eventually need to learn C or C++. If your goal is to become a web developer or data scientist, then start with Python or JavaScript. If you just want a job in the programming industry, I don't know, learn Java or C#, both great languages that get a lot of undeserved hate, but offer a lot of job opportunities. Look, here's the thing: if you're just starting out in programming, yeah, it's gonna be hard no matter what language you choose. I'd actually argue that starting with C or C++ is far better than starting with something that obscures a lot of the underlying concepts of programming, I'd argue further that by starting with Python or Javascript you are doing yourself a disservice in the long run and trading off the pain of learning something when your understanding of a topic is still fresh and malleable for the pain of learning something later when you have a lot more invested in your current understanding of programming. But hey, that's just my opinion. ## C++ vs Rust: Friends or Rivals? Rust has earned a lot of love in recent years, and for good reason. It takes memory safety seriously, and its borrow checker enforces discipline that C++ often leaves to the programmer. That said, Rust is still building its ecosystem, and the learning curve can feel just as steep — just in different ways. C++ may not prevent you from shooting yourself in the foot, but it gives you decades of battle-tested tooling, compilers, and libraries that power everything from Chrome to Unreal Engine. In practice, many teams use Rust and C++ together rather than treating them as enemies. Rust shines in new projects where safety is the priority, while C++ continues to dominate legacy systems and performance-critical domains. ## Is C++ Still Used in 2025? The short answer: absolutely. Despite the constant chatter that it’s outdated, C++ remains one of the most widely used languages in the world. Major browsers like Chrome and Firefox are still written in it. Game engines like Unreal run on it. Automotive systems, financial trading platforms, and even AI frameworks lean heavily on C++ for performance and control. New standards (C++20, C++23) keep modernizing the language, ensuring it stays competitive with younger alternatives. If you peel back the layers of most large-scale systems we rely on daily, you’ll almost always find C++ humming away under the hood. ## Conclusion C++ is a powerful and versatile programming language that has stood the test of time. While it does have its complexities and challenges, it remains a relevant and widely used language in today's tech landscape. With the right approach and mindset, C++ can be a joy to work with and can yield high-performance and efficient applications. So next time you hear someone criticize C++, take a moment to consider the strengths and capabilities of this venerable language before dismissing it outright. Hope you enjoyed this blog post. If you did, please consider sharing it with your friends and colleagues. If you have any questions or comments, please feel free to reach out to me on [Twitter](https://twitter.com/dayvster).

0 views

Implementing Forth in Go and C

I first ran into Forth about 20 years ago when reading a book about designing embedded hardware . The reason I got the book back then was to actually learn more about the HW aspects, so having skimmed the Forth chapter I just registered an "oh, this is neat" mental note and moved on with my life. Over the last two decades I heard about Forth a few more times here and there, such as that time when Factor was talked about for a brief period, maybe 10-12 years ago or so. It always occupied a slot in the "weird language" category inside my brain, and I never paid it much attention. Until June this year, when a couple of factors combined fortuitously: And something clicked. I'm going to implement a Forth, because... why not? So I spent much of my free hacking time over the past two months learning about Forth and implementing two of them. It's useful to think of Forth (at least standard Forth , not offshoots like Factor) as having two different "levels": Another way to look at it (useful if you belong to a certain crowd) is that user-level Forth is like Lisp without macros, and hacker-level Forth has macros enabled. Lisp can still be great and useful without macros, but macros take it to an entire new level and also unlock the deeper soul of the language. This distinction will be important when discussing my Forth implementations below. There's a certain way Forth is supposed to be implemented; this is how it was originally designed, and if you get closer to the hacker level, it becomes apparent that you're pretty much required to implement it this way - otherwise supporting all of the language's standard words will be very difficult. I'm talking about the classical approach of a linked dictionary, where a word is represented as a "threaded" list [1] , and this dictionary is available for user code to augment and modify. Thus, much of the Forth implementation can be written in Forth itself. The first implementation I tried is stubbornly different. Can we just make a pure interpreter? This is what goforth is trying to explore (the Go implementation located in the root directory of that repository). Many built-in words are supported - definitely enough to write useful programs - and compilation (the definition of new Forth words using : word ... ; ) is implemented by storing the actual string following the word name in the dictionary, so it can be interpreted when the word is invoked. This was an interesting approach and in some sense, it "works". For the user level of Forth, this is perfectly usable (albeit slow). However, it's insufficient for the hacker level, because the host language interpreter (the one in Go) has all the control, so it's impossible to implement IF...THEN in Forth, for example (it has to be implemented in the host language). That was a fun way to get a deeper sense of what Forth is about, but I did want to implement the hacker level as well, so the second implementation - ctil - does just that. It's inspired by the jonesforth assembly implementation, but done in C instead [2] . ctil actually lets us implement major parts of Forth in Forth itself. For example, variable : Conditionals: These are actual examples of ctil's "prelude" - a Forth file loaded before any user code. If you understand Forth, this code is actually rather mind-blowing. We compile IF and the other words by directly laying our their low-level representation in memory, and different words communicate with each other using the data stack during compilation . Forth made perfect sense in the historic context in which it was created in the early 1970s. Imagine having some HW connected to your computer (a telescope in the case of Forth's creator), and you have to interact with it. In terms of languages at your disposal - you don't have much, even BASIC wasn't invented yet. Perhaps your machine still didn't have a C compiler ported to it; C compilers aren't simple, and C isn't very great for exploratory scripting anyway. So you mostly just have your assembly language and whatever you build on top. Forth is easy to implement in assembly and it gives you a much higher-level language; you can use it as a calculator, as a REPL, and as a DSL for pretty much anything due to its composable nature. Forth certainly has interesting aspects; it's a concatenative language , and thus inherently point-free . A classical example is that instead of writing the following in a more traditional syntax: You just write this: There is no need to explicitly pass parameters, or to explicitly return results. Everything happens implicitly on the stack. This is useful for REPL-style programming where you use your language not necessarily for writing large programs, but more for interactive instructions to various HW devices. This dearth of syntax is also what makes Forth simple to implement. All that said, in my mind Forth is firmly in the "weird language" category; it's instructive to learn and to implement, but I wouldn't actually use it for anything real these days. The stack-based programming model is cool for very terse point-free programs, but it's not particularly readable and hard to reason about without extensive comments, in my experience. Consider the implementation of a pretty standard Forth word: +! . It expects and address at the top of stack, and an addend below it. It adds the addend to the value stored at that address. Here's a Forth implementation from ctil's prelude: Look at that stack wrangling! It's really hard to follow what goes where without the detailed comments showing the stack layout on the right of each instruction (a common practice for Forth programs). Sure, we can create additional words that would make this simpler, but that just increases the lexicon of words to know. My point is, there's fundamental difficulty here. When you see this C code: Even without any documentation, you can immediately know several important things: Written in Forth [3] : How can you know the arity of the functions without adding explicit comments? Sure, if you have a handful of words like bar and foo you know like the back of your hand, this is easy. But imagine reading a large, unfamiliar code base full of code like this and trying to comprehend it. The source code of my goforth project is on GitHub ; both implementations are there, with a comprehensive test harness that tests both. The learn Forth itself, I found these resources very useful: To learn how to implement Forth: Implementing Forth is a great self-improvement project for a coder; there's a pleasantly challenging hump of understanding to overcome, and you gain valuable insights into stack machines, interpretation vs. compilation and mixing these levels of abstraction in cool ways. Also, implementing programming languages from scratch is fun! It's hard to beat the feeling of getting to interact with your implementation for the first time, and then iterating on improving it and making it more featureful. Just one more word ! Which is another deviation from the norm. Forth is really supposed to be implemented in assembly - this is what it was designed for, and it's very clear from its structure that it must be so in order to achieve peak performance. But where's the fun in doing things the way they were supposed to be done? Besides, jonesforth is already a perfectly fine Forth implementation in assembly, so I wouldn't have learned much by just copying it. I had a lot of fun coding in C for this one; it's been a while since I last wrote non-trivial amounts of C, and I found it very enjoyable.

0 views
Xe Iaso 1 months ago

Final Fantasy 14 on macOS with a 36 key keyboard

Earlier this year, I was finally sucked into Final Fantasy 14. I've been loving my time in it, but most of my playtime was on my gaming tower running Fedora. I knew that the game does support macOS, and I did get it working on my MacBook for travel, but there was one problem: I wasn't able to get my bars working with mouse and keyboard. A 36 key keyboard and MMO mouse combination for peak gaming. Final Fantasy 14 has a ridiculous level of customization. Every UI element can be moved and resized freely. Every action your player character can take is either bindable to arbitrary keybinds or able to be put in hotbars. Here's my hotbars for White Mage: My bars for the White Mage job, showing three clusters of actions along with a strip of quick actions up top. My bars have three "layers" to them: I have things optimized so that the most common actions I need to do are on the base layer. This includes spells like my single target / area of effect healing spells and my burst / damage over time spells. However, critical things like health regeneration, panic button burst healing, shields, and status dispelling are all in the shift and control layers. When I don't have instinctive access to these spells with button combos, I have to manually click on the buttons. This sucks. I ended up fixing this by installing Karabiner Elements , giving it access to the accessibility settings it needs, and enabling my mouse to be treated as a keyboard in its configuration UI. There's some other keyboard hacks that I needed to do. My little split keyboard runs QMK , custom keyboard firmware written in C that has a stupid number of features. In order to get this layout working with FFXIV, I had to use a combination of the following features: Here is what my keymap looks like: I use the combination of this to also do programming. I've been doing a few full blown Anubis features via this keyboard such as log filters . I'm still not up to full programming speed with it, but I'm slowly internalizing the keymap and getting faster with practice. Either way, Final Fantasy 14 is my comfort game and now I can play it on the go with all the buttons I could ever need. I hope this was interesting and I'm going to be publishing more of these little "how I did a thing" posts like this in the future. Let me know what you think about this!

0 views
matklad 2 months ago

Zig's Lovely Syntax

It’s a bit of a silly post, because syntax is the least interesting detail about the language, but, still, I can’t stop thinking how Zig gets this detail just right for the class of curly-braced languages, and, well, now you’ll have to think about that too. On the first glance, Zig looks almost exactly like Rust, because Zig borrows from Rust liberally. And I think that Rust has great syntax, considering all the semantics it needs to express (see “Rust’s Ugly Syntax” ). But Zig improves on that, mostly by leveraging simpler language semantics, but also through some purely syntactical tasteful decisions. How do you spell a number ninety-two? Easy, . But what type is that? Statically-typed languages often come with several flavors of integers: , , . And there’s often a syntax for literals of a particular types: , , . Zig doesn’t have suffixes, because, in Zig, all integer literals have the same type: : The value of an integer literal is known at compile time and is coerced to a specific type on assignment or ascription: To emphasize, this is not type inference, this is implicit comptime coercion. This does mean that code like generally doesn’t work, and requires an explicit type. Raw or multiline strings are spelled like this: This syntax doesn’t require a special form for escaping itself: It nicely dodges indentation problems that plague every other language with a similar feature. And, the best thing ever: lexically, each line is a separate token. As Zig has only line-comments, this means that is always whitespace. Unlike most other languages, Zig can be correctly lexed in a line-by-line manner. Raw strings is perhaps the biggest improvement of Zig over Rust. Rust brute-forces the problem with syntax, which does the required job, technically, but suffers from the mentioned problems: indentation is messy, nesting quotes requires adjusting hashes, unclosed raw literal breaks the following lexical structure completely, and rustfmt’s formatting of raw strings tends to be rather ugly. On the plus side, this syntax at least cannot be expressed by a context-free grammar! For the record, Zig takes C syntax (not that C would notice): The feels weird! It will make sense by the end of the post. Here, I want only to note part, which matches the assignment syntax . This is great! This means that grepping for gives you all instances where a field is written to. This is hugely valuable: most of usages are reads, but, to understand the flow of data, you only need to consider writes. Ability to mechanically partition the entire set of usages into majority of boring reads and a few interesting writes does wonders for code comprehension. Where Zig departs from C the most is the syntax for types. C uses a needlessly confusing spiral rule. In Zig, all types are prefix: While pointer type is prefix, pointer dereference is postfix, which is a more natural subject-verb order to read: Zig has general syntax for “raw” identifiers: It is useful to avoid collisions with keywords, or for exporting a symbol whose name is otherwise not a valid Zig identifier. It is a bit more to type than Kotlin’s delightful , but manages to re-use Zig’s syntax for built-ins ( ) and strings. Like, Rust, Zig goes for function declaration syntax. This is such a massive improvement over C/Java style function declarations: it puts token (which is completely absent in traditional C family) and function name next to each other, which means that textual search for allows you to quickly find the function. Then Zig adds a little twist. While in Rust we write The arrow is gone! Now that I’ve used this for some time, I find arrow very annoying to type, and adding to the visual noise. Rust needs the arrow: Rust has lambdas with an inferred return type, and, in a lambda, the return type is optional. So you need some sort of an explicit syntax to tell the parser if there is return type: And it’s understandable that lambdas and functions would want to use compatible syntax. But Zig doesn’t have lambdas, so it just makes the type mandatory. So the main is Related small thing, but, as name of the type, I think I like more than . Zig is using and for binding values to names: This is ok, a bit weird after Rust’s, whose would be in Zig, but not really noticeable after some months. I do think this particular part is not great, because , the more frequent one, is longer. I think Kotlin nails it: , , . Note all three are monosyllable, unlike and ! Number of syllables matters more than the number of letters! Like Rust, Zig uses syntax for ascribing types, which is better than because optional suffixes are easier to parse visually and mechanically than optional prefixes. Zig doesn’t use and and spells the relevant operators as and : This is easier to type and much easier to read, but there’s also a deeper reason why they are not sigils. Zig marks any control flow with a keyword. And, because boolean operators short-circuit, they are control flow! Treating them as normal binary operator leads to an entirely incorrect mental model. For bitwise operations, Zig of course uses and . Both Zig and Rust have statements and expressions. Zig is a bit more statement oriented, and requires explicit returns: Furthermore, because there are no lambdas, scope of return is always clear. Relatedly, the value of a block expression is void. A block is a list of statements, and doesn’t have an optional expression at the end. This removes the semicolon problem — while Rust rules around semicolons are sufficiently clear (until you get to macros), there’s some constant mental overhead to getting them right all the time. Zig is more uniform and mechanical here. If you need a block that yields a value, Zig supports a general syntax for breaking out of a labeled block: Rust makes pedantically correct choice regarding s: braces are mandatory: This removes the dreaded “dangling else” grammatical ambiguity. While theoretically nice, it makes -expression one-line feel too heavy. It’s not the braces, it’s the whitespace around them: But the ternary is important! Exploding a simple choice into multi-line condition hurts readability. Zig goes with the traditional choice of making parentheses required and braces optional: By itself, this does create a risk of style bugs. But in Zig formatter (non-configurable, user-directed) is a part of the compiler, and formatting errors that can mask bugs are caught during compilation. For example, is an error due to inconsistent whitespace around the minus sign, which signals a plausible mixup of infix and binary minus. No such errors are currently produced for incorrect indentation (the value add there is relatively little, given ), but this is planned. NB: because Rust requires branches to be blocks, it is forced to make synonym with . Otherwise, the ternary would be even more unusable! Syntax design is tricky! Whether you need s and whether you make or mandatory in ifs are not orthogonal! Like Python, Zig allows on loops. Unlike Python, loops are expressions, which leads to a nicely readable imperative searches: Zig doesn’t have syntactically-infinite loop like Rust’s or Go’s . Normally I’d consider that a drawback, because these loops produce different control flow, affecting reachability analysis in the compiler, and I don’t think it’s great to make reachability dependent on condition being visibly constant. But! As Zig places semantics front and center, and the rules for what is and isn’t a comptime constant are a backbone of every feature, “anything equivalent to ” becomes sufficiently precise. Incidentally, these days I tend to write “infinite” loops as Almost always there is an up-front bound for the number of iterations until the break, and its worth asserting this bound, because debugging crashes is easier than debugging hangs. , , , , and all use the same Ruby/Rust inspired syntax for naming captured values: I like how the iterator comes first, and then the name of an item follows, logically and syntactically. I have a very strong opinion about variable shadowing. It goes both ways: I spent hours debugging code which incorrectly tried to use a variable that was shadowed by something else, but I also spent hours debugging code that accidentally used a variable that should have been shadowed! I really don’t know whether on balance it is better to forbid or encourage shadowing! Zig of course forbids shadowing, but what’s curious is that it’s just one episode of the large crusade against any complexity in name resolution. There’s no “prelude”, if you want to use anything from std, you need to import it: There are no glob imports, if you want to use an item from std, you need to import it: Zig doesn’t have inheritance, mixins, argument-dependent lookup, extension functions, implicit or traits, so, if you see , that is guaranteed to be a boring method declared on type. Similarly, while Zig has powerful comptime capabilities, it intentionally disallows declaring methods at compile time. Like Rust, Zig used to allow a method and a field to share a name, because it actually is syntactically clear enough at the call site which is which. But then this feature got removed from Zig. More generally, Zig doesn’t have namespaces. There can be only one kind of in scope, while Rust allows things like I am astonished at the relative lack of inconvenience in Zig’s approach. Turns out that is all the syntax you’ll ever need for accessing things? For the historically inclined, see “The module naming situation” thread in the rust mailing list archive to learn the story of how rust got its syntax. The lack of namespaces touches on the most notable (by its absence) feature of Zig syntax, which deeply relates to the most profound aspect of Zig’s semantics. Everything is an expression. By which I mean, there’s no separate syntactic categories of values, types, and patterns. Values, types, and patterns are of course different things. And usually in the language grammar it is syntactically obvious whether a particular text fragment refers to a type or a value: So the standard way is to have separate syntax families for the three categories, which need to be internally unambiguous, but can be ambiguous across the categories because the place in the grammar dictates the category: when parsing , everything until is a pattern, stuff between and is a type, and after we have a value. There are two problems here. First, there’s a combinatorial explosion of sorts in the syntax, because, while three categories describe different things, it turns out that they have the same general tree-ish shape. The second problem is that it might be hard to maintain category separation in the grammar. Rust started with the three categories separated by a bright line. But then, changes happen. Originally, Rust only allowed syntax for assignment. But today you can also write to do unpacking like Similarly, the turbofish used to move the parser from the value to the type mode, but now const parameters are values that can be found in the type position! The alternative is not to pick this fight at all. Rather than trying to keep the categories separately in the syntax, use the same surface syntax to express all three, and categorize later, during semantic analysis. In fact, this is already happens in the example — these are different things! One is a place (lvalue) and another is a “true” value (rvalue), but we use the same syntax for both. I don’t think such syntactic unification necessarily implies semantic unification, but Zig does treat everything uniformly, as a value with comptime and runtime behavior (for some values, runtime behavior may be missing, for others — comptime): The fact that you can write an where a type goes is occasionally useful. But the fact that simple types look like simple values syntactically consistently make the language feel significantly less busy. As a special case of everything being an expression, instances of generic types look like this: Just a function call! Though, there’s some resistance to trickery involved to make this work. Usually, languages rely on type inference to allow eliding generic arguments. That in turn requires making argument syntax optional, and that in turn leads to separating generic and non-generic arguments into separate parameter lists and some introducer sigil for generics, like or . Zig solves this syntactic challenge in the most brute-force way possible. Generic parameters are never inferred, if a function takes 3 comptime arguments and 2 runtime arguments, it will always be called with 5 arguments syntactically. Like with the (absence of) importing flourishes, a reasonable reaction would be “wait, does this mean that I’ll have to specify the types all the time?” And, like with import, in practice this is a non-issue. The trick are comptime closures. Consider a generic : We have to specify type when creating an instance of an . But subsequently, when we are using the array list, we don’t have to specify the type parameter again, because the type of variable already closes over . This is the major truth of object-orienting programming, the truth so profound that no one even notices it: in real code, 90% of functions are happiest as (non-virtual) methods. And, because of that, the annotation burden in real-world Zig programs is low. While Zig doesn’t have Hindley-Milner constraint-based type inference, it relies heavily on one specific way to propagate types. Let’s revisit the first example: This doesn’t compile: and are different values, we can’t select between two at runtime because they are different. We need to coerce the constants to a specific runtime type: But this doesn’t kick the can sufficiently far enough and essentially reproduces the with two incompatible branches. We need to sink coercion down the branches: And that’s exactly how Zig’s “Result Location Semantics” works. Type “inference” runs a simple left-to-right tree-walking algorithm, which resembles interpreter’s . In fact, is exactly what happens. Zig is not a compiler, it is an interpreter. When evaluates an expression, it gets: When interpreting code like the interpreter passes the result location ( ) and type down the tree of subexpressions. If branches store result directly into object field (there’s a inside each branch, as opposed to one after the ), and each coerces its comptime constant to the appropriate runtime type of the result. This mechanism enables concise syntax for specifying enums: When evaluates the switch, it first evaluates the scrutinee, and realizes that it has type . When evaluating arm, it sets result type to for the condition, and a literal gets coerced to . The same happens for the second arm, where result type further sinks down the . Result type semantics also explains the leading dot in the record literal syntax: Syntactically, we just want to disambiguate records from blocks. But, semantically, we want to coerce the literal to whatever type we want to get out of this expression. In Zig, is a shorthand for . I must confess that did weird me out a lot at first during writing code (I don’t mind reading the dot). It’s not the easiest thing to type! But that was fixed once I added snippet, expanding to . The benefits to lightweight record literal syntax are huge, as they allow for some pretty nice APIs. In particular, you get named and default arguments for free: I don’t really miss the absence of named arguments in Rust, you can always design APIs without them. But they are free in Zig, so I use them liberally. Syntax wise, we get two features (calling functions and initializing objects) for the price of one! Finally, the thing that weirds out some people when they see Zig code, and makes others reconsider their choice GitHub handles, even when they haven’t seen any Zig: syntax for built-in functions. Every language needs to glue “userspace” code with primitive operations supported by the compiler. Usually, the gluing is achieved by making the standard library privileged and allowing it to define intrinsic functions without bodies, or by adding ad-hoc operators directly to the language (like Rust’s ). And Zig does have a fair amount of operators, like or . But the release valve for a lot of functionality are built-in functions in distinct syntactic namespace, so Zig separates out , , , , , , , , , and . There’s no need to overload casting when you can give each variant a name. There’s also for type ascription. The types goes first, because the mechanism here is result type semantics: evaluates the first argument as a type, and then uses that as the type for the second argument. Curiously, I think actually can be implemented in the userspace: In Zig, a type of function parameter may depend on values of preceding (comptime) ones! My favorite builtin is . First, it’s the most obvious way to import code: Its crystal clear where the file comes from. But, second, it is an instance of reverse syntax sugar. You see, import isn’t really a function. You can’t do The argument of has to be a string, syntactically. It really is syntax, except that the function-call form is re-used, because it already has the right shape. So, this is it. Just a bunch of silly syntactical decisions, which add up to a language which is positively enjoyable to read. As for big lessons, obviously, the less features your language has, the less syntax you’ll need. And less syntax is generally good, because varied syntactic constructs tend to step on each other toes. Languages are not combinations of orthogonal aspects. Features tug and pull the language in different directions and their combinations might turn to be miraculous features in their own right, or might drag the language down. Even with a small feature-set fixed, there’s still a lot of work to pick a good concrete syntax: unambiguous to parse, useful to grep, easy to read and not to painful to write. A smart thing is of course to steal and borrow solutions from other languages, not because of familiarity, but because the ruthless natural selection tends to weed out poor ideas. But there’s a lot of inertia in languages, so there’s no need to fear innovation. If an odd-looking syntax is actually good, people will take to it. Is there anything about Zig’s syntax I don’t like? I thought no, when starting this post. But in the process of writing it I did discover one form that annoys me. It is the while with the increment loop: This is two-thirds of a C-style loop (without the declarator), and it sucks for the same reason: control flow jumps all over the place and is unrelated to the source code order. We go from condition, to the body, to the increment. But in the source order the increment is between the condition and the body. In Zig, this loop sucks for one additional reason: that separating the increment I think is the single example of control flow in Zig that is expressed by a sigil, rather than a keyword. This form used to be rather important, as Zig lacked a counting loop. It has form now, so I am tempted to call the while-with-increment redundant. Annoyingly, is almost equivalent to But not exactly: if contains a , or , the version would run the one extra time, which is useless and might be outright buggy. Oh well.

0 views
Julia Evans 4 months ago

Using `make` to compile C programs (for non-C-programmers)

I have never been a C programmer but every so often I need to compile a C/C++ program from source. This has been kind of a struggle for me: for a long time, my approach was basically “install the dependencies, run , if it doesn’t work, either try to find a binary someone has compiled or give up”. “Hope someone else has compiled it” worked pretty well when I was running Linux but since I’ve been using a Mac for the last couple of years I’ve been running into more situations where I have to actually compile programs myself. So let’s talk about what you might have to do to compile a C program! I’ll use a couple of examples of specific C programs I’ve compiled and talk about a few things that can go wrong. Here are three programs we’ll be talking about compiling: This is pretty simple: on an Ubuntu system if I don’t already have a C compiler I’ll install one with: This installs , , and . The situation on a Mac is more confusing but it’s something like “install xcode command line tools”. Unlike some newer programming languages, C doesn’t have a dependency manager. So if a program has any dependencies, you need to hunt them down yourself. Thankfully because of this, C programmers usually keep their dependencies very minimal and often the dependencies will be available in whatever package manager you’re using. There’s almost always a section explaining how to get the dependencies in the README, for example in paperjam ’s README, it says: To compile PaperJam, you need the headers for the libqpdf and libpaper libraries (usually available as libqpdf-dev and libpaper-dev packages). You may need (found in AsciiDoc ) for building manual pages. So on a Debian-based system you can install the dependencies like this. If a README gives a name for a package (like ), I’d basically always assume that they mean “in a Debian-based Linux distro”: if you’re on a Mac will not work. I still have not 100% gotten the hang of developing on a Mac yet so I don’t have many tips there yet. I guess in this case it would be if you’re using Homebrew. Some C programs come with a and some instead come with a script called . For example, if you download sqlite’s source code , it has a script in it instead of a Makefile. My understanding of this script is: I think there might be some options you can pass to get the script to produce a different but I have never done that. The next step is to run to try to build a program. Some notes about : Here’s an error I got while compiling on my Mac: Over the years I’ve learned it’s usually best not to overthink problems like this: if it’s talking about , there’s a good change it just means that I’ve done something wrong with how I’m including the dependency. Now let’s talk about some ways to get the dependency included in the right way. Before we talk about how to fix dependency problems: building C programs is split into 2 steps: It’s important to know this when building a C program because sometimes you need to pass the right flags to the compiler and linker to tell them where to find the dependencies for the program you’re compiling. If I run on my Mac to install , I get this error: This is not because is not installed on my system (it actually is!). But the compiler and linker don’t know how to find the library. To fix this, we need to: And we can get to pass those extra parameters to the compiler and linker using environment variables! To see how this works: inside ’s Makefile you can see a bunch of environment variables, like here: Everything you put into the environment variable gets passed to the linker ( ) as a command line argument. sometimes define their own environment variables that they pass to the compiler/linker, but also has a bunch of “implicit” environment variables which it will automatically pass to the C compiler and linker. There’s a full list of implicit environment variables here , but one of them is , which gets automatically passed to the C compiler. (technically it would be more normal to use for this, but this particular hardcodes so setting was the only way I could find to set the compiler flags without editing the ) I learned thanks to @zwol that there are actually two ways to pass environment variables to : The difference between them is that will override the value of set in the but won’t. I’m not sure which way is the norm but I’m going to use the first way in this post. Now that we’ve talked about how and get passed to the compiler and linker, here’s the final incantation that I used to get the program to build successfully! This passes to the compiler and to the linker. Also I don’t want to pretend that I “magically” knew that those were the right arguments to pass, figuring them out involved a bunch of confused Googling that I skipped over in this post. I will say that: Yesterday I discovered this cool tool called qf which you can use to quickly open files from the output of . is in a big directory of various tools, but I only wanted to compile . So I just compiled , like this: Basically if you know (or can guess) the output filename of the file you’re trying to build, you can tell to just build that file by running I sometimes write 5-line C programs with no dependencies, and I just learned that if I have a file called , I can just compile it like this without creating a : It gets automaticaly expanded to , which saves a bit of typing. I have no idea if I’m going to remember this (I might just keep typing anyway) but it seems like a fun trick. If you’re having trouble building a C program, maybe other people had problems building it too! Every Linux distribution has build files for every package that they build, so even if you can’t install packages from that distribution directly, maybe you can get tips from that Linux distro for how to build the package. Realizing this (thanks to my friend Dave) was a huge ah-ha moment for me. For example, this line from the nix package for says: This is basically saying “pass the linker flag to build this on a Mac”, so that’s a clue we could use to build it. That same file also says . I’m not sure what this means, but when I try to build the package I do get an error about something called a , so I guess that’s somehow related to the “PointerHolder transition”. Once you’ve managed to compile the program, probably you want to install it somewhere! Some s have an target that let you install the tool on your system with . I’m always a bit scared of this (where is it going to put the files? what if I want to uninstall them later?), so if I’m compiling a pretty simple program I’ll often just manually copy the binary to install it instead, like this: Once I figured out how to do all of this, I realized that I could use my new knowledge to contribute a package to Homebrew! Then I could just on future systems. The good thing is that even if the details of how all of the different packaging systems, they fundamentally all use C compilers and linkers. I think all of this is an interesting example of how it can useful to understand some basics of how C programs work (like “they have header files”) even if you’re never planning to write a nontrivial C program if your life. It feels good to have some ability to compile C/C++ programs myself, even though I’m still not totally confident about all of the compiler and linker flags and I still plan to never learn anything about how autotools works other than “you run to generate the ”. Two things I left out of this post:

0 views
Corrode 5 months ago

Flattening Rust's Learning Curve

I see people make the same mistakes over and over again when learning Rust. Here are my thoughts (ordered by importance) on how you can ease the learning process. My goal is to help you save time and frustration. Stop resisting. That’s the most important lesson. Accept that learning Rust requires adopting a completely different mental model than what you’re used to. There are a ton of new concepts to learn like lifetimes, ownership, and the trait system. And depending on your background, you’ll need to add generics, pattern matching, or macros to the list. Your learning pace doesn’t have much to do with whether you’re smart or not or if you have a lot of programming experience. Instead, what matters more is your attitude toward the language . I have seen junior devs excel at Rust with no prior training and senior engineers struggle for weeks/months or even give up entirely. Leave your hubris at home. Treat the borrow checker as a co-author, not an adversary. This reframes the relationship. Let the compiler do the teaching: for example, this works great with lifetimes, because the compiler will tell you when a lifetime is ambiguous. Then just add it but take the time to reason about why the compiler couldn’t figure it out itself. If you try to compile this, the compiler will ask you to add a lifetime parameter. It provides this helpful suggestion: So you don’t have to guess what the compiler wants and can follow its instructions. But also sit down and wonder why the compiler couldn’t figure it out itself. Most of the time when fighting the compiler it is actually exposing a design flaw. Similarly, if your code gets overly verbose or looks ugly, there’s probably a better way. Declare defeat and learn to do it the Rust way. If you come from a dynamic language like Python, you’ll find that Rust is more verbose in general. Most of it just comes from type annotations, though. Some people might dismiss Rust as being “unelegant” or “ugly”, but the verbosity actually serves a good purpose and is immensely helpful for building large-scale applications: Turn on all clippy lints on day one – even the pedantic ones. Run the linter and follow the suggestions religiously. Don’t skip that step once your program compiles. Resistance is futile. The longer you refuse to learn, the longer you will suffer; but the moment you let your guard down is the moment you’ll start to learn. Forget what you think you knew about programming and really start to listen to what the compiler, the standard library, and clippy are trying to tell you. I certainly tried to run before I could walk. That alone cost me a lot of precious time. Don’t make it too hard on yourself in the beginning. Here are some tips: Don’t introduce too many new concepts at the same time! Instead, while you learn about a new concept, have an editor open and write out a few examples. What helped was to just write some code in the Rust playground and try to get it to compile. Write super small snippets (e.g., one for one concept) instead of using one big “tutorial” repo. Get into the habit of throwing most of your code away. I still do that and test out ideas in the playground or when I brainstorm with clients. For instance, here’s one of my favorite code snippets to explain the concept of ownership: Can you fix it? Can you explain it? Ask yourself what would change if was an . If Rust code looks scary to you, break it down . Write your own, simpler version, then slowly increase the complexity. Rust is easier to write than to read. By writing lots of Rust, you will also learn how to read it better as well. How you do anything is how you do everything. – An ancient Rust proverb You can be sloppy in other languages, but not in Rust. That means you have to be accurate while you code or the code just won’t compile. The expectation is that this approach will save you debugging time in the future. I found that the people who learn Rust the fastest all have great attention to detail. If you try to just get things done and move on, you will have a harder time than if you aim to do things right on your first try. You will have a much better time if you re-read your code to fix stupid typos before pressing “compile.” Also build a habit of automatically adding and where necessary as you go. A good example of someone who thinks about these details while coding is Tsoding. For example, watch this stream where he builds a search engine in Rust from scratch to see what I mean. I think you can learn this skill as long as you’re putting in your best effort and give it some time. With today’s tooling it is very easy to offload the bulk of the work to the computer. Initially, it will feel like you’re making quick progress, but in reality, you just strengthen bad habits in your workflow. If you can’t explain what you wrote to someone else or if you don’t know about the tradeoffs/assumptions a part of your code makes, you took it too far. Often, this approach stems from a fear that you’re not making progress fast enough. But you don’t have to prove to someone else that you’re clever enough to pick up Rust very quickly. To properly learn Rust you actually have to write a lot of code by hand. Don’t be a lurker on Reddit, reading through other people’s success stories. Have some skin in the game! Put in the hours because there is no silver bullet. Once it works, consider open sourcing your code even if you know it’s not perfect. LLMs are like driving a car on auto-pilot. It’s comfortable at first, but you won’t feel in control and slowly, that uneasy feeling will creep in. Turn off the autopilot while learning. A quick way to set you up for success is to learn by writing code in the Rust Playground first. Don’t use LLMs or code completion. Just type it out! If you can’t, that means you haven’t fully internalized a concept yet. That’s fine! Go to the standard library and read the docs. Take however long it takes and then come back and try again. Slow is steady and steady is fast. Muscle memory in programming is highly underrated. People will tell you that this is what code completion is for, but I believe it’s a requirement to reach a state of flow: if you constantly blunder over syntax errors or, worse, just wait for the next auto-completion to make progress, that is a terrible developer experience. When writing manually, you will make more mistakes. Embrace them! These mistakes will help you learn to understand the compiler output. You will get a “feeling” for how the output looks in different error scenarios. Don’t gloss over these errors. Over time you will develop an intuition about what feels “rustic.” Another thing I like to do is to run “prediction exercises” where I guess if code will compile before running it. This builds intuition. Try to make every program free of syntax errors before you run it. Don’t be sloppy. Of course, you won’t always succeed, but you will get much better at it over time. Read lots of other people’s code. I recommend , for example, which is some of the best Rust code out there. Don’t be afraid to get your hands dirty. Which areas of Rust do you avoid? What do you run away from? Focus on that. Tackle your blind spots. Track your common “escape hatches” (unsafe, clone, etc.) to identify your current weaknesses. For example, if you are scared of proc macros, write a bunch of them. After you’re done with an exercise, break it! See what the compiler says. See if you can explain what happens. A poor personal version is better than a perfect external crate (at least while learning). Write some small library code yourself as an exercise. Notable exceptions are probably and , which can save you time dealing with JSON inputs and setting up error handling that you can spend on other tasks as long as you know how they work. Concepts like lifetimes are hard to grasp. Sometimes it helps to draw how data moves through your system. Develop a habit to explain concepts to yourself and others through drawing. I’m not sure, but I think this works best for “visual”/creative people (in comparison to highly analytical people). I personally use excalidraw for drawing. It has a “comicy” feel, which takes the edge off a bit. The implication is that it doesn’t feel highly accurate, but rather serves as a rough sketch. Many good engineers (as well as great Mathematicians and Physicists) are able to visualize concepts with sketches. In Rust, sketches can help to visualize lifetimes and ownership of data or for architecture diagrams. Earlier I said you should forget everything you know about programming. How can I claim now that you should build on top of what you already know? What I meant is that Rust is the most different in familiar areas like control flow handling and value passing. E.g., mutability is very explicit in Rust and calling a function typically “moves” its arguments. That’s where you have to accept that Rust is just different and learn from first principles . However, it is okay to map Rust concepts to other languages you already know. For instance, “a trait is a bit like an interface” is wrong, but it is a good starting point to understand the concept. Here are a few more examples: And if you have a functional background, it might be: The idea is that mapping concepts helps fill in the gaps more quickly. Map what you already know from another language (e.g., Python, TypeScript) to Rust concepts. As long as you know that there are subtle differences, I think it’s helpful. I don’t see people mention this a lot, but I believe that Rosetta Code is a great resource for that. You basically browse their list of tasks, pick one you like and start comparing the Rust solution with the language you’re strongest in. Also, port code from a language you know to Rust. This way, you don’t have to learn a new domain at the same time as you learn Rust. You can build on your existing knowledge and experience. Finally, find other people who come from the same background as you. Read their blogs where they talk about their experiences learning Rust. Write down your experiences as well. I find that people who tend to guess their way through challenges often have the hardest time learning Rust. In Rust, the details are everything. Don’t gloss over details, because they always reveal some wisdom about the task at hand. Even if you don’t care about the details, they will come back to bite you later. For instance, why do you have to call on a thing that’s already a string? Those stumbling blocks are learning opportunities. It might look like a waste of time to ask these questions and means that it will take longer to finish a task, but it will pay off in the long run. Reeeeeally read the error messages the compiler prints. Everyone thinks they do this, but time and again I see people look confused while the solution is right there in their terminal. There are as well; don’t ignore those. This alone will save you sooo much time. Thank me later. You might say that is true for every language, and you’d be right. But in Rust, the error messages are actually worth your time. Some of them are like small meditations: opportunities to think about the problem at a deeper level. If you get any borrow-checker errors, refuse the urge to guess what’s going on. Instead of guessing, walk through the data flow by hand (who owns what and when). Try to think it through for yourself and only try to compile again once you understand the problem. The key to good Rust code is through its type system. It’s all in the type system. Everything you need is hidden in plain sight. But often, people skip too much of the documentation and just look at the examples. What few people do is read the actual function documentation . You can even click through the standard library all the way to the source code to read the thing they are using. There is no magic (and that’s what’s so magical about it). You can do that in Rust much better than in most other languages. That’s because Python for example is written in C, which requires you to cross that language boundary to learn what’s going on. Similarly, the C++ standard library isn’t a single, standardized implementation, but rather has several different implementations maintained by different organizations. That makes it super hard to know what exactly is going on . In Rust, the source code is available right inside the documentation. Make good use of that! Function signatures tell a lot! The sooner you will embrace this additional information, the quicker you will be off to the races with Rust. If you have the time, read interesting parts of the standard library docs. Even after years, I always learn something when I do. Try to model your own projects with types first. This is when you start to have way more fun with the language. It feels like you have a conversation with the compiler about the problem you’re trying to solve. For example, once you learn how concepts like expressions, iterators and traits fit together, you can write more concise, readable code. Once you learn how to encode invariants in types, you can write more correct code that you don’t have to run to test. Instead, you can’t compile incorrect code in the first place. Learn Rust through “type-driven development” and let the compiler errors guide your design. Before you start, shop around for resources that fit your personal learning style. To be honest, there is not that much good stuff out there yet. On the plus side, it doesn’t take too long to go through the list of resources before settling on one specific platform/book/course. The right resource depends on what learner you are. In the long run, finding the right resource saves you time because you will learn quicker. I personally don’t like doing toy exercises that others have built out for me. That’s why I don’t like Rustlings too much; the exercises are not “fun” and too theoretical. I want more practical exercises. I found that Project Euler or Advent of Code work way better for me. The question comes up quite often, so I wrote a blog post about my favorite Rust learning resources . I like to watch YouTube, but exclusively for recreational purposes. In my opinion, watching ThePrimeagen is for entertainment only. He’s an amazing programmer, but trying to learn how to program by watching someone else do it is like trying to learn how to become a great athlete by watching the Olympics. Similarly, I think we all can agree that Jon Gjengset is an exceptional programmer and teacher, but watching him might be overwhelming if you’re just starting out. (Love the content though!) Same goes for conference talks or podcasts: they are great for context, and for soft-skills, but not for learning Rust. Instead, invest in a good book if you can. Books are not yet outdated and you can read them offline, add personal notes, type out the code yourself and get a “spatial overview” of the depth of the content by flipping through the pages. Similarly, if you’re serious about using Rust professionally, buy a course or get your boss to invest in a trainer. Of course, I’m super biased here as I run a Rust consultancy, but I truly believe that it will save you and your company countless hours and will set you up for long-term success. Think about it: you will work with this codebase for years to come. Better make that experience a pleasant one. A good trainer, just like a good teacher, will not go through the Rust book with you, but watch you program Rust in the wild and give you personalized feedback about your weak spots. “Shadow” more experienced team members or friends. Don’t be afraid to ask for a code review on Mastodon or the Rust forum and return the favor and do code reviews there yourself. Take on opportunities for pair programming. This is such a great way to see if you truly understood a concept. Don’t be afraid to say “I don’t know.” Then go and explore the answer together by going straight to the docs. It’s way more rewarding and honest. Help out with OSS code that is abandoned. If you put in a solid effort to fix an unmaintained codebase, you will help others while learning how to work with other people’s Rust code. Read code out loud and explain it. There’s no shame in that! It helps you “serialize” your thoughts and avoid skipping important details. Take notes. Write your own little “Rust glossary” that maps Rust terminology to concepts in your business domain. It doesn’t have to be complete and just has to serve your needs. Write down things you found hard and things you learned. If you find a great learning resource, share it! If you learn Rust because you want to put it on your CV, stop. Learn something else instead. I think you have to actually like programming (and not just the idea of it) to enjoy Rust. If you want to be successful with Rust, you have to be in it for the long run. Set realistic expectations: You won’t be a “Rust grandmaster” in a week but you can achieve a lot in a month of focused effort. There is no silver bullet, but if you avoid the most common ways to shoot yourself in the foot, you pick up the language much faster. Rust is a day 2 language. You won’t “feel” as productive as in your first week of Go or Python, but stick it out and it will pay off. Good luck and have fun!

0 views
Uros Popovic 5 months ago

Bare metal printf - C standard library without OS

Guide to implementing a C standard library for bare metal hardware using Newlib, enabling powerful functions like printf and malloc without relying on an operating system. This step-by-step guide demonstrates how to configure and integrate Newlib on RISC-V architecture, showing you how to redirect standard I/O through UART for embedded systems development.

0 views
Schneems 7 months ago

Installing the sassc Ruby gem on a Mac. A debugging story

I’m not exactly sure about the timeline, but at some point, stopped working for me on my Mac (ARM). Initially, I thought this was because that gem was no longer maintained, and the last release was in 2020, but I was wrong. It’s 100% installable today. Read the rest to find out the real culprit and how to fix it. FWIW some folks on lobste.rs suggested switching to sass-embedded for sass needs. This post still, works but into the future it might not. In this post I’ll explain some things about native extensions libraries in Ruby and in the process tell you how to fix this error below if you’re getting it on your Mac: You can install the on your Mac by: If you want to know more about native compilation or my debugging process, read on! There might be a simpler way to solve the problem (such as directly editing the rbconfig file), but I’m comfortable sharing the above steps because that’s what I’ve done. If you fixed this differently, post the solution on your own site or in the comments somewhere. When I get an error, it makes sense to search for it and ask an LLM (if that’s your thing). I did both. GitHub copilot suggested that I make sure command-line tools are installed and that is installed via homebrew. This was unhelpful, but it’s worth double-checking. Searching brought me to https://github.com/sass/sassc-ruby/issues/248 . This brought me to https://github.com/sass/sassc-ruby/issues/225#issuecomment-2391129846 . Suggesting that the problem is related to RbConfig and native extensions. These have the fix in there, but don’t go into detail on the why the fix works. This post attempts to dig deeper using a debugging mindset. Meta narrative: This article skips between explaining things I know to be true and debugging via doing. Skip any explanations you feel are tedious. Skip if: You know what a native extension is Most Ruby libraries are plain Ruby code. For an example look at https://github.com/zombocom/mini_histogram . When you , it downloads the source code, and that’s all that’s needed to run it (well, that and a Ruby version installed on the machine). The term “native extensions” refers to libraries that use Ruby’s C API or FFI in some way. There are a few reasons why someone would want to do this: For developers who haven’t used much C or C++, it’s useful to know that system-installed packages are how they (mostly) share code. There’s no rubygems.org for C packages. Things like for Ubuntu might be conflated as a “C package manager,” but it’s really like (for Mac), where it installs things globally. Then, when you compile a program in C, it can dynamically or statically link to other libraries to use them. Back to native extensions: When a gem with a native extension is installed the source code is downloaded but then a secondary compilation process is invoked. Here’s a tutorial on creating a native extension . It utilizes a tool called rake-compiler . But under the hood it effectively boils down to when you it will run compilation code such as on the system. This process generates compiled binaries, these binaries are compiled against a specific CPU architecture that is native to the machine you’re on, hence why they’re called native extensions. You’re using native (binary) code to extend Ruby’s capabilities. Skip if: You understand why CPP files would be found in the gem Compiling code is hard. Or rather, dependency management is hard, and compiling code requires that the platform have certain dependencies installed; therefore, compiling code is hard. To make life easier, one common pattern that Ruby developers do is to vendor in dependencies into their native extension gem. Rather than assuming is installed on the system in a location that is easy to find, it can instead bring that code along with it. Here you can see that sassc from brings C++ source code from libsass: In this case may have dependencies that it hasn’t vendored and it expects to find on the system, but the key here is that when you it needs to not just its own bridge code (using Ruby’s C API), but it also needs to compile as well. That is where the errors are coming from, it’s not able to compile these C++ files: For completeness: There’s another type of vendoring that native-extension gems can do. They can statically compile and vendor in a binary. This bypasses the need to and is much faster, but moves the burden to the gem maintainer. Here’s an example where Nokogiri 1.18.4 is precompiled to run on my ARM Mac . You don’t need to know this for debugging the install problem, since that process isn’t being used here. When debugging, I like to remove layers of abstraction when possible to boil the problem down to its core essence. You might think “I cannot run “ is the problem, but really that’s the context; the real problem is that within that process, the command fails. The output of the command isn’t terribly well structured, but there are hints that this is the core problem: This is saying, “When I am in this directory” and “I run this command ” then I get this output. When someone is experiencing an exception on their Rails app, I encourage them to try copying that code into a session to reproduce the problem without the overhead of the request/response cycle. This helps reduce the scope and removes a layer of abstraction. Here removing abstraction will be manually go into that directory and run . Doing this gave me the same error: I was curious about how to get more information out of and found a SO post suggesting that will list out the commands. From or I see this description: Running that gave me some output: If you’re familiar with the output above you probably spotted the problem. If not, let’s detour and explain what this make tool even is. Skip this if you know what make is and how to write a GNU make describes itself as: GNU Make  is a tool which controls the generation of executables and other non-source files of a program from the program’s source files. The library Rake is a similar concept implemented in Ruby. The name “Rake” is short for “Ruby (M)ake.” In Rake, you can define a task and its prerequisites. The Rake tool will resolve those to ensure they’re run in the correct order without having to run them multiple times. This is commonly used for database migrations and generating assets for a web app, such as CSS and JS. Technically, that’s all Make does as well, it allows you to define tasks in a reusable way, and it handles some of the logic of execution. In practice, make has become the go-to composition tool for compiling C programs. In that world there are projects that don’t even tell you how to build the binaries because they expect you to in the same way some Ruby developers might forget instructions on adding a gem to the Gemfile in the README of their rubygem. You can see a makefile in action following Ruby’s instructions on compilation At the end of the day, does very little. It’s almost more like its own language that happens to be useful for compiling code rather than a “compiling code” tool. The result is that the bulk of the logic comes from the contents of the Makefile and what the developer put in there rather than the Make tool itself. The output ends up being indistinguishable from a bunch of shell scripts in a trenchcoat. Now we know that does very little and we have its output we see two lines (I added a space for clarity): We could remove from the equation by running them directly: That worked as expected, what about the next line? It starts with which, from the manual page So no matter what comes after this command, it will simply exit non-zero. This command can never work. This seems odd, definetly not what the author of this makefile intended. If this is the bug, and I think it is, where is that coming from? Is it dynamic from something in the environment (environment variables) or is it coming from shelling out to some other utility on disk or is it coming from some config file? Or is it static? Is it baked in already. Re-running with env vars (mentioned in the GitHub comments) such as has no effect. It’s the same output. This leads me to believe it’s something static. Looking at the contents of the Makefile: Huh, that’s weird. Where is that used? That last line comes from this code in the Makefile: Skip this if you know make syntax To understand what this is doing, we can write a tiny make program: The indentation under the should be a tab, but your editor or my blogging process might have converted it into a space. Now when we run that: It printed the command and then the output of that command. We’re not limited to static commands though. Modify the file: Here, we’ve extracted the command into a variable and are using that to produce the same effective command. What that means is tells make to replace with which is not what we want. But where did come from? I’m glad you asked. If you search the source code for that line, you won’t find it. That’s because this Makefile is generated. When we looked at native extensions before, notice that I talked about and not about hand-rolling a . Even when we looked at -s Makefile, it wasn’t hardcoded; it came to be after calling and . This Makefile is generated at install time. When you compile Ruby it needs to gather information about the system in order to know how to compile itself. Things like “what compiler are you using” (it could be gcc or clang, for example). Ruby isn’t the only program that needs to know this stuff; native extension code that compiles needs to know it, too. When you compile Ruby it generates a file that contains information that Ruby users can access via RbConfig . From the docs: The module storing Ruby interpreter configurations on building. This file was created by mkconfig.rb when ruby was built. It contains build information for ruby which is used e.g. by mkmf to build compatible native extensions. Any changes made to this file will be lost the next time ruby is built. So that info is what Ruby used at compile time. Where is it? When I looked at that file I saw something alarming: When Ruby was compiled it came to the conclusion that it should use to compile C code: But it mistakenly concluded that it should use the command to compile C++ code (the meaning of these environment variables). It SHOULD be or something like , but it’s not. When the Makefile for the gem is generated it hardcodes into it by mistake because it is pulling that information from the module generated by Ruby at compile time. Why did it record ? Well, I don’t know. I assume it has something to do with the interplay between Ruby’s configuration script and Xcode developer tools. I didn’t debug down that pathway. Since we can fix the problem by re-installing the same version of Ruby with a newer version of the Xcode developer tools, it seems that the problem is in Xcode, but there might be a more complicated interaction involved (perhaps Ruby is doing something Xcode didn’t expect, for example). Thankfully others came before me and came to the conclusion about where the problem was coming from and how to fix it. They suggested what I did above: After doing this you can inspect the file: Lookin good. It no longer reports . I mentioned above that it might be possible to manually edit these files to fix the problem. That would save the time and energy for re-compiling your Rubies. But you definitely want to upgrade your Xcode developer tools and ensure that future ruby installs have the right information. Going through the motions of this full process for at least one Ruby version (assuming you’re using a version switcher like chruby or asdf ) is recommended. Personally, I uninstalled everything to decrease the chances that I have to re-learn about this problem and find this blog post X months/years in the future because I missed something in my process. For those of you without this problem: Hopefully, this was educational. You might be wondering why I decided to blog about this specific topic (of all things). Well, I’ve got to do something while I’m recompiling all those rubies, and learning-via-teaching is a core pedagogy of mine. If you enjoyed this post consider:

0 views
Julia Evans 7 months ago

Standards for ANSI escape codes

Hello! Today I want to talk about ANSI escape codes. For a long time I was vaguely aware of ANSI escape codes (“that’s how you make text red in the terminal and stuff”) but I had no real understanding of where they were supposed to be defined or whether or not there were standards for them. I just had a kind of vague “there be dragons” feeling around them. While learning about the terminal this year, I’ve learned that: So I wanted to put together a list for myself of some standards that exist around escape codes, because I want to know if they have to feel unreliable and frustrating, or if there’s a future where we could all rely on them with more confidence. Have you ever pressed the left arrow key in your terminal and seen ? That’s an escape code! It’s called an “escape code” because the first character is the “escape” character, which is usually written as , , , , or . Escape codes are how your terminal emulator communicates various kinds of information (colours, mouse movement, etc) with programs running in the terminal. There are two kind of escape codes: Now let’s talk about standards! The first standard I found relating to escape codes was ECMA-48 , which was originally published in 1976. ECMA-48 does two things: The formats are extensible, so there’s room for others to define more escape codes in the future. Lots of escape codes that are popular today aren’t defined in ECMA-48: for example it’s pretty common for terminal applications (like vim, htop, or tmux) to support using the mouse, but ECMA-48 doesn’t define escape codes for the mouse. There are a bunch of escape codes that aren’t defined in ECMA-48, for example: I believe (correct me if I’m wrong!) that these and some others came from xterm, are documented in XTerm Control Sequences , and have been widely implemented by other terminal emulators. This list of “what xterm supports” is not a standard exactly, but xterm is extremely influential and so it seems like an important document. In the 80s (and to some extent today, but my understanding is that it was MUCH more dramatic in the 80s) there was a huge amount of variation in what escape codes terminals actually supported. To deal with this, there’s a database of escape codes for various terminals called “terminfo”. It looks like the standard for terminfo is called X/Open Curses , though you need to create an account to view that standard for some reason. It defines the database format as well as a C library interface (“curses”) for accessing the database. For example you can run this bash snippet to see every possible escape code for “clear screen” for all of the different terminals your system knows about: On my system (and probably every system I’ve ever used?), the terminfo database is managed by ncurses. I think it’s interesting that there are two main approaches that applications take to handling ANSI escape codes: Some examples of programs/libraries that take approach #2 (“don’t use terminfo”) include: I got curious about why folks might be moving away from terminfo and I found this very interesting and extremely detailed rant about terminfo from one of the fish maintainers , which argues that: [the terminfo authors] have done a lot of work that, at the time, was extremely important and helpful. My point is that it no longer is. I’m not going to do it justice so I’m not going to summarize it, I think it’s worth reading. I was just talking about the idea that you can use a “common set” of escape codes that will work for most people. But what is that set? Is there any agreement? I really do not know the answer to this at all, but from doing some reading it seems like it’s some combination of: and maybe ultimately “identify the terminal emulators you think your users are going to use most frequently and test in those”, the same way web developers do when deciding which CSS features are okay to use I don’t think there are any resources like Can I use…? or Baseline for the terminal though. (in theory terminfo is supposed to be the “caniuse” for the terminal but it seems like it often takes 10+ years to add new terminal features when people invent them which makes it very limited) I also asked on Mastodon why people found terminfo valuable in 2025 and got a few reasons that made sense to me: The way that ncurses uses the environment variable to decide which escape codes to use reminds me of how webservers used to sometimes use the browser user agent to decide which version of a website to serve. It also seems like it’s had some of the same results – the way iTerm2 reports itself as being “xterm-256color” feels similar to how Safari’s user agent is “Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15”. In both cases the terminal emulator / browser ends up changing its user agent to get around user agent detection that isn’t working well. On the web we ended up deciding that user agent detection was not a good practice and to instead focus on standardization so we can serve the same HTML/CSS to all browsers. I don’t know if the same approach is the future in the terminal though – I think the terminal landscape today is much more fragmented than the web ever was as well as being much less well funded. A few more documents and standards related to escape codes, in no particular order: I sometimes see people saying that the unix terminal is “outdated”, and since I love the terminal so much I’m always curious about what incremental changes might make it feel less “outdated”. Maybe if we had a clearer standards landscape (like we do on the web!) it would be easier for terminal emulator developers to build new features and for authors of terminal applications to more confidently adopt those features so that we can all benefit from them and have a richer experience in the terminal. Obviously standardizing ANSI escape codes is not easy (ECMA-48 was first published almost 50 years ago and we’re still not there!). I don’t even know what all of the challenges are. But the situation with HTML/CSS/JS used to be extremely bad too and now it’s MUCH better, so maybe there’s hope.

0 views
Andre Garzia 8 months ago

Creating a gamebook engine

I made this some time ago but never blogged about it. Unfortunately, I lost some of the source code, but that would be easy to rebuild. I decided to check what were the development options for the Playdate handheld console by Panic after receiving an email from them (I’m on the mailing list for the device). The offering is just too damn polished. Check out Develop for Playdate page. Like everything Panic does, it is damn well done. You can use the SDK to develop using C or Lua or a combination of both. They also offer an web IDE called Pulp that is similar to a pico-8 development workflow with tools for crafting fonts, screens, sprites, audio, and scripting. I went ahead and downloaded the SDK. I already had a license for Nova — which is the fancy development editor they make — and they ship lots of integrations for that editor with the SDK. Everything works out of the box. This is an example source loaded in Nova (using the Playdate theme and the extension). I’m using the editor task system to run the sample in the included emulator. It just works… I might be coding a gamebook engine for the Playdate… that was not in my strategy post, but the muse calls me and it is rude not to answer her call. And this is what my gamebook editor looks like:

0 views

Use Monoids for Construction

There’s a common anti-pattern I see in beginner-to-intermediate Haskell programmers that I wanted to discuss today. It’s the tendency to conceptualize the creation of an object by repeated mutation. Often this takes the form of repeated insertion into an empty container, but comes up under many other guises as well. This anti-pattern isn’t particularly surprising in its prevalence; after all, if you’ve got the usual imperative brainworms, this is just how things get built. The gang of four “builder pattern” is exactly this; you can build an empty object, and setters on such a thing change the state but return the object itself. Thus, you build things by chaning together setter methods: Even if you don’t ascribe to the whole OOP design principle thing, you’re still astronomically likely to think about building data structures like this: To be more concrete, maybe instead of doodads and widgets you have s and s. Or dictionaries and key-value pairs. Or graphs and edges. Anywhere you look, you’ll probably find examples of this sort of code. Maybe you’re thinking to yourself “I’m a hairy-chested functional programmer and I scoff at patterns like these.” That might be true, but perhaps you too are guilty of writing code that looks like: Just because it’s dressed up with functional combinators doesn’t mean you’re not still writing C code. To my eye, the great promise of functional programming is its potential for conceptual clarity, and repeated mutation will always fall short of the mark. The complaint, as usual, is that repeated mutation tells you how to build something, rather than focusing on what it is you’re building. An algorithm cannot be correct in the absence of intention—after all, you must know what you’re trying to accomplish in order to know if you succeeded. What these builder patterns, for loops, and s all have in common is that they are algorithms for strategies for building something. But you’ll notice none of them come with comments. And therefore we can only ever guess at what the original author intended, based on the context of the code we’re looking at. I’m sure this all sounds like splitting hairs, but that’s because the examples so far have been extremely simple. But what about this one? which I found by grepping through for , and then mangled to remove the suggestive variable names. What does this one do? Based solely on the type we can presume it’s using that function to partition the list somehow. But how? And is it correct? We’ll never know—and the function doesn’t even come with any tests! The shift in perspective necessary here is to reconceptualize building-by-repeated-mutation as building-by-combining. Rather than chiseling out the object you want, instead find a way of gluing it together from simple, obviously-correct pieces. The notion of “combining together” should evoke in you a cozy warm fuzzy feeling. Much like being in a secret pillow form. You must come to be one with the monoid. Once you have come to embrace monoids, you will have found inner programming happiness. Monoids are a sacred, safe place, at the fantastic intersection of “overwhelming powerful” and yet “hard to get wrong.” As an amazingly fast recap, a monoid is a collection of three things: some type , some value of that type , and binary operation over that type , subject to a bunch of laws: which is to say, does nothing and doesn’t care where you stick the parentheses. If you’re going to memorize any two particular examples of monoids, it had better be these two: The first says that lists form a monoid under the empty list and concatenation. The second says that products preserve monoids. The list monoid instance is responsible for the semantics of the ordered, “sequency” data structures. That is, if I have some sequential flavor of data structure, its monoid instance should probably satisfy the equation . Sequency data structures are things like lists, vectors, queues, deques, that sort of thing. Data structures where, when you combine them, you assume there is no overlap. The second monoid instance here, over products, is responsible for pretty much all the other data structures. The first thing we can do with it is remember that functions are just really, really big product types, with one “slot” for every value in the domain. We can show an isomorphism between pairs and functions out of booleans, for example: and under this isomorphism, we should thereby expect the instance to agree with . If you generalize this out, you get the following instance: which combines values in the codomain monoidally. We can show the equivalence between this monoid instance and our original product preservation: which is a little proof that our function monoid agrees with the preservation-of-products monoid. The same argument works for any type in the domain of the function, but showing it generically is challenging. Anyway, I digresss. The reason to memorize this instance is that it’s the monoid instance that every data structure is trying to be. Recall that almost all data structures are merely different encodings of functions, designed to make some operations more efficient than they would otherwise be. Don’t believe me? A is an encoding of the function optimized to efficiently query which values map to something. That is to say, it’s a sparse representation of a function. What does all of this look like in practice? Stuff like worrying about is surely programming-in-the-small, which is worth knowing, but isn’t the sort of thing that turns the tides of a successful application. The reason I’ve been harping on about the function and product monoids is that they are compositional. The uninformed programmer will be surprised by just far one can get by composing these things. At work, we need to reduce a tree (+ nonlocal references) into an honest-to-goodness graph. While we’re doing it, we need to collect certain nodes. And the tree has a few constructors which semantically change the scope of their subtrees, so we need to preserve that information as well. It’s actually quite the exercise to sketch out an algorithm that will accomplish all of these goals when you’re thinking about explicit mutation. Our initial attempts at implementing this were clumsy. We’d fold the tree into a graph, adding fake nodes for the construcotrs. Then we’d filter all the nodes in the graph, trying to find the ones we needed to collect. Then we’d do a graph traversal from the root, trying to find these nodes, and propagating their information downstream. Rather amazingly, this implementation kinda sorta worked! But it was slow, and took \(O(10k)\) SLOC to implement. The insight here is that everything we needed to collect was monoidal: where the stanza gives us the semigroup and monoid instances that we’d expect from being the product of a bunch of other monoids. And now for the coup de grace : we hook everything up with the monad. is a chronically slept-on type, because most people seem to think it’s useful only for logging, and, underwhelming at doing logging compared to a real logger type. But the charm is in the details: is a monad whenever is a monoid , which makes it the perfect monad for solving data-structure-creation problems like the one we’ve got in mind. Such a thing gives rise to a few helper functions: each of which is responsible for adding a little piece to the final solution. Our algorithm is thus a function of the type: which traverses the , recursing with a different whenever it comes across a constructor, and calling our helper functions as it goes. At each step of the way, the only thing it needs to return is the root of the section of the graph it just built, which recursing calls can use to break up the problem into inductive pieces. This new implementation is roughly 20x smaller, coming in at @O (500)@ SLOC, and was free of all the bugs we’d been dilligently trying to squash under the previous implementation. Chalk it down to another win for induction!

0 views
Lambda Land 1 years ago

How to Make Racket Go (Almost) As Fast As C

I recently wrote about using first-class functions to help make a BF interpreter . This is a follow-up post to describe a nifty solution to a tricky problem that made my program go 2–5× faster and put it about on-par with an interpreter written in pure C. A basic interpreter works by walking down the AST and evaluating nodes recursively: when the interpreter encounters an expression, it dispatches on the type of expression to decide how to perform the evaluation. Here’s the key insight to get a massive speed bump with very little effort: that dispatch is expensive and can be performed ahead-of-time . We can walk through the code once and precompute all the dispatching work. This is not a new idea. The first description that I’m aware of comes from Feeley and Lapalme [ 1 ]. A name for this technique is making a threaded interpreter . It’s nowhere near as fast as a native code compiler, but interpreters are easy to write, and this is a very simple way to get a very big boost in performance. Please see the appendix for the full code for this interpreter. Tested to run with Racket v8.14 [cs]. Here’s a simple language and a (simplified) interpreter: We can now build and run simple programs: There is nothing particularly special about this interpreter: there’s a basic representation for programs, and the function walks the AST recursively to evaluate the program. To make this interpreter go faster, we need bypass the statement by pre-computing the code to run. We can do this by building up a closure that calls the next thing to run in tail-position. Racket has a proper tail-call optimization, so function calls in tail position will be optimized to instructions and they won’t grow the stack. Having known jump targets is also really good for modern CPUs which do speculative execution; known jump targets means no branch mis-predictions. We do this by breaking up the function: instead of taking an expression and an environment, we want a function that just takes an expression. This should return a function that we can give an environment, which will the compute the value of the program. We will call this new function . Note how the code follows the same structure as the basic interpreter, but the return type has changed: instead of a value, you get a function in the form of . Also, instead of calling on subexpressions, you call , and pass the environment to those subexpressions to get the value out. There’s a lot more that we could here to improve things. The easiest thing would be to track where variables will be in the environment and optimize variable lookups with direct jumps into the environment structure. This saves us from having to walk the environment linearly on every variable lookup. I won’t implement that here, but that’s some pretty low-hanging fruit. So, we get a lot of oomph by turning everything into tail calls. But loops (which, in BF, are the only branching mechanism) present a tricky problem: you either need to call the function that encodes the loop body repeatedly or call the function that encodes whatever comes after the loop. Moreover, once the loop body is done, it needs to jump back to the first function that encodes the choice of whether to keep going in the loop or exit. Here’s my function for my basic threaded interpreter for BF: I match on each of the characters of the program. In my interpreter I do a little bit of preprocessing to turn and into and structs respectively. That way, I don’t have to spend a ton of time scanning the program for matching brackets. The interesting bit is the clause for the construct: to build the closure I need at a , I need to be able to refer to the loop body (computed and stored in ) as well as the end of the loop (computed and stored in ). When I build the closure, pass and (which is the rest of the program after the loop) into the function as the parameter. When the compiler encounters the matching instruction, it uses the parameter to get the and to decide whether or not to redo the loop or not. I’ve since improved this code, and now the check only happens once. I also parse the program so that every instruction gets turned into a struct—rather than left as a bare character. See interp_threaded_opt.rkt in my Brainfreeze repo for the current version. The and functions need to reference each other to be able to continue or abort the loop. lets me build functions that can reference each other in a clean, functional way. So how much faster does threading make the code go? Here is a BF program that renders a Mandelbrot set. I can run this with my basic and my threaded interpreter on my M1 Pro MacBook Pro to get an idea: vs is a big difference! That’s a solid 2× speedup! This actually put my threaded Racket interpreter on par with a C-based threaded interpreter that one of my classmates built and ran with the same benchmarks. If I recall, his was only about a second or two faster. The compile step opens up a big opportunity for optimizations. I’ve been working on some domain-specific optimizations for BF and my interpreter can run the same benchmark in a blazing . (And, of course, none of this really holds a candle to a proper compiler; I’ve to a compiler that takes the optimized code from the threaded interpreter and emits machine code. It can run in a mere !) I hope you take away a few things from this post: Proper tail-calls make stuff go fast. If your language supports proper tail-call optimization, take advantage of it. Racket is really good for writing interpreters and compilers! You can get very fast performance in the comfort of a high-level garbage-collected functional programming language. Racket will never be able to match the best-written C code in terms of speed. But Racket is far easier to debug and a lot more fun to write—and for a lot of applications, Racket is still more than fast enough. A bad day writing code in Scheme is better than a good day writing code in C. David Stigant Proper tail-calls make stuff go fast. If your language supports proper tail-call optimization, take advantage of it. Racket is really good for writing interpreters and compilers! You can get very fast performance in the comfort of a high-level garbage-collected functional programming language.

0 views
fnands 1 years ago

Speeding up CRC-32 calculations in Mojo

In a previous post on parsing PNG images in I very briefly mentioned cyclic redundancy checks, and posted a rather cryptic looking function which I claimed was a bit inefficient. In this post I want to follow up on that a bit and see how we can speed up these calculations. For reference, this post was done with Mojo 24.5, so a few language details have changed since my last post (e.g.  got moved to the top-level and a few of it’s functions have been renamed). I actually wrote most of this post in June with Mojo 24.4, but ran into a bit of an issue which has now been resolved. It even resulted in a new unit test for Mojo, so thanks to Laszlo Kindrat for fixing the issue, and for soraros and martinvuyk for helping figure out what the actual issue was. But first, let’s go through a bit of background so we know what we’re dealing with. CRCs are error detecting codes that are often used to detect corruption of data in digital files, an example of which is PNG files. In the case of PNGs for example the CRC-32 is calculated for the data of each chunk and appended to the end of the chunk, so that the person reading the file can verify whether the data they read was the same as the data that was written. A CRC check technically does “long division in the ring of polynomials of binary coefficients ( )” 😳. It’s not as complicated as it sounds. I found the Wikipedia article on Polynomial long division to be helpful, and if you want an in depth explanation then this post by Kareem Omar does an excellent job of explaining both the concept and implementation considerations. I won’t go deep into the explanations, so I recommend you read at least the first part of Kareem’s post for more background. I pretty much use his post as a guide. Did you read that post? Then welcome back, and well continue from there. But tl;dr: XOR is equivalent to polynomial long division (over a finite field) for binary numbers, and XOR is a very efficient operation to calculate. Essentially what a CRC check does in practice is to run through a sequence of bytes, and iteratively perform a lot of XORs and bit-shifts. By iteratively going through each bit, one can come up with a value that will (nearly certainly) change if data is corrupted somehow. The CRC-32 check from my previous post looked something like this: I’ll step through this in a moment, but the first thing you might notice here is that I am reversing a lot of bits here. The table argument is a bit of future proofing, which we won’t need for now, but will become apparent soon. Just ignore it for now. This is because when I was implementing this function (based off a C example), I implemented a little-endian version of the algorithm, while PNGs are encoded as big-endian. It’s not a huge deal, but it does mean that I am constantly reversing bytes, and then reversing the output again. We can make this better by implementing the big-endian version: This is very similar, and just entails that we use a the reverse of the polynomial we did before (if you bit reverse you get ). This also saves us one 24-bit bit-shift, as we are now working on the bottom 8 bits of the instead of the top 8. Just to verify that these implementations are equivalent, let’s do a quick test: And there we go, a more elegant version of the CRC-32 check I implemented last time. As the theme of today’s post is trying to speed things up, let’s do a little bit of benchmarking to see if this change has saved us any time. As we are doing one fewer bit reverse and bit shift per byte, as well as a final reverse, we should see a bit of a performance uplift. So let’s define a benchmarking function. This function will take two version of the CRC32 function and benchmark their runtimes. We have to a little bit of work first. The functions we test need to return , so we need to wrap our functions in functions with no return value. Note the calls: the compiler will realize that is never used and will compile this away unless you instruct it to keep them: Next we need a test case. I’m not sure if there is a nicer way to fill a List with random values yet, but for now we can list create an , alloc some space, and fill it with random numbers using . Then we can init a using that data: And finally we are ready to benchmark: Nice! So just by avoiding a few bit reversals and bit shifts we get about a 30% uplift in performance depending on the run. Note: I am doing this in a Jupyter notebook, so there is a bit of variance from run to run. While we’re checking performance, let’s see how this implementation would perform in Python: And let’s just do a sanity check to assure ourselves that we produce the same CRC-32 value given the same bytestream: That’s pretty slow in fact this means that: So that’s a nice little speedup we can get by writing essentially the same logic in Mojo. Now this is a bit unfair. If you actually wrote the this function in Python for anything other than educational purposes, then you are using Python wrong, but I’ll get back to how you would actually do it later. Now, the majority of CRC-32 implementations you see will usually use a table. This is called the Sarwate algorithm and exploits the fact that as we are operating on data one byte at a time, there are only 256 unique values that the calculation in the inner loop of the algorithm can take. This is a small enough table (1024 bytes for 256 unsigned 32-bit integers) that it can easily be kept in the L1 cache during computation. Let’s try it out to see how much faster this gets us. First, we need to pre-compute the table: We’ve now effectively amortized the cost of the inner loop and replaced it with a table lookup which should in principle be faster. Our CRC-32 algorithm now take this form: Now we can test it out! So a speedup of around 4-5 times as fast as when we started. This is already a pretty good result, but can we do better? In principle, we could load the data as a 16-bit integer at a time and use the same trick above to build a table. However, such a table would have 65536 entries, resulting in a 256 KB table, which is a lot bigger than the 48 KB of L1 cache on my machine. It turns out, we can still do two bytes at a time . Following the description given by Kareem Omar , we realize that if we have two successive bytes in our message, let’s call them and : if we pad these messages with zeroes What this means is that if we have a CRC algorithm that works on a 16-bit message called M then: For the operation on , leading zeros don’t affect the calculation, so we can just use the 8-bit CRC algorithm we have developed above: The same does not hold for trailing zeroes. However, as will always be of the form and there can only be 256 unique values for , we can build a new table with 256 entries to look up these values. We can build two separate 256 entry tables, or we can just build one 512 entry table, So let’s construct this new table: We have to now update our algorithm to do two table lookups instead of one. Additionally, we have to modify our algorithm to read the data two bytes at a time. So let’s see if this speeds things up: Nice, now we are at an around 7x speedup to where we began. But can we go further? There’s nothing in the above that forces us to only use two bytes at a time. Nothing stops us from doing 4 bytes, i.e. 32-bits at a time with similar logic to above. I’ll quickly create a function that will fill an arbitrarily sized table: Now to create the 4-byte function: And presto, it still works! Let’s see how much faster we are now? A 14 times speedup over where we started! But why stop there? We can in principle explicitly write out version that take 8, 16 or however many bytes at time. This get’s a little long-winded, so I’ll write some generic functions to make functions of arbitrary size. Let’s do a quick sanity check to see if this works: Let’s try and increase the table size as far as we can. We’ll go up in powers of two, and see how far we can go. And there it is. At least on my machine, 32 bytes is the limit, maxing out at a roughly 40 times speedup over the naive implementation. After that we start to see a performance decrease. Let’s plot this to see the trend: As you can see from the above, with a 32-byte table we hit our maximum speedup of around 40 times as fast as the naive implementation. After that, the performance falls off a cliff. Why would this be? If you read the blog post I linked above you already know the answer: Cache. In Kareem Omar’s original post the recommendation is to not go above a 16-byte table, as this will take up approximately half of the standard 32 KB of L1 cache on most computers. However, since this post was written in 2019, L1 cache sizes have increased a bit, and on my machine with a 48 KB L1 cache the 32-byte table performs best, but it’s clear that once you go past that you run into cache issues. This is actually a place then where some compile-time metaprogramming might help: depending on the size of your L1 cache, you might want to compile your application with a 16-byte or a 32-byte table. Or if you want to future proof your application for the (presumable not so distant) future case where the CPUs have at least 128 KB of L1 cache, then you can even add the option of a 64-byte table. At some point Mojo had an autotune functionality that would have been able to do this, but it was removed and we’ll have to wait for it to be added back. Now if you read Kareem’s post, you might realize he went even further by calling some (hardware accelerated) intrinsic functions to go even faster. There is a caveat here in that in that case, the polynomial used is baked into the silicone, and the variation for which this works is called where the C stands for Castagnoli, and importantly this is not the variation that is used for PNG checksums, so I won’t go further with this. Taking our best result from above, we get: So an astounding 500 times speedup over pure Python. Now this is a completely unfair comparison, as I am comparing the naive Python implementation to the optimized Mojo one. Now as I hinted before, no-one in their right mind would write a CRC-32 check like I did above in Python. What one would really do is use zlib from Python’s standard library. In this case we get: So when using Python as you realistically would, Python is actually still more than three times faster than our best effort so far! Of course, the counterpoint to this is that the zlib implementation is done in C, not Python, so we are effectively comparing Mojo to C at this point. The above begs the question however, why is the zlib version still faster than Mojo? What kind of black magic is happening in zlib? Well, this lead me down a rabbit hole and I ended up reading a pretty informative whitepaper by Andrew Kadatch and Bob Jenkins titled Everything We Know About CRC But Afraid To Forget . Now, I can’t find where this was officially published, and the only record of this seems to be in some guy’s github repo. I’m kidding a little on the last point, it’s in the zlib repo which is maintained by Mark Adler . Update: I have been informed that it was originally published as part of the release of crcutil on Google Code . Thanks to jorams on HN who pointed this out. The zlib CRC32 implementation is written in C that has been optimized to within an inch of it’s life, and has so many statement in there that it’s hard to know which way is up. In any case, there is a commit by Mark Adler from 2018 that’s titled Speed up software CRC-32 computation by a factor of 1.5 to 3. . Well, that’s about the amount of performance I am missing, so I guess that’s where I need to start looking. The commit message states: Use the interleaved method of Kadatch and Jenkins in order to make use of pipelined instructions through multiple ALUs in a single core. This also speeds up and simplifies the combination of CRCs, and updates the functions to pre-calculate and use an operator for CRC combination. It’s a pretty large commit and reads about as easily as hieroglyphics, so it might take me a moment to digest what’s going on there. The whole thing that kicked this work off was reading PNG image in Mojo. So how much faster does this improved CRC32 check make reading PNGs? Not much it seems. Reading a pixel PNG image with Mimage is now about 3.5% faster. I suspect the majority of the time reading PNGs is either spent reading from disk or decompressing the actual data. But hey, faster is faster. I’m wondering if/when Mojo will get it’s equivalent of zlib, and what shape that might take.

0 views
Bill Mill 1 years ago

Comparing golang sqlite to C sqlite

Following a question on reddit , I got curious about how well golang sqlite bindings (and translations, and WASM embeddings) compare to using C directly. I'm competent with C, but no expert, so I used the help of an LLM to translate a golang sqlite benchmark to C and tried to make it as equivalent to the golang as possible in terms of the work it was doing. The test I copied is pretty simple: insert a million users into a database, then query the users table and create user structs for each. The result is just one test, but it suggests that all sqlite bindings (or translations) have trouble keeping up with sqlite in C. All times here are in milliseconds, and I picked the best of two runs: The tests were run with go 1.23.1 and clang 16.0.0 on an apple m1 mac with 32gb of ram. This is a deeply non-serious benchmark - I was browsing the web while I was running it, for starters. Still, the magnitude of the results indicates that sqlite in C is still quite a bit faster than any of the alternatives in go. Somebody on mastodon asked me how this compares to previous versions of go, so I did a brief test of mattn and crawshaw on go 1.19, and found that they were in the range of 10-15% slower, so real progress on making cgo faster has been made in a pretty short timeframe. The code is available here in a gist. For kicks, I added a python version to the table above; source is in the same gist 2025 Jun 19 : noticed that the python ratios were not correct and fixed them; I did not re-run any tests

0 views
Ginger Bill 1 years ago

Marketing the Odin Programming Language is Weird

[Originally from a Twitter Thread] Original Twitter Post Odin is a weird programming language to advertise/market for. Odin is very pragmatic in terms of its design and overall philosophy. Unlike all popular languages out there, it has no “killer feature”. I’ve tried to design it to solve actual problems with actual solutions. Those languages with a “killer feature” to them do make them “standout” and consequentially more “hypeable”. The problem is that those “killer features” are usually absolute nonsense, very niche, or they rarely have any big benefit. Hype doesn’t make software better. Odin isn’t a “big idea” language but rather make an alternative to C on modern systems; it tries to solve the problems that other systems languages have failed to address. The problems are usually small but unrelated to each other; not solveable with a single “big idea” 1 . And before people say: “Odin’s ‘killer feature’ is that it has none”, how the heck do you market that? That seems like an anti-marketing feature. There isn’t an example of a popular programming language out there which hasn’t got a “killer feature”, even C’s was that in many way it is a “portable assembly” (even if that is not actually true). I know I have a bad habit when people ask me “why should I use Odin?”: I ask them what their actual needs are, and then tell them to try Odin as well as the other “competition” to see which they prefer. I know that’s honest English politeness to a tee, but that’s me as a man. I want Odin to be as best as it can be, but without trying to sell the world to someone in the process. I want to show people the trade-offs that each design has, even in other languages, and not ignore those trade-offs. There are no solutions, only trade-offs. The lack of “hypeableness” that Odin offers is kind of reflected in the people that seem to be attracted to Odin in the first place. They seem very pragmatic, and just want to get on with programming. And as such, they are the kind of people who don’t even share/hype Odin. I guess I don’t want to easily attract the people who are more driven by hype the pragmatic concerns. I want to make software a better place, and attracting such people is detrimental to that endeavour in the first place. Just look at all the hyped JavaScript frameworks out there, and do they really make better software? Or do they just optimize for a mythical “Developer Experience (DX)” 2 which just results in more crap which gets slower, bulkier, and offer less for the actual user? This is probably why some of my “hot takes” have been doing the rounds every now and then. I am trying to find out what the actual problems are and see what possible options there are to either solve or mitigate them on a case-by-case basis. A lot of those “hot takes” have been a form of marketing, and I am trying to at least give myself some exposure. Every single of them is just my opinion, which I usually think is quite mundane too. The web is huge, and thus there will be people who think those takes are shocking. And to be clear I don’t want to make Odin “hypeable” in the first place. I am glad with the steady, stable, and albeit slow, growth that Odin has been getting. The people who try Odin out, pretty much always stay for the longhaul as they fall in love with the language since it does bring them the “joy of programming” back. Something which I do advertise on the Odin website is the “joy of programming” aspect, but that is something that cannot be explained in words, but rather has to be experienced to believe. Another issue with advertising/marketing a systems-level programming language is that it is niche. It has manual memory management, high control over memory layout, S I M D and SOA support, etc. Great for people who need that level of control, but not needed for the general webdev. Obviously that isn’t the intended audience for Odin, but the problem is that in the social media landscape, those voices are the loudest and many will actually shutdown the voices of people who disagree with them just because they are not in the webdev domain. A minor issue that people are starting to think that Odin is “just” for gamedev. It makes me laugh because gamedev is pretty much the most wide domain possible where you will do virtually every area of programming possible. So “just” is a huge compliment but clearly the wrong image. It’s like saying C++ is “just” for gamedev, when obviously it can be used for anything. The same as Odin, because it’s a systems programming language. Odin does bundle with many game/application oriented packages but they are just that: packages. This is another problem. “Odin” can be thought of in a few different ways: When people speak of Python, they usually think of the entire ecosystem. I’ve worked with people who honestly thought Python was Numpy etc, and that you just had to download them. They had no distinction between any of the concepts, “Python” was just the “tool itself”. Since I am an originally C programmer (and language designer), all of those distinctions are made obviously clear to me. There is no single C compiler, and they are all different. There stdlib is dreadful and you want to replace it with your own thing straightaway. But C still prevails. I make those distinctions because I believe it makes things a lot clearer about programming itself, and helps you understand what the flaws are in the tool; thus know what you can do to mitigate/workaround those issues. But this does require a higher quality standard that than of the norm. Another issue is that Odin is free . As weird as it sounds but since about 20 years ago, it’s nigh impossible to sell a compiler. People expect a programming language and compiler to be free; without caring how much time, money, and effort that goes into building a tool. Odin does have a GitHub Sponsors page ( https://github.com/sponsors/odin-lang/ ) but we make very little, and definitely not enough to pay anyone full-time yet. We will pay for the odd paid work when we have the money, but only for a few weeks here and there. I would love to have a few people working full-time on Odin, but it’s something we cannot afford. It’s also one of the main motivations too: to actually pay people for their work. So I ask you fellow internet users: How the heck do you advertise/market Odin (a systems-level programming language) when it does not have a discernable “killer feature” nor is “hypeable” by its very nature of being a pragmatic language? [This bit is rhetorical; I won’t reply to it]. Name a popular language out here today, and I can name the “killer feature” for that language and why it became popular because of it. People maybe complain about the “feature” after many years but it’s what brought them to it.  ↩︎ Developer Experience (DX) is probably a psyop that makes software even worse; at expense of making the programmer think he is being more productive when in reality he is being less so because the DX is optimizing for that dopamine hit of “felt-productivity” than actual productivity and quality.  ↩︎ The language itself The language+the compiler The language+the+compiler+core library+vendor library The entire ecosystem [This bit is rhetorical; I won’t reply to it]. Name a popular language out here today, and I can name the “killer feature” for that language and why it became popular because of it. People maybe complain about the “feature” after many years but it’s what brought them to it.  ↩︎ Developer Experience (DX) is probably a psyop that makes software even worse; at expense of making the programmer think he is being more productive when in reality he is being less so because the DX is optimizing for that dopamine hit of “felt-productivity” than actual productivity and quality.  ↩︎

0 views